PreLort: Prefix-Nested LoRA for Federated Fine-Tuning under Rank Heterogeneity

The article introduces PreLort, a novel approach for federated fine-tuning of large language models using a prefix-nested low-rank adaptation (LoRA) strategy. This method addresses the challenges posed by heterogeneous hardware in federated settings by organizing adapter dimensions into a prefix hierarchy, enabling effective aggregation of task-relevant information while maintaining performance across varying adapter ranks. Experimental results indicate that PreLort outperforms existing federated LoRA methods in accuracy and ROUGE-L metrics, providing a more efficient use of shared low-rank representations, which is crucial for practitioners working with federated learning and model adaptation in diverse environments.