Subspace-Constrained Federated Learning with Low-Rank Adaptation

The paper presents a subspace-regularized federated low-rank adaptation (LoRA) method that addresses geometric misalignment in federated learning, which can hinder convergence and aggregation. The proposed method was empirically evaluated on RoBERTa-large and SmolLM-360M models, demonstrating superior performance over FedAvg and other baselines, achieving the highest accuracy and lowest loss metrics, as well as near-perfect basis overlap. This work is significant for practitioners as it enhances the efficiency of fine-tuning large models in federated settings, particularly under conditions of data heterogeneity.