FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRA

FedRot-LoRA is a new framework designed to enhance Federated LoRA by addressing the issue of rotational misalignment in local updates during model fine-tuning. It utilizes orthogonal transformations to align client updates prior to aggregation, mitigating aggregation errors without increasing communication costs or limiting model expressivity. Experimental results show that FedRot-LoRA significantly outperforms existing federated LoRA methods across various tasks and levels of data heterogeneity, making it a valuable advancement for practitioners working with decentralized training of large language models.