SDFLoRA: Selective Decoupled Federated LoRA for Privacy-preserving Fine-tuning with Heterogeneous Clients

The article presents Selective Decoupled Federated LoRA (SDFLoRA), a novel framework designed for privacy-preserving fine-tuning of large language models in federated learning environments. SDFLoRA addresses challenges related to rank and data heterogeneity by decoupling client updates into shared and private components, allowing only the shared component to participate in aggregation while maintaining local semantics in the private component. Experimental results indicate that SDFLoRA significantly improves the utility-privacy trade-off compared to existing federated LoRA methods, making it a valuable approach for practitioners dealing with heterogeneous client data in federated learning scenarios.