Which LoRA? An Empirical Study on the Effectiveness of LoRA Techniques During Multilingual Instruction Tuning

The study published in arXiv investigates the effectiveness of various LoRA variants in multilingual instruction tuning, revealing no significant advantages of more complex LoRA techniques over basic LoRA. Experiments conducted on two datasets across diverse languages indicate that layer-wise language representations remain consistent across models fine-tuned with different LoRA methods. This finding suggests that practitioners may not need to adopt more complex LoRA variants for improving cross-lingual transfer and knowledge retention in multilingual tasks.