Beyond LoRA: Is Sparsity-Induced Adaptation Better?

The paper presents new approaches to parameter-efficient fine-tuning (PEFT) by introducing sparsity-induced adaptations of Low-rank Adaptation (LoRA) techniques, specifically Cheap LoRA (cLA) and the chained circulant variant, ${c}^3$LA. These methods aim to improve generalization and reduce training costs by inducing sparsity in low-rank updates, achieving up to 10% reduction in training time and 15% in peak GPU memory usage. The study evaluates 11 fine-tuning methods across various datasets and models, providing theoretical bounds and empirical evidence that these adaptations can maintain competitive performance compared to traditional methods, which is critical for practitioners seeking efficient model training strategies.