On-the-Fly Adaptation to Quantization: Configuration-Aware LoRA for Efficient Fine-Tuning of Quantized LLMs

The paper introduces CoA-LoRA, a configuration-aware method for adapting LoRA adapters to various quantization settings of large language models without the need for repeated fine-tuning. It utilizes a Pareto-based configuration search to optimize a training configuration set, enabling efficient low-rank adjustments across different bit-widths. This approach significantly reduces computational costs while maintaining or improving performance compared to existing methods that require separate fine-tuning for each quantization configuration, making it valuable for deploying quantized models on edge devices.