Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMs

The article presents Attention-Spectrum Regularization (ASR), a novel framework for replay-free continual learning in multimodal large language models (MLLMs) that preserves skill-conditioned structures of cross-modal attention. ASR utilizes spectral statistics of cross-attention maps to maintain prototypes of skills, effectively controlling the drift of these prototypes during adaptation to new tasks. Experimental results on benchmarks such as VQA v2 and CoIN demonstrate that ASR enhances performance and reduces forgetting compared to existing methods, making it a lightweight solution for practitioners working with continual learning in MLLMs.