Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMs

The paper critiques the MR-LoRA method for continual learning in multimodal large language models (MLLMs), asserting that routing does not necessitate an MLLM and can be effectively achieved with a training-free, replay-free prototypical routing method called RePRo, which operates at a significantly lower computational cost. It also argues that shared experts do not enhance continual learning due to structural limitations in the MLLM-CL benchmark, which fails to account for task separability and fixed task orders, leading to a need for redesigned benchmarks that better reflect genuine continual transfer across diverse learning scenarios. This has implications for practitioners by highlighting the need for more robust evaluation metrics in continual learning for MLLMs.