Sparse Neuron Ablation Triggers Catastrophic Collapse of the Language Core in Large Vision-Language Models

The article presents a method called Consistently Activated Neurons (CAN) for neuron ablation in Large Vision-Language Models (LVLMs) to identify critical neurons whose removal leads to catastrophic performance collapse. Experiments with 7B parameter models like \texttt{LLaVA-1.5-7b-hf} and \texttt{InstructBLIP-vicuna-7b} demonstrate that removing as few as four neurons can trigger significant degradation, revealing that critical neurons are primarily located in the language model's down-projection layer. This research highlights the vulnerability of LVLMs and underscores the importance of understanding the structural dependencies within these models for practitioners aiming to enhance their robustness.