Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation

The paper presents a function-space theory for understanding catastrophic forgetting in continual adaptation, focusing on the behavior of neural tangent kernels (NTK). It introduces a closed-form predictor for the forgetting vector that operates in the NTK regime, demonstrating that forgetting is concentrated in a few NTK eigenmodes, particularly in scenarios involving frozen-backbone linear heads. This insight is crucial for practitioners as it suggests the need for targeted spectral regularization to mitigate forgetting, enhancing model performance in continual learning tasks.