Can Scale Save Us From Plasticity Loss in Large Language Models?

The study investigates plasticity loss in GPT-style Transformer models, focusing on their ability to adapt to new information after prior learning. Analyzed models ranged from 5M to 314M parameters, revealing that plasticity loss occurs even in larger architectures and follows a sublinear scaling law with model size. These findings indicate that while larger models may mitigate the effects of plasticity loss, simply increasing parameter count is insufficient to prevent this issue, impacting the design of continual learning systems in natural language processing.