Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformers

The article presents a new framework for scaling linear mode connectivity (LMC) to facilitate the merging of billion-parameter pretrained transformers. It introduces a dual learning procedure that allows both models to optimize their weight transformations toward a shared interpolation path, achieving near-zero loss barriers on language model benchmarks and maintaining high accuracy in vision tasks like ImageNet classification. This advancement is significant for practitioners as it enhances the reliability of model merging, potentially improving the performance and efficiency of large-scale AI systems.