Unifying Local Communications and Local Updates for LLM Pretraining

The paper introduces GASLoC, a decentralized pre-training algorithm for large language models (LLMs) that enhances communication efficiency by allowing local optimizer steps and utilizing gossip-based peer communication. It demonstrates superior performance over existing decentralized methods, particularly in heterogeneous bandwidth scenarios, and achieves competitive results with DiLoCo while enabling multiple local updates. This advancement is significant for practitioners as it optimizes LLM training across distributed environments, alleviating bottlenecks associated with synchronous All-Reduce operations.