LoRDO: Distributed Low-Rank Optimization with Infrequent Communication

LoRDO is a novel framework that integrates low-rank optimization with infrequent communication to enhance distributed training of foundation models, addressing bandwidth limitations in data parallelism. The method achieves near-equivalent performance to low-rank Distributed Data Parallel (DDP) on language modeling tasks with model sizes ranging from 125M to 720M, while reducing communication overhead by approximately 10 times. This approach is particularly beneficial for practitioners working with low-memory environments, as it allows for effective optimization without the full-batch gradient access typically required for low-rank methods.