LAYUP: Asynchronous decentralized gradient descent with LAYer-wise UPdates

LAYUP introduces an asynchronous decentralized stochastic gradient descent (SGD) method with layer-wise updates, designed to mitigate the communication overhead associated with synchronous, centralized training. By utilizing randomized gossip communication, LAYUP allows for immediate application of layer-wise updates during backpropagation, achieving convergence up to 32% faster than traditional synchronous data parallel training and 27% faster than existing communication-efficient algorithms, while improving robustness against stragglers. This approach enhances model FLOPs utilization and provides a viable solution for practitioners seeking efficient distributed training without compromising accuracy.