FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs

FoMoE introduces a novel approach to training Mixture-of-Experts (MoEs) by partitioning expert layers across distributed workers, allowing for the omission of non-resident experts during local training. This system achieves up to 1.42x reduction in communication costs compared to existing methods and 45.44x compared to Distributed Data Parallelism, while also providing empirical throughput speedups of up to 1.4x. The implications for practitioners include enhanced scalability and efficiency in training large models on loosely connected commodity hardware, particularly for configurations scaling to 100 billion parameters.