Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments

Mosaic is a new data-free knowledge distillation framework designed for heterogeneous distributed environments in Federated Learning (FL). It employs local generative models to create synthetic data, facilitating the formation of a Mixture-of-Experts (MoE) architecture that consolidates client models into a robust global model. Experimental results indicate that Mosaic outperforms existing methods on standard image and multimodal benchmarks, addressing challenges of model and data heterogeneity, which is critical for practitioners aiming to enhance FL performance while maintaining data privacy.