dMoE: dLLMs with Learnable Block Experts

The dMoE framework introduces a block-level Mixture-of-Experts (MoE) architecture designed for Diffusion Large Language Models (dLLMs), addressing the inefficiencies of token-level expert selection during parallel decoding. By aggregating token-level expert distributions into a unified block-level distribution, dMoE significantly decreases the number of uniquely activated experts from 69.5 to 14.6, while maintaining 99.11% of original performance. This innovation results in a 76.64% to 79.84% reduction in memory usage and a latency speedup of 1.14× to 1.66×, making it a valuable advancement for practitioners working with large-scale language models.