Decoupled Mixture-of-Experts for Parametric Knowledge Injection

The paper introduces Decoupled Mixture-of-Experts (DMoE), a new modular architecture designed for parametric knowledge injection into large language models (LLMs). DMoE separates expert modules and the routing mechanism from the base model, allowing for independent updates of knowledge without risking catastrophic forgetting. Experimental results demonstrate that DMoE outperforms traditional retrieval-augmented generation and adapter-based methods on knowledge-intensive benchmarks, making it a promising approach for enhancing LLMs with external knowledge while maintaining efficient inference.