Routing-Aware Expert Calibration for Machine Unlearning in Mixture-of-Experts Language Models

The paper presents TRACE (Targeted Routing-Aware Calibration of Experts), a novel approach to machine unlearning in Mixture-of-Experts (MoE) language models. TRACE addresses the routing mismatch between forget and retain data by reweighting token-level retain losses to ensure better calibration of forget-critical experts, resulting in a 9% relative utility improvement over existing methods on benchmarks like WMDP and MUSE-BOOKS. This advancement is significant for practitioners as it enhances the efficiency of unlearning processes in MoE architectures, which is crucial for maintaining model integrity and compliance with data privacy regulations.