Toward Calibrated Mixture-of-Experts Under Distribution Shift

The paper presents a study on the calibration of mixture-of-experts (MoE) models under distribution shift, revealing that while expert-level calibration ensures overall model calibration in hard-routed models, it fails in soft-routed models. The authors propose an adversarial reweighting method to penalize calibration errors, enhancing the accuracy-calibration tradeoff across various model classes and tasks. This research is significant for practitioners as it provides insights into improving MoE model performance in dynamic environments, critical for applications relying on accurate probabilistic predictions.