Does the Same Token Mean the Same State? MoE Routing as Signal for Reasoning Control

The paper introduces Routing Agreement Decoding (RAD), a novel approach for controlling reasoning in sparse Mixture-of-Experts (MoE) language models by leveraging router states associated with fixed token anchors. RAD operates without parsing or evaluating answer strings, instead using anchor-window MoE routing states to identify the densest route-basin center, achieving competitive performance compared to traditional majority voting across various configurations and datasets. This method is particularly beneficial for tasks where exact-string voting is challenging, enhancing selection processes in code generation and other applications.