From Routing Differences to Candidate Selection in MoE Reasoning
Abstract
Mixture-of-Experts (MoE) routing carries meaningful structure and is increasingly treated as an actionable interface. Targeted expert activation or deactivation can steer model behavior, motivating us to ask whether naturally occurring routing variation also reflects answer correctness. Under stochastic sampling, the same question can yield both correct and incorrect solutions, creating a practical need to identify which candidates are worth selecting or retaining. Using a paired-trajectory design, we compare correct and incorrect candidates for the same question across five MoE models and five reasoning datasets, connecting routing differences, cross-question readout, and candidate selection. Routing differences exceed a matched random-label baseline in 21 of 24 settings after Holm correction. Supervised probes recover correctness-related information on unseen questions, with readout strength varying across models and tasks. Adding routing to candidate scoring improves both Top-1 accuracy and Coverage@2 for four of five DAPO-Math models, including a 7.3 percentage-point Coverage@2 gain for GLM. Retention gains broadly follow relative probe advantages across models and tasks. Mean routing distance offers little additional benefit over text diversity or existing scores and lowers rare-correct retention in all nine evaluated settings. These results establish routing as a source of learnable candidate quality information whose selection value depends on both signal strength and how it is used.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.