acceptodds
Under review as a conference paper at ICLR 2027

Does the Same Token Mean the Same State? MoE Routing as Signal for Reasoning Control

Abstract

In sparse Mixture-of-Experts language models, does the same token id imply the same router state and the same experts producing it? Holding the emitted token id fixed at repeated anchors, we find it does not: the experts that produce it still separate task context, trajectory history, and reasoning-effort mode. This residual structure supports test-time control: near *boundary* anchors (the final-response transition) and *delimiter* anchors (which open the answer, e.g. `\boxed{` or code fences), routing neighborhoods already align with final-answer basins at a marker-only readout and strongest when the routing is read at the answer opening. We operationalize this as **RAD (Routing Agreement Decoding)**, an answer-string-free multi-rollout selector: it locates a fixed anchor, represents each rollout by its anchor-window MoE routing states, and returns the densest Weighted-Jaccard *K*-NN route-basin center, without parsing, normalizing, executing, or voting over answer strings. Across 10 sparse-MoE configurations (gpt-oss, Qwen3-MoE) and 6 datasets spanning math, GPQA, and code, RAD is on par with Majority where string voting is well-posed, with small positive paired deltas (RAD 73.9 / RAD+DC 74.2 vs. Majority 73.6). Like majority voting, RAD is not a verifier: a dense *wrong* basin can still win. Its value is the interface: the same selector gives direct pass@1 on code, where exact-string voting is ill-defined, and the same routing-density principle, re-anchored to the agentic boundary, improves best-of-16 patch selection on SWE-bench Verified over random, where patches have no answer string to vote on.

Primary area:
interpretability and explainable AI
Keywords:
Mixture-of-Experts, Sparse MoE, Router States, Test-Time Reasoning, Routing Agreement Decoding, Multi-Rollout Selection, Reasoning Control, Model Routing, SWE-bench Verified, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_woWbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents that in MoE models, the same token id can be generated by different expert routes, and that routing near the answer-opening position aligns with final-answer clusters. Based on this, it proposes RAD: each of N=64 rollouts is represented by its averaged routing over a 16-token window after an anchor (e.g., \boxed{), and the rollout with the highest Weighted-Jaccard K-NN density is selected, without reading answer strings. RAD matches textual Majority on math/GPQA, gives above-random pass@1 on LiveCodeBench, and improves SWE-bench Verified best-of-16 selection.

Critical strengths

The “same token, different experts” analysis is clean, and is a useful observation about MoE routing in its own right.

Critical weaknesses
  • On math and GPQA, RAD performs about the same as majority voting (73.9% vs. 73.6%, not significant). It is slightly worse with fewer rollouts (m ≤ 32), and the version that reads no answer tokens drops to 71.1%. The authors are upfront about this, but it means the method does not beat the simplest baseline in its main setting.
  • The paper argues that RAD is most useful for code and SWE-bench, where majority voting does not work. However, in these settings it is only compared with FMV and random selection. I would expect simpler baselines that also do not need answer strings, such as picking the densest output by text embedding or edit distance, Universal Self-Consistency, or an LLM judge. Without these, I cannot tell whether routing gives anything beyond plain output similarity.
  • I am also not convinced that consensus is the right tool for code. In practice, people sample more to raise pass@k and then pick a solution by running tests or using a verifier. The paper’s own composition control shows RAD does worse than random when only a few rollouts are correct, which is the usual case on hard coding problems. Its gains on code are also much smaller than on math.
  • Several prior works already use MoE routing at test time, such as RICE (https://arxiv.org/abs/2505.14681), C3PO (https://arxiv.org/abs/2504.07964), SCMoE (https://arxiv.org/abs/2405.14507), and R2-T2 (https://arxiv.org/abs/2502.20395), and MoEE (https://arxiv.org/abs/2410.10814) uses routing weights as embeddings. These methods add some inference cost, but they work on a single generation and need no answer anchor, so they apply more widely. The paper does not discuss or compare with any of them.
  • RAD only works after all rollouts are finished. The authors show that reading routing from the first 32 to 256 tokens is close to random, so it cannot be used for early stopping or pruning like many other test-time methods in the related work. The claim that RAD is “free” therefore only covers the selection step, while the main cost, generating 64 full rollouts, stays the same.
Questions for authors
  • How does RAD compare with prior routing-based test-time methods (e.g., SCMoE, RICE, C3PO) and with using routing weights as embeddings (MoEE)? Could routing-intervention methods be combined with RAD?
  • Can the main tables report Δsel instead of lift over Avg@64, to separate genuine selection from the anchor-completion filter?
  • The answer-string-free interface should matter most for open-ended generation (e.g., long-form QA, summarization, story writing), yet these are not evaluated. Does RAD work there, given there is no natural answer-opening anchor and no single “answer basin”? For creative tasks, would selecting the densest (most typical) rollout actually hurt quality or diversity
2 comments

We thank the reviewer for the thoughtful and constructive assessment, particularly for recognizing the value of the "same token, different experts" observation. To clarify:

1. RAD is not intended to outperform Majority or replace execution-based verification.

Our central question is whether native MoE routing decisions can serve as a useful agreement signal for selecting completed rollouts without comparing their answer strings. The comparable performance to textual Majority (73.9% vs. 73.6%, not statistically significant) is evidence that this sparse internal signal can recover much of the consensus structure accessible through explicit answer grouping, rather than a claim of superior accuracy.

Importantly, RAD can complement execution-based selection rather than compete with it. Appendix A.4 (Table 6) already reports that applying RAD within FMV's winning execution-equivalence class improves accuracy from 71.88% to 73.68% (+1.80 pp, exact McNemar p=0.0084) on the same 1,554 covered model–problem cells, without additional test executions.

This result is particularly relevant to the reviewer's practical concern: even after execution-based consensus has been established, routing agreement can provide an additional selection signal among candidates indistinguishable under the generated tests.

2. The contribution is a distinct use of routing information, not the first discovery that routing is informative.

We appreciate the references to RICE, C3PO, SCMoE, R2-T2, and MoEE, and agree that these should be discussed explicitly.

Our distinction is operational and methodological: several of these methods modify routing or output distributions during generation, whereas RAD uses the routing decisions already produced by completed rollouts, without modifying the model or generation process. MoEE is especially relevant as prior work demonstrating that routing weights can serve as representations, and we do not claim that routing-based representations are new.

Our specific contribution is to establish and characterize anchor-conditioned routing agreement as a candidate-selection signal, including where this signal emerges, its relationship to answer basins, and its systematic failure under incorrect consensus.

We also agree that text embeddings, edit-distance selection, and LLM judges are relevant alternatives. Their omission limits claims of comparative superiority, and we will make this explicit rather than imply that routing necessarily outperforms textual similarity. Nevertheless, this does not invalidate the demonstrated routing–answer-basin relationship or the existing selection results. We view the comparison as important for establishing relative advantages, not as a prerequisite for the narrower empirical finding that sparse routing supports selection.

3. The limitations are genuine, but they define the scope rather than contradict the contribution.

We agree that RAD is neither an early-stopping method nor a correctness verifier. It selects from completed rollout pools, and its effectiveness depends on the structure of the candidate distribution.

Indeed, our composition analysis (Fig. 4, Appendix A.15) explicitly shows that routing consensus can fail when correct candidates are rare. We regard this as a substantive characterization of what routing agreement measures: shared solution-basin structure rather than independent correctness.

Regarding the anchor filter, Appendices A.9 and A.11 already separate the eligibility contribution from genuine selection gains through

Δsel=AccRAD−Avg@cohort.\Delta_{\mathrm{sel}}=\mathrm{Acc}_{\mathrm{RAD}}-\mathrm{Avg@cohort}.

We agree that this distinction should be more prominent in the main paper and will revise the presentation accordingly.

We will also replace potentially misleading descriptions of RAD as "free" with the more precise statement that selection requires no additional model forward pass, while explicitly acknowledging the approximately 7% routing-capture overhead reported in Appendix A.5.

Finally, Appendix A.7 already examines EOS-aligned routing, which achieves 70.37% versus 71.32% for marker-based RAD on the evaluated pool. This suggests that the readout is not intrinsically restricted to a particular answer delimiter. However, we agree that effectiveness on long-form and creative generation remains unestablished and will state this boundary clearly.

We appreciate the reviewer's suggestions and will strengthen the related-work discussion, emphasize the existing selection-gain decomposition and FMV complementarity results, and clarify the computational and applicability boundaries.

We hope these clarifications help establish that the paper makes a meaningful contribution as an empirical study and operationalization of MoE routing agreement, even without claiming universal superiority over alternative selectors. We would be grateful if the reviewer could reconsider the assessment in light of this more precise scope.

Reviewer_woWbOpenReview ↗

Final Decision

Thank you for the response. The rebuttal narrows the claim rather than addressing my concerns, and no new experiments are provided. The comparison with text embedding or edit-distance selection could be run directly on the existing rollouts at low cost, and it is central to whether routing offers anything beyond output similarity. Without it, the FMV complementarity result also cannot show that routing is needed, since a text-similarity tie-breaker might give the same gain. With the claim reduced to “routing supports selection” and RAD matching but not exceeding majority voting, the contribution is mainly an observation rather than a method. I keep my score and support rejection.

Reviewer_SsWiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Routing Agreement Decoding (RAD), a training-free method for selecting among completed outputs of sparse MoE models. RAD compares expert-routing patterns near aligned response markers and selects the candidate with the strongest local agreement. Experiments on mathematics, GPQA, code generation, and software repair show performance close to textual majority voting on canonical-answer tasks and improvements over random selection on code and patches. Additional analyses examine where the routing signal is strongest and when it follows an incorrect consensus.

Critical strengths

S1. Reusing expert-routing decisions for candidate selection is a practical idea. The method requires no additional model forward pass and can operate when candidate outputs are difficult to group by their final answer strings.

S2. The evaluation covers multiple MoE architectures and several task types. The marker-only results provide useful evidence that the selection signal is already present before the final answer region begins.

S3. The paper examines anchor position, window size, layer depth, routing weights, candidate count, and incorrect consensus. These experiments help explain both the effectiveness and the limitations of routing agreement.

Critical weaknesses

W1. RAD adopts NAD’s agreement rule, making the routing representation its main methodological change. The evaluation therefore needs a direct comparison with NAD or a simple hidden-state selector on the same candidates and anchors, reporting accuracy, latency, and memory use.

W2. The SWE-bench baselines are limited to random selection and step-count heuristics. Confidence-based and patch-similarity selectors would provide stronger comparisons on the same candidate pools.

W3. Counting FMV’s abstentions as failures drives the reported parity on code. On the shared covered subset, FMV leads RAD by 3.8 points (Table 6). Including a fallback for uncovered cases and total selection costs would clarify the practical tradeoff.

Questions for authors

Q1. How does RAD compare with NAD or a pooled hidden-state selector when the candidate pool, anchor positions, and readout window are matched? Please report selection accuracy and measured overhead.

Q2. On SWE-bench, how does RAD compare with token-confidence selection and consensus based on patch text or changed-line overlap? Are the gains stable under problem-level resampling?

2 comments

We thank the reviewer for recognizing the practical routing interface, the marker-only evidence, and the analyses of where routing agreement succeeds and fails. We would like to clarify the contribution supported by the existing results and revise the statements whose scope is broader than the evidence.

W1/Q1 — Contribution beyond NAD and the role of hidden-state comparisons.

RAD deliberately adopts NAD’s local agreement rule, as stated in Section 2. Our contribution is the empirical characterization of native MoE routing as a consensus readout: whether it retains useful structure after fixing emitted token identity, where that structure becomes useful for selection, and which parts of the routing record are needed.

The existing evidence includes token-controlled analyses (§3.2), readout-position controls (§3.3), and strictly marker-only selection (Appendix A.2). In particular, marker-only RAD reaches 71.1% on math+GPQA against the 66.3% random-selection baseline (Table 3), without including answer-region tokens in its readout. The kernel and layer-depth ablations further characterize the representation (Appendices A.13–A.14).

We agree that a matched NAD/hidden-state comparison would be needed to establish comparative accuracy or efficiency superiority; that comparison is not available in the current paper. We will restrict the corresponding claims accordingly. Appendix A.5 reports approximately 7% routing-capture overhead and selection in tens of milliseconds on a CPU thread. These are measurements of our implementation, not evidence of a head-to-head advantage over hidden-state selection. The supported contribution is a characterized and operationally useful routing readout.

W2/Q2 — What the SWE-bench results establish.

We agree that patch-similarity consensus is a meaningful baseline even when exact-string voting degenerates. The step-count controls address trajectory length and do not replace confidence-based or patch-similarity comparisons.

The existing SWE-bench results show that the routing-density principle applies to multi-step agent trajectories: on all 500 problems per model, RAD improves over Avg@16 by 2.2, 2.4, and 2.9 percentage points across the three evaluated backbones (Table 2). We will describe these as evidence of applicability to agentic patch selection, without claiming superiority over the additional selectors suggested by the reviewer. Problem-level resampling stability has not been established for this experiment; the positive gains across the three fixed candidate pools should be interpreted within that scope.

W3 — FMV coverage, parity, and complementary selection information.

We agree with the reviewer’s interpretation of Table 6. On the common covered subset, FMV achieves 71.88% versus RAD’s 68.08%. The near-equality on the full set depends on counting FMV abstentions as failures. We will replace the unqualified “on par with FMV” wording with this explicit comparison and will not extend it to FMV with a fallback or to matched total-cost performance.

The existing FMV+RAD result provides a separate finding: covered accuracy increases from 71.88% to 73.68% at exactly the same coverage. This method applies RAD within the candidate set sharing FMV’s selected execution vector; it does not use RAD as an abstention fallback (Appendix A.4). The improvement therefore demonstrates useful additional selection information within an execution-consensus class, independently of abstention handling.

Together, the results support native routing as a useful consensus signal, identify its dependence on readout position and its failure under incorrect consensus, and demonstrate applications to both direct and agentic candidate selection.

We would appreciate reconsideration of the overall assessment in light of this clarified contribution and the evidence already reported. Our revisions will make the empirical scope and practical tradeoffs explicit, while preserving the central finding that aligned MoE routing supports useful candidate selection.

Thank you for the clarifications. While these address some of my concerns, I still have some reservations and will maintain my rating of 2.

Reviewer_iQ4SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether expert-routing decisions produced during MoE inference can select among multiple candidate solutions. Routing Agreement Decoding (RAD) aggregates routing states over a short window at a fixed anchor, such as an answer-opening delimiter, and selects the candidate with the highest local kNN agreement under weighted Jaccard similarity. It does not compare normalized answer strings or execute candidate programs.

Routing retains contextual information even when the emitted token is held fixed, and neighborhoods near answer openings align with final-answer classes. On mathematics and GPQA, RAD is comparable to majority voting: 73.9% versus 73.6% pooled accuracy, with no significant difference. It improves over random selection on LiveCodeBench and SWE-bench Verified. Readout ablations, eligibility-gain decomposition, and candidate-composition controls show that routing agreement captures consensus rather than correctness.

Critical strengths
  • S1: RAD reuses sparse routing decisions already computed during MoE inference. It needs neither an additional selector network nor candidate execution at selection time, which is useful for outputs that are difficult to canonicalize or settings where sandbox execution is undesirable.

  • S2: The paper rigorously tests both effectiveness and failure conditions. Random-position controls, anchor readouts, marker-only ablations, eligibility-gain decomposition (Δanchor\Delta_{\text{anchor}} vs. Δsel\Delta_{\text{sel}}), and controlled correct-candidate fractions explain when the method works and why it fails. The false-consensus analysis is especially valuable because it provides mechanistic insight beyond headline accuracy tables.

Critical weaknesses
  • W1: No matched anchor-window comparison against internal-representation baselines. NAD already combines internal activations, Jaccard similarity, and kNN agreement. RAD substitutes MoE routing and restricts readouts to a marker-only or 16-token anchor window, but never compares against NAD neuron sets or last-layer hidden states at the same anchor positions (Section 6). Because the reported storage savings (Appendix A.5) come primarily from reading a 16-token window rather than the full trace, pooling hidden states or top-kk neurons over the same window would be similarly lightweight while also applying to dense models. A head-to-head comparison at matched anchor windows is needed to isolate the value of MoE routing from anchor alignment.

  • W2: Weak code and SWE-bench evidence due to eligibility filtering, unmitigated abstentions, and missing baselines. (1) In Table 20, LiveCodeBench gains over Avg@64 come largely from mechanical eligibility filtering (Δanchor\Delta_{\text{anchor}}, emitting a code fence); pure routing selection (Δsel\Delta_{\text{sel}}) is near zero or negative on all four Qwen configurations (−1.6,+0.9,+0.6,−0.5-1.6, +0.9, +0.6, -0.5 pp). (2) In Table 6, RAD (66.71%) appears level with execution-based FMV (66.89%) on all 1,670 cells only because FMV's 116 abstentions (6.95%) are scored as 0. On the 1,554 covered cells, FMV leads by +3.80 pp (p=3.8×10−5p=3.8\times 10^{-5}), and a simple RAD fallback on abstentions lifts FMV to ~70.2% on the full set. (3) On SWE-bench Verified (Table 2, Table 13), RAD is compared only against random selection (Avg@16) and step count, omitting execution-free confidence baselines (DeepConf-argmax, RAD+DC) evaluated elsewhere in the paper.

Questions for authors
  • Q1 (Matched internal-representation baselines): Could the authors compare RAD, NAD neuron sets, and simple hidden-state cosine/kNN similarity on one representative MoE model and the same candidate pool, using the exact same anchor positions and window widths (delimiter@marker and delimiter@16)? Please report accuracy alongside capture, storage, and selection costs to isolate the contribution of MoE routing from anchor-window alignment.

  • Q2 (FMV coverage and fallback): Why are the 116 uncovered FMV cells in Table 6 scored as incorrect rather than falling back to random selection or RAD? Given that FMV with a RAD fallback on abstentions (FMV on the 1,554 covered cells + RAD on the 116 uncovered cells) would reach ~70.2% vs. RAD's 66.71% on the full denominator, how should we interpret the claim that RAD is "level with execution-based functional majority voting"?

  • Q3 (SWE-bench baselines and statistical testing): On the SWE-bench Verified pools, what are the problem-level paired bootstrap confidence intervals and McNemar p-values for the +2.2+2.2 to +2.9+2.9 pp end-to-end gains? On the same 16-rollout pools, how does RAD compare against execution-free confidence selection (DeepConf-argmax / RAD+DC, which requires only token logprobs already produced during generation) or trajectory-embedding similarity baselines?

1 comment

We thank the reviewer for the thoughtful assessment and for recognizing the value of reusing MoE routing signals and systematically analyzing their failure conditions. We agree that several comparisons deserve clearer qualification. We would like to clarify why these limitations do not undermine the central empirical contribution.

W1 / Q1: Comparison with NAD and Hidden States

We agree that matched anchor-window baselines would be valuable for determining the relative effectiveness of different internal representations.

However, RAD does not claim that routing is more informative than neuron activations or hidden states. Our central question is whether the sparse routing decisions already computed during MoE inference are sufficient to support meaningful consensus selection.

Section 6 explicitly acknowledges the absence of matched representation baselines. The contribution is therefore operational sufficiency, not representational superiority.

The existing experiments independently establish that:

  1. Identical emitted tokens retain distinct contextual routing states (Section 3.2).
  2. Answer-opening routing neighborhoods exhibit substantially stronger answer-basin alignment than random-position controls (Figure 2).
  3. Marker-only routing already contains a meaningful consensus signal without conditioning on answer-region tokens (Appendix A.2).

We also agree that the storage reduction in Appendix A.5 is relative to full-token routing capture, not windowed hidden-state representations, and will clarify this distinction.

Thus, the requested comparison would establish the relative merits of routing against other representations, rather than invalidate the demonstrated existence and usefulness of the routing-agreement signal.

W2 / Q2: Eligibility Filtering and FMV

We agree that eligibility filtering contributes to the LiveCodeBench gains. Appendix Table 20 explicitly decomposes the total improvement into mechanical eligibility gain and routing-only selection gain.

Importantly, all six gpt-oss configurations exhibit positive routing-only gains of +2.9 to +6.9 pp, measured against random selection within the identical eligible candidate pool. Thus, eligibility filtering does not explain all observed improvements. We acknowledge that the Qwen results are substantially weaker and do not claim universal improvements.

Regarding FMV, we agree that treating abstentions as failures does not represent the strongest execution-based pipeline. The reviewer's proposed FMV fallback would achieve approximately 70.2% according to the reported Table 6 aggregates, outperforming standalone RAD. We will remove the overly broad suggestion of parity with execution-based selection.

Nevertheless, the existing FMV+RAD experiment demonstrates complementary information: RAD reranking within FMV's selected execution-equivalence class improves covered accuracy from 71.88% to 73.68% (+1.80 pp; reported exact McNemar p=0.0084).

The relevant contribution is therefore not superiority over execution-based methods, but an execution-free consensus signal that can also complement execution-derived information.

Q3: SWE-bench Baselines and Statistical Testing

We agree that DeepConf-argmax, RAD+DC, and problem-level paired uncertainty estimates would strengthen the SWE-bench comparison.

Our intended claim is narrower than superiority over these unreported baselines. The SWE-bench experiments investigate whether the routing-agreement principle extends to multi-step agentic trajectories, where exact-string voting over patches is ineffective.

Using the same boundary-based selection rule without per-model tuning, RAD produces positive end-to-end point estimates of +2.2, +2.4, and +2.9 pp over Avg@16 across three distinct backbones (Table 2).

Appendix A.10 further examines trajectory-length confounding, showing that step-count-only selection does not explain the observed gains.

We agree that these modest improvements should not be presented as statistically established superiority over alternative confidence-based selectors without the requested comparisons. We will clarify that the SWE results provide initial evidence of transferability.

Summary

We appreciate the reviewer's suggestions and will sharpen the paper's claims accordingly.

Our contribution is not a new agreement objective, a correctness verifier, or proof that routing dominates alternative internal representations. Rather, it is the identification and systematic characterization of native MoE routing as an actionable, answer-string-free consensus signal, including controlled evidence of when it works and when false consensus causes failure.

We hope these clarifications address the concerns about the scope and interpretation of our claims, and that the demonstrated empirical findings can be evaluated independently of the additional comparisons needed to establish relative superiority.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.