- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents that in MoE models, the same token id can be generated by different expert routes, and that routing near the answer-opening position aligns with final-answer clusters. Based on this, it proposes RAD: each of N=64 rollouts is represented by its averaged routing over a 16-token window after an anchor (e.g., \boxed{), and the rollout with the highest Weighted-Jaccard K-NN density is selected, without reading answer strings. RAD matches textual Majority on math/GPQA, gives above-random pass@1 on LiveCodeBench, and improves SWE-bench Verified best-of-16 selection.
Critical strengths
The “same token, different experts” analysis is clean, and is a useful observation about MoE routing in its own right.
Critical weaknesses
- On math and GPQA, RAD performs about the same as majority voting (73.9% vs. 73.6%, not significant). It is slightly worse with fewer rollouts (m ≤ 32), and the version that reads no answer tokens drops to 71.1%. The authors are upfront about this, but it means the method does not beat the simplest baseline in its main setting.
- The paper argues that RAD is most useful for code and SWE-bench, where majority voting does not work. However, in these settings it is only compared with FMV and random selection. I would expect simpler baselines that also do not need answer strings, such as picking the densest output by text embedding or edit distance, Universal Self-Consistency, or an LLM judge. Without these, I cannot tell whether routing gives anything beyond plain output similarity.
- I am also not convinced that consensus is the right tool for code. In practice, people sample more to raise pass@k and then pick a solution by running tests or using a verifier. The paper’s own composition control shows RAD does worse than random when only a few rollouts are correct, which is the usual case on hard coding problems. Its gains on code are also much smaller than on math.
- Several prior works already use MoE routing at test time, such as RICE (https://arxiv.org/abs/2505.14681), C3PO (https://arxiv.org/abs/2504.07964), SCMoE (https://arxiv.org/abs/2405.14507), and R2-T2 (https://arxiv.org/abs/2502.20395), and MoEE (https://arxiv.org/abs/2410.10814) uses routing weights as embeddings. These methods add some inference cost, but they work on a single generation and need no answer anchor, so they apply more widely. The paper does not discuss or compare with any of them.
- RAD only works after all rollouts are finished. The authors show that reading routing from the first 32 to 256 tokens is close to random, so it cannot be used for early stopping or pruning like many other test-time methods in the related work. The claim that RAD is “free” therefore only covers the selection step, while the main cost, generating 64 full rollouts, stays the same.
Questions for authors
- How does RAD compare with prior routing-based test-time methods (e.g., SCMoE, RICE, C3PO) and with using routing weights as embeddings (MoEE)? Could routing-intervention methods be combined with RAD?
- Can the main tables report Δsel instead of lift over Avg@64, to separate genuine selection from the anchor-completion filter?
- The answer-string-free interface should matter most for open-ended generation (e.g., long-form QA, summarization, story writing), yet these are not evaluated. Does RAD work there, given there is no natural answer-opening anchor and no single “answer basin”? For creative tasks, would selecting the densest (most typical) rollout actually hurt quality or diversity
2 comments
Final Decision
Thank you for the response. The rebuttal narrows the claim rather than addressing my concerns, and no new experiments are provided. The comparison with text embedding or edit-distance selection could be run directly on the existing rollouts at low cost, and it is central to whether routing offers anything beyond output similarity. Without it, the FMV complementarity result also cannot show that routing is needed, since a text-similarity tie-breaker might give the same gain. With the claim reduced to “routing supports selection” and RAD matching but not exceeding majority voting, the contribution is mainly an observation rather than a method. I keep my score and support rejection.
We thank the reviewer for the thoughtful and constructive assessment, particularly for recognizing the value of the "same token, different experts" observation. To clarify:
1. RAD is not intended to outperform Majority or replace execution-based verification.
Our central question is whether native MoE routing decisions can serve as a useful agreement signal for selecting completed rollouts without comparing their answer strings. The comparable performance to textual Majority (73.9% vs. 73.6%, not statistically significant) is evidence that this sparse internal signal can recover much of the consensus structure accessible through explicit answer grouping, rather than a claim of superior accuracy.
Importantly, RAD can complement execution-based selection rather than compete with it. Appendix A.4 (Table 6) already reports that applying RAD within FMV's winning execution-equivalence class improves accuracy from 71.88% to 73.68% (+1.80 pp, exact McNemar p=0.0084) on the same 1,554 covered model–problem cells, without additional test executions.
This result is particularly relevant to the reviewer's practical concern: even after execution-based consensus has been established, routing agreement can provide an additional selection signal among candidates indistinguishable under the generated tests.
2. The contribution is a distinct use of routing information, not the first discovery that routing is informative.
We appreciate the references to RICE, C3PO, SCMoE, R2-T2, and MoEE, and agree that these should be discussed explicitly.
Our distinction is operational and methodological: several of these methods modify routing or output distributions during generation, whereas RAD uses the routing decisions already produced by completed rollouts, without modifying the model or generation process. MoEE is especially relevant as prior work demonstrating that routing weights can serve as representations, and we do not claim that routing-based representations are new.
Our specific contribution is to establish and characterize anchor-conditioned routing agreement as a candidate-selection signal, including where this signal emerges, its relationship to answer basins, and its systematic failure under incorrect consensus.
We also agree that text embeddings, edit-distance selection, and LLM judges are relevant alternatives. Their omission limits claims of comparative superiority, and we will make this explicit rather than imply that routing necessarily outperforms textual similarity. Nevertheless, this does not invalidate the demonstrated routing–answer-basin relationship or the existing selection results. We view the comparison as important for establishing relative advantages, not as a prerequisite for the narrower empirical finding that sparse routing supports selection.
3. The limitations are genuine, but they define the scope rather than contradict the contribution.
We agree that RAD is neither an early-stopping method nor a correctness verifier. It selects from completed rollout pools, and its effectiveness depends on the structure of the candidate distribution.
Indeed, our composition analysis (Fig. 4, Appendix A.15) explicitly shows that routing consensus can fail when correct candidates are rare. We regard this as a substantive characterization of what routing agreement measures: shared solution-basin structure rather than independent correctness.
Regarding the anchor filter, Appendices A.9 and A.11 already separate the eligibility contribution from genuine selection gains through
Δsel=AccRAD−Avg@cohort.We agree that this distinction should be more prominent in the main paper and will revise the presentation accordingly.
We will also replace potentially misleading descriptions of RAD as "free" with the more precise statement that selection requires no additional model forward pass, while explicitly acknowledging the approximately 7% routing-capture overhead reported in Appendix A.5.
Finally, Appendix A.7 already examines EOS-aligned routing, which achieves 70.37% versus 71.32% for marker-based RAD on the evaluated pool. This suggests that the readout is not intrinsically restricted to a particular answer delimiter. However, we agree that effectiveness on long-form and creative generation remains unestablished and will state this boundary clearly.
We appreciate the reviewer's suggestions and will strengthen the related-work discussion, emphasize the existing selection-gain decomposition and FMV complementarity results, and clarify the computational and applicability boundaries.
We hope these clarifications help establish that the paper makes a meaningful contribution as an empirical study and operationalization of MoE routing agreement, even without claiming universal superiority over alternative selectors. We would be grateful if the reviewer could reconsider the assessment in light of this more precise scope.