- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper looks at the “best expert for each token” idea for MoE models, where each candidate expert is tried, and the best one is kept, all on the same tokens. The authors argue that the observed effects are mostly due to selection noise. To show this, they select the expert on one half of the text and score it on the other half, and they compare against a matched-noise baseline that reproduces the original numbers. They also introduce an identity-preserving swap as a control, and recommend separating selection from scoring in future works.
Critical strengths
S1: The paper is well-written, engages with related works, and analyses current strategies.
S2: The authors propose an identity-preserving swap that changes only which expert gets a routing weight and leaves everything else fixed.
Critical weaknesses
W1: The cross-fit tests whether the chosen expert helps consistently across different downstream tokens after the intervention. The cross-fit and its detection floor are reported for the whole suffix, where a real gain on only the first few tokens could be hidden. The paper’s own results show the forcing/swap ratio depends strongly on scoring horizon, so it is unclear to me how robust the null is at short horizons.
W2: The matched-noise null relies on the two halves being exchangeable. The paper states this assumption, and the interleaved halves with matched distance profiles make it plausible, but I did not see a direct test of it. The cross-fit uses the same split, so the two main pieces of evidence share this assumption and are not fully independent. If the halves differ, agreement between the null and the observed values could be partly an artifact.
Questions for authors
Q1: Do the results rule out token-specific headroom that a learned selector could exploit, or only persistent headroom that carries over to other tokens?
Q2: Which version of the ratio (5.48, 3.63, or the weighted range) is the main one? Why does it differ so much between models, and does it also hold within each model, not only when averaged?
2 comments
Thank you for your response and the newly added experiments, which resolved my concerns. I will maintain my positive score.
Response to Reviewer H2qY: Short-Horizon Effects, Matched-Noise Assumptions, and Headroom Interpretation
Thank you for the thoughtful review and for identifying two important limits of the original analysis: short-horizon effects and the assumptions behind the matched-noise reference. Both concerns led to substantive revisions and additional experiments.
W1 — Short-horizon effects. We now evaluate scoring horizons from 2 to 64 suffix tokens, alongside alternative block-interleaved and contiguous splits (Section 5.3; Appendix B.3, Tables 12–13). The partial trace collection does not show detectably positive pooled restricted-candidate cross-fitted headroom, but its sensitivity is substantially weaker at short horizons. We therefore do not interpret non-detection over a full suffix as evidence that short-lived effects are absent.
Importantly, Qwen1.5-MoE layer 10 shows a positive short-horizon forcing effect. An initial two-token estimate of +0.052 [+0.028, +0.075] nats/site was followed by a preregistered fresh-document replication of +0.0189 [+0.0002, +0.0375] across 1,837 sites and 56 documents. The replicated estimate is smaller, neither seed alone excludes zero, and the full-suffix effect is not detected. This establishes a setting-specific exception to an overly broad interpretation of the pooled result, not universal short-horizon transfer.
W2 — Exchangeability and matched-noise validity. We directly examined the assumptions of the matched-noise construction. The interleaved halves differ substantially in contrast variance: half-A/half-B ratios are 11.4 [8.9, 14.4] for forcing and 3.7 [2.2, 5.1] for the swap (Appendix B.1, Table 9). The reference also shares its split with the cross-fit, so the two are not independent confirmations.
We have consequently removed the original interpretation of the reference as an exact calibrated null. It is now explicitly an approximate, uncalibrated descriptive comparison. The observed paired split-half ratio is 3.63 versus a reference ratio of 3.97; this does not establish that all apparent headroom is noise (Section 5.2).
Independent-continuation validation. We additionally tested whether candidate effects transfer across independently model-sampled continuations of the same prefix. Candidates were selected on four continuations and evaluated on four others. Two of three preregistered cells passed the instrument gate, contributing 437 sites. Forcing retains +0.0087 [+0.0060, +0.0119] nats/site, driven by OLMoE-0924 layer 7, whereas the swap gives −0.0001 [−0.0003, +0.0001]. The excluded cell and post-hoc replication are separately disclosed (Section 5.4; Appendix B.2). The positive forcing result is relative to its intervention-specific reference, not necessarily the native route. It demonstrates why the original within-continuation non-detection cannot be generalized to every evaluation protocol.
Q1 — Can a learned selector exploit token-specific opportunity? Our results do not rule this out. The action-class ceiling applies only to the specified identity-preserving swap family, candidate population, and utility (Section 3). A tested frozen identity-only selector produced +0.00086 [+0.00025, +0.00152] nats/site on one sealed split, but two fresh-cohort evaluations (+0.00016 and −0.00003) did not replicate a detectable gain (Appendix D). These findings constrain that particular selector, not every learned routing policy. Moreover, an exploratory exhaustive-pool test finds a positive swap cross-fit of +0.0044 [+0.0003, +0.0088] in one Qwen3 layer with 128 candidates, although its uncertainty and limited document coverage prevent a general claim.
Q2 — Which ratio is primary, and does it hold within models? The primary estimate is 5.48, defined as the ratio of summed cell means across the 202-cell registry. The site-pooled ratio of 5.65 uses different weighting, while 3.63 comes from a separate 115-cell paired split-half collection. They are not interchangeable estimates (Section 5.1; Table 2; Appendix B.5).
Within the eight evaluated checkpoints, the primary forcing-to-swap ratio ranges from 1.12 to 11.90 and exceeds one for every checkpoint. Forcing exceeds the swap in 199/202 individual cells. The magnitude varies with architecture and routing semantics; Section 6 shows that changes in candidate-effect dispersion are strongly associated with the reported ratio (Spearman 0.92), without claiming a complete causal decomposition.
We appreciate these questions. The revised manuscript retains the central intervention-dependent measurement finding while explicitly recognizing persistent effects, statistical limitations, and the boundaries of what the experiments establish.