- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies token-level routing of forward-KL coverage transfer in on-policy distillation. Starting from EOPD, the authors explore several increasingly sophisticated routing mechanisms, including soft entropy gating, distribution-aware routing, outcome-conditioned routing, branch-value conditioning, and a cover-then-sharpen curriculum. Their main empirical conclusion is negative: more selective routing does not outperform EOPD, and large-k coverage appears to depend more strongly on the total forward-KL budget than on where that budget is allocated. The paper attributes the failure of outcome-based routing to finite-rollout miscoverage: low-success problems are both more likely to receive no positive routing evidence and to have the highest marginal value for Pass@k.
Critical strengths
- The paper performs a fairly extensive controlled study around a common OPD/EOPD training protocol and includes substantially more seed and uncertainty analysis than is typical for this type of experiment.
- The finite-rollout miscoverage explanation is intuitive and provides a coherent mechanism for why outcome-conditioned gates can suppress coverage on difficult problems.
- The authors are unusually transparent about limitations, reproduction discrepancies, post-hoc additional runs, and where their statistical resolution is insufficient.
Critical weaknesses
-
Limited novelty and significance. EOPD already investigates entropy-based routing, forward-KL strength, and related design choices. Although this paper explores additional routing mechanisms and a cover-then-sharpen curriculum, none demonstrates a robust improvement over EOPD. The main conclusions are largely negative or parity results, and the proposed budget-limited interpretation offers limited new scientific insight.
-
Insufficient evidence for the central budget principle. The paper demonstrates that seed variability can reverse seemingly meaningful comparisons, yet several key experiments supporting the broader budget-limited conclusion, including the routing cascade and coefficient/dose analyses, remain single-seed. This weakens the generality of the proposed explanation.
-
Reporting and reproducibility concerns. The CTS cover-87 three-seed AIME results in Tables 1 and 8 contain values inconsistent with conventional rounding of integer correctness counts (e.g., AIME25 Avg@8 = 6.66 and Pass@8 = 22.21). More importantly, the additional CTS seeds 41/43 are retained only as summary-level metrics despite supporting a headline multi-seed equivalence claim. The statistical aggregation and uncertainty estimation require clarification. Separately, the claimed "strict reproduction" of EOPD does not reproduce its published absolute performance.
Questions for authors
-
What is the main new scientific insight beyond the routing and forward-KL analyses already established by EOPD, given that none of the proposed methods robustly improves over it?
-
Given the demonstrated seed sensitivity, how do the authors justify the broader budget-limited conclusion using predominantly single-seed mechanism and dose experiments?
-
Could the authors provide the exact per-seed integer correctness counts for CTS cover-87 (seeds 41/42/43), clarify the aggregation and rounding used in Tables 1 and 8, and explain how the seed-mean equivalence confidence intervals were computed given the summary-only records for seeds 41/43? A matched three-seed CTS-versus-EOPD comparison would also be helpful.
-
Minor: what does the undefined “Arc-27 protocol” refer to in Appendix F?
10 comments
W1. Limited novelty beyond EOPD and no robust performance gain [1/1]
W1. Limited novelty beyond EOPD and no robust performance gain [1/1]
Concern. EOPD already examines entropy routing, FKL strength, and related choices, while the additional routes and curriculum do not robustly improve it. The incremental scientific contribution needs to be clearer.
Our contribution is a controlled diagnosis of reliability, evidence availability, dose/schedule dependence, and sampling-budget-dependent coverage—not the introduction of entropy-conditioned FKL or a superior performance frontier. We appreciate the need to separate inherited and new elements. EOPD already provides the entropy/FKL objective and threshold, coefficient, and retained-support ablations. We do not claim priority for those ideas or for the general observation that FKL strength matters.
The first additional finding is a concrete evaluation-reliability failure in this routing family. Soft versus OPD gives +2.09 Avg@8 [95% problem interval +0.9,+3.4] on seed 42 and −1.57 [−3.1,−0.2] on seed 41. Against EOPD on common seeds 41/42/43, soft gives 29.99/49.33 versus 30.02/50.66. The new dose panel adds a second concrete reversal: α=0.5 exceeds α=2.0 on seed 41 but is lower on both metrics on seed 43. The contribution is not “seeds matter” in the abstract; it is evidence that plausible routing and dose selections change ordering in this specific design space.
The second finding links positive-evidence routing to a measurable failure region. A gate requiring one success among four independent rollouts closes with probability (1−p)^4, or 0.6561 at p=0.1. Marginal Pass@8 sensitivity is 8(1−p)^7 under the same model. The logged 28–49% closed-group fractions make this relevant to the evaluated utility/branch pilots. Grouping confounds and the c>0 benchmark-diagnostic population remain explicit, so this is a bounded mechanism rather than a theorem that all routing is futile.
The third finding is that method conclusions depend on schedule, model setting, and sampling budget. CTS retains its cover-87/116 inversion. On seata2 T4, OPD/EOPD/soft give 31.02/30.46/30.72 Avg@8 and 47.38/49.11/47.51 Pass@8; on Llama they give 17.02/16.93/16.28 and 39.64/38.28/37.75. Soft does not jointly win either line. At pass@128, however, Llama-soft is no lower than both baselines at all three measured endpoints, while T4-soft improves AMC23/AIME25 but loses AIME24. This shows why a single model, seed, or k cannot carry the full design conclusion.
The significance is an experimentally grounded design lesson rather than a SOTA claim. A signal closer to task success is not automatically better supervision if finite sampling makes it unavailable on difficult prompts. A gate changes token selection, realized coefficient exposure, prompt representation, and subsequent optimization together. Our controls make those coupled effects measurable and identify where apparently attractive choices fail stronger seed, environment, and coverage checks. The unsupported historical CTS parity claim is excluded from this contribution.
Planned manuscript revision — introduction, related work, and discussion. We will center these tested findings, credit EOPD's established analyses, and include the new dose reversal and mixed large-k evidence. The reporting audit supports transparency but is not presented as an algorithmic advance.
Proposed revised-manuscript text.
Our contribution is a controlled study of selective routing and cross-seed reliability, finite-positive-evidence failure, dose/schedule dependence, and coverage under changing sampling budgets. It is not a new sufficient statistic for coverage or a universally superior gate.
W2. Single-seed mechanism and dose evidence [1/1]
W2. Single-seed mechanism and dose evidence [1/1]
Concern. Seed sensitivity is demonstrated in the main comparisons, but predominantly single-seed routing and dose analyses support the broader budget interpretation.
We agree that seed sensitivity must constrain explanatory claims as strongly as performance claims. A plausible single-seed pattern cannot become decisive because it is convenient. The new two-seed/two-dose results improve the evidence base, but their seed-43 reversal further weakens any universal dose interpretation.
The retained single-seed observations contain their own limits. The routing cascade gives Pass@8 50.00, 48.38, 48.29, and 47.13; the middle 0.09-point difference is too small for an independent mechanistic conclusion. CTS cover 87 gives 50.04 and cover 116 gives 49.80, ruling out strict monotonicity. The cascade changes weights and exposure together, while CTS changes timing and optimization history. Neither is a matched-exposure causal intervention that excludes routing.
The completed coefficient panel uses matched training seeds and one evaluation domain. Both doses use Qwen3-1.7B-Base/Qwen3-8B, 174 updates, Pro6000 evaluation, and decoding seed 42:
| Dose | Seed 41 Avg@8 / Pass@8 | Seed 43 Avg@8 / Pass@8 | Two-seed mean Avg@8 / Pass@8 |
|---|---|---|---|
| α=0.5 | 31.00 / 51.47 | 29.39 / 48.33 | 30.19 / 49.90 |
| α=2.0 | 30.48 / 48.93 | 29.56 / 49.18 | ≈30.02 / ≈49.06 |
The α=0.5 mean uses unrounded count-based records; its sample SD is 1.14 Avg@8 and 2.22 Pass@8. The α=2.0 means are computed from the displayed run summaries. On seed 41, α=0.5 is higher by +0.52/+2.54; on seed 43, it is lower by −0.17/−0.85. The modestly higher two-seed means therefore coexist with opposite per-seed orderings. We do not describe α=0.5 as consistently better on both seeds or claim a strict monotonic dose law.
The new α=0.5 spectra measure evaluation coverage, not an additional dose control. Seed 41 gives pass@128 92.5/46.67/40.00 on AMC23/AIME24/AIME25. Independent evaluations of the same seed-43 checkpoint give 97.5/40.00/43.33 and 92.5/43.33/40.00. These runs quantify evaluation variability on one checkpoint; they are not extra training seeds. No matched α=2.0 spectra are available, so they cannot establish a between-dose large-k effect.
Historical endpoints remain environment-separated. Traceable original α=0 and α=1 scores are 28.54/48.62 and 30.63/50.00, evaluated in the historical A6000 setting. We do not merge them with the Pro6000 panel to construct a four-dose common-environment curve. The original α=1 token-share and performance entries also came from different training seeds, and the post-inspection 1/24 ordering argument is withdrawn.
The mechanism evidence remains local. Positive-evidence closure has a precise independent-sampling identity, but grouped outcome arms also reduce distinct prompts. The sensitivity-weighted closure diagnostic is calculated only on observed-solvable problems with c>0 among eight completions. With detached weights and fixed contexts, the FKL gradient is Σg_t∇L_t; equal B=Σg_t does not equalize token gradients, lengths, optimizer state, or subsequent on-policy contexts. Replication and causal isolation remain necessary.
Planned manuscript revision — dose and mechanism analyses. We will report all four new dose outcomes, the per-seed reversal, and the evaluation-repeat distinction; preserve the cascade/schedule counterexamples; and replace the broad budget law with a conditional research interpretation.
Proposed revised-manuscript text.
The two-dose panel provides replicated observations but not a universal dose ordering: α=0.5 is higher on seed 41 and lower on seed 43 for both reported metrics. Exposure, routing, timing, and optimization are therefore not treated as independently identified causes.
Response to Reviewer PPE4 — Overview and index
Response to Reviewer PPE4
Thank you for the detailed examination of novelty, seed sensitivity, and numerical reproducibility. Your review has led us to sharpen the scientific contribution and to audit the provenance behind every uncertainty statement. The new evidence adds complete same-environment three-arm model comparisons, a two-seed/two-dose panel, large-k spectra, and new CTS n=128 integer-count records. It also preserves the negative results: there is no robust EOPD overtaking, the dose ordering reverses on seed 43, and the unsupported historical CTS three-seed aggregate remains withdrawn.
The evidence cutoff is 13 November 2026. Scores are percentages and differences are percentage points. Macros equally average six unrounded benchmark rates. Training seeds, decoding seeds, n=8/n=128 sample budgets, and evaluation environments are distinguished throughout. This is a proposed local response: no OpenReview posting or completed manuscript update is implied. “Planned manuscript revision” and “Proposed revised-manuscript text” identify changes to synchronize with final LaTeX. The final CTS-58 seed-43 A800face six-benchmark evaluation is now included as a separately labeled observation; it is not pooled with records from other evaluation environments.
Reading guide
The response is organized by the original review items. No substantive content is omitted from the complete response. Continuations preserve all evidence, tables, derivations, and proposed manuscript text in order. Proposed revisions are not represented as completed LaTeX edits. All reported numerical panels have been filled and numerically checked. This package has not been posted to OpenReview.
There are 7 W/Q threads and 9 W/Q comments in total:
Including this overview, there are 10 comments for this reviewer.
Please read each thread in ascending [part/total] order.