acceptodds
Under review as a conference paper at ICLR 2027

Every Fork Has a Price: The Precision–Coverage Trade-off in On-Policy Distillation of Reasoning Models

Abstract

How much of a teacher's reasoning diversity should a student keep? In on-policy distillation, a routing rule decides which tokens receive coverage transfer. The dominant answer, entropy-aware on-policy distillation (EOPD), applies forward KL wherever teacher entropy exceeds a threshold. We test whether smarter routing can do better. Five routing mechanisms of increasing sophistication are compared in a controlled design-space study under one strictly reproduced protocol (Qwen3-1.7B student, Qwen3-8B teacher, six math benchmarks, identical trajectory budgets). What survives is a principle: coverage transfer is budget-limited, not routing-limited. Mechanistically, gates conditioned on trajectory outcomes remove forward-KL mass precisely from low-success problems whose marginal Pass@k value is highest. Whole prompt groups lose all coverage signal at a rate matching a compounding miscoverage model ((1-p)^n). Budget-preserving reweighting cannot recover the coverage that closed prompt groups never receive. EOPD's crude threshold keeps the coverage lead because it spends the most: Pass@k follows the budget, not the gate. Three measurements make the principle concrete: a soft continuous entropy gate matches EOPD on macro Avg@8 within a ±3-point equivalence margin across seeds (29.99±0.46 vs. 30.15±0.54) at zero extra rollout cost. A cover-then-sharpen curriculum turns the budget into a dial: a half-schedule cover holds coverage level with the full-budget endpoint over three seeds (50.21±0.60 vs. 50.68±0.28 Pass@8). The forward-KL coefficient alone moves macro Avg@8 monotonically up to the default dose (28.54 → 30.63), confirming that outcomes track the budget, not the gate. Our code is available at the anonymous repository: https://anonymous.4open.science/r/TSA-OPD-for-ICLR-2027-50BD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Large Language Models, Mathematical Reasoning, Precision–Coverage Trade-off

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PPE4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies token-level routing of forward-KL coverage transfer in on-policy distillation. Starting from EOPD, the authors explore several increasingly sophisticated routing mechanisms, including soft entropy gating, distribution-aware routing, outcome-conditioned routing, branch-value conditioning, and a cover-then-sharpen curriculum. Their main empirical conclusion is negative: more selective routing does not outperform EOPD, and large-k coverage appears to depend more strongly on the total forward-KL budget than on where that budget is allocated. The paper attributes the failure of outcome-based routing to finite-rollout miscoverage: low-success problems are both more likely to receive no positive routing evidence and to have the highest marginal value for Pass@k.

Critical strengths
  1. The paper performs a fairly extensive controlled study around a common OPD/EOPD training protocol and includes substantially more seed and uncertainty analysis than is typical for this type of experiment.
  2. The finite-rollout miscoverage explanation is intuitive and provides a coherent mechanism for why outcome-conditioned gates can suppress coverage on difficult problems.
  3. The authors are unusually transparent about limitations, reproduction discrepancies, post-hoc additional runs, and where their statistical resolution is insufficient.
Critical weaknesses
  1. Limited novelty and significance. EOPD already investigates entropy-based routing, forward-KL strength, and related design choices. Although this paper explores additional routing mechanisms and a cover-then-sharpen curriculum, none demonstrates a robust improvement over EOPD. The main conclusions are largely negative or parity results, and the proposed budget-limited interpretation offers limited new scientific insight.

  2. Insufficient evidence for the central budget principle. The paper demonstrates that seed variability can reverse seemingly meaningful comparisons, yet several key experiments supporting the broader budget-limited conclusion, including the routing cascade and coefficient/dose analyses, remain single-seed. This weakens the generality of the proposed explanation.

  3. Reporting and reproducibility concerns. The CTS cover-87 three-seed AIME results in Tables 1 and 8 contain values inconsistent with conventional rounding of integer correctness counts (e.g., AIME25 Avg@8 = 6.66 and Pass@8 = 22.21). More importantly, the additional CTS seeds 41/43 are retained only as summary-level metrics despite supporting a headline multi-seed equivalence claim. The statistical aggregation and uncertainty estimation require clarification. Separately, the claimed "strict reproduction" of EOPD does not reproduce its published absolute performance.

Questions for authors
  1. What is the main new scientific insight beyond the routing and forward-KL analyses already established by EOPD, given that none of the proposed methods robustly improves over it?

  2. Given the demonstrated seed sensitivity, how do the authors justify the broader budget-limited conclusion using predominantly single-seed mechanism and dose experiments?

  3. Could the authors provide the exact per-seed integer correctness counts for CTS cover-87 (seeds 41/42/43), clarify the aggregation and rounding used in Tables 1 and 8, and explain how the seed-mean equivalence confidence intervals were computed given the summary-only records for seeds 41/43? A matched three-seed CTS-versus-EOPD comparison would also be helpful.

  4. Minor: what does the undefined “Arc-27 protocol” refer to in Appendix F?

10 comments
AuthorsOpenReview ↗

Response to Reviewer PPE4 — Overview and index

Response to Reviewer PPE4

Thank you for the detailed examination of novelty, seed sensitivity, and numerical reproducibility. Your review has led us to sharpen the scientific contribution and to audit the provenance behind every uncertainty statement. The new evidence adds complete same-environment three-arm model comparisons, a two-seed/two-dose panel, large-k spectra, and new CTS n=128 integer-count records. It also preserves the negative results: there is no robust EOPD overtaking, the dose ordering reverses on seed 43, and the unsupported historical CTS three-seed aggregate remains withdrawn.

The evidence cutoff is 13 November 2026. Scores are percentages and differences are percentage points. Macros equally average six unrounded benchmark rates. Training seeds, decoding seeds, n=8/n=128 sample budgets, and evaluation environments are distinguished throughout. This is a proposed local response: no OpenReview posting or completed manuscript update is implied. “Planned manuscript revision” and “Proposed revised-manuscript text” identify changes to synchronize with final LaTeX. The final CTS-58 seed-43 A800face six-benchmark evaluation is now included as a separately labeled observation; it is not pooled with records from other evaluation environments.


Reading guide

The response is organized by the original review items. No substantive content is omitted from the complete response. Continuations preserve all evidence, tables, derivations, and proposed manuscript text in order. Proposed revisions are not represented as completed LaTeX edits. All reported numerical panels have been filled and numerically checked. This package has not been posted to OpenReview.

There are 7 W/Q threads and 9 W/Q comments in total:

  • W1 — 1 comment: Limited novelty beyond EOPD and no robust performance gain
  • W2 — 1 comment: Single-seed mechanism and dose evidence
  • W3 — 2 comments: CTS reporting, uncertainty, and reproduction fidelity
  • Q1 — 1 comment: The main new insight beyond EOPD
  • Q2 — 1 comment: Justifying a broad conclusion from single-seed mechanism experiments
  • Q3 — 2 comments: Integer counts, rounding, confidence intervals, and matched seeds
  • Q4 — 1 comment: Meaning of “Arc-27 protocol” in Appendix F

Including this overview, there are 10 comments for this reviewer.

Please read each thread in ascending [part/total] order.

AuthorsOpenReview ↗

W1. Limited novelty beyond EOPD and no robust performance gain [1/1]

W1. Limited novelty beyond EOPD and no robust performance gain [1/1]

Concern. EOPD already examines entropy routing, FKL strength, and related choices, while the additional routes and curriculum do not robustly improve it. The incremental scientific contribution needs to be clearer.

Our contribution is a controlled diagnosis of reliability, evidence availability, dose/schedule dependence, and sampling-budget-dependent coverage—not the introduction of entropy-conditioned FKL or a superior performance frontier. We appreciate the need to separate inherited and new elements. EOPD already provides the entropy/FKL objective and threshold, coefficient, and retained-support ablations. We do not claim priority for those ideas or for the general observation that FKL strength matters.

The first additional finding is a concrete evaluation-reliability failure in this routing family. Soft versus OPD gives +2.09 Avg@8 [95% problem interval +0.9,+3.4] on seed 42 and −1.57 [−3.1,−0.2] on seed 41. Against EOPD on common seeds 41/42/43, soft gives 29.99/49.33 versus 30.02/50.66. The new dose panel adds a second concrete reversal: α=0.5 exceeds α=2.0 on seed 41 but is lower on both metrics on seed 43. The contribution is not “seeds matter” in the abstract; it is evidence that plausible routing and dose selections change ordering in this specific design space.

The second finding links positive-evidence routing to a measurable failure region. A gate requiring one success among four independent rollouts closes with probability (1−p)^4, or 0.6561 at p=0.1. Marginal Pass@8 sensitivity is 8(1−p)^7 under the same model. The logged 28–49% closed-group fractions make this relevant to the evaluated utility/branch pilots. Grouping confounds and the c>0 benchmark-diagnostic population remain explicit, so this is a bounded mechanism rather than a theorem that all routing is futile.

The third finding is that method conclusions depend on schedule, model setting, and sampling budget. CTS retains its cover-87/116 inversion. On seata2 T4, OPD/EOPD/soft give 31.02/30.46/30.72 Avg@8 and 47.38/49.11/47.51 Pass@8; on Llama they give 17.02/16.93/16.28 and 39.64/38.28/37.75. Soft does not jointly win either line. At pass@128, however, Llama-soft is no lower than both baselines at all three measured endpoints, while T4-soft improves AMC23/AIME25 but loses AIME24. This shows why a single model, seed, or k cannot carry the full design conclusion.

The significance is an experimentally grounded design lesson rather than a SOTA claim. A signal closer to task success is not automatically better supervision if finite sampling makes it unavailable on difficult prompts. A gate changes token selection, realized coefficient exposure, prompt representation, and subsequent optimization together. Our controls make those coupled effects measurable and identify where apparently attractive choices fail stronger seed, environment, and coverage checks. The unsupported historical CTS parity claim is excluded from this contribution.

Planned manuscript revision — introduction, related work, and discussion. We will center these tested findings, credit EOPD's established analyses, and include the new dose reversal and mixed large-k evidence. The reporting audit supports transparency but is not presented as an algorithmic advance.

Proposed revised-manuscript text.

Our contribution is a controlled study of selective routing and cross-seed reliability, finite-positive-evidence failure, dose/schedule dependence, and coverage under changing sampling budgets. It is not a new sufficient statistic for coverage or a universally superior gate.

AuthorsOpenReview ↗

W2. Single-seed mechanism and dose evidence [1/1]

W2. Single-seed mechanism and dose evidence [1/1]

Concern. Seed sensitivity is demonstrated in the main comparisons, but predominantly single-seed routing and dose analyses support the broader budget interpretation.

We agree that seed sensitivity must constrain explanatory claims as strongly as performance claims. A plausible single-seed pattern cannot become decisive because it is convenient. The new two-seed/two-dose results improve the evidence base, but their seed-43 reversal further weakens any universal dose interpretation.

The retained single-seed observations contain their own limits. The routing cascade gives Pass@8 50.00, 48.38, 48.29, and 47.13; the middle 0.09-point difference is too small for an independent mechanistic conclusion. CTS cover 87 gives 50.04 and cover 116 gives 49.80, ruling out strict monotonicity. The cascade changes weights and exposure together, while CTS changes timing and optimization history. Neither is a matched-exposure causal intervention that excludes routing.

The completed coefficient panel uses matched training seeds and one evaluation domain. Both doses use Qwen3-1.7B-Base/Qwen3-8B, 174 updates, Pro6000 evaluation, and decoding seed 42:

DoseSeed 41 Avg@8 / Pass@8Seed 43 Avg@8 / Pass@8Two-seed mean Avg@8 / Pass@8
α=0.531.00 / 51.4729.39 / 48.3330.19 / 49.90
α=2.030.48 / 48.9329.56 / 49.18≈30.02 / ≈49.06

The α=0.5 mean uses unrounded count-based records; its sample SD is 1.14 Avg@8 and 2.22 Pass@8. The α=2.0 means are computed from the displayed run summaries. On seed 41, α=0.5 is higher by +0.52/+2.54; on seed 43, it is lower by −0.17/−0.85. The modestly higher two-seed means therefore coexist with opposite per-seed orderings. We do not describe α=0.5 as consistently better on both seeds or claim a strict monotonic dose law.

The new α=0.5 spectra measure evaluation coverage, not an additional dose control. Seed 41 gives pass@128 92.5/46.67/40.00 on AMC23/AIME24/AIME25. Independent evaluations of the same seed-43 checkpoint give 97.5/40.00/43.33 and 92.5/43.33/40.00. These runs quantify evaluation variability on one checkpoint; they are not extra training seeds. No matched α=2.0 spectra are available, so they cannot establish a between-dose large-k effect.

Historical endpoints remain environment-separated. Traceable original α=0 and α=1 scores are 28.54/48.62 and 30.63/50.00, evaluated in the historical A6000 setting. We do not merge them with the Pro6000 panel to construct a four-dose common-environment curve. The original α=1 token-share and performance entries also came from different training seeds, and the post-inspection 1/24 ordering argument is withdrawn.

The mechanism evidence remains local. Positive-evidence closure has a precise independent-sampling identity, but grouped outcome arms also reduce distinct prompts. The sensitivity-weighted closure diagnostic is calculated only on observed-solvable problems with c>0 among eight completions. With detached weights and fixed contexts, the FKL gradient is Σg_t∇L_t; equal B=Σg_t does not equalize token gradients, lengths, optimizer state, or subsequent on-policy contexts. Replication and causal isolation remain necessary.

Planned manuscript revision — dose and mechanism analyses. We will report all four new dose outcomes, the per-seed reversal, and the evaluation-repeat distinction; preserve the cascade/schedule counterexamples; and replace the broad budget law with a conditional research interpretation.

Proposed revised-manuscript text.

The two-dose panel provides replicated observations but not a universal dose ordering: α=0.5 is higher on seed 41 and lower on seed 43 for both reported metrics. Exposure, routing, timing, and optimization are therefore not treated as independently identified causes.

Reviewer_u55tOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the precision–coverage trade-off in on-policy distillation. The authors evaluate different forward-KL routing strategies and argue that coverage transfer depends more on the forward-KL supervision budget than on the routing strategy itself. They also propose a soft entropy gate and a cover-then-sharpen training strategy.

Critical strengths
  1. The paper addresses an interesting and relevant problem in on-policy distillation, particularly the trade-off between precision and coverage.

  2. The authors investigate several routing strategies, providing useful empirical insights into forward-KL supervision.

  3. The paper presents extensive experimental analyses, including ablation studies, multi-seed evaluations, and Pass@k analysis across different sampling budgets.

Critical weaknesses
  1. The experiments are limited to a single teacher–student pair and mathematical reasoning benchmarks. It remains unclear whether the findings generalize to other models and domains, such as agentic tasks. Additional experiments across more diverse settings would help establish the generality of the conclusions.

  2. The central claim that "coverage transfer is budget-limited, not routing-limited" is not sufficiently supported by the experimental evidence. The failure of several routing strategies to improve performance does not necessarily imply that routing is unimportant. Could the authors provide stronger evidence to support this conclusion?

  3. The proposed methods do not demonstrate clear improvements over EOPD. In the multi-seed results, EOPD achieves higher average performance on both Avg@8 and Pass@8. This raises concerns about whether the proposed methods effectively improve the precision–coverage trade-off. What practical advantages do these methods offer over EOPD?

Questions for authors

See weaknesses!

5 comments
AuthorsOpenReview ↗

Response to Reviewer u55t — Overview and index

Response to Reviewer u55t

We are grateful for your careful distinction between observing failed routing extensions and explaining why they fail. Your review pushed us to apply the same evidential standard to performance, mechanism, and practical value. The completed experiments now include both additional three-arm model lines and a two-seed, two-dose panel. They strengthen the diagnosis while also supplying counterexamples: soft does not win the same-environment Avg@8/Pass@8 comparisons, and the α=0.5 versus α=2.0 ordering reverses on seed 43.

Your Questions for Authors field says “See weaknesses!”; W1–W3 therefore answer the questions embedded in all three weaknesses. The evidence cutoff is 13 November 2026. Scores are percentages, differences are percentage points, and six-benchmark macros equally average unrounded benchmark rates. Training seeds, decoding seeds, sample budgets, and evaluation environments are separated. This package is a local proposed response; it does not imply that comments or manuscript changes have been posted. Revision passages below are proposed text for final LaTeX synchronization.


Reading guide

The response is organized by the original review items. No substantive content is omitted from the complete response. Continuations preserve all evidence, tables, derivations, and proposed manuscript text in order. Proposed revisions are not represented as completed LaTeX edits. All reported numerical panels have been filled and numerically checked. This package has not been posted to OpenReview.

There are 3 W/Q threads and 4 W/Q comments in total:

  • W1 — 1 comment: A single model pair and mathematical tasks
  • W2 — 2 comments: The evidence does not show that routing is unimportant
  • W3 — 1 comment: No clear improvement over EOPD or practical advantage

Including this overview, there are 5 comments for this reviewer.

Please read each thread in ascending [part/total] order.

AuthorsOpenReview ↗

W1. A single model pair and mathematical tasks [1/1]

W1. A single model pair and mathematical tasks [1/1]

Concern. The single original teacher–student pair and mathematical benchmarks leave transfer to other models and domains, including agentic tasks, unclear.

We have completed the additional three-arm model comparisons, while task-domain transfer remains an explicit open question. Thank you for separating these requirements. The original study uses Qwen3-1.7B-Base/Qwen3-8B on mathematical reasoning. The new T4 and Llama experiments test changed teachers and families, but they do not establish agentic-task performance.

Direct same-environment evidence now includes all three methods. T4 uses Qwen3-1.7B-Base/Qwen3-4B-Instruct-2507; the family-change line uses Llama-3.2-3B-Instruct/Llama-3.1-8B-Instruct. Training seed 42 and 174 MATH updates are used. All six rows below are evaluated on seata2 with eight completions per problem, decoding seed 42, temperature/top-p 1.0/0.8, and the common mathematical adapter.

Setting, seata2 evaluationOPD Avg@8 / Pass@8EOPD Avg@8 / Pass@8Soft Avg@8 / Pass@8
Qwen 1.7B / 4B teacher31.02 / 47.3830.46 / 49.1130.72 / 47.51
Llama 3B / 8B teacher17.02 / 39.6416.93 / 38.2816.28 / 37.75

The result is not a uniform transfer of the original recommendation. T4 has the highest Avg@8 under OPD and highest Pass@8 under EOPD; Llama has OPD highest on both. Soft does not overtake both baselines on either line. These are fixed-recipe, single-training-seed observations, not multi-seed family-level significance tests.

Complementary large-k measurements reveal a different aspect of transfer. With n=128, temperature/top-p 1.0/0.8, and decoding seed 42, Llama-soft reaches 100.0/50.00/23.33 pass@128 on AMC23/AIME24/AIME25, versus OPD 95.0/33.33/23.33 and EOPD 95.0/43.33/20.00. It is no lower than both baselines at all three endpoints, but AIME25 is a tie with OPD. T4-soft instead gives 85.0/20.00/40.00 versus OPD 82.5/30.00/36.67 and EOPD 82.5/26.67/36.67, retaining the AIME24 negative result. The spectra therefore support setting-dependent coverage at large sampling budgets, not general superiority or uniform low-k gains.

Evaluation environments and repetitions are not pooled. The n=128 spectra use Pro6000 for T4 and TSA-A800 for Llama, whereas the six-benchmark n=8 table uses seata2. A repeated evaluation of a checkpoint is not another training seed. Earlier cross-machine comparisons that appeared to show a T4-soft official win are superseded by the same-seata2 table.

The confounds remain scientifically relevant. Teacher training version changes with the 4B model, and the Llama student is instruction-tuned rather than base-initialized. Holding the entropy threshold fixed does not necessarily hold selected tokens or realized FKL exposure fixed. The experiments reveal limits of fixed-recipe transfer without isolating parameter count, model family, or an exposure-mediated cause.

Regarding agentic tasks, the available evidence supports only a hypothesis. Sparse positive-evidence gates could suppress feedback on difficult interactive states, but tool use adds changing state distributions, delayed outcomes, and multi-step credit assignment that our experiments do not measure. A favorable large-k mathematical result would not resolve those differences. We retain the absence of non-mathematical and interactive-agent evaluations as a substantive limitation.

Planned manuscript revision — model generality and limitations. We will add the completed same-environment three-arm table and large-k spectra, distinguish them by sample budget and environment, and state the model/task confounds directly.

Proposed revised-manuscript text.

The additional models show conditional applicability and setting-dependent orderings rather than universal transfer. Large-k verifier coverage is reported separately from eight-sample performance, and neither mathematical result substitutes for an agentic-task evaluation.

AuthorsOpenReview ↗

W2. The evidence does not show that routing is unimportant [1/2]

W2. The evidence does not show that routing is unimportant [1/2]

Concern. Failed routing strategies do not establish that coverage is budget-limited rather than routing-limited; stronger evidence is needed for the central interpretation.

We agree, and we withdraw the broad “budget-limited, not routing-limited” conclusion. The defensible claim is conditional: the tested selective gates can remove supervision, while coefficient exposure, timing, prompt diversity, and routing interact. The new dose panel itself shows a cross-seed ordering reversal, so it cannot support a universal dose law. Routing is not excluded as a cause.

The routing cascade remains direct evidence about the tested components. All rows use the same six-benchmark evaluation:

Routing variantMacro Avg@8Macro Pass@8
Soft entropy gate30.6350.00
Entropy + JS weighting29.4548.38
JS + teacher-mass weighting30.5148.29
Full composite gate29.4547.13

Adding components reduces Pass@8 by 2.87 points from soft to the full gate without an accuracy gain above the soft starting point. The 0.09-point middle change is not independently resolved. Each component changes allocation and coefficient exposure together, so this is not a matched-exposure isolation of routing.

The schedule evidence likewise contains a counterexample to strict monotonicity. Historical seed-42 CTS gives 29.98/47.31, 30.64/50.04, and 29.23/49.80 Avg@8/Pass@8 at 58, 87, and 116 cover updates; full EOPD gives 29.32/51.10. Cover 87 exceeds cover 116 on both displayed metrics. The archived launcher also starts a fresh soft-stage job from merged cover weights, so optimizer/scheduler/data-loader continuity is not established. Timing and optimization history cannot be collapsed into one accumulated-FKL scalar.

The new coefficient panel supplies two training seeds at each of two doses in one evaluation domain. Both use the original Qwen3-1.7B-Base/Qwen3-8B pair, 174 updates, and Pro6000 evaluation. Decoding seed 42 is distinct from training seeds 41/43.

DoseTraining seedMacro Avg@8Macro Pass@8
α=0.54131.0051.47
α=0.54329.3948.33
α=2.04130.4848.93
α=2.04329.5649.18
α=0.5, two-seed mean ± sample SD41/4330.19 ± 1.1449.90 ± 2.22
α=2.0, two-seed mean, approximate41/4330.0249.06
Reviewer_DBsGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper conducts a controlled empirical study on token‑level routing mechanisms for entropy‑aware on‑policy distillation (EOPD) for math reasoning. Its proposes a soft continuous entropy gate as a zero‑cost drop‑in alternative to EOPD’s hard threshold and a cover‑then‑sharpen curriculum that makes forward‑KL budget an explicit tunable dial. Multi‑seed experiments across math benchmarks show that Pass@k performance tracks accumulated forward‑KL spend instead of gate sophistication.

Critical strengths

The paper proposes a zero‑cost soft entropy gate and a cover‑then‑sharpen curriculum that treats forward‑KL spend as an explicit tuning dial, offering practically valuable insights for on‑policy distillation research on LLM reasoning.

Critical weaknesses
  1. All core empirical conclusions are constrained to math benchmarks, leaving unknown whether the budget‑limited principle transfers to non‑mathematical domains.

  2. In Table 1, the results show that the metrics for the proposed method are not significantly better than those of other methods.

Questions for authors
  1. Beyond the Qwen3-1.7B / Qwen3-8B models, do the conclusions still hold when using different combinations of "student" and "teacher" model scales?

  2. Please provide more comparisons with other SOTA on‑policy distillation or RLVR methods.

  3. The experiments focus on rule‑verifiable mathematical reasoning. What outcomes should be expected for open-ended natural-language tasks that lack a mechanism for verifying definitive answers?

8 comments
AuthorsOpenReview ↗

Response to Reviewer DBsG — Overview and index

Response to Reviewer DBsG

We sincerely thank you for recognizing the practical importance of the precision–coverage trade-off and for asking us to distinguish performance improvement, model transfer, and task-domain transfer. Your comments prompted us to complete the missing three-arm model comparisons and to separate two questions that had become conflated: whether a method wins at Avg@8/Pass@8, and whether it improves solution coverage when many more samples are available. The completed evidence supports a controlled diagnosis of selective routing, not a universally superior gate.

The evidence cutoff for this response is 13 November 2026. Scores are percentages, differences are percentage points, and a six-benchmark macro equally averages six unrounded benchmark scores. Training seeds, decoding seeds, sample budgets, and evaluation environments are identified separately. This is a proposed author response: no OpenReview posting or manuscript update is implied. “Planned manuscript revision” and “Proposed revised-manuscript text” describe text to synchronize with the final LaTeX before submission.


Reading guide

The response is organized by the original review items. No substantive content is omitted from the complete response. Continuations preserve all evidence, tables, derivations, and proposed manuscript text in order. Proposed revisions are not represented as completed LaTeX edits. All reported numerical panels have been filled and numerically checked. This package has not been posted to OpenReview.

There are 5 W/Q threads and 5 W/Q comments in total:

  • W1 — 1 comment: Evidence is restricted to mathematical reasoning
  • W2 — 1 comment: Table 1 does not show a significant improvement
  • Q1 — 1 comment: Different student/teacher scales and model families
  • Q2 — 1 comment: Comparisons with other on-policy distillation and RLVR methods
  • Q3 — 1 comment: Open-ended natural-language tasks without definitive verification

Including this overview, there are 6 comments for this reviewer.

Please read each thread in ascending [part/total] order.

AuthorsOpenReview ↗

W1. Evidence is restricted to mathematical reasoning [1/1]

W1. Evidence is restricted to mathematical reasoning [1/1]

Concern. The mathematical benchmarks do not establish whether the proposed budget interpretation transfers to other task domains.

Our direct answer is that the measured findings remain specific to the evaluated mathematical-reasoning recipes; transfer beyond mathematics is still unverified. Thank you for insisting on this distinction. Six mathematical benchmarks broaden problem source and difficulty, but they are not open-ended generation or agentic tasks. The completed teacher and Llama comparisons address model dependence, not task-domain dependence.

Within the tested domain, the new evidence makes the scope more—not less—conditional. The completed same-environment three-arm comparisons on T4 and Llama do not give soft a joint Avg@8/Pass@8 victory. In the original setting, EOPD has the highest retained multi-seed means, and the routing cascade lowers seed-42 Pass@8 from 50.00 to 47.13. The CTS schedule also retains its 87/116 inversion. These observations constrain any claim that one routing rule or one scalar budget explains coverage across settings.

The large-k experiments add a narrower, useful result. On Llama, soft reaches pass@128 values of 100.0/50.00/23.33 on AMC23/AIME24/AIME25, compared with OPD 95.0/33.33/23.33 and EOPD 95.0/43.33/20.00. Thus soft is no worse than either baseline at all three @128 endpoints, although AIME25 is a tie with OPD rather than a strict win. On T4, soft improves AMC23 and AIME25 at @128 but drops to 20.00 on AIME24 versus 30.00/26.67 for OPD/EOPD. This is evidence of setting-dependent solution coverage at a larger sampling budget, not evidence of universal domain transfer or semantic diversity.

The sampling explanation is a transferable hypothesis with explicit conditions. If an outcome gate requires at least one success among four independent rollouts, a prompt with success probability p=0.1 produces an all-failure group with probability (1−p)^4=0.6561. That group supplies no positive outcome evidence for opening the gate. Under the same model, marginal Pass@k sensitivity is k(1−p)^(k−1), which is larger at low p for k>1. This identifies a possible failure of positive-evidence routing on difficult prompts; it does not prove that the mechanism dominates another task or that one FKL update would recover the withheld prompt.

The boundary is explicit for open-ended and interactive settings. A different task may have noisy success labels, several incomparable quality dimensions, delayed outcomes, or changing states. Teacher entropy remains computable without a correctness verifier, but computability does not predict factuality, usefulness, or semantic coverage. No completed non-mathematical experiment is being claimed, and the model results are not a substitute for that missing domain comparison.

Planned manuscript revision — empirical scope and limitations. We will state that all completed tasks use rule-verifiable mathematics, report the setting-dependent model and large-k findings separately, and present exposure/timing as measured explanatory variables rather than a domain-independent law.

Proposed revised-manuscript text.

All completed tasks use rule-verifiable mathematics. Teacher entropy is available without a verifier, but its computability does not predict factuality, usefulness, or semantic coverage in open-ended generation. The model and large-k results do not substitute for a non-mathematical task comparison.

AuthorsOpenReview ↗

W2. Table 1 does not show a significant improvement [1/1]

W2. Table 1 does not show a significant improvement [1/1]

Concern. The proposed variants do not significantly outperform the methods in the submitted main table.

We agree: neither the original multi-seed table nor the completed supplementary comparisons establishes a robust performance improvement over EOPD. Your observation correctly separates a useful experimental control from a better-performing replacement. We retain the unfavorable results and do not relabel a favorable comparison against OPD as a win against EOPD.

The original-setting results remain the primary multi-seed evidence. Brackets below are observed training-seed ranges, not confidence intervals; every run uses the common six-benchmark evaluation with eight completions per problem.

MethodTraining seedsMacro Avg@8, mean [range]Macro Pass@8, mean [range]
OPD41, 42, 4329.60 [28.54, 31.15]48.56 [48.10, 48.97]
EOPD41, 42, 43, 4430.15 [29.32, 30.73]50.68 [50.35, 51.10]
Soft entropy gate41, 42, 4329.99 [29.58, 30.63]49.33 [48.98, 50.00]

Restricting all three methods to common seeds 41/42/43 gives OPD 29.60/48.56, EOPD 30.02/50.66, and soft 29.99/49.33. The paired soft-minus-EOPD mean differences are −0.031593 Avg@8 and −1.328286 Pass@8. Near-identical average accuracy therefore coexists with lower observed coverage; this is not an artifact of EOPD's additional seed 44.

The corrected uncertainty analysis supports only a conditional tolerance statement. Using strict UTF-8 decoding, all expected problems and eight verdicts, 10,000 dataset-stratified resamples, and random-number seed 42, the three-soft/four-EOPD retained-mean 90% intervals are [−1.981960,+1.628357] for Avg@8 and [−4.148239,+1.433067] for Pass@8. The first lies narrowly inside an exploratory ±2-point tolerance, but not ±1; the second is not inside ±3. Problems are resampled within retained runs, not training seeds across a population. These intervals neither establish general equivalence nor a performance win.

The strongest reliability result is the cross-seed reversal. Soft exceeds OPD by +2.09 Avg@8 on seed 42, with a retained-model paired 95% interval [+0.9,+3.4], but falls below OPD by −1.57 [−3.1,−0.2] on seed 41. A seemingly well-resolved comparison on one trained model therefore does not settle the method ordering after retraining.

The new same-environment model results also do not support a soft victory claim. On seata2, T4 OPD/EOPD/soft give 31.02/30.46/30.72 Avg@8 and 47.38/49.11/47.51 Pass@8. Llama OPD/EOPD/soft give 17.02/16.93/16.28 and 39.64/38.28/37.75. OPD has the highest Avg@8 in both lines; EOPD has the highest T4 Pass@8 and OPD the highest Llama Pass@8. Earlier appearances of a T4-soft “official overtaking” came from comparing different machines and are not used here.

Large-k coverage is a different practical question, not a way to reverse Table 1. Llama-soft is no lower than both baselines at all three measured pass@128 endpoints, including a tie on AIME25; T4-soft has both improvements and a substantial AIME24 regression. These results motivate reporting performance as a function of sampling budget, but they do not show a superior Avg@8/Pass@8 frontier, a runtime saving, or statistically established dominance at large k.

Planned manuscript revision — main results and statistical appendix. We will preserve the full seed memberships, common-seed comparison, corrected intervals, same-environment model table, and mixed large-k outcomes. The contribution will be framed as an interpretable continuous control, a temporal control, and a reproducible reliability/coverage diagnosis rather than a new best method.

Proposed revised-manuscript text.

EOPD has the highest observed coverage in the original setting. The soft gate does not establish a joint accuracy–coverage improvement there or in the supplementary same-environment model comparisons. Its large-k coverage results are setting-dependent and are reported separately from Avg@8/Pass@8.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions and Responses to All Reviewers

We thank Reviewers DBsG, u55t, and PPE4 for their constructive comments. We have updated the title, abstract, full manuscript, and supplementary material, and posted itemized responses to every weakness and question. The latest revision also includes a new nine-run, training-seed-level intervention study. The main revisions are summarized below.

1. Explicit scope and contribution. We have revised the title to Every Fork Has a Price: The Precision–Coverage Trade-off in On-Policy Distillation of Mathematical Reasoning Models. The abstract, introduction, and contribution statements now explicitly identify mathematical reasoning as the empirical scope, with cross-domain transfer identified as a direction for future validation. The paper’s primary contribution is direct, controlled evidence of a supervision-induced precision–coverage trade-off, together with an analysis of the conditions under which this trade-off can be measured.

2. A mechanism-centered analysis. For success probability pp, nn independent rollouts yield zero successes with probability (1−p)n(1-p)^n. This mechanism creates finite-evidence gate closure on difficult prompts with high marginal Pass@kk sensitivity, k(1−p)k−1k(1-p)^{k-1}. The evaluated logs contain 28–49% fully closed prompt groups. Grouped controls support this account and identify reduced prompt diversity as an important experimental factor. The revised paper therefore evaluates selective signals through both their relation to success and the availability of supervision under finite sampling.

3. New nine-run controlled intervention. We added three intervention arms paired across three training seeds: the original gate, failed-group supervision restoration, and random matched exposure. Relative to the original gate, failed-group restoration changes Avg@8/Pass@8 by −0.625/+2.685 points, while random matched exposure changes them by −0.613/+2.407 points. Both directions repeat on every training seed, and the paired training-seed 95% intervals confirm both directions at the training-seed level. The targeted and random interventions produce closely matched effects, supporting supervision exposure as the common driver of the observed precision–coverage trade-off.

4. Evidence across models, doses, and sampling budgets. We added a matched two-dose, two-seed panel, complete same-environment OPD/EOPD/soft comparisons for T4 and Llama, and separately labeled, count-verified CTS evaluations. Llama-soft reaches pass@128 100.0/50.00/23.33 on AMC23/AIME24/AIME25, whereas T4-soft reaches 20.00 on AIME24. These results characterize how coverage orderings vary with model setting and sampling budget. We present soft and CTS as interpretable controls and diagnostics for studying routing reliability and coverage.

5. Statistical and reproducibility improvements. A strict UTF-8 audit, complete verdict checks, corrected intervals, and common-seed comparisons are now included. Training seeds, decoding seeds, evaluation repetitions, sampling budgets, and evaluation environments are distinguished explicitly. The CTS analysis uses individually verified records and count-level evidence. Reproduction-fidelity claims report absolute-score offsets explicitly.

6. Presentation and supplementary material. Figure 2 and its caption have been redesigned, and the supplementary material now contains full numerical tables, count-level audits, statistical analyses, model-transfer and large-kk results, the nine-run intervention study, and explicit evidence boundaries.

Responses to the reviews. For DBsG, we address domain/model transfer, improvement over EOPD, broader baselines, and open-ended tasks through the revised scope and controlled positioning. For u55t, we address training-level uncertainty and reproducibility using common-seed analyses and the new nine-run intervention, which directly tests the supervision-exposure mechanism across training seeds. For PPE4, we clarify novelty beyond EOPD, add replicated dose evidence, report available CTS counts, explain aggregation and uncertainty, correct the statistical audit, disclose reproduction offsets, and define “Arc-27.”

Collectively, these revisions align the paper’s scope and claims with its controlled evidence. They establish finite-evidence gate closure as a measurable design risk, demonstrate a training-seed-consistent supervision-induced precision–coverage trade-off, and provide the controls and evaluation protocols needed to study routing reliability and coverage rigorously.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.