- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies what on-policy distillation transfers from Qwen3-8B to smaller Qwen3 students, distinguishing problem solving from reasoning termination. Across three student sizes, thinking and non-thinking modes, and two supervised initialization routes, distillation improves single-sample accuracy but remains below the starting student’s multi-sample accuracy. In thinking mode, smaller students increasingly fail to close their reasoning and produce a final answer. Teacher-probability probes suggest that supervision reinforces existing stopping locations while providing little stopping signal along unfinished trajectories. Further diagnostics examine answer marking, correct-value occurrences, repetition, and forced termination. The paper argues that distillation improves solving while filtering existing stops rather than teaching students where to stop.
Critical strengths
I find the separation of answer marking, correctness, and reasoning closure useful, because aggregate accuracy obscures substantially different failure modes. The contrast between modest accuracy gains and sharply reduced answer production in the smaller thinking students makes this diagnostic practically informative.
The stopping analysis is supported by several complementary observations, including training trajectories, teacher probabilities, initialization controls, and forced termination. I particularly appreciate the officially post-trained 0.6B control, which makes it harder to attribute the entire phenomenon to an inadequate initial thinking format.
Critical weaknesses
Showing that distilled p@1 remains below starting p@32 or p@48 establishes an empirical comparison under this recipe, but does not establish the starting student’s reachable set as a fundamental limit, especially when Section 3 reports AIME coverage increasing from 0.400 to 0.700. Given that the paper’s related-work discussion already identifies sampling-efficiency gains and stopping failures, I see the contribution primarily as a useful characterization of this setting rather than a sufficiently established general account of distillation’s limits.
Lenient scoring credits any correct marked answer, even when a student subsequently rejects it, and Section 5 reports roughly 150–186 marked answers per sample for the smallest teacher-SFT students. Although the paper acknowledges this behavior and provides stricter scoring, I would want committed-answer performance to anchor the central conclusions, since correct-value occurrence above chance does not establish that the model has completed or recognized a valid solution.
Low teacher stopping probability along unfinished student trajectories is consistent with the account, but it does not establish why training shifts students toward those trajectories or demonstrate that missing stopping supervision is the decisive cause. The uniform-lengthening control rejects one simple alternative, while the forced-stop experiment demonstrates partial recoverability; neither directly tests whether adding stopping supervision during training prevents the collapse.
The main study uses one model family, one teacher, mathematical training data, and a fixed clipped, sampled reverse-KL recipe, while thinking and non-thinking routes also differ in initialization and training duration. I therefore find the observed size dependence informative, but remain unconvinced that “distillation transfers solving but not stopping” is sufficiently separated from properties of this particular training setup.
Questions for authors
How do the central solving and coverage conclusions change when committed-answer scoring replaces lenient scoring throughout the main comparisons? This would clarify which gains represent reliable answer production rather than successful candidates embedded in unfinished or contradictory outputs.
Would adding explicit stopping supervision at selected student prefixes prevent closure collapse under otherwise matched training? Such an intervention would provide a stronger test of the proposed mechanism than teacher-probability measurements alone.
What evidence supports interpreting the starting student’s finite-sample coverage as a ceiling rather than an empirical reference point? A sampling-budget sensitivity analysis, together with a closer examination of newly solved problems, would help distinguish limited exploration from a substantive restriction on capability transfer.
New experiments and answers to Q1–Q3: sampling-budget sensitivity, strict scoring, stopping supervision
Thank you for the detailed and constructive review; your three questions shaped the new experiments. The revised paper, now titled More Reliable, Not More Capable: On-Policy Distillation at Small Scale, is uploaded; all numbers and table numbers below refer to it.
To strengthen the conclusions and the evidence behind them, we added substantial new experiments for the revision:
(1) each student, at all three sizes, fine-tuned on the teacher's own solutions before distillation, so that one teacher supplies all supervision after pretraining;
(2) an additional on-policy distillation run with a second, smaller teacher: the 1.7B student (fine-tuned as in (1)) distilled from Qwen3-4B instead of Qwen3-8B, with everything else unchanged;
(3) pass@120 for every model on every problem: each problem is sampled up to 120 times wherever some model has not solved it reliably (665k scored answers in total), and every answer is re-scored under answer budgets from 1k to 30k tokens (an answer counts only if it stops within the budget);
(4) AIME 2026 sampled up to 480 times per problem;
(5) BBH, three non-mathematical tasks, for every model, with 120 attempts at 4B.
Main finding. For small students that think before they answer (Qwen3 4B, 1.7B and 0.6B in thinking mode, distilled from Qwen3-8B), on-policy distillation raises single-attempt accuracy but not the ceiling.
(i) Given a long answer budget, a single attempt succeeds far more often (at 30k tokens, AMC23 at 4B rises from 52% to 72%). On AMC23 and AIME 2026, the gain is smaller at 16k and disappears at 8k.
(ii) The ceiling, the share of problems solved within 120 attempts at a given answer budget, does not clearly rise above the best performance before distillation: the higher of the pretrained model and the fine-tuned start (the student after fine-tuning, before distillation).
Both findings also hold with the smaller teacher and on BBH.
W1 / Q3 (a ceiling or an empirical reference point; sensitivity to the sampling budget). The revised paper defines the ceiling as a measured quantity: the share of problems solved within 120 attempts at a given answer budget, from 1k to 30k tokens. New experiments 3 and 4 are the sensitivity analysis you ask for. Every model is compared at 120 attempts on every problem (problems every model already solves reliably keep 48 samples, since more cannot change whether they are solved).
At 30k, the student after distillation solves no clearly more problems than the best performance before distillation.
At 16k, its ceiling is lower in every setting.
On AIME 2026 at 4B, the student's lead of four problems shrinks to two when the models before distillation are sampled up to 480 times.
Newly solved problems, one by one (Appendix Table 17). Of the eight AIME 2026 problems solved by a student after distillation but never by the pretrained model, seven are already solved by a fine-tuned start; one is solved only after distillation.
Where the gains come from. On both fine-tuning routes, of single-attempt gains of up to 22 points, at most 0.7 points come from problems that no model before distillation had solved in any sample (Appendix Table 13).
W2 / Q1 (committed-answer scoring). Every number in the revised paper is strict: a sample counts only if it stops (ends before the token limit, with its reasoning closed) and its marked answer is correct. Counting, in addition, correct answers in samples that do not stop would change pass@1 by at most 0.4 points and leave pass@120 unchanged (Appendix B).
W3 / Q2 (explicit stopping supervision). In the reframed paper, the claims concern reliability and the ceiling, and stopping is a descriptive observation. The appendix reports the intervention you suggest, on the 1.7B student fine-tuned on DeepSeek-R1 solutions. Every 1,024 tokens the student is paused and asked for an answer; if the answer is correct, the end-of-reasoning token is inserted at that point and given a positive advantage. Over the last five steps, 10.2% of responses stop within the budget, against 9.4% in an otherwise identical run that only truncates at the same points. It is a 40-step run, reported in Appendix D as a description.
W4 (one setup; the two modes differ in more than the mode). The claims concern students in thinking mode; Section 6.2 reports students in non-thinking mode separately. Within thinking mode, the new experiments vary:
the fine-tuning solutions (the teacher's own or DeepSeek-R1's): the two starts differ, yet after distillation they reach comparable pass@1 and pass@120;
the teacher (Qwen3-8B or Qwen3-4B): the result is the same.
These experiments make the evidence broader and more direct than in the submission. We hope they address your concerns, and we would be grateful if you would take them into account in your final assessment.