acceptodds
Under review as a conference paper at ICLR 2027

Solving Without Stopping: On-Policy Distillation at Small Scale

Abstract

On-policy distillation, where a student learns from a stronger teacher's feedback on its own outputs, is a common way to pass reasoning to smaller models. We analyze what it transfers at small scale, distilling Qwen3-8B into Qwen3 4B, 1.7B and 0.6B students, in thinking mode (reason at length, then end the reasoning and answer) and, for comparison, in non-thinking mode (no separate reasoning phase). Long reasoning needs two abilities, solving a problem and knowing when it is solved, and we find that distillation transfers the first, but in thinking mode not the second. Solving improves at every size, up to two ceilings, which we measure comprehensively across both modes and all student sizes: a student's single attempt never exceeds what it could already reach in many attempts before training, and the smaller the student, the further it stays below the teacher. Stopping is where the modes part. In non-thinking mode every student keeps stopping; in thinking mode students stop ending their reasoning early in training, and the smaller the student, the less of this ability survives: the teacher signals a stop almost only where a student already ends its reasoning, so distillation teaches no new stops; it only keeps the student's existing stops that land on a right answer, and a weak student has few such stops. The smallest students often reach the right value but do not commit to it: they either rarely mark it or mark it and write past it. Together, these results describe how small students behave under on-policy distillation, and a diagnostic that separates answer marking, correctness and stopping.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, knowledge distillation, small language models, reasoning, thinking mode, stopping, test-time scaling, pass@k

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dXGbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies what on-policy distillation transfers from Qwen3-8B to smaller Qwen3 students, distinguishing problem solving from reasoning termination. Across three student sizes, thinking and non-thinking modes, and two supervised initialization routes, distillation improves single-sample accuracy but remains below the starting student’s multi-sample accuracy. In thinking mode, smaller students increasingly fail to close their reasoning and produce a final answer. Teacher-probability probes suggest that supervision reinforces existing stopping locations while providing little stopping signal along unfinished trajectories. Further diagnostics examine answer marking, correct-value occurrences, repetition, and forced termination. The paper argues that distillation improves solving while filtering existing stops rather than teaching students where to stop.

Critical strengths

I find the separation of answer marking, correctness, and reasoning closure useful, because aggregate accuracy obscures substantially different failure modes. The contrast between modest accuracy gains and sharply reduced answer production in the smaller thinking students makes this diagnostic practically informative.

The stopping analysis is supported by several complementary observations, including training trajectories, teacher probabilities, initialization controls, and forced termination. I particularly appreciate the officially post-trained 0.6B control, which makes it harder to attribute the entire phenomenon to an inadequate initial thinking format.

Critical weaknesses

Showing that distilled p@1 remains below starting p@32 or p@48 establishes an empirical comparison under this recipe, but does not establish the starting student’s reachable set as a fundamental limit, especially when Section 3 reports AIME coverage increasing from 0.400 to 0.700. Given that the paper’s related-work discussion already identifies sampling-efficiency gains and stopping failures, I see the contribution primarily as a useful characterization of this setting rather than a sufficiently established general account of distillation’s limits.

Lenient scoring credits any correct marked answer, even when a student subsequently rejects it, and Section 5 reports roughly 150–186 marked answers per sample for the smallest teacher-SFT students. Although the paper acknowledges this behavior and provides stricter scoring, I would want committed-answer performance to anchor the central conclusions, since correct-value occurrence above chance does not establish that the model has completed or recognized a valid solution.

Low teacher stopping probability along unfinished student trajectories is consistent with the account, but it does not establish why training shifts students toward those trajectories or demonstrate that missing stopping supervision is the decisive cause. The uniform-lengthening control rejects one simple alternative, while the forced-stop experiment demonstrates partial recoverability; neither directly tests whether adding stopping supervision during training prevents the collapse.

The main study uses one model family, one teacher, mathematical training data, and a fixed clipped, sampled reverse-KL recipe, while thinking and non-thinking routes also differ in initialization and training duration. I therefore find the observed size dependence informative, but remain unconvinced that “distillation transfers solving but not stopping” is sufficiently separated from properties of this particular training setup.

Questions for authors

How do the central solving and coverage conclusions change when committed-answer scoring replaces lenient scoring throughout the main comparisons? This would clarify which gains represent reliable answer production rather than successful candidates embedded in unfinished or contradictory outputs.

Would adding explicit stopping supervision at selected student prefixes prevent closure collapse under otherwise matched training? Such an intervention would provide a stronger test of the proposed mechanism than teacher-probability measurements alone.

What evidence supports interpreting the starting student’s finite-sample coverage as a ceiling rather than an empirical reference point? A sampling-budget sensitivity analysis, together with a closer examination of newly solved problems, would help distinguish limited exploration from a substantive restriction on capability transfer.

1 comment
AuthorsOpenReview ↗

New experiments and answers to Q1–Q3: sampling-budget sensitivity, strict scoring, stopping supervision

Thank you for the detailed and constructive review; your three questions shaped the new experiments. The revised paper, now titled More Reliable, Not More Capable: On-Policy Distillation at Small Scale, is uploaded; all numbers and table numbers below refer to it.

To strengthen the conclusions and the evidence behind them, we added substantial new experiments for the revision:

  • (1) each student, at all three sizes, fine-tuned on the teacher's own solutions before distillation, so that one teacher supplies all supervision after pretraining;

  • (2) an additional on-policy distillation run with a second, smaller teacher: the 1.7B student (fine-tuned as in (1)) distilled from Qwen3-4B instead of Qwen3-8B, with everything else unchanged;

  • (3) pass@120 for every model on every problem: each problem is sampled up to 120 times wherever some model has not solved it reliably (665k scored answers in total), and every answer is re-scored under answer budgets from 1k to 30k tokens (an answer counts only if it stops within the budget);

  • (4) AIME 2026 sampled up to 480 times per problem;

  • (5) BBH, three non-mathematical tasks, for every model, with 120 attempts at 4B.

Main finding. For small students that think before they answer (Qwen3 4B, 1.7B and 0.6B in thinking mode, distilled from Qwen3-8B), on-policy distillation raises single-attempt accuracy but not the ceiling.

  • (i) Given a long answer budget, a single attempt succeeds far more often (at 30k tokens, AMC23 at 4B rises from 52% to 72%). On AMC23 and AIME 2026, the gain is smaller at 16k and disappears at 8k.

  • (ii) The ceiling, the share of problems solved within 120 attempts at a given answer budget, does not clearly rise above the best performance before distillation: the higher of the pretrained model and the fine-tuned start (the student after fine-tuning, before distillation).

Both findings also hold with the smaller teacher and on BBH.

W1 / Q3 (a ceiling or an empirical reference point; sensitivity to the sampling budget). The revised paper defines the ceiling as a measured quantity: the share of problems solved within 120 attempts at a given answer budget, from 1k to 30k tokens. New experiments 3 and 4 are the sensitivity analysis you ask for. Every model is compared at 120 attempts on every problem (problems every model already solves reliably keep 48 samples, since more cannot change whether they are solved).

  • At 30k, the student after distillation solves no clearly more problems than the best performance before distillation.

  • At 16k, its ceiling is lower in every setting.

  • On AIME 2026 at 4B, the student's lead of four problems shrinks to two when the models before distillation are sampled up to 480 times.

  • Newly solved problems, one by one (Appendix Table 17). Of the eight AIME 2026 problems solved by a student after distillation but never by the pretrained model, seven are already solved by a fine-tuned start; one is solved only after distillation.

  • Where the gains come from. On both fine-tuning routes, of single-attempt gains of up to 22 points, at most 0.7 points come from problems that no model before distillation had solved in any sample (Appendix Table 13).

W2 / Q1 (committed-answer scoring). Every number in the revised paper is strict: a sample counts only if it stops (ends before the token limit, with its reasoning closed) and its marked answer is correct. Counting, in addition, correct answers in samples that do not stop would change pass@1 by at most 0.4 points and leave pass@120 unchanged (Appendix B).

W3 / Q2 (explicit stopping supervision). In the reframed paper, the claims concern reliability and the ceiling, and stopping is a descriptive observation. The appendix reports the intervention you suggest, on the 1.7B student fine-tuned on DeepSeek-R1 solutions. Every 1,024 tokens the student is paused and asked for an answer; if the answer is correct, the end-of-reasoning token is inserted at that point and given a positive advantage. Over the last five steps, 10.2% of responses stop within the budget, against 9.4% in an otherwise identical run that only truncates at the same points. It is a 40-step run, reported in Appendix D as a description.

W4 (one setup; the two modes differ in more than the mode). The claims concern students in thinking mode; Section 6.2 reports students in non-thinking mode separately. Within thinking mode, the new experiments vary:

  • the fine-tuning solutions (the teacher's own or DeepSeek-R1's): the two starts differ, yet after distillation they reach comparable pass@1 and pass@120;

  • the teacher (Qwen3-8B or Qwen3-4B): the result is the same.

These experiments make the evidence broader and more direct than in the submission. We hope they address your concerns, and we would be grateful if you would take them into account in your final assessment.

Reviewer_8kqhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work aims to analyze which abilities teachers transfer to students during on-policy distillation (OPD) training. They train Qwen3 4B, 1.7B, and 0.6B models with the Qwen3-8B model as the teacher on the 13.5k DAPO-Math-17k problems. They train students in reasoning and non-reasoning modes and evaluate the performance on GSM8K-200, AIME 2026, and AMC. After training, they analyze the students' responses, separating answer correctness from stopping. They find that there are two ceilings to the performance increase: the base student's pass@n and the teacher's performance. They find that as the student model size decreases, the share of the teacher's accuracy that the student reaches also decreases. They analyze the responses generated by the student's training in reasoning mode and find that they do not learn to stop.

Critical strengths
  1. The authors identify that there is a ceiling of pass@n to the performance improvement via OPD. This could be a very informative claim if well supported. See the weakness point below.
Critical weaknesses
  1. The main claim that performance improvement in students via OPD training has a ceiling of pass@n of the base model is a profound claim. This would mean that distillation does not induce any new reasoning behavior in the student and only incentivizes what was already present in the base model. While the authors have done thorough experiments in 1 domain, the generalizability of this finding is questionable. Prior works using OPD distillation have shown substantial performance gains via OPD training [1,2]. [1] paper even claimed that while GRPO-style RL does not exceed the capacity of base models, distillation can be used to induce novel reasoning. The authors should discuss in detail how their findings relate to the broader prior works.

  2. The analysis does not consider the distributional differences between the training data and evaluation data. All the models are trained on DAPA-Math-17 problems and then evaluated on AMC, AIME 2026, and GSM8K-200. While all the problems are from the math domain, there is no controlled notion of in-distribution and out-of-distribution; thus, it is hard to comment on whether the results are due to out-of-distribution evaluations or are general findings about OPD training.

  3. In distillation, the objective is to minimize the distance between the student and teacher distributions. In on-policy training, the teacher never directly gives targets to students; thus, treating stop tokens as special is not warranted. With training, the aim is that students eventually go closer to the teacher's distribution and thus learn to generate similar traces and stops. The experiments on which the claim about OPD training inducing solving but not inducing stopping is based have confounding factors. The base model, when fine-tuned on teacher-generated traces, generates stops at the same rate as OPD (Figure 6). Even off-policy training is not able to induce stopping in the base model. This suggests that the student models do not have the capability to mimic the teacher's reasoning, as opposed to something specific to OPD training. This could also explain why the student's performance does not get close to the teacher's performance. The authors also draw an artificial distinction between knowing how to solve and stopping. If the model knows how to solve, then once it solves the problem, it should be confident to stop. The authors should provide a hypothesis or some results that show that this distinction is warranted.

References -

  1. Yue, Yang, et al. "Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?." Advances in Neural Information Processing Systems 38 (2026): 57654-57689.
  2. Shenfeld, Idan, et al. "Self-distillation enables continual learning." arXiv preprint arXiv:2601.19897 (2026).
Questions for authors
  1. Why did the authors use different values of n to identify the ceiling? Could you have kept n constant and established an upper bound?
2 comments
AuthorsOpenReview ↗

Part 1/2: New experiments; the ceiling holds statistically (W1, W2, Q1)

We thank the reviewer for a critical review. We ran seven new experiments (E1–E7); all samples, logs and scripts will be released with the final version.

Summary. (i) The ceiling holds statistically; at 480 samples the start and distilled ceilings on AIME are close, and on easier benchmarks they coincide. (ii) Results hold out of domain benchmarks (BBH). (iii) The stopping failure is not caused by the SFT start: the teacher teaches stopping when the start fits the budget (Part 2). (iv) Stopping is a separate ability the per-token objective cannot supply (Part 2).

W1: scope of the ceiling claim

We do not claim "distillation creates no new reasoning"; we narrow the claim to what the data support. Statistically, on every benchmark and size, distilled pass@1 stays below the start's own pass@n (lenient; max ratio 0.96 over 27 cells), and pass@n moves by about ±0.1.

E2/E3: AIME 2026, 240 samples per problem (strict; 4B on the external-SFT route, i.e. our main route, cap 30,720; pretrained non-thinking reference at cap 8,192):

kk4B external-SFT start4B after OPD4B pretrained, non-thinking
10.1580.3120.081
160.3730.5750.279
640.4970.6990.367
2400.6330.7330.433
4800.700 (21/30)0.767 (23/30)–

The start's curve is still rising at k=240k=240, so on problems unsolved in the first 240 samples we drew 240 more (row 480; problems already solved count as solved). At 480 samples the two ceilings are close (0.700 vs. 0.767). On easier benchmarks the start converges much faster and the ceilings already coincide (4B, lenient, start vs. distilled: AMC23 0.925 vs. 0.900 at n=48n=48; GSM8K 0.980 vs. 0.965 at n=32n=32). The AIME gap at small kk reflects slow convergence on the hardest problems, not a higher ceiling.

Problem by problem, the gap is one problem, not a new capability. Every problem the start solves, the distilled model also solves. Of the two that separate them, one is solved by the distilled model in 11/480 samples (2.3%) while the start was still gaining problems (19 → 21 from 240 to 480 samples), i.e. at the edge of its reach. Only one (18/240 vs. 0/480) is clearly outside it.

Relation to Yue et al. and Shenfeld et al. Our claim is that at small scale the gains are bounded by the student's own pass@n. Neither work studies on-policy distillation from a stronger teacher. Yue et al.'s "new reasoning" comes from offline distillation (SFT on stronger traces), contrasted with RLVR. OPD samples like RL but is supervised per token by a teacher, so which regime it falls into is open. Shenfeld et al. study self-distillation against SFT, on new-task accuracy and forgetting. There is no stronger teacher, no own-pass@n ceiling and no analysis of stopping. Their ablation, in which offline distillation from the same teacher underperforms the on-policy version, agrees with our view that what OPD learns is shaped by the student's own trajectories. We will add this discussion.

W2: in- vs. out-of-distribution

New benchmark E4: BIG-Bench Hard (BBH), three non-math free-answer tasks (object counting, word sorting, Dyck languages), 250 problems × 8 samples:

4B1.7B0.6Bteacher
start0.4060.2320.1480.802
after OPD0.4740.2690.176/

Same direction as in-domain: gains at every size, and a gap to the teacher that grows as the student shrinks. The stopping failure also transfers outside math: on the two long-output tasks, the 1.7B and 0.6B hit the cap on 37–82% of samples before distillation and 88–97% after.

Q1: different nn

nn differs only across benchmarks (32 on GSM8K, 48 elsewhere). GSM8K is easy and saturates by n=32n=32 (lenient start pass@32 0.90–0.99 for every model that reliably answers). Within a benchmark, start and distilled models share the same nn. At a common k=32k=32 (unbiased estimator 1−(n−ck)/(nk)1-\binom{n-c}{k}/\binom{n}{k}) the conclusion is unchanged (4B start lenient pass@32 0.757 vs. distilled pass@1 0.651). On AIME we went to 480 samples (above). The revision reports all ceilings at a common kk.

(4B start lenient pass@32 0.757 vs. distilled pass@1 0.651, three-benchmark mean)

AuthorsOpenReview ↗

Part 2/2: Stopping is an ability the teacher can teach only within budget (W3)

W3: solving vs. stopping, and the SFT-start confound

(1) The teacher can teach stopping, if the start fits the budget. We agree teacher-trace SFT also stops rarely. E1 is a new SFT start from the teacher's own shortest correct solutions to the same 5,878 problems as our external SFT data, with the identical OPD recipe (142 steps, 7,168-token budget). The starts differ mainly in length:

SFT startmean tokenstest closure, start → after OPD
external solutions< 2,400stays high on GSM8K (1.7B 0.87 → 0.81)
teacher, same problems (E1)< 2,400rises: 4B AIME 0.24 → 0.43; 1.7B GSM8K 0.63 → 0.87; 0.6B GSM8K 0.43 → 0.63
teacher traces, original~4,8001.7B, 0.6B fall to 0.00

The two short starts begin far apart (e.g. 1.7B GSM8K strict pass@1 .705 vs. .566) yet end within about 0.05 of each other in strict score on every benchmark and size (e.g. .734 vs. .786; full table in the revision). A start already at ~4.8k tokens is truncated before it can finish, and OPD lengthens it further; a start that fits learns to stop from the same teacher. Offline SFT on long traces fails differently: it copies a length the student cannot sustain.

What is specific to OPD is that the teacher does teach stopping on the student's own trajectories, wherever the student can solve. Grouping the 270 test problems by whether the start already solved them (E7), right stops rise at every size on problems the student can solve: on "sometimes solved" problems, 4B 0.200 → 0.481, 1.7B 0.193 → 0.246, 0.6B 0.199 → 0.301; on "mostly solved" problems, 0.6B 0.684 → 0.858 and 1.7B 0.821 → 0.851 (4B already saturated at 0.95). Wrong stops in these groups fall to 0.01–0.22. On problems the start never solved, right stops stay at 0.00–0.04. The teacher loses this ability exactly where the student is truncated or cannot solve.

(2) Why matching the teacher cannot repair this. The stop token is one position among ~10410^4, with a per-token advantage of about +0.03+0.03. On truncated rollouts, the teacher's maximum stop probability has a median of ~10−810^{-8}: it never reaches such states, so it never signals a stop. E5 (1.7B, 40 steps) tries to teach stopping directly: every 1,024 tokens we pause the student and force an answer; if it is already correct, we insert the stop token there and reward it. Closure is unchanged (0.102 vs. 0.094 for plain truncation). E6 explains why: teaching a stop requires a point where the student has already reached the right answer. Early in training, 23–26% of truncated rollouts contain such a point; after 142 steps only 4–6% do. Most truncated rollouts never reach the answer, so the stop has no correct place to be taught: the student is truncated not because it fails to stop after solving, but because it has not solved within the budget.

(3) Stopping is an ability: knowing the boundary of one's own competence. Whether a model stops, right or wrong, expresses its judgment of what it can and cannot solve. A model with this judgment stops once it has solved, and does not commit to an answer it has not reached. Without it, two failures follow: overconfident stops (closing with a wrong answer) and wasted computation (continuing after the answer is reached, or running to the cap). The teacher has this judgment; the small students largely do not:

  • Overconfident stops: when the teacher closes it is right 88.5% of the time; the starting students 52% (4B), 13% (1.7B), 4% (0.6B) on their training rollouts.
  • Wasted computation: early in training 23–26% of truncated responses already contained the correct answer (E6), yet the student kept writing to the cap. After OPD, students write far more than the teacher (AMC23 mean tokens: teacher 8.7k; 4B 14.4k; 1.7B 21.8k).
  • Solves without ever stopping: the 1.7B distilled from the long teacher-trace start writes a correct answer at lenient pass@1 0.394 but closes 0% of the time.
  • The boundary is sharpened only where it exists (E7, above).

Stopping is thus not a by-product of solving but a judgment about one's own boundary, essential for both reliability and token efficiency. OPD transfers it only where the student can already solve within the budget.

What is specific to OPD: (i) the ceiling measured against the student's own pass@n, per problem, with up to 480 samples; (ii) a truncated student gets no stop signal from a teacher that never reaches such states, and neither the objective nor explicit stop supervision repairs this; (iii) the same teacher does teach stopping wherever the student can solve and the start fits the budget. None of this is observable in offline SFT or RLVR, which lack a per-token teacher signal on the student's trajectory. We will add E1–E7, with all samples and scripts, to the final version.

Reviewer_P33uOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies what a small student actually learns from a larger thinking teacher under on-policy distillation, arguing that long reasoning needs two abilities, solving a problem and recognizing that it is solved, which end-point scores mix together. The authors distill Qwen3-8B into Qwen3 4B, 1.7B and 0.6B students in thinking mode, with non-thinking mode as a reference, and track each student at evaluation and through training with a diagnostic that separates whether a sample marks an answer, whether that answer is right, and whether it stops. Some important findings: solving improves at every size but never beyond what the student could already reach with many attempts, and smaller students stay further below the teacher. In thinking mode, students stop closing their reasoning early in training, more so the smaller they are, which the authors attribute to distillation filtering existing stops rather than teaching new ones.

Critical strengths
  1. The question is practical: separating "can solve" from "knows when to stop" addresses a real gap in how distillation is evaluated.
  2. The diagnostic is simple and could easily be adopted by other distillation papers.
  3. The stopping collapse is replicated across length budgets, starting points and the officially post-trained 0.6B, so the phenomenon itself is robust.
Critical weaknesses

There are a few things that I want to point out:

  1. In my opinion, the main issue is that the paper's own numbers contradict its central mechanism. The abstract says distillation "only keeps the student's existing stops that land on a right answer", but Section 4 reports that late in training, stopped responses are right at most 70% of the time at 4B, about 26% at 1.7B, and almost never at 0.6B. At 4B, this means more than half of the right stops are lost as well, so distillation loses most stops of both kinds rather than keeping the right ones. The filter does seem to hold for the officially post-trained 0.6B, but not for the main external-SFT students that the paper's story is about. Since this filter story is the paper's explanation for why small students cannot stop, it needs to be revised.
  2. The thinking vs. non-thinking comparison changes more than the mode. The non-thinking route starts from the base model without SFT, uses a non-thinking teacher with short responses, and trains for 212 steps instead of 142. Any of these could explain why non-thinking students keep stopping, and distilling a non-thinking student from the thinking teacher would help separate them.
  3. I also wonder how much "solving improves at every size" depends on lenient scoring, which counts a sample as correct if any of its marked answers is right, even when the student never closes its reasoning. The 1.7B and 0.6B distilled from the teacher-SFT start never finish a sample at evaluation, yet mark around 150–185 answers per sample, so lenient scoring can credit them for listing many candidates.
Questions for authors
  1. What fraction of the surviving stops are correct late in training, for each run?
  2. Can you distill a non-thinking student from the thinking teacher to separate teacher length from student mode?
  3. Can you report strict or committed accuracy for every row of Table 1?
1 comment
AuthorsOpenReview ↗

Thank you; new experiments, strict scoring throughout, and answers to Q1–Q3

Thank you for the careful reading, the precise questions and the positive assessment. Your comments led us to a substantial amount of new work for the revision. The revised paper, now titled More Reliable, Not More Capable: On-Policy Distillation at Small Scale, is uploaded; all numbers and table numbers below refer to it.

To strengthen the conclusions and the evidence behind them, we added substantial new experiments for the revision: (1) each student, at all three sizes, fine-tuned on the teacher's own solutions before distillation, so that one teacher supplies all supervision after pretraining; (2) an additional on-policy distillation run with a second, smaller teacher: the 1.7B student (fine-tuned as in (1)) distilled from Qwen3-4B instead of Qwen3-8B, with everything else unchanged; (3) pass@120 for every model on every problem: each problem is sampled up to 120 times wherever some model has not solved it reliably (665k scored answers in total), and every answer is re-scored under answer budgets from 1k to 30k tokens (an answer counts only if it stops within the budget); (4) AIME 2026 sampled up to 480 times per problem; (5) BBH, three non-mathematical tasks, for every model, with 120 attempts at 4B.

Main finding. For small students that think before they answer (Qwen3 4B, 1.7B and 0.6B in thinking mode, distilled from Qwen3-8B), on-policy distillation raises single-attempt accuracy but not the ceiling. (i) Given a long answer budget, a single attempt succeeds far more often (at 30k tokens, AMC23 at 4B rises from 52% to 72%). On AMC23 and AIME 2026, the gain is smaller at 16k and disappears at 8k. (ii) The ceiling, the share of problems solved within 120 attempts at a given answer budget, does not clearly rise above the best performance before distillation: the higher of the pretrained model and the fine-tuned start (the student after fine-tuning, before distillation). At 30k the student solves no clearly more problems (AMC23: 95%, 72% and 52% against 92%, 88% and 62%, from 4B to 0.6B), and at 16k it solves fewer in every setting. Both findings also hold with the smaller teacher and on BBH.

W1 / Q1 (the filtering account; which stops are correct). The revised paper rests no claim on a stopping mechanism and reports stopping descriptively, at evaluation, on the same problems before and after distillation. A sample stops if generation ends before the token limit and the reasoning is closed. On AMC23 (Appendix Table 21), answers that stop with a correct answer do not decrease in any run, from 4B to 0.6B:

  • fine-tuned on the teacher's own solutions: 51→72%, 21→38%, 7→16%;
  • fine-tuned on DeepSeek-R1 solutions: 54→72%, 29→33%, 15→15%.

Answers that stop with a wrong answer fall or stay (4→3%, 7→7%, 8→4%; 20→4%, 27→8%, 41→7%). For Q1, the share of stopped answers that are correct rises in every run (first 48 samples per problem):

students fine-tuned on4B1.7B0.6B
the teacher's own solutions92% → 96%76% → 85%45% → 78%
DeepSeek-R1 solutions73% → 95%51% → 81%26% → 67%

W2 / Q2 (thinking and non-thinking students differ in more than the mode). We ran the closest test of student mode: the students fine-tuned on either source, before and after distillation, answering under the non-thinking chat template (Section 6.2, Appendix Table 23). They keep reasoning at length: stopping rate, score and length match thinking mode at the same 8k limit (e.g. 4B after distillation on AIME 2026: 9% stop and 8% correct, against 8% and 8% in thinking mode). A student's mode is thus set by its fine-tuning, not by the template, which is why the non-thinking route is trained separately, from the non-thinking teacher. The paper's claims concern students in thinking mode, and no conclusion depends on comparing the two modes. Within thinking mode, the new experiments vary one factor at a time:

  • the source of the fine-tuning solutions (new experiment 1): the two starts differ, yet after distillation they reach comparable pass@1 and pass@120;
  • the teacher (new experiment 2): the result is the same.

W3 / Q3 (strict accuracy for every row). Every number in the revised paper, including every cell of Table 1, is strict: a sample counts only if it stops and its marked answer is correct. The difference between strict and lenient scoring is exactly the share of samples that mark a correct answer but do not stop. Appendix Table 18 lists it for every route, size and benchmark: at most 0.3% of samples. Counting them would change pass@1 by at most 0.4 points and leave pass@120 unchanged (Appendix B). All students are fine-tuned on short solutions: the teacher's shortest correct solutions (new experiment 1; mean 2,180 tokens) or DeepSeek-R1 solutions of at most 2,048 tokens.

These experiments make the evidence broader and more direct than in the submission. We are grateful for your review: it made the paper clearer and better supported.

Comments on the paper

1 comment
AuthorsOpenReview ↗

The ceiling is now the main finding, supported by new experiments (Q1, out of domain, E5, E6)

Thank you for the careful follow-up and for pointing to the ceiling as the most interesting finding. Our revised paper: distillation moves answers the student could already reach within many attempts into its first attempt, and does not clearly add new problems it can solve. We reframed the paper around this finding, now titled More Reliable, Not More Capable: On-Policy Distillation at Small Scale; its claims are about reliability and the ceiling, and stopping is a description. The revised paper is uploaded; all numbers and table numbers below refer to it.

To strengthen the conclusions and the evidence behind them, we added substantial new experiments for the revision:

  • (1) each student, at all three sizes, fine-tuned on the teacher's own solutions before distillation, so that one teacher supplies all supervision after pretraining;

  • (2) an additional on-policy distillation run with a second, smaller teacher: the 1.7B student distilled from Qwen3-4B instead of Qwen3-8B, with everything else unchanged;

  • (3) pass@120 for every model on every problem (665k scored answers), each answer re-scored under answer budgets from 1k to 30k tokens;

  • (4) AIME 2026 sampled up to 480 times per problem;

  • (5) BBH, three non-mathematical tasks, for every model, with 120 attempts at 4B.

Main finding. For small students that think before they answer (Qwen3 4B, 1.7B and 0.6B in thinking mode, distilled from Qwen3-8B), on-policy distillation raises single-attempt accuracy but not the ceiling.

  • (i) Given a long answer budget, a single attempt succeeds far more often (at 30k tokens, AMC23 at 4B rises from 52% to 72%, and AIME 2026 from 16% to 30%). On AMC23 and AIME 2026, the gain is smaller at 16k and disappears at 8k.

  • (ii) The ceiling, the share of problems solved within 120 attempts at a given answer budget, does not clearly rise above the best performance before distillation (the higher of the pretrained model and the fine-tuned start).

Both findings also hold with the smaller teacher and on BBH.

1. A common number of attempts (your Q1; new experiments 3 and 4). Every model is now compared at 120 attempts on every problem. Problems that every model already solves reliably keep 48 samples (32 on GSM8K).

  • At 30k, after vs. before distillation: AMC23 95% vs. 92% (4B), 72% vs. 88% (1.7B), 52% vs. 62% (0.6B). On AIME 2026 at 4B, a lead of four problems shrinks to two when the models before distillation are sampled up to 480 times.

  • At 16k, the ceiling after distillation is lower in every setting.

2. Where the gains come from. As you describe, distillation raises what was reachable at pass@k to pass@1. On both fine-tuning routes, of single-attempt gains of up to 22 points, at most 0.7 points come from problems that no model before distillation had solved in any sample (Appendix Table 13).

3. Out of domain (new experiment 5). Training uses mathematics only. BBH differs from it in task and answer type: three BIG-Bench Hard tasks (counting objects, sorting words, closing bracket sequences; 250 problems each), with free-form, non-mathematical answers checked exactly. Single-attempt accuracy rises at every size. At 4B, within 8, 48 and 120 attempts, the student solves 71%, 81% and 83% of the problems, against 60%, 74% and 79% before distillation. The lead falls from 11 to 7 to 4 points as attempts grow, the same pattern as in mathematics.

4. Teacher and start (new experiments 1 and 2).

  • With Qwen3-4B as teacher, the result is the same (1.7B on AMC23: 78% within 120 attempts, against 88% before distillation).

  • Two fine-tuning sources. Students fine-tuned on the teacher's own solutions and students fine-tuned on DeepSeek-R1 solutions start differently, but after distillation reach comparable pass@1 and pass@120: within 120 attempts on AMC23, 72% vs. 70% at 1.7B and 52% vs. 58% at 0.6B; at most 3 of 40 problems are solved by only one of the two.

5. Stopping, forced answering and wording (your points 3–5). In the reframed paper, the claims concern reliability and the ceiling; stopping is a descriptive observation, with a sample counted as stopped if generation ends before the token limit and the reasoning is closed. Stop supervision (E5) is a 40-step run, reported in Appendix D as a description. The drop in forced-answer accuracy (E6) agrees with the main finding: responses that do not stop fall almost entirely on problems the fine-tuned start solved in fewer than half of its attempts, where distillation adds little. Throughout, the revised paper describes measured behaviour: whether a response ends within the limit, and whether its answer is correct.

We hope this addresses your remaining concerns, and we would be grateful if you would consider it in your final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.