acceptodds
Under review as a conference paper at ICLR 2027

Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision

Abstract

Current post-training methods in verifiable settings fall into two categories. Reinforcement learning (RLVR) relies on binary rewards, which are broadly applicable and powerful, but provide only sparse supervision during training. Distillation provides dense token-level supervision, typically obtained from an external teacher or using high-quality demonstrations. Collecting such supervision can be costly or unavailable. We propose Self-Distillation Zero (SD-Zero), a method that is substantially more training sample-efficient than RL and does not require an external teacher or high-quality demonstrations. SD-Zero trains a single model to play two roles: a Generator, which produces an initial response, and a Reviser, which conditions on that response and its binary reward to produce an improved response. We then perform on-policy self-distillation to distill the reviser into the generator, using the reviser's token distributions conditioned on the generator's response and its reward as supervision. In effect, SD-Zero trains the model to transform binary rewards into dense token-level self-supervision. On math and code reasoning benchmarks with Qwen3-4B-Instruct and Olmo-3-7B-Instruct, SD-Zero improves performance by at least 10% over the base models and outperforms strong baselines, including Rejection Fine-Tuning (RFT), GRPO, and On-Policy Self-Distillation (OPSD), under the same question set and training sample budget. Extensive ablation studies show two novel characteristics of our proposed algorithm: (a) token-level self-localization, where the reviser can identify the key tokens that need to be revised in the generator's response based on reward, and (b) iterative self-evolution, where the improved ability to revise answers can be distilled back into generation performance with regular teacher synchronization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy self-distillation, self-revision, self-critique, dense supervision

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_Q1ZgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Self-Distillation Zero (SD-Zero), a two-stage method that converts binary outcome rewards into dense token-level supervision without an external teacher. In the first stage, the model is trained on correctness-filtered self-revision traces so that it can revise an initial response conditioned on the response and its binary reward. In the second stage, a frozen copy of this revision-capable model supervises the generator on the generator's own trajectories through token-level distillation. Experiments on math and code with Qwen3-4B-Instruct and Olmo-3-7B-Instruct show consistent improvements over the base models and stronger performance than RFT, GRPO, OPSD, and several related self-distillation baselines under roughly matched generation and training-token budgets. The paper also includes extensive ablations of the two training stages, the teacher context, data allocation, objective design, and teacher synchronization.

Critical strengths
  • The central hypothesis is interesting: a model that cannot solve a problem on its first attempt may still provide a better conditional distribution after seeing its own attempt and outcome, and this revision behavior can then supervise one-pass generation. The two-stage design is intuitive and addresses a real limitation of both sparse-reward RLVR and teacher-based distillation.
  • The performance claim is supported by a relatively comprehensive set of experiments. The gains hold across two model families and both math and code, and the paper compares against strong RL and self-distillation baselines while accounting for both generated samples and approximate token cost. The phase, objective, teacher-context, data-allocation, and filtering ablations also show that the reported result is not explained by one obvious implementation choice.
Critical weaknesses
  • The experiments do not establish the proposed mechanism of token-level self-localization. Figure 4 shows that KL mass is concentrated on a few positions, but the same pattern would arise if the reviser ignored the erroneous reasoning and branched into a different solution. There is no ground-truth error location, controlled token corruption, or counterfactual intervention showing that high-KL positions identify tokens whose correction improves the answer. Similarly, shorter responses and fewer explicit revision keywords do not demonstrate that revision has been “internalized”; they may only indicate a change in output style. These mechanism claims are central to the paper but are supported mainly by proxies and one qualitative example.
  • The dense information does not come from the binary reward alone. Phase 1 samples multiple revisions, retains only correct ones, and trains on these high-quality selected traces; the reviser also receives additional inference compute and a hand-written revision instruction. SD-Zero therefore removes an external teacher, but it does not remove high-quality filtered supervision. The iterative self-evolution claim is also not isolated: teacher synchronization is followed by additional training, without a matched control that continues for the same number of steps with the frozen teacher, and the reported synchronization result is on training questions.
Questions for authors
  1. Can the authors evaluate localization against known error positions, for example by introducing controlled token-level corruptions and testing whether the largest KL changes identify the corrupted tokens or predict the effect of repairing them? Comparing against a reviser that cannot see the student's attempt would also show whether the attempt is actually used rather than ignored.
  2. For teacher synchronization, what happens if training continues for the same number of steps and on the same data while the original SRT teacher remains frozen? Please report held-out rather than only training-question accuracy. This control would determine whether the extra gain comes from synchronization or simply from continued optimization.
3 comments
AuthorsOpenReview ↗

Author Response 1

Thank you for the insightful review. We are glad you found the central hypothesis interesting and the experiments comprehensive. Please find our response below.

W1, Q1: Can high-KL positions identify errors and guide repairs, and does the reviser use the student’s attempt?

Yes. We conducted controlled corruption, context-ablation, and repair experiments. We selected 160 verified-correct, completed solutions to OpenR1 questions outside the SRT data, each containing a boxed answer within 8,192 tokens. We replaced exactly one digit or arithmetic operator in each solution. We then evaluated the corrupted solutions using the SRT checkpoint for both student and teacher, as at the initialization of Phase 2.

1. The KL score localizes injected errors.

CorruptionCasesLargest per-token KL at the corrupted token
Digit8065/80 (81%)
Operator8073/80 (91%)
All160138/160 (86%)

The largest per-token KL identifies the injected error in 86% of cases, compared with approximately 0.02% by chance over solutions averaging 5,082 tokens. We also compare the corrupted token with the same position in the clean solution. After standardizing the student-minus-teacher log-probability gap within each solution to obtain z, the median is −39.7 at the corrupted token, versus +0.10 at the clean position. This controlled comparison shows that the injected error produces a localized change in the scoring signal.

2. The teacher’s scoring depends on the supplied attempt.

We varied the teacher’s context while keeping the SRT model fixed on 42 solutions:

Teacher’s contextMedian z at the corrupted tokenCases with z < −3
The student’s own attempt−31.342/42
Another problem’s attempt−9.035/42
No attempt−3.924/42

The signal at the corrupted token is strongest when the teacher sees the student’s own attempt. Its magnitude decreases with an unrelated attempt and decreases further with no attempt. These results show that the teacher’s evaluation responds to the content of the supplied attempt.

3. Repairing the largest-KL position improves final accuracy.

For each of the 160 corruptions, we set the token at the selected position and regenerated the continuation, using three samples per case. Among the 138 correctly localized corruptions, the teacher’s top token matches the original token in 85 cases.

Token interventionLargest-KL position: final accuracy (%)Random position: final accuracy (%)
Keep the token from the corrupted solution92.995.6
Insert the teacher’s top token95.094.8
Restore the original token94.094.2

Inserting the teacher’s top token at the largest-KL position raises final accuracy by 2.1 percentage points. The corresponding change at a random position is −0.8 points, giving a 2.9-point difference in the accuracy changes, paired over the 160 cases. This intervention connects the localization signal to the benefit of a targeted repair.

4. Disagreement is sparse and localized along the student’s trajectory.

We also examined the shape of disagreement on 368 incorrect attempts using the Qwen3-4B-Instruct reviser: 79 completed responses and 289 that reached the length cap. In both groups, the median longest run of consecutive high-KL tokens is two tokens, and the top 1% of positions account for 85% and 84% of KL mass, respectively. These measurements show that disagreement is sparse and localized along the student’s trajectory.

For “internalization,” we have updated the paper to state the accuracy-based claim directly: Phase 2 improves first-attempt accuracy through revision-conditioned self-distillation. In Table 9, SRT achieves 66.7% on the first attempt and 71.7% after revision. SD-Zero achieves 68.3% on the first attempt, a 1.6-point improvement over SRT’s first attempt.

AuthorsOpenReview ↗

Author Response 2

W2: Sources of dense supervision and isolation of self-evolution

We agree that the source of supervision should be stated more precisely. Phase 1 is a one-time warm-up that uses binary outcome feedback to select successful self-generated revisions for SRT. Phase 2 then obtains dense token-level supervision from the teacher’s learned revision-conditioned distributions. We will clarify that SD-Zero removes the need for an external teacher while using filtered, self-generated supervision.

We also added a control that isolates teacher synchronization within Phase 2. Please see below.

W2, Q2: Does synchronization improve held-out accuracy beyond continued optimization with a frozen teacher?

Thank you for suggesting this control. We compared periodic teacher synchronization with continued training using the original SRT teacher frozen throughout, starting from the same SRT checkpoint on Olmo3-7B-Instruct and using the same training data and number of steps. In the synchronized configuration, we update the teacher at steps 500, 1000, and 1125. We evaluate both configurations on held-out OpenR1 questions.

Teacher configurationStep 0Step 500Step 1000Step 1125Step 1250
Periodically synchronized teacher, frozen between updates52.455.357.558.058.3
Original SRT teacher frozen throughout52.455.356.256.456.3

Accuracy (%) on held-out OpenR1 questions.

Both configurations reach 55.3% at step 500, before the first teacher update. Continuing optimization with the frozen teacher raises accuracy to 56.3% at step 1250, while periodic synchronization reaches 58.3%, a 2.0-percentage-point advantage. From step 500 onward, synchronization yields a 3.0-point gain, compared with 1.0 point from continued optimization with the frozen teacher. These matched results support an additional benefit from teacher synchronization beyond continued optimization, with the improvement measured on held-out questions.

Most of my concerns have been addressed and I remain my positive judgement to the work.

Reviewer_nNcCOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Self-Distillation Zero, a two-stage post-training method that translates binary feedback into token-level credit assignment. The first phase (Self-Revision Training) collects (prompt, answer, feedback, revision) tuples where the revision was successful and trains the model to generate a correct revision given (prompt, answer, feedback) and to generate (answer, feedback, revision) given the initial prompt and is responsible for the key gains in performance and increase in response length. In the second stage the student model learns the (prompt, answer, feedback) conditioned teacher/reviser policy through self-distillation. Here the main effect is a further improvement in model capabilities alongside a significant reduction in response length. The method is primarily evaluated on Qwen3-4B-Instruct and Olmo3-7B-Instruct and focuses on hard Math and Code benchmarks. The authors consider thinking models out of scope for this study.

Critical strengths

S1: The paper is very cleanly written and easy to follow. Every decision is well motivated and experimentally validated. Overall it is a pleasure to read.

S2: Using Self-Revision Training to prepare a model for self-distillation seems to be a distinctive contribution. The results suggest that is is one effective way to improve self-distillation on reasoning-heavy tasks like mathematical reasoning and competition coding.

S3: The finding that SRT boosts self-revision capabilities is very interesting. Do those self-revision capabilities generalize across benchmarks, e.g. math → code?

S4: The controlled ablations are very extensive and cover most questions about the method's design and the contributions of its components.

Critical weaknesses

W1: While thinking models are declared out of scope, the authors limit their analysis to models that may not have been extensively trained to reason and self-correct. Testing whether SRT provides additional benefits in a setting where models already have such capabilities would strengthen the claims of the paper.

Questions for authors

Q1: Do those self-revision capabilities obtained from SRT generalize across benchmarks, e.g. math → code?

Q2: Does SRT provide additional self-revision benefits for thinking models that already exhibit self-correction behavior?

Q3: Beyond the periodic synchronization experiment, how does performance depend on update frequency? In particular, how does a frozen teacher compare with using the current student as teacher with stopped gradients?

5 comments
AuthorsOpenReview ↗

Author Response

Thank you for the encouraging review. We are glad you found the paper clear, SRT a distinctive contribution, and the ablations extensive. Please find our response below.

W1, Q2: Does SRT provide additional benefits for thinking models?

Thank you for suggesting this evaluation. In our observed generations, Qwen3-4B-Instruct exhibits reasoning and self-correction behavior, including phrases such as “wait” and “let me correct it.” SRT aims to strengthen the model’s ability to use outcome feedback to revise its response. Appendix F, Table 20 examines OPSD on Qwen3-4B with thinking enabled during training, providing context for distillation in this setting.

To directly assess SRT’s additional benefit for models with existing reasoning capabilities, we evaluated SRT on Qwen3-1.7B, Qwen3-8B, and Qwen3-14B in thinking mode:

Backbone (thinking mode)ModelAIME24AIME25MATHOpenR1HMMT25AMOBenchAvg.
Qwen3-1.7BBase53.835.089.343.423.33.041.3
Qwen3-1.7BSRT50.435.892.549.528.77.844.1
Qwen3-8BBase75.465.094.853.043.816.858.1
Qwen3-8BSRT72.363.197.958.842.116.458.4
Qwen3-14BBase79.272.595.555.449.618.361.7
Qwen3-14BSRT78.673.697.560.654.016.763.5

Accuracy is reported in percentages. Avg. is the unweighted mean across the six benchmarks, computed before rounding.

SRT improves average accuracy across all three thinking-mode backbones, by 2.8, 0.3, and 1.8 percentage points, respectively. These results show that SRT can provide additional benefits for models with existing reasoning capabilities, extending its applicability to thinking-mode models across the tested scales.

Q1: Do SRT’s benefits generalize across domains, e.g. math → code?

We evaluated cross-domain transfer by training SRT separately on mathematics and code, then evaluating each checkpoint in both domains. The mathematics score averages AIME24, AIME25, and MATH; the code score measures accuracy on Codeforces.

ModelMath: AIME24, AIME25, MATH (%)Code: Codeforces (%)
Base model63.331.7
SRT trained on math72.834.5
SRT trained on code67.643.0

SRT improves cross-domain accuracy in both directions. Math-trained SRT raises Codeforces accuracy from 31.7% to 34.5%, a 2.8-point gain. Code-trained SRT raises average mathematics accuracy from 63.3% to 67.6%, a 4.3-point gain. These results show that SRT’s benefits extend beyond its training domain. These measurements establish cross-domain transfer of downstream performance. They support the generality of SRT’s benefits.

Q3: How does teacher update frequency affect performance?

We compared periodic teacher synchronization (Table 14) with using the current student as the teacher at every training step, with stopped gradients on Olmo3-7B-Instruct. In the periodic configuration, the teacher remains frozen between updates at steps 500, 1000, and 1125. Both configurations start from the same SRT checkpoint.

Teacher configurationStep 0Step 500Step 1000Step 1125Step 1250
Periodically synchronized teacher, frozen between updates52.455.357.558.058.3
Current student as teacher, with stopped gradients52.456.258.559.059.1

Accuracy (%) on held-out OpenR1 questions.

Per-step teacher updates yield higher accuracy at every checkpoint after initialization. During the first 500 steps, when the periodic teacher remains frozen at the original SRT checkpoint, using the current student achieves 56.2%, compared with 55.3%. At step 1250, it reaches 59.1%, exceeding periodic synchronization by 0.8 percentage points. These results support more frequent teacher updates in this setting. We appreciate the reviewer's point, and will add this synchronized-teacher experiment to the paper.

Thank you for the detailed experiments. My questions regarding generality and stability w.r.t. update teacher update frequency (Q1 and Q3) have been sufficiently addressed.

My remaining concern is that the reported numbers for Qwen3-14B base (thinking enabled) align exactly with the Qwen3-14B base (thinking disabled) from Table 13 in the appendix. Could the authors clarify whether “thinking mode disabled” in Table 13 refers to SRT training, evaluation, or both? For the new table, was thinking enabled during SRT data generation and training, and were Base and SRT evaluated under the same thinking-mode and sampling settings?

Thank you for catching this! We re-ran the Qwen3-14B evaluations and confirmed that the current numbers (61.7%) correspond to thinking mode on. We apologize for mislabeling Table 13, and have updated the paper. Detailed settings are:

  • SRT data generation. Model produced its initial attempts and revisions in thinking mode (temperature 1.0, top-p 0.9, up to 16,384 tokens).
  • SRT training. Every training target begins with a thinking block, and the prompt template leaves thinking on.
  • Evaluation. Base and SRT used the identical prompt and sampling settings (temperature 1.0, top-p 1.0, up to 32,768 tokens, 8 samples per problem). Every sample from both models opens a thinking block, and for Qwen3-14B, 97–98% close it before answering.
Reviewer_BJMzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SD-ZERO consists of the SRT phase followed by the Self-Distillation phase. The SRT phase trains generation and outcome-conditioned revision using self-generated traces, with revised answers that pass verification. The Self-Distillation phase transfers the reviser's token-level feedback to the generator, without an external teacher or supplied gold solutions. Experiments on mathematics and coding evaluate accuracy, self-revision, and efficiency.

Critical strengths
  1. Practical empirical improvements.

Table 1 reports the highest aggregate accuracy among the compared methods for both main backbones. Figure 3 and Table 9 also show that the Self-Distillation phase improves first-attempt accuracy and reduces response length compared with the SRT model on Qwen3-4B-Instruct/AIME24. These results support the method's practical value.

  1. Useful controlled comparisons.

Table 11 compares objectives on the same revision data, while Table 16 evaluates the Self-Distillation phase without the SRT phase. These controls help establish the SRT phase's contribution beyond simply obtaining better training responses.

Critical weaknesses
  1. Sensitivity to training-data selection remains unclear.

The main mathematics experiments use OpenR1-Math, with SRT data collected from the first 10K questions. The paper does not report full SD-ZERO comparisons across independently sampled question subsets or alternative mathematics training datasets. Since SRT retains only successful revisions, the resulting data yield and performance gains may depend on training-data selection. This leaves the robustness of the reported sample efficiency insufficiently established.

  1. Experimental accounting and reporting need clarification.

C.1 reports 6K retained revision traces, while Table 3 counts 6K training questions, without clarifying how multiple successful revisions per question are handled. Table 3 also lists one SRT training epoch, whereas Table 4 lists three. Furthermore, Table 18 reports averages but omits individual benchmark scores for two variants. Although the paper analyzes response length, comparable evaluation-length results across benchmarks and for the Table 18 variants are missing. These gaps limit reproducibility and interpretation of the accuracy–efficiency comparison.

Questions for authors
  1. SRT data construction

How are multiple successful revisions per question handled? Do the 6K retained traces correspond to 6K distinct questions?

  1. Training-data sensitivity

Are there any existing results comparing: (a) different randomly sampled 10K-question subsets of OpenR1-Math or (b) an alternative mathematics training dataset, such as DeepScaleR, using the complete SD-ZERO pipeline? If available, the retained SRT trace counts and final accuracy would be informative. Otherwise, briefly acknowledging this limitation would be sufficient for rebuttal.

  1. Training budget

Could you reconcile the SRT epoch counts in Tables 3 and 4 and clarify which count is reflected in C.2's training-token accounting?

  1. Evaluation reporting

Could you provide the missing per-benchmark scores in Table 18? Existing response-length statistics for the Table 18 variants and for both main backbones across the Table 1 benchmarks would also help assess the claimed efficiency benefits.

4 comments
AuthorsOpenReview ↗

Author Rebuttal Response

Thank you for the careful review. We are glad you found the empirical gains practical and the controlled comparisons in Tables 11 and 16 useful. Please find our response below.

W2, Q1. Retained traces versus distinct questions

For each question, we retain only its first successful revision. After inferencing on the initial 10K questions and filtering, we select 6K questions with successful revisions and keep one trace per question. Thus, the 6K training traces correspond to 6K distinct questions. (The counts in the table below describe the pool of questions with successful revisions before the 6K selection.)

W1, Q2. Sensitivity to training-data selection

We evaluated the complete SD-Zero pipeline on two random 10K-question OpenR1-Math subsets and DeepScaleR-Preview-Dataset. Accuracies below average all eight Table 1 benchmarks on Qwen3-4B-Instruct:

Training-data variantQuestions with successful revisionsSRT accuracy (%)SD-Zero accuracy (%)
First 10K OpenR1-Math questions (original)6,13457.660.3
Random 10K OpenR1-Math questions, seed 16,25357.860.7
Random 10K OpenR1-Math questions, seed 26,05757.259.8
DeepScaleR-Preview-Dataset6,79356.258.8

The random subsets yield final accuracy within 0.5 percentage points of the original selection. DeepScaleR has a different training-data distribution from OpenR1-Math, yet SD-Zero reaches 58.8% and improves over SRT by 2.6 points. Across all four settings, self-distillation adds 2.6 to 2.9 points, supporting robustness to question selection and the tested change in training-data distribution.

W2, Q3. Training epochs and token accounting

We train SRT, SFT, and RFT for one epoch. Appendix C.2's training-token accounting also uses one epoch. We apologize for the typo in Table 4, which was generated from a wrong version of the training script. We have updated the paper to correct it to one epoch.

W2, Q4. Per-benchmark accuracy and response length

The missing Table 18 accuracies are:

Phase-2 variantAIME24AIME25HMMT25AMOBenchOpenR1MATHAvg.
Forward KL, importance-weighted67.161.341.716.357.894.356.4
SFT on teacher revisions68.858.335.018.562.497.856.8

SD-Zero retains the highest six-benchmark average at 57.3%, although individual variants perform better on some benchmarks.

The following average response lengths cover all Table 18 variants on Qwen3-4B-Instruct/AIME24, using the same Generate-then-Revise evaluation:

Phase-2 variantFirst-attempt tokensRevised-attempt tokens
None (SRT model)8,4588,137
Forward KL, importance-weighted7,9328,036
Teacher on a fixed self-revision4,9615,820
SFT on teacher revisions4,3285,903
Reverse KL, on-policy (SD-Zero)3,5183,314

SD-Zero produces the shortest first and revised attempts among these variants, reducing their lengths by 58.4% and 59.3%, respectively, relative to SRT. These measurements support the token-efficiency benefit on Qwen3-4B-Instruct/AIME24.

Thank you for the rebuttal, which addresses most of my concerns.

Could you clarify why the forward-KL average in Table 18 changed from 56.2 to 56.4, and whether this was a reporting correction or a new run/evaluation?

Also, the revised Table 4 still lists 3 epochs, even though your response specifies 1.

Thank you for the prompt reply!

The change in the number in Table 18 is due to a new evaluation run.

We've updated the paper PDF again. Table 4 should list 1 epoch now!

Thank you for the clarification and correction. The rebuttal has addressed my main concerns, and I am updating my rating to weak acceptance.

Reviewer_ktCwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SD-ZERO introduces a two-phase post-training framework that teaches language models to internalize self-correction without relying on external expert supervision or reinforcement learning. Phase 1 fine-tunes the model on verified self-revision trajectories to master explicit error correction, while Phase 2 uses reverse KL divergence against a frozen, revision-conditioned teacher to distill those multi-turn reasoning loops into direct, single-pass generations.

Critical strengths
  1. The idea of building self-revision training set is interesting.
  2. The Context-Conditioned Distillation help transforms a multi-turn generation task into a token-level error-localization mechanism during student rollouts.
Critical weaknesses
  1. While the authors make a commendable effort to equalize generation and token budgets in Appendix C.2, the comparison remains methodologically confounded. SD-ZERO benefits from a supervised, rejection-filtered warm-up phase (SRT) that discards 34K failed rollouts, whereas the primary RL baseline (GRPO) is cold-started from the base model for only 1 single epoch. As demonstrated by the authors' own findings in Appendix G.2, RL algorithms benefit substantially when initialized from filtered revision data. A truly rigorous comparison would require benchmarking SD-ZERO against GRPO initialized from the same Phase 1 SRT checkpoint and trained across sufficient policy iterations.
  2. The baseline implementation of GRPO is severely constrained by its group sampling size (G=4G=4) and training duration (1 epoch). In verifiable mathematical reasoning tasks where baseline success rates are often below 10–15%, a group size of 4 guarantees that a large fraction of prompt batches will yield all-zero rewards, resulting in vanished advantages and wasted compute. Standard GRPO literature uses group sizes of G∈[8,64]G \in [8, 64] to ensure robust intra-group contrast. While the authors evaluate G=8G=8 in Appendix D.4, they counter-balance this by halving the optimization steps to 0.5 epochs. This under-samples and under-trains the RL baseline compared to established post-training protocols.
  3. The "Generate-then-Revise" evaluation (Figure 3, Table 9) is fundamentally unfair: it tests zero-shot baselines on an outcome-conditioned revision format they were never trained to perform, whereas SRT was explicitly fine-tuned on this exact objective.
  4. Figure 2 is visually cluttered and conceptually misleading, directly contradicting the paper's actual implementation and mathematical formulation. The diagram defines the teacher as πT:=πθSRT(yrevised∣x,yinit,Pr)\pi_T := \pi_{\theta_{\text{SRT}}}(y_{\text{revised}} \mid x, y_{\text{init}}, P_r) and explicitly claims in its caption that the teacher generates a revised response, erroneously depicting an impossible KL(πS ∥ πT)\text{KL}(\pi_S \,\Vert{}\, \pi_T) minimization between two distinct sequences (yinity_{\text{init}} and yrevisedy_{\text{revised}}). In reality, as detailed in Section 2.2, the teacher never generates a revision; it merely evaluates next-token probability distributions along the student's own on-policy trajectory. Furthermore, the schematic completely omits the dual nature of LSRT\mathcal{L}_{\text{SRT}}—obscuring how Lgeneration\mathcal{L}_{\text{generation}} trains the model on full concatenated traces—and relies on tangled, ambiguous dependency arrows that obfuscate active parameter updates versus frozen evaluations.
Questions for authors

Please refer to the weaknesses.

4 comments
AuthorsOpenReview ↗

Author Rebuttal Response

Thank you for the detailed feedback. We address all four concerns below.

W1. GRPO initialized from the same SRT checkpoint

We evaluated GRPO initialized from the same Phase-1 SRT checkpoint as SD-Zero, using both the same and twice SD-Zero's Phase-2 rollout budget. We use Qwen3-4B-Instruct and report avg@8 on AIME24, AIME25, and MATH, containing 30, 30, and 500 questions, respectively.

InitializationPhase 2Phase-2 rollout budgetAIME24AIME25MATHAvg.
SRTNone066.759.292.472.8
SRTGRPO1x SD-Zero67.556.791.471.9
SRTGRPO2x SD-Zero69.660.492.174.0
SRTSD-Zero Phase 21x SD-Zero68.360.093.674.0

Avg. is the unweighted mean of the three reported benchmark scores. The rollout budgets cover Phase 2; all rows share the SRT warm-up.

At the matched rollout budget, SD-Zero outperforms SRT-initialized GRPO by 2.1 percentage points on average. Doubling GRPO's rollout budget raises its average to 74.0%, matching SD-Zero at the reported precision. This comparison controls for SRT initialization and supports SD-Zero's rollout efficiency: on these benchmarks, it reaches the same reported average with half as many Phase-2 rollouts.

W2. Larger GRPO groups without reducing training duration

Appendix D.4, Table 10 already includes GRPO with G = 8 for a full epoch, alongside the half-epoch run. We additionally evaluated G = 16 for a full epoch:

MethodGroup sizeEpochsTotal generationsAverage math accuracy (%)
GRPO41.060K51.7
GRPO80.560K50.6
GRPO81.0120K52.3
GRPO (new)161.0240K53.7
SD-ZeroN/AN/A49K57.3

These averages cover all six math benchmarks on Qwen3-4B-Instruct, whereas W1 reports three benchmarks. SD-Zero's 49K generations include SRT data collection and Phase-2 rollouts.

Increasing G from 4 to 16 while retaining a full epoch improves GRPO from 51.7% to 53.7%. SD-Zero remains 3.6 points higher, while GRPO uses approximately 4.9 times its total generations. Thus, the advantage persists in the tested larger-group settings without the half-epoch adjustment raised in the review.

W3. Revision-format familiarity and Generate-then-Revise evaluation

Our claim is that SRT elicits the revision capability needed for Phase 2. Figure 3 and Table 9 measure whether a checkpoint can use outcome feedback to improve its response, the behavior Phase 2 relies on for supervision. This makes the evaluation relevant to choosing a warm-up, even if adaptation to the revision format contributes to the capability.

We expect format familiarity to make a limited contribution because the prompt adds one short instruction: "Wait, this response is not correct, let me start over" or "Let me rephrase the above solution."

To test the effect of limited revision-format training, we added two controls: GRPO followed by SRT on 1K questions, and SRT on 1K questions followed by GRPO. Both use the same Generate-then-Revise evaluation as Figure 3 and Table 9.

ModelFirst-attempt accuracy (%)Revised-attempt accuracy (%)Revision gain (points)
SRT66.771.7+5.0
GRPO -> 1K-question SRT65.167.8+2.7
1K-question SRT -> GRPO62.163.6+1.5
SRT -> SD-Zero68.373.6+5.3

Both controls receive explicit training on the revision format, achieving revision gains of 2.7 and 1.5 percentage points. Full SRT achieves a larger gain of 5.0 points, demonstrating that its advantage persists against checkpoints with prior exposure to the same revision objective. These results support SRT’s effectiveness in developing the revision capability needed for Phase 2.

W4. Figure 2 and the actual distillation computation

We agree that the original Figure 2 misrepresented Phase 2. We have revised the diagram and caption to match Section 2.2 and Eq. (4):

  • The student samples a response yy. At each position tt, the teacher evaluates πθSRT(⋅∣x,y,Pr,y<t)\pi_{\theta_{\mathrm{SRT}}}(\cdot\mid x,y,P_r,y_{<t}) and the student evaluates πθ(⋅∣x,y<t)\pi_\theta(\cdot\mid x,y_{<t}). The KL compares these next-token distributions along the same student trajectory. The teacher does not generate a separate revision during this training step.
  • Phase 1 now shows both SRT losses. The generation loss trains the model to predict the full concatenated trace (initial attempt, outcome-conditioned prompt, and revised response) given only the question. The revision loss trains the model to predict the revised response given the question, initial attempt, and outcome-conditioned prompt.
  • The diagram explicitly marks the student as trainable and the SRT teacher as frozen. Only the student receives gradients. The self-revision sampling step and caption now use notation consistent with Section 2.
Reviewer_ktCwOpenReview ↗

Official Comment by Authors

I thank authors for detailed response.I have some further questions below.

  1. How do you define rollout budget, you mentioned GRPO catch up the performance when doubling the budget. Does that mean increase group size from 4 to 8 or increase training samples/update steps.
  2. Based on the additional results, SD-Zero efficiently achieves strong performance comparable to GRPO. However, we still don't know how their performance ceilings compare. Since doubling the training budget doesn't necessarily mean doubling the training time, I'd like to understand the headroom so we can decide if SD-Zero is worth pursuing.

Thank you for the valuable follow-up questions. Please see our response below.

1. Definition of the rollout budget

We define rollout budget as the number of model generations during training. In SD-Zero (Phase 2), this counts the number of student generations. In GRPO, this counts all sampled responses.

The doubled GRPO budget comes from increased update steps while keeping the group size G = 8 fixed. The 1× run uses 50 training steps, and the 2× run uses 100 training steps.

2. Extended training

We agree that longer-training performance is a valuable metric. We extended the comparison on Olmo-3-7B-Instruct, using SRT-initialized checkpoint and held-out OpenR1 evaluation set.

SRT-initialized GRPO

Training step050100125150
Accuracy (%)52.454.455.856.956.8

SD-Zero Phase 2

Teacher configurationStep 0Step 500Step 1000Step 1125Step 1250
Original SRT teacher frozen throughout52.455.356.256.456.3
Periodically synchronized teacher52.455.357.558.058.3
Current student as teacher, with stopped gradients52.456.258.559.059.1

We observed three advantages of SD-Zero during the longer training runs:

  1. Accuracy advantage at an equal generation budget, even with extended training. Both methods use 128 prompts per step, with eight responses per prompt for GRPO and one for SD-Zero, so 1000 SD-Zero steps and 125 GRPO steps both generate 128K responses. At this budget, SD-Zero achieves 57.5% with periodic teacher updates and 58.5% with per-step updates, compared with 56.9% for GRPO.

  2. Higher best-observed performance, suggesting higher saturation accuracy. Both methods improve more slowly toward the end of training. SD-Zero reaches a best-observed accuracy of 59.1%, exceeding GRPO’s 56.9% by 2.2 percentage points. Together with the slowing gains, these results suggest a higher saturation accuracy for SD-Zero in this setting.

  3. Lower measured training time. At the matched budget of 128K generations, SD-Zero uses approximately 230 H100 GPU-hours, compared with 420 GPU-hours for GRPO. Across the full extended runs, SD-Zero takes only 60% of GRPO’s training time on the same hardware, with approximately the same number of generated responses.

Dear reviewer ktCw,

As the discussion period comes to an end (around 9 hours left), we would like to sincerely ask if our reply has addressed your remaining concerns. If so, we would appreciate it if you would reconsider the assessment. Thank you so much for your helpful suggestions throughout the discussion!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.