- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Self-Distillation Zero (SD-Zero), a two-stage method that converts binary outcome rewards into dense token-level supervision without an external teacher. In the first stage, the model is trained on correctness-filtered self-revision traces so that it can revise an initial response conditioned on the response and its binary reward. In the second stage, a frozen copy of this revision-capable model supervises the generator on the generator's own trajectories through token-level distillation. Experiments on math and code with Qwen3-4B-Instruct and Olmo-3-7B-Instruct show consistent improvements over the base models and stronger performance than RFT, GRPO, OPSD, and several related self-distillation baselines under roughly matched generation and training-token budgets. The paper also includes extensive ablations of the two training stages, the teacher context, data allocation, objective design, and teacher synchronization.
Critical strengths
- The central hypothesis is interesting: a model that cannot solve a problem on its first attempt may still provide a better conditional distribution after seeing its own attempt and outcome, and this revision behavior can then supervise one-pass generation. The two-stage design is intuitive and addresses a real limitation of both sparse-reward RLVR and teacher-based distillation.
- The performance claim is supported by a relatively comprehensive set of experiments. The gains hold across two model families and both math and code, and the paper compares against strong RL and self-distillation baselines while accounting for both generated samples and approximate token cost. The phase, objective, teacher-context, data-allocation, and filtering ablations also show that the reported result is not explained by one obvious implementation choice.
Critical weaknesses
- The experiments do not establish the proposed mechanism of token-level self-localization. Figure 4 shows that KL mass is concentrated on a few positions, but the same pattern would arise if the reviser ignored the erroneous reasoning and branched into a different solution. There is no ground-truth error location, controlled token corruption, or counterfactual intervention showing that high-KL positions identify tokens whose correction improves the answer. Similarly, shorter responses and fewer explicit revision keywords do not demonstrate that revision has been “internalized”; they may only indicate a change in output style. These mechanism claims are central to the paper but are supported mainly by proxies and one qualitative example.
- The dense information does not come from the binary reward alone. Phase 1 samples multiple revisions, retains only correct ones, and trains on these high-quality selected traces; the reviser also receives additional inference compute and a hand-written revision instruction. SD-Zero therefore removes an external teacher, but it does not remove high-quality filtered supervision. The iterative self-evolution claim is also not isolated: teacher synchronization is followed by additional training, without a matched control that continues for the same number of steps with the frozen teacher, and the reported synchronization result is on training questions.
Questions for authors
- Can the authors evaluate localization against known error positions, for example by introducing controlled token-level corruptions and testing whether the largest KL changes identify the corrupted tokens or predict the effect of repairing them? Comparing against a reviser that cannot see the student's attempt would also show whether the attempt is actually used rather than ignored.
- For teacher synchronization, what happens if training continues for the same number of steps and on the same data while the original SRT teacher remains frozen? Please report held-out rather than only training-question accuracy. This control would determine whether the extra gain comes from synchronization or simply from continued optimization.
3 comments
Author Response 2
W2: Sources of dense supervision and isolation of self-evolution
We agree that the source of supervision should be stated more precisely. Phase 1 is a one-time warm-up that uses binary outcome feedback to select successful self-generated revisions for SRT. Phase 2 then obtains dense token-level supervision from the teacher’s learned revision-conditioned distributions. We will clarify that SD-Zero removes the need for an external teacher while using filtered, self-generated supervision.
We also added a control that isolates teacher synchronization within Phase 2. Please see below.
W2, Q2: Does synchronization improve held-out accuracy beyond continued optimization with a frozen teacher?
Thank you for suggesting this control. We compared periodic teacher synchronization with continued training using the original SRT teacher frozen throughout, starting from the same SRT checkpoint on Olmo3-7B-Instruct and using the same training data and number of steps. In the synchronized configuration, we update the teacher at steps 500, 1000, and 1125. We evaluate both configurations on held-out OpenR1 questions.
| Teacher configuration | Step 0 | Step 500 | Step 1000 | Step 1125 | Step 1250 |
|---|---|---|---|---|---|
| Periodically synchronized teacher, frozen between updates | 52.4 | 55.3 | 57.5 | 58.0 | 58.3 |
| Original SRT teacher frozen throughout | 52.4 | 55.3 | 56.2 | 56.4 | 56.3 |
Accuracy (%) on held-out OpenR1 questions.
Both configurations reach 55.3% at step 500, before the first teacher update. Continuing optimization with the frozen teacher raises accuracy to 56.3% at step 1250, while periodic synchronization reaches 58.3%, a 2.0-percentage-point advantage. From step 500 onward, synchronization yields a 3.0-point gain, compared with 1.0 point from continued optimization with the frozen teacher. These matched results support an additional benefit from teacher synchronization beyond continued optimization, with the improvement measured on held-out questions.
Most of my concerns have been addressed and I remain my positive judgement to the work.
Author Response 1
Thank you for the insightful review. We are glad you found the central hypothesis interesting and the experiments comprehensive. Please find our response below.
W1, Q1: Can high-KL positions identify errors and guide repairs, and does the reviser use the student’s attempt?
Yes. We conducted controlled corruption, context-ablation, and repair experiments. We selected 160 verified-correct, completed solutions to OpenR1 questions outside the SRT data, each containing a boxed answer within 8,192 tokens. We replaced exactly one digit or arithmetic operator in each solution. We then evaluated the corrupted solutions using the SRT checkpoint for both student and teacher, as at the initialization of Phase 2.
1. The KL score localizes injected errors.
The largest per-token KL identifies the injected error in 86% of cases, compared with approximately 0.02% by chance over solutions averaging 5,082 tokens. We also compare the corrupted token with the same position in the clean solution. After standardizing the student-minus-teacher log-probability gap within each solution to obtain z, the median is −39.7 at the corrupted token, versus +0.10 at the clean position. This controlled comparison shows that the injected error produces a localized change in the scoring signal.
2. The teacher’s scoring depends on the supplied attempt.
We varied the teacher’s context while keeping the SRT model fixed on 42 solutions:
The signal at the corrupted token is strongest when the teacher sees the student’s own attempt. Its magnitude decreases with an unrelated attempt and decreases further with no attempt. These results show that the teacher’s evaluation responds to the content of the supplied attempt.
3. Repairing the largest-KL position improves final accuracy.
For each of the 160 corruptions, we set the token at the selected position and regenerated the continuation, using three samples per case. Among the 138 correctly localized corruptions, the teacher’s top token matches the original token in 85 cases.
Inserting the teacher’s top token at the largest-KL position raises final accuracy by 2.1 percentage points. The corresponding change at a random position is −0.8 points, giving a 2.9-point difference in the accuracy changes, paired over the 160 cases. This intervention connects the localization signal to the benefit of a targeted repair.
4. Disagreement is sparse and localized along the student’s trajectory.
We also examined the shape of disagreement on 368 incorrect attempts using the Qwen3-4B-Instruct reviser: 79 completed responses and 289 that reached the length cap. In both groups, the median longest run of consecutive high-KL tokens is two tokens, and the top 1% of positions account for 85% and 84% of KL mass, respectively. These measurements show that disagreement is sparse and localized along the student’s trajectory.
For “internalization,” we have updated the paper to state the accuracy-based claim directly: Phase 2 improves first-attempt accuracy through revision-conditioned self-distillation. In Table 9, SRT achieves 66.7% on the first attempt and 71.7% after revision. SD-Zero achieves 68.3% on the first attempt, a 1.6-point improvement over SRT’s first attempt.