- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies on-policy self-distillation (OPSD), where a frozen copy of the student conditioned on the reference solution teaches the student via forward KL on student-generated prefixes. It identifies "entropy overshoot" (student token entropy rising above the teacher's and staying there) as a failure mode and attributes it to (i) answer-conditioned teacher guidance that transfers poorly to student prefixes and (ii) forward KL's one-way mode-covering pressure. E2-OPSD replaces the reference solution in the teacher context with a retrieved solved neighboring problem (EGT) and routes each token to forward or reverse KL by the sign of the student-teacher entropy gap, weighted by its normalized magnitude (BER + AGD). On four backbones (Qwen3-0.6B/1.7B thinking, Qwen3-4B-Instruct, Olmo-3-7B-Instruct) trained 360 steps on DeepMath, it reports math mean@16 gains of +0.6 to +4.3 over OPSD and OOD gains over Base on ZebraLogic/HumanEval, plus ablations and mechanism probes (teacher-context KL, gradient-share by entropy bucket, KL-direction dynamics).
Critical strengths
-
The diagnosis is concrete and the fix is cheap. The entropy-overshoot phenomenon is shown directly (Figures 1, 4, 6), the teacher-context probe (Table 13: 25.9% lower forward KL on student prefixes, teacher own-prefix length 836 vs student 9,140 tokens) gives a plausible mechanism for why answer-conditioned teachers mismatch student states, and the method adds no forward passes or networks. Figure 6 (FKL vs RKL vs fixed mixture vs BER under a fixed teacher) is a clean, controlled demonstration that state-dependent direction routing behaves differently from any fixed divergence.
-
The ablation is additive and consistent on Qwen3-1.7B (Table 11): EGT +1.4, BER +3.0, AGD +0.8, combined +4.3 over OPSD, with per-benchmark gains on all four math sets and small evaluation-seed variance. On this model the result is convincing.
Critical weaknesses
-
All baselines are essentially flat relative to Base, which makes the comparison uninformative. Across the four backbones, OPSD, SDPO, RLSD and OPSD+EOPD move average math mean@16 by at most +0.8 and often by -0.1 to +0.5 (Table 1), whereas their original papers report substantial gains. Under the shared protocol (lr 1e-6, 360 steps, frozen teacher), the competitors appear not to be learning at all, so "beating OPSD" reduces to "beating Base." Either the budget/hyperparameters are unsuited to the baselines or the implementations are off; in either case the headline "up to 4.3 over OPSD" is not evidence that E2-OPSD outperforms properly-tuned self-distillation. The OOD tables (Tables 2, 10) compound this by comparing only to Base, with no baseline rows.
-
Gains are robust on only one of four backbones, from a single training run each. On Qwen3-0.6B (+0.6 avg over OPSD), Olmo-3-7B (+1.9, with HMMT25 below OPSD and SDPO) and to a lesser extent Qwen3-4B (+2.2, mostly from AIME25), the improvements are within one to two evaluation-seed standard deviations and there are no training-seed repeats. The "consistent across families and modes" claim rests largely on Qwen3-1.7B, which is also the only model used for all ablations and probes. Relatedly, the causal link between entropy overshoot and accuracy loss is asserted rather than established: in Table 1 OPSD does not degrade below Base despite overshooting, and the teacher used as the entropy reference has privileged answer access, so a student exceeding its entropy is not obviously pathological.
A further concern on novelty: the cited "direction-adaptive self-distillation" (Zhang et al., 2026) appears to already switch KL direction based on student/teacher uncertainty, and retrieved-exemplar context for training-time guidance (RA-RFT, XRPO) is also prior work. Neither is compared against, so the incremental contribution of BER and EGT over the closest prior methods is unclear.
Questions for authors
Clarification requests:
- Why do OPSD, SDPO and RLSD show near-zero change from Base under your protocol, when their original papers report multi-point gains? Did you tune learning rate or step budget per baseline, and do the baselines improve with a larger budget (e.g., 1,000+ steps) or the learning rates used in their papers?
- Please add baseline rows to the OOD tables (ZebraLogic, HumanEval). Gains over Base alone do not show that E2-OPSD transfers better than OPSD/SDPO/RLSD.
- How similar are retrieved neighbors to the target problem in DeepMath (which contains many near-duplicate templates)? Please report (a) a leakage check, e.g., fraction of neighbors sharing the final answer or differing only in constants, and (b) a control with a randomly chosen (non-neighbor) solved problem, to separate "remove the current answer" from "provide a transferable strategy."
- How does E2-OPSD compare to Zhang et al. (2026) direction-adaptive self-distillation, and what is the distinction between BER and that method?
- Can you provide training-seed repeats (at least 2-3 runs) for one small configuration to quantify run-to-run variance relative to the reported gains?
Suggestions for future work: Validating that the method still helps when OPSD itself is run to the point where "gains deteriorate as training proceeds" (longer budgets) would directly support the paper's motivation.
3 comments
Q6. How does E²-OPSD compare with Zhang et al. (2026), and what distinguishes BER from that method?
We thank the reviewer for this important question. Although both methods use entropy-aware routing, they address fundamentally different problems.
- Failure mode. DASD addresses entropy suppression and diminished exploration, whereas E²-OPSD targets entropy overshoot—the student becoming more uncertain than its teacher.
- Teacher relationship. DASD makes the teacher attractive at low-entropy tokens but repulsive at high-entropy tokens. E²-OPSD always preserves the teacher as the target: BER selects between two teacher-attractive KL directions, while AGD controls update strength.
- Supervision. DASD is verifier-anchored and applies GRPO-style advantages to sampled tokens. E²-OPSD is reward- and verifier-free, distills the teacher’s full top- distribution at every position, and improves its context through EGT.
Thus, E²-OPSD provides a more faithful and broadly applicable solution to OPSD’s intrinsic instability, without repelling the teacher or requiring external verification. We will add this distinction to the related-work section.
Suggestion for future work: validate that the method still helps when OPSD is run until its gains deteriorate.
We thank the reviewer for this insightful suggestion. We extended OPSD to 480 steps and branched the overshooting run to BER at step 240 (Figure 9). BER rapidly eliminates the entropy gap and consistently outperforms continued OPSD, while E²-OPSD prevents the degradation altogether, finishing 2.8–3.6 points higher across two seeds (Table R3). This provides strong direct validation of E²-OPSD’s central design: preventing entropy overshoot is substantially more effective than correcting it afterward.
Table R1. Results over three training seeds (Qwen3-4B-Instruct-2507, math mean@16, %). Seed 1 is the run in Table 1; benchmark columns average the three seeds.
| Method | AMC23 | AIME24 | AIME25 | HMMT25 | Seed 1 | Seed 2 (new) | Seed 3 (new) | Mean ± Std. |
|---|---|---|---|---|---|---|---|---|
| Base | 85.39 | 61.74 | 46.46 | 29.03 | — | — | — | 55.65 |
| GRPO | 84.99 | 61.65 | 47.41 | 29.19 | 55.19 | 56.18 | 56.05 | 55.81 ± 0.54 |
| OPSD | 85.53 | 61.90 | 46.76 | 29.49 | 55.58 | 55.65 | 56.54 | 55.92 ± 0.54 |
| E²-OPSD | 86.86 | 62.66 | 50.46 | 30.39 | 57.78 | 57.23 | 57.77 | 57.59 ± 0.31 |
Table R2. Out-of-domain results (Qwen3-4B-Instruct-2507, training seed 2, mean@16, %). ± is over three evaluation seeds.
| Method | ZebraLogic | HumanEval | LiveCodeBench v6 | Avg. |
|---|---|---|---|---|
| Base | 78.98 ± 0.84 | 84.04 ± 0.09 | 33.27 ± 0.30 | 65.43 |
| GRPO | 79.27 ± 0.12 | 83.40 ± 0.08 | 33.77 ± 0.17 | 65.48 |
| OPSD | 79.85 ± 0.52 | 86.12 ± 0.27 | 33.83 ± 0.30 | 66.60 |
| E²-OPSD | 82.65 ± 0.11 | 85.10 ± 0.25 | 35.55 ± 0.26 | 67.76 |
Table R3. Training dynamics over 480 steps (Qwen3-4B-Instruct-2507, validation mean@8, %; gap = student − teacher entropy, nats).
| Method (seed) | Gap > 0 from | Best (step) | Step 480 | Change | Gap at best | Gap at 480 |
|---|---|---|---|---|---|---|
| OPSD (2) | step 128 | 56.7 (240) | 53.4 | −3.3 | +0.019 | +0.034 |
| OPSD (3) | step 128 | 57.2 (184) | 53.8 | −3.4 | +0.013 | +0.035 |
| BER only (2) | never | 57.4 (184) | 54.7 | −2.7 | −0.015 | −0.014 |
| E²-OPSD (2) | never | 58.8 (304) | 56.2 | −2.6 | −0.008 | −0.008 |
| E²-OPSD (3) | never | 58.2 (240) | 57.3 | −0.9 | −0.009 | −0.008 |
Looking for your response
Dear Reviewer,
I hope this message finds you well. We sincerely appreciate your valuable feedback. We have carefully addressed all your comments point by point and made substantial revisions to the manuscript with significant effort.
As the discussion period is nearing its end with less than three days remaining, we want to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.
Thank you for your time and effort in reviewing our paper.
We thank the reviewer for the careful and constructive review. All new experiments are in Appendix D of the revised PDF, and new text is shown in blue. Tables R1–R3 are reproduced at the end of this response for your convenience.
Q1. Why do OPSD, SDPO, and RLSD show near-zero change from Base, and do they improve with tuned learning rates or larger budgets?
We thank the reviewer for this important question. Within our fixed compute budget, we carefully tuned each baseline individually, including its learning rate and training steps, and allocated sufficient training budget to all methods. We report the best validation-selected configuration for each baseline. Since our data and experimental protocol differ from those in the original papers, the absolute gains are not directly comparable.
Q2. Please add baseline rows to the OOD tables.
We thank the reviewer for the suggestion. We added GRPO and OPSD baselines and introduced LiveCodeBench v6 to strengthen the OOD evaluation (Table R2; Table 17 in the revised PDF). E²-OPSD achieves the highest three-benchmark average (67.76 vs. 66.60 for OPSD and 65.48 for GRPO), with particularly strong gains on ZebraLogic and LiveCodeBench v6. These results provide stronger evidence that E²-OPSD delivers robust transfer beyond its in-domain advantages, especially on challenging multi-step reasoning tasks.
Q3. Can you provide training-seed repeats to quantify run-to-run variance?
We repeated GRPO, OPSD, and E²-OPSD on Qwen3-4B-Instruct-2507 with two additional training seeds (Table R1, Table 16 in the revised PDF). Across three seeds, E²-OPSD reaches 57.59 ± 0.31, compared with 55.92 ± 0.54 for OPSD and 55.81 ± 0.54 for GRPO. It is best on every seed and on all four benchmarks on average, and its margin over each baseline exceeds the training-seed standard deviation of every method.
Q4. The "consistent across families and modes" claim rests largely on Qwen3-1.7B, which is also the only model used for ablations and probes.
We thank the reviewer for raising this important concern. We substantially expanded the evidence beyond Qwen3-1.7B along two dimensions.
Together, these results show that both the diagnosed failure and E²-OPSD’s gains are consistent across model scales, families, modes, and random seeds, providing substantially stronger support for our central claims.
Q5. The causal link between overshoot and accuracy loss is asserted rather than established: OPSD does not fall below Base in Table 1, and the teacher has privileged access, so exceeding its entropy is not obviously pathological.
A: We thank the reviewer for this thoughtful question.