- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether every token position along a student-generated trajectory should be supervised during Direct On-Policy Distillation (Direct-OPD, Section 3, Eqs. 1–4), where a student is trained using the top- () token-level log-ratio reward between a post-RL teacher and its pre-RL reference . In Section 4.1 (Eqs. 5–7 and Appendix B Proposition 1), the authors construct a clean family of distributions showing a probability-mass mismatch: as the total mass assigned by and to the student's top- candidates shrinks toward zero, the true behavioral divergence (and both KL directions) vanishes linearly in , yet the log-ratio rewards and the local Direct-OPD logit gradient (Eq. 17) remain completely unchanged.
Motivated by this mismatch, the authors propose S2D-OPD (Section 4.2, Figure 1, Eqs. 8–12), which forms -way coarsened distributions over the student's top- tokens plus a single unnormalized residual bucket (Eq. 8), scores each state by the coarsened Jensen–Shannon divergence (Eq. 9), and retains only the top highest-JSD positions per response (Eq. 10) for both the policy-shift reward update and the student-anchor KL penalty (Eqs. 11–12 and Appendix A.2 Eq. 14). Evaluated across two 1.5B teacher–reference pairs (R1-Distill-1.5B -> JustRL-1.5B and Nemotron-1.5B -> QuestA-1.5B) and four students (Qwen3-1.7B, Qwen3-4B, Qwen3-8B, R1-Distill-7B) on 93 held-out competition math problems (AIME26 [30 problems] and HMMT Nov 2025 + Feb 2026 [63 problems], Table 1), S2D-OPD improves held-out Test Avg. over dense Direct-OPD in 7 of 8 configurations and ties in the 8th, lifting the 8-setting grand mean from 46.36% to 47.31% (+0.95 percentage points) with zero extra forward passes.
Critical strengths
1. Elegant theoretical diagnosis of tail-mass insensitivity in log-ratio distillation rewards.
Proposition 1 (Section 4.1, Eqs. 5–6; Appendix B, Eqs. 15–18) pinpoints a genuine flaw in top- log-ratio objectives: when both and assign tiny probability (e.g., vs. ) to a token in the student's top-16 candidate set, produces a huge noisy reward even though the teacher's actual behavior barely moved in total variation (Figure 5). Compressing the off-top- probability mass into a single residual event without renormalizing (Eq. 8) is a zero-overhead way to make JSD sensitive to how much probability mass actually supports the teacher shift.
2. Clean, zero-extra-forward-pass implementation with consistent directional improvements and strong decile diagnostics.
Because Direct-OPD already queries and on the student's rollout tokens, computing the 17-way coarsened JSD adds virtually no wall-clock cost. The decile experiment in Figure 3 ( across deciles, with bottom deciles collapsing and Random 10% matching dense training) strongly supports the thesis that supervising all 100% of tokens is at best redundant and at worst harmful, and across the 8 student–teacher pairs in Table 1, S2D-OPD never underperforms dense Direct-OPD (7 wins, 1 tie).
Critical weaknesses
1. Modest average test gain (+0.95 pp) without multi-seed training variance, especially under JustRL-1.5B.
While the validation curves in Figure 2 show clear separation on AIME24/AIME25, the held-out Test Avg. improvements in Table 1—evaluated at the best validation checkpoint—average +0.95 percentage points overall and are practically flat (+0.2, +0.2, 0.0 points) on three of the four students when distilling from JustRL-1.5B (Figure 4b). Because RL/OPD training with batch size 128 and rollouts over 300 steps has non-trivial run-to-run trajectory and checkpoint-selection variance, bootstrapping over the 93 test problems from a single training seed per cell (Section 5.2, lines 308–312) does not establish that +0.2 to +1.7 point differences survive across random training seeds.
2. Missing empirical comparisons against prior selective-distillation signals and missing reward-vs.-KL masking ablation.
Section 2 (pp. 2–3) surveys a rich set of token-selection criteria for distillation—including student entropy/uncertainty (Tavor et al., 2026; Ko et al., 2026), teacher confidence/margin (Jin et al., 2026; Zhou et al., 2026), token-importance weighting (TIP, Xu et al., 2026; TA-OPD, Wang et al., 2026b), and sign-agreement gating (OPD2, Heo et al., 2026)—yet Section 5 only compares S2D-OPD against dense Direct-OPD, Random 10% (on a single setting in Figure 3a), and forward/reverse KL between and (Figure 6a). Especially since Random 10% in Figure 3a (51.2) already gets within 0.1 points of dense Direct-OPD (51.3) simply by masking 90% of positions, we need (a) an ablation that masks only the reward in Eq. (12) while keeping the student-anchor KL penalty unmasked across all tokens, and (b) direct empirical comparisons on the same setup against natural selection heuristics such as expected reward magnitude, student entropy, or OPD2 / TA-OPD.
Questions for authors
-
Assessment-critical (Reward masking vs. anchor-KL masking): In Eq. (11), Eq. (12), and Appendix A.2 Eq. (14), zeroes out both the Direct-OPD reward gradient and the student-anchor KL penalty at 90% of token positions (while in Eq. 13 still tracks the all-token mean shift ). Notably, in Figure 3a, even
Random 10%(51.2) nearly matches 100% dense training (51.3). What happens onQwen3-1.7BandQwen3-4Bif you apply the top-10% JSD mask only to the reward term while keeping the KL anchor active on all tokens (or if you applyRandom 10%on the Table 1 held-out benchmarks)? -
Assessment-critical (Alternative token-selection baselines): On
Qwen3-1.7BandQwen3-4B, how does coarsened teacher–reference JSD (, Eq. 9) compare at the same 10% budget against selecting tokens by (i) student entropy, (ii) expected reward magnitude , or (iii) teacher–student disagreement /TA-OPD/OPD2? -
Assessment-critical (Multi-seed training robustness): Can you report mean std across at least 3 independent training seeds for a core subset of Table 1 (e.g.,
Qwen3-1.7BandQwen3-4Bunder bothJustRL-1.5BandQuestA-1.5B) to verify that the held-out gains (particularly the +0.2 pp margin onQwen3-4BunderJustRL) hold across training seeds rather than only problem-level bootstrap resamples?
6 comments
Response [2/2]
W2.2 & Q1. Reward masking vs. anchor-KL masking
We apply the top-10% JSD mask to each term separately. A masked term is averaged over the retained positions, and an unmasked term over all valid positions. As you note, the adaptive KL coefficient (Eq. 13) is computed from all valid positions in every variant, so it is identical across them.
| Variant | Reward term | KL term | AIME26 | HMMT | Test Avg. | Δ (pp) |
|---|---|---|---|---|---|---|
| Direct-OPD | all | all | 45.42 | 32.04 | 36.36 | — |
| Reward-Mask-Only | top-10% | all | 47.19 | 31.85 | 36.80 | +0.44 |
| KL-Mask-Only | all | top-10% | 43.75 | 30.46 | 34.74 | −1.62 |
| S²D-OPD | top-10% | top-10% | 47.40 | 32.94 | 37.60 | +1.24 |
- Filtering the reward is the beneficial component. Masking only the reward improves accuracy. Masking only the KL anchor keeps the low-JSD rewards, removes their regularization, and degrades accuracy. Weaker regularization therefore does not explain the gain.
- Joint masking adds a further gain. With reward-only masking, the anchor acts on all positions but the reward on only 10% of them, which likely over-regularizes the sparse reward. Joint masking keeps both terms on the same positions, as in Direct-OPD. The gap between +0.44 and +1.24 suggests that this balance also contributes. These are single runs, so we treat the size of the gap with caution.
- On Random-10. You rightly note that Random-10 (51.2) nearly matches dense training (51.3). Random-10 uses exactly the same masking and normalization as S²D-OPD, including masking both terms. It differs only in which states are retained. Masking 90% of positions therefore does not by itself improve or regularize training. The gain of S²D-OPD (52.9) comes from retaining high-JSD states, and the low-JSD deciles under the same masking fall below the initial student.
Dear Authors,
Thank you for the detailed responses. You engaged seriously with all three of my questions. You trained Qwen3-4B/JustRL with four seeds, compared four alternative selectors, and masked the reward and KL terms separately. You also clarified how each masked term is normalized, and that the adaptive KL coefficient is computed from all valid positions in every variant. The gap to TA-Direct and OPD²-gate is the clearest new result: S²D-OPD leads them by 1.98 and 2.11 pp in single runs (OPD²-gate at its own retention rate). However, the core issue is unchanged. The gains are small and are not yet separated from training noise, so I keep my rating at 2 for three reasons.
-
The four-seed result does not separate the gain from variation across seeds. The gain is +0.64 ± 0.37 pp (s.e.), and it comes from one setting only. The four per-seed differences are +0.20, 0.00, +0.67 and +1.68, so their standard deviation is about 0.75, larger than the mean gain. Seed 45 alone accounts for 1.68 of the total 2.55. Without it, the mean gain is 0.29. In your reply to me, you acknowledge that "the seed-level spread is still comparable to the gain". The p = 0.048 comes from an exact one-sided sign-flip test over the 93 problems, applied to seed-averaged differences. As you note in Response [2/3] to Reviewer yDxL, this test "still measures uncertainty over test problems". The null distribution of a sign-flip test is symmetric, so the two-sided value is 2 × 0.048 = 0.096. In your later reply to Reviewer yDxL, the same result appears as "+0.64 ± 0.37 pp (p = 0.048 < 0.05)" without "one-sided", which reads as seed-level significance. The General Response describes it accurately as problem-level but still omits one-sided.
-
The new ablations are single runs, and the key differences in the reward/KL ablation fall within run-to-run noise. Your own K sweep gives a rough noise scale. Scoring with K = 16 and K = 32 yields masks that overlap the full-vocabulary mask by 95.40% and 98.08%, so the two runs should be nearly identical. Their Test Avg. still differs by 0.87 (37.60 vs. 36.73). Reward-Mask-Only gains +0.44 over dense Direct-OPD, and its gap to S²D-OPD is 0.80. Both are smaller than 0.87. Even the gain of S²D-OPD over dense in this setting, +1.24, is only about 1.4 times this difference. KL-Mask-Only (−1.62) is the clearest result, and it is evidence against weaker regularization alone. It does not show that removing the KL anchor at the other 90% of positions is harmless once the reward is filtered. That is exactly the step from Reward-Mask-Only to S²D-OPD. Under your normalization, the same step also moves the KL average from all positions to the retained ones. The reward/KL confound is therefore neither excluded nor confirmed. For Random-10, the argument rests on the same masking and normalization, but the only numbers given are the validation peaks from Fig. 3a (51.2 vs. 51.3), and checkpoints are selected on that same validation set.
-
The selector comparison is clear against the adapted baselines but harder to read against the simple ones. JSD leads student entropy by 1.14 and reward magnitude by 1.38, again in single runs. My Q2 asked about the sum of p̄|Δ| over the top-K candidates. Response [3/3] to Reviewer yDxL instead gives the absolute value of the sum of p̄Δ, in which positive and negative log-ratios can cancel. Response [2/2] to Reviewer 8A13 gives |Δ|. Both responses report the same 36.22, so it is unclear which formula was used. Your reply to me says that the retention rate of OPD²-gate is "not fixed at 10%". Response [1/3] to Reviewer yDxL notes this exception but then says that "every selector discards 90% of states", and the General Response lists OPD²-gate among the selectors compared "Under the same 10% budget". The actual retention rate is needed to read this comparison.
Proposition 1 and the decile analysis remain the strongest parts of the paper, and S²D-OPD adds little cost. I realize the discussion window may close soon, so I list the analyses and clarifications that bear on my assessment; the new seeds and the held-out Random-10 results could change it, depending on the results:
- At least three seeds of dense Direct-OPD, S²D-OPD and Reward-Mask-Only on Qwen3-1.7B/JustRL. In your reply to Reviewer yDxL, you offered to prioritize additional seeds in a specific setting of concern. This is the setting I would name.
- Per-seed validation peaks and selected steps, with seed-level SD rather than s.e. labelled as spread.
- Which reward-magnitude formula was used, and the actual retention rate of OPD²-gate.
- Held-out results for Random-10.
- Stating consistently that p = 0.048 comes from a one-sided, problem-level test.
Response [1/2]
Thank you for the careful review and for recognizing both the theoretical diagnosis and the decile analysis. We address the seed robustness concern (W1/Q3) first, then the missing baselines (W2.1/Q2) and the reward-vs.-KL ablation (W2.2/Q1). Unless stated otherwise, the new experiments use Qwen3-1.7B/JustRL (the setting of Fig. 3, single run). All new results will be included in the revision.
W1 & Q3. Multi-seed training robustness
We retrained Qwen3-4B/JustRL with four seeds. This is one of the smallest-gain settings in Table 1 (+0.2 pp), and seed 42 is the run reported there. S²D-OPD outperforms Direct-OPD in three seeds and ties in one, with a mean gain of +0.64 ± 0.37 pp (s.e.). The +0.2 pp in Table 1 is thus at the lower end of the seed distribution.
Mean ± s.e. across four seeds.
We then averaged each problem's paired difference over the four seeds. An exact one-sided sign-flip test over the 93 problems gives p=0.048. With four seeds, the seed-level spread is still comparable to the gain, so we present this as consistent but modest evidence that the improvement is not an artifact of a single run. We will add seeds for further settings in the revision.
The smaller gains under JustRL are consistent with our analysis in Sec. 5.3. JustRL has higher teacher–reference JSD than QuestA (Fig. 4a), so dense supervision contains fewer low-divergence states for masking to remove, and the held-out gains are correspondingly smaller (Fig. 4b).
W2.1 & Q2. Alternative token-selection baselines
We compare S²D-OPD with representative token selectors under the same Direct-OPD objective and a fixed 10% per-response supervision budget:
Reward magnitude: Selects tokens with the largest expected absolute Direct-OPD reward magnitude over the student's top-K candidates. Formally: ∑v∈Vk∣pΔ∣
Student entropy: Selects tokens with the highest student predictive entropy.
TA-Direct: Adapts TA-OPD's teachability score, which combines teacher–student disagreement with teacher coverage of the student's top-K candidates, to Direct-OPD.
We additionally include OPD$^2$-gate, which uses sign agreement between the centered teacher–reference and teacher–student log-ratios to determine token retention. Unlike the other selectors, OPD$^2$-gate has no manually adjustable retention budget and therefore operates at its naturally determined retention rate rather than a fixed 10%.
JSD is the only selector that clearly improves over dense Direct-OPD. Since every selector discards most states, the gain cannot come from sparsity alone. It depends on whether the selection signal measures how much RL changed the teacher. The reward-magnitude result is especially informative. Selecting the largest Direct-OPD rewards does not help, which is consistent with Sec. 4.1: a large log-ratio need not reflect a large change in the teacher's distribution. In cost, S²D-OPD adds about 6% time per step over Direct-OPD and needs no extra forward passes. OPD²-gate is 4.4× slower than S²D-OPD because centering over 1,024 student candidates requires three additional scoring passes per rollout.