acceptodds
Under review as a conference paper at ICLR 2027

Not Every Token Is Worth Distilling: Selective Supervision for Direct-OPD

Abstract

Direct On-Policy Distillation (Direct-OPD) transfers reinforcement-learning-induced policy improvements from a small model to a larger student by using the token-level log-ratio between post-RL and pre-RL checkpoints as dense supervision on the student's own rollouts. This transfer rewards the policy shift at every state, yet the log-ratio measures only relative change: it can stay fixed even as the probability mass that both checkpoints assign to the student's candidate tokens vanishes. Through an exact construction, we show that the Direct-OPD reward and its update can remain unchanged while the Jensen-Shannon divergence (JSD) and both KL directions between the checkpoints vanish with this mass, and we note that a small JSD bounds how much the teacher's behavior changed. Motivated by this analysis, we propose Selective Supervision for Direct-OPD (SD-OPD), which ranks student-sampled states by their teacher-reference JSD and masks Direct-OPD supervision at low-divergence states, retaining only the top 10% of states per response. Across two teacher pairs and four student models ranging from 1.7B to 8B parameters, SD-OPD improves held-out accuracy over dense Direct-OPD on AIME and HMMT benchmarks in seven of eight settings and matches it in the eighth, without extra forward passes. Our code is available at https://anonymous.4open.science/r/S2D-OPD-8868.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DgWMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether every token position along a student-generated trajectory should be supervised during Direct On-Policy Distillation (Direct-OPD, Section 3, Eqs. 1–4), where a student πθ\pi _ \theta is trained using the top-KK (K=16K=16) token-level log-ratio reward Δt(v∣st)=log⁡πT(v∣st)πref(v∣st)\Delta _ t(v \mid s _ t) = \log \frac{\pi _ {\mathrm{T}}(v \mid s _ t)}{\pi _ {\mathrm{ref}}(v \mid s _ t)} between a post-RL teacher πT\pi _ {\mathrm{T}} and its pre-RL reference πref\pi _ {\mathrm{ref}}. In Section 4.1 (Eqs. 5–7 and Appendix B Proposition 1), the authors construct a clean family of distributions (Pϵ,Qϵ)(P _ \epsilon, Q _ \epsilon) showing a probability-mass mismatch: as the total mass ϵ∈(0,1)\epsilon \in (0, 1) assigned by πT\pi _ {\mathrm{T}} and πref\pi _ {\mathrm{ref}} to the student's top-KK candidates VK(st)\mathcal{V} _ K(s _ t) shrinks toward zero, the true behavioral divergence DJS(Pϵ,Qϵ)=ϵDJS(t,q)D _ {\mathrm{JS}}(P _ \epsilon, Q _ \epsilon) = \epsilon D _ {\mathrm{JS}}(\mathbf{t}, \mathbf{q}) (and both KL directions) vanishes linearly in ϵ\epsilon, yet the log-ratio rewards Δϵ(v∣st)=log⁡(tv/qv)\Delta _ \epsilon(v \mid s _ t) = \log(t _ v / q _ v) and the local Direct-OPD logit gradient gϵ=g≠0\mathbf{g} _ \epsilon = \mathbf{g} \ne \mathbf{0} (Eq. 17) remain completely unchanged.

Motivated by this mismatch, the authors propose S2D-OPD (Section 4.2, Figure 1, Eqs. 8–12), which forms (K+1)(K+1)-way coarsened distributions π~T,π~ref\widetilde{\pi} _ {\mathrm{T}}, \widetilde{\pi} _ {\mathrm{ref}} over the student's top-KK tokens plus a single unnormalized residual bucket vov _ o (Eq. 8), scores each state by the coarsened Jensen–Shannon divergence d~t=DJS(π~T∥π~ref)\tilde{d} _ t = D _ {\mathrm{JS}}(\widetilde{\pi} _ {\mathrm{T}} \Vert \widetilde{\pi} _ {\mathrm{ref}}) (Eq. 9), and retains only the top ρ=10%\rho = 10\% highest-JSD positions per response (Eq. 10) for both the policy-shift reward update and the student-anchor KL penalty (Eqs. 11–12 and Appendix A.2 Eq. 14). Evaluated across two 1.5B teacher–reference pairs (R1-Distill-1.5B -> JustRL-1.5B and Nemotron-1.5B -> QuestA-1.5B) and four students (Qwen3-1.7B, Qwen3-4B, Qwen3-8B, R1-Distill-7B) on 93 held-out competition math problems (AIME26 [30 problems] and HMMT Nov 2025 + Feb 2026 [63 problems], Table 1), S2D-OPD improves held-out Test Avg. over dense Direct-OPD in 7 of 8 configurations and ties in the 8th, lifting the 8-setting grand mean from 46.36% to 47.31% (+0.95 percentage points) with zero extra forward passes.

Critical strengths

1. Elegant theoretical diagnosis of tail-mass insensitivity in log-ratio distillation rewards.
Proposition 1 (Section 4.1, Eqs. 5–6; Appendix B, Eqs. 15–18) pinpoints a genuine flaw in top-KK log-ratio objectives: when both πT\pi _ {\mathrm{T}} and πref\pi _ {\mathrm{ref}} assign tiny probability (e.g., 10−410^{-4} vs. 10−610^{-6}) to a token in the student's top-16 candidate set, log⁡(πT/πref)\log(\pi _ {\mathrm{T}}/\pi _ {\mathrm{ref}}) produces a huge noisy reward even though the teacher's actual behavior barely moved in total variation (Figure 5). Compressing the off-top-KK probability mass into a single residual event vov _ o without renormalizing (Eq. 8) is a zero-overhead way to make JSD sensitive to how much probability mass actually supports the teacher shift.

2. Clean, zero-extra-forward-pass implementation with consistent directional improvements and strong decile diagnostics.
Because Direct-OPD already queries πT\pi _ {\mathrm{T}} and πref\pi _ {\mathrm{ref}} on the student's rollout tokens, computing the 17-way coarsened JSD d~t\tilde{d} _ t adds virtually no wall-clock cost. The decile experiment in Figure 3 (R2=0.97R^2 = 0.97 across deciles, with bottom deciles collapsing and Random 10% matching dense training) strongly supports the thesis that supervising all 100% of tokens is at best redundant and at worst harmful, and across the 8 student–teacher pairs in Table 1, S2D-OPD never underperforms dense Direct-OPD (7 wins, 1 tie).

Critical weaknesses

1. Modest average test gain (+0.95 pp) without multi-seed training variance, especially under JustRL-1.5B.
While the validation curves in Figure 2 show clear separation on AIME24/AIME25, the held-out Test Avg. improvements in Table 1—evaluated at the best validation checkpoint—average +0.95 percentage points overall and are practically flat (+0.2, +0.2, 0.0 points) on three of the four students when distilling from JustRL-1.5B (Figure 4b). Because RL/OPD training with batch size 128 and n=4n=4 rollouts over 300 steps has non-trivial run-to-run trajectory and checkpoint-selection variance, bootstrapping over the 93 test problems from a single training seed per cell (Section 5.2, lines 308–312) does not establish that +0.2 to +1.7 point differences survive across random training seeds.

2. Missing empirical comparisons against prior selective-distillation signals and missing reward-vs.-KL masking ablation.
Section 2 (pp. 2–3) surveys a rich set of token-selection criteria for distillation—including student entropy/uncertainty (Tavor et al., 2026; Ko et al., 2026), teacher confidence/margin (Jin et al., 2026; Zhou et al., 2026), token-importance weighting (TIP, Xu et al., 2026; TA-OPD, Wang et al., 2026b), and sign-agreement gating (OPD2, Heo et al., 2026)—yet Section 5 only compares S2D-OPD against dense Direct-OPD, Random 10% (on a single setting in Figure 3a), and forward/reverse KL between π~T\widetilde{\pi} _ {\mathrm{T}} and π~ref\widetilde{\pi} _ {\mathrm{ref}} (Figure 6a). Especially since Random 10% in Figure 3a (51.2) already gets within 0.1 points of dense Direct-OPD (51.3) simply by masking 90% of positions, we need (a) an ablation that masks only the reward gtRg _ t^R in Eq. (12) while keeping the student-anchor KL penalty d^t(θ)\widehat{d} _ t(\theta) unmasked across all tokens, and (b) direct empirical comparisons on the same setup against natural selection heuristics such as expected reward magnitude, student entropy, or OPD2 / TA-OPD.

Questions for authors
  1. Assessment-critical (Reward masking vs. anchor-KL masking): In Eq. (11), Eq. (12), and Appendix A.2 Eq. (14), mt=1{t∈I}m _ t = \mathbf{1}\lbrace t \in \mathcal{I} \rbrace zeroes out both the Direct-OPD reward gradient gtRg _ t^R and the student-anchor KL penalty d^t(θ)\widehat{d} _ t(\theta) at 90% of token positions (while αm+1\alpha _ {m+1} in Eq. 13 still tracks the all-token mean shift rˉm\bar{r} _ m). Notably, in Figure 3a, even Random 10% (51.2) nearly matches 100% dense training (51.3). What happens on Qwen3-1.7B and Qwen3-4B if you apply the top-10% JSD mask only to the reward term gtRg _ t^R while keeping the KL anchor active on all tokens (or if you apply Random 10% on the Table 1 held-out benchmarks)?

  2. Assessment-critical (Alternative token-selection baselines): On Qwen3-1.7B and Qwen3-4B, how does coarsened teacher–reference JSD (d~t\tilde{d} _ t, Eq. 9) compare at the same 10% budget against selecting tokens by (i) student entropy, (ii) expected reward magnitude ∑v∈VKpˉt(v)∣Δt(v∣st)∣\sum _ {v\in\mathcal{V} _ K} \bar{p} _ t(v)\lvert \Delta _ t(v\mid s _ t) \rvert, or (iii) teacher–student disagreement / TA-OPD / OPD2?

  3. Assessment-critical (Multi-seed training robustness): Can you report mean ±\pm std across at least 3 independent training seeds for a core subset of Table 1 (e.g., Qwen3-1.7B and Qwen3-4B under both JustRL-1.5B and QuestA-1.5B) to verify that the held-out gains (particularly the +0.2 pp margin on Qwen3-4B under JustRL) hold across training seeds rather than only problem-level bootstrap resamples?

6 comments
AuthorsOpenReview ↗

Response [1/2]

Thank you for the careful review and for recognizing both the theoretical diagnosis and the decile analysis. We address the seed robustness concern (W1/Q3) first, then the missing baselines (W2.1/Q2) and the reward-vs.-KL ablation (W2.2/Q1). Unless stated otherwise, the new experiments use Qwen3-1.7B/JustRL (the setting of Fig. 3, single run). All new results will be included in the revision.

W1 & Q3. Multi-seed training robustness

We retrained Qwen3-4B/JustRL with four seeds. This is one of the smallest-gain settings in Table 1 (+0.2 pp), and seed 42 is the run reported there. S²D-OPD outperforms Direct-OPD in three seeds and ties in one, with a mean gain of +0.64 ± 0.37 pp (s.e.). The +0.2 pp in Table 1 is thus at the lower end of the seed distribution.

SeedMethodAIME26HMMT (Nov / Feb)Test Avg.Δ vs. Direct (pp)
42Direct-OPD66.7747.12 (52.60 / 42.14)53.46—
42S²D-OPD68.6546.53 (51.88 / 41.67)53.66+0.20
43Direct-OPD66.5646.48 (51.77 / 41.67)52.96—
43S²D-OPD66.5646.48 (53.13 / 40.44)52.960.00
44Direct-OPD66.2546.33 (51.35 / 41.76)52.76—
44S²D-OPD66.0447.42 (53.54 / 41.86)53.43+0.67
45Direct-OPD65.3145.73 (51.25 / 40.72)52.05—
45S²D-OPD69.5846.18 (51.56 / 41.29)53.73+1.68
MeanDirect-OPD66.2246.42 (51.74 / 41.57)52.81 ± 0.29—
MeanS²D-OPD67.7146.65 (52.53 / 41.32)53.44 ± 0.17+0.64 ± 0.37

Mean ± s.e. across four seeds.

We then averaged each problem's paired difference over the four seeds. An exact one-sided sign-flip test over the 93 problems gives p=0.048p = 0.048. With four seeds, the seed-level spread is still comparable to the gain, so we present this as consistent but modest evidence that the improvement is not an artifact of a single run. We will add seeds for further settings in the revision.

The smaller gains under JustRL are consistent with our analysis in Sec. 5.3. JustRL has higher teacher–reference JSD than QuestA (Fig. 4a), so dense supervision contains fewer low-divergence states for masking to remove, and the held-out gains are correspondingly smaller (Fig. 4b).

W2.1 & Q2. Alternative token-selection baselines

We compare S²D-OPD with representative token selectors under the same Direct-OPD objective and a fixed 10% per-response supervision budget:

  • Reward magnitude: Selects tokens with the largest expected absolute Direct-OPD reward magnitude over the student's top-KK candidates. Formally: ∑v∈Vk∣pΔ∣\sum_{v\in\mathcal V_k} |p \Delta|

  • Student entropy: Selects tokens with the highest student predictive entropy.

  • TA-Direct: Adapts TA-OPD's teachability score, which combines teacher–student disagreement with teacher coverage of the student's top-KK candidates, to Direct-OPD.

We additionally include OPD$^2$-gate, which uses sign agreement between the centered teacher–reference and teacher–student log-ratios to determine token retention. Unlike the other selectors, OPD$^2$-gate has no manually adjustable retention budget and therefore operates at its naturally determined retention rate rather than a fixed 10%.

SelectorAIME26HMMTTest Avg.Δ vs. Direct (pp)Time/step (s)
Dense Direct-OPD45.4232.0436.36—49.59
Reward magnitude47.0831.0536.22−0.1451.15
Student entropy46.7731.5536.46+0.1052.35
TA-Direct (K = 16)45.4230.9535.62−0.7454.56
OPD²-gate44.3831.2535.49−0.87232.80
S²D-OPD (JSD)47.4032.9437.60+1.2452.56

JSD is the only selector that clearly improves over dense Direct-OPD. Since every selector discards most states, the gain cannot come from sparsity alone. It depends on whether the selection signal measures how much RL changed the teacher. The reward-magnitude result is especially informative. Selecting the largest Direct-OPD rewards does not help, which is consistent with Sec. 4.1: a large log-ratio need not reflect a large change in the teacher's distribution. In cost, S²D-OPD adds about 6% time per step over Direct-OPD and needs no extra forward passes. OPD²-gate is 4.4× slower than S²D-OPD because centering over 1,024 student candidates requires three additional scoring passes per rollout.

AuthorsOpenReview ↗

Response [2/2]

W2.2 & Q1. Reward masking vs. anchor-KL masking

We apply the top-10% JSD mask to each term separately. A masked term is averaged over the retained positions, and an unmasked term over all valid positions. As you note, the adaptive KL coefficient (Eq. 13) is computed from all valid positions in every variant, so it is identical across them.

VariantReward termKL termAIME26HMMTTest Avg.Δ (pp)
Direct-OPDallall45.4232.0436.36—
Reward-Mask-Onlytop-10%all47.1931.8536.80+0.44
KL-Mask-Onlyalltop-10%43.7530.4634.74−1.62
S²D-OPDtop-10%top-10%47.4032.9437.60+1.24
  • Filtering the reward is the beneficial component. Masking only the reward improves accuracy. Masking only the KL anchor keeps the low-JSD rewards, removes their regularization, and degrades accuracy. Weaker regularization therefore does not explain the gain.
  • Joint masking adds a further gain. With reward-only masking, the anchor acts on all positions but the reward on only 10% of them, which likely over-regularizes the sparse reward. Joint masking keeps both terms on the same positions, as in Direct-OPD. The gap between +0.44 and +1.24 suggests that this balance also contributes. These are single runs, so we treat the size of the gap with caution.
  • On Random-10. You rightly note that Random-10 (51.2) nearly matches dense training (51.3). Random-10 uses exactly the same masking and normalization as S²D-OPD, including masking both terms. It differs only in which states are retained. Masking 90% of positions therefore does not by itself improve or regularize training. The gain of S²D-OPD (52.9) comes from retaining high-JSD states, and the low-JSD deciles under the same masking fall below the initial student.

Dear Authors,

Thank you for the detailed responses. You engaged seriously with all three of my questions. You trained Qwen3-4B/JustRL with four seeds, compared four alternative selectors, and masked the reward and KL terms separately. You also clarified how each masked term is normalized, and that the adaptive KL coefficient is computed from all valid positions in every variant. The gap to TA-Direct and OPD²-gate is the clearest new result: S²D-OPD leads them by 1.98 and 2.11 pp in single runs (OPD²-gate at its own retention rate). However, the core issue is unchanged. The gains are small and are not yet separated from training noise, so I keep my rating at 2 for three reasons.

  1. The four-seed result does not separate the gain from variation across seeds. The gain is +0.64 ± 0.37 pp (s.e.), and it comes from one setting only. The four per-seed differences are +0.20, 0.00, +0.67 and +1.68, so their standard deviation is about 0.75, larger than the mean gain. Seed 45 alone accounts for 1.68 of the total 2.55. Without it, the mean gain is 0.29. In your reply to me, you acknowledge that "the seed-level spread is still comparable to the gain". The p = 0.048 comes from an exact one-sided sign-flip test over the 93 problems, applied to seed-averaged differences. As you note in Response [2/3] to Reviewer yDxL, this test "still measures uncertainty over test problems". The null distribution of a sign-flip test is symmetric, so the two-sided value is 2 × 0.048 = 0.096. In your later reply to Reviewer yDxL, the same result appears as "+0.64 ± 0.37 pp (p = 0.048 < 0.05)" without "one-sided", which reads as seed-level significance. The General Response describes it accurately as problem-level but still omits one-sided.

  2. The new ablations are single runs, and the key differences in the reward/KL ablation fall within run-to-run noise. Your own K sweep gives a rough noise scale. Scoring with K = 16 and K = 32 yields masks that overlap the full-vocabulary mask by 95.40% and 98.08%, so the two runs should be nearly identical. Their Test Avg. still differs by 0.87 (37.60 vs. 36.73). Reward-Mask-Only gains +0.44 over dense Direct-OPD, and its gap to S²D-OPD is 0.80. Both are smaller than 0.87. Even the gain of S²D-OPD over dense in this setting, +1.24, is only about 1.4 times this difference. KL-Mask-Only (−1.62) is the clearest result, and it is evidence against weaker regularization alone. It does not show that removing the KL anchor at the other 90% of positions is harmless once the reward is filtered. That is exactly the step from Reward-Mask-Only to S²D-OPD. Under your normalization, the same step also moves the KL average from all positions to the retained ones. The reward/KL confound is therefore neither excluded nor confirmed. For Random-10, the argument rests on the same masking and normalization, but the only numbers given are the validation peaks from Fig. 3a (51.2 vs. 51.3), and checkpoints are selected on that same validation set.

  3. The selector comparison is clear against the adapted baselines but harder to read against the simple ones. JSD leads student entropy by 1.14 and reward magnitude by 1.38, again in single runs. My Q2 asked about the sum of p̄|Δ| over the top-K candidates. Response [3/3] to Reviewer yDxL instead gives the absolute value of the sum of p̄Δ, in which positive and negative log-ratios can cancel. Response [2/2] to Reviewer 8A13 gives |Δ|. Both responses report the same 36.22, so it is unclear which formula was used. Your reply to me says that the retention rate of OPD²-gate is "not fixed at 10%". Response [1/3] to Reviewer yDxL notes this exception but then says that "every selector discards 90% of states", and the General Response lists OPD²-gate among the selectors compared "Under the same 10% budget". The actual retention rate is needed to read this comparison.

Proposition 1 and the decile analysis remain the strongest parts of the paper, and S²D-OPD adds little cost. I realize the discussion window may close soon, so I list the analyses and clarifications that bear on my assessment; the new seeds and the held-out Random-10 results could change it, depending on the results:

  • At least three seeds of dense Direct-OPD, S²D-OPD and Reward-Mask-Only on Qwen3-1.7B/JustRL. In your reply to Reviewer yDxL, you offered to prioritize additional seeds in a specific setting of concern. This is the setting I would name.
  • Per-seed validation peaks and selected steps, with seed-level SD rather than s.e. labelled as spread.
  • Which reward-magnitude formula was used, and the actual retention rate of OPD²-gate.
  • Held-out results for Random-10.
  • Stating consistently that p = 0.048 comes from a one-sided, problem-level test.
Reviewer_yDxLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies selective supervision for Direct-OPD, which transfers a small teacher's reinforcement-learning improvements to a larger student using the token-level log-ratio between the teacher's post-RL and pre-RL checkpoints. It constructs a case where this reward and its local logit gradient remain unchanged as the probability mass supporting the teacher's change vanishes. Motivated by this observation, S²D-OPD ranks student-generated positions by teacher–reference JSD over the student's top-16 candidates and a residual category, then applies both the reward and student-anchor KL only at the top 10% of positions per response. Across two teacher pairs and four students, it reports seven improvements and one tie over dense Direct-OPD, with a mean held-out gain of 0.95 percentage points. Additional experiments examine divergence bins, random selection, alternative divergences, and retention rates.

Critical strengths
  1. The paper identifies a concrete limitation of the transferred reward. Direct-OPD scores relative probability changes, while the proposed construction jointly shrinks the teacher's and reference's probability mass on the student's candidates. The reward remains fixed even as both KL directions and JSD vanish. This cleanly explains why a large log-ratio need not represent a substantial change in the teacher's distribution, and motivates a selector that preserves absolute probability mass rather than renormalizing it away.

  2. The evaluation provides encouraging evidence beyond a single model comparison. The paper separates checkpoint selection on AIME 2024/2025 from evaluation on 93 later AIME/HMMT problems, and tests eight teacher–student combinations. The random-10% control and divergence-bin experiments additionally distinguish reducing supervision from choosing which supervision to retain. The mostly positive held-out results and sensitivity analyses make the proposed modification worth investigating.

Critical weaknesses
  1. The idea that not all tokens are equally valuable for learning is already well explored, so the incremental novelty needs clearer justification. S²D-OPD retains only the 10% of positions with the largest teacher–reference JSD of Direct-OPD [1]. However, Beyond the 80/20 Rule [6] already shows that restricting policy-gradient updates to a minority of high-entropy tokens can maintain or improve reasoning performance; Sparse but Critical [2] identifies sparse, consequential RL-induced changes; and TIP [4] and TA-OPD [5] study selective supervision in on-policy distillation. The distinctive contribution here is therefore the teacher–reference JSD criterion and its mass-mismatch analysis, rather than unequal token importance itself. The paper should clarify what these add to prior understanding and compare against representative existing selectors, since dense and random-selection baselines alone do not establish that added value.

  2. The experiments do not isolate filtering the reward from changing the regularization. Equations 11–12 mask both the policy-shift reward and the KL anchor to the initial student, then average over retained positions. Consequently, selection changes the distribution of reward updates and regularization simultaneously. Likewise, training exclusively on a low-JSD decile amplifies its relative weight compared with dense training. The resulting gains and low-bin collapses therefore do not by themselves establish that removing harmful reward supervision is the main mechanism. Separate reward-mask and KL-mask controls are needed for this interpretation.

  3. The small gains and stability claims lack training-seed evidence. The reported mean improvement is 0.95 points, with gains of only 0.2 points for the JustRL/Qwen3-4B and JustRL/Qwen3-8B combinations. Appendix F explicitly states that each configuration is trained once. The paired problem bootstrap appropriately quantifies uncertainty over evaluation problems for these checkpoints, but does not establish reproducibility across training runs. Repeated runs on representative settings are particularly important for the smaller improvements and the claim of more stable training.

  4. The empirical connection between the mass-mismatch construction and supervision utility remains incomplete. The construction holds the student's preferences fixed while reducing the mass that both teacher checkpoints assign to its candidates. Yet a change that is rare under the teacher can still matter at an action frequently considered by the student. The paper correctly acknowledges that the construction does not prove a transfer benefit, but does not quantify how often the constructed low-mass regime occurs or distinguish its effects from low divergence caused by teacher–reference agreement at substantial mass. The decile study supports JSD as a useful heuristic in one setting; it does not yet identify probability-mass mismatch as the source of the gains.

  5. The selector's dependence on the candidate set is not evaluated. S²D-OPD computes JSD using 16 student candidates and merges all remaining tokens into one category. As the paper correctly notes, this measures change only at that resolution and can hide redistribution within the tail. The robustness experiments vary divergence type, selection scope, and retention rate, but keep this support fixed. Without a support-size or mask-agreement analysis, it remains unclear how much the selected positions and transfer gains depend on the top-16 implementation.

[1] Weak-to-Strong Generalization via Direct On-Policy Distillation. 2026.

[2] Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs. 2026.

[3] On-Policy Delta Distillation. 2026.

[4] TIP: Token Importance in On-Policy Distillation. 2026.

[5] Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation. 2026.

[6] Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. 2025.

Questions for authors
  1. What new insight does teacher–reference JSD provide beyond existing findings on unequal token importance and selective updates [2,4–6]? Under a matched Direct-OPD objective and retention budget, how does it compare with representative entropy- or support-based selectors and OPD²'s signal-agreement gate [3]?

  2. Do the gains persist when only the reward is masked and the KL anchor remains dense? What happens when only the KL term is masked? Please make the normalization of both terms explicit in these comparisons.

  3. Can representative configurations be repeated across training seeds, including one setting with a small reported improvement? Please separate variation across training runs from the existing uncertainty over test problems.

  4. Among discarded positions, how many have low candidate mass under both checkpoints, versus substantial mass but similar distributions? Do these groups differ in their effect on learning, and does candidate mass add explanatory value beyond JSD alone?

  5. Keeping the Direct-OPD candidate support fixed, how sensitive are selection masks to the support size used for scoring JSD? A comparison against full-vocabulary scores on a fixed rollout sample, followed by a representative training check, would clarify this dependence.

5 comments
AuthorsOpenReview ↗

Response [1/3]

Thank you for the detailed and constructive review. Below we address each weakness and question. Unless stated otherwise, the new experiments use the setting of Fig. 3 (Qwen3-1.7B student, JustRL teacher pair, single run). All new results will be included in the revision.

W1 & Q1. Novelty relative to existing token-selection work

We agree that unequal token importance is well established, and we do not claim it as our contribution. Our contribution is a selection criterion tailored to transferring a policy shift, together with evidence for why it is needed.

A different signal. Existing selectors are based on the student, the teacher, or the disagreement between them. Student-based selectors use entropy or uncertainty, as in Beyond the 80/20 Rule [6]. Teacher-based selectors use the teacher's confidence. Disagreement-based selectors compare teacher and student, as in TIP [4] and TA-OPD [5]. In Direct-OPD, however, the supervision is the change between two teacher checkpoints, and none of these signals measures how much RL changed the teacher at a given state. Sparse but Critical [2] (Meng et al., 2026, which we build on in Sec. 5.1) shows that RL-induced changes are concentrated in a small fraction of tokens. Its contribution is analytical. We show that Direct-OPD's dense reward does not respect this sparsity, and that enforcing it improves transfer. OPD² [3] also relies on log-ratios, so it inherits the mass-insensitivity described in Sec. 4.1.

A diagnosis of why selection is needed. Prop. 1 identifies a specific failure mode of the log-ratio reward. The reward can stay large while the teacher's behavioral change vanishes. This argues against selecting by reward magnitude and in favor of selecting by divergence.

Matched comparison. Following your suggestion, we compare selectors under the same Direct-OPD objective and the same 10% per-response budget. OPD²-gate is the exception: it uses its own sign-agreement rule, so its retention rate is not fixed at 10%.

SelectorAIME26HMMTTest Avg.Δ vs. Direct (pp)Time/step (s)
Dense Direct-OPD45.4232.0436.36—49.59
Reward magnitude47.0831.0536.22−0.1451.15
Student entropy (cf. [6])46.7731.5536.46+0.1052.35
TA-Direct ([5], adapted, K = 16)45.4230.9535.62−0.7454.56
OPD²-gate [3]44.3831.2535.49−0.87232.80
S²D-OPD (JSD)47.4032.9437.60+1.2452.56

Only JSD selection clearly improves over dense Direct-OPD. At the same budget, the student-entropy criterion of [6] changes little, and the other selectors underperform dense training. Since every selector discards 90% of states, the improvement cannot come from sparsity alone. It depends on whether the selection signal measures the teacher's change. S²D-OPD costs about 6% more time per step than Direct-OPD and needs no extra forward passes. OPD²-gate is 4.4× slower because of its additional scoring passes. We will add [6] and this comparison to the paper.

References

[1] Feng et al. Weak-to-Strong Generalization via Direct On-Policy Distillation. 2026.

[2] Meng et al. Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs. ICLR 2026.

[3] Heo et al. On-Policy Delta Distillation. 2026.

[4] Xu et al. TIP: Token Importance in On-Policy Distillation. 2026.

[5] Wang et al. Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation. 2026.

[6] Wang et al. Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. 2025.

AuthorsOpenReview ↗

Response [2/3]

W2 & Q2. Separating reward filtering from regularization

We apply the top-10% JSD mask to each term separately. In each variant, a masked term is averaged over the retained positions and an unmasked term over all valid positions. Thus Reward-Mask-Only averages the reward over retained positions and the KL anchor over all positions, and KL-Mask-Only does the reverse. In all variants, the adaptive KL coefficient is computed from all valid positions, as in App. A.

VariantReward termKL termAIME26HMMTTest Avg.Δ (pp)
Direct-OPDallall45.4232.0436.36—
Reward-Mask-Onlytop-10%all47.1931.8536.80+0.44
KL-Mask-Onlyalltop-10%43.7530.4634.74−1.62
S²D-OPDtop-10%top-10%47.4032.9437.60+1.24
  • Reward filtering is the beneficial component. Masking only the reward improves accuracy, whereas masking only the KL anchor degrades it. The second variant keeps the low-JSD rewards while removing their regularization. Weaker regularization therefore does not explain the gain.
  • Joint masking adds a further gain. Under reward-only masking, the KL anchor acts on all positions but the reward on only 10% of them, which likely over-regularizes the sparse reward. Joint masking keeps the reward and anchor on the same positions, as in Direct-OPD. The gap between the two variants (+0.44 vs. +1.24) suggests that this balance also contributes, so we do not attribute the full gain to reward filtering. These are single runs, so we treat the size of this gap with caution.
  • Reweighting does not explain the low-bin collapse. Random-10 uses exactly the same normalization as the decile runs: both terms are averaged over 10% of positions, which gives each retained position the same weight. Yet Random-10 matches dense Direct-OPD (51.2 vs. 51.3; Fig. 3a), whereas the lowest deciles collapse. The harm comes from which states are retained, not from how they are weighted.

W3 & Q3. Training-seed evidence

We retrained Qwen3-4B/JustRL with four seeds. This is one of the smallest-gain settings in Table 1 (+0.2 pp), and seed 42 is the run reported there. S²D-OPD outperforms Direct-OPD in three seeds and ties in one, with a mean gain of +0.64 ± 0.37 pp (s.e.). In the second step, we averaged each problem's paired difference over the four seeds. An exact one-sided sign-flip test over the 93 problems then gives p=0.048p = 0.048. This test still measures uncertainty over test problems. Across training runs, the seed-level spread (±0.37 pp) is not small relative to the gain. We therefore view this as consistent but modest evidence and will report it as such.

SeedMethodAIME26HMMT (Nov / Feb)Test Avg.Δ vs. Direct (pp)
42Direct-OPD66.7747.12 (52.60 / 42.14)53.46—
42S²D-OPD68.6546.53 (51.88 / 41.67)53.66+0.20
43Direct-OPD66.5646.48 (51.77 / 41.67)52.96—
43S²D-OPD66.5646.48 (53.13 / 40.44)52.960.00
44Direct-OPD66.2546.33 (51.35 / 41.76)52.76—
44S²D-OPD66.0447.42 (53.54 / 41.86)53.43+0.67
45Direct-OPD65.3145.73 (51.25 / 40.72)52.05—
45S²D-OPD69.5846.18 (51.56 / 41.29)53.73+1.68
MeanDirect-OPD66.2246.42 (51.74 / 41.57)52.81 ± 0.29—
MeanS²D-OPD67.7146.65 (52.53 / 41.32)53.44 ± 0.17+0.64 ± 0.37

Mean ± s.e. across four seeds.

AuthorsOpenReview ↗

Response [3/3]

W4 & Q4. How often does the mass mismatch arise, and what does JSD remove?

We agree that Prop. 1 is an idealized boundary case and does not by itself show where the gains come from. We therefore ask two empirical questions: does the underlying mechanism, large log-ratios on candidates that carry almost no teacher or reference mass, occur in practice, and are such large relative changes worth supervising?

(a) Candidate-level mismatch is common among discarded states. On 512 training rollouts (Qwen3-1.7B/JustRL), we counted, at each state, the student's top-16 candidates for which both πT\pi_{\text{T}} and πref\pi_{\text{ref}} are below 10−310^{-3} yet ∣Δt∣>ln⁡2|\Delta_t| > \ln 2. Among the states S²D-OPD discards, 67.47% have at least 8 such candidates and 85.73% have at least 4. These results show that large log-ratios on low-probability candidates frequently occur among discarded states.

(b) Selecting by reward magnitude does not help. If large relative changes were valuable regardless of the mass behind them, selecting states by expected reward magnitude ∑v∈VK∣pˉt(v)Δt(v∣st)∣\sum_{v\in\mathcal{V}_K}|\bar{p}_t(v)\Delta_t(v\mid s_t)| should work at least as well as JSD. Under the same 10% budget, it does not:

SelectionAIME26HMMTTest Avg.
Dense Direct-OPD (100%)45.4232.0436.36
Reward magnitude (10%)47.0831.0536.22
S²D-OPD (JSD, 10%)47.4032.9437.60

Reward-magnitude selection is no better than dense training (36.22 vs. 36.36), while JSD selection gains +1.38 pp over it (+1.24 pp over dense). This is consistent with large relative shifts not being reliable supervision when little probability mass supports them.

W5 & Q5. Sensitivity to the scoring support

Here we keep the Direct-OPD training support fixed at K = 16 and vary only the support used to compute JSD. On fixed rollouts, we compare the resulting masks with the full-vocabulary JSD mask. We then train with each scoring support.

Scoring KMask overlap with full vocab.Full-vocab. JSD mass retainedSpearman ρTest Avg.
890.16%98.37%0.993037.27
16 (default)95.40%99.50%0.997137.60
3298.08%99.85%0.998236.73
6499.19%99.93%0.998937.67
12899.65%99.96%0.9992—

Dense Direct-OPD: 36.36.

  • The masks barely depend on the support. At K = 16, 95.4% of selected positions coincide with full-vocabulary selection (Spearman ρ = 0.997). Coarsening the tail into one residual token therefore rarely changes which states are retained.
  • The gains hold across support sizes. Every scoring support from K = 8 to 64 improves over dense Direct-OPD, with Test Avg. ranging from 36.73 to 37.67. We see no monotone trend with K. We keep K = 16 because it reuses the candidate set Direct-OPD already computes.
Reviewer_8A13OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a token-selection method for on-policy distillation. The method only keeps tokens with large reference-teacher JSD for distillation. Empirical study shows that the teacher to student distillation performance is highly correlated with this metric and the resulting method is stronger than direct OPD.

Critical strengths
  1. The method is well-motivated and the performance compared to direct OPD is stronger.
  2. They found that reference-teacher token JSD has a strong correlation with the distillation effectiveness.
Critical weaknesses
  1. The evaluation is limited. Only two held-out datasets are used and the number of problems in the two datasets is only 93 in total, which is small.
  2. Yet the method is well-motivated, the mechanism of JSD selection is still unclear.
Questions for authors
  1. For random 10, which tokens are selected? I'm curious how many of them are of high JSD.
3 comments
AuthorsOpenReview ↗

Response [1/2]

Thank you for the careful review and for recognizing the motivation of our method and the correlation between JSD and transfer. We address the size of the evaluation (W1), the mechanism of JSD selection (W2), and the composition of Random-10 (Q1) below. All new results will be included in the revision.

W1. Limited evaluation

Choice of benchmarks. The held-out set consists of three competitions: AIME 2026 (30 problems), HMMT Nov. 2025 (30) and HMMT Feb. 2026 (33); the two HMMT sets are merged into one column in Table 1. Following MathArena [1], we restrict held-out evaluation to competitions released after every student model, so that gains cannot come from data contamination. For this reason, we excluded older benchmarks such as MATH-500 [2], AMC23 and OlympiadBench [3]. Recent OPD studies [5, 6] use the same competitions.

Statistical reliability. We agree that 93 problems is a small set, and we account for this in two ways. First, each problem is evaluated with 32 samples (avg@32 [4]), and the comparison covers eight teacher–student settings. Pooled over these settings, a paired bootstrap over problems gives a mean gain of +0.95 pp (95% CI [0.40, 1.54]; exact one-sided sign-flip test p=6.4×10−4p = 6.4\times10^{-4}; Sec. 5.2).

Second, the test above only captures uncertainty over test problems, so we also retrained Qwen3-4B/JustRL with four seeds. This is one of the smallest-gain settings in Table 1 (+0.2 pp), and seed 42 is the run reported there. S²D-OPD outperforms Direct-OPD in three seeds and ties in one, with a mean gain of +0.64 ± 0.37 pp (s.e.).

SeedMethodAIME26HMMT (Nov / Feb)Test Avg.Δ vs. Direct (pp)
42Direct-OPD66.7747.12 (52.60 / 42.14)53.46—
42S²D-OPD68.6546.53 (51.88 / 41.67)53.66+0.20
43Direct-OPD66.5646.48 (51.77 / 41.67)52.96—
43S²D-OPD66.5646.48 (53.13 / 40.44)52.960.00
44Direct-OPD66.2546.33 (51.35 / 41.76)52.76—
44S²D-OPD66.0447.42 (53.54 / 41.86)53.43+0.67
45Direct-OPD65.3145.73 (51.25 / 40.72)52.05—
45S²D-OPD69.5846.18 (51.56 / 41.29)53.73+1.68
MeanDirect-OPD66.2246.42 (51.74 / 41.57)52.81 ± 0.29—
MeanS²D-OPD67.7146.65 (52.53 / 41.32)53.44 ± 0.17+0.64 ± 0.37

Mean ± s.e. across four seeds.

We then averaged each problem's paired difference over the four seeds. An exact one-sided sign-flip test over the 93 problems gives p=0.048p = 0.048. The improvement is modest, but it is consistent across settings and training seeds.

References

[1] Balunović et al. MathArena: Evaluating LLMs on Uncontaminated Math Competitions. NeurIPS 2025.

[2] Lightman et al. Let's Verify Step by Step. ICLR 2024.

[3] He et al. OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems. ACL 2024.

[4] Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. NeurIPS 2025.

[5] Hu et al. Teaching a Moving Student: Rethinking the Curriculum of On-Policy Distillation. arXiv:2609.25048, 2026.

[6] Hou et al. Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation. arXiv:2610.21601, 2026.

AuthorsOpenReview ↗

Response [2/2]

W2. Mechanism of JSD selection

Our evidence supports a single mechanism. JSD selection removes states where Direct-OPD makes a substantial update even though the teacher's behavior barely changed, and these updates actively harm the student.

(i) Reward and behavioral change can decouple. The Direct-OPD reward is a log-ratio. It measures relative probability change and ignores the probability mass behind it. Prop. 1 shows that the reward and its update can stay fixed while the teacher–reference JSD vanishes. Conversely, a small JSD bounds how much the teacher's probability of any token can change (Eq. 7). Fig. 5 gives a real example: RL lowers the sampled token's probability from 0.12 to 2×10−52\times10^{-5}. This gives the state the largest reward magnitude among the response's 8,192 states, but the total variation is only 0.12, so JSD masks it.

(ii) JSD is not a proxy for large rewards. To test this, we select the top 10% of states per response by expected reward magnitude ∑v∈VK∣pΔ∣\sum_{v\in\mathcal V_K}|p\Delta| and compare it with JSD selection. The setting is that of Fig. 3 (Qwen3-1.7B/JustRL, single run):

Selection (10% of states)AIME26HMMTTest Avg.
Dense Direct-OPD (100%)45.4232.0436.36
Reward magnitude47.0831.0536.22
S²D-OPD (JSD)47.4032.9437.60

Selecting the largest rewards is no better than dense Direct-OPD, whereas JSD selection improves Test Avg. by 1.24 pp. What matters for transfer is therefore whether a reward reflects a substantial change in the teacher's distribution, not how large the reward is.

(iii) Low-JSD updates are harmful, not merely weak. When the student trains on a single JSD decile alone (Fig. 3a), performance rises with JSD percentile. The three lowest deciles end below the initial student, and the two lowest collapse. Top-16 overlap shows why (Fig. 7). The top decile moves the student toward the teacher while preserving its overlap with the reference. Lower deciles move the student away from the reference, with much smaller gains in teacher overlap. A new ablation points the same way: applying the JSD mask only to the reward improves Test Avg. by 0.44 pp over Direct-OPD. Applying the mask only to the KL anchor lowers it by 1.62 pp, because this keeps the low-JSD rewards and removes their regularization. Finally, masking helps more under the lower-divergence teacher pair (QuestA; Fig. 4). This is consistent with S²D-OPD working mainly by removing harmful low-divergence updates, rather than by concentrating on the largest shifts.

Q1. Which tokens does Random-10 select?

Random-10 samples 10% of valid positions uniformly within each response, so its selected states follow the same JSD distribution as dense supervision. In expectation, only 10% of them come from the top-JSD decile that S²D-OPD retains. The remaining 90% come from lower deciles, including about 30% from the three lowest deciles, which are harmful when trained on alone. This explains why Random-10 behaves like dense Direct-OPD (peak validation accuracy 51.2 vs. 51.3) rather than like the top decile (52.9): it reduces the amount of supervision without changing its composition. Consistent with this, Random-10 lies on the same trend line between mean training JSD and performance gain as the ten deciles (Fig. 3b).

Thank you for your response. Although I find the method to be somewhat insightful, the evaluation and the analysis are still too weak for the readers to better understand the method. Therefore, I will keep my score as weak rejection.

Comments on the paper

3 comments

Dear Reviewers,

I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their careful and constructive feedback. Reviewer 8A13 particularly appreciated the clear motivation, consistent improvement over Direct-OPD, and correlation between JSD and distillation effectiveness. Reviewer yDxL acknowledged our theoretical finding that log-ratio rewards are insensitive to absolute probability mass, as well as the encouraging results across eight teacher–student configurations. Reviewer DgWM highlighted the elegant theoretical analysis, efficient implementation without additional forward passes, and decile experiments demonstrating the limitations of dense supervision.

We addressed the main concerns with additional experiments. Unless noted, they use Qwen3-1.7B/JustRL and a 10% per-response supervision budget.

  • Reviewers yDxL and DgWM questioned the statistical reliability of our results, and Reviewer 8A13 raised concerns about benchmark coverage. To avoid data contamination, we evaluate only on competitions released after all student models: AIME 2026 and HMMT Nov. 2025 / Feb. 2026, with 93 problems in total, each sampled 32 times. Pooled across eight teacher–student settings, S$^2$D-OPD improves over Direct-OPD by **+0.95 pp (95% CI [0.40, 1.54], p = 6.4 × 10$^{-4})∗∗.Toaccountfortrainingrandomness,weretrainedQwen3−4B/JustRL,oneofthesmallest−gainsettings(+0.2ppinTable1),withfourseeds.S)**. To account for training randomness, we retrained Qwen3-4B/JustRL, one of the smallest-gain settings (+0.2 pp in Table 1), with four seeds. S^2$D-OPD improves in three seeds and ties in one, with a mean gain of +0.64 ± 0.37 pp (s.e.). A problem-level sign-flip test on seed-averaged differences gives p = 0.048. The gains are modest but consistent.

  • Reviewers yDxL and DgWM questioned the novelty and added value of JSD selection over existing token-selection criteria. Existing selectors score states using the student (entropy), the teacher (confidence), or teacher–student disagreement (TIP, TA-OPD). Unlike these criteria, teacher–reference JSD directly measures the magnitude of RL-induced distributional changes while accounting for absolute probability mass. Under the same 10% budget, S$^2$D-OPD reaches 37.60% accuracy, outperforming dense Direct-OPD (36.36%), reward-magnitude selection (36.22%), student-entropy selection (36.46%), TA-OPD adapted to Direct-OPD (35.62%). OPD$^2$ gate achieves 35.49% using its own sign-agreement rule, which determines retention during training. Every selector discards most states, yet only JSD clearly improves, indicating that supervision sparsity alone cannot explain the gains.

  • Reviewer 8A13 asked why JSD selection works, and Reviewers yDxL and DgWM asked whether the gain comes from reward filtering or from weaker KL regularization. Our experiments suggest that the gain comes from which states are supervised, not merely from how many. Training on a low-JSD decile reduces performance below the initial student. Randomly keeping 10% of states, with identical masking and normalization, performs similarly to dense training (51.2 vs. 51.3). Masking only the reward improves accuracy to 36.80%, masking only the KL anchor lowers it to 34.74%, and masking both reaches 37.60%. Reward filtering is therefore a key contributor, and weaker regularization alone cannot explain the gain. Together, these results are consistent with JSD selection improving supervision quality by prioritizing states with meaningful teacher–reference distributional changes.

  • Reviewer yDxL questioned whether the probability-mass mismatch identified in Proposition 1 occurs in practice. In 512 training rollouts, among the states that S$^2$D-OPD discards, 67.47% contain at least eight of the student's top-16 candidates with large log-ratios (∣Δ∣>ln⁡2|\Delta| > \ln 2) but negligible teacher and reference probabilities (< 10$^{-3}$). Consistent with this, selecting the states with the largest reward magnitudes performs no better than dense training (36.22% vs. 36.36%). This supports the practical relevance of the identified failure mode, although we do not claim it is the sole causal explanation for the gains.

  • Reviewer yDxL questioned the sensitivity of JSD selection to the number of candidate tokens K used to compute it. With the default K = 16, 95.40% of selected positions coincide with full-vocabulary JSD selection (Spearman ρ = 0.997). Every tested value from K = 8 to 64 outperforms dense Direct-OPD (36.73–37.67% vs. 36.36%). Because S$^2$D-OPD reuses the candidates Direct-OPD already computes, it requires no additional forward passes and adds only approximately 6% time per training step.

We believe these additional experiments and clarifications strengthen the evidence for S$^2$D-OPD as a simple, efficient, and effective approach that selects informative distillation supervision based on substantial teacher–reference distributional changes. We sincerely hope they address the reviewers' concerns and respectfully invite the reviewers and the Area Chair to reconsider their assessment of our work.

AuthorsOpenReview ↗

Summary of Paper and Rebuttal

Dear PC, SAC, and AC,

We thank the reviewers for their constructive comments and suggestions, and we appreciate the substantial workload Area Chairs are carrying this year. Below, we briefly summarize our core contributions, followed by the key points raised by reviewers and our responses.

Core Contributions and Significance

  • Direct-OPD enables weak-to-strong transfer by distilling a small model's RL-induced policy shift to a larger student. We identify a fundamental limitation: its log-ratio rewards remain unchanged even as the supporting probability mass and teacher–reference divergence vanish. We formalize this as a probability-mass mismatch (Prop. 1; recognized by Reviewers yDxL and DgWM) and show through controlled decile diagnostics that such low-divergence supervision is actively harmful (acknowledged by all reviewers).

  • Based on this diagnosis, we propose S²D-OPD, which retains only the top-10% highest-JSD states per response at zero extra forward-pass cost. Across 8 teacher–student settings (1.7B–8B), it consistently improves over dense Direct-OPD (7 wins, 1 tie; acknowledged by all reviewers) while stabilizing late-stage training.

Concerns Addressed During Rebuttal

  • Evaluation Setup: We clarified our evaluation protocol. We selected three recent benchmarks specifically to mitigate data contamination, aligning with standard practice in recent literature. Furthermore, reporting avg@32 substantially reduces evaluation variance.

  • Stability & Statistical Significance: To address concerns regarding the smallest observed gain (Qwen3-4B with the JustRL teacher), we conducted experiments across four random seeds. Our method achieves 3 wins and 1 tie against the baseline, and a problem-level sign-flip test on the seed-averaged differences yields p=0.048p = 0.048. These results indicate that the improvements are modest yet consistent, achieved with negligible computational overhead.

  • Comparison with Alternative Selective Mechanisms: We empirically compared our approach against all selective mechanisms suggested by the reviewers, even those not originally designed for the Direct-OPD setting. The results demonstrate that our JSD-based filtering mechanism is the most effective, whereas alternative mechanisms often degrade performance.

  • Ablation on Reward vs. KL Masking: As requested, we added ablations that mask only the reward term or only the KL term. Masking only the KL term degrades performance because it effectively removes the anchor constraint, while masking only the reward term yields only marginal gains because it effectively strengthens the KL constraint by a factor of 10.

  • Empirical Validation of Theoretical Motivation: To directly validate our motivation, we analyzed token-level statistics and found that in 67.47% of states, at least 8 of the top-16 tokens have negligible teacher and reference probabilities (<10−3< 10^{-3}) while simultaneously exhibiting large reward magnitudes (∣Δ∣>ln⁡2|\Delta| > \ln 2).

  • Sensitivity to Top-KK JSD Approximation: We evaluated the sensitivity of top-KK JSD computation across K∈{8,16,32,64}K \in \{8, 16, 32, 64\}. All configurations maintain over 90% overlap with full-vocabulary JSD and consistently outperform Direct-OPD in average test performance, confirming the robustness of our method.

Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.