acceptodds
Under review as a conference paper at ICLR 2027

Not All Disagreement Is Equally Teachable: Token Teachability in On-Policy Distillation

Abstract

On-policy distillation (OPD) trains a student on its own rollouts with token-level teacher supervision. Recent selective OPD methods exploit the non-uniformity of OPD signals by prioritizing high-entropy or high-disagreement tokens. We revisit this principle and ask: which teacher signals are more teachable under the student's current predictive state? Using a fixed-context diagnostic that measures same-context teacher–student KL reduction, we show that raw disagreement conflates two qualitatively distinct cases: support-aligned corrections, where teacher mass lies within the student's top-\(K\) candidates, and support-misaligned corrections, which place relatively more mass beyond its local support. We formalize this distinction as token teachability and show that it reveals heterogeneity within high-disagreement regions. The analysis shows that support-conditioned disagreement improves sparse-selection utility, while acting as a refinement rather than a replacement for KL disagreement. Motivated by these findings, we propose Teachability-Aware OPD (TA-OPD), which applies OPD loss to high-teachability positions without reward models or verifiers. Across Qwen2.5 and Qwen3 settings, TA-OPD improves over entropy- and divergence-based selectors, often matching or surpassing full-token OPD by selecting 5%-10% high-teachability tokens. These results show that OPD depends on how disagreement aligns with the student's local support, not merely dense supervision or high-disagreement tokens. Code and infrastucture are released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy distillation, LLM, Token Teachability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rjA7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies which teacher signals are useful in on-policy distillation. It distinguishes disagreement aligned with the student’s local support from disagreement outside that support, and proposes TA-OPD to select teachable token positions. Experiments show that supervising a small subset of token positions can match or outperform full-token OPD.

Critical strengths

The distinction between disagreement magnitude and support alignment is interesting. The fixed-context diagnostics and controlled ablations provide useful evidence, and the method is simple and empirically evaluated across multiple teacher–student settings.

Critical weaknesses

The paper prioritizes support-aligned corrections, but its explanation of why support-misaligned corrections are less teachable relies mainly on empirical observations and lacks a clear mechanistic explanation.

Questions for authors

Why are support-misaligned corrections less teachable? Could they also help OPD, and under what conditions would they be beneficial?

4 comments

Thank you for your insightful comment and for highlighting the need for a clearer mechanistic explanation. We have revised the PDF to strengthen this discussion and added a figure (Appendix D, Figure 7) illustrating the state-dependent nature of token teachability.

W&Q: Why are support-misaligned corrections less teachable?

The key mechanism arises from the optimization geometry of reverse-KL OPD and its finite-sampling implementation. For the reverse-KL objective in Eq. (6), the gradient with respect to a student logit zvz_v is

$ \frac{\partial D_{\mathrm{KL}}(p_\theta|p_T)}{\partial z_v} =p_\theta(v)\left[\log\frac{p_\theta(v)}{p_T(v)}-D_{\mathrm{KL}}(p_\theta|p_T)\right]. $

When a teacher-preferred token v⋆v^\star has very low student probability, its direct logit-gradient contribution is suppressed by $ p_\theta (v^\star) $, despite potentially large disagreement. In sampled-token OPD, this alternative is also rarely sampled, further limiting how frequently its teacher–student discrepancy directly contributes to stochastic updates. In contrast, support-aligned corrections involve alternatives already assigned appreciable student probability, making their feedback more readily accessible under finite sampling.

Can support-misaligned corrections still help? Yes. Updates on other sampled tokens can indirectly redistribute probability toward v⋆v^\star. Moreover, teachability is state-dependent: as v⋆v^\star gains probability and enters the student's top-(K), both teacher support coverage and its direct sampling frequency can increase. Our new schematic illustrates this possible transition under a fixed teacher distribution. Greater exploration, continued training, or direct supervision, such as full-distribution distillation may therefore make initially off-support corrections more useful.

Experiment results: Appendix D, Table 11 provides a matched-ratio Q3 intervention. High-DID^I positions achieved a positive bootstrap gain (+0.006±0.003)(+0.006\pm0.003), but a slightly negative matched gain (−0.003)(-0.003). High-DLD^L positions yielded larger gains (+0.015±0.010+0.015\pm0.010 and +0.010+0.010, respectively). These observations are consistent with weaker, rather than absent, learning from support-misaligned corrections. We emphasize that the gradient analysis provides a local optimization mechanism, while the schematic is illustrative rather than an empirical training trajectory. Together, they clarify why support alignment can matter without implying that off-support corrections are intrinsically unlearnable.

Dear Reviewer,

Thank you so much for your thoughtful initial reviews!

With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.

Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!

Warm regards

Thank you for the clarification. The reverse-KL gradient analysis and finite-sampling explanation substantially address my concern about the mechanism. I will maintain my score.

Thank you again for your careful follow-up and constructive discussion. We are glad that our responses and analysis have addressed your main concerns. The revised manuscript now includes the reverse-KL gradient analysis and finite-sampling explanation.

Reviewer_SCGuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies which teacher signals are useful for on-policy distillation. It introduces a fixed-context diagnostic and proposes TA-OPD, which selects supervision positions using teacher–student disagreement weighted by teacher probability mass on the student’s top-K candidates. Experiments show that sparse support-aware supervision can match or improve full-token OPD, with benefits depending on the model pair and supervision ratio.

Critical strengths
  1. Studies the usefulness beyond raw disagreement magnitude, with a simple position-selection rule.
  2. Uses fixed contexts, held-out prompts, selector-neutral validation, and clustered uncertainty estimates.
  3. Evaluates four teacher–student pairs and six benchmarks, transparently reporting cases where KL-only or other selectors remain stronger.
Critical weaknesses
  1. The downstream advantage over KL-only is not broadly established. Tables 3 and 4 report this comparison for one model pair at a 10% supervision ratio. The macro-average gain of 0.89 is driven by IFEval (+5.55), while the reasoning average decreases by 0.21. Comparisons on additional model pairs would clarify the generality of this benefit.

  2. Practical guidance on the supervision ratio is limited. Table 2 shows that TA-OPD outperforms TIP at 5% supervision but falls behind at 30% in the 8B-to-4B setting. The paper acknowledges the non-monotonic behavior, but gives limited guidance on choosing the ratio before downstream evaluation.

Questions for authors
  1. Does TA-OPD retain its downstream advantage over KL-only on other main model pairs?
  2. What explains the ratio-dependent ranking in Table 2, and how should the supervision ratio be selected using training or validation information?
7 comments

Thank you for the insightful review and for recognizing our strength.

Your comments identify two important questions: whether the downstream benefit over KL-only extends beyond the original comparison, and how the supervision ratio can be chosen without using the final test scores. We added two supplemental KL-only training runs and eight fresh TA-OPD ratio configurations, together with a validation-based selection audit. The latter evaluates all four pre-specified ratios for two teacher–student pairs, using 8000 validation responses on 500 shared held-out questions. We report the complete candidate results, paired prompt-bootstrap intervals, selection frequencies, realized supervision budgets, and fresh downstream evaluations below.

The revised manuscript has been submitted.

W1:

We agree that the original 14B→4B result does not establish general downstream superiority over KL-only. Its +0.89 macro-average difference is largely attributable to IFEval (+5.55), while the reasoning average decreases by 0.21. We have revised the manuscript to distinguish support-dependent heterogeneity, which the controlled diagnostics investigate, from the stronger claim that multiplying disagreement by support coverage must improve downstream performance across model pairs and tasks. We do not make the latter claim.

This distinction also closes the loop with the selector-neutral analysis: KL-only is a stronger ranker of fixed-context KL reduction in the reported confirmatory settings, whereas support conditioning is a complementary supervision-allocation prior. The original downstream ablation illustrates that the two rankings can differ; it is not evidence of universal dominance or, by itself, a demonstrated training-stability mechanism.

Q1: Does TA-OPD retain its advantage on additional main model pairs?

We expanded the KL-only evaluation to Qwen3-4B→1.7B and Qwen3-8B-GRPO→4B, using a 10% supervision ratio and exactly 30 updates. Table R1 reports all six benchmarks rather than only their macro-average.

Table R1. Supplemental KL-only results. Entries are mean ± sample SD across two decoding seeds of one trained checkpoint. All scores are percentages. The aggregate SDs are computed across the corresponding decoding-level aggregates, not by averaging benchmark SDs.

MetricQwen3-4B→1.7BQwen3-8B-GRPO→4B
AIME2416.66 ± 4.7221.66 ± 2.35
AIME255.00 ± 2.3620.00 ± 0.00
GPQA-Diamond29.55 ± 3.9241.41 ± 7.86
HumanEval59.46 ± 0.4379.27 ± 0.86
IFEval67.66 ± 2.8878.19 ± 4.71
MATH-50074.20 ± 0.8581.30 ± 0.14
Reasoning average31.35 ± 1.3641.09 ± 1.41
Six-benchmark macro-average42.09 ± 0.5053.64 ± 0.01

These runs broaden the baseline coverage, but they are not paired replications of the original TA-OPD results. We therefore do not combine these new KL-only measurements with historical TA-OPD averages to claim an additional controlled improvement.

For completeness, the fresh 30-update 8B-GRPO→4B TA-OPD checkpoint selected at 10% in the validation audit below scores 52.66 ± 0.32, while the new KL-only checkpoint scores 53.64 ± 0.01. These separately reported runs are not an established matched-seed comparison, and they do not support a TA-OPD advantage in this setting. Thus, the answer supported by the current evidence is not a uniform advantage over KL-only. We retain the positive original 14B→4B result with its task-level qualification, report the additional baseline results transparently, and restrict the claim to a competitive support-aware allocation rule rather than a universally superior selector.

W2:

The reversal highlighted in the review is not primarily a collapse of TA-OPD as the ratio increases. In the original 8B-GRPO→4B experiment, moving from 5% to 30% changes TA-OPD from 57.35 to 57.28 (−0.07), whereas TIP increases from 56.20 to 58.46 (+2.26). TA-OPD again has the higher point estimate at 50%. Table R2 shows the full pattern for both teacher settings, including ratios unfavorable to TA-OPD.

Table R2. Original downstream ratio sweep, reproduced from Table 2. These are historical main-experiment macro-averages, not scores assigned to the new validation-audit checkpoints. Differences are descriptive point estimates, not significance claims.

Teacher→Qwen3-4BRatioEntropyTIPTA-OPDTA−TIP
Qwen3-8B-GRPO5%56.9556.2057.35+1.15
Qwen3-8B-GRPO10%56.6956.8156.87+0.06
Qwen3-8B-GRPO30%57.5658.4657.28−1.18
Qwen3-8B-GRPO50%53.9055.7457.90+2.16
Qwen3-14B5%53.5653.8353.58−0.25
Qwen3-14B10%53.3953.6254.65+1.03
Qwen3-14B30%53.6253.9352.81−1.12
Qwen3-14B50%52.7652.1252.16+0.04

Mechanistically, increasing the ratio changes both which positions receive supervision and their relative weight in the update. Equation (7) averages the loss over selected positions. For a fixed model state and batch, expanding a selector's top-ranked set from n1n_1 to n2n_2 positions gives $ \mathcal L_{n_2} =\frac{n_1}{n_2}\mathcal L_{n_1} +\frac{n_2-n_1}{n_2}\mathcal L_{\mathrm{added}}, $

where Ladded\mathcal L_{\mathrm{added}} is the average loss on the additional positions. Thus, a larger ratio does not preserve the original update and merely append extra supervision: it reweights it. Because TA-OPD and TIP rank positions differently, their added supervision differs as well. Their on-policy trajectories also evolve separately during training.

This explains why neither monotonic performance nor a fixed cross-selector ranking is guaranteed. It does not identify the precise cause of TIP's numerical improvement, and we do not attribute that improvement to unmeasured gradient noise, interference, or instability. The practical implication is to treat the ratio as a validation-selected hyperparameter, rather than infer a universally optimal value from the test-set sweep.

Q2: How can the ratio be selected using validation information?

Selection protocol

We conducted a new validation-based audit, rather than retrospectively redefining the original ratio choice. For each of Qwen3-8B-GRPO→4B and Qwen3-14B→4B, we trained the four pre-specified ratios {5%,10%,30%,50%}\{5\%,10\%,30\%,50\%\} for 30 updates, using training seed 1234. Every checkpoint was evaluated on the same 500 held-out DAPO questions, screened for lexical overlap with the training data and all six test sets, with decoding seeds 42 and 43.

The selection rule maximizes the integer validation correct count across the two decodings, with the smaller ratio breaking exact ties. The ratio was fixed before fresh six-benchmark inference. No test score enters this selection rule. We use task correctness rather than the selector's masked training loss or fixed-context KL reduction, consistent with the paper's distinction between local diagnostic utility and downstream task utility.

Complete candidate results

Table R3. All validation candidates. Each decoding seed evaluates 500 questions. Correct counts pool 1,000 responses per configuration; these are two responses per question, not 1,000 independent questions. Bold marks the ratio selected by the fixed rule.

Teacher→Qwen3-4BRatioSeed 42 accuracySeed 43 accuracyCorrect / 1,000 responsesPooled accuracy
Qwen3-8B-GRPO5%42.0044.4043243.20
Qwen3-8B-GRPO10%45.0047.4046246.20
Qwen3-8B-GRPO30%44.0045.6044844.80
Qwen3-8B-GRPO50%41.0043.6042342.30
Qwen3-14B5%48.8046.4047647.60
Qwen3-14B10%46.2044.0045145.10
Qwen3-14B30%46.8047.8047347.30
Qwen3-14B50%48.2046.8047547.50

Selection uncertainty

To distinguish a clear preference from a near-tie, we additionally report paired prompt-bootstrap differences from 10% and bootstrap winner frequencies. Each prompt is resampled with both decoding outputs retained together.

Table R4. Validation differences and selection stability. Differences and intervals are in percentage points. Winner frequency is an empirical resampling frequency, not a posterior probability of being globally optimal. Intervals condition on the trained checkpoints and do not capture training-seed variation.

Teacher→Qwen3-4BRatioDifference from 10%95% paired prompt intervalWinner frequency
Qwen3-8B-GRPO5%−3.00[−5.70, −0.20]0.9%
Qwen3-8B-GRPO10%0.00[0.00, 0.00]82.5%
Qwen3-8B-GRPO30%−1.40[−4.30, +1.60]16.4%
Qwen3-8B-GRPO50%−3.90[−7.10, −0.80]0.2%
Qwen3-14B5%+2.50[−0.40, +5.40]38.8%
Qwen3-14B10%0.00[0.00, 0.00]0.7%
Qwen3-14B30%+2.20[−0.80, +5.10]28.1%
Qwen3-14B50%+2.40[−0.60, +5.30]32.4%

For 8B-GRPO→4B, 10% has the highest validation accuracy under both decoding seeds and wins 82.5% of prompt resamples. Its pooled advantage is 3.00 points over 5% and 3.90 points over 50%; 30% remains a competitive alternative given the interval crossing zero.

For 14B→4B, the fixed rule selects 5%, but the validation profile is nearly flat at 5%, 30%, and 50% (47.60%, 47.30%, and 47.50%). The difference between 5% and 50% is only one correct response out of 1,000 decodings, and the 5% winner frequency is 38.8%. We therefore report a near-tied validation region, not a uniquely optimal ratio. This is useful guidance in itself: the validation procedure exposes when a sharp optimum is not supported, rather than presenting a test-tuned ratio as universal.

Fresh downstream evaluation after ratio selection

Only the validation-selected checkpoints were subsequently evaluated on the six test benchmarks.

Table R5. Fresh downstream results of validation-selected TA-OPD checkpoints. Downstream entries are mean ± sample SD across two decoding seeds. Validation accuracy is selection-conditioned; it is not an unbiased estimate of the selected model's generalization accuracy.

Teacher→Qwen3-4BValidation-selected ratioValidation accuracyFresh six-benchmark macro-average
Qwen3-8B-GRPO10%46.2052.66 ± 0.32
Qwen3-14B5%47.6054.10 ± 1.37

This establishes an executable ratio-selection protocol that does not use the final test scores, followed by fresh downstream evaluation. It does not establish the downstream-optimal ratio, an improvement over a fresh 10% checkpoint for 14B→4B, or superiority over independently tuned TIP/KL-only. A math validation set provides task-aligned tuning guidance for the present training setting, not a guarantee of maximizing every capability or the six-benchmark macro-average.

Realized supervision-budget audit

We also checked that the configured ratios correspond to the actual training masks across all eight configurations, rather than relying on nominal hyperparameters alone.

Table R6. Realized supervision over 30 updates. Actual percentages are rounded from the recorded ratios. Every configuration has zero additional selected positions from the minimum-one-token safeguard.

Teacher→Qwen3-4BConfigured ratioValid positionsSelected positionsActual selected percentage
Qwen3-8B-GRPO5%205,43110,2855.007%
Qwen3-8B-GRPO10%235,53523,56910.007%
Qwen3-8B-GRPO30%211,51663,46830.006%
Qwen3-8B-GRPO50%206,189103,10350.004%
Qwen3-14B5%236,96811,8645.007%
Qwen3-14B10%226,70622,68410.006%
Qwen3-14B30%223,96567,20130.005%
Qwen3-14B50%223,088111,55150.003%

The audit covers 1769398 valid positions and 413725 selected positions in total. The realized ratios closely track the configured values, with small deviations consistent with integer rounding. The different position totals reflect separate on-policy rollouts; this is an update-budget-controlled audit, not a claim of identical generated-token budgets or proportional compute savings.

In summary, the revised manuscript separates three questions that should not be conflated: whether support alignment reveals heterogeneity in local learning, whether a particular selector improves downstream performance, and how its supervision ratio is chosen. The additional experiments substantially expand the reported evidence and provide concrete validation-based guidance, while making the limits of the cross-pair performance claim explicit.

Reviewer_7ptLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper starts from the observation that token importance in OPD is not uniform: positions where the teacher’s probability mass lies within the student’s top-K support show larger fixed-context KL reduction after training than positions where it does not. Based on this, the paper defines token teachability and proposes a method that filters a subset of tokens by this score for OPD training. The method is evaluated on math, coding, factual QA, and instruction following.

Critical strengths
  1. Easy to read and follow. The message comes across clearly.
  2. The formulation of teachability is reasonable and intuitive.
Critical weaknesses
  1. D~\tilde{D} and C~\tilde{C} are first used in Eq. (5) but defined only later in Eq. (11), which makes Section 3 hard to follow on a first read.
  2. The gains in Table 1 look marginal. In two of the four settings (8B-GRPO->4B and 14B->4B), TA-OPD is within 0.1 points of the best baseline, which is smaller than the reported standard deviations.
Questions for authors
  1. Could the authors compare their method with SelecTKD (Huang et al., 2026)? In my understanding, both prioritize tokens by how much the student’s and teacher’s supports overlap. What is the contribution of this paper beyond SelecTKD, and is there a reason it was not included as a baseline? A direct comparison showing an advantage would make the results more convincing.
  2. TSD-KD (Kim and Baek, ICLR 2026) also restricts teacher feedback to the student’s own top-k candidates, which seems related to the support-based view of this paper. Could the authors discuss the relation to TSD-KD?

references

  • SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMs. Huang et al., CVPR 2026.
  • Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation. Kim and Baek, ICLR 2026.
6 comments

Thank you for your insightful review and for recognizing the clarity of the paper and the intuition behind teachability.

Your comments identify two points that need a sharper answer: what the paper adds beyond existing selective distillation, and how strongly the empirical results support the method. We have clarified the definitions and performance claims, expanded the related-work discussion, and added a direct comparison with the published Greedy Top-k variant of SelecTKD on the two teacher–student settings highlighted in W2. Below we report the complete six-benchmark results, every decoding-seed macro difference, and the supervision-weight audit, while separating the new comparison from the original Table 1.

The revised manuscript has been submitted.

W1:

We agree. We have moved the batch-wise robust normalization definition to Section 3.2, immediately before the disagreement decomposition (Eq. (5) in the submission). Both D~t=Norm⁡B(Dt)\widetilde D_t=\operatorname{Norm}_B(D_t) and C~t=Norm⁡B(Ct)\widetilde C_t=\operatorname{Norm}_B(C_t) are now defined when first used, including the 5th/95th-percentile clipping and numerical stabilizer. Section 4.2 refers back to this definition. This is a presentation correction; the score and implementation are unchanged.

W2:

We agree that the +0.06 and +0.01 differences should not be interpreted as reliable superiority over the strongest baseline. We now describe these two settings as numerically comparable, rather than statistically better. For clarity, Table R1 separates the four original settings instead of treating every positive difference as an equally strong result.

Table R1. Original Table 1: comparison with the strongest non-TA distillation baseline. Values are reported macro-averages; differences are descriptive, not significance claims. The baseline pool is Full OPD, Entropy, and TIP, excluding the untrained Base and TA-OPD+Entropy.

Teacher → StudentStrongest non-TA baselineBaseline Avg.TA-OPD Avg.Difference
Qwen3-4B → Qwen3-1.7BTIP43.0544.89+1.84
Qwen3-8B-GRPO → Qwen3-4BTIP56.8156.87+0.06
Qwen3-14B → Qwen3-4BFull OPD54.6454.65+0.01
DeepSeek-R1-Distill-Qwen-14B → Qwen2.5-3BFull OPD28.7630.62+1.86

For the last setting, the difference from the untrained Base is +0.64 (30.62 versus 29.98), distinct from the +1.86 improvement over Full OPD. The original table therefore supports setting-dependent numerical improvements and competitive sparse supervision, not a universal performance advantage. The selected-position ratio is a supervision allocation parameter, not a proportional reduction in compute.

To address the missing related-method comparison directly, we additionally evaluated SelecTKD-G on the same two teacher–student pairs singled out in this comment. The new comparison yields +1.33 and +1.20 macro-average differences in favor of TA-OPD, with all individual benchmarks and decoding-seed differences reported under Q1. These are separate 30-update results; we do not combine their baselines with the historical TA-OPD scores in Table R1.

Q1: Contribution beyond SelecTKD and a direct comparison

1. Relationship and scientific contribution

SelecTKD is directly relevant, and we agree that citing it without an empirical comparison left the distinction insufficiently established. Our contribution is not the general idea of selective distillation or teacher–student compatibility.

Using our notation, the published Greedy verifier checks whether the student's most likely token belongs to the teacher's top-kk:

$ w_t^{\mathrm{G}}=\beta+(1-\beta) \mathbf 1!\left[ \arg\max_v p_\theta(v\mid c_t)\in S_t^T(k) \right]. $

In contrast, TA-OPD measures teacher probability mass on the student's candidate set and combines it with local disagreement:

$ C_t=\sum_{v\in S_t^S(K)}p_T(v\mid c_t), \qquad s_t^{\mathrm{teach}}=\widetilde D_t\widetilde C_t. $

Both mechanisms allocate supervision across positions, but they are not the same overlap test. A position can pass the Greedy verifier while differing substantially from another accepted position in disagreement and coverage. TA-OPD ranks those positions rather than assigning all accepted positions the same multiplier. Here, SelecTKD-G's k=5k=5 is the teacher's verification set size; TA-OPD's KK defines the student's local support.

The associated scientific question is whether disagreement of similar magnitude can have different learning utility depending on the student's predictive support. Our fixed-context diagnostic measures later KL reduction on the same prefixes, separating this measurement from changes in the evaluated contexts. The development-bank analysis controls for student entropy, teacher entropy, and token position; its decomposition associates the support-aligned component with a larger standardized coefficient across support sizes:

Table R2. Existing diagnostic evidence from Appendix F, Table 14—not additional rebuttal training. Intervals are the reported prompt-cluster bootstrap intervals for the coefficient gap.

Support size KKCoefficient of DLD^LCoefficient of DID^IGap95% interval
80.0860.045+0.041[0.039, 0.044]
160.0870.043+0.044[0.041, 0.047]
320.0870.043+0.044[0.041, 0.047]

The matched-ratio Q3 intervention complements this association analysis, while the selector-neutral audit in Appendix B establishes an important boundary: raw KL can remain the stronger fixed-context gain ranker. Thus, the contribution is a characterization of conditional learning heterogeneity and its implications for supervision allocation, not a claim that support weighting universally predicts improvement better than KL. The new comparison below tests the resulting allocation rule against a published compatibility-based alternative.

2. Direct comparison

We independently implemented the published Greedy Top-k verifier (SelecTKD-G; k=5,β=0.01k=5,\beta=0.01), following Section 4.1.2, Eq. (9), and Algorithm 1, and adapted it to the same sampled-token OPD trainer. TA-OPD uses a fixed 10% ratio in both settings. The methods share initialization, training data, optimizer, training seed 1234, and a 30-update horizon. Evaluation covers the six full benchmarks with decoding seeds 42 and 43.

We preserve SelecTKD-G's original accepted/rejected weights and full-position normalization; we neither truncate it to 10% nor normalize by the weight sum. Accordingly, this is a common-training-protocol comparison, not a matched-supervision-count ablation or a reproduction of SelecTKD's original end-to-end training recipe.

Table R3. Complete new downstream comparison. Scores are percentages, reported as mean ± sample SD across two decoding seeds for one trained checkpoint per configuration. The higher mean within each teacher setting is bold. SDs do not estimate training-seed uncertainty.

Teacher → Qwen3-4BMethodAvg.AIME24AIME25GPQA-D.HumanEvalIFEvalMATH-500
Qwen3-8B-GRPOTA-OPD (10%)52.66 ± 0.3220.00 ± 4.7115.00 ± 2.3643.18 ± 2.5078.05 ± 1.7378.56 ± 5.4981.20 ± 0.85
Qwen3-8B-GRPOSelecTKD-G51.33 ± 0.0616.67 ± 0.0011.66 ± 2.3544.19 ± 5.3678.35 ± 0.4375.42 ± 7.0681.70 ± 0.14
Qwen3-14BTA-OPD (10%)53.59 ± 1.8418.33 ± 7.0716.66 ± 4.7243.94 ± 2.8679.88 ± 1.7380.13 ± 0.4082.60 ± 0.00
Qwen3-14BSelecTKD-G52.39 ± 0.7616.66 ± 9.4318.33 ± 7.0739.14 ± 0.3580.80 ± 1.2977.82 ± 4.4481.60 ± 0.85

TA-OPD has the higher macro-average in both settings: +1.33 points with the GRPO-trained 8B teacher and +1.20 with the 14B teacher. The benchmark-level pattern is not uniformly positive: TA-OPD leads on three of six benchmarks in the first setting and four of six in the second, while SelecTKD-G is stronger on HumanEval in both. We report those trade-offs rather than only the aggregate.

3. Are the differences confined to IFEval or one decoding seed?

Table R4. Benchmark-level and aggregate differences, TA-OPD minus SelecTKD-G, in percentage points. Benchmark differences and the five-benchmark average use the rounded means in Table R3; the full macro differences use the supplied unrounded decoding aggregates. The exclusion analysis is descriptive and does not replace the original six-benchmark metric.

MetricQwen3-8B-GRPO → 4BQwen3-14B → 4B
AIME24+3.33+1.67
AIME25+3.34−1.67
GPQA-Diamond−1.01+4.80
HumanEval−0.30−0.92
IFEval+3.14+2.31
MATH-500−0.50+1.00
Six-benchmark macro-average+1.33+1.20
Five-benchmark average excluding IFEval≈ +0.97≈ +0.98

The positive aggregate differences are therefore not solely due to IFEval. In the 8B-GRPO setting, the two AIME benchmarks also contribute positively; with the 14B teacher, GPQA-Diamond, AIME24, and MATH-500 contribute positively.

Table R5. All decoding-seed macro comparisons. Every row uses training seed 1234. Differences are computed before rounding. These are two decodings of each trained model, not independent training replications.

Teacher → Qwen3-4BDecoding seedTA-OPD Avg.SelecTKD-G Avg.Difference
Qwen3-8B-GRPO4252.8951.38+1.52
Qwen3-8B-GRPO4352.4451.29+1.15
Qwen3-14B4252.2951.85+0.44
Qwen3-14B4354.8952.93+1.96

All four differences are positive, so the reported mean improvements are not produced by one decoding seed reversing the aggregate direction. The different magnitudes nevertheless matter, particularly in the 14B setting; we do not infer statistical significance or training-seed robustness from two decoding seeds.

4. What does the implementation audit reveal?

Table R6. Supervision allocation in the new comparison. SelecTKD-G acceptance rates are measured; down-weighted fractions are their complements. TA-OPD's 10% is the configured selection ratio.

Teacher → Qwen3-4BSelecTKD-G full-weight acceptanceSelecTKD-G down-weighted positionsRejected-position weightTA-OPD configured selected ratio
Qwen3-8B-GRPO99.89%0.11%0.0110%
Qwen3-14B99.84%0.16%0.0110%

The Greedy verifier gives full weight to nearly all positions in these settings; rejected positions still retain nonzero weight. This provides a concrete distinction between accepting a plausible student proposal and ranking positions for a restricted supervision budget. An accepted proposal need not indicate that its position has the same learning utility as other accepted positions. TA-OPD continues to rank positions by support-conditioned disagreement when the binary Greedy verifier offers little differentiation.

This audit describes how the rules operate; it does not establish that acceptance saturation caused the score differences or isolate support conditioning from every other aspect of loss weighting. The new result is therefore specific: TA-OPD improves the observed macro-average over our Greedy-verifier adaptation in both tested 30-update settings. It does not establish superiority over SelecTKD's Non-greedy Spec-k variant, which was not evaluated, or over longer training and multiple training seeds. We explicitly identify the comparator as SelecTKD-G throughout.

Q2: Relationship to TSD-KD

We agree that TSD-KD is closely related and have added an explicit discussion. Its indirect component uses teacher preference ranking over the student's top-k candidates within a selected early reasoning segment. Its direct component uses an uncertainty-gap gate on JSD-based distillation, together with selective entropy regularization. Thus, TSD-KD also selects positions; the distinction is not simply “vocabulary selection versus position selection.”

In TA-OPD, the student top-K set is used to compute the coverage statistic for allocating supervision. It does not restrict the OPD loss to that candidate set or introduce a teacher-ranking objective. Selected positions retain the standard sampled-token reverse-KL OPD objective, with allocation determined by disagreement and support coverage rather than an entropy-gap gate. Our fixed-context analysis additionally asks when support alignment is informative about local learning and reports where it fails to improve the ranking over raw KL.

The shared motivation is to account for the student's predictive state; the distinction is the diagnostic question, allocation signal, and training objective. We discuss this relationship without claiming empirical superiority over TSD-KD, which the new experiments do not test.

Comments on the paper

1 comment

We thank the reviewers for the constructive discussion. To connect the individual responses to the paper’s central research question, we summarize the contribution, supporting evidence, and rebuttal updates below.

1. Core contribution: understanding which disagreement produces useful learning.
The paper studies how the student’s current predictive support relates to the usefulness of teacher supervision. Our central finding is support-dependent learning heterogeneity within high-disagreement regions: disagreement magnitude can conceal differences in learning response associated with where teacher probability mass lies relative to the student’s local support. Token teachability provides a concrete formulation of this distinction.

2. Evidence: measuring learning responses and testing their boundaries.
The fixed-context diagnostic measures KL reduction on unchanged prefixes. Controlled analyses and matched-ratio Q3 interventions examine learning responses under different supervision choices (§3; Appendices D and F). The selector-neutral audit further tests how these findings translate into selection and identifies settings where KL-only remains the stronger local-gain ranker (Appendix B). These analyses establish the empirical scope of the finding and its limitations.

3. Practical significance: turning the finding into supervision allocation.
TA-OPD instantiates this insight as a simple position-selection rule. Across four teacher–student pairs and six benchmarks, the experiments show that supervising 5–10% of positions can match or improve full-token OPD in several settings. This demonstrates a practical use of the diagnostic insight. The evidence for additional downstream gains over KL-only remains limited, and we distinguish that algorithmic question from the contribution to understanding OPD.

4. Rebuttal updates addressing the reviewers’ questions.

  • Mechanism: We clarified the local reverse-KL explanation and the state-dependent nature of teachability, with an explicitly illustrative support-coverage schematic.
  • Related methods: We clarified the relationships to SelecTKD and TSD-KD, added direct SelecTKD-G comparisons on two model pairs, and moved the normalization definitions before their first use.
  • Ratio selection: We evaluated all four pre-specified ratios for two model pairs on held-out validation prompts, fixed the choices before downstream evaluation, and reported selection uncertainty.

We hope the final assessment considers the diagnostic findings, their controlled evaluation, and the practical results together. The contribution is an empirically grounded account of how student support relates to learning from disagreement, with TA-OPD providing a testable application and explicitly reported boundaries.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.