acceptodds
Desk rejected at ICLR 2027

ResOPD: Tail Residualization for Sparse On-Policy Distillation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy distillation (OPD) trains a student model on self-generated trajectories, but transmitting dense teacher distributions across long reasoning traces creates prohibitive communication and memory bottlenecks. Practical systems therefore rely on sparse teacher interfaces, typically transmitting either the sampled-token score or a small Top- distribution. However, this sparse setting faces a fundamental dilemma: sampled-token estimators are unbiased but suffer from severe gradient variance, whereas directly optimizing Top- objectives introduces systematic bias. To improve this trade-off, we propose ResOPD (On-Policy Distillation with Tail Residualization), which achieves unbiased full-vocabulary reverse KL gradient estimation with substantial variance reduction under the same sparse payload. ResOPD aggregates the unobserved vocabulary into an observable coarse tail event, computes its exact aggregate gradient, and samples only the fine-grained within-tail residual, which requires no additional teacher queries or forward passes. Extensive experiments demonstrate that ResOPD substantially reduces gradient variance, stabilizes online training dynamics, and improves downstream performance across the evaluated settings. These results establish ResOPD as an efficient, plug-and-play variance reduction primitive for sparse on-policy distillation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, On-Policy Distillation, Policy gradient

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_NF5jOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper considers sparse OPD settings where only top-kk tokens are available on the interface, which is very often employed in practice. They theoretically demonstrate the variance decomposition of it, propose an unbiased estimator (ResOPD), and show the empirical performance improvements across multiple domains.

Critical strengths
  • This paper provides bias and variance analysis of sparse OPD where the methods (HETS, SC-TA) proposed in prior works are proven to be special cases of them. ResOPD improves from HETS by reducing the variance from which tail token is sampled. This is a clear contribution.
  • This paper shows the empirical validity, including cross-domain evaluation results, of ResOPD using Qwen3, Qwen3.5 models with sufficient size on math, science, and code domains, and also reports token-distribution dynamics at training time.
Critical weaknesses
  • It would be better to show the results on vOPD in Qwen3 models (Table.4) since vOPD is relatively closer to ResOPD in Table.3. This is also related to the question below.
Questions for authors
  • The variance of vOPD is larger than HETS (Table.1) but evaluation results of vOPD is better than HETS (Table.3). Could the authors provide the thoughts on this?
6 comments
AuthorsOpenReview ↗

Response to Reviewer NF5j (1/1)

W1 & Q1: It would be better to show the results on vOPD in Qwen3 models (Table.4) since vOPD is relatively closer to ResOPD in Table.3. The variance of vOPD is larger than HETS (Table.1) but evaluation results of vOPD is better than HETS (Table.3). Could the authors provide the thoughts on this?

We thank the reviewer for this suggestion. We added Teacher-Top-4 vOPD to the Qwen3 cross-domain experiment. The evaluation results are shown below:

modelGPQA‑DSuperGPQALCBv5LCBv6Avg.ScoreAUC
Student45.2532.4621.7624.3830.96--
vOPD39.8931.3841.1240.5738.2434.44
ResOPD47.1237.5448.9043.0544.1542.65

Our experiment shows vOPD also improved substantially over the student base model on code (LiveCodeBench), but experienced severe performance degradation on GPQA‑D and SuperGPQA (both degraded notably in the later stages of training, even falling below the base student), leading to lower Avg.Score and AUC compared with HETS and ResOPD. This provides complementary evidence that estimator variance and downstream ranking are not universally aligned: vOPD performs better than HETS in the math setting, whereas the ranking reverses under this Qwen3 cross-domain recipe. In theory, vOPD remains an unbiased estimator of the target reverse-KL gradient by subtracting the local KL divergence (a scalar) between the student and teacher over the teacher's Top-kk tokens. However, we suspect that this scalar subtraction does not fully exploit the fine-grained distribution geometry across these Top-kk tokens, which may ultimately hinder fine-grained student-teacher matching in certain scenarios. We have added these results to Table 4 in our revised manuscript.

I appreciate the authors for clarification and additional experiments. I could confirm that ResOPD is better than vOPD at Qwen3 evaluation. However, the claim that scalar KL subtraction fails to capture fine-grained token-level information remains speculative. Could the authors provide some empirical evidence for this claim, such as analysis of token-level teacher–student distribution alignment during training, or detailed explanations? In particular, what makes ResOPD more effective than vOPD at capturing this distribution structure?

We thank the reviewer for this thoughtful follow-up. We agree that our earlier statement regarding scalar KL subtraction was overly broad: vOPD retains token-specific teacher–student feedback and effectively learns the teacher's distribution. The distinction we intend to convey is that scalar centering does not explicitly average the revealed token-specific head gradients or regulate head–tail mass allocation, whereas ResOPD explicitly integrates the revealed head gradients alongside a head–tail event control.


1. Gradient-Level Mechanism: Explicit Head Integration and Head–Tail Event Control

At a fixed prefix, let SS be the teacher Top-kk set, P=p(S)P = p(S), Q=q(S)Q = q(S), ri=log⁡(pi/qi)r_i = \log(p_i/q_i), and si=∇log⁡pis_i = \nabla\log p_i. Teacher-Top-kk vOPD computes:

gv(a)=(ra−b)sa,b=KL(p(⋅∣S) ∣∣q(⋅∣S)),g_v(a) = (r_a - b)s_a, \qquad b = \mathrm{KL}\bigl(p(\cdot\mid S)\ || q(\cdot\mid S)\bigr),

with a detached baseline bb. While subtracting bb centers the reward and reduces gradient variance, the single-sample update still depends on the sampled token's specific gradient direction sas_a and does not explicitly control the probability mass split between the head SS and the unseen tail.

ResOPD instead computes the exact conditional head gradient by averaging across all revealed Top-kk tokens:

μH=∑i∈SpiPrisi.\mu_H = \sum_{i\in S}\frac{p_i}{P}r_i s_i.

Combined with head–tail event control, any sample a∈Sa \in S produces the same deterministic estimate PμH+(1−P)μBP\mu_H + (1-P)\mu_B, where μB=log⁡[(1−P)/(1−Q)]E[sa∣a∉S]\mu_B = \log[(1-P)/(1-Q)]\mathbb{E}[s_a \mid a \notin S] is the tail-bucket gradient. Both estimators share the same expected local reverse-KL gradient; the distinction lies in finite-sample estimation, where ResOPD removes intra-head sampling noise and directly constrains the head–tail probability mass balance.


2. Empirical Evidence: Distribution Alignment Across Training and Shared Prefixes

A. Head–Tail Calibration Along Training Trajectories

To track how probability mass allocation evolves during Qwen3 training, we measured the relative tail-mass mismatch ℓ=log⁡1−P1−Q\ell = \log\frac{1-P}{1-Q} (where ℓ>0\ell > 0 indicates P<QP < Q, i.e., over-allocating probability mass to the unseen tail) across training intervals:

Training-Trajectory DiagnosticResOPDvOPD
Relative tail-mass mismatch (Steps 1–50)0.2950.302
Relative tail-mass mismatch (Steps 51–100)0.1540.397
Teacher-Top-4 hit rate95.33%93.57%

While both methods exhibit comparable tail-mass alignment during the first half of training (0.295 vs. 0.302 in Steps 1–50), their trajectories diverge in later updates: ResOPD continues to tighten its head–tail calibration down to 0.154 in Steps 51–100, whereas vOPD exhibits a larger tail-mass mismatch (0.397). Throughout training, ResOPD also yields a higher Teacher-Top-4 hit rate (95.33% vs. 93.57%).

B. Controlled Evaluation on Shared Prefixes

To isolate distribution alignment on identical contexts, we evaluated the final checkpoints on 4,096 shared prefixes sampled from initial-student trajectories across four benchmarks (GPQA, SuperGPQA, LCBv5 and LCBv6).

We report three overall diagnostics: (1) the absolute head–tail mass gap ∣P−Q∣|P - Q|, (2) the conditional-head reverse KL KL(p(⋅∣S)∥∣q(⋅∣S))\mathrm{KL}(p(\cdot\mid S)\||q(\cdot\mid S)) within the Top-4 tokens, and (3) the overall 5-way reverse KL D5D_5 (over the 4 head tokens plus 1 combined tail bucket), which decomposes as $ D_5 = P\mathrm{KL}\bigl(p(\cdot\mid S)||q(\cdot\mid S)\bigr) + \mathrm{KL}\bigl(\mathrm{Bern}(P)||\mathrm{Bern}(Q)\bigr). $

Shared-Prefix DiagnosticResOPDvOPD
Absolute head–tail mass gap (pp)6.119.15
Conditional-head reverse KL0.86480.8558
5-way reverse KL D5D_51.1041.267

These diagnostics reveal two distinct patterns consistent with our theoretical decomposition. (1) Tighter Head–Tail Mass Calibration: Consistent with its explicit head–tail event control, ResOPD reduces the overall absolute head–tail mass gap ∣P−Q∣|P - Q| from 9.15 to 6.11 percentage points compared to vOPD. (2) Comparable Within-Head Alignment vs. Lower Overall D5D_5: Both methods achieve nearly identical conditional-head reverse KL, confirming that vOPD effectively learns relative token preferences within the Top-4 set. However, as shown by the D5D_5 decomposition above, ResOPD's tighter calibration of the head–tail probability split KL(Bern(P)∥∣Bern(Q))\mathrm{KL}(\mathrm{Bern}(P)\||\mathrm{Bern}(Q)) yields a lower overall 5-way divergence D5D_5 (1.104 vs. 1.267).

In summary, ResOPD's advantage lies in its explicit control: it exactly averages the revealed head gradients, removing intra-head sampling noise, and explicitly constrains head–tail mass allocation through event control, yielding more stable alignment of the teacher's distribution.

Reviewer_dvDVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies on-policy distillation with sparse teacher feedback consisting of Top-k probabilities and the probability of the sampled token. ResOPD computes the gradient of a coarse distribution containing the head tokens and a tail bucket, then adds a residual correction on tail samples. Under exact on-policy sampling, it is unbiased for the full-vocabulary reverse-KL gradient at a fixed prefix. The paper derives conditions for variance reduction relative to HETS and evaluates the method through exact gradient audits and distillation experiments on mathematical reasoning, science, and coding tasks.

Critical strengths
  1. The variance decomposition clearly identifies when ResOPD improves over HETS using existing sparse teacher feedback. Exact gradient audits support the mechanism without requiring additional teacher queries.
  2. ResOPD improves the mean math score to 50.12, compared with 45.72 for HETS and 48.85 for vOPD. Full-support math experiments also favor ResOPD over HETS, while cross-domain evaluations provide further evidence of practical value.
Critical weaknesses
  1. The main comparisons do not report variability across independent training runs. Some gains are modest: the math score exceeds vOPD by 1.27 points, while the AUC exceeds TA-OPD by only 0.02 points. Their reproducibility therefore remains uncertain.
  2. The coverage of strong baselines is limited across settings. Appendix Table 8 compares SC-TA and favors ResOPD at the final math checkpoint, but neither SC-TA nor vOPD is included in the full-support math or cross-domain experiments. It remains unclear whether ResOPD’s advantage over these alternatives extends to those settings.
Questions for authors
  1. How many independent training runs underlie the main results? Can the authors report uncertainty for the key final-score and AUC differences, distinguishing evaluation sampling noise from training variability?
  2. Under the same full-support sampling configuration and teacher-feedback budget, how does ResOPD compare with SC-TA and vOPD in either the math or cross-domain setting? This comparison would help establish the scope of its practical advantage.
  3. In the existing frozen-prefix audits, what fraction of prefixes have a higher gradient covariance trace under ResOPD than under HETS? What characterizes these cases? This would help assess how reliably the coarse-tail control reduces variance beyond the reported aggregate averages.
4 comments
AuthorsOpenReview ↗

Response to Reviewer dvDV (1/3)

W1 & Q1: The main comparisons do not report variability across independent training runs. Some gains are modest: the math score exceeds vOPD by 1.27 points, while the AUC exceeds TA-OPD by only 0.02 points. Their reproducibility therefore remains uncertain. How many independent training runs underlie the main results? Can the authors report uncertainty for the key final-score and AUC differences, distinguishing evaluation sampling noise from training variability?

We thank the reviewer for pointing this out. The main results in the original submission were reported from a single run. We have therefore conducted one additional experiment targeting evaluation sampling noise and training variability, and report both below.


(1) Evaluation noise

To quantify evaluation sampling stochasticity on the evaluated checkpoints, we compute 95% paired bootstrap confidence intervals. Differences are reported as ResOPD minus baseline (pp):

BaselineΔ\Delta Avg.Score [95% CI]Δ\Delta AUC [95% CI]
TA-OPD+1.96 [−0.62, 5.54]+0.03 [−1.49, 1.54]
Sampled-token OPD+6.09 [3.07, 9.21]+1.27 [-0.57, 3.14]
vOPD+1.28 [−1.81, 4.39]+0.47 [−1.42, 2.36]
HETS+4.40 [1.10, 7.78]+1.78 [0.06, 3.52]

As observed, ResOPD demonstrates statistically significant gains over HETS under evaluation resampling. Against the competitive baselines (TA-OPD, Sampled-token OPD and vOPD), point estimates consistently favor ResOPD, while the wider intervals reflect inherent test-set sampling and decoding noise, with confidence intervals that include zero, leaving uncertainty about the magnitude and direction of the underlying differences.


(2) Training variability

We added two training runs per method, using seeds 41 and 43, and report three runs in total, including the originally submitted seed-42 run. Per-seed metrics and aggregate statistics (mean ± sample standard deviation, n=3n=3) are:

MethodAvg.Score (S41)Avg.Score (S42)Avg.Score (S43)Avg.Score (Mean ± Std)AUC (S41)AUC (S42)AUC (S43)AUC (Mean ± Std)
TA-OPD46.2848.1746.3346.93 ± 1.0846.8447.9445.6946.82 ± 1.13
Sampled-token OPD43.1544.0347.2844.82 ± 2.1844.5346.6945.2345.48 ± 1.10
vOPD46.8348.8547.5147.73 ± 1.0345.3147.4947.2846.69 ± 1.20
HETS47.2445.7245.8846.28 ± 0.8445.3946.1845.7645.78 ± 0.40
ResOPD50.0550.1249.3249.83 ± 0.4447.4247.9648.2347.87 ± 0.41

Across the three seeds, ResOPD achieves the highest mean Avg. and AUC. It has the lowest observed standard deviation in Avg., while its AUC standard deviation is comparable to HETS (0.41 vs. 0.40). These results support the consistency of the observed gains across the three tested seeds, although additional runs would be needed to estimate training variability more precisely.


Taken together, the two experiments address the two sources of uncertainty separately and support the same conclusion. The bootstrap intervals exclude zero for both metrics against HETS and for the final score against sampled-token OPD. Comparisons with TA-OPD and vOPD remain inconclusive under evaluation resampling, despite positive point estimates. The three-seed study in (2) shows that they are not an artifact of a favorable training seed. We would also emphasize that, although reseeding does produce run-to-run fluctuation in the final metrics, ResOPD consistently achieves a closer student–teacher distribution fit throughout training in every run (e.g., smaller student-teacher mass gap in Figure 3). The advantage is thus already visible in the training-time objective the method directly optimizes, rather than appearing only in end-of-training evaluation. We have added these results to Appendix B.2.

AuthorsOpenReview ↗

Response to Reviewer dvDV (2/3)

W2 & Q2: The coverage of strong baselines is limited across settings. Appendix Table 8 compares SC-TA and favors ResOPD at the final math checkpoint, but neither SC-TA nor vOPD is included in the full-support math or cross-domain experiments. It remains unclear whether ResOPD’s advantage over these alternatives extends to those settings. Under the same full-support sampling configuration and teacher-feedback budget, how does ResOPD compare with SC-TA and vOPD in either the math or cross-domain setting? This comparison would help establish the scope of its practical advantage.

We thank the reviewer for the helpful suggestion. Due to computational resource constraints, we prioritized evaluating SC-TA and vOPD under the cross-domain setting with identical full-support sampling configurations and teacher-feedback budgets. The complete results across GPQA‑Diamond, SuperGPQA, LiveCodeBench v5 (LCBv5), and LiveCodeBench v6 (LCBv6) are summarized below:

ModelGPQA‑DSuperGPQALCBv5LCBv6Avg.ScoreAUC
Student45.2532.4621.7624.3830.96--
Teacher56.0648.8745.5141.7148.04--
Sampled-OPD50.3036.8743.3139.4342.4841.21
HETS48.8936.1046.3143.2443.6342.13
vOPD39.8931.3841.1240.5738.2434.44
SC-TA49.2937.0245.3041.7143.3342.16
ResOPD47.1237.5448.9043.0544.1542.65

As shown in the table, under the same feedback budget, ResOPD achieves the highest overall Average score (44.15) and AUC (42.65), outperforming both SC-TA (43.33 / 42.16) and vOPD (38.24 / 34.44). In particular, vOPD suffers notable degradation in later training stages, even falling below the base student on scientific reasoning benchmarks like SuperGPQA and GPQA‑D, highlighting ResOPD's robustness and clear practical advantage in the cross-domain regime. We have included these results in our revised manuscript (Table 4).

AuthorsOpenReview ↗

Response to Reviewer dvDV (3/3)

Q3: In the existing frozen-prefix audits, what fraction of prefixes have a higher gradient covariance trace under ResOPD than under HETS? What characterizes these cases? This would help assess how reliably the coarse-tail control reduces variance beyond the reported aggregate averages.

Thank you for this suggestion. We analyzed the per-prefix covariance traces. In the DeepMath Qwen3.5-4B/27B Top-4 panel (8,192 prefixes), ResOPD has a higher trace than HETS on 752 prefixes (9.18%, under strict FP32 comparison). The exact condition is VResOPD−VHETS=Pτp(∣μB∣2−2⟨μtail,μB⟩),\mathcal V_{\mathrm{ResOPD}}-\mathcal V_{\mathrm{HETS}}=P\tau_p(|\mu_B|^2-2\langle\mu_{\mathrm{tail}},\mu_B\rangle), so degradation occurs when the tail-bucket control predicts the true tail mean worse than the zero control.

Beyond this condition, the clearest empirical signature is an aggregate student–teacher mass mismatch, i.e., the student assigns more probability mass than the teacher to the teacher’s Top-kk tokens: 678 of the 752 prefixes (90.2%) satisfy Pstudent>PteacherP_{student}>P_{teacher}, equivalently τp<τq\tau_p<\tau_{q}. Thus, these cases are associated with poorer tail-bucket alignment, although we observe no robust semantic or positional pattern. Since the student is also less likely to sample from the teacher tail in these prefixes, their aggregate practical impact is limited.

Thank you for the detailed responses and additional experiments. These have addressed most of my concerns, and I will maintain my positive rating.

Reviewer_hDjrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ResOPD, a variance-reduced gradient estimator for sparse on-policy distillation (OPD). Unlike existing methods that either introduce bias through Top-k truncation or suffer from high gradient variance, ResOPD aggregates the unobserved tail into a pseudo-token and computes its coarse gradient analytically, while using sampled residual corrections to preserve unbiasedness. The paper provides a theoretical variance analysis and shows that ResOPD can reduce gradient variance without additional teacher queries or forward passes. Empirically, experiments on Qwen models demonstrate lower gradient variance and improved overall performance on math, coding, and science benchmarks.

Critical strengths
  1. The problem is well-motivated and practically important, as sparse teacher feedback is widely used to reduce communication costs in OPD.

  2. The empirical results are promising. The exact gradient variance analysis and downstream evaluations provide useful evidence for the effectiveness of ResOPD.

  3. The proposed method is simple and practical, requiring no additional teacher queries or forward passes.

Critical weaknesses
  1. The proposed method appears closely related to standard control variate techniques. The main idea of analytically computing the coarse gradient and sampling the residual seems relatively straightforward. I am not fully convinced about the methodological novelty compared to existing variance-reduction methods.

  2. Although the authors provide an ablation on different Top-k values, these experiments are conducted with only one training epoch, while the main experiments use three epochs. Given that one of the key motivations of ResOPD is to improve training stability and mitigate late-stage performance degradation, I wonder whether the conclusions regarding different k values still hold under the full training budget. It would be helpful to provide a more comprehensive analysis of how k affects gradient variance and downstream performance throughout training.

  3. The experimental evaluation is somewhat limited in terms of model diversity. Although the authors consider different model pairs in the gradient variance analysis and cross-domain experiments, the main end-to-end experiments primarily focus on a single teacher-student pair (Qwen3.5-27B to Qwen3.5-4B). It remains unclear whether the proposed method generalizes to different model families and teacher-student scales. It would be helpful to include more diverse model configurations to further validate the generalizability of ResOPD.

Questions for authors

See weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer hDjr (1/3)

W1: The proposed method appears closely related to standard control variate techniques. The main idea of analytically computing the coarse gradient and sampling the residual seems relatively straightforward. I am not fully convinced about the methodological novelty compared to existing variance-reduction methods.

We thank the reviewer for this remark. We agree that the general control-variate (CV) principle is classical. Its use in LLM post-training, however, is still largely confined to scalar, action-independent baselines subtracted from a sequence- or token-level reward (PPO's critic, GRPO's group mean, RLOO, ReMax). For OPD specifically, the only CV construction we are aware of is vOPD, which subtracts the detached per-token value bt=Vπθ(ct)=−DKL(πθ∣πT)b_t=V^{\pi_\theta}(c_t)=-D_{\mathrm{KL}}(\pi_\theta|\pi_T) from the reward. This is the classical baseline transplanted to OPD: it rescales the scalar coefficient of a single sampled score, while the head is never integrated, so every coordinate of the gradient still comes from one draw. The remaining classical recipes are not instantiable under a sparse interface at all: full-vocabulary OPD requires the dense teacher distribution, and learned or relaxed CVs (MuProp/REBAR/RELAX) require extra teacher evaluations.

In ResOPD, the coarse gradient and tail sampling mainly make the estimator implementable and unbiased: the coarse term provides the computable expectation, while the sampled tail restores the fine-grained full-vocabulary gradient. The core technical contribution lies in constructing a prefix-dependent vector control variate over the tail bucket tailored to the sparse Top-kk interface. Writing ResOPD against the zero-control estimator HETS makes the difference precise: g^ResOPD=g^HETS−(J−τp)μB,J=I[A∉S].\hat{\boldsymbol g}^{\rm ResOPD}=\hat{\boldsymbol g}^{\rm HETS}-(J-\tau_p)\boldsymbol\mu_B,\qquad J=\mathbb{I}[A\notin S]. Here the textbook optimal coefficient Cov(g^HETS,J)/Var(J)\mathrm{Cov}(\hat{\boldsymbol g}^{\rm HETS},J)/\mathrm{Var}(J) evaluates exactly to μtail\boldsymbol\mu_{\rm tail}, i.e. the fine-tail mean gradient that the sparse interface hides. Nor can it be fitted, since each prefix supplies a single rollout draw and the coefficient is a parameter-space vector that moves with every update. Our μB=log⁡(τp/τq)∇θlog⁡τp\boldsymbol\mu_B=\log(\tau_p/\tau_q)\nabla_\theta\log\tau_p is instead available in closed form from the same payload: it makes the deterministic part the exact (k+1)(k{+}1)-way coarse-rKL gradient, and it equals μtail\boldsymbol\mu_{\rm tail} exactly when student and teacher share the conditional tail shape.

This control consequently reduces a different variance component from vOPD's. The sampled-token variance splits as PVhead+τpVtail+Pτp∥μhead−μtail∥2P\mathcal V_{\rm head}+\tau_p\mathcal V_{\rm tail}+P\tau_p\|\boldsymbol\mu_{\rm head}-\boldsymbol\mu_{\rm tail}\|^2; a scalar baseline rescales the reward, whereas ResOPD integrates the head and shrinks the head–tail event term to Pτp∥μtail−μB∥2P\tau_p\|\boldsymbol\mu_{\rm tail}-\boldsymbol\mu_B\|^2. We derive this decomposition and the exact reduction condition, and validate them through frozen-prefix audits and end-to-end experiments: relative to sampled-token OPD, the payload-matched vOPD removes 8–19% of the covariance trace, HETS 27–56%, and ResOPD 53–75% (Tab. 1).

In summary, rather than a naive application of classical control variates, ResOPD is an adaptation to a constraint the classical recipes do not cover: under the sparse Top-kk interface the optimal coefficient is unobservable and unfittable, and the payload's aggregate tail masses supply a closed-form substitute, with provable variance guarantees and consistent empirical gains.

AuthorsOpenReview ↗

Response to Reviewer hDjr (2/3)

W2: Although the authors provide an ablation on different Top-k values, these experiments are conducted with only one training epoch, while the main experiments use three epochs. Given that one of the key motivations of ResOPD is to improve training stability and mitigate late-stage performance degradation, I wonder whether the conclusions regarding different k values still hold under the full training budget. It would be helpful to provide a more comprehensive analysis of how k affects gradient variance and downstream performance throughout training.

We thank the reviewer for this constructive comment. We appreciate the suggestion to examine the Top-kk ablation under the full training budget, and we agree that it is important to verify whether the conclusions hold when training proceeds beyond a single epoch. Below we address each point in turn.

1. Three-epoch results for different kk.

We have supplemented the Top-kk ablation with 3-epoch training runs. Due to computational resource constraints, we reduced the amount of training data, while keeping all other configurations identical to the main experiments. The results are summarized below:

kkAIME26 (HETS / ResOPD)HMMT26 (HETS / ResOPD)OlymMath (HETS / ResOPD)Avg (HETS / ResOPD)Δ\Delta Avg
157.29 / 57.2936.88 / 37.6934.25 / 35.7542.80 / 43.58+0.78
456.04 / 57.2937.12 / 39.0236.50 / 37.5043.22 / 44.60+1.38
1658.75 / 56.6738.45 / 40.9134.50 / 36.5043.90 / 44.69+0.79
3258.54 / 58.3339.58 / 39.9633.75 / 36.7543.96 / 45.02+1.06

As shown, ResOPD consistently outperforms HETS on the aggregate metric across all tested values of kk under the 3-epoch training budget, which is consistent with the conclusions drawn from the 1-epoch ablation. This suggests that the advantage of ResOPD is not an artifact of the shortened training schedule.

2. Purpose of the Top-kk ablation.

We would like to clarify that the Top-kk ablation was designed to evaluate the estimator-level variance-reduction mechanism rather than to prescribe a universally optimal kk across all training budgets:

  • Theoretical trend (Figure 4): As kk grows, the probability that the student samples outside the teacher's Top-kk support decreases. Consequently, the tail event that ResOPD explicitly corrects occurs less frequently, naturally narrowing its variance-reduction advantage over HETS.
  • Downstream performance (Table 5): Downstream gains depend on complex optimization dynamics and specific task metrics, and thus do not strictly mirror the estimator variance trend monotonically. Nevertheless, ResOPD consistently outperforms HETS on the main aggregate metric across all tested values of kk.

3. Clarification on late-stage performance degradation.

We would like to clarify that late-stage performance degradation was an empirical phenomenon observed specifically in our math experiments. It is tied to particular experimental configurations, such as the base model, task domain, and optimization settings, rather than being a universal behavior. For example, our cross-domain experiments on Qwen3 did not exhibit this degradation (Figure 8). ResOPD is primarily designed to reduce estimator variance and enhance training stability, while whether late-stage degradation occurs remains setting-dependent.

We hope these additional results and clarifications address the reviewer's concern.

AuthorsOpenReview ↗

Response to Reviewer hDjr (3/3)

W3: The experimental evaluation is somewhat limited in terms of model diversity. Although the authors consider different model pairs in the gradient variance analysis and cross-domain experiments, the main end-to-end experiments primarily focus on a single teacher-student pair (Qwen3.5-27B to Qwen3.5-4B). It remains unclear whether the proposed method generalizes to different model families and teacher-student scales. It would be helpful to include more diverse model configurations to further validate the generalizability of ResOPD.

We thank the reviewer for this constructive suggestion regarding model diversity and generalizability.

First, we would like to clarify that our existing cross-domain experiments (Table 4) were specifically designed to evaluate the generalizability of ResOPD across model families, architectures, teacher-student scales, and tasks. In those experiments, we shifted from the Qwen3.5 family (27B →\rightarrow 4B) to the Qwen3 family (30B-A3B →\rightarrow 4B) and validated ResOPD beyond mathematics on code generation and scientific reasoning benchmarks.

Second, to further address your concern and provide more comprehensive evidence under the exact same setting as our main experiments, we conducted additional end-to-end evaluations covering both a larger student scale (Qwen3.5-27B →\rightarrow Qwen3.5-9B) and a distinct model family (Llama series, DeepSeek-R1-Distill-70B →\rightarrow DeepSeek-R1-Distill-8B). Given the substantial computational cost, we focus our comparison on our primary baseline HETS.

As shown in the tables below, ResOPD improves average accuracy over both base students and achieves higher average accuracy and AUC than HETS in both configurations across both new configurations:

1. Different Teacher-Student Scale (Qwen3.5-27B →\rightarrow Qwen3.5-9B)

ModelDeepMathAIME26HMMT26OlymMathAvg.ScoreAUC
Student (Qwen3.5-9B)64.6865.6243.1842.7554.06-
Teacher (Qwen3.5-27B)89.2980.6262.8863.2574.01-
+ HETS67.4665.6245.0847.2556.3554.90
+ ResOPD (Ours)70.4467.7147.7347.2558.2856.99

2. Different Model Family (DeepSeek-R1-Distill-70B →\rightarrow DeepSeek-R1-Distill-8B)

ModelDeepMathAIME26HMMT26OlymMathAvg.ScoreAUC
Student (DeepSeek-R1-Distill-8B)59.6738.7520.2416.5033.79-
Teacher (DeepSeek-R1-Distill-70B)85.7160.4231.6334.5053.06-
+ HETS64.4839.0821.9717.2535.7035.27
+ ResOPD (Ours)68.2538.7522.1618.5036.9236.23

These results further confirm that ResOPD generalizes effectively across varying model scales and families. We have incorporated these results into Appendix D.1.

Dear Reviewer hDjr,

Thank you for your constructive review. As the discussion period will end soon, we would greatly appreciate it if you could take a moment to review our rebuttal and additional results.

If our responses have addressed your concerns, we would be grateful if you could consider raising your score. We would also be happy to clarify any remaining questions.

Thank you very much for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.