- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper considers sparse OPD settings where only top- tokens are available on the interface, which is very often employed in practice. They theoretically demonstrate the variance decomposition of it, propose an unbiased estimator (ResOPD), and show the empirical performance improvements across multiple domains.
Critical strengths
- This paper provides bias and variance analysis of sparse OPD where the methods (HETS, SC-TA) proposed in prior works are proven to be special cases of them. ResOPD improves from HETS by reducing the variance from which tail token is sampled. This is a clear contribution.
- This paper shows the empirical validity, including cross-domain evaluation results, of ResOPD using Qwen3, Qwen3.5 models with sufficient size on math, science, and code domains, and also reports token-distribution dynamics at training time.
Critical weaknesses
- It would be better to show the results on vOPD in Qwen3 models (Table.4) since vOPD is relatively closer to ResOPD in Table.3. This is also related to the question below.
Questions for authors
- The variance of vOPD is larger than HETS (Table.1) but evaluation results of vOPD is better than HETS (Table.3). Could the authors provide the thoughts on this?
6 comments
I appreciate the authors for clarification and additional experiments. I could confirm that ResOPD is better than vOPD at Qwen3 evaluation. However, the claim that scalar KL subtraction fails to capture fine-grained token-level information remains speculative. Could the authors provide some empirical evidence for this claim, such as analysis of token-level teacher–student distribution alignment during training, or detailed explanations? In particular, what makes ResOPD more effective than vOPD at capturing this distribution structure?
We thank the reviewer for this thoughtful follow-up. We agree that our earlier statement regarding scalar KL subtraction was overly broad: vOPD retains token-specific teacher–student feedback and effectively learns the teacher's distribution. The distinction we intend to convey is that scalar centering does not explicitly average the revealed token-specific head gradients or regulate head–tail mass allocation, whereas ResOPD explicitly integrates the revealed head gradients alongside a head–tail event control.
1. Gradient-Level Mechanism: Explicit Head Integration and Head–Tail Event Control
At a fixed prefix, let be the teacher Top- set, , , , and . Teacher-Top- vOPD computes:
with a detached baseline . While subtracting centers the reward and reduces gradient variance, the single-sample update still depends on the sampled token's specific gradient direction and does not explicitly control the probability mass split between the head and the unseen tail.
ResOPD instead computes the exact conditional head gradient by averaging across all revealed Top- tokens:
Combined with head–tail event control, any sample produces the same deterministic estimate , where is the tail-bucket gradient. Both estimators share the same expected local reverse-KL gradient; the distinction lies in finite-sample estimation, where ResOPD removes intra-head sampling noise and directly constrains the head–tail probability mass balance.
2. Empirical Evidence: Distribution Alignment Across Training and Shared Prefixes
A. Head–Tail Calibration Along Training Trajectories
To track how probability mass allocation evolves during Qwen3 training, we measured the relative tail-mass mismatch (where indicates , i.e., over-allocating probability mass to the unseen tail) across training intervals:
| Training-Trajectory Diagnostic | ResOPD | vOPD |
|---|---|---|
| Relative tail-mass mismatch (Steps 1–50) | 0.295 | 0.302 |
| Relative tail-mass mismatch (Steps 51–100) | 0.154 | 0.397 |
| Teacher-Top-4 hit rate | 95.33% | 93.57% |
While both methods exhibit comparable tail-mass alignment during the first half of training (0.295 vs. 0.302 in Steps 1–50), their trajectories diverge in later updates: ResOPD continues to tighten its head–tail calibration down to 0.154 in Steps 51–100, whereas vOPD exhibits a larger tail-mass mismatch (0.397). Throughout training, ResOPD also yields a higher Teacher-Top-4 hit rate (95.33% vs. 93.57%).
B. Controlled Evaluation on Shared Prefixes
To isolate distribution alignment on identical contexts, we evaluated the final checkpoints on 4,096 shared prefixes sampled from initial-student trajectories across four benchmarks (GPQA, SuperGPQA, LCBv5 and LCBv6).
We report three overall diagnostics: (1) the absolute head–tail mass gap , (2) the conditional-head reverse KL within the Top-4 tokens, and (3) the overall 5-way reverse KL (over the 4 head tokens plus 1 combined tail bucket), which decomposes as $ D_5 = P\mathrm{KL}\bigl(p(\cdot\mid S)||q(\cdot\mid S)\bigr) + \mathrm{KL}\bigl(\mathrm{Bern}(P)||\mathrm{Bern}(Q)\bigr). $
| Shared-Prefix Diagnostic | ResOPD | vOPD |
|---|---|---|
| Absolute head–tail mass gap (pp) | 6.11 | 9.15 |
| Conditional-head reverse KL | 0.8648 | 0.8558 |
| 5-way reverse KL | 1.104 | 1.267 |
These diagnostics reveal two distinct patterns consistent with our theoretical decomposition. (1) Tighter Head–Tail Mass Calibration: Consistent with its explicit head–tail event control, ResOPD reduces the overall absolute head–tail mass gap from 9.15 to 6.11 percentage points compared to vOPD. (2) Comparable Within-Head Alignment vs. Lower Overall : Both methods achieve nearly identical conditional-head reverse KL, confirming that vOPD effectively learns relative token preferences within the Top-4 set. However, as shown by the decomposition above, ResOPD's tighter calibration of the head–tail probability split yields a lower overall 5-way divergence (1.104 vs. 1.267).
In summary, ResOPD's advantage lies in its explicit control: it exactly averages the revealed head gradients, removing intra-head sampling noise, and explicitly constrains head–tail mass allocation through event control, yielding more stable alignment of the teacher's distribution.
Response to Reviewer NF5j (1/1)
We thank the reviewer for this suggestion. We added Teacher-Top-4 vOPD to the Qwen3 cross-domain experiment. The evaluation results are shown below:
Our experiment shows vOPD also improved substantially over the student base model on code (LiveCodeBench), but experienced severe performance degradation on GPQA‑D and SuperGPQA (both degraded notably in the later stages of training, even falling below the base student), leading to lower Avg.Score and AUC compared with HETS and ResOPD. This provides complementary evidence that estimator variance and downstream ranking are not universally aligned: vOPD performs better than HETS in the math setting, whereas the ranking reverses under this Qwen3 cross-domain recipe. In theory, vOPD remains an unbiased estimator of the target reverse-KL gradient by subtracting the local KL divergence (a scalar) between the student and teacher over the teacher's Top-k tokens. However, we suspect that this scalar subtraction does not fully exploit the fine-grained distribution geometry across these Top-k tokens, which may ultimately hinder fine-grained student-teacher matching in certain scenarios. We have added these results to Table 4 in our revised manuscript.