- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper observes the position bias of OPD, and studies the effect of learning only the early tokens in the generated response. The paper proposes a constrained optimization perspective, and derived the optimal solution to this problem. It also motivates the importance-weighted OPD that re-weights the gradient contribution of different positions in the generated response. Experiments show that the small student can obtain larger gains through this improved OPD algorithm.
Critical strengths
- The question studied in the paper is clear and meaningful. The experiments indeed showed that supervising the early tokens is better than supervising the late tokens, and it motivates the problem of re-weighting different positions.
- The experiments are extremely extensive. Experiments on many different models and tasks are presented to support the effect of importance re-weighting. Ablation studies also demonstrates the effectiveness of the the algorithm design.
Critical weaknesses
There are huge gaps between the theory and the algorithm.
Proposition 1 gives the optimal solution to a constrained optimization problem. After this, you claimed that it explains the position bias in Figure 1a. This is not rigorous, because you only gave the optimal solution to the optimization problem, but it is not directly related to the training trajectory, and it cannot justify training only the early tokens have smaller KL divergence. You are measuring the expected reward (not the KL divergence) in Figure 1a. I would recommend adding the KL divergence curve.
In addition, the constrained optimization problem and this proposition does not justify the benefit of weight decay. You did not show the specific design of your weight decay will yield a smaller KL divergence, or a higher expected reward.
You proposed a new optimization problem in equation (7). I don't understand why learning this objective is superior to the original KL divergence objective. You can't learn because it is not a parameterized policy, and you can only learn , so what is point of optimizing the KL divergence between and ? The role of is also not explained in the objective.
Even if you decided to consider the optimization problem in equation (7), there are so many approximations from equation (9) to equation (15). You are eventually using something so different from your theory that it makes me wonder if your theory is the correct way to explain your algorithm.
The way you proposed the exact form of your weight decay through positions is not satisfactory. It is not guaranteed by any theory and I wonder whether your re-weighting is nearly optimal, or there are other better ways to do the re-weighting that you haven't found. Why you make all the weights greater than 1 is also not convincing because it is not supported by any theory.
The exact gradient of OPD is in equation (57), but in practice this is approximated. This choice makes the gradient biased, and the bias may increase in . Intuitively, the variance also increases in . Therefore, my understanding of the re-weighting is that it reduces the bias and variance caused by late tokens. I am also interested in the performance of using the precise gradient in (57) to do the optimization, and see if it better or worse than optimizing the approximated objective, and if re-weighting still helps.
There is also a typo after equation (9). You forgot to write in the definition of .
Questions for authors
I already wrote everything in the Critical Weaknesses section. Please refer to that.
8 comments
Dear Reviewer mYUU (2/8)
W2: In addition, the constrained optimization problem and this proposition does not justify the benefit of weight decay. You did not show the specific design of your weight decay will yield a smaller KL divergence, or a higher expected reward.
In addition, the constrained optimization problem and this proposition does not justify the benefit of weight decay.
A2.1: The overall decay emerges from accumulated teacher–student likelihood ratios on student rollouts.
1. Theory: the prefix-ratio factor decays with position in expectation. Proposition 1 gives ratio-based weights, and the prefix-ratio factor in Eq. 9 is Under student sampling, each term is non-positive in expectation: The accumulated log-ratio thus becomes more negative along student rollouts, giving a downward trend in the prefix-ratio factor whose shape follows the sampled trajectory, not a predefined schedule over positions.
2. Experiments: the decay is observed on student rollouts. Over 30 random student rollouts, the mean Spearman correlation between and token position is for all , since rank correlation is invariant to the monotone map . Mean Pearson correlations are also negative (Table R2). Consistently, Fig. 3c shows teacher prefix probabilities decrease faster than student ones.
Table R2. Mean within-rollout correlations between and token position (same 30 rollouts).
| α | 1 | 0.1 | 0.01 | 0.001 |
|---|---|---|---|---|
| Pearson | -0.273 | -0.435 | -0.921 | -0.974 |
| Spearman | -0.964 | -0.964 | -0.964 | -0.964 |
You did not show the specific design of your weight decay will yield a smaller KL divergence, or a higher expected reward.
A2.2: IW-OPD lowers the final mean token-level reverse KL from 0.47 to 0.41 vs. standard OPD (Table R3, App. H). The stabilization-step ablation (Table R4) shows IW-OPD raises accuracy, lowers KL and reduces mean gradient variance. Tables 1–3 further show consistent accuracy gains across the evaluated teacher–student settings.
Table R3. Mean token-level reverse KL during training (lower is better).
| Step | 0 | 10 | 30 | 50 |
|---|---|---|---|---|
| OPD | 0.77 | 0.56 | 0.52 | 0.47 |
| IW-OPD | 0.76 | 0.41 | 0.43 | 0.41 |
Table R4. Component ablation. Ratio-based variants use α=0.001 unless noted.
| Variant | Log scale + Normalization | Unsigned accumulation | OPD blend | Mean gradient variance↓ | AIME25 mean@32 (%)↑ | Final Reverse KL↓ |
|---|---|---|---|---|---|---|
| OPD | — | — | — | 24.9 | 17.1 | 0.47 |
| Prefix ratio (α=1) | ✗ | ✗ | ✗ | 192.4 | 5.1 | 0.69 |
| Prefix ratio (α=0.001) | ✗ | ✗ | ✗ | 37.8 | 16.5 | 0.54 |
| Ratio + blend | ✗ | ✗ | ✓ | 25.1 | 17.8 | 0.45 |
| Signed | ✓ | ✗ | ✗ | 24.3 | 18.0 | 0.46 |
| Signed + blend | ✓ | ✗ | ✓ | 19.5 | 18.3 | 0.43 |
| Unsigned | ✓ | ✓ | ✗ | 21.4 | 18.3 | 0.44 |
| IW-OPD | ✓ | ✓ | ✓ | 19.1 | 19.0 | 0.41 |
Dear Reviewer mYUU (3/8)
W3: You proposed a new optimization problem in equation (7). I don't understand why learning this objective is superior to the original KL divergence objective. You can't learn because it is not a parameterized policy, and you can only learn , so what is point of optimizing the KL divergence between and ? The role of is also not explained in the objective.
You proposed a new optimization problem in equation (7). I don't understand why learning this objective is superior to the original KL divergence objective.
A3.1: We believe that Eq. 7 is superior to the standard OPD objective under a finite update budget for the following reasons:
1. Supervision value depends on the student prefix. OPD uses teacher next-token logits conditioned on student-generated prefixes, but standard OPD weights all token-level KL terms uniformly. Supervising only the first 30% of response tokens yields higher reward and lower KL than supervising the last 30%. This motivates allocating supervision according to prefix-dependent value. Supported by Fig. 1 and Table R1.
2. Directly optimizing the residual teacher–student gap yields a sampling distribution that matches this motivation. KL reduction during standard OPD plateaus with a nonzero residual. Eq. 5 models this limited progress by considering distributions within a local KL budget around the student, and Eq. 7 directly minimizes the remaining gap by updating the student parameters . The resulting sampling distribution is a likelihood-ratio reweighting of student rollouts (Eq. 6), which allocates the limited update budget according to the prefix-dependent supervision value in point 1. Consistently, both prefix ratios and practical weights correlate positively with teacher continuation accuracy at matched response positions. Supported by Fig. 3a–b, Eq. 5–7 and Prop. 1, and the matched-position correlations in App. G, Fig. 5.
3. IW-OPD improves training outcomes. Compared with standard OPD, IW-OPD achieves lower final reverse KL, higher task accuracy across the evaluated model settings, and lower gradient variance. These results support the practical benefit of reallocating supervision. Supported by Table R3 (reverse KL), Tables 1–3 (task accuracy), and Table R4 (gradient variance).
You can't learn because it is not a parameterized policy, and you can only learn , so what is point of optimizing the KL divergence between and ?
A3.2: We use a change of measure to express the projected objective using student samples. No separately trained policy is required.
App. A.2, Eq. 38, gives, for a sample loss ,
The weight is computed from the student and teacher probabilities via Eq. 6. Applying this identity to the KL decomposition in Eq. 8 yields the student-sampled objective in App. A.2, Eq. 40.
The role of is also not explained in the objective.
A3.3: is the effective local update budget. It limits how far can move from the current student ( in Eq. 5), modeling the limited progress of OPD in Fig. 3a–b. Through Prop. 1, enters the objective via . By Eq. 29, for small ,
so fixing either or determines the other, and a larger budget corresponds to a larger , i.e., stronger reweighting toward teacher-compatible trajectories. As , and , so Eq. 7 reduces to the standard OPD objective.
Dear Reviewer mYUU (1/8)
We are grateful to the reviewer for the detailed examination of our derivation, which give us a chance to state the link between Prop. 1, the training objective, and the practical weights more explicitly. Our point-by-point responses follow.
W1: There are huge gaps between the theory and the algorithm. Proposition 1 gives the optimal solution to a constrained optimization problem. After this, you claimed that it explains the position bias in Figure 1a. This is not rigorous, because you only gave the optimal solution to the optimization problem, but it is not directly related to the training trajectory, and it cannot justify training only the early tokens have smaller KL divergence.
A1.1: Prop. 1 enters the training process through Eq. 7. At every training step, Eq. 7 takes the projection qθ⋆ of the current student given by Prop. 1 (Eqs. 5–6) and directly minimizes its remaining KL gap to the teacher:
minθDKL(qθ⋆∥πT).
On student rollouts, Prop. 2 turns this objective into a token-level objective in which each KL term is weighted by the prefix-ratio factor rtα (Eq. 8–9):
JIW⋆(θ)=−Ey∼πθ[∑t=1Tsg[Zα,trtα]logπT(yt∣y<t)πθ(yt∣y<t)],
where Zα,t=Ey<t∼πθ[rtα] is the position-wise normalizer, sg[⋅] is the stop-gradient operator, and
rtα=(πθ(y<t)πT(y<t))α=exp[α∑k<tlogπθ(yk∣y<k)πT(yk∣y<k)].
Under student sampling, each term of this sum is non-positive in expectation:
Eyk∼πθ(⋅∣y<k)[logπθ(yk∣y<k)πT(yk∣y<k)]=−DKL(πθ(⋅∣y<k)∥πT(⋅∣y<k))≤0.
The weight rtα therefore decays with position for any student, and hence at every training step, so the objective that Eq. 7 optimizes is weighted toward early tokens. This matches the KL curves in A1.2: Prefix-30 reaches a KL comparable to full-token OPD, whereas Suffix-30 remains clearly higher (Table R1).
We have revised Sec. 3.2 and Sec. 4.1 to state this link explicitly.
A1.2: We show that Prefix-30 achieves lower mean token-level reverse KL than Suffix-30 and reaches a level comparable to full-token OPD (Table R1). The corresponding KL curves in App. H, Fig. 6a directly evaluate student–teacher distribution matching under different supervision positions.
Table R1. Mean token-level reverse KL for full-token, prefix-only, and suffix-only OPD at selected training steps. Lower values are better.