acceptodds
Under review as a conference paper at ICLR 2027

On the Position Bias of On-Policy Distillation

Abstract

On-Policy Distillation (OPD) improves the learning efficiency of standard reinforcement learning through dense, token-level supervision from teachers. In the standard KL objective of OPD, token-level losses are uniformly averaged, implying equal weights for all tokens. However, we discover that not all tokens are created equal: as student rollouts grow longer, they deviate further from the teacher's distribution, leading to degraded supervision quality at later positions. As a result, OPD using only the first 30% of tokens can perform comparably to using all tokens, whereas OPD using only the last 30% of tokens barely learns anything. In this work, we provide a principled understanding of this issue through the lens of constrained optimization. Based on these insights, we derive Importance-Weighted On-Policy Distillation (IW-OPD), in which the weight assigned to each token depends on the accumulated discrepancy between the student's and teacher's distributions, naturally upweighting earlier tokens and downweighting later ones with larger deviations. We show that IW-OPD converges significantly faster than OPD, with better learning efficiency, and achieves better final performance than standard OPD in both same-size and cross-scale settings, improving performance by points on AIME-2025.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Credit Assignment, Large Language Model Post-Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mYUUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper observes the position bias of OPD, and studies the effect of learning only the early tokens in the generated response. The paper proposes a constrained optimization perspective, and derived the optimal solution to this problem. It also motivates the importance-weighted OPD that re-weights the gradient contribution of different positions in the generated response. Experiments show that the small student can obtain larger gains through this improved OPD algorithm.

Critical strengths
  1. The question studied in the paper is clear and meaningful. The experiments indeed showed that supervising the early tokens is better than supervising the late tokens, and it motivates the problem of re-weighting different positions.
  2. The experiments are extremely extensive. Experiments on many different models and tasks are presented to support the effect of importance re-weighting. Ablation studies also demonstrates the effectiveness of the the algorithm design.
Critical weaknesses

There are huge gaps between the theory and the algorithm.

Proposition 1 gives the optimal solution to a constrained optimization problem. After this, you claimed that it explains the position bias in Figure 1a. This is not rigorous, because you only gave the optimal solution to the optimization problem, but it is not directly related to the training trajectory, and it cannot justify training only the early tokens have smaller KL divergence. You are measuring the expected reward (not the KL divergence) in Figure 1a. I would recommend adding the KL divergence curve.

In addition, the constrained optimization problem and this proposition does not justify the benefit of weight decay. You did not show the specific design of your weight decay will yield a smaller KL divergence, or a higher expected reward.

You proposed a new optimization problem in equation (7). I don't understand why learning this objective is superior to the original KL divergence objective. You can't learn qq because it is not a parameterized policy, and you can only learn π(θ)\pi(\theta), so what is point of optimizing the KL divergence between qq and πT\pi_T? The role of ρ\rho is also not explained in the objective.

Even if you decided to consider the optimization problem in equation (7), there are so many approximations from equation (9) to equation (15). You are eventually using something so different from your theory that it makes me wonder if your theory is the correct way to explain your algorithm.

The way you proposed the exact form of your weight decay through positions is not satisfactory. It is not guaranteed by any theory and I wonder whether your re-weighting is nearly optimal, or there are other better ways to do the re-weighting that you haven't found. Why you make all the weights greater than 1 is also not convincing because it is not supported by any theory.

The exact gradient of OPD is in equation (57), but in practice this is approximated. This choice makes the gradient biased, and the bias may increase in tt. Intuitively, the variance also increases in tt. Therefore, my understanding of the re-weighting is that it reduces the bias and variance caused by late tokens. I am also interested in the performance of using the precise gradient in (57) to do the optimization, and see if it better or worse than optimizing the approximated objective, and if re-weighting still helps.

There is also a typo after equation (9). You forgot to write α\alpha in the definition of rtr_t.

Questions for authors

I already wrote everything in the Critical Weaknesses section. Please refer to that.

8 comments
AuthorsOpenReview ↗

Dear Reviewer mYUU (1/8)

We are grateful to the reviewer for the detailed examination of our derivation, which give us a chance to state the link between Prop. 1, the training objective, and the practical weights more explicitly. Our point-by-point responses follow.

W1: There are huge gaps between the theory and the algorithm. Proposition 1 gives the optimal solution to a constrained optimization problem. After this, you claimed that it explains the position bias in Figure 1a. This is not rigorous, because you only gave the optimal solution to the optimization problem, but it is not directly related to the training trajectory, and it cannot justify training only the early tokens have smaller KL divergence.

This is not rigorous, because you only gave the optimal solution to the optimization problem, but it is not directly related to the training trajectory, and it cannot justify training only the early tokens have smaller KL divergence.

A1.1: Prop. 1 enters the training process through Eq. 7. At every training step, Eq. 7 takes the projection qθ⋆q_\theta^\star of the current student given by Prop. 1 (Eqs. 5–6) and directly minimizes its remaining KL gap to the teacher:

min⁡θDKL(qθ⋆∥πT).\min_\theta D_{\mathrm{KL}}(q_\theta^\star \Vert \pi_T).

On student rollouts, Prop. 2 turns this objective into a token-level objective in which each KL term is weighted by the prefix-ratio factor rtαr_t^\alpha (Eq. 8–9):

JIW⋆(θ)=−Ey∼πθ[∑t=1Tsg[rtαZα,t]log⁡πθ(yt∣y<t)πT(yt∣y<t)],\mathcal J_{\mathrm{IW}}^\star(\theta)=-\mathbb E_{y\sim\pi_\theta}\left[\sum_{t=1}^{T}\mathrm{sg}\left[\frac{r_t^\alpha}{Z_{\alpha,t}}\right]\log\frac{\pi_\theta(y_t\mid y_{ < t})}{\pi_T(y_t\mid y_{ < t})}\right],

where Zα,t=Ey<t∼πθ[rtα]Z_{\alpha,t}=\mathbb E_{y_{ < t}\sim\pi_\theta}[r_t^\alpha] is the position-wise normalizer, sg[⋅]\mathrm{sg}[\cdot] is the stop-gradient operator, and

rtα=(πT(y<t)πθ(y<t))α=exp⁡[α∑k<tlog⁡πT(yk∣y<k)πθ(yk∣y<k)].r_t^\alpha=\left(\frac{\pi_T(y_{ < t})}{\pi_\theta(y_{ < t})}\right)^\alpha=\exp\left[\alpha\sum_{k < t}\log\frac{\pi_T(y_k\mid y_{ < k})}{\pi_\theta(y_k\mid y_{ < k})}\right].

Under student sampling, each term of this sum is non-positive in expectation:

Eyk∼πθ(⋅∣y<k)[log⁡πT(yk∣y<k)πθ(yk∣y<k)]=−DKL(πθ(⋅∣y<k)∥πT(⋅∣y<k))≤0.\mathbb E_{y_k\sim\pi_\theta(\cdot\mid y_{ < k})}\left[\log\frac{\pi_T(y_k\mid y_{ < k})}{\pi_\theta(y_k\mid y_{ < k})}\right]=-D_{\mathrm{KL}}\big(\pi_\theta(\cdot\mid y_{ < k}) \Vert \pi_T(\cdot\mid y_{ < k})\big)\le 0 .

The weight rtαr_t^\alpha therefore decays with position for any student, and hence at every training step, so the objective that Eq. 7 optimizes is weighted toward early tokens. This matches the KL curves in A1.2: Prefix-30 reaches a KL comparable to full-token OPD, whereas Suffix-30 remains clearly higher (Table R1).

We have revised Sec. 3.2 and Sec. 4.1 to state this link explicitly.

You are measuring the expected reward (not the KL divergence) in Figure 1a. I would recommend adding the KL divergence curve.

A1.2: We show that Prefix-30 achieves lower mean token-level reverse KL than Suffix-30 and reaches a level comparable to full-token OPD (Table R1). The corresponding KL curves in App. H, Fig. 6a directly evaluate student–teacher distribution matching under different supervision positions.

Table R1. Mean token-level reverse KL for full-token, prefix-only, and suffix-only OPD at selected training steps. Lower values are better.

SupervisionStep 0Step 10Step 30Step 50
Full-token OPD0.770.560.520.47
Prefix-30 (first 30%)0.750.520.480.48
Suffix-30 (last 30%)0.740.680.620.63
AuthorsOpenReview ↗

Dear Reviewer mYUU (2/8)

W2: In addition, the constrained optimization problem and this proposition does not justify the benefit of weight decay. You did not show the specific design of your weight decay will yield a smaller KL divergence, or a higher expected reward.

In addition, the constrained optimization problem and this proposition does not justify the benefit of weight decay.

A2.1: The overall decay emerges from accumulated teacher–student likelihood ratios on student rollouts.

1. Theory: the prefix-ratio factor decays with position in expectation. Proposition 1 gives ratio-based weights, and the prefix-ratio factor in Eq. 9 is rtα=(πT(y<t)πθ(y<t))α=exp⁡[α∑k<tlog⁡πT(yk∣y<k)πθ(yk∣y<k)].r_t^\alpha=\left(\frac{\pi_T(y_{< t})}{\pi_\theta(y_{< t})}\right)^\alpha=\exp\Big[\alpha\sum_{k< t}\log\frac{\pi_T(y_k\mid y_{< k})}{\pi_\theta(y_k\mid y_{< k})}\Big]. Under student sampling, each term is non-positive in expectation: Eyk∼πθ(⋅∣y<k)[log⁡πT(yk∣y<k)πθ(yk∣y<k)]=−DKL(πθ(⋅∣y<k)∥πT(⋅∣y<k))≤0.\mathbb E_{y_k\sim\pi_\theta(\cdot\mid y_{< k})}\Big[\log\frac{\pi_T(y_k\mid y_{< k})}{\pi_\theta(y_k\mid y_{< k})}\Big]=-D_{\mathrm{KL}}(\pi_\theta(\cdot\mid y_{< k})\Vert\pi_T(\cdot\mid y_{< k}))\le0. The accumulated log-ratio thus becomes more negative along student rollouts, giving a downward trend in the prefix-ratio factor whose shape follows the sampled trajectory, not a predefined schedule over positions.

2. Experiments: the decay is observed on student rollouts. Over 30 random student rollouts, the mean Spearman correlation between rtαr_t^\alpha and token position is −0.964-0.964 for all α\alpha, since rank correlation is invariant to the monotone map rt↦rtαr_t\mapsto r_t^\alpha. Mean Pearson correlations are also negative (Table R2). Consistently, Fig. 3c shows teacher prefix probabilities decrease faster than student ones.

Table R2. Mean within-rollout correlations between rtαr_t^\alpha and token position (same 30 rollouts).

α10.10.010.001
Pearson-0.273-0.435-0.921-0.974
Spearman-0.964-0.964-0.964-0.964

You did not show the specific design of your weight decay will yield a smaller KL divergence, or a higher expected reward.

A2.2: IW-OPD lowers the final mean token-level reverse KL from 0.47 to 0.41 vs. standard OPD (Table R3, App. H). The stabilization-step ablation (Table R4) shows IW-OPD raises accuracy, lowers KL and reduces mean gradient variance. Tables 1–3 further show consistent accuracy gains across the evaluated teacher–student settings.

Table R3. Mean token-level reverse KL during training (lower is better).

Step0103050
OPD0.770.560.520.47
IW-OPD0.760.410.430.41

Table R4. Component ablation. Ratio-based variants use α=0.001 unless noted.

VariantLog scale + NormalizationUnsigned accumulationOPD blendMean gradient variance↓AIME25 mean@32 (%)↑Final Reverse KL↓
OPD———24.917.10.47
Prefix ratio (α=1)✗✗✗192.45.10.69
Prefix ratio (α=0.001)✗✗✗37.816.50.54
Ratio + blend✗✗✓25.117.80.45
Signed✓✗✗24.318.00.46
Signed + blend✓✗✓19.518.30.43
Unsigned✓✓✗21.418.30.44
IW-OPD✓✓✓19.119.00.41
AuthorsOpenReview ↗

Dear Reviewer mYUU (3/8)

W3: You proposed a new optimization problem in equation (7). I don't understand why learning this objective is superior to the original KL divergence objective. You can't learn qq because it is not a parameterized policy, and you can only learn π(θ)\pi(\theta), so what is point of optimizing the KL divergence between qq and πT\pi_T? The role of ρ\rho is also not explained in the objective.

You proposed a new optimization problem in equation (7). I don't understand why learning this objective is superior to the original KL divergence objective.

A3.1: We believe that Eq. 7 is superior to the standard OPD objective under a finite update budget for the following reasons:

1. Supervision value depends on the student prefix. OPD uses teacher next-token logits conditioned on student-generated prefixes, but standard OPD weights all token-level KL terms uniformly. Supervising only the first 30% of response tokens yields higher reward and lower KL than supervising the last 30%. This motivates allocating supervision according to prefix-dependent value. Supported by Fig. 1 and Table R1.

2. Directly optimizing the residual teacher–student gap yields a sampling distribution that matches this motivation. KL reduction during standard OPD plateaus with a nonzero residual. Eq. 5 models this limited progress by considering distributions qq within a local KL budget around the student, and Eq. 7 directly minimizes the remaining gap DKL(qθ⋆∥πT)D_{\mathrm{KL}}(q_\theta^\star \Vert \pi_T) by updating the student parameters θ\theta. The resulting sampling distribution qθ⋆q_\theta^\star is a likelihood-ratio reweighting of student rollouts (Eq. 6), which allocates the limited update budget according to the prefix-dependent supervision value in point 1. Consistently, both prefix ratios and practical weights correlate positively with teacher continuation accuracy at matched response positions. Supported by Fig. 3a–b, Eq. 5–7 and Prop. 1, and the matched-position correlations in App. G, Fig. 5.

3. IW-OPD improves training outcomes. Compared with standard OPD, IW-OPD achieves lower final reverse KL, higher task accuracy across the evaluated model settings, and lower gradient variance. These results support the practical benefit of reallocating supervision. Supported by Table R3 (reverse KL), Tables 1–3 (task accuracy), and Table R4 (gradient variance).

You can't learn qq because it is not a parameterized policy, and you can only learn π(θ)\pi(\theta), so what is point of optimizing the KL divergence between qq and πT\pi_T?

A3.2: We use a change of measure to express the projected objective using student samples. No separately trained policy is required.

App. A.2, Eq. 38, gives, for a sample loss ff,

Ey∼qθ⋆[f(y)]=Ey∼πθ[qθ⋆(y)πθ(y)f(y)].\mathbb E_{y\sim q_\theta^\star}[f(y)]=\mathbb E_{y\sim\pi_\theta}\left[\frac{q_\theta^\star(y)}{\pi_\theta(y)}f(y)\right].

The weight qθ⋆/πθq_\theta^\star/\pi_\theta is computed from the student and teacher probabilities via Eq. 6. Applying this identity to the KL decomposition in Eq. 8 yields the student-sampled objective in App. A.2, Eq. 40.

The role of ρ\rho is also not explained in the objective.

A3.3: ρ\rho is the effective local update budget. It limits how far qq can move from the current student (DKL(q∥πθ)≤ρD_{\mathrm{KL}}(q \Vert \pi_\theta)\leq\rho in Eq. 5), modeling the limited progress of OPD in Fig. 3a–b. Through Prop. 1, ρ\rho enters the objective via α\alpha. By Eq. 29, for small α\alpha,

ρ≈12α2Var(log⁡r),\rho\approx\tfrac12\alpha^2\mathrm{Var}(\log r),

so fixing either α\alpha or ρ\rho determines the other, and a larger budget corresponds to a larger α\alpha, i.e., stronger reweighting toward teacher-compatible trajectories. As ρ→0\rho\to 0, α→0\alpha\to 0 and qθ⋆→πθq_\theta^\star\to\pi_\theta, so Eq. 7 reduces to the standard OPD objective.

Reviewer_kDVXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies position bias in OPD: supervision on early response tokens is more effective than supervision on late tokens. It motivates prefix weighting through a KL-constrained projection and proposes IW-OPD, which weights standard OPD advantages using normalized cumulative absolute teacher–student log-probability differences. Experiments report improved early-checkpoint and final performance across several Qwen3 teacher–student configurations.

Critical strengths
  1. The prefix-versus-suffix experiment provides a clear and useful diagnostic of where OPD supervision helps (Figure 1).
  2. The method reuses existing teacher scores and covers several model scales, with main results averaged over three seeds. Table 4 includes fixed-prefix, linear-decay, and curriculum controls.
Critical weaknesses
  1. The interpretation of the practical weights needs stronger support. The theory motivates likelihood-ratio weighting, while Eqs. 14–15 use normalized cumulative absolute disagreement. Proportionally scaling all disagreements in a response leaves its weights unchanged, despite changing total disagreement. Table 4 supports the heuristic’s usefulness, but does not directly establish that its weights measure teacher supervision quality.

  2. Some reported averages appear inconsistent. Assuming Avg is the unweighted mean of the five benchmark columns, Table 1’s final 0.6B IW-OPD row averages 20.64, rather than 20.2; Table 2’s 4B OPD-at-step-10 row averages 53.54, rather than 54.0. These differences exceed rounding and affect the reported gain comparisons.

Questions for authors
  1. Can the weights in Eq. 15 be validated against teacher continuation accuracy at matched response positions, using the diagnostic in Figure 1b?
  2. How is Avg computed in Tables 1–2, and could the two inconsistent rows and their reported gains be checked?
6 comments
AuthorsOpenReview ↗

Dear Reviewer kDVX (1/4)

We sincerely appreciate the reviewer’s close reading, including the recomputation of our reported averages and the recognition of the position-bias diagnosis, and we address each concern in turn below.

W1: The practical weights (Eqs. 14–15) deviate from the theoretical likelihood-ratio weighting, and their interpretation as teacher-supervision quality lacks direct evidence.

The interpretation of the practical weights needs stronger support. The theory motivates likelihood-ratio weighting, while Eqs. 14–15 use normalized cumulative absolute disagreement.

A1.1: Thank you for this point. Below, we first explain the role of α\alpha in the ideal weight, which is the core difficulty, and then show that the practical weight follows from the ideal weight in Prop. 1 through three transformations. Each has a clear motivation and validity condition, and each is supported by the ablation in Table R1.

Table R1. Ablation of the transformations (teacher: Qwen3-4B-Instruct-2507; student: Qwen3-0.6B). Mean gradient variance is averaged over all training steps.

VariantLog scaling + normalizationUnsigned accumulationOPD blendMean gradient variance ↓AIME25 mean@32 (%) ↑
OPD———24.917.1
Prefix ratio (α=1\alpha=1)✗✗✗192.45.1
Prefix ratio (α=0.001\alpha=0.001)✗✗✗37.816.5
Ratio + blend✗✗✓25.117.8
Signed✓✗✗24.318.0
Signed + blend✓✗✓19.518.3
Unsigned✓✓✗21.418.3
IW-OPD✓✓✓19.119.0

Let St=∑k<tAkOPD=log⁡rtS_t=\sum_{k < t}A_k^{\mathrm{OPD}}=\log r_t and Dt=∑k<t∣AkOPD∣D_t=\sum_{k < t}\lvert A_k^{\mathrm{OPD}}\rvert. The ideal weight in Prop. 1 is rtα=eαStr_t^{\alpha}=e^{\alpha S_t}.

The role of α\alpha. The coefficient α\alpha in the ideal weight is not a free hyperparameter. By Eq. 29, it is tied to the trust-region budget ρ\rho and the likelihood ratio rr; for small α\alpha,

ρ≈12α2σ2,σ2=Var(log⁡r).\rho\approx\tfrac12\alpha^2\sigma^2,\qquad \sigma^2=\mathrm{Var}(\log r).

Using the ideal weight therefore requires fixing either α\alpha or ρ\rho, yet neither can be fixed reliably. The budget ρ\rho is implicit and not observed. A fixed α\alpha would have to match the discrepancy scale, which varies 3–5× across responses and shifts about 5× during training (Table R2). This explains why the ideal weight is so sensitive to α\alpha (Fig. 4a), and it motivates the transformations below.

Table R2. Accumulated discrepancy on training rollouts (teacher: Qwen3-4B-Instruct-2507; student: Qwen3-0.6B), reported as the 10th–90th percentile range across responses at each training step.

Step 0Step 10Step 20Step 30Step 40Step 50
Total ∣ST∣\lvert S_T\rvert (p10–p90)373–11731802–63211077–4557808–4165788–3812756–3869
Per token (p10–p90)0.46–1.420.36–0.790.29–0.530.30–0.610.29–0.510.28–0.50
AuthorsOpenReview ↗

Dear Reviewer kDVX (2/4)

(a) Log scaling.

Motivation. The exponential weight is unstable. Because ∣St∣\lvert S_t\rvert grows to hundreds or thousands along a response (Fig. 3c), the weights either collapse to zero or become nearly flat (Fig. 4a–b). Direct ratio weighting therefore has higher variance and lower accuracy than OPD.

Transformation. We use the first-order Taylor expansion

eαSt≈1+αSt.e^{\alpha S_t}\approx 1+\alpha S_t .

The constant term 11 is standard OPD, and αSt\alpha S_t is the log-scaled discrepancy (Eq. 12).

Condition. The expansion requires αSt≈0\alpha S_t\approx 0. By the relation above, αSt\alpha S_t is of order 2ρ\sqrt{2\rho}, which is small in the local regime where OPD operates (Fig. 3a–b). A fixed α\alpha violates this on high-discrepancy responses; normalization (b) addresses this.

(b) Normalization.

Motivation. The linear weight still contains α\alpha, which cannot be fixed reliably (see above). It must also remain non-negative, since a negative weight would reverse the OPD gradient. Because the discrepancy scale varies widely (Table R2), a fixed α\alpha that keeps all weights non-negative would make the correction negligible on most responses.

Transformation. We rescale within each response:

r~tnorm=St−Smin⁡Smax⁡−Smin⁡∈[0,1].\tilde r_t^{\mathrm{norm}}=\frac{S_t-S_{\min}}{S_{\max}-S_{\min}}\in[0,1].

The common factor α\alpha cancels in this ratio, so neither α\alpha nor ρ\rho needs to be specified, and scaling all discrepancies by a constant leaves the weights unchanged. Together with (a), this reduces gradient variance from 37.8 to 24.3 and raises accuracy from 16.5% to 18.0% (Table R1). The two steps are ablated jointly because the linear form is invalid without normalization.

(c) Unsigned accumulation.

Motivation. With signed accumulation, a later token on which the teacher is more confident can cancel earlier drift, so a drifted prefix can look compatible again (rebounds in Fig. 4b). This overestimates the compatibility of drifted prefixes, whose teacher supervision is less reliable (Fig. 1b).

Transformation. Since AkOPD≥−∣AkOPD∣A_k^{\mathrm{OPD}}\ge-\lvert A_k^{\mathrm{OPD}}\rvert, we replace StS_t by its lower bound:

St≥−Dt.S_t \ge -D_t .

The weight becomes non-increasing along the response, and normalization simplifies to r~tnorm=1−Dt/DT\tilde r_t^{\mathrm{norm}}=1-D_t/D_T (Eq. 14). The lower bound may underestimate compatibility but never overestimates it, and the two errors are not equally costly. Overestimation gives extra weight to a drifted prefix and amplifies a noisy signal, whereas underestimation only withholds part of the extra weight, while the OPD term still supervises every token. The bound is therefore a conservative choice. It improves accuracy from 18.0% to 18.3% without the blend and from 18.3% to 19.0% with it, and it reduces gradient variance in both cases (Table R1).

Condition. The bound is tight when positive advantages are small in total, since the gap is 2∑k<tmax⁡(AkOPD,0)2\sum_{k < t}\max(A_k^{\mathrm{OPD}},0). This holds on student rollouts (Sec. 4.2, Fig. 3c): about 15% of tokens have positive advantages, but their magnitudes are small (Table R3).

Table R3. Share of positive advantages in ∑k∣AkOPD∣\sum_k\lvert A_k^{\mathrm{OPD}}\rvert on training rollouts (same setting as Table R2).

Training step01020304050Mean
Positive-advantage share of ∑k∣AkOPD∣\sum_k\lvert A_k^{\mathrm{OPD}}\rvert (%)2.23.74.13.43.52.93.3

The bound therefore differs from STS_T by 6% of DTD_T on average throughout training.

Final form. The practical weight is r~tIW-OPD=1+γr~tnorm\tilde r_t^{\mathrm{IW\text{-}OPD}}=1+\gamma\tilde r_t^{\mathrm{norm}}. The constant 11 is the zeroth-order term from (a), i.e., standard OPD. γ\gamma controls how much update budget is reallocated, and γ=0\gamma=0 recovers OPD. Keeping this term preserves supervision on late tokens, where r~tnorm→0\tilde r_t^{\mathrm{norm}}\to 0, and it helps in every configuration of Table R1.

Revision. This analysis is now included in the paper (App. A.4).

AuthorsOpenReview ↗

Dear Reviewer kDVX (3/4)

Proportionally scaling all disagreements in a response leaves its weights unchanged, despite changing total disagreement.

A1.2: This invariance is a deliberate design trade-off, and the total disagreement still enters the update.

1. Scale invariance is a deliberate design trade-off. In the ideal weight eαSte^{\alpha S_t}, the absolute scale of the disagreement matters only through α\alpha, which is tied to the budget ρ\rho (A1.1, Eq. 29). Neither can be set reliably: ρ\rho is implicit and not observed, and a fixed α\alpha cannot match a discrepancy scale that varies 3–5× across responses and shifts about 5× during training (Table R2). We therefore deliberately give up the scale-dependent part of the ideal weight and keep the part that can be computed reliably, namely the relative ordering of prefixes within a response. The cost is that the weight no longer reflects differences in total disagreement between responses. In return, neither α\alpha nor ρ\rho needs to be specified, and jointly with log scaling, normalization reduces gradient variance from 37.8 to 24.3 and raises accuracy from 16.5% to 18.0% (Table R1).

2. The total disagreement is not discarded. The weight only allocates the update across positions within a response. The IW-OPD advantage is r~tIW-OPDAtOPD\tilde r_t^{\mathrm{IW\text{-}OPD}}A_t^{\mathrm{OPD}}, so the magnitude of each token's disagreement still enters the update through AtOPDA_t^{\mathrm{OPD}}: responses with larger disagreement still receive larger updates.

3. IW-OPD is compatible with trajectory-level selection. FiRe-OPD filters student rollouts using the teacher’s average token log-probability. We apply this filter and use IW-OPD weights within the retained responses. The resulting IW-FiRe-OPD outperforms both FiRe-OPD and IW-OPD on all three benchmarks (Table R4), improving over FiRe-OPD by 1.8, 1.1, and 0.8 percentage points on AIME25, BrowseComp, and DeepPlanning, respectively. Together with IW-OPD’s gains over standard OPD without filtering (Table R1), these results support the standalone value of IW-OPD’s within-response optimization and its compatibility with trajectory selection.

Table R4. Trajectory filtering combined with IW token weighting. Results from Sec. 5.3, Table 5. All scores are percentages.

MethodAIME25BrowseCompDeepPlanning
FiRe-OPD50.637.621.3
IW-OPD49.738.121.9
IW-FiRe-OPD52.438.722.1

Table 4 supports the heuristic’s usefulness, but does not directly establish that its weights measure teacher supervision quality.

A1.3: We show that the practical weight correlates positively with teacher continuation accuracy, an observable proxy for supervision quality, as the ideal likelihood ratio does. Following the diagnostic in Fig. 1b, we pair each student-generated prefix with the teacher's accuracy when continuing from it. Both the ideal prefix likelihood ratio and the practical weight have positive Spearman correlations with this accuracy at all three matched response positions (Table R5; App. G, Fig. 5). Position-only schedules, such as the linear decay in Table 4, assign a constant weight at a fixed position and cannot produce this association. The correlation therefore reflects the trajectory-dependent part of the weight rather than position alone.

Table R5. Spearman correlation between the weights and teacher continuation accuracy at matched response positions (App. G).

Weighting signal30% position50% position70% position
Prefix likelihood ratio0.7670.4330.635
Practical weight0.7090.3970.633
Reviewer_bZGrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a position bias in On-Policy Distillation (OPD). The authors observe that OPD's token-level reverse-KL objective weights all tokens uniformly, yet supervision quality degrades along the rollout: with a fixed 30% token budget, supervising only the first 30% of tokens matches or exceeds full supervision, while supervising only the last 30% barely learns (Fig. 1a); correspondingly, the teacher's ability to reach correct answers from student-generated prefixes decays as prefixes grow (Fig. 1b). The authors formalize OPD as a trust-region problem — minimizing KL(q‖πT) subject to KL(q‖πθ) ≤ ρ — whose closed-form optimum reweights the student distribution by the teacher-to-student likelihood ratio, explaining why uniform weighting misallocates the update budget. They derive Importance-Weighted OPD (IW-OPD), which multiplies each token's OPD advantage by a prefix-importance weight. The practical estimator accumulates unsigned token-level teacher–student discrepancies over the prefix, normalizes within each response, and interpolates with standard OPD (a single hyperparameter γ, no additional teacher evaluations). Across two teacher regimes (Qwen3-4B-Instruct, Qwen3-30B-A3B) and three student scales (0.6B/1.7B/4B), plus a 235B-teacher/30B-student setting, IW-OPD improves both sample efficiency (e.g., +6.9 AIME25 at step 10) and final performance, with larger relative gains for stronger teachers and smaller students; ablations isolate trajectory-adaptive weighting from fixed or linear schedules, and an appendix study shows the weighting composes with ExOPD.

Critical strengths

1. A well-characterized phenomenon with a principled explanation. The prefix/suffix controlled study (Fig. 1a), the teacher-capability decay curve (Fig. 1b), and the KL-plateau observation (Fig. 3) together establish the position bias convincingly; Proposition 1 then explains it from a trust-region/constrained-projection perspective with a closed-form optimum. This is consequential because it changes how one should think about OPD supervision — as a finite-budget allocation problem rather than uniform token matching — and provides a theoretical language for a whole family of reweighted objectives.

2. A zero-overhead, immediately adoptable method with consistent, well-ablated gains. The prefix weights reuse advantage terms that standard OPD already computes, so IW-OPD requires no additional teacher evaluations and reduces to OPD at γ = 0. Gains are consistent across all six teacher–student pairs and benchmarks, and the trend that stronger teachers and smaller students benefit more matches the theory's prediction (larger drift ⇒ weighting matters more). The ablations cleanly separate trajectory-adaptive weighting (+5.6) from fixed-prefix amplification (+0.4) and linear decay (+0.7). Consequential: if the standalone advantage over existing reweighting schemes holds up, the method is practically useful at essentially no cost.

Critical weaknesses

1. Insufficient comparison with existing reweighting methods — the paper's central claim is not established against other baselines. The contribution is a token-reweighting scheme, yet no published token-reweighting method for OPD appears in any comparison table. Notably, TIP (Token Importance in OPD), REOPOLD, FiRe-OPD (trajectory filtering plus soft token reweighting) and EOPD — all OPD-native reweighting baselines, the same family this paper belongs to — are neither compared; the ablations only test hand-designed variants. I acknowledge that some might be concurrent work, but more baselines should be considered indeed. Moreover, the one published-method comparison (ExOPD) is relegated to an appendix combination study: reading Tables 1 and 10 together, ExOPD alone attains a higher average than IW-OPD alone for both the 4B student (57.8 vs 57.1) and the 1.7B student (41.7 vs 40.3), and IW-OPD's demonstrated value rests on stacking with ExOPD (+1.0/+1.5). This is consequential because the reader cannot determine whether IW-OPD's advantage comes from the specific prefix-likelihood-ratio weighting or would be matched by any reasonable token weighting — which is precisely what the paper claims to improve over.

2. The gap between the presented theory and the method actually used. The theoretically derived likelihood-ratio weight is unstable and underperforms standard OPD (−1.2 in Table 4); the effective method is a four-step surrogate (log scaling, unsigned accumulation, per-response normalization, OPD blend) that the authors themselves describe as "a conservative prefix-compatibility proxy..., not an exact density-ratio correction." The theory therefore functions mainly as motivation, and the practical method's success is an empirical finding. Consequential: the paper's claim of a "principled understanding" is weaker than stated — it is unclear which parts of the derivation are load-bearing or what falsifiable prediction the theory makes, which also makes it hard to transfer the insight to future designs.

Questions for authors
  1. Please provide a direct comparison, in one table under matched training budgets, of IW-OPD against published OPD reweighting methods — in particular FiRe-OPD (trajectory filtering + soft token reweighting), TIP (Token Importance in OPD), REOPOLD, and EOPD — including IW-OPD alone, ExOPD alone, and IW-ExOPD side by side. Why are these closely related methods neither cited nor compared, given that FiRe-OPD addresses the same question of how to allocate supervision within rollouts? (Clarification request — material to whether IW-OPD has standalone value.)
  2. Why does ExOPD alone outperform IW-OPD alone on average in both student settings? What capability does prefix weighting add that reward scaling does not?
  3. The ideal weight fails in practice (Table 4). What falsifiable prediction does the constrained-projection theory make, and which of the four stabilization steps (log scaling / unsigned accumulation / normalization / OPD blend) is essential to recover the benefit? Does the normalization step effectively remove the α parameter, and is that consistent with the theory?
  4. The Limitations state that experiments are "conducted at the 4B student scale," but Table 3 reports a 30B-A3B student — please clarify the intended scope of the claims.
8 comments
AuthorsOpenReview ↗

Dear Reviewer bZGr (1/7)

We thank the reviewer for the constructive review. The requested comparison with published OPD reweighting methods and the questions about which parts of the theory are load-bearing have helped us strengthen the paper, and we respond to each point below.

W1: Insufficient comparison with existing reweighting methods.

Insufficient comparison with existing reweighting methods — the paper's central claim is not established against other baselines. The contribution is a token-reweighting scheme, yet no published token-reweighting method for OPD appears in any comparison table. Notably, TIP (Token Importance in OPD), REOPOLD, FiRe-OPD (trajectory filtering plus soft token reweighting) and EOPD — all OPD-native reweighting baselines, the same family this paper belongs to — are neither compared; the ablations only test hand-designed variants. I acknowledge that some might be concurrent work, but more baselines should be considered indeed.

A1.1: The direct comparisons under matched training budgets demonstrate IW-OPD’s standalone value, especially on the two agentic benchmarks. We thank the reviewer for suggesting this comparison. Table R1 shows gains over OPD on all three benchmarks and the best standalone scores among the compared methods on BrowseComp and DeepPlanning.

Table R1. Direct comparison under matched settings within each benchmark. All scores are percentages; bold marks the best standalone result.

MethodAIME25BrowseCompDeepPlanning
OPD48.035.719.6
TIP48.536.920.4
REOPOLD49.136.018.4
FiRe-OPD50.637.621.3
EOPD48.335.517.7
ExOPD50.137.121.0
IW-OPD49.738.121.9
IW-ExOPD51.739.022.1
IW-FiRe-OPD52.438.722.1

All methods share the teacher, student initialization, training data, generated-token budget, and rollout/evaluation settings within each benchmark. Filtered rollouts count toward the budget, and agentic tools and call limits are fixed. AIME25 uses a Qwen3-30B-A3B-Instruct-2507 teacher and Qwen3-4B student. BrowseComp and DeepPlanning use a Qwen3.5-35B-A3B teacher and Qwen3.5-9B student, reporting task accuracy and the official DeepPlanning aggregate score over English travel and shopping levels 1–3, respectively.

TIP emphasizes token entropy and teacher–student disagreement. FiRe-OPD combines trajectory filtering with uncertainty-based token weighting. REOPOLD adjusts token selection and feedback strength over training. EOPD adds forward KL at high-teacher-entropy positions.

We have added this comparison to the main text (Sec. 5.3, Table 5), together with an analysis of how these methods differ and the corresponding citations in Related Work (Sec. 6). App. F gives the detailed settings.

Moreover, the one published-method comparison (ExOPD) is relegated to an appendix combination study: reading Tables 1 and 10 together, ExOPD alone attains a higher average than IW-OPD alone for both the 4B student (57.8 vs 57.1) and the 1.7B student (41.7 vs 40.3), and IW-OPD's demonstrated value rests on stacking with ExOPD (+1.0/+1.5).

A1.2: IW-OPD alone outperforms ExOPD on both agentic benchmarks, BrowseComp and DeepPlanning.

  • Method Comparison: ExOPD’s reward scaling can encourage exploration by increasing reward strength relative to KL regularization, but it does not introduce a dedicated mechanism for allocating supervision across token positions within an update. In multi-turn agentic tasks, earlier actions shape later observations and decisions, making the causal role of prefixes more important.
  • Results: Table R1 shows that IW-OPD alone outperforms ExOPD by 1.0 and 0.9 percentage points on BrowseComp and DeepPlanning, respectively. These results support the distinct roles of reward scaling and prefix-dependent supervision allocation. Combining the two further improves both scores, supporting their compatibility. This distinction allows prefix weighting to be applied on top of ExOPD’s reward design, as described in App. E.
AuthorsOpenReview ↗

Dear Reviewer bZGr (2/7)

This is consequential because the reader cannot determine whether IW-OPD's advantage comes from the specific prefix-likelihood-ratio weighting or would be matched by any reasonable token weighting — which is precisely what the paper claims to improve over.

A1.3: IW-OPD weights tokens by the reliability of teacher supervision on the student's prefix. IW-OPD is motivated by differences in the value of teacher supervision across student-generated prefixes. In OPD, the teacher provides next-token prediction logits conditioned on each prefix of a student rollout. When a rollout drifts into a prefix that the teacher cannot reliably continue, those logits can become less useful for learning.

The weights track supervision quality, which position-only weighting cannot capture. At matched response positions, both the prefix likelihood ratio and the practical weight correlate positively with teacher continuation accuracy (App. G; Table R5). A position-only schedule gives all prefixes at the same position the same weight and cannot capture this, which is consistent with the much smaller gains of hand-designed schedules in Table 4.

IW-OPD addresses a different aspect of supervision allocation than FiRe-OPD and ExOPD, and adding it to either method yields further gains. FiRe-OPD selects trajectories and weights tokens by local uncertainty, and ExOPD rescales the reward uniformly across positions. IW-OPD instead distributes supervision along a trajectory according to the discrepancy accumulated over its prefix. If another weighting scheme could replace IW-OPD, stacking the two would bring little benefit; in practice, adding IW-OPD improves both methods on all three benchmarks (Table R1).

We have added design comparison to the main text (Sec. 5.3).

AuthorsOpenReview ↗

Dear Reviewer bZGr (3/7)

W2: The gap between the presented theory and the method actually used.

The gap between the presented theory and the method actually used. … the effective method is a four-step surrogate (log scaling, unsigned accumulation, per-response normalization, OPD blend) that the authors themselves describe as "a conservative prefix-compatibility proxy..., not an exact density-ratio correction." The theory therefore functions mainly as motivation, and the practical method's success is an empirical finding. Consequential: the paper's claim of a "principled understanding" is weaker than stated — it is unclear which parts of the derivation are load-bearing.

A2.1: Thank you for this point. Below, we first explain the role of α\alpha in the ideal weight, which is the core difficulty, and then show that the practical weight follows from the ideal weight in Prop. 1 through three transformations. Each has a clear motivation and validity condition, and each is supported by the ablation in Table R2.

Table R2. Ablation of the transformations (teacher: Qwen3-4B-Instruct-2507; student: Qwen3-0.6B). Mean gradient variance is averaged over all training steps.

VariantLog scaling + normalizationUnsigned accumulationOPD blendMean gradient variance ↓AIME25 mean@32 (%) ↑
OPD———24.917.1
Prefix ratio (α=1\alpha=1)✗✗✗192.45.1
Prefix ratio (α=0.001\alpha=0.001)✗✗✗37.816.5
Ratio + blend✗✗✓25.117.8
Signed✓✗✗24.318.0
Signed + blend✓✗✓19.518.3
Unsigned✓✓✗21.418.3
IW-OPD✓✓✓19.119.0

Let St=∑k<tAkOPD=log⁡rtS_t=\sum_{k < t}A_k^{\mathrm{OPD}}=\log r_t and Dt=∑k<t∣AkOPD∣D_t=\sum_{k < t}\lvert A_k^{\mathrm{OPD}}\rvert. The ideal weight in Prop. 1 is rtα=eαStr_t^{\alpha}=e^{\alpha S_t}.

The role of α\alpha. The coefficient α\alpha in the ideal weight is not a free hyperparameter. By Eq. 29, it is tied to the trust-region budget ρ\rho and the likelihood ratio rr; for small α\alpha,

ρ≈12α2σ2,σ2=Var(log⁡r).\rho\approx\tfrac12\alpha^2\sigma^2,\qquad \sigma^2=\mathrm{Var}(\log r).

Using the ideal weight therefore requires fixing either α\alpha or ρ\rho, yet neither can be fixed reliably. The budget ρ\rho is implicit and not observed. A fixed α\alpha would have to match the discrepancy scale, which varies 3–5× across responses and shifts about 5× during training (Table R3). This explains why the ideal weight is so sensitive to α\alpha (Fig. 4a), and it motivates the transformations below.

Table R3. Accumulated discrepancy on training rollouts (teacher: Qwen3-4B-Instruct-2507; student: Qwen3-0.6B), reported as the 10th–90th percentile range across responses at each training step.

Step 0Step 10Step 20Step 30Step 40Step 50
Total ∣ST∣\lvert S_T\rvert (p10–p90)373–11731802–63211077–4557808–4165788–3812756–3869
Per token (p10–p90)0.46–1.420.36–0.790.29–0.530.30–0.610.29–0.510.28–0.50

Comments on the paper

1 comment
AuthorsOpenReview ↗

Paper Update

We sincerely thank all reviewers for their careful reading and valuable comments. We have responded to each concern and incorporated the suggestions into the updated manuscript. All revisions are marked in blue. The main revisions are:

  • Sec. 3.2 & 4.1: clarify that Prop. 1 applies to the current student at every training step, and explain why the constrained optimum concentrates its mass on the few teacher-compatible prefixes along student rollouts.
  • Sec. 5.3 (new!): comparison with recent OPD methods under matched settings on AIME25, BrowseComp, and DeepPlanning (Table 5)
    • Token selection and reweighting: TIP, REOPOLD, FiRe-OPD
    • Objective modification: EOPD, ExOPD
    • Combinations: IW-ExOPD and IW-FiRe-OPD
  • Sec. 6: add these methods to Related Work and clarify how IW-OPD differs from them.
  • Tables 1–4, Sec. 5.2 & 5.4, Table 14: correct rounding inconsistencies in averages and gains, and define how Avg is computed.
  • Appendix A.4 (new!): derivation from the ideal weight to the practical weight
    • why α\alpha cannot be fixed in advance
    • the derivation and validity condition of each step: log scaling, unsigned accumulation, normalization, and OPD blending
    • component ablation, discrepancy scale, and positive-advantage share (Tables 6–8)
  • Appendix F (new!): experimental settings and implementation details for the comparison in Sec. 5.3
  • Appendix G (new!): weights versus teacher continuation accuracy at matched response positions
  • Appendix H (new!): reverse-KL training dynamics
    • prefix-only versus suffix-only supervision
    • OPD versus IW-OPD
  • Appendix I (new!): gradient variance across token positions during training
  • Appendix J (new!): comparison of token-local and full return-to-go updates

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.