acceptodds
Under review as a conference paper at ICLR 2027

When Does Teacher Feedback Hurt? Outcome-Aligned Reweighting for On-Policy Distillation

Abstract

On-policy distillation (OPD) provides dense teacher feedback on student-generated trajectories, but not every teacher-induced update improves task success. We measure the causal utility of this feedback by applying the OPD update contributed by a single token position and measuring the resulting change in the student's probability of solving the same prompt. These interventions reveal that most token updates have small causal effects on accuracy, while a few produce large improvements or substantial degradation. To estimate utility without performing costly interventions, we derive a first-order score based on the alignment between each token's distillation update and the student's outcome policy gradient. The scores correlate strongly with measured intervention effects across training checkpoints, with Pearson correlations of over . We then propose ALIGN-OPD (outcome-ALIGNed On-Policy Distillation), which uses these scores to reweight token-level teacher feedback during training. Across instruction following, code, and math domains, ALIGN-OPD improves mean domain accuracy by points over the RL teacher, over vanilla OPD, and over the strongest competing baseline.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Language modeling, on-policy distillation, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ztqXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether teacher feedback in on-policy distillation (OPD) always improves the student's task performance. They measure the utility of individual token updates by applying each parameter update separately and evaluating its effect on the student's probability of answering questions correctly. They then derive a first-order approximation based on alignment between the token's OPD update and the gradient of expected task success. Using this score, they reweight teacher feedback during training and experiments on SmolLM3 and Qwen3 across instruction following, code, and math show improvements over vanilla OPD and several competing baselines.

Critical strengths
  • generally i think the paper is focusing on an interesting research question: whether learning from a teacher signal actually improves the student's ability to solve a problem, and it proposes a direct way to evaluate that in a token-wise manner.

  • it also provides a way to connect teacher-induced token-wise updates with task success, though subject to a certain amount of computational effort, and it has been shown to gain performance improvements across different domains and two model backbones

Critical weaknesses
  • the utility scores used during actual training seem not to be the same as those used to demonstrate the strong correlation between estimated token utility and the actual change in answer correctness after applying an individual token update, as shown in fig 4 and in sec 4.1. It thus raises a concern about whether the training-time estimate is still reliable under this approximation.

  • the additional computational (training) cost (e.g., estimating outcome gradients and computing token-level updates) induced by this method makes it difficult to assess relative to the modest performance gains (Vanilla OPD with residual clipping seems to also work comparably well). Can the authors provide a clear measurement of the actual training-time overhead introduced by the practical score estimation? This matters since the performance gains are relatively small (see Tables 1 and 2), so it would be informative to see such a report of extra cost.

  • i find the token-level intervention analysis interesting; however, the observation that only some token positions have a large effect on task outcome is not entirely surprising given earlier findings on token importance in reasoning and opd. I think a systematic investigation of what distinguishes tokens with large positive utility, large negative utility, and near-zero utility would make this paper more informative. For example, are harmful updates concentrated in particular types of tokens? How do the distributions of positive and negative utility differ between correct and incorrect responses? Are the patterns related to token position?

  • The presentation of the core training procedure and some math details could be made clearer. E.g., I found sec 4.2 is a bit difficult to follow; for instance, why is twofold scoring used? Also, this paper uses terms that may have ambiguity, like gradient: outcome gradient, outcome policy gradient, and token OPD gradient, etc. They may have different roles in different contexts, so their use in the text should avoid any ambiguity. Also, Eq. 2 and eq 1 use different subscripts with no explanation. Also, this paper states that negative and near-zero feedback is suppressed. However, the implemented weighting function (w_t=\sigma(3(\hat s_t+1))) assigns approximately 0.95 to a token with zero alignment score. Can the authors clarify this?

Questions for authors
  • Since residual clipping seems to work comparably well, can the authors clarify what exactly the clipping is doing, how it differs, and how low the practical cost of using clipping is compared to that of the proposed methods?

  • Can the authors explain why a response should be scored using the opposite fold in sec 4.2? The procedure here needs to be described more clearly

  • How noisy could the estimate of outcome gradients be? especially when nearly all responses are correct or incorrect

  • can the authors show how the distribution of absolute teacher–student residuals evolves alongside utility in Fig3, to examine whether their changes follow similar trends during training?

  • sec 3.3 states that positions with negligible parameter updates and near-zero utility are excluded, so what are the thresholds?

  • eq 13 assigns zero weight to a target fold when the opposite fold is all correct or incorrect, but it may imply we are using less amount of training data; could the authors report the percentage of dropped folds across training? furthermore, it would be useful to compare against vanilla OPD trained with the same fold-selection mechanism, but without the token-level weights applied, in order to distinguish improvements from selective removal of training examples.

  • since the causal utility in eq 7 is defined wrt one prompt, whether the authors can do a small cross-prompt analysis, in which we measure the effects of high- and low-utility updates on held-out prompts, and this could help whether the causal utility is primarily prompt-specific or not?

4 comments

We thank the reviewer for the positive assessment of our work and the constructive questions on the practical estimator, training overhead, and token-level dynamics. Because several weaknesses and questions touch on closely related parts of the method, we group them below for brevity while addressing every point directly.


W1: Validation of the training-time score (BF16 FD & fewer probes)

This is a valid and important concern. To directly validate the training-time estimator, we ran an update-level experiment on 16 prompts (Qwen3-4B). For each prompt, we split NN training responses into two equal folds and scored each fold using the opposite fold's outcome gradient via our training-time BF16 Richardson FD procedure (N/2N/2 probes per fold, mixed outcomes so no folds were dropped). Starting from the same checkpoint and fresh AdamW state, we compared one optimization step of unweighted OPD against BF16 FD-weighted OPD on the exact same responses, evaluating before/after accuracy with 100 paired rollouts per prompt:

Total responses NNProbes per foldUnweighted OPD (%)BF16 FD-weighted OPD (%)Gain from weighting
16813.6323.13+9.50 pp
64 (training setting)3212.9425.25+12.31 pp
25612811.6933.63+21.94 pp
1,02451211.5638.19+26.63 pp

Even with 8–32 probes and BF16 finite differences, weighting consistently improves the actual parameter update (+12.31 pp at our training setting of 32 probes/fold). While larger NN yields higher single-prompt gains here, under a fixed rollout budget during training, N=64N=64 vs. 1,0241,024 trades off 16×16\times prompt diversity; we therefore use N=32–64N=32\text{--}64 in training to balance probe quality with prompt coverage.

W2 & Q1: Training-time computational overhead and comparison with residual clipping

We profiled wall-clock time on our Qwen training run using 8×H200 GPUs (N=64N=64, 1,024 responses/step):

StageAvg. time / step (min)Share of step time
Rollout generation (shared with Vanilla OPD)11.4149.6%
Student, teacher & base-model log-probs (shared)2.7111.7%
Richardson FD (4 BF16 forward passes)2.9212.7%
Probe gradient, scoring & communication2.8412.3%
Final parameter update, reward & logging (shared)3.1613.7%
Total (ALIGN-OPD vs. Vanilla OPD)23.03 vs. 17.28100% (1.33×1.33\times Vanilla)

Because our two-fold split reuses existing training rollouts rather than generating separate probe batches, and directional FD avoids per-token backward passes, the scoring pipeline adds 5.76 min/step (+33% wall-clock time).

How residual clipping differs (Q1). Residual clipping (u~t=clip(ut,−c,c)\tilde{u}_t = \mathrm{clip}(u_t, -c, c)) is computationally free (an element-wise op), but it only caps the magnitude of extreme teacher–student disagreement utu_t (and does not bound the factor ∇θlog⁡pθ\nabla_\theta \log p_\theta). As Figures 2 and 3 show, even among tokens with large ∣ut∣\lvert u_t\rvert, the induced updates can be strongly beneficial, near-zero, or strongly harmful. Consequently, clipping indiscriminately weakens large-residual updates that are actually helpful, while still allowing harmful updates to proceed with capped magnitude cc in the wrong direction. ALIGN-OPD instead evaluates whether the update direction dtOPDd_t^{\mathrm{OPD}} aligns with improving task success (⟨g^x/∥g^x∥2,dtOPD⟩\langle \hat{g}_x/\lVert\hat{g}_x\rVert_2, d_t^{\mathrm{OPD}}\rangle), retaining beneficial high-residual updates in full while suppressing harmful ones towards zero.

W3: Characteristics of helpful and harmful token updates

We agree that a closer look at what distinguishes helpful and harmful updates makes the analysis much more informative. We therefore expanded our investigation on Figure 3 and the new 16-prompt Qwen3-4B audit (16×6416 \times 64 responses, mentioned in our response to W1). We will incorporate these findings into the revised manuscript:

  • Utility may not be fixed by token type: In Figure 3, the same token "Wait" appears at both the positive and negative extremes. We found that this happens because parameter updates alter broader generation habits rather than just local semantics: encouraging reconsideration tokens can help the student recover from mistakes, whereas, in some other cases, triggering extra continuation on an already-correct trajectory can cause length truncation before the final answer is reached.

  • Correct vs. incorrect responses & update direction: Across our 16-prompt audit, about 6% of tokens are strongly downweighted (wt≤0.1w_t \le 0.1), at similar rates in both correct and incorrect responses. Over 70% of these downweighted updates are cases where the teacher tries to decrease a token's probability (ut<0u_t < 0), but the outcome gradient favors keeping it.

Below, we provide three examples from real responses illustrate how token position, trajectory context, and error type may cause these weights in practice (the studied token is marked with ⟦·⟧):

1. Blocking the teacher from penalizing a correct answer token (wt=0w_t = 0). When counting non-isomorphic well-orderings of N\mathbb{N}, the student outputs the correct answer \boxed{\aleph_⟦1⟧}. Because the teacher may prefer alternative phrasing or formatting at this prefix, it assigns a strong negative residual to 1, whereas the outcome gradient strongly supports it. ALIGN-OPD sets wt=0w_t = 0 to block this harmful suppression (on this prompt, one-step update improves accuracy from 19% to 92%). Notably, position matters even for the exact same token: when \aleph_⟦1⟧ appears again just a few tokens later in the post-answer explanation, the teacher agrees with the student (ut=0u_t = 0) and the token is retained (wt≈0.95w_t \approx 0.95).

... Therefore, the number of **non-isomorphic well-orderings of $\mathbb{N}$** is **uncountably infinite**.

### ✅ Final Answer:
$$
\boxed{\aleph_⟦1⟧}   <-- [u_t < 0, outcome opposes teacher -> w_t = 0 (downweighted)]
$$
(i.e., the cardinality of the set of non-isomorphic well-orderings of $\mathbb{N}$ is the cardinality of the first uncountable cardinal, $\aleph_⟦1⟧$, assuming the Axiom of Choice.)<|im_end|>   <-- [u_t = 0 -> w_t = 0.95 (retained)]

2. Dampening unproductive trial-and-error (wt≈0w_t \approx 0). On a functional-equation prompt, the student guesses g(x)=xg(x)=x, sees it fails, and writes "So ⟦that⟧ doesn't work. Try g(x) = x^2." Here the teacher reinforces the guess-and-test transition (that, ut>0u_t > 0), while the outcome direction opposes this, possibly because the outcome gradient supports a more efficient method than this trial-and-error way.

... If we try $g(x) = x$, then:
$$
g(xy) + g(x(1-y)) = xy + x(1-y) = x \neq xy \quad (\text{unless } y = \frac{1}{2})
$$
So ⟦that⟧ doesn't work.

Try $g(x) = x^2$. Then: ...

3. Contrast-retaining teacher penalties on actual errors (wt=1w_t = 1). Unlike Case 1, when the teacher penalizes an actual mathematical mistake, ALIGN-OPD keeps the update. On a binary-tree recursion prompt where the base case is D1=1D_1 = 1, a student mistakenly writes "D_1 = ⟦0⟧". Here both the teacher and the outcome direction push 0 down, yielding wt=1w_t = 1 and preserving the corrective signal.

... A tree with 1 inner vertex (just the root) has a left subtree with 0 inner vertices, which is valid (since 0 is even).
So we must define:
$$
D_1 = ⟦0⟧ \quad (\text{since the root has a left tree of } 0 \dots)
$$

W4 & Q2: Two-fold scoring, gradient notation, and sigmoid weighting

  • Why score with the opposite fold? (W4, Q2): If a target response yjy_j is included when computing the outcome gradient g^x=1M∑i(Ri−Rˉx)∇θlog⁡pθ(yi∣x)\hat{g}_x = \frac{1}{M}\sum_i (R_i - \bar{R}_x)\nabla_\theta \log p_\theta(y_i\mid x), scoring yjy_j against g^x\hat{g}_x introduces self-influence, as the response's own trajectory contributes to the reference direction. Leave-one-out scoring would avoid this but requires NN separate outcome-gradient backward passes per prompt. Splitting the NN responses into two folds (F1F_1 scores F2F_2, and F2F_2 scores F1F_1) eliminates self-influence while requiring only 2 outcome-gradient computations per prompt. We will revise Section 4.2 to make this motivation and procedure clear.
  • Gradient terminology and Eq. (1)–(2) notation (W4): In Equations (1) and (2), LOPD\mathcal{L}_{\mathrm{OPD}} and LRKL\mathcal{L}_{\mathrm{RKL}} refer to the same reverse-KL objective; we will unify the subscript to LOPD\mathcal{L}_{\mathrm{OPD}} and explicitly indicate evaluation at θ=θˉ\theta=\bar{\theta}. Throughout the manuscript, we will also standardize our terminology to strictly distinguish three quantities: the population outcome policy gradient gx=∇θVx(θˉ)g_x = \nabla_\theta V_x(\bar{\theta}), the empirical estimated outcome gradient g^x\hat{g}_x, and the token OPD update dtOPD=ut∇θlog⁡pθ(yt∣zt)∣θ=θˉd_t^{\mathrm{OPD}} = u_t\nabla_\theta \log p_\theta(y_t\mid z_t)\big\rvert_{\theta=\bar{\theta}}.
  • Sigmoid weighting behavior (W4): We agree that our description in Section 4.2 stating that "near-zero feedback is suppressed" was inaccurate. Under our implemented weight function, a score of s^t=0\hat{s}_t = 0 receives wt=σ(3)≈0.95w_t = \sigma(3) \approx 0.95 (and wt=0.5w_t = 0.5 at s^t=−1\hat{s}_t = -1). The function suppress strongly negative updates while retaining most of the weight on near-zero and positively aligned feedback. We will correct this description in Section 4.2 to match this.

Q3 & Q6: Outcome-gradient noise and homogeneous-fold dropping

How noisy are the outcome-gradient estimates? (Q3) On our 16-prompt Qwen3-4B set, we split the NN responses of each prompt into two equal folds (N/2N/2 probes per fold) and computed the full-parameter cosine similarity cos⁡(g^x(1),g^x(2))\cos(\hat{g}_x^{(1)}, \hat{g}_x^{(2)}) between the two folds' outcome gradients:

Total rollouts NNProbes / foldMean cos⁡\cosMin / max across 16 prompts
1680.015-0.044 / 0.093
64 (training)320.045-0.007 / 0.138
2561280.1570.006 / 0.493
1,0245120.3600.060 / 0.797
  • All-correct / all-incorrect folds: When a fold is completely homogeneous, centered rewards vanish (Ri−Rˉx=0R_i - \bar{R}_x = 0) and g^x=0\hat{g}_x = 0. When outcomes are heavily skewed (e.g., a prompt with 90% accuracy), the minority outcome has very few samples in a finite fold, potentially yielding noisier directions (cos⁡=−0.002\cos = -0.002 at N=64N=64 and 0.0980.098 at N=1,024N=1,024).
  • Large cross-prompt variation, weak link to difficulty: Cosine varies widely across prompts, and correlates weakly with overall prompt accuracy (ρ=0.09\rho = 0.09 at N=64N=64; ρ=0.26\rho = 0.26 at N=1,024N=1,024). Instead, longer responses accumulate more variance (ρ=−0.39\rho = -0.39 to −0.52-0.52 with mean token length). For example, at ~52% accuracy, a shorter-response prompt (705 tokens) reaches cos⁡=0.138\cos = 0.138 (N=64N=64) and 0.5060.506 (N=1,024N=1,024), whereas a longer-response prompt (2,958 tokens) stays at 0.0110.011 and 0.0600.060.
  • Effectiveness despite full-parameter noise: Although full-parameter cosine in a 4B space averages 0.0450.045 at N=64N=64, our response to W1 shows that projecting token updates onto this 32-probe outcome gradient already improves one-step accuracy by +12.31 pp.

Homogeneous-fold dropping rate and isolation from token weighting (Q6). In our N=64N=64 Qwen3-4B Math training, ~15% of folds (weighted by valid tokens) are dropped. While compute constraints during the rebuttal window prevented us from conducting a full set of training runs of vanilla OPD with fold-dropping, two existing experiments directly isolate token-level weighting from fold removal:

  • W1 has 0% dropped folds: In the experiments described in our response to W1, all 16 prompts have mixed outcomes in both folds (no responses dropped), yet token weighting alone improves one-step accuracy from 12.94% to 25.25% (+12.31 pp) at N=64N=64.
  • Table 3 ablations share the exact same fold dropping: In Section 5.2 (Table 3), Reverse reweighting uses the exact same homogeneous-fold dropping rule as ALIGN-OPD, but inverting the token weights degrades accuracy by 3.91 points.

Q4: How absolute residuals ∣ut∣\vert{}u_t\vert{} relate to and evolve alongside utility

The residual magnitude ∣ut∣\vert{}u_t\vert{} plays an important role in causal utility because it directly scales the token update dtOPD=ut∇θlog⁡pθ(yt∣zt)d_t^{\mathrm{OPD}} = u_t \nabla_\theta \log p_\theta(y_t \mid z_t). More specifically:

  • When ∣ut∣\vert{}u_t\vert{} is near zero, utility can hardly be large: A near-zero residual induces a negligible parameter step, keeping Δt\Delta_t close to zero.
  • When ∣ut∣\vert{}u_t\vert{} is large, utility is not necessarily large (or positive): A large ∣ut∣\vert{}u_t\vert{} provides a non-trivial step size, but the actual accuracy change still depends on ∇θlog⁡pθ(yt∣zt)\nabla_\theta \log p_\theta(y_t \mid z_t) and how the update direction aligns with the outcome gradient gxg_x.

This can explain why utility becomes smaller at most positions at later training steps in Figure 3, since the absolute teacher-student residuals get smaller overall. Additionally, this relationship can be supported by the following evidence:

  • Full-response interventions (Figure 2): Positions with small ∣ut∣\vert{}u_t\vert{} consistently have near-zero utility, whereas positions with large residuals (∣ut∣≥0.2\vert{}u_t\vert{} \ge 0.2) produce near-zero effects at most tokens alongside a few large positive (+12.87%) or negative (-5.37%) changes.

  • Evolution across checkpoints (Figure 3): At every checkpoint, Figure 3 already restricts evaluation to the 10 positions with the largest ∣ut∣\vert{}u_t\vert{} per response. Even within this high-∣ut∣\vert{}u_t\vert{} subset, most utilities remain clustered near zero (with the interquartile range narrowing slightly as training progresses), while large positive and negative outliers persist throughout training.

  • Real response audit (our response to W3): As illustrated before, large teacher–student residuals can correspond to fixing an actual error (helpful), encouraging trial-and-error (unhelpful), or penalizing a valid answer format (harmful).

Q5: What thresholds were used for Figure 3 in Section 3.3?

We did not apply a hard threshold on parameter-update norm or utility. Instead, for each response we simply selected the 10 token positions with the largest ∣ut∣\vert{}u_t\vert{}. Because tokens with small ∣ut∣\vert{}u_t\vert{} induce negligible updates and near-zero utility (as discussed in Q4), selecting the top-10 ∣ut∣\vert{}u_t\vert{} positions was for focusing the intervention budget on non-trivial updates and keeping Figure 3 readable without being dominated by near-zero points. In Section 3.3, the phrase "excluding positions with negligible parameter updates and near-zero utility" was intended to explain the rationale for taking the top-10 ∣ut∣\vert{}u_t\vert{} positions rather than describing a separate filtering step. We will revise this sentence to avoid ambiguity.

Q7: Does single-prompt utility transfer across prompts?

To test cross-prompt transfer, we extended our 16-prompt setup (N=64N=64, BF16 Richardson FD): we applied a single-prompt AdamW update on 1 source prompt and evaluated accuracy changes across the other 15 prompts (16×15=24016 \times 15 = 240 directed source–target pairs):

Single-step update methodSame-prompt utility (pp)Cross-prompt utility (pp)Cross-prompt diff. vs. Unweighted (pp)Cross-prompt 95% CI
Unweighted OPD+12.94+9.30——
ALIGN-OPD+25.25+9.51+0.21[−1.67, +2.08]

While ALIGN-OPD substantially boosts same-prompt utility (+25.25 vs. +12.94 pp), it shows no clear advantage over vanilla OPD on single-step cross-prompt utility (+9.51 vs. +9.30 pp). This is likely because the evaluation covers only 15 other prompts, which may simply require behavioral improvements unrelated to the current source prompt. More broadly, ALIGN-OPD does not explicitly target cross-prompt utility by definition; instead, generalizable gains emerge over full training, as the model sees a wide range of behaviors and suppressing harmful updates across diverse prompt batches accumulates over time. Extending our formulation to directly capture cross-prompt utility is a promising direction for future work, and we will add this experiment and discussion to the revision.

Reviewer_Jkp1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors provide an approach (ALIGN-OPD) to measure how much each token’s teacher feedback actually helps by applying that token’s update alone and checking whether the student solves the same prompt often. The method shows tha this effect equals how well the token’s update aligns with an outcome-based policy gradient.

It uses this alignment score to downweight teacher feedback that points against task success.

The main idea comes from Armandpour et al. (2026), "Unmasking On-Policy Distillation", which already scores each token's distillation gradient by its alignment with the success-probability gradient. ALIGN-OPD is essentially that score turned into a training weight, combined with gradient-similarity loss weighting (e.g. Du et al. (2018), "Adapting Auxiliary Losses Using Gradient Similarity", not cited), so its novelty is limited to the intervention study and the cheap score estimator.

Critical strengths
  • The single-token intervention study is careful and informative. It shows that most teacher updates barely change accuracy, while a few have large positive or negative effects. It also shows that large teacher–student disagreement alone does not predict whether an update helps.

  • The score is practical to compute during training. It reuses the distillation rollouts and needs only one probe gradient and four extra forward passes per fold, instead of a gradient for every token.

Critical weaknesses
  1. Limited novelty. The main idea comes from Armandpour et al. (2026), "Unmasking On-Policy Distillation", which already scores each token's distillation gradient by its alignment with the success-probability gradient. ALIGN-OPD turns that score into a training weight using gradient-similarity loss weighting (Du et al. (2018), "Adapting Auxiliary Losses Using Gradient Similarity", not cited).

  2. Some equations do not follow:

  • Eq. (13): The text says negative and near-zero feedback is suppressed. With the chosen α=3, τ=1, the weight is 0.95 at s=0 and 0.5 at s=−1, so this feedback is kept.
  • Eq. (11)–(13): The score is not normalized, and Figure 4 shows values from −120 to 240. A fixed τ=1 is scale-dependent, so the sigmoid acts almost as a hard sign gate.
  1. Regarding the experiments:
  • The score is validated on only two responses with 30 tokens each, and the score actually used in training (16–32 probes, finite differences) is never validated.
  • All results are single-seed with no error bars, and gains of 1–2 AIME points are within noise.
  • There is no analysis of which tokens get downweighted during training or why this improves accuracy.
Questions for authors

Please see Weaknesses.

3 comments

We thank the reviewer for the thoughtful assessment and recognition of the intervention study and practical estimator. We would like to address their concerns one by one below.

Relationship to Unmasking OPD and gradient-similarity weighting

We would like to clarify that Unmasking OPD and ALIGN-OPD evaluate different effects, not the same score. Unmasking OPD computes cosine similarity between the distillation and ideal success gradients with respect to next-token logits at a fixed prefix (Armandpour et al., 2026, Eqs. 2.3 and 2.12). It asks whether teacher feedback shifts the next-token distribution toward more successful continuations from that prefix.

ALIGN-OPD instead asks whether actually updating the student's shared parameters using one token's feedback improves its probability of solving the entire prompt again. We measure this effect directly and derive its first-order approximation:

Δt=Vx(θ+ηdtOPD)−Vx(θ)≈η⟨∇θVx,dtOPD⟩.\Delta_t = V_x(\theta+\eta d_t^{\mathrm{OPD}})-V_x(\theta) \approx \eta\left\langle\nabla_\theta V_x, d_t^{\mathrm{OPD}}\right\rangle.

Because shared parameters couple positions and prefixes, a locally beneficial next-token shift can still induce a harmful parameter update, so the two criteria need not agree. Crucially, this difference in the score definition itself is what makes ALIGN-OPD practical during training. Because Unmasking OPD defines alignment in next-token logit space at each individual prefix, estimating its ideal gradient requires branch-conditioned success rates from intermediate prefixes. This means many targeted continuations per branching token, which is why the authors note themselves the cost makes their approach more suitable for offline diagnostics. In contrast, ALIGN-OPD instead scores all tokens against a single shared prompt-level gradient ∇θVx\nabla_\theta V_x estimated from complete responses, naturally avoiding per-token branching rollouts so we can reuse existing on-policy responses and score all tokens together via directional finite differences.

We will also cite Du et al. (2018): unlike weighting a separate auxiliary loss, our focus is on token-level parameter-update utility in OPD and its practical estimation during training. We have clarified these distinctions in the revised manuscript.

Weighting rule and score scale

The weighting rule deliberately retains some mildly negative feedback while nearly eliminating strongly negative feedback:

\qquad w(-1)=0.5, \qquad w(0)\approx0.95.$$ A score predicts utility for the current prompt; a mildly negative score does not rule out benefits on other prompts. We therefore downweight such feedback rather than discard it entirely. The statement that near-zero feedback is suppressed was inaccurate and has been corrected. Figure 4 selects positions with the largest absolute teacher–student residuals, focusing on the strongest feedback that impacts the student. Most training-token scores instead lie approximately between −3 and 3. The rule has a graded transition: for example, a score of −1 retains half the feedback. We did not extensively tune this sigmoid mapping or claim it is optimal. Our focus is estimating feedback utility; gains with this simple mapping demonstrate practical usefulness without extensive tuning. Better score-to-weight mappings remain possible. We intentionally do not normalize scores separately within each response. Responses can differ in their overall alignment strength; per-response normalization would remove magnitude differences that we want the weights to preserve.

Validation of the training-time score

We agree that Figure 4 alone does not validate the practical estimator: it uses 1,024 probes and direct token gradients, whereas training uses 16-32 probes and finite differences. Its coverage is limited because each token intervention requires 4,096 paired evaluation rollouts.

We therefore added an update-level validation on 16 prompts, testing whether cross-fitted BF16 finite-difference scores improve actual parameter updates across different rollout budgets.

For each prompt, we varied the total number of training responses N and split them into two equal folds. Each fold's outcome gradient was used to score the other fold through the training-time BF16 Richardson FD procedure, giving N/2 probes per score estimate. At each N, weighted and unweighted OPD used identical training responses and independently took one AdamW step from the same initial Qwen3-4B checkpoint and fresh optimizer state. The prompts were selected to have mixed outcomes in both folds at every N, so no responses were dropped.

We measured same-prompt success before and after each update using 100 paired evaluation rollouts per prompt. The table reports mean one-step utility across the 16 prompts, in percentage points:

Total responses NProbes per foldUnweighted OPDBF16 FD-weighted OPDGain from weighting
16813.62523.125+9.500
643212.93825.250+12.313
25612811.68833.625+21.938
1,02451211.56338.188+26.625

Weighting improves observed utility at every tested budget. In particular, 32 probes per fold—the training-time setting—yield an additional 12.31 pp over unweighted OPD. This directly tests the usefulness of practical scores beyond the two illustrative responses in Figure 4.

Larger N yields greater same-prompt utility here, but introduces a prompt-diversity trade-off during training. Under a fixed total response budget, increasing N from 64 to 1,024 leaves room for 16 times fewer prompts. We therefore use smaller N to balance useful within-prompt feedback against broader prompt coverage, rather than maximize the one-step gain on each individual prompt.

Training-seed and evaluation variability

For AIME24/25, we use avg@16 for SmolLM3 and avg@32 for Qwen3, corresponding to 16 and 32 sampled answers per problem. This relatively large number of evaluations per problem reduces generation-level sampling noise.

To further validate the training improvement, within the available rebuttal time, we completed a three-seed comparison of ALIGN-OPD and all five baselines on SmolLM3 instruction following. Results below report the average of IFEval and IFBench strict prompt-level accuracy, evaluated using avg@4 with mean ± standard deviation across three different seeds.

MethodIF accuracy: mean ± SD
Vanilla OPD48.61 ± 0.96
OPD + residual clipping48.67 ± 0.18
OPD + GRPO49.22 ± 0.01
SG-OPD49.49 ± 0.15
TIP48.94 ± 0.68
ALIGN-OPD50.01 ± 1.16

ALIGN-OPD still achieves the highest mean accuracy, improving by 1.39 percentage points over vanilla OPD and 0.52 points over SG-OPD, the strongest baseline. It also outperforms vanilla OPD in each of the three seeds.

Which token updates are downweighted, and why?

Figure 3 labels tokens with extreme positive and negative measured utility; notably, “Wait” appears at both ends and hence kept sometimes and downweighted at other times. One possible explanation is that updates involving such tokens alter reconsideration: additional reasoning can correct mistakes, but unnecessary continuation on otherwise successful solutions can lead to truncation before the final answer. We will add this discussion.

We agree that the existing analysis was insufficient. We have therefore added new experiments inspecting real responses from our our 16-prompt Qwen3-4B audit. We discuss some interesting findings below, and will incorporate them in our revised manuscript.

In our 16-prompt analysis, about 6% of tokens are strongly downweighted (wt≤0.1w_t \le 0.1), at similar rates in correct and incorrect responses. Interestingly, over 70% of these downweighted updates are cases where the teacher tries to decrease a token's probability (ut<0u_t < 0), but the outcome direction favors keeping it. Three examples from real responses illustrate how this works in practice (target token marked with ⟦·⟧):

  • 1. Blocking the teacher from penalizing a correct answer token (wt=0w_t = 0). When counting non-isomorphic well-orderings of N\mathbb{N}, the student outputs the correct answer \boxed{\aleph_⟦1⟧}. Because the teacher prefers alternative phrasing or formatting at this prefix, it assigns a strong negative residual to 1, whereas the outcome gradient strongly supports it. ALIGN-OPD sets wt=0w_t = 0 to block this harmful suppression (and on this prompt, weighted OPD improves one-step accuracy from 19% to 92%). In contrast, when \aleph_⟦1⟧ appears again a few tokens later in the explanation text, the teacher agrees with the student (ut=0u_t = 0) and wt≈0.95w_t \approx 0.95.
... Therefore, the number of **non-isomorphic well-orderings of $\mathbb{N}$** is **uncountably infinite**.

### ✅ Final Answer:
$$
\boxed{\aleph_⟦1⟧}   <-- [u_t < 0, outcome opposes teacher -> w_t = 0 (downweighted)]
$$
(i.e., the cardinality of the set of non-isomorphic well-orderings of $\mathbb{N}$ is the cardinality of the first uncountable cardinal, $\aleph_⟦1⟧$, assuming the Axiom of Choice.)<|im_end|>   <-- [u_t = 0 -> w_t = 0.95 (retained)]

  • 2. Dampening unproductive trial-and-error (wt≈0w_t \approx 0). On a functional-equation prompt, the student guesses g(x)=xg(x)=x, sees it fails, and writes "So ⟦that⟧ doesn't work. Try g(x) = x^2." Here the teacher reinforces the guess-and-test transition (that, ut>0u_t > 0), while the outcome direction opposes imitating this trial-and-error branch, so ALIGN-OPD suppresses the update.
... If we try $g(x) = x$, then:
$$
g(xy) + g(x(1-y)) = xy + x(1-y) = x \neq xy \quad (\text{unless } y = \frac{1}{2})
$$
So ⟦that⟧ doesn't work.

Try $g(x) = x^2$. Then: ...
  • 3. Contrast-retaining teacher penalties on actual errors (wt=1w_t = 1). Unlike Case 1, when the teacher penalizes an actual mathematical mistake, ALIGN-OPD keeps the update. On a binary-tree recursion prompt where the base case is D1=1D_1 = 1, a student mistakenly writes "D_1 = ⟦0⟧". Here both the teacher and the outcome direction push 0 down, yielding wt=1w_t = 1 and preserving the corrective signal.
... A tree with 1 inner vertex (just the root) has a left subtree with 0 inner vertices, which is valid (since 0 is even).
So we must define:
$$
D_1 = ⟦0⟧ \quad (\text{since the root has a left tree of } 0 \dots)
$$

[1] Armandpour et al. (2026). Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why. arXiv:2605.10889.

[2] Du et al. (2018). Adapting Auxiliary Losses Using Gradient Similarity. arXiv:1812.02224.

Reviewer_cjokOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new method, ALIGN-OPD, based on their initial work on applying OPD updates to individual to individual tokens to see which have the largest impact (positive or negative) on the final answer. They then build a predictor based on these scores, to create their new OPD weighting scheme. They find small gains on known models on math and code when running their method vs previous methods.

Critical strengths

The paper's proposal to look at token-level importance and weight accordingly seems smart and intuitive, and their method for measuring it seems well designed. Their finding that some teacher updates are useful and some are harmful is also interesting. On their benchmarks, their method is also better on nearly all of them vs vanilla OPD.

Critical weaknesses

The experimental settings are under-developed, and there are some inconsistencies in the paper.

Experimental:

  • The Pearson score comes from just two responses with 30 tokens, and the text says 3 checkpoints while the figure shows 2. Most of the Pearson is also concentrated in a few extreme tokens, and the rest are near zero
  • The actual difference step is near bf16's precision limit, so it's not clear that's doing anything?
  • Everything is from a single run with no seed or other hyperparameter changes (LR, etc), so a difference of ~1 point is fully within noise. The evaluations themselves, like AIME, when evaluated with a temperature above 0, have a noise floor per run of more than the increase shown from the method

Section 4.2 says prompt screening was used, but 5.2 says it wasn't? And the normalizer Z is defined differently in equations 14 and 18

Questions for authors

None

2 comments

We thank the reviewer for the careful assessment and constructive feedback. We have therefore conducted additional experiments and corrected the inconsistencies identified in the manuscript.


Coverage and robustness of the correlation analysis

“Three checkpoints” was an error and has been corrected to two. Figure 4 evaluates 30 token positions per response, with each intervention requiring 4,096 paired evaluation rollouts to estimate utility with meaningful confidence intervals. This cost constrained the analysis to two responses.

We agree that extreme effects contribute strongly to Pearson correlation. Accurately identifying these effects is nevertheless important: these updates have the largest beneficial or harmful impact on task success and are particularly valuable to retain or suppress. The Spearman correlations of 0.839 and 0.706 additionally indicate substantial rank agreement within the evaluated responses, although near-zero effects are harder to distinguish under finite evaluation rollouts, which can reduce the reliability of the measured rankings. We acknowledge that these examples do not establish comprehensive validation across prompts. The additional experiments below directly examine numerical accuracy and the usefulness of practical weighting.

Numerical accuracy of BF16 finite differences

We thank the reviewer for noticing this nuance and raising the concern. We use BF16 finite-difference forward passes for its higher computational efficiency at the cost of accuracy. To assess whether this accuracy-efficiency trade-off makes sense, we conducted two complementary experiments:

Experiment 1: Does BF16 FD recover the intended score?

Using the initial Qwen3-4B checkpoint across 16 prompts (N=64N=64, Richardson step h=10−5h=10^{-5}, wt=σ(3(st+1))w_t=\sigma(3(s_t+1))), we evaluated BF16 and FP32 FD against exact FP32 autograd over 4,096 randomly sampled tokens (plus 1,024 diagnostic tokens). Averaged across the 16 prompts:

Metric (vs. exact FP32 autograd)BF16 FDFP32 FD
Score relative L2L_2 error49.18%2.34%
Score cosine similarity0.88670.9996
Score Spearman correlation0.68670.9798
Weight MAE0.031350.000948
Tokens with weight error >0.1> 0.16.03%0.17%
FD time / prompt (perturbation + 4 forwards)17.5 s81.7 s (4.7×)
Total scoring time / prompt (incl. BF16 probe grad)48.5 s107.1 s (2.2×)

BF16 FD does introduce non-negligible numerical error compared to FP32 FD, which closely tracks exact autograd. However, BF16 FD is far from pure noise: it retains strong directional (cos⁡=0.89\cos = 0.89) and rank (ρ=0.69\rho = 0.69) agreement with the true score, and the sigmoid mapping compresses most discrepancies into a small weight MAE (0.0310.031, with only 6.03%6.03\% of tokens shifting by >0.1>0.1). In our current implementation, where BF16 benefits from FlashAttention and FP32 can only use native SDPA, BF16 reduces the FD forward cost by 4.7×4.7\times and total scoring time by 2.2×2.2\times (excluding rollout generation and optimizer steps).

Experiment 2: Do the resulting weights improve an actual model update?

We selected 16 prompts that contain correct and incorrect responses in both folds. This eliminates fold dropping as an explanation for differences between methods.

For each prompt and total response count N, all three methods used the same training responses and independently performed one AdamW update from the same initial Qwen3-4B checkpoint and fresh optimizer state. Unweighted (vanilla) OPD assigned all tokens weight 1. The two weighted variants split the responses into equal folds, used each fold’s outcome gradient to score the other fold, and differed only in the precision of the FD forward passes. Each probe therefore contained N/2 responses. This diagnostic used the same FD step size and optimization settings across variants.

For each prompt, we generated 100 fresh evaluation responses before and after each update using paired seeds. The table reports the change in same-prompt success rate, averaged equally over the 16 prompts, in percentage points.

Total responses NUnweightedBF16 FDFP32 FDBF16 − UnweightedFP32 − Unweighted
1613.62523.12523.000+9.500+9.375
6412.93825.25025.625+12.313+12.688
25611.68833.62536.250+21.938+24.563
1,02411.56338.18840.625+26.625+29.063

Even with BF16 precision noise, BF16 FD improves substantially over unweighted OPD at every budget and closely tracks FP32 at training-time sizes (N=16,64N=16, 64). FP32 only opens a ~2.5 pp gap at larger NN.

We have added these new analyses to our revised manuscript.

Reliability across training seeds and evaluation samples.

For AIME24/25, we use avg@16 for SmolLM3 and avg@32 for Qwen3, corresponding to 16 and 32 sampled answers per problem. This relatively large number of evaluations per problem reduces generation-level sampling noise.

To further validate the training improvement, within the available rebuttal time, we completed a three-seed comparison of ALIGN-OPD and all five baselines on SmolLM3 instruction following. Results below report the average of IFEval and IFBench strict prompt-level accuracy, evaluated using avg@4 with mean ± standard deviation across three different seeds.

MethodIF accuracy: mean ± SD
Vanilla OPD48.61 ± 0.96
OPD + residual clipping48.67 ± 0.18
OPD + GRPO49.22 ± 0.01
SG-OPD49.49 ± 0.15
TIP48.94 ± 0.68
ALIGN-OPD50.01 ± 1.16

ALIGN-OPD still achieves the highest mean accuracy, improving by 1.39 percentage points over vanilla OPD and 0.52 points over SG-OPD, the strongest baseline. It also outperforms vanilla OPD in each of the three seeds.

Prompt screening and loss normalization

We apologize for the inconsistent descriptions. Prompt screening was used for Qwen experiments, where many easy prompts produced uniformly correct probe responses, but not for SmolLM3. Within each Qwen setting, all compared methods used the same screened prompt pool. We have clarified this distinction.

Across all experiments, the loss was averaged first over valid tokens within each response, then over retained responses. Eq. (18) incorrectly described a global token-mean reduction, while Eq. (14) omitted the within-response length normalization. We have corrected both descriptions. The corresponding ALIGN-OPD objective is

LALIGN(θ)=−1Z∑j∈R1Tj∑t∈valid(j)sg(wj,tuj,t)ℓj,t(θ),Z=∣R∣.\mathcal{L}_{\mathrm{ALIGN}}(\theta) = -\frac{1}{Z}\sum_{j\in\mathcal{R}}\frac{1}{T_j}\sum_{t\in\mathrm{valid}(j)}\mathrm{sg}(w_{j,t}u_{j,t})\ell_{j,t}(\theta), \quad Z = \lvert\mathcal{R}\rvert.

Here, RR denotes the retained responses, and TjT_j is the number of valid tokens in response j.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We thank all three reviewers for their thoughtful feedback and constructive suggestions. We are encouraged that all reviewers recognized the importance of understanding when teacher feedback helps or harms the student, and found our token-level intervention analysis interesting and informative.

Our central contribution is showing how to estimate whether each token's OPD update helps or hurts task success and using that estimate to reweight teacher feedback during training. Unlike prefix-level offline diagnostics such as Unmasking OPD, ALIGN-OPD formulates utility in parameter space against a single shared prompt-level outcome gradient. This avoids per-token branching rollouts and reuses existing on-policy responses directly during training, adding only ~33% wall-clock overhead over vanilla OPD.

In response to the reviewers, we substantially strengthened the empirical evidence and revised the manuscript:

  • Training-time estimator: Validated BF16 Richardson finite differences against exact FP32 autograd and demonstrated that the resulting weights consistently improve actual parameter updates at training-time probe budgets (N=64N=64 instead of 1,0241,024).
  • Robustness: Added a three-seed comparison against all five baselines, where ALIGN-OPD retains the highest mean accuracy and outperforms vanilla OPD on every seed.
  • Interpretability and generalization: Analyzed which token updates are downweighted with concrete examples, quantified outcome-gradient noise and fold dropping, and examined cross-prompt utility.
  • Clarity and consistency: Corrected all reported text and notation inconsistencies (including loss normalization, checkpoint counts, prompt screening scope, and sigmoid wording) and clarified the two-fold procedure and training overhead.

We hope these experiments and revisions resolve the reviewers' main concerns, and this summary is helpful for the assessment. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.