- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether teacher feedback in on-policy distillation (OPD) always improves the student's task performance. They measure the utility of individual token updates by applying each parameter update separately and evaluating its effect on the student's probability of answering questions correctly. They then derive a first-order approximation based on alignment between the token's OPD update and the gradient of expected task success. Using this score, they reweight teacher feedback during training and experiments on SmolLM3 and Qwen3 across instruction following, code, and math show improvements over vanilla OPD and several competing baselines.
Critical strengths
-
generally i think the paper is focusing on an interesting research question: whether learning from a teacher signal actually improves the student's ability to solve a problem, and it proposes a direct way to evaluate that in a token-wise manner.
-
it also provides a way to connect teacher-induced token-wise updates with task success, though subject to a certain amount of computational effort, and it has been shown to gain performance improvements across different domains and two model backbones
Critical weaknesses
-
the utility scores used during actual training seem not to be the same as those used to demonstrate the strong correlation between estimated token utility and the actual change in answer correctness after applying an individual token update, as shown in fig 4 and in sec 4.1. It thus raises a concern about whether the training-time estimate is still reliable under this approximation.
-
the additional computational (training) cost (e.g., estimating outcome gradients and computing token-level updates) induced by this method makes it difficult to assess relative to the modest performance gains (Vanilla OPD with residual clipping seems to also work comparably well). Can the authors provide a clear measurement of the actual training-time overhead introduced by the practical score estimation? This matters since the performance gains are relatively small (see Tables 1 and 2), so it would be informative to see such a report of extra cost.
-
i find the token-level intervention analysis interesting; however, the observation that only some token positions have a large effect on task outcome is not entirely surprising given earlier findings on token importance in reasoning and opd. I think a systematic investigation of what distinguishes tokens with large positive utility, large negative utility, and near-zero utility would make this paper more informative. For example, are harmful updates concentrated in particular types of tokens? How do the distributions of positive and negative utility differ between correct and incorrect responses? Are the patterns related to token position?
-
The presentation of the core training procedure and some math details could be made clearer. E.g., I found sec 4.2 is a bit difficult to follow; for instance, why is twofold scoring used? Also, this paper uses terms that may have ambiguity, like gradient: outcome gradient, outcome policy gradient, and token OPD gradient, etc. They may have different roles in different contexts, so their use in the text should avoid any ambiguity. Also, Eq. 2 and eq 1 use different subscripts with no explanation. Also, this paper states that negative and near-zero feedback is suppressed. However, the implemented weighting function (w_t=\sigma(3(\hat s_t+1))) assigns approximately 0.95 to a token with zero alignment score. Can the authors clarify this?
Questions for authors
-
Since residual clipping seems to work comparably well, can the authors clarify what exactly the clipping is doing, how it differs, and how low the practical cost of using clipping is compared to that of the proposed methods?
-
Can the authors explain why a response should be scored using the opposite fold in sec 4.2? The procedure here needs to be described more clearly
-
How noisy could the estimate of outcome gradients be? especially when nearly all responses are correct or incorrect
-
can the authors show how the distribution of absolute teacher–student residuals evolves alongside utility in Fig3, to examine whether their changes follow similar trends during training?
-
sec 3.3 states that positions with negligible parameter updates and near-zero utility are excluded, so what are the thresholds?
-
eq 13 assigns zero weight to a target fold when the opposite fold is all correct or incorrect, but it may imply we are using less amount of training data; could the authors report the percentage of dropped folds across training? furthermore, it would be useful to compare against vanilla OPD trained with the same fold-selection mechanism, but without the token-level weights applied, in order to distinguish improvements from selective removal of training examples.
-
since the causal utility in eq 7 is defined wrt one prompt, whether the authors can do a small cross-prompt analysis, in which we measure the effects of high- and low-utility updates on held-out prompts, and this could help whether the causal utility is primarily prompt-specific or not?
4 comments
W3: Characteristics of helpful and harmful token updates
We agree that a closer look at what distinguishes helpful and harmful updates makes the analysis much more informative. We therefore expanded our investigation on Figure 3 and the new 16-prompt Qwen3-4B audit ( responses, mentioned in our response to W1). We will incorporate these findings into the revised manuscript:
-
Utility may not be fixed by token type: In Figure 3, the same token
"Wait"appears at both the positive and negative extremes. We found that this happens because parameter updates alter broader generation habits rather than just local semantics: encouraging reconsideration tokens can help the student recover from mistakes, whereas, in some other cases, triggering extra continuation on an already-correct trajectory can cause length truncation before the final answer is reached. -
Correct vs. incorrect responses & update direction: Across our 16-prompt audit, about 6% of tokens are strongly downweighted (), at similar rates in both correct and incorrect responses. Over 70% of these downweighted updates are cases where the teacher tries to decrease a token's probability (), but the outcome gradient favors keeping it.
Below, we provide three examples from real responses illustrate how token position, trajectory context, and error type may cause these weights in practice (the studied token is marked with ⟦·⟧):
1. Blocking the teacher from penalizing a correct answer token (). When counting non-isomorphic well-orderings of , the student outputs the correct answer \boxed{\aleph_⟦1⟧}. Because the teacher may prefer alternative phrasing or formatting at this prefix, it assigns a strong negative residual to 1, whereas the outcome gradient strongly supports it. ALIGN-OPD sets to block this harmful suppression (on this prompt, one-step update improves accuracy from 19% to 92%). Notably, position matters even for the exact same token: when \aleph_⟦1⟧ appears again just a few tokens later in the post-answer explanation, the teacher agrees with the student () and the token is retained ().
... Therefore, the number of **non-isomorphic well-orderings of $\mathbb{N}$** is **uncountably infinite**.
### ✅ Final Answer:
$$
\boxed{\aleph_⟦1⟧} <-- [u_t < 0, outcome opposes teacher -> w_t = 0 (downweighted)]
$$
(i.e., the cardinality of the set of non-isomorphic well-orderings of $\mathbb{N}$ is the cardinality of the first uncountable cardinal, $\aleph_⟦1⟧$, assuming the Axiom of Choice.)<|im_end|> <-- [u_t = 0 -> w_t = 0.95 (retained)]
2. Dampening unproductive trial-and-error (). On a functional-equation prompt, the student guesses , sees it fails, and writes "So ⟦that⟧ doesn't work. Try g(x) = x^2." Here the teacher reinforces the guess-and-test transition (that, ), while the outcome direction opposes this, possibly because the outcome gradient supports a more efficient method than this trial-and-error way.
... If we try $g(x) = x$, then:
$$
g(xy) + g(x(1-y)) = xy + x(1-y) = x \neq xy \quad (\text{unless } y = \frac{1}{2})
$$
So ⟦that⟧ doesn't work.
Try $g(x) = x^2$. Then: ...
3. Contrast-retaining teacher penalties on actual errors (). Unlike Case 1, when the teacher penalizes an actual mathematical mistake, ALIGN-OPD keeps the update. On a binary-tree recursion prompt where the base case is , a student mistakenly writes "D_1 = ⟦0⟧". Here both the teacher and the outcome direction push 0 down, yielding and preserving the corrective signal.
... A tree with 1 inner vertex (just the root) has a left subtree with 0 inner vertices, which is valid (since 0 is even).
So we must define:
$$
D_1 = ⟦0⟧ \quad (\text{since the root has a left tree of } 0 \dots)
$$
W4 & Q2: Two-fold scoring, gradient notation, and sigmoid weighting
- Why score with the opposite fold? (W4, Q2): If a target response is included when computing the outcome gradient , scoring against introduces self-influence, as the response's own trajectory contributes to the reference direction. Leave-one-out scoring would avoid this but requires separate outcome-gradient backward passes per prompt. Splitting the responses into two folds ( scores , and scores ) eliminates self-influence while requiring only 2 outcome-gradient computations per prompt. We will revise Section 4.2 to make this motivation and procedure clear.
- Gradient terminology and Eq. (1)–(2) notation (W4): In Equations (1) and (2), and refer to the same reverse-KL objective; we will unify the subscript to and explicitly indicate evaluation at . Throughout the manuscript, we will also standardize our terminology to strictly distinguish three quantities: the population outcome policy gradient , the empirical estimated outcome gradient , and the token OPD update .
- Sigmoid weighting behavior (W4): We agree that our description in Section 4.2 stating that "near-zero feedback is suppressed" was inaccurate. Under our implemented weight function, a score of receives (and at ). The function suppress strongly negative updates while retaining most of the weight on near-zero and positively aligned feedback. We will correct this description in Section 4.2 to match this.
Q3 & Q6: Outcome-gradient noise and homogeneous-fold dropping
How noisy are the outcome-gradient estimates? (Q3) On our 16-prompt Qwen3-4B set, we split the responses of each prompt into two equal folds ( probes per fold) and computed the full-parameter cosine similarity between the two folds' outcome gradients:
| Total rollouts | Probes / fold | Mean | Min / max across 16 prompts |
|---|---|---|---|
| 16 | 8 | 0.015 | -0.044 / 0.093 |
| 64 (training) | 32 | 0.045 | -0.007 / 0.138 |
| 256 | 128 | 0.157 | 0.006 / 0.493 |
| 1,024 | 512 | 0.360 | 0.060 / 0.797 |
- All-correct / all-incorrect folds: When a fold is completely homogeneous, centered rewards vanish () and . When outcomes are heavily skewed (e.g., a prompt with 90% accuracy), the minority outcome has very few samples in a finite fold, potentially yielding noisier directions ( at and at ).
- Large cross-prompt variation, weak link to difficulty: Cosine varies widely across prompts, and correlates weakly with overall prompt accuracy ( at ; at ). Instead, longer responses accumulate more variance ( to with mean token length). For example, at ~52% accuracy, a shorter-response prompt (705 tokens) reaches () and (), whereas a longer-response prompt (2,958 tokens) stays at and .
- Effectiveness despite full-parameter noise: Although full-parameter cosine in a 4B space averages at , our response to W1 shows that projecting token updates onto this 32-probe outcome gradient already improves one-step accuracy by +12.31 pp.
Homogeneous-fold dropping rate and isolation from token weighting (Q6). In our Qwen3-4B Math training, ~15% of folds (weighted by valid tokens) are dropped. While compute constraints during the rebuttal window prevented us from conducting a full set of training runs of vanilla OPD with fold-dropping, two existing experiments directly isolate token-level weighting from fold removal:
- W1 has 0% dropped folds: In the experiments described in our response to W1, all 16 prompts have mixed outcomes in both folds (no responses dropped), yet token weighting alone improves one-step accuracy from 12.94% to 25.25% (+12.31 pp) at .
- Table 3 ablations share the exact same fold dropping: In Section 5.2 (Table 3),
Reverse reweightinguses the exact same homogeneous-fold dropping rule asALIGN-OPD, but inverting the token weights degrades accuracy by 3.91 points.
Q4: How absolute residuals relate to and evolve alongside utility
The residual magnitude plays an important role in causal utility because it directly scales the token update . More specifically:
- When is near zero, utility can hardly be large: A near-zero residual induces a negligible parameter step, keeping close to zero.
- When is large, utility is not necessarily large (or positive): A large provides a non-trivial step size, but the actual accuracy change still depends on and how the update direction aligns with the outcome gradient .
This can explain why utility becomes smaller at most positions at later training steps in Figure 3, since the absolute teacher-student residuals get smaller overall. Additionally, this relationship can be supported by the following evidence:
-
Full-response interventions (Figure 2): Positions with small consistently have near-zero utility, whereas positions with large residuals () produce near-zero effects at most tokens alongside a few large positive (+12.87%) or negative (-5.37%) changes.
-
Evolution across checkpoints (Figure 3): At every checkpoint, Figure 3 already restricts evaluation to the 10 positions with the largest per response. Even within this high- subset, most utilities remain clustered near zero (with the interquartile range narrowing slightly as training progresses), while large positive and negative outliers persist throughout training.
-
Real response audit (our response to W3): As illustrated before, large teacher–student residuals can correspond to fixing an actual error (helpful), encouraging trial-and-error (unhelpful), or penalizing a valid answer format (harmful).
Q5: What thresholds were used for Figure 3 in Section 3.3?
We did not apply a hard threshold on parameter-update norm or utility. Instead, for each response we simply selected the 10 token positions with the largest . Because tokens with small induce negligible updates and near-zero utility (as discussed in Q4), selecting the top-10 positions was for focusing the intervention budget on non-trivial updates and keeping Figure 3 readable without being dominated by near-zero points. In Section 3.3, the phrase "excluding positions with negligible parameter updates and near-zero utility" was intended to explain the rationale for taking the top-10 positions rather than describing a separate filtering step. We will revise this sentence to avoid ambiguity.
Q7: Does single-prompt utility transfer across prompts?
To test cross-prompt transfer, we extended our 16-prompt setup (, BF16 Richardson FD): we applied a single-prompt AdamW update on 1 source prompt and evaluated accuracy changes across the other 15 prompts ( directed source–target pairs):
| Single-step update method | Same-prompt utility (pp) | Cross-prompt utility (pp) | Cross-prompt diff. vs. Unweighted (pp) | Cross-prompt 95% CI |
|---|---|---|---|---|
| Unweighted OPD | +12.94 | +9.30 | — | — |
| ALIGN-OPD | +25.25 | +9.51 | +0.21 | [−1.67, +2.08] |
While ALIGN-OPD substantially boosts same-prompt utility (+25.25 vs. +12.94 pp), it shows no clear advantage over vanilla OPD on single-step cross-prompt utility (+9.51 vs. +9.30 pp). This is likely because the evaluation covers only 15 other prompts, which may simply require behavioral improvements unrelated to the current source prompt. More broadly, ALIGN-OPD does not explicitly target cross-prompt utility by definition; instead, generalizable gains emerge over full training, as the model sees a wide range of behaviors and suppressing harmful updates across diverse prompt batches accumulates over time. Extending our formulation to directly capture cross-prompt utility is a promising direction for future work, and we will add this experiment and discussion to the revision.
We thank the reviewer for the positive assessment of our work and the constructive questions on the practical estimator, training overhead, and token-level dynamics. Because several weaknesses and questions touch on closely related parts of the method, we group them below for brevity while addressing every point directly.
This is a valid and important concern. To directly validate the training-time estimator, we ran an update-level experiment on 16 prompts (Qwen3-4B). For each prompt, we split N training responses into two equal folds and scored each fold using the opposite fold's outcome gradient via our training-time BF16 Richardson FD procedure (N/2 probes per fold, mixed outcomes so no folds were dropped). Starting from the same checkpoint and fresh AdamW state, we compared one optimization step of unweighted OPD against BF16 FD-weighted OPD on the exact same responses, evaluating before/after accuracy with 100 paired rollouts per prompt:
Even with 8–32 probes and BF16 finite differences, weighting consistently improves the actual parameter update (+12.31 pp at our training setting of 32 probes/fold). While larger N yields higher single-prompt gains here, under a fixed rollout budget during training, N=64 vs. 1,024 trades off 16× prompt diversity; we therefore use N=32–64 in training to balance probe quality with prompt coverage.
We profiled wall-clock time on our Qwen training run using 8×H200 GPUs (N=64, 1,024 responses/step):
Because our two-fold split reuses existing training rollouts rather than generating separate probe batches, and directional FD avoids per-token backward passes, the scoring pipeline adds 5.76 min/step (+33% wall-clock time).
How residual clipping differs (Q1). Residual clipping (u~t=clip(ut,−c,c)) is computationally free (an element-wise op), but it only caps the magnitude of extreme teacher–student disagreement ut (and does not bound the factor ∇θlogpθ). As Figures 2 and 3 show, even among tokens with large ∣ut∣, the induced updates can be strongly beneficial, near-zero, or strongly harmful. Consequently, clipping indiscriminately weakens large-residual updates that are actually helpful, while still allowing harmful updates to proceed with capped magnitude c in the wrong direction. ALIGN-OPD instead evaluates whether the update direction dtOPD aligns with improving task success (⟨g^x/∥g^x∥2,dtOPD⟩), retaining beneficial high-residual updates in full while suppressing harmful ones towards zero.