acceptodds
Under review as a conference paper at ICLR 2027

ACT:Selective Temporal Feedback for On-Policy Distillation

Abstract

On-policy distillation (OPD) provides dense token-level supervision on student-generated trajectories, yet dense supervision can still be poorly allocated. Two positions with similar local teacher–student disagreement may have different trajectory contexts: the disagreement may quickly dissipate at one position but persist into the future at another. Exploiting this temporal context requires distinguishing where to intervene from what feedback to apply: position selection should preserve disagreement before sign cancellation, whereas feedback construction should retain its direction and net effect. Based on this distinction, we introduce Advantage Credit Transfer (ACT), which decouples position selection from feedback construction. Over a short future window, ACT uses unsigned future mismatch to select a sparse set of positions and mixes signed future feedback into the shaped directional coefficients at those positions, leaving all other shaped directional coefficients unchanged. ACT reuses the teacher and student log probabilities already computed by OPD, requiring no additional teacher queries, trajectory sampling, or value network. Controlled experiments show that future-mismatch selection outperforms alternative selectors at equal coverage, signed future feedback outperforms unsigned replacements under a fixed gate, and selective correction outperforms uniform allocation under matched total correction magnitude. With a 1.5B student, ACT achieves the highest macro average among the evaluated methods across five benchmarks spanning mathematical, coding, and scientific reasoning, under both math-domain and multi-domain distillation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, large language models, selective temporal feedback, token-level credit assignment, knowledge distillation, reasoning models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_eaScOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

On-policy distillation provides dense token-level feedback to student generations but assigning this feedback across the generation across tokens can be tricky. Uniform allocation is bad because not all tokens carry the same importance for a sequence and not all tokens differ between student and teacher similarly. So making OPD better or more efficient is down to figuring out which tokens to grade and how to grade them. ACT uses a known technique of using a future window for grading tokens but presents a novel grading method that decouples position selection from feedback construction. The position selection preserves disagreement before the sign cancellation and feedback retains it's direction and net effect. ACT reuses the teacher and student logprobs already computed for OPD (by computing shaped directional, unsigned future Mismatch and signed future feedback). Controlled experiments justify the effect of each design choice and the overall methods provides a clear win in the models and settings tested.

Critical strengths

To start with, the core idea is developed with a scientific mindset and the paper is very clearly written which makes it easy to follow.

Explanation of why later feedback can act on earlier actions is well-explained and there are clear ablations on this.

ACT separate two decisions: where to provide feedback and what feedback to provide. ACT develops a statistic for each and motivates what each statistic M_t and F_t measures and how M_t-|F_t|, the cancellation gap, leads to the behaviours observed in practice (teacher alternating between supporting and suppressing student action in a window) which can sometimes be non-intuitive. Unsigned future mismatch M_t defines where to intervene and signed future feedback F_t defines how to intervene (constructs the feedback to apply at that point)

There is evidence presented in the paper in the form of experiments that future information contributes to both position selection and feedback aggregation.

The core idea is well-motivated idea that has been thoughtfully developed and clearly articulated in the paper. Each concept is justified and explained and then ablated cleanly, making the paper a good piece of scientific exploration.

Experiments or justifications for feedback control, gate control and local vs future mismatch are clearly presented.

Design choices are nicely ablated and experimental results are convincing even if not exhaustive.

Critical weaknesses

Some of the visualizations are a bit misleading. Figure 1 done as a table instead of a bar chart (given the axes are clipped) would red differently (results would not look as impressive), however this is a cosmetic issue.

Figure 3 is again not easy to interpret. Would benefit from a longer caption explaining the colouring and blocking.

There isn't substantial evidence that the method scales to larger student/teacher sizes (ack that compute availability could be a blocker)

Questions for authors

Can you comment (not necessarily do new experiments) on how the results will scale to sparser feedback situations and larger student model sizes, especially multi-modal ones where the tokens could be either a few characters or an image patch etc?

Reviewer_x1zPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OPD obtains dense supervision using teacher-student log-ratio. While two positions might exhibit the same teacher-student disagreement, in one case the disagreement can fade away quickly, and it can persist in the other. This paper introduces Advantage Credit Transfer (ACT), which uses the short H horizon future tokens along with a discount factor to decide which positions to apply correction to and what correction update to apply.

Critical strengths
  • The perspective on purely local updates being myopic to subsequent trajectory changes is insightful. ACT introduces two statistics: i) Unsigned future mismatch to identify token positions to apply correction and ii) Signed future feedback, which takes into account a local horizon of subsequent tokens, to determine the correction update. Ablation studies (figure 4) validate this design choice.

  • ACT reuses teacher-student log-ratio already computed by OPD and does not require additional teacher queries, rollouts or an auxiliary value network.

Critical weaknesses
  • The gains over baselines in Table 1 (except AIME 2024, which has 30 samples) are marginal. The lack of standard deviation and multi-seed training runs make it hard to assess if the gains are within noise. Additionally, ACT-Directional, which isolates the temporal component scores very close to ACT (except AIME 2024). ACT-Directional is not reported in Table 2.

  • On Multi-domain distillation, OPD average score is lower than the student model. The hyper-parameters in ACT (\rho, H, \beta) were subjected to extensive tuning. Baselines do not appear to be treated with the same rigor.

Questions for authors
  1. The introduction section mentions TOPD, another recent and relevant baseline that also uses near future trajectory for guiding OPD update.
  2. Did the authors also try a different student-teacher pair (e.g. Qwen 3 series used in prior work)?
1 comment

I thank the authors for their response and new experiments. Majority of my concerns are addressed with the new experiments. I have updated my rating to 3.

Reviewer_9jqDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Advantage Credit Transfer (ACT), which utilizes teacher-student disagreement in future trajectories in addition to local token-level feedback in On-Policy Distillation (OPD). ACT separates position selection from feedback construction, selecting a subset of positions based on unsigned mismatch computed as a discounted weighted average of future token-level disagreement magnitudes. At the selected positions, signed future feedback is mixed with the original token-level coefficients for training.

Critical strengths

S1. The high level approach of considering the downstream effects or future influence of a token, rather than relying solely on local teacher-student disagreement, is a reasonable direction for improving OPD.

S2. ACT provides a computationally efficient way to incorporate temporal information into existing OPD token-level supervision without requiring additional teacher queries or rollouts.

Critical weaknesses

W1. The paper emphasizes that local OPD does not consider future information, but does not clearly demonstrate what problems this causes. The fact that future information differs from local feedback alone does not justify the need to incorporate it. The lack of direct analysis of what meaningful supervision local OPD misses and how ACT addresses this makes the motivation for the proposed research question unclear.

W2. ACT computes future mismatch MtM_t based on teacher-student log probability differences over future tokens and treats positions with high MtM_t as important targets for correction. However, the validity of this selection criterion is not clearly examined. Since the teacher is conditioned on student-generated prefixes, disagreement can remain small even after an incorrect reasoning step if both the teacher and student continue following the same assumption. Moreover, there is insufficient evidence that high future disagreement indicates reasoning errors or important positions requiring correction, or that it identifies such positions better than local disagreement. A more direct validation would be to examine the correlation between MtM_t and incorrect reasoning steps identified by an independent LLM judge, or between MtM_t and improvements in accuracy following teacher interventions at the selected positions.

W3 One of ACT's key motivations, the sign cancellation problem in signed disagreement and the use of unsigned discrepancy to address it, has already been discussed in IW-OPD [1]. IW-OPD also makes a conceptual distinction between using unsigned discrepancy for supervision allocation and signed local feedback for updates. Therefore, the paper should more clearly explain how ACT differs from IW-OPD and clarify its contributions relative to this prior work.

[1] On the Position Bias of On-Policy Distillation, https://arxiv.org/abs/2606.22600

Questions for authors

Please see the weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.