acceptodds
Under review as a conference paper at ICLR 2027

Stochastic Teacher Intervention for Agentic On-Policy Distillation

Abstract

On-policy distillation (OPD) efficiently transfers capabilities from a stronger teacher to a student language model through dense token-level supervision on student-generated rollouts and has shown promise on complex tasks such as mathematical reasoning. However, in multi-turn agentic tasks, student decisions shape subsequent observations, causing early errors to accumulate across turns. The resulting trajectories can drift away from the teacher's rollout distribution, making the teacher's token-level supervision less reliable or even counterproductive for OPD training. To address this issue, we introduce STI-OPD, a stochastic teacher intervention framework for multi-turn agentic OPD. During multi-turn interaction, STI-OPD uses teacher intervention guided by teacher–student policy discrepancy to replace the student's proposed action with a teacher-generated one to maximize the acquisition of reliable supervision. We further develop a stochastic intervention strategy, addressing the limitations of previous threshold-based or fixed-schedule approaches, that estimates policy discrepancy using KL divergence and maps it to an intervention probability. By sampling whether to intervene from this probability, STI-OPD adaptively balances teacher control with student exploration. To learn from the resulting mixed-policy trajectories, we introduce an Importance-Weighted Reverse KL objective that corrects the token sampling mismatch between teacher-generated responses and the student policy to preserve the original OPD objective. Across tool-integrated reasoning and long-horizon interaction, STI-OPD outperforms the strongest prior OPD baseline on every evaluated benchmark and student size. Ablations further show that both discrepancy-guided intervention and importance weighting contribute to these gains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, On-Policy Distillation, Agentic

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FpNsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses trajectory drift in multi-turn on-policy distillation (OPD): because each student action shapes the next observation, early errors compound across turns, pushing the student into histories where the teacher's token-level supervision becomes unreliable. Existing mitigations either reweight/mask the loss (SAGE-OPD, SCOPE) or control rollout depth/teacher-mix via curricula (TCOD, Guided-OPD, TurnOPD, ReOPD), but none adaptively decides whether the current turn needs correction based on instantaneous policy discrepancy, and none correctly handles the token-sampling mismatch that arises when teacher-generated tokens enter a reverse-KL objective designed for student-sampled tokens.

Critical strengths

The importance-weighting correction for teacher-generated tokens is a principled, previously under-addressed technical contribution. The paper correctly identifies that intervention-based OPD methods (Guided-OPD, Relay-OPD, FutureBridge-OPD) introduce teacher-sampled tokens into a reverse-KL objective designed for student-sampled tokens, and that this creates a sampling-distribution mismatch that prior work does not correct.

Comprehensive experimental scope with proper controls. The paper evaluates on two distinct settings (tool-integrated reasoning with 6 benchmarks; long-horizon agents with 6 ALFWorld categories + ScienceWorld), two student scales (0.6B, 1.7B), two teacher scales (4B, 32B), and six baselines including all major intervention-based competitors (Guided-OPD, Relay-OPD, FutureBridge-OPD).

Critical weaknesses

The importance-weighting implementation deviates from the theory, and the deviation is unquantified. The theoretical justification (Eq. 10) shows that the importance-weighted integrand is an unbiased estimator of the conditional reverse KL. In practice (Appendix B.3), the importance ratio is PPO-style clipped "to enable stable and controllable step-wise optimization." Clipping breaks the unbiasedness identity: the clipped objective estimates a truncated version of the reverse KL, not the reverse KL itself. The paper does not report the clip range, the fraction of ratios that exceed the clip bounds, the variance of the unclipped ratio, or the magnitude of the bias introduced by clipping.

Seed count and evaluation protocol are under-specified, and some gains over the strongest baseline are modest. The paper reports means ± standard deviations but does not state the number of seeds or whether evaluation uses greedy or sampling decoding. The standard deviations are small (±0.3–0.9) and uniform across very different benchmarks, which is unusual for agentic tasks.

Questions for authors

What is the PPO clip range? What fraction of importance ratios w = π_θ(x|c)/π_T(x|c) exceed the clip bounds in practice (at early, mid, and late training)? What is the variance of the unclipped ratio?

How many independent seeds were used for each result? Is evaluation greedy or sampled (the stds suggest sampling)?

Have you tested alternative stochastic mappings (e.g., sigmoid(s/τ), min(1, s/τ), or a learned mapping)?

The teacher regenerates the entire turn even when discrepancy is concentrated in one decision. Have you considered (or is it feasible to implement) partial-turn intervention, where only the discrepant tokens are replaced?

ReOPD (Liao et al., 2026) addresses the same trajectory-drift problem via prefix replay rather than intervention. Why is it not included as a baseline?

5 comments
AuthorsOpenReview ↗

Response to Reviewer FpNs (Part 1 / 3)

Thank you for your thoughtful review and for recognizing the importance-weighting correction for teacher-generated tokens as a principled and previously under-addressed contribution, as well as our comprehensive experimental scope with proper controls. We address your concerns point by point below.

W1 & Q1: Clipping for the importance-weighted reverse KL

Clipping does not affect our claims. We adopt the clipping in our implementation but It is never active in our setting. Therefore, it introduces no bias, and the optimized objective is exactly the unclipped importance-weighted objective in Equations (10)–(11). The detailed theoretical analysis is provided below:

Student tokens. We perform a single gradient update per batch, so $\theta = \theta_{\text{old}}$ at the time of the update and the PPO ratio for student-generated tokens equals 1.

Teacher tokens. The ratio is $\omega^T = \pi_\theta / \pi_T$, which may deviate from 1. However, the advantage is $A = -\log \omega^T$, computed at $\theta_{\text{old}}$ and detached, so $A > 0$ exactly when $\omega^T < 1$, and $A < 0$ exactly when $\omega^T > 1$. The PPO objective $\min(\omega^T A, \text{clip}(\omega^T, 1-\epsilon, 1+\epsilon) A)$ cuts the gradient only when $\omega^T > 1+\epsilon$ with $A > 0$, or when $\omega^T < 1-\epsilon$ with $A < 0$. Neither case can occur, so the minimum always selects the unclipped term.

In the revision, we have stated that the clipped form is retained only for consistency with standard PPO implementations and is inactive in our setting.

W2 & Q2: Random seeds and sampling settings

We evaluate each model with 32 independent runs on the math benchmarks and 8 on GPQA, LiveCodeBench, ALFWorld, and ScienceWorld, sampling with temperature 0.3, top-p 0.5 referring to [5], and a maximum generation length of 32,768 tokens. Averaging over multiple independent samples per problem [1, 2, 3] and a 32,768-token generation limit [1, 2] are common practice in evaluating reasoning models. Sampling parameters are not standardized across reasoning evaluations [4], so we fix one configuration and apply it identically to all methods. This setup keeps the results stable in three ways, leading to relatively low and consistent standard deviation:

  • Multiple runs. We use as many evaluation runs as our computational budget allows, which reduces the influence of randomness.
  • Low sampling variance. The low temperature and top-p reduce the variability introduced by sampling.
  • No truncation errors. The generation limit is long enough to avoid errors caused by truncation: with a limit of 512 tokens, about 30% of generations would be truncated and scored as incorrect.

[1] Guo, Daya, et al. "DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning." arXiv preprint arXiv:2501.12948 (2025).
[2] Yang, An, et al. "Qwen3 technical report." arXiv preprint arXiv:2505.09388 (2025).
[3] Yu, Qiying, et al. "DAPO: An open-source LLM reinforcement learning system at scale." arXiv preprint arXiv:2503.14476 (2025).
[4] Hochlehnert, Andreas, et al. "A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility." arXiv preprint arXiv:2504.07086 (2025).
[5] Choudhury, Sanjiban, and Paloma Sodhi. "Better than your teacher: LLM agents that learn from privileged AI feedback." International Conference on Learning Representations (2025).

AuthorsOpenReview ↗

Response to Reviewer FpNs (Part 2 / 3)

Q3: Alternative stochastic mappings

Our mapping outperforms both suggested alternatives on Qwen3-0.6B (24.8 vs. 21.4 and 22.6 average math accuracy) and consistently outperforms deterministic thresholding (Appendix C.3), without any hyperparameter.

Our mapping $\alpha_t = s_t/(1+s_t)$, equivalently $\sigma(\log s_t)$ , i.e., a sigmoid applied to the log-discrepancy, and the $\tau = 1$ instance of $s_t/(\tau + s_t)$, satisfies three properties:

  • $\alpha_t = 0$ when $s_t = 0$, so no intervention occurs when the policies agree.
  • $\alpha_t$ increases monotonically with $s_t$.
  • $\alpha_t < 1$ for any finite $s_t$, so the student always retains a nonzero probability of executing its own turn.

The alternatives violate these properties or require a task-dependent scale:

  • Sigmoid mapping. $\text{sigmoid}(s_t/\tau) \geq 0.5$ for all $s_t \geq 0$, so it intervenes in at least half of the turns even when the student matches the teacher. Avoiding this requires an offset, which effectively acts as a threshold.
  • Clipped linear mapping. $\min(1, s_t/\tau)$ saturates at 1 whenever $s_t \geq \tau$, turning intervention into a deterministic takeover in high-discrepancy turns, and $\tau$ must be tuned for each setting.
  • Deterministic thresholding. It is the limit of a steep sigmoid with an offset, which we compare against in Appendix C.3.

A learned mapping would require an additional objective for deciding when to intervene. We leave it and other formulations to future work. Also we acknowledge that there may be other feasible mapping, we provide one simple and effective mapping in the submission.

The results of alternate mappings are shown below, where Avg. Math is the average over AIME24, AIME25, AIME26, and HMMT26 (the s/(1+s) entry corresponds to Table 1). We have added corresponding results and discussions in Appendix C.4

MappingHyperparametersAvg. Math
s/(1+s)none24.8
sigmoid(s/τ)τ = 121.4
min(1, s/τ)τ = 122.6

Q4: Partial-turn intervention

We intervene on whole turns because partial replacement breaks the consistency between reasoning and action, and token-level intervention decisions are noisy. STI-OPD also outperforms baselines that insert shorter teacher segments.

Partial replacement creates inconsistent contexts. The student's subsequent tokens are conditioned on the tokens being replaced, so splicing teacher tokens into a student response leaves the remaining student tokens inconsistent with them. For example, if only the tokens specifying the action are replaced, the student's reasoning may support a different action from the one executed, and training on such turns teaches the student to decouple reasoning from action. Replacing the entire response keeps them consistent (Section 3.2).

Token-level decisions are noisy. A single-token discrepancy estimate has high variance, and a large discrepancy at one token does not necessarily indicate an error, since many tokens admit several valid choices. Our turn-level score averages the $k_3$ estimates over the entire response (Equation (7)), yielding a stable signal.

STI-OPD outperforms segment-level baselines. Guided-OPD intervenes on whole turns, whereas Relay-OPD and FutureBridge-OPD insert shorter teacher segments (Appendix B.2). STI-OPD outperforms both kinds of baselines (Tables 1 and 2).

As noted in Appendix D, turn-level intervention spends teacher supervision on the whole response even when the discrepancy is concentrated in a single decision. We will leave the finer-grained intervention that preserves this consistency is future work.

AuthorsOpenReview ↗

Response to Reviewer FpNs (Part 3 / 3)

Q5: ReOPD as a baseline

Thank you for the suggestion. We added ReOPD as a baseline: STI-OPD outperforms it on the six benchmarks. We use the same teacher and student, with a prefix pool collected by the teacher in each environment. Owing to limited time and compute, we ran this comparison on Qwen3-0.6B in the tool-integrated reasoning (TIR) setting.

We did not include ReOPD in the submission because it addresses a different setting. We agree that ReOPD targets the same trajectory-drift problem, but the two settings differ:

  • ReOPD is offline. It replays prefixes from a fixed pool of pre-collected teacher trajectories under a static step-decaying schedule, without new environment interaction during training, and the student acts only at the supervised step.
  • Our intervention baselines are online. In Guided-OPD, Relay-OPD, and FutureBridge-OPD, the student interacts with the environment across multiple turns during training, and the teacher intervenes on trajectories as they are generated. Within this setting, STI-OPD decides at each turn whether to intervene based on the instantaneous teacher–student discrepancy.
ModelMethodAIME24 Avg@32AIME25 Avg@32AIME26 Avg@32HMMT26 Avg@32GPQA Avg@8LiveCodeBench v6 Avg@8
Qwen3-0.6BReOPD17.0 ± 0.825.2 ± 0.617.5 ± 0.418.0 ± 0.528.4 ± 0.728.9 ± 0.3
Qwen3-0.6BSTI-OPD24.2 ± 0.433.5 ± 0.422.7 ± 0.318.6 ± 0.332.5 ± 0.338.8 ± 0.4

We add the comparison with ReOPD in Appendix C.4.


We hope our responses could address your concerns. If you have any further questions, please do not hesitate to let us know.

Reviewer_jVEpOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method STI-OPD to address the problem that early student actions cause trajectories to drift away from the teacher distribution in multi-turn agentic OPD. It uses a k3 reverse-KL discrepancy to generate a stochastic, discrepancy-conditioned teacher takeover probability, and applies importance weighting to teacher-generated tokens. The authors give a very comprehensive family of benchmark to compare multiple Qwen3 student/teacher sizes on tool reasoning, ALFWorld, and ScienceWorld, and report component ablations, deterministic thresholds, matched intervention budgets, training dynamics, and cost analyses.

Critical strengths
  • The technical problem is clearly decomposed, and the design is internally coherent. k3 estimation, smooth stochastic intervention, and teacher-branch sampling correction correspond respectively to stable routing, trajectory correction, and token-source mismatch; this is more clearly novel than simply applying loss masking or a fixed curriculum.
  • The empirical coverage goes well beyond a single benchmark. Multiple student/teacher sizes, tool reasoning, long-horizon interactive tasks, and controls for components, thresholds, and matched intervention budgets suggest that the method is not effective only by chance in one setting.
Critical weaknesses
  • The central theoretical claim does not hold for the actual algorithm. The importance identity holds only for fixed contexts and absolute continuity; adaptive takeover changes the subsequent history/environment distribution, and the implementation is no longer unbiased after ratio clipping. The paper therefore cannot describe the current implementation as recovering the complete original OPD objective.
  • The strong performance and efficiency claims lack rigorous statistical and computational support. The teacher both scores trajectories and may regenerate an entire round, then supplies probabilities again during optimization; the same intervention count is not the same as the same number of teacher tokens or wall-clock time. The main paper also does not clearly report the number of independent training seeds for each result cell. For a method whose selling point is more effective allocation of teacher computation, this is an acceptance-threshold issue.
Questions for authors
  1. Under actual ratio clipping and the adaptive context distribution, would the authors revise “recovering the original OPD objective” to “an approximate correction at the fixed-context token-sampling level”? If not, please provide an error or unbiasedness proof for the full trajectory objective.
  2. Under matched teacher tokens, teacher forward passes, wall-clock time, and independent training seeds, does STI-OPD still outperform the strongest intervention baseline? If the gap disappears, the assessment of the substantive contribution would change.
  3. Does α=s/(1+s) require calibration by task, sequence length, or tokenizer? If the method works only in Qwen3/text-only settings, should the paper narrow its generality claims?
5 comments
AuthorsOpenReview ↗

Response to Reviewer jVEp (Part 1 / 3)

Thank you for your thoughtful review and for recognizing the clear problem decomposition and internally coherent design of STI-OPD, as well as its broad empirical coverage across model sizes, task types, and controlled comparisons. We address your concerns point by point below.

W1 & Q1: The claim of recovering the original OPD objective

Importance weighting corrects the token-sampling mismatch at fixed contexts, it does not recover the original OPD objective over full trajectories. We study teacher-intervention OPD [1, 2, 3] to improve agentic training. Intervention makes the executed tokens in intervened turns off-policy with respect to the student, so the standard OPD loss on these tokens no longer estimates the intended conditional reverse KL. Importance-weighted reverse KL corrects this mismatch at each fixed context. We replace the corresponding wording in the abstract and introduction with "a correction of the token-sampling mismatch at fixed contexts", consistent with the scope stated at the end of revised Section 3.3.

Regarding the three conditions you raise:

  • Ratio clipping does not affect our theoretical claims and conclusion. Clipping is never active in our implementation (see our response to Reviewer FpNs, W1), so the fixed-context correction is exact. We perform a single gradient update per batch, so the ratio for student tokens equals 1. For teacher tokens, the advantage equals $-\log \omega^T$, so $A > 0$ exactly when $\omega^T < 1$ and $A < 0$ exactly when $\omega^T > 1$, and the minimum in the PPO objective always selects the unclipped term.
  • Absolute continuity holds. Both the student and the teacher are softmax policies over a shared vocabulary that assign nonzero probability to every token. All training rollouts, including teacher turns, are sampled with temperature 1.0 and top-p 1.0 (Table 4), so executed teacher tokens are drawn from $\pi_T$ itself rather than from a truncated distribution.
  • The context shift is not corrected, by design. Keeping the student in states closer to successful trajectories is the purpose of intervention [4]. Correcting this shift exactly would require trajectory-level importance weights, whose variance grows exponentially with the horizon [5], which is impractical for long-horizon multi-turn tasks.

Empirically, removing importance weighting reduces performance by 2.4–11.8 points on every ALFWorld task type and ScienceWorld (Table 3).

[1] Li, Gengsheng, et al. "On-policy distillation with curriculum turn-level guidance for multi-turn agents." arXiv preprint arXiv:2606.15912 (2026).
[2] Xu, Haolei, et al. "Pass the baton: Trajectory-relayed on-policy distillation." arXiv preprint arXiv:2607.26057 (2026).
[3] Chen, Chishui, et al. "Look ahead before you distill: Future trajectory validation of teacher guidance for agentic on-policy distillation." arXiv preprint arXiv:2608.01953 (2026).
[4] Ross, Stéphane, Geoffrey Gordon, and Drew Bagnell. "A reduction of imitation learning and structured prediction to no-regret online learning." Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (2011).
[5] Liu, Qiang, et al. "Breaking the curse of horizon: Infinite-horizon off-policy estimation." Advances in Neural Information Processing Systems 31 (2018).

AuthorsOpenReview ↗

Response to Reviewer jVEp (Part 2 / 3)

W2 & Q2: Comparison under matched computation, and statistical details

Thank you for this important point. We agree that the same number of interventions does not imply the same teacher cost. However, it is difficult to align the intervention tokens due to distinct intervention strategies proposed by each method. We therefore regard end-to-end training wall-clock time as the most comparable cost measure in the submission, because all of these teacher computations are executed during training and all methods are measured on the same hardware and serving framework.

As shown in Figure 5 (reproduced in the table below, where performance is the ScienceWorld success rate from Table 2), STI-OPD achieves the best performance efficiency than all baselines. FutureBridge-OPD, while requiring less training time at both student sizes. STI-OPD therefore Pareto-dominates FutureBridge-OPD, and a matched wall-clock comparison would require truncating FutureBridge-OPD’s training, which is unlikely to improve its performance.

ModelMethodWall-clock Training Time (h)ScienceWorld Success RateSuccess Rate / Hour
Qwen3-0.6BGRPO68.413.80.20
Qwen3-0.6BOPD43.47.30.17
Qwen3-0.6BFutureBridge-OPD52.415.40.29
Qwen3-0.6BSTI-OPD45.918.70.41
Qwen3-1.7BGRPO77.320.10.26
Qwen3-1.7BOPD51.717.10.33
Qwen3-1.7BFutureBridge-OPD62.822.20.35
Qwen3-1.7BSTI-OPD53.527.50.51

STI-OPD also relies on fewer teacher-generated tokens. As shown in Figure 4, the fraction of executed tokens generated by the teacher decreases steadily over training for STI-OPD, whereas it remains roughly constant and consistently higher for FutureBridge-OPD. This follows from the intervention rule: the teacher takes over more often when the policy discrepancy is large and cedes control as the student aligns, and the discrepancy decreases much faster for STI-OPD than for FutureBridge-OPD. Because STI-OPD achieves higher performance with fewer teacher-generated tokens, matching teacher tokens is unlikely to close the gap. In addition, Appendix C.4 shows that STI-OPD remains consistently better under the same teacher-intervention budget.

Regarding random seeds: because large-scale LLM training is costly, we follow common practice [1,2,3] in the baselines and train each model once, then run independent evaluations under multiple random seeds: 32 for the math benchmarks, and 8 each for GPQA, LiveCodeBench, ALFWorld, and ScienceWorld. We report the mean and standard deviation across these evaluations to reduce the effect of evaluation randomness.

[1] Li, Gengsheng, et al. "On-policy distillation with curriculum turn-level guidance for multi-turn agents." arXiv preprint arXiv:2606.15912 (2026).
[2] Xu, Haolei, et al. "Pass the baton: Trajectory-relayed on-policy distillation." arXiv preprint arXiv:2607.26057 (2026).
[3] Chen, Chishui, et al. "Look ahead before you distill: Future trajectory validation of teacher guidance for agentic on-policy distillation." arXiv preprint arXiv:2608.01953 (2026).

AuthorsOpenReview ↗

Response to Reviewer jVEp (Part 3 / 3)

Q3: The choice of α = s/(1+s) and the scope of STI-OPD

Equation (8) requires no calibration by task or sequence length, and we use it unchanged across all benchmarks. It sets the intervention odds equal to $s_t$, and $s_t$ is a length-normalized per-token KL estimate (Equation (7)), so it has the same units across tasks and does not scale with response length. The same mapping, without any tuning, is used for both tool-integrated reasoning and long-horizon agentic interaction.

Other monotone mappings are compatible with our framework and we provide one simple and effective mapping. Our mapping $\alpha_t = s_t/(1+s_t)$, equivalently $\sigma(\log s_t)$, satisfies three properties without any hyperparameter. First, $\alpha_t = 0$ when $s_t = 0$, so no intervention occurs when the policies agree. Second, $\alpha_t$ increases monotonically with $s_t$. Third, $\alpha_t < 1$ for any finite $s_t$, so the student always retains a chance to execute its own turn. The alternatives either violate these properties or require a task-dependent scale:

  • $\text{sigmoid}(s_t/\tau) \geq 0.5$ for all $s_t \geq 0$, so it intervenes in at least half of the turns even when the student matches the teacher.
  • $\min(1, s_t/\tau)$ saturates at 1 whenever $s_t \geq \tau$, removing student execution in high-discrepancy turns, and $\tau$ must be tuned for each setting.
  • Deterministic thresholding is the limit of a steep sigmoid with an offset, and our mapping consistently outperforms it (Appendix C.3).

Results on Qwen3-0.6B (Avg. Math is the average over AIME24, AIME25, AIME26, and HMMT26):

MappingHyperparametersAvg. Math
s/(1+s)none24.8
sigmoid(s/τ)τ = 121.4
min(1, s/τ)τ = 122.6

Tokenizer. The token-level discrepancy requires the teacher and student to share a vocabulary. This is a requirement of token-level OPD in general rather than of our mapping, and cross-tokenizer settings require additional alignment techniques that are orthogonal to our contribution.

Scope. Our experiments use Qwen3 models in text-based agentic environments, as noted in Appendix D. We chose Qwen3 because it is a widely used open-source model family with strong long-reasoning ability, and its models of different sizes share a vocabulary, as token-level OPD requires. Multimodal settings are beyond the scope of this paper, and we state this scope in the main text.

We thank the reviewer for the suggestion on the probability mapping. A learned mapping would require an additional objective for deciding when to intervene. We will discuss it, together with other mappings and multimodal extensions, in future work.


We hope our responses could address your concerns. If you have any further questions, please do not hesitate to let us know.

Reviewer_JKz7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I like that this paper attempts to correct compounding errors by sampling a teacher for parts of a trajectory. There are improvements over baselines.

Critical strengths

There is a formal fix for using the teacher’s tokens in lieu of the student’s with equation 10. I like the various baselines and ablations used to validate the method.

Critical weaknesses

Equation 10 does not use a a ppo clip but the ppo clip is used in practice. It is plausible that the student is using a different correct strarefy than the teacher but sampling the teacher would switch out a correct token. There is a lack of discussion on the effect of privileged information distillation: https://arxiv.org/abs/2602.04942.

Questions for authors

Equation 11 needs a rethink. Overall this paper does have performance gains but the method has some gaps like Equation 10 and whether the student’s policy is being shifted to imitate the teacher or to a correct optimal policy. I also think a privleged info baseline would help situate whether this sampling is a good direction.

5 comments
AuthorsOpenReview ↗

Response to Reviewer JKz7 (Part 1 / 2)

Thank you for your thoughtful review and for recognizing our approach to correcting compounding errors with external guidance, the formal correction in Equation (10), and the range of baselines and ablations used to validate the method. We address your concerns point by point below.

W1: Equation (10) omits the PPO clip that is used in practice

PPO clipping does not affect our conclusions. We explain this theoretically below: under our single-update training setup, the clipping branch is never active for either student or teacher tokens, so the gradient is the same as that of the unclipped objective in Equations (10)–(11).

Student tokens. We perform a single gradient update per batch, so $\theta = \theta_{\text{old}}$ at the time of the update and the PPO ratio $\pi_\theta / \pi_{\theta_{\text{old}}}$ equals 1.

Teacher tokens. The advantage is $A = -\ell_{\theta_{\text{old}}} = -\log \omega^T$, so $A > 0$ exactly when $\omega^T < 1$, and $A < 0$ exactly when $\omega^T > 1$. The PPO objective $\min(\omega^T A, \text{clip}(\omega^T, 1-\epsilon, 1+\epsilon) A)$ cuts the gradient only when $\omega^T > 1+\epsilon$ with $A > 0$, or when $\omega^T < 1-\epsilon$ with $A < 0$. Neither case can occur, so the minimum always selects the unclipped term.

Also, we actually use PPO clipping in our implementation and, removing importance weighting reduces performance by 2.4–11.8 points on every ALFWorld task type and ScienceWorld (Table 3). These results show that clipping does not resolve the token-sampling mismatch between teacher-generated tokens and the student policy.

We retain the clipped form only for consistency with standard PPO implementations and have clarified this in the revised Section 3.3.

W2 & Q1: The student may follow a different correct strategy, and intervention may replace its correct tokens

The student is shifted toward the teacher by design, but it is not forced to copy the teacher's strategy. OPD aims to move the student efficiently toward a stronger teacher, which serves as a proxy for a better policy [1, 2], discovering an optimal policy independently is the role of RL.

Valid alternative strategies are not penalized. The reverse KL used in OPD is mode-seeking: it encourages the student to generate outputs that the teacher considers likely, within the student's own capacity, rather than covering all of the teacher's behaviors [3]. A valid alternative strategy to which the teacher assigns non-negligible probability incurs relatively little penalty.

The student keeps executing its own turns. Because $\alpha_t < 1$ for any finite $s_t$ (Equation (8)), the student retains a nonzero probability of executing its own turn even when its strategy diverges from the teacher's (Section 3.2).

The trade-off favors efficiency. STI-OPD may discard some correct student tokens, but the teacher's dense token-level supervision on student-visited states trains the student faster than a sparse outcome reward. STI-OPD achieves a better trade-off between performance and training time than GRPO (Figure 5 of the submission, reproduced in the table below).

ModelMethodWall-clock Training Time (h)ScienceWorld Success Rate
Qwen3-0.6BOPD43.47.3
Qwen3-0.6BSTI-OPD45.918.7
Qwen3-0.6BFutureBridge-OPD52.415.4
Qwen3-0.6BGRPO68.413.8
Qwen3-1.7BOPD51.717.1
Qwen3-1.7BSTI-OPD53.527.5
Qwen3-1.7BFutureBridge-OPD62.822.2
Qwen3-1.7BGRPO77.320.1

[1] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." arXiv preprint arXiv:1503.02531 (2015).
[2] Agarwal, Rishabh, et al. "On-policy distillation of language models: Learning from self-generated mistakes." International Conference on Learning Representations (2024).
[3] Gu, Yuxian, et al. "MiniLLM: Knowledge distillation of large language models." International Conference on Learning Representations (2024).

AuthorsOpenReview ↗

Response to Reviewer JKz7 (Part 2 / 2)

W3 & Q2: Relation to privileged information distillation

Thank you for pointing us to this work, which we discuss in the revision. We view privileged information (PI) distillation as orthogonal and complementary to STI-OPD. PI distillation augments the teacher’s inputs with privileged information, while STI-OPD determines when the teacher intervenes (Section 3.2) and corrects the resulting token-sampling mismatch (Section 3.3). The two are complementary: STI-OPD can use a PI-conditioned teacher for both intervention (Equation 7) and supervision (Equation 9).

Following your suggestion, we add a PI baseline (we add these results in Appendix C.8), OPD with a PI-conditioned teacher, and a variant of STI-OPD that uses the same PI-conditioned teacher. Results blow (the OPD and STI-OPD rows are reproduced from Table 1) show that PI could be well integrated with STI-OPD to further improve its performance

First, STI-OPD without PI already outperforms OPD with PI across benchmarks, indicating that intervention-based sampling contributes substantially more than PI in this setting. Second, combining the two yields the best results on all six benchmarks, which suggests that the two approaches are complementary. Third, PI brings a larger gain on average when combined with STI-OPD than with standard OPD, possibly because in STI-OPD the PI-conditioned teacher not only scores the student’s tokens but also generates the intervened turns.

ModelMethodAIME24 Avg@32AIME25 Avg@32AIME26 Avg@32HMMT26 Avg@32GPQA Avg@8LiveCodeBench v6 Avg@8
Qwen3-0.6BOPD11.2 ± 0.518.5 ± 0.512.4 ± 0.411.8 ± 0.627.1 ± 0.623.5 ± 0.7
Qwen3-0.6BOPD w/ PI13.8 ± 0.319.3 ± 0.613.3 ± 0.212.2 ± 0.428.2 ± 0.324.2 ± 0.5
Qwen3-0.6BSTI-OPD24.2 ± 0.433.5 ± 0.422.7 ± 0.318.6 ± 0.332.5 ± 0.338.8 ± 0.4
Qwen3-0.6BSTI-OPD w/ PI25.3 ± 0.235.0 ± 0.523.5 ± 0.619.1 ± 0.433.9 ± 0.540.5 ± 0.3

We hope our responses could address your concerns. If you have any further questions, please do not hesitate to let us know.

AuthorsOpenReview ↗

Follow-up of Rebuttal

Dear Reviewer JKz7,

As the discussion period is about to close, we would like to kindly draw your attention to our response above. In response to your reviews, we have shown that PPO clipping is never active in our setting and does not affect our conclusions, while importance weighting remains necessary, as removing it costs 2.4–11.8 points. We have also clarified that intervention does not force the student to abandon a valid alternative strategy and that STI-OPD achieves a better trade-off between performance and training time than GRPO, and we have added experiments with a privileged-information teacher, showing that PI is complementary to STI-OPD and that combining the two gives the best results on all six benchmarks.

Could you please let us know whether these additions have addressed your concerns regarding PPO clipping, alternative student strategies, and privileged information distillation? If so, we would appreciate it if you could consider updating your score.

If any concerns remain, we would be glad to clarify them further while the discussion period is still open.

Sincerely,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Author Wrap-up & Reviewer Discussion Summary

Dear PCs, SACs, ACs, and Reviewers

As the discussion period comes to a close, we provide a brief summary of the review discussion.

We thank you and the reviewers (JKz7, jVEp, and FpNs) for the constructive discussion. Following our rebuttal, Reviewers JKz7 and FpNs raised their ratings from 2 to 3. All three reviewers now rate the paper 3 (Weak Acceptance) and explicitly confirmed that our rebuttal resolved their concerns, with JKz7 and jVEp stating that the additional experiments support the paper's claims.


Reviewer Consensus Highlights

  • Reviewer FpNs (Rating: raised from 2 to 3, Weak Acceptance): Recognized the importance-weighted correction as a "principled, previously under-addressed technical contribution" that prior intervention-based OPD methods lack. Praised the comprehensive experiments with proper controls. After the rebuttal, stated that they were satisfied with the response and raised the score.
  • Reviewer JKz7 (Rating: raised from 2 to 3, Weak Acceptance): Appreciated the formal fix for learning from teacher tokens (Eq. 10), the consistent gains over baselines, and the range of baselines and ablations. After the rebuttal, confirmed that the additional experiments satisfy all asks in the review and support the paper's claims, and raised the score.
  • Reviewer jVEp (Rating: 3, Weak Acceptance): Highlighted the clear problem decomposition and internally coherent design. k3 estimation, stochastic intervention, and teacher-branch correction address stable routing, trajectory correction, and token-source mismatch, which the reviewer found "more clearly novel than simply applying loss masking or a fixed curriculum." Noted that the broad empirical coverage shows the gains are robust across settings. Confirmed that our responses answered all questions and support the paper's claims.

Summary of Resolution on Key Concerns

  • PPO Clipping (JKz7, jVEp, FpNs): Clipping is never active in our setting (measured clipping fraction 0%), so the optimized objective is exactly Eq. (11). With one gradient update per batch, student-token ratios equal 1. For teacher tokens, the sign of the detached advantage A=−log⁡ωTA=-\log\omega^T rules out both clipping cases. Importance weighting is essential: removing it costs 2.4–11.8 points on every long-horizon task (Table 3).
  • Scope of the Correction (jVEp): We state the claim as a correction of the token-sampling mismatch at each fixed context. Absolute continuity holds because teacher turns are sampled from full-support softmax policies (temperature 1.0, top-p 1.0). The context shift is intended, since steering the student toward successful states is the purpose of intervention.
  • Cost and Teacher Usage (jVEp): STI-OPD Pareto-dominates FutureBridge-OPD in end-to-end wall-clock time, with 0.41 / 0.51 ScienceWorld success per training hour vs. 0.29 / 0.35. It adds only 3–6% training time over standard OPD. Under a matched teacher-intervention budget, it outperforms FutureBridge-OPD on all seven tasks by 4.2 points on average.
  • Intervention Mapping (jVEp, FpNs): αt=st/(1+st)\alpha_t = s_t/(1+s_t) is hyperparameter-free and is used unchanged across all tasks, since sts_t is length-normalized. It outperforms sigmoid(s/τ)\mathrm{sigmoid}(s/\tau) and min⁡(1,s/τ)\min(1, s/\tau) (24.8 vs. 21.4 / 22.6 Avg. Math). It also outperforms every deterministic threshold on all long-horizon tasks.

Key Revisions in the Updated Paper

  • Stated that PPO clipping is inactive and reported the clip range and the 0% clipping fraction (§3.3, Appendix B.3, Table 4).
  • Added the matched-budget comparison (Appendix C.5) and the wall-clock efficiency analysis (§4.4, Appendix C.6).
  • Added the alternative stochastic mappings (Appendix C.4).
  • Added the ReOPD comparison (Appendix C.7) and the PI-teacher experiments (Appendix C.8), and discussed PI distillation in Related Work.

Thank you again for your time and guidance throughout the review process.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.