- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses trajectory drift in multi-turn on-policy distillation (OPD): because each student action shapes the next observation, early errors compound across turns, pushing the student into histories where the teacher's token-level supervision becomes unreliable. Existing mitigations either reweight/mask the loss (SAGE-OPD, SCOPE) or control rollout depth/teacher-mix via curricula (TCOD, Guided-OPD, TurnOPD, ReOPD), but none adaptively decides whether the current turn needs correction based on instantaneous policy discrepancy, and none correctly handles the token-sampling mismatch that arises when teacher-generated tokens enter a reverse-KL objective designed for student-sampled tokens.
Critical strengths
The importance-weighting correction for teacher-generated tokens is a principled, previously under-addressed technical contribution. The paper correctly identifies that intervention-based OPD methods (Guided-OPD, Relay-OPD, FutureBridge-OPD) introduce teacher-sampled tokens into a reverse-KL objective designed for student-sampled tokens, and that this creates a sampling-distribution mismatch that prior work does not correct.
Comprehensive experimental scope with proper controls. The paper evaluates on two distinct settings (tool-integrated reasoning with 6 benchmarks; long-horizon agents with 6 ALFWorld categories + ScienceWorld), two student scales (0.6B, 1.7B), two teacher scales (4B, 32B), and six baselines including all major intervention-based competitors (Guided-OPD, Relay-OPD, FutureBridge-OPD).
Critical weaknesses
The importance-weighting implementation deviates from the theory, and the deviation is unquantified. The theoretical justification (Eq. 10) shows that the importance-weighted integrand is an unbiased estimator of the conditional reverse KL. In practice (Appendix B.3), the importance ratio is PPO-style clipped "to enable stable and controllable step-wise optimization." Clipping breaks the unbiasedness identity: the clipped objective estimates a truncated version of the reverse KL, not the reverse KL itself. The paper does not report the clip range, the fraction of ratios that exceed the clip bounds, the variance of the unclipped ratio, or the magnitude of the bias introduced by clipping.
Seed count and evaluation protocol are under-specified, and some gains over the strongest baseline are modest. The paper reports means ± standard deviations but does not state the number of seeds or whether evaluation uses greedy or sampling decoding. The standard deviations are small (±0.3–0.9) and uniform across very different benchmarks, which is unusual for agentic tasks.
Questions for authors
What is the PPO clip range? What fraction of importance ratios w = π_θ(x|c)/π_T(x|c) exceed the clip bounds in practice (at early, mid, and late training)? What is the variance of the unclipped ratio?
How many independent seeds were used for each result? Is evaluation greedy or sampled (the stds suggest sampling)?
Have you tested alternative stochastic mappings (e.g., sigmoid(s/τ), min(1, s/τ), or a learned mapping)?
The teacher regenerates the entire turn even when discrepancy is concentrated in one decision. Have you considered (or is it feasible to implement) partial-turn intervention, where only the discrepant tokens are replaced?
ReOPD (Liao et al., 2026) addresses the same trajectory-drift problem via prefix replay rather than intervention. Why is it not included as a baseline?
5 comments
Response to Reviewer FpNs (Part 2 / 3)
Q3: Alternative stochastic mappings
Our mapping outperforms both suggested alternatives on Qwen3-0.6B (24.8 vs. 21.4 and 22.6 average math accuracy) and consistently outperforms deterministic thresholding (Appendix C.3), without any hyperparameter.
Our mapping $\alpha_t = s_t/(1+s_t)$, equivalently $\sigma(\log s_t)$ , i.e., a sigmoid applied to the log-discrepancy, and the $\tau = 1$ instance of $s_t/(\tau + s_t)$, satisfies three properties:
- $\alpha_t = 0$ when $s_t = 0$, so no intervention occurs when the policies agree.
- $\alpha_t$ increases monotonically with $s_t$.
- $\alpha_t < 1$ for any finite $s_t$, so the student always retains a nonzero probability of executing its own turn.
The alternatives violate these properties or require a task-dependent scale:
- Sigmoid mapping. $\text{sigmoid}(s_t/\tau) \geq 0.5$ for all $s_t \geq 0$, so it intervenes in at least half of the turns even when the student matches the teacher. Avoiding this requires an offset, which effectively acts as a threshold.
- Clipped linear mapping. $\min(1, s_t/\tau)$ saturates at 1 whenever $s_t \geq \tau$, turning intervention into a deterministic takeover in high-discrepancy turns, and $\tau$ must be tuned for each setting.
- Deterministic thresholding. It is the limit of a steep sigmoid with an offset, which we compare against in Appendix C.3.
A learned mapping would require an additional objective for deciding when to intervene. We leave it and other formulations to future work. Also we acknowledge that there may be other feasible mapping, we provide one simple and effective mapping in the submission.
The results of alternate mappings are shown below, where Avg. Math is the average over AIME24, AIME25, AIME26, and HMMT26 (the s/(1+s) entry corresponds to Table 1). We have added corresponding results and discussions in Appendix C.4
| Mapping | Hyperparameters | Avg. Math |
|---|---|---|
| s/(1+s) | none | 24.8 |
| sigmoid(s/τ) | τ = 1 | 21.4 |
| min(1, s/τ) | τ = 1 | 22.6 |
Q4: Partial-turn intervention
We intervene on whole turns because partial replacement breaks the consistency between reasoning and action, and token-level intervention decisions are noisy. STI-OPD also outperforms baselines that insert shorter teacher segments.
Partial replacement creates inconsistent contexts. The student's subsequent tokens are conditioned on the tokens being replaced, so splicing teacher tokens into a student response leaves the remaining student tokens inconsistent with them. For example, if only the tokens specifying the action are replaced, the student's reasoning may support a different action from the one executed, and training on such turns teaches the student to decouple reasoning from action. Replacing the entire response keeps them consistent (Section 3.2).
Token-level decisions are noisy. A single-token discrepancy estimate has high variance, and a large discrepancy at one token does not necessarily indicate an error, since many tokens admit several valid choices. Our turn-level score averages the $k_3$ estimates over the entire response (Equation (7)), yielding a stable signal.
STI-OPD outperforms segment-level baselines. Guided-OPD intervenes on whole turns, whereas Relay-OPD and FutureBridge-OPD insert shorter teacher segments (Appendix B.2). STI-OPD outperforms both kinds of baselines (Tables 1 and 2).
As noted in Appendix D, turn-level intervention spends teacher supervision on the whole response even when the discrepancy is concentrated in a single decision. We will leave the finer-grained intervention that preserves this consistency is future work.
Response to Reviewer FpNs (Part 3 / 3)
Q5: ReOPD as a baseline
Thank you for the suggestion. We added ReOPD as a baseline: STI-OPD outperforms it on the six benchmarks. We use the same teacher and student, with a prefix pool collected by the teacher in each environment. Owing to limited time and compute, we ran this comparison on Qwen3-0.6B in the tool-integrated reasoning (TIR) setting.
We did not include ReOPD in the submission because it addresses a different setting. We agree that ReOPD targets the same trajectory-drift problem, but the two settings differ:
- ReOPD is offline. It replays prefixes from a fixed pool of pre-collected teacher trajectories under a static step-decaying schedule, without new environment interaction during training, and the student acts only at the supervised step.
- Our intervention baselines are online. In Guided-OPD, Relay-OPD, and FutureBridge-OPD, the student interacts with the environment across multiple turns during training, and the teacher intervenes on trajectories as they are generated. Within this setting, STI-OPD decides at each turn whether to intervene based on the instantaneous teacher–student discrepancy.
| Model | Method | AIME24 Avg@32 | AIME25 Avg@32 | AIME26 Avg@32 | HMMT26 Avg@32 | GPQA Avg@8 | LiveCodeBench v6 Avg@8 |
|---|---|---|---|---|---|---|---|
| Qwen3-0.6B | ReOPD | 17.0 ± 0.8 | 25.2 ± 0.6 | 17.5 ± 0.4 | 18.0 ± 0.5 | 28.4 ± 0.7 | 28.9 ± 0.3 |
| Qwen3-0.6B | STI-OPD | 24.2 ± 0.4 | 33.5 ± 0.4 | 22.7 ± 0.3 | 18.6 ± 0.3 | 32.5 ± 0.3 | 38.8 ± 0.4 |
We add the comparison with ReOPD in Appendix C.4.
We hope our responses could address your concerns. If you have any further questions, please do not hesitate to let us know.
Response to Reviewer FpNs (Part 1 / 3)
Thank you for your thoughtful review and for recognizing the importance-weighting correction for teacher-generated tokens as a principled and previously under-addressed contribution, as well as our comprehensive experimental scope with proper controls. We address your concerns point by point below.
W1 & Q1: Clipping for the importance-weighted reverse KL
Clipping does not affect our claims. We adopt the clipping in our implementation but It is never active in our setting. Therefore, it introduces no bias, and the optimized objective is exactly the unclipped importance-weighted objective in Equations (10)–(11). The detailed theoretical analysis is provided below:
Student tokens. We perform a single gradient update per batch, so $\theta = \theta_{\text{old}}$ at the time of the update and the PPO ratio for student-generated tokens equals 1.
Teacher tokens. The ratio is $\omega^T = \pi_\theta / \pi_T$, which may deviate from 1. However, the advantage is $A = -\log \omega^T$, computed at $\theta_{\text{old}}$ and detached, so $A > 0$ exactly when $\omega^T < 1$, and $A < 0$ exactly when $\omega^T > 1$. The PPO objective $\min(\omega^T A, \text{clip}(\omega^T, 1-\epsilon, 1+\epsilon) A)$ cuts the gradient only when $\omega^T > 1+\epsilon$ with $A > 0$, or when $\omega^T < 1-\epsilon$ with $A < 0$. Neither case can occur, so the minimum always selects the unclipped term.
In the revision, we have stated that the clipped form is retained only for consistency with standard PPO implementations and is inactive in our setting.
W2 & Q2: Random seeds and sampling settings
We evaluate each model with 32 independent runs on the math benchmarks and 8 on GPQA, LiveCodeBench, ALFWorld, and ScienceWorld, sampling with temperature 0.3, top-p 0.5 referring to [5], and a maximum generation length of 32,768 tokens. Averaging over multiple independent samples per problem [1, 2, 3] and a 32,768-token generation limit [1, 2] are common practice in evaluating reasoning models. Sampling parameters are not standardized across reasoning evaluations [4], so we fix one configuration and apply it identically to all methods. This setup keeps the results stable in three ways, leading to relatively low and consistent standard deviation:
[1] Guo, Daya, et al. "DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning." arXiv preprint arXiv:2501.12948 (2025).
[2] Yang, An, et al. "Qwen3 technical report." arXiv preprint arXiv:2505.09388 (2025).
[3] Yu, Qiying, et al. "DAPO: An open-source LLM reinforcement learning system at scale." arXiv preprint arXiv:2503.14476 (2025).
[4] Hochlehnert, Andreas, et al. "A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility." arXiv preprint arXiv:2504.07086 (2025).
[5] Choudhury, Sanjiban, and Paloma Sodhi. "Better than your teacher: LLM agents that learn from privileged AI feedback." International Conference on Learning Representations (2025).