acceptodds
Under review as a conference paper at ICLR 2027

Teacher–Student Gaps Are Not Enough: Outcome-Guided On-Policy Distillation for Multi-Turn Autonomous Agents

13 views

Abstract

On-policy distillation (OPD) trains a student on its own trajectories with dense teacher supervision. Recent work on OPD for multi-turn autonomous agents often treats large teacher–student token-level distributional gaps as promising intervention points, linking larger gaps to a greater need for correction. Yet, our empirical analysis reveals a **supervision–benefit mismatch**: large gaps can be benign, while small gaps can be outcome-critical. Teacher–student gaps capture differences at the current turn, whereas the benefit of teacher guidance depends on how the current student interacts with the environment afterward. The student may still succeed despite choosing an action that differs from the teacher's, while a teacher-preferred action may lead to a state from which the student cannot complete the task. Local gaps alone are therefore not enough to determine whether teacher guidance benefits the current student. Effective supervision should instead emphasize guidance that the current student can translate into better final task outcomes. Accordingly, we propose **Outcome-Guided On-Policy Distillation (OG-OPD)**, which applies trajectory-relative weighting to teacher supervision and calibrates these weights using final task outcomes from paired student continuations. This calibration selectively strengthens supervision on the student's original trajectories at turns where teacher guidance benefits the current student. Across ALFWorld, ScienceWorld, and WebShop, OG-OPD consistently outperforms baselines under diverse settings. It improves task success rates by 3.6–17.7 percentage points over vanilla OPD and by up to 7.0 percentage points over the strongest baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy distillation, Multi-turn autonomous agents, Selective supervision, Outcome-based calibration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2L4iOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates on-policy distillation (OPD) for multi-turn autonomous agents and identifies a core "supervision-benefit mismatch": token-level distributional gaps between teacher and student fail to reliably signal actual task benefits. To address this, the authors propose Outcome-Guided On-Policy Distillation (OG-OPD). The method incorporates trajectory-relative weighting to unify supervision scale across turns, filters candidate turns via gap jumps, and performs outcome-based calibration using paired continuations with the current frozen student model. Supervision is selectively upweighted only when a teacher action allows the student to turn a failing trajectory into a success. Validation across ALFWorld, ScienceWorld, and WebShop across multiple model scales demonstrates consistent improvements in task success rate (e.g., up to +17.7% on WebShop) over Vanilla OPD and existing baselines.

Critical strengths
  • High-Leverage Insight into Multi-Turn Agent Distillation: The paper offers a compelling analysis of why classical token-level gaps fail in multi-turn environments. In sequential decision-making, small deviations early on cause compounding errors over steps. These small gaps often represent high-leverage learning points—fully within the student's current capacity yet decisive for task outcome. Conversely, large gaps frequently force the student into states it cannot handle (teacher-only completions), resulting in teacher harm. Grounding supervision in actual student continuations effectively resolves this dilemma.

  • Effective and Outcome-Grounded Calibration: OG-OPD provides a pragmatic solution. By evaluating paired continuations with the frozen current student, the method guarantees that upweighted teacher actions are actually executable by the student at its current learning stage. Furthermore, the candidate selection mechanism successfully filters out roughly 75% of unpromising turns, keeping training rollout overhead to a manageable level (~1.4×) while yielding substantial empirical gains.

Critical weaknesses
  • Increased Training Cost and Complexity: Although candidate selection prunes around 75% of turns, running paired rollouts using the student model still introduces a ~1.4× to 1.5× wall-clock training time overhead and requires managing environment resets and parallel continuations during training.

  • Heuristic Candidate Selection and Threshold Sensitivity: The candidate selection step relies on specific quantile thresholds for absolute gap and abrupt increase. The paper lacks a thorough sensitivity analysis showing how robust these thresholds are across environments with different episode lengths, noisy observation spaces, or varying action granularities.

Questions for authors
  • Recall of Outcome-Critical Small Gaps: What estimated proportion of outcome-critical small gaps are successfully captured by the abrupt increase criterion versus filtered out?

  • Potential for Early Stopping in Rollouts: Could the paired continuation overhead (~1.4×) be further reduced by implementing early stopping during rollouts when trajectory divergence becomes irreversible?

  • Threshold Robustness: How robust are the default quantile thresholds when transferred to environments with significantly longer horizons or sparse rewards?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 2L4i | Weakness 1. Training cost under matched budgets

To account for calibration costs, we compare methods on WebShop under two separate resource caps. We separately cap total training time and total environment interactions, allowing vanilla OPD to spend the resources saved on calibration on additional updates. We also include Teacher-success upweighting to test whether a different upweighting rule changes performance under the same caps.

All methods use the same budget limits. We use a Qwen3-32B teacher and a Qwen3-1.7B student. All three methods use the same hardware, initialization, training tasks, and evaluation protocol. We set the caps by rounding up the resources used for 200 updates with default OG-OPD, to 5.60 hours and 20,000 environment steps. The caps are applied separately, so methods can perform different numbers of updates. For each seed, we use cumulative resource logs to retrospectively select the latest saved checkpoint from a completed update within the relevant cap.

OG-OPD and Teacher-success upweighting use the same procedure for paired student continuations. They also share candidate selection, pre-calibration weights, teacher response evaluation, and the weight floor. To limit the cost of state restoration and two-branch execution, the procedure checks at most two candidates and pairs at most one turn per trajectory. We verify the starting state by replaying the prefix. From that state, the student’s original action and the teacher-proposed action are executed in separate environment instances. The same frozen student policy continues both branches. If replay or execution fails, calibration is skipped, but consumed time and environment steps still count toward the budget. Teacher-success upweighting requires only success after the teacher-proposed action, whereas OG-OPD also requires failure after the student’s original action.

1. OG-OPD leads at the same training-time cap. The 5.60-hour cap covers original trajectory generation, teacher feedback, state replay, paired student continuations, parameter updates, waiting, and checkpoint saving.

\small\begin{array}{lrcrrcr} \hline \textbf{Method}&&\textbf{Resource Use}&&&\textbf{WebShop}&\cr \hline &\textbf{Training Time (h)}&\textbf{Updates}&\textbf{Environment Steps}&\textbf{SR (\\%) ↑}&\textbf{Score ↑}&\textbf{Rounds ↓}\cr \hline \text{Vanilla OPD}&5.58±0.01&299.0±5.0&\text{15,665 ± 260.0}&34.0±2.0&33.0±1.9&6.2±0.2\cr \text{Teacher-success upweighting}&5.58±0.01&199.0±1.0&\text{19,900 ± 80.0}&40.0±1.0&38.8±1.2&5.9±0.2\cr \text{OG-OPD}&5.59±0.01&202.0±3.0&\text{20,106 ± 100.0}&\boldsymbol{45.0±1.0}&\boldsymbol{43.8±1.3}&\boldsymbol{5.7±0.2}\cr \hline \end{array}

Under the same time cap, vanilla OPD makes 299 updates on average, 97 more than OG-OPD. OG-OPD still achieves 45.0% mean SR, 11.0 percentage points above vanilla OPD’s 34.0%.

2. OG-OPD leads at the same environment-step cap. The 20,000-step cap includes all actual interactions from original student trajectories, both branches, and state replay. We count each step once, including the first action of each branch, steps that yield no valid calibration result, and steps executed but not yet used for an update.

\small\begin{array}{lrcrrcr} \hline \textbf{Method}&&\textbf{Resource Use}&&&\textbf{WebShop}&\cr \hline &\textbf{Training Time (h)}&\textbf{Updates}&\textbf{Environment Steps}&\textbf{SR (\\%) ↑}&\textbf{Score ↑}&\textbf{Rounds ↓}\cr \hline \text{Vanilla OPD}&7.11±0.09&381.0±5.0&\text{19,961 ± 35.5}&36.0±2.0&35.0±2.0&6.1±0.2\cr \text{Teacher-success upweighting}&5.61±0.01&200.0±1.0&\text{19,973 ± 24.3}&39.3±1.5&38.2±1.3&5.9±0.2\cr \text{OG-OPD}&5.54±0.03&200.0±3.0&\text{19,907 ± 80.5}&\boldsymbol{45.3±3.1}&\boldsymbol{43.4±1.4}&\boldsymbol{5.7±0.2}\cr \hline \end{array}

Under the same interaction cap, vanilla OPD allocates its budget to more original trajectories and averages 381 updates, about 1.9× as many as OG-OPD. Despite spending some interactions on calibration, OG-OPD reaches 45.3% SR, 9.3 percentage points above vanilla OPD’s 36.0%.

On WebShop, OG-OPD achieves higher SR and scores with fewer rounds, even when the budget accounts for calibration and vanilla OPD can make more updates. With the same procedure for paired student continuations, OG-OPD also achieves a mean SR 5.0 percentage points higher than Teacher-success upweighting under the time cap and 6.0 points higher under the total interaction cap. This suggests that checking whether the student can already succeed after its original action helps identify where extra supervision is useful. In the revised version, Section 4.4 summarizes the two budget comparisons, and Appendix F.2 reports the resource accounting and full results in Table 10.

AuthorsOpenReview ↗

Response to Reviewer 2L4i | Weakness 2. Candidate selection and threshold sensitivity

We apply the same looser and stricter candidate settings to ScienceWorld and WebShop to see whether OG-OPD still outperforms vanilla OPD. On ScienceWorld, we stratify evaluation tasks by reference path length, defined as the number of actions until the gold action sequence first succeeds. We also append controlled text distractors to test whether observation noise widens the performance gap between default and stricter screening.

1. Gains persist across nearby thresholds and weight floors.

OG-OPD computes dynamic quantile thresholds within each training batch, so actual cutoffs vary with the batch distribution even when quantiles are fixed. Each valid trajectory has the same total statistical weight in the gap threshold, preventing longer trajectories from dominating simply because they have more turns. Only valid positive changes contribute to the abrupt-increase threshold. With the weight floor fixed at 1.50, we jointly vary the default gap and abrupt-increase quantiles (0.5, 0.6) to (0.4, 0.5) or (0.6, 0.7). We also fix the default quantiles and raise only the floor from 1.50 to 1.75. We use a Qwen3-32B teacher and Qwen3-1.7B student, evaluate at step 200, and report the mean ± sample standard deviation over three training seeds.

\small\begin{array}{lrcrcrcr} \hline \textbf{Method}&&\kern 0.282em\textbf{Screening Parameters}\kern -0.282em&&\kern 2.637em\textbf{ScienceWorld}\kern -2.637em&&\kern 2.587em\textbf{WebShop}\kern -2.587em&\cr \hline &\textbf{Gap}&\textbf{Increase}&\textbf{Floor}&\textbf{SR (\\%) ↑}&\textbf{Score ↑}&\textbf{SR (\\%) ↑}&\textbf{Score ↑}\cr \hline \text{Vanilla OPD}&\text{—}&\text{—}&\text{—}&13.0±1.4&48.1±1.2&28.0±2.6&27.5±2.6\cr \text{OG-OPD}&0.5&0.6&1.50&16.6±0.6&52.0±0.7&45.7±1.2&44.3±1.5\cr \text{OG-OPD (looser)}&0.4&0.5&1.50&16.8±0.7&52.2±0.8&46.3±1.5&44.7±1.2\cr \text{OG-OPD (stricter)}&0.6&0.7&1.50&16.0±0.8&51.2±1.0&44.0±2.0&42.9±1.8\cr \text{OG-OPD (higher weight floor)}&0.5&0.6&1.75&16.7±0.7&52.1±0.8&46.7±1.2&44.6±1.3\cr \hline \end{array}

Looser screening improves mean SR over the default by 0.2 and 0.6 percentage points on ScienceWorld and WebShop, whereas stricter screening reduces it by 0.6 and 1.7 points. Raising only the weight floor to 1.75 improves mean SR by 0.1 and 1.0 points. Stricter screening has the largest effect. Even under that setting, OG-OPD leads vanilla OPD by 3.0 and 16.0 percentage points, respectively.

2. Stricter screening hurts longer-path tasks more.

We partition the 1,308 ScienceWorld evaluation tasks by the number of actions until the fixed reference path first succeeds. The resulting groups contain 693 shorter tasks (≤26 steps) and 615 longer tasks (>26 steps). For each configuration and training seed, we compute SR for both groups from per-task results from the same trained model, then aggregate over three seeds.

\small\begin{array}{lrcr} \hline \textbf{Method}&&\textbf{ScienceWorld · SR (\\%) ↑}&\cr \hline &\textbf{Shorter}&\textbf{Longer}&\textbf{Overall}\cr \hline \text{Vanilla OPD}&16.6±2.0&8.9±0.7&13.0±1.4\cr \text{OG-OPD}&21.5±0.7&11.0±0.5&16.6±0.6\cr \text{OG-OPD (looser)}&21.7±0.8&11.3±0.8&16.8±0.7\cr \text{OG-OPD (stricter)}&21.6±0.9&9.7±0.7&16.0±0.8\cr \hline \end{array}

OG-OPD exceeds vanilla OPD by 4.9 and 2.1 percentage points on the shorter and longer groups, respectively. With shorter reference paths, mean SR stays between 21.5% and 21.7% across the three settings. With longer paths, it drops from 11.3% under looser screening to 9.7% under stricter screening. Relative to the default, stricter screening leaves the shorter group nearly unchanged and reduces SR by 1.3 points in the longer group. Within the tested task lengths, the longer group accounts for most of the overall SR decrease under stricter screening.

AuthorsOpenReview ↗

Response to Reviewer 2L4i | Weakness 2 (continued). Sensitivity to observation noise

3. Text distractors lower SR under both settings.

We append 64 tokens of task-irrelevant text to each ScienceWorld observation. The template, length, and injection frequency are fixed. We ensure essential task information is not truncated and retain the same legal actions and success criteria. The teacher and student receive the same perturbed observation at each state. Training, evaluation, state replay, and paired student continuations follow the same reproducible perturbation protocol. We run three training seeds per configuration, for six runs total.

\small\begin{array}{lcr} \hline \textbf{Observation Condition}&\kern 3.150em\textbf{ScienceWorld · SR (\\%) ↑}\kern -3.150em&\cr \hline &\textbf{Default}&\textbf{Stricter}\cr \hline \text{Original ScienceWorld}&16.6±0.6&16.0±0.8\cr \text{Task-Irrelevant Text}&14.8±0.7&14.1±0.8\cr \hline \end{array}

Text distractors reduce mean SR by 1.8 and 1.9 percentage points for default and stricter screening, respectively, changing the gap between the two from 0.6 to 0.7 points. Overall performance declines, but the two configurations are affected similarly under this perturbation.

All three candidate-threshold settings retain an SR advantage over vanilla OPD in the original ScienceWorld and WebShop environments. Stricter screening reduces SR more on tasks with longer reference paths. Adding task-irrelevant text lowers SR under both settings, but the gap between them changes little. ScienceWorld and WebShop use different native action interfaces, so the gains with the same quantiles in both offer preliminary evidence that the thresholds can transfer across interfaces. In the revised version, Appendix F.4 reports the threshold and observation-noise comparisons in Tables 14 and 15, and Appendix F.5 reports the reference-path stratification in Table 16.

Reviewer_Mt67OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how teacher supervision can be better allocated in on-policy distillation for multi-turn agents. The key observation is that the size of the teacher–student gap at a given turn does not necessarily reflect whether following the teacher would improve the student’s eventual task outcome. To address this, the authors introduce Outcome-Guided On-Policy Distillation (OG-OPD), which uses gap-based signals to identify candidate turns and then compares paired continuations starting from the student action and the teacher action. If the teacher action leads to a better final outcome, the corresponding turn receives stronger supervision while optimization is still performed on the student’s original trajectory. Experiments show consistent improvements over vanilla OPD and several recent multi-turn distillation baselines.

Critical strengths
  1. The paper identifies a clear limitation of using teacher–student gaps alone to guide supervision in multi-turn OPD. The empirical analysis shows that large gaps are not always beneficial intervention points, while smaller gaps can still be important for final task outcomes.
  2. The proposed method is simple and well aligned with this motivation. OG-OPD directly checks whether a teacher action improves the student’s final outcome before increasing supervision at that turn, and the experiments show consistent gains over several strong OPD baselines.
Critical weaknesses
  1. OG-OPD uses final task outcomes as an additional supervision signal beyond the teacher feedback used in standard OPD. As a result, it is not fully clear how much of the gain comes from the proposed calibration method itself and how much comes from having access to this extra outcome feedback and additional rollouts. A simpler outcome-aware OPD baseline would make this comparison clearer.
  2. The calibration rule only adds extra weight when the teacher action allows the current student to succeed. However, a teacher action may still be correct and useful to learn even if the student fails later because it cannot yet handle the remaining steps. This may bias the method toward teacher guidance that is already within the current student’s capability and overlook harder but useful guidance.
Questions for authors
  1. Since the calibration uses only one pair of continuations at each selected turn, could the result be highly sensitive to rollout randomness and therefore unstable?
  2. The analysis shows that some small-gap turns can still matter for the final outcome, but candidate selection mainly focuses on large gaps and sudden increases. How often might this miss useful low-gap turns?
  3. All experiments use Qwen3 teacher–student pairs. What about other model families? Evaluating this would help show whether the method generalizes beyond a single model family.
  4. OG-OPD has about 1.4–1.5x the training cost of vanilla OPD. How does it compare with the baselines under the same training time or interaction budget?
  5. How sensitive are the results to the candidate thresholds and the weight used for positive outcomes?
9 comments
AuthorsOpenReview ↗

Response to Reviewer Mt67 | Weakness 1. Separating the calibration rule from extra outcome feedback

Thank you for suggesting a simpler outcome-aware baseline. To distinguish the effect of the calibration rule from that of access to task outcomes and additional rollouts, we use two comparisons. The first keeps the paired student continuation procedure fixed and varies only the upweighting rule, isolating the effect of outcome-based calibration. The second applies separate caps on training time and total environment interactions on WebShop to test whether OG-OPD retains its advantage when the budget includes the additional rollouts.

1. Comparing both outcomes improves upweighting. We introduced Teacher-success upweighting. At each selected turn, both methods start two branches from the same state, execute the student’s original action in one and the teacher-proposed action in the other, and let the same frozen student policy continue both. Both methods obtain the final task outcome of each branch. Teacher-success upweighting gives extra weight whenever the branch following the teacher-proposed action succeeds; OG-OPD additionally requires the branch following the student’s original action to fail. Both methods use the same candidate selection, trajectory-relative weighting, teacher response evaluation, weight floor, and token-normalized distillation objective. They also use the same remaining interaction budget, decoding settings, and seed protocol for each pair.

We use a Qwen3-32B teacher and a Qwen3-1.7B student, evaluate at step 200, and report the mean ± sample standard deviation over three training seeds.

\small\begin{array}{lrcrrcr} \hline \textbf{Method}&&\textbf{ScienceWorld}&&&\textbf{WebShop}&\cr \hline &\textbf{SR (\\%) ↑}&\textbf{Score ↑}&\textbf{Rounds ↓}&\textbf{SR (\\%) ↑}&\textbf{Score ↑}&\textbf{Rounds ↓}\cr \hline \text{Vanilla OPD}&13.0\pm 1.4&48.1\pm 1.2&12.6\pm 0.3&28.0\pm 2.6&27.5\pm 2.6&6.8\pm 0.2\cr \text{Upweighting without verification}&14.9\pm 0.9&49.9\pm 1.1&12.3\pm 0.2&35.0\pm 2.0&34.5\pm 2.1&6.1\pm 0.2\cr \text{Teacher-success upweighting}&15.2\pm 0.8&50.7\pm 0.9&12.0\pm 0.2&40.3\pm 1.5&39.1\pm 1.4&5.9\pm 0.2\cr \textbf{OG-OPD}&\boldsymbol{16.6\pm 0.6}&\boldsymbol{52.0\pm 0.7}&\boldsymbol{11.6\pm 0.2}&\boldsymbol{45.7\pm 1.2}&\boldsymbol{44.3\pm 1.5}&\boldsymbol{5.6\pm 0.2}\cr \hline \end{array}

Compared with upweighting without verification, Teacher-success upweighting improves mean SR by 0.3 and 5.3 percentage points on ScienceWorld and WebShop, respectively. Under the same procedure for paired student continuations, requiring the branch following the student’s original action to fail gives OG-OPD a further gain of 1.4 and 5.4 points over Teacher-success upweighting, with higher scores and fewer rounds on both benchmarks.

2. OG-OPD leads under both budget caps. On WebShop, we round up the reference cost of 200 OG-OPD updates to set separate caps of 5.60 training hours and 20,000 environment steps. This lets vanilla OPD use the resources saved on calibration for additional updates. Training time includes everything from trajectory generation to checkpoint saving, including paired student continuations and waiting during training, but excludes separate evaluation. Environment steps include actual interactions from original trajectories, state replay, and both branches, including their initial actions and interactions that did not yield valid calibration results. All three methods use the same hardware, initialization, training tasks, and evaluation protocol. For each seed, we apply the relevant cap independently and evaluate the latest saved checkpoint from a completed update within that cap.

\small\begin{array}{lrcr} \hline \textbf{WebShop Budget Cap}&&\textbf{WebShop · SR (\\%) ↑}&\cr \hline &\textbf{Vanilla OPD}&\textbf{Teacher-success upweighting}&\textbf{OG-OPD}\cr \hline \text{5.60 training hours}&34.0\pm 2.0&40.0\pm 1.0&\boldsymbol{45.0\pm 1.0}\cr \text{20,000 environment steps}&36.0\pm 2.0&39.3\pm 1.5&\boldsymbol{45.3\pm 3.1}\cr \hline \end{array}

Under the time and total interaction caps, OG-OPD exceeds vanilla OPD by 11.0 and 9.3 percentage points in mean SR, respectively, and Teacher-success upweighting by 5.0 and 6.0 points, respectively.

Together, these comparisons show that checking whether the student can succeed after its original action helps identify where extra supervision is useful. On WebShop, OG-OPD’s gains persist when the common budget includes the cost of obtaining task outcomes. In the revised version, Section 4.4 summarizes both comparisons. Appendix F.1 reports the Teacher-success upweighting control in Table 8, and Appendix F.2 reports the budget comparisons in Table 10.

AuthorsOpenReview ↗

Response to Reviewer Mt67 | Weakness 2. Useful teacher guidance beyond the student’s current capabilities

A teacher-proposed action may be correct and valuable to learn even when the current student cannot complete the remaining steps. OG-OPD keeps base teacher supervision and uses the paired outcomes only to decide where to add weight. It first sets each turn’s pre-calibration weight through trajectory-relative weighting, then calibrates that weight using paired student continuations. At valid turns where teacher rescue is not observed, including when the branch following the teacher-proposed action ultimately fails, OG-OPD retains the pre-calibration weight. It neither removes the distillation loss nor further reduces that weight. Teacher rescue is evidence that the current student can benefit from the teacher-proposed action; it is not a complete judgment of the action’s correctness or long-term learning value.

The teacher-only completion analysis in Section 3.1 illustrates this distinction. After the same teacher-proposed action, the teacher often completes the task, but the current student rarely does. A teacher-proposed action can therefore be part of a successful solution without immediately helping the current student finish the task. This does not mean the guidance is not worth learning. As training progresses, OG-OPD evaluates newly collected turns with the updated student policy and can assign them extra weight if it observes teacher rescue. In the revised version, Section 3.3 explains the default weighting rule, and Appendix F.3 discusses how the rule continues to evaluate newly collected turns as the student improves.

AuthorsOpenReview ↗

Response to Reviewer Mt67 | Q1. Sensitivity to rollout randomness

To assess single-pair calibration, we repeated paired student continuations at fixed turns to measure how often teacher rescue recurs, then trained with more pairs per turn to assess the effect on final performance and execution cost.

1. Repeated trials often confirm teacher rescue.

We use a Qwen3-32B teacher and a Qwen3-1.7B student. On each benchmark, ScienceWorld and WebShop, we randomly sample 100 turns at each of training steps 20, 100, and 180. All sampled turns pass candidate selection and teacher response evaluation. For each turn, we fix the student checkpoint, environment state and history, the student’s original response and the teacher response, decoding settings, and the remaining interaction budget, then collect ten valid pairs. The same frozen student policy continues both branches, using matched seeds within a pair and independent seeds across repeats. Replay and runtime failures are recorded separately and are not counted as task failures.

We record teacher rescue when the branch following the student’s original action fails and the branch following the teacher-proposed action succeeds. Rescue frequency is the proportion of valid pairs that show teacher rescue. To compute label disagreement, we compare the rescue labels across all 45 distinct pairs of repeats at each turn, then average the disagreement rates across turns. For positive reconfirmation, we take each repeat that shows teacher rescue and compare it with the other nine repeats at the same turn. We pool these comparisons and report the fraction in which the other repeat also shows teacher rescue.

\small\begin{array}{lrrrr} \hline \textbf{Benchmark}&\textbf{Training Step}&\textbf{Rescue Frequency}&\textbf{Label disagreement ↓}&\textbf{Positive reconfirmation ↑}\cr \hline \text{ScienceWorld}&\text{20}&\text{5.8\\%}&\text{4.8\\%}&\text{58.2\\%}\cr \text{ScienceWorld}&\text{100}&\text{7.3\\%}&\text{5.1\\%}&\text{64.8\\%}\cr \text{ScienceWorld}&\text{180}&\text{6.9\\%}&\text{4.1\\%}&\text{70.2\\%}\cr \text{WebShop}&\text{20}&\text{9.1\\%}&\text{7.4\\%}&\text{59.1\\%}\cr \text{WebShop}&\text{100}&\text{11.7\\%}&\text{8.0\\%}&\text{65.9\\%}\cr \text{WebShop}&\text{180}&\text{10.9\\%}&\text{6.2\\%}&\text{71.6\\%}\cr \hline \end{array}

Rescue events are sparse, which partly explains the low overall disagreement rates. Positive reconfirmation therefore provides a more direct measure of the reproducibility of the signal that triggers upweighting. Positive reconfirmation ranges from 58.2% to 71.6%. A single observation is therefore informative but not fully reproducible.

2. More pairs yield modest gains at higher cost.

We compare one, three, and five pairs per selected turn to test whether reducing local variability improves training. With multiple pairs, we scale the full increment that a single rescue would trigger by the fraction of pairs at that turn that show rescue. For a fixed turn, responses, student policy, and pre-calibration weight, averaging independent repeats preserves the expected calibration increment while reducing sampling variance. We evaluate at step 200 and report the mean ± sample standard deviation over three training seeds.

\small\begin{array}{lrcrrrcrr} \hline \textbf{Pairs per Turn}&&\textbf{ScienceWorld}&&&&\textbf{WebShop}&&\cr \hline &\textbf{SR (\\%) ↑}&\textbf{Score ↑}&\textbf{Rounds ↓}&\textbf{Time (Relative; h)}&\textbf{SR (\\%) ↑}&\textbf{Score ↑}&\textbf{Rounds ↓}&\textbf{Time (Relative; h)}\cr \hline \text{1 (default)}&16.6±0.6&52.0±0.7&11.6±0.2&\text{1.00× (≈12.42 h)}&45.7±1.2&44.3±1.5&5.6±0.2&\text{1.00× (≈5.54 h)}\cr \text{3}&16.7±0.5&52.1±0.7&11.5±0.2&\text{1.05× (13.04 h)}&45.7±0.6&44.6±1.4&5.6±0.2&\text{1.07× (5.93 h)}\cr \text{5}&16.9±0.5&52.3±0.6&11.5±0.2&\text{1.10× (13.66 h)}&46.3±1.2&44.9±1.4&5.5±0.2&\text{1.12× (6.20 h)}\cr \hline \end{array}

With the same candidate thresholds, three pairs improve mean ScienceWorld SR by 0.1 percentage points; WebShop SR is 45.7 ± 0.6%, matching the single-pair mean of 45.7%. Five pairs improve mean SR by 0.3 and 0.6 points, respectively, and slightly improve scores on both benchmarks. More repeats yield modest gains, with no consistent, substantial reduction in variability across training runs. Increasing from one to five pairs raises the number of continuations per selected turn from two to ten and increases total training time by about 10% on ScienceWorld and 12% on WebShop.

A single observation of teacher rescue is noisy, but more pairs yielded only modest improvements in the tested settings. One pair achieved comparable final performance with fewer continuations and lower training time, so we retain it as the default. In the revised version, Appendix F.3 reports the fixed-turn repeatability diagnostics in Table 11 and the training and cost comparisons in Table 12.

Reviewer_tAmgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when teacher guidance is useful for multi-turn agents. It proposes OG-OPD, which reweights on-policy distillation across turns and uses paired student continuations to identify teacher interventions that lead to success. These outcomes increase the distillation weight on the student's original response, without adding the teacher branch to the training data. Experiments cover ALFWorld, ScienceWorld and WebShop.

Critical strengths
  1. The motivation is clear. A teacher-preferred action can be difficult for the current student to follow up, so comparing both actions under the same student continuation is useful.
  2. The paper evaluates two teacher–student settings and reports three-seed results, component ablations and training costs. The WebShop improvement for the smaller student is substantial.
Critical weaknesses
  1. My concern is whether outcome-based calibration retains its advantage at equal cost. The reported per-step cost is 1.42–1.48 times that of vanilla OPD for the smaller student. The fixed-step comparison supports the complete method, but does not isolate its benefit under equal total training or interaction budgets.
  2. The rescue rule is intentionally one-sided: a successful teacher intervention increases the weight, while harm does not decrease it. Appendix A correctly distinguishes rescue probability from net benefit. With only one pair tested at a selected position, the reliability of this selection rule remains unclear.
Questions for authors
  1. How does OG-OPD compare with vanilla OPD and an outcome-aware baseline under matched total training and continuation budgets?
  2. Does the gain remain when the number of upweighted turns and the weight distribution are matched, but selected turns or outcome labels are shuffled?
  3. Across repeated paired continuations, how stable is the rescue signal, and would explicitly downweighting harmful teacher interventions improve training?
5 comments
AuthorsOpenReview ↗

Response to Reviewer tAmg | Weakness 1 + Q1. Matched training and interaction budgets

To test whether OG-OPD’s gains persist after accounting for the cost of paired student continuations, we compare methods on WebShop under separate caps on training time and total environment steps. Vanilla OPD can use the resources saved on calibration for more updates. We also include Teacher-success upweighting, which follows the same procedure for paired student continuations, to test whether a different upweighting rule changes performance under the same budget.

All methods use the same budget limits. We use a Qwen3-32B teacher and a Qwen3-1.7B student. All three methods use the same hardware, initialization, training tasks, and evaluation protocol. We set the caps by rounding up the resources used for 200 updates with default OG-OPD, to 5.60 hours and 20,000 environment steps. The caps are applied separately, so methods can perform different numbers of updates. For each seed, we use cumulative resource logs to retrospectively select the latest saved checkpoint from a completed update within the relevant cap.

Teacher-success upweighting and OG-OPD use the same candidate selection, pre-calibration weights, teacher response evaluation, and weight floor. The same frozen student policy continues both branches, one after the student’s original action and the other after the teacher-proposed action. Teacher-success upweighting gives extra weight whenever the latter branch succeeds; OG-OPD additionally requires the former branch to fail.

1. OG-OPD leads at the same training-time cap. The 5.60-hour cap covers original trajectory generation, teacher feedback, state replay, paired student continuations, parameter updates, waiting, and checkpoint saving.

\hline \textbf{Method}&&\textbf{Resource Use}&&&\textbf{WebShop}&\cr \hline &\textbf{Training Time (h)}&\textbf{Updates}&\textbf{Environment Steps}&\textbf{SR (\\%) ↑}&\textbf{Score ↑}&\textbf{Rounds ↓}\cr \hline \text{Vanilla OPD}&5.58 \pm 0.01&299.0 \pm 5.0&15{,}665 \pm 260.0&34.0 \pm 2.0&33.0 \pm 1.9&6.2 \pm 0.2\cr \text{Teacher-success upweighting}&5.58 \pm 0.01&199.0 \pm 1.0&19{,}900 \pm 80.0&40.0 \pm 1.0&38.8 \pm 1.2&5.9 \pm 0.2\cr \text{OG-OPD}&5.59 \pm 0.01&202.0 \pm 3.0&20{,}106 \pm 100.0&\boldsymbol{45.0 \pm 1.0}&\boldsymbol{43.8 \pm 1.3}&\boldsymbol{5.7 \pm 0.2}\cr \hline \end{array}$$ Under the same time cap, vanilla OPD makes 299 updates on average, 97 more than OG-OPD. OG-OPD still achieves 45.0% mean SR, 11.0 percentage points above vanilla OPD’s 34.0%. **2. OG-OPD leads at the same environment-step cap.** The 20,000-step cap includes all actual interactions from original student trajectories, both branches, and state replay. We count each step once, including the first action of each branch, steps that yield no valid calibration result, and steps executed but not yet used for an update. $$\small\begin{array}{lrcrrcr} \hline \textbf{Method}&&\textbf{Resource Use}&&&\textbf{WebShop}&\cr \hline &\textbf{Training Time (h)}&\textbf{Updates}&\textbf{Environment Steps}&\textbf{SR (\\%) ↑}&\textbf{Score ↑}&\textbf{Rounds ↓}\cr \hline \text{Vanilla OPD}&7.11 \pm 0.09&381.0 \pm 5.0&19{,}961 \pm 35.5&36.0 \pm 2.0&35.0 \pm 2.0&6.1 \pm 0.2\cr \text{Teacher-success upweighting}&5.61 \pm 0.01&200.0 \pm 1.0&19{,}973 \pm 24.3&39.3 \pm 1.5&38.2 \pm 1.3&5.9 \pm 0.2\cr \text{OG-OPD}&5.54 \pm 0.03&200.0 \pm 3.0&19{,}907 \pm 80.5&\boldsymbol{45.3 \pm 3.1}&\boldsymbol{43.4 \pm 1.4}&\boldsymbol{5.7 \pm 0.2}\cr \hline \end{array}$$ Under the same interaction cap, vanilla OPD allocates its budget to more original trajectories and averages 381 updates, about 1.9× as many as OG-OPD. Despite spending some interactions on calibration, OG-OPD reaches 45.3% SR, 9.3 percentage points above vanilla OPD’s 36.0%. On WebShop, OG-OPD achieves higher SR and scores with fewer rounds, even when the budget accounts for calibration and vanilla OPD can make more updates. With the same procedure for paired student continuations, OG-OPD also achieves a mean SR 5.0 percentage points higher than Teacher-success upweighting under the time cap and 6.0 points higher under the total interaction cap. This suggests that checking whether the student can already succeed after its original action helps identify where extra supervision is useful. In the revised version, Section 4.4 summarizes the two budget comparisons, and Appendix F.2 reports the resource accounting and full results in Table 10.
AuthorsOpenReview ↗

Response to Reviewer tAmg | Weakness 2. Reliability of the one-sided rescue rule

To assess single-pair calibration, we repeated paired student continuations at fixed turns to measure how often teacher rescue recurs, then trained with more pairs per turn to assess the effect on final performance and execution cost.

1. Repeated trials often confirm teacher rescue.

We use a Qwen3-32B teacher and a Qwen3-1.7B student. On each benchmark, ScienceWorld and WebShop, we randomly sample 100 turns at each of training steps 20, 100, and 180. All sampled turns pass candidate selection and teacher response evaluation. For each turn, we fix the student checkpoint, environment state and history, the student’s original response and the teacher response, decoding settings, and the remaining interaction budget, then collect ten valid pairs. The same frozen student policy continues both branches, using matched seeds within a pair and independent seeds across repeats. Replay and runtime failures are recorded separately and are not counted as task failures.

We record teacher rescue when the branch following the student’s original action fails and the branch following the teacher-proposed action succeeds. Rescue frequency is the proportion of valid pairs that show teacher rescue. To compute label disagreement, we compare the rescue labels across all 45 distinct pairs of repeats at each turn, then average the disagreement rates across turns. For positive reconfirmation, we take each repeat that shows teacher rescue and compare it with the other nine repeats at the same turn. We pool these comparisons and report the fraction in which the other repeat also shows teacher rescue.

\hline \textbf{Benchmark}&\textbf{Training Step}&\textbf{Rescue Frequency}&\textbf{Label disagreement ↓}&\textbf{Positive reconfirmation ↑}\cr \hline \text{ScienceWorld}&\text{20}&\text{5.8\\%}&\text{4.8\\%}&\text{58.2\\%}\cr \text{ScienceWorld}&\text{100}&\text{7.3\\%}&\text{5.1\\%}&\text{64.8\\%}\cr \text{ScienceWorld}&\text{180}&\text{6.9\\%}&\text{4.1\\%}&\text{70.2\\%}\cr \text{WebShop}&\text{20}&\text{9.1\\%}&\text{7.4\\%}&\text{59.1\\%}\cr \text{WebShop}&\text{100}&\text{11.7\\%}&\text{8.0\\%}&\text{65.9\\%}\cr \text{WebShop}&\text{180}&\text{10.9\\%}&\text{6.2\\%}&\text{71.6\\%}\cr \hline \end{array}$$ Rescue events are sparse, which partly explains the low overall disagreement rates. Positive reconfirmation therefore provides a more direct measure of the reproducibility of the signal that triggers upweighting. Positive reconfirmation ranges from 58.2% to 71.6%. A single observation is therefore informative but not fully reproducible. **2. More pairs yield modest gains at higher cost.** We compare one, three, and five pairs per selected turn to test whether reducing local variability improves training. With multiple pairs, we scale the full increment that a single rescue would trigger by the fraction of pairs at that turn that show rescue. For a fixed turn, responses, student policy, and pre-calibration weight, averaging independent repeats preserves the expected calibration increment while reducing sampling variance. We evaluate at step 200 and report the mean ± sample standard deviation over three training seeds. $$\small\begin{array}{lrcrrrcrr} \hline \textbf{Pairs per Turn}&&\textbf{ScienceWorld}&&&&\textbf{WebShop}&&\cr \hline &\textbf{SR (\\%) ↑}&\textbf{Score ↑}&\textbf{Rounds ↓}&\textbf{Time (Relative; h)}&\textbf{SR (\\%) ↑}&\textbf{Score ↑}&\textbf{Rounds ↓}&\textbf{Time (Relative; h)}\cr \hline \text{1 (default)}&16.6\pm 0.6&52.0\pm 0.7&11.6\pm 0.2&\text{1.00× (≈12.42 h)}&45.7\pm 1.2&44.3\pm 1.5&5.6\pm 0.2&\text{1.00× (≈5.54 h)}\cr \text{3}&16.7\pm 0.5&52.1\pm 0.7&11.5\pm 0.2&\text{1.05× (13.04 h)}&45.7\pm 0.6&44.6\pm 1.4&5.6\pm 0.2&\text{1.07× (5.93 h)}\cr \text{5}&16.9\pm 0.5&52.3\pm 0.6&11.5\pm 0.2&\text{1.10× (13.66 h)}&46.3\pm 1.2&44.9\pm 1.4&5.5\pm 0.2&\text{1.12× (6.20 h)}\cr \hline \end{array}$$ With the same candidate thresholds, three pairs improve mean ScienceWorld SR by 0.1 percentage points; WebShop SR is 45.7 ± 0.6%, matching the single-pair mean of 45.7%. Five pairs improve mean SR by 0.3 and 0.6 points, respectively, and slightly improve scores on both benchmarks. More repeats yield modest gains, with no consistent, substantial reduction in variability across training runs. Increasing from one to five pairs raises the number of continuations per selected turn from two to ten and increases total training time by about 10% on ScienceWorld and 12% on WebShop. A single observation of teacher rescue is noisy, but more pairs yielded only modest improvements in the tested settings. One pair achieved comparable final performance with fewer continuations and lower training time, so we retain it as the default. In the revised version, Appendix F.3 reports the fixed-turn repeatability diagnostics in Table 11 and the training and cost comparisons in Table 12.
AuthorsOpenReview ↗

Response to Reviewer tAmg | Q2. Shuffling under matched turn weights

Thank you for suggesting a matched-weight control. On WebShop, we use three shuffling controls to test whether the gains depend on assigning extra weight to turns where we observe teacher rescue. All three retain trajectory-relative weighting, candidate selection, teacher response evaluation, and paired student continuations. In Shuffled outcome labels, we permute the teacher rescue labels and recompute each increment using the receiving turn’s pre-calibration weight. This preserves label counts, but the increments can change. In Shuffled calibration increments, we permute the increments themselves, including zeros. This preserves the number of upweighted turns and the distribution of increments, but can change the distribution of final weights. Matched-weight shuffling permutes rescue labels only among turns with identical pre-calibration weights. It preserves the number of upweighted turns, the increment distribution, and the final turn-weight distribution while changing which turns receive some of the increments. We also measure how often an upweighting slot actually moves to a non-rescue turn, using all originally upweighted turns as the denominator. On average, 64.0% ± 4.0% of originally upweighted turns have an eligible non-rescue turn to swap with. In practice, 38.0% ± 3.0% of upweighting slots move to non-rescue turns.

We use a Qwen3-32B teacher and Qwen3-1.7B student, evaluate at step 200, and report the mean ± sample standard deviation over three training seeds.

\hline \textbf{Method}&&\textbf{WebShop}&\cr \hline &\textbf{SR (\\%) ↑}&\textbf{Score ↑}&\textbf{Rounds ↓}\cr \hline \text{Vanilla OPD}&28.0\pm 2.6&27.5\pm 2.6&6.8\pm 0.2\cr \text{Shuffled outcome labels}&37.7\pm 1.5&36.7\pm 1.6&6.1\pm 0.2\cr \text{Shuffled calibration increments}&39.0\pm 1.0&38.0\pm 1.2&5.9\pm 0.2\cr \text{Matched-weight shuffling}&41.0\pm 1.0&40.0\pm 1.3&5.8\pm 0.2\cr \text{OG-OPD}&\boldsymbol{45.7\pm 1.2}&\boldsymbol{44.3\pm 1.5}&\boldsymbol{5.6\pm 0.2}\cr \hline \end{array}$$ All three shuffling controls achieve higher mean WebShop SR than vanilla OPD but fall short of OG-OPD. Matched-weight shuffling offers the most direct test of whether the turns that receive extra weight matter. Moving about 38% of the slots to non-rescue turns yields a mean SR of 41.0%, 4.7 percentage points below OG-OPD’s 45.7%. The score is 4.3 points lower, and the mean number of rounds is 0.2 higher. Even with the same number of upweighted turns and the same final turn-weight distribution, assigning extra weight to turns where paired student continuations show teacher rescue is more effective in this setting than redistributing it among turns with identical pre-calibration weights. In the revised version, Section 4.4 summarizes the matched-weight control, and Appendix F.1 describes all three shuffling controls and reports their results in Table 9.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.