- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates on-policy distillation (OPD) for multi-turn autonomous agents and identifies a core "supervision-benefit mismatch": token-level distributional gaps between teacher and student fail to reliably signal actual task benefits. To address this, the authors propose Outcome-Guided On-Policy Distillation (OG-OPD). The method incorporates trajectory-relative weighting to unify supervision scale across turns, filters candidate turns via gap jumps, and performs outcome-based calibration using paired continuations with the current frozen student model. Supervision is selectively upweighted only when a teacher action allows the student to turn a failing trajectory into a success. Validation across ALFWorld, ScienceWorld, and WebShop across multiple model scales demonstrates consistent improvements in task success rate (e.g., up to +17.7% on WebShop) over Vanilla OPD and existing baselines.
Critical strengths
-
High-Leverage Insight into Multi-Turn Agent Distillation: The paper offers a compelling analysis of why classical token-level gaps fail in multi-turn environments. In sequential decision-making, small deviations early on cause compounding errors over steps. These small gaps often represent high-leverage learning points—fully within the student's current capacity yet decisive for task outcome. Conversely, large gaps frequently force the student into states it cannot handle (teacher-only completions), resulting in teacher harm. Grounding supervision in actual student continuations effectively resolves this dilemma.
-
Effective and Outcome-Grounded Calibration: OG-OPD provides a pragmatic solution. By evaluating paired continuations with the frozen current student, the method guarantees that upweighted teacher actions are actually executable by the student at its current learning stage. Furthermore, the candidate selection mechanism successfully filters out roughly 75% of unpromising turns, keeping training rollout overhead to a manageable level (~1.4×) while yielding substantial empirical gains.
Critical weaknesses
-
Increased Training Cost and Complexity: Although candidate selection prunes around 75% of turns, running paired rollouts using the student model still introduces a ~1.4× to 1.5× wall-clock training time overhead and requires managing environment resets and parallel continuations during training.
-
Heuristic Candidate Selection and Threshold Sensitivity: The candidate selection step relies on specific quantile thresholds for absolute gap and abrupt increase. The paper lacks a thorough sensitivity analysis showing how robust these thresholds are across environments with different episode lengths, noisy observation spaces, or varying action granularities.
Questions for authors
-
Recall of Outcome-Critical Small Gaps: What estimated proportion of outcome-critical small gaps are successfully captured by the abrupt increase criterion versus filtered out?
-
Potential for Early Stopping in Rollouts: Could the paired continuation overhead (~1.4×) be further reduced by implementing early stopping during rollouts when trajectory divergence becomes irreversible?
-
Threshold Robustness: How robust are the default quantile thresholds when transferred to environments with significantly longer horizons or sparse rewards?
7 comments
Response to Reviewer 2L4i | Weakness 2. Candidate selection and threshold sensitivity
We apply the same looser and stricter candidate settings to ScienceWorld and WebShop to see whether OG-OPD still outperforms vanilla OPD. On ScienceWorld, we stratify evaluation tasks by reference path length, defined as the number of actions until the gold action sequence first succeeds. We also append controlled text distractors to test whether observation noise widens the performance gap between default and stricter screening.
1. Gains persist across nearby thresholds and weight floors.
OG-OPD computes dynamic quantile thresholds within each training batch, so actual cutoffs vary with the batch distribution even when quantiles are fixed. Each valid trajectory has the same total statistical weight in the gap threshold, preventing longer trajectories from dominating simply because they have more turns. Only valid positive changes contribute to the abrupt-increase threshold. With the weight floor fixed at 1.50, we jointly vary the default gap and abrupt-increase quantiles (0.5, 0.6) to (0.4, 0.5) or (0.6, 0.7). We also fix the default quantiles and raise only the floor from 1.50 to 1.75. We use a Qwen3-32B teacher and Qwen3-1.7B student, evaluate at step 200, and report the mean ± sample standard deviation over three training seeds.
\small\begin{array}{lrcrcrcr} \hline \textbf{Method}&&\kern 0.282em\textbf{Screening Parameters}\kern -0.282em&&\kern 2.637em\textbf{ScienceWorld}\kern -2.637em&&\kern 2.587em\textbf{WebShop}\kern -2.587em&\cr \hline &\textbf{Gap}&\textbf{Increase}&\textbf{Floor}&\textbf{SR (\\%) ↑}&\textbf{Score ↑}&\textbf{SR (\\%) ↑}&\textbf{Score ↑}\cr \hline \text{Vanilla OPD}&\text{—}&\text{—}&\text{—}&13.0±1.4&48.1±1.2&28.0±2.6&27.5±2.6\cr \text{OG-OPD}&0.5&0.6&1.50&16.6±0.6&52.0±0.7&45.7±1.2&44.3±1.5\cr \text{OG-OPD (looser)}&0.4&0.5&1.50&16.8±0.7&52.2±0.8&46.3±1.5&44.7±1.2\cr \text{OG-OPD (stricter)}&0.6&0.7&1.50&16.0±0.8&51.2±1.0&44.0±2.0&42.9±1.8\cr \text{OG-OPD (higher weight floor)}&0.5&0.6&1.75&16.7±0.7&52.1±0.8&46.7±1.2&44.6±1.3\cr \hline \end{array}
Looser screening improves mean SR over the default by 0.2 and 0.6 percentage points on ScienceWorld and WebShop, whereas stricter screening reduces it by 0.6 and 1.7 points. Raising only the weight floor to 1.75 improves mean SR by 0.1 and 1.0 points. Stricter screening has the largest effect. Even under that setting, OG-OPD leads vanilla OPD by 3.0 and 16.0 percentage points, respectively.
2. Stricter screening hurts longer-path tasks more.
We partition the 1,308 ScienceWorld evaluation tasks by the number of actions until the fixed reference path first succeeds. The resulting groups contain 693 shorter tasks (≤26 steps) and 615 longer tasks (>26 steps). For each configuration and training seed, we compute SR for both groups from per-task results from the same trained model, then aggregate over three seeds.
\small\begin{array}{lrcr} \hline \textbf{Method}&&\textbf{ScienceWorld · SR (\\%) ↑}&\cr \hline &\textbf{Shorter}&\textbf{Longer}&\textbf{Overall}\cr \hline \text{Vanilla OPD}&16.6±2.0&8.9±0.7&13.0±1.4\cr \text{OG-OPD}&21.5±0.7&11.0±0.5&16.6±0.6\cr \text{OG-OPD (looser)}&21.7±0.8&11.3±0.8&16.8±0.7\cr \text{OG-OPD (stricter)}&21.6±0.9&9.7±0.7&16.0±0.8\cr \hline \end{array}
OG-OPD exceeds vanilla OPD by 4.9 and 2.1 percentage points on the shorter and longer groups, respectively. With shorter reference paths, mean SR stays between 21.5% and 21.7% across the three settings. With longer paths, it drops from 11.3% under looser screening to 9.7% under stricter screening. Relative to the default, stricter screening leaves the shorter group nearly unchanged and reduces SR by 1.3 points in the longer group. Within the tested task lengths, the longer group accounts for most of the overall SR decrease under stricter screening.
Response to Reviewer 2L4i | Weakness 2 (continued). Sensitivity to observation noise
3. Text distractors lower SR under both settings.
We append 64 tokens of task-irrelevant text to each ScienceWorld observation. The template, length, and injection frequency are fixed. We ensure essential task information is not truncated and retain the same legal actions and success criteria. The teacher and student receive the same perturbed observation at each state. Training, evaluation, state replay, and paired student continuations follow the same reproducible perturbation protocol. We run three training seeds per configuration, for six runs total.
\small\begin{array}{lcr} \hline \textbf{Observation Condition}&\kern 3.150em\textbf{ScienceWorld · SR (\\%) ↑}\kern -3.150em&\cr \hline &\textbf{Default}&\textbf{Stricter}\cr \hline \text{Original ScienceWorld}&16.6±0.6&16.0±0.8\cr \text{Task-Irrelevant Text}&14.8±0.7&14.1±0.8\cr \hline \end{array}
Text distractors reduce mean SR by 1.8 and 1.9 percentage points for default and stricter screening, respectively, changing the gap between the two from 0.6 to 0.7 points. Overall performance declines, but the two configurations are affected similarly under this perturbation.
All three candidate-threshold settings retain an SR advantage over vanilla OPD in the original ScienceWorld and WebShop environments. Stricter screening reduces SR more on tasks with longer reference paths. Adding task-irrelevant text lowers SR under both settings, but the gap between them changes little. ScienceWorld and WebShop use different native action interfaces, so the gains with the same quantiles in both offer preliminary evidence that the thresholds can transfer across interfaces. In the revised version, Appendix F.4 reports the threshold and observation-noise comparisons in Tables 14 and 15, and Appendix F.5 reports the reference-path stratification in Table 16.
Response to Reviewer 2L4i | Weakness 1. Training cost under matched budgets
To account for calibration costs, we compare methods on WebShop under two separate resource caps. We separately cap total training time and total environment interactions, allowing vanilla OPD to spend the resources saved on calibration on additional updates. We also include Teacher-success upweighting to test whether a different upweighting rule changes performance under the same caps.
All methods use the same budget limits. We use a Qwen3-32B teacher and a Qwen3-1.7B student. All three methods use the same hardware, initialization, training tasks, and evaluation protocol. We set the caps by rounding up the resources used for 200 updates with default OG-OPD, to 5.60 hours and 20,000 environment steps. The caps are applied separately, so methods can perform different numbers of updates. For each seed, we use cumulative resource logs to retrospectively select the latest saved checkpoint from a completed update within the relevant cap.
OG-OPD and Teacher-success upweighting use the same procedure for paired student continuations. They also share candidate selection, pre-calibration weights, teacher response evaluation, and the weight floor. To limit the cost of state restoration and two-branch execution, the procedure checks at most two candidates and pairs at most one turn per trajectory. We verify the starting state by replaying the prefix. From that state, the student’s original action and the teacher-proposed action are executed in separate environment instances. The same frozen student policy continues both branches. If replay or execution fails, calibration is skipped, but consumed time and environment steps still count toward the budget. Teacher-success upweighting requires only success after the teacher-proposed action, whereas OG-OPD also requires failure after the student’s original action.
1. OG-OPD leads at the same training-time cap. The 5.60-hour cap covers original trajectory generation, teacher feedback, state replay, paired student continuations, parameter updates, waiting, and checkpoint saving.
\small\begin{array}{lrcrrcr} \hline \textbf{Method}&&\textbf{Resource Use}&&&\textbf{WebShop}&\cr \hline &\textbf{Training Time (h)}&\textbf{Updates}&\textbf{Environment Steps}&\textbf{SR (\\%) ↑}&\textbf{Score ↑}&\textbf{Rounds ↓}\cr \hline \text{Vanilla OPD}&5.58±0.01&299.0±5.0&\text{15,665 ± 260.0}&34.0±2.0&33.0±1.9&6.2±0.2\cr \text{Teacher-success upweighting}&5.58±0.01&199.0±1.0&\text{19,900 ± 80.0}&40.0±1.0&38.8±1.2&5.9±0.2\cr \text{OG-OPD}&5.59±0.01&202.0±3.0&\text{20,106 ± 100.0}&\boldsymbol{45.0±1.0}&\boldsymbol{43.8±1.3}&\boldsymbol{5.7±0.2}\cr \hline \end{array}
Under the same time cap, vanilla OPD makes 299 updates on average, 97 more than OG-OPD. OG-OPD still achieves 45.0% mean SR, 11.0 percentage points above vanilla OPD’s 34.0%.
2. OG-OPD leads at the same environment-step cap. The 20,000-step cap includes all actual interactions from original student trajectories, both branches, and state replay. We count each step once, including the first action of each branch, steps that yield no valid calibration result, and steps executed but not yet used for an update.
\small\begin{array}{lrcrrcr} \hline \textbf{Method}&&\textbf{Resource Use}&&&\textbf{WebShop}&\cr \hline &\textbf{Training Time (h)}&\textbf{Updates}&\textbf{Environment Steps}&\textbf{SR (\\%) ↑}&\textbf{Score ↑}&\textbf{Rounds ↓}\cr \hline \text{Vanilla OPD}&7.11±0.09&381.0±5.0&\text{19,961 ± 35.5}&36.0±2.0&35.0±2.0&6.1±0.2\cr \text{Teacher-success upweighting}&5.61±0.01&200.0±1.0&\text{19,973 ± 24.3}&39.3±1.5&38.2±1.3&5.9±0.2\cr \text{OG-OPD}&5.54±0.03&200.0±3.0&\text{19,907 ± 80.5}&\boldsymbol{45.3±3.1}&\boldsymbol{43.4±1.4}&\boldsymbol{5.7±0.2}\cr \hline \end{array}
Under the same interaction cap, vanilla OPD allocates its budget to more original trajectories and averages 381 updates, about 1.9× as many as OG-OPD. Despite spending some interactions on calibration, OG-OPD reaches 45.3% SR, 9.3 percentage points above vanilla OPD’s 36.0%.
On WebShop, OG-OPD achieves higher SR and scores with fewer rounds, even when the budget accounts for calibration and vanilla OPD can make more updates. With the same procedure for paired student continuations, OG-OPD also achieves a mean SR 5.0 percentage points higher than Teacher-success upweighting under the time cap and 6.0 points higher under the total interaction cap. This suggests that checking whether the student can already succeed after its original action helps identify where extra supervision is useful. In the revised version, Section 4.4 summarizes the two budget comparisons, and Appendix F.2 reports the resource accounting and full results in Table 10.