acceptodds
Under review as a conference paper at ICLR 2027

UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agents

Abstract

On-policy distillation (OPD) trains a student on its own rollouts using dense supervision from a teacher. In multi-turn environments, a mistake at a critical decision step can redirect the subsequent rollout toward poor outcomes. We use low teacher confidence on student actions to select high-uncertainty steps for correction. In a controlled ALFWorld study, a single teacher correction at a low-confidence step improves subsequent student behavior and task success, motivating selective intervention during distillation. We propose UOPD, an uncertainty-aware intervention method for on-policy distillation. At low-uncertainty turns, UOPD executes student actions and applies the standard OPD loss. At high-uncertainty turns, it samples and executes teacher actions and trains the student to imitate them through supervised fine-tuning, which minimizes forward Kullback–Leibler in expectation. UOPD adjusts an uncertainty threshold online to target a scheduled intervention rate. Empirically, we evaluate UOPD across a broad range of agentic tasks, including ALFWorld, WebShop, and Search, and student models of varying sizes, demonstrating its superior performance over OPD methods and their variants. UOPD improves WebShop score by up to relative to standard OPD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, large language model, reinforcement learning, agentic rl, knowledge distillation, multi-turn agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aeT7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work addresses the problem that early errors in multi-turn agent OPD can affect subsequent interaction trajectories. It proposes UOPD, which decides whether to intervene before execution based on the teacher's confidence in the student's candidate action. At an intervention step, the teacher's action is executed and learned through SFT, with the corresponding objective being forward KL. At a non-intervention step, the student's action is executed and the standard OPD loss is applied. UOPD consistently outperforms standard OPD on ALFWorld, WebShop, and multi-hop Search, while requiring fewer interaction rounds.

Critical strengths
  1. Teacher intervention before action execution not only changes the training objective at the current step, but also affects subsequent rollouts. Compared with methods such as SAGE-OPD, which adjust supervision signals based on existing trajectories, this design can directly change subsequent interaction states.

  2. UOPD outperforms standard OPD on ALFWorld, WebShop, and multi-hop Search. Table 2 further presents ablation studies on the intervention signal, teacher action execution, and SFT weight to analyze the impact of different components on performance.

Critical weaknesses
  1. The related work section discusses how SAGE-OPD uses teacher judgments and confidence to adjust turn-level supervision, but the main experiments do not directly compare against it. Although The work compares UOPD with related methods such as FTB-OPD, a comparison with SAGE-OPD under the same settings would still help determine to what extent the gains from UOPD come from teacher action replacement before execution, rather than selective supervision alone.

  2. The conclusions rely on a key assumption: the average advantage of the teacher's action over the student's action is positive at selected intervention steps. The experiments in Section 3 and Table 2 provide some evidence, but do not directly verify this theoretical assumption or sufficiently establish whether confidence can consistently identify steps with greater intervention gains across different tasks.

Questions for authors
  1. Could UOPD be directly compared with SAGE-OPD under the same settings to further clarify its independent contribution?

  2. Do the key experimental findings remain stable across multiple independent training seeds, especially the performance gains on multi-hop Search?

2 comments

We are very grateful for your valuable comments. Below, we summarize your comments in quotes and provide our point-by-point responses.

W1 / Q1 / Q2: A direct comparison with SAGE-OPD under the same settings is needed. Do the findings remain stable across independent training seeds, especially on multi-hop Search?

We thank the reviewer for this suggestion. We have added comparisons with SAGE-OPD and completed three independent training runs for each method on ALFWorld, WebShop, and multi-hop Search, using the same student and environment-specific teacher models. The table reports averages over the three training runs. ALFWorld and WebShop use 3B students, and Search uses the paper's 1.5B base student.

MethodALFWorld seen SR (%)ALFWorld unseen SR (%)WebShop scoreWebShop SR (%)Multihop Search EM (%)
OPD87.785.077.366.217.4
SAGE-OPD84.282.672.059.816.3
UOPD89.688.681.369.521.9

UOPD outperforms both baselines across all reported metrics. Compared with SAGE-OPD, it improves unseen ALFWorld success by 6.0 percentage points, WebShop score by 9.3 points, and multi-hop Search EM by 5.6 percentage points.

SAGE-OPD underperforms OPD in our setup. A practical challenge is its dependence on rubric-based teacher judgments. The teacher must label each student response as SKIP, WEAK, or STRONG. These labels determine whether and how strongly the turn receives distillation supervision. Our teachers are RL policies specialized for environment interaction. A teacher that acts effectively is not necessarily a reliable judge of another agent’s actions.

For example, in one SAGE-OPD training run on ALFWorld, the mean SKIP rate rose from 29.0% in the first 20 rollout batches to 68.7% in the last 20. Since SKIP assigns zero distillation weight, this reduces teacher supervision and may help explain SAGE-OPD’s weaker performance than OPD in our setting.

SAGE-OPD asks the teacher to explicitly judge whether a student turn needs supervision under a predefined rubric. UOPD instead uses the teacher's likelihood of the student response to select intervention turns and samples corrective actions directly from the teacher. This avoids requiring specialized expert policies to also perform rubric-based evaluation.

W2: The experiments do not directly verify whether selected teacher actions have positive average advantage, or whether confidence identifies larger intervention gains consistently across tasks.

We agree that aggregate final performance does not directly verify this assumption. The condition concerns whether a teacher correction improves downstream return under subsequent student continuation. We therefore directly test whether low teacher confidence identifies states with greater correction benefit at multiple training checkpoints.

Specifically, we evaluate OPD checkpoints at steps 25, 50, 75, and 100 on ALFWorld and WebShop. At each checkpoint, we freeze the student and collect student-only trajectories on 128 training tasks per benchmark. We score each student response under the teacher and select the lowest- and highest-confidence turns from the same trajectory.

At each selected state, we compare two branches with the same environment state and interaction prefix: one executes the original student proposal, while the other executes one teacher correction. Both branches then generate fresh continuations with the same frozen student, without further teacher assistance. We run three paired comparisons per task using different random seeds and average their results. The table reports correction gains relative to each selected state’s student-continuation control.

BenchmarkStepLow-confidence correction gainHigh-confidence correction gain
ALFWorld25+7.8+6.3
ALFWorld50+7.0+2.6
ALFWorld75+3.10.0
ALFWorld100+1.6+0.1
WebShop25+2.7+0.2
WebShop50+7.5+5.8
WebShop75+5.2+3.0
WebShop100+1.7+0.4

Low-confidence corrections yield positive average gains and outperform high-confidence corrections at every tested checkpoint on both benchmarks. This directly links teacher confidence to realized correction benefit, providing empirical support for the practical gate’s ability to identify beneficial intervention states. The smaller correction gains in later training also align with the rationale for our intervention schedule, which gradually reduces the intervention rate as the student becomes more capable.

Besides these replications and baseline comparisons, our response to Reviewer qD4L describes the action-only SFT ablation, and our response to Reviewer WENY describes the action-only uncertainty gate. These experiments address different questions about what the teacher supplies, how the correction is learned, and where intervention is selected.

Dear Reviewer,

As the discussion period is nearing its end, we wanted to check whether we have adequately addressed your concerns. If you have any remaining questions or feedback, please let us know. We would be happy to address them.

Thank you again for your time and valuable feedback!

Reviewer_qD4LOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles correcting uncertain local actions with a teacher called UOPD annealed over time. Performance improves and total turns taken decreased.

Critical strengths

Local action errors compound over time. Distilling the teacher’s selection can improve this which I find interesting. There are real statistical performance improvements from this method.

Critical weaknesses

The related-work mentions privileged context but has not engaged enough in methods to distill such. An empirical comparison would be even better. https://arxiv.org/abs/2602.04942. Further, the lack of training seed variation instead of eval seed variation and seed uncertainty intervals induces difficulty when reading the results. A random turn baseline instead would demonstrate directly the benefit of low-confidence turn interventions.

Questions for authors

How does UOPD compare to relate to privileged-information distillation? Perhaps performance is due to reasoning ability and length instead of the action selected. How much of the performance comes from reasoning tokens versus the executed action?

3 comments

We are very grateful for your valuable comments. Below, we summarize your comments in quotes and provide our point-by-point responses.

W1: The discussion of privileged-information distillation is insufficient, and an empirical comparison would be valuable.

Thank you for highlighting this related work. Our paper focuses on standard on-policy distillation, where a separately trained expert teacher is already available and remains fixed during student training. Within this setting, UOPD studies when the teacher should intervene during interaction and how the student should learn from the resulting corrections. The teacher and student receive the same interaction history, so our method does not require additional privileged information as teacher input.

Privileged Information Distillation for Language Models studies a related self-distillation setting: the teacher and student share model parameters, and additional training-time information conditions the teacher to provide a stronger learning signal.

The two approaches are complementary. UOPD's intervention mechanism could in principle use a privileged-information-conditioned teacher to score student proposals and supply corrective actions, while the student continues to act without privileged context. Our present scope is distillation from an existing expert teacher; extending selective intervention to privileged-information self-distillation is a natural direction. We will clarify this connection and the scope of our study in the revision.

W2: Evaluation-seed variation does not replace independent training-seed variation, and the uncertainty of the reported gains is unclear.

We have completed two additional independent training runs for each of OPD, UOPD, and the action-only SFT variant. The results are reported in our response to Q1.

W3: A random-turn intervention baseline would more directly demonstrate the benefit of selecting low-confidence turns.

We agree that this is an important control. Table 2 of the paper includes a random-turn intervention baseline that retains teacher execution and teacher SFT under the same target intervention schedule (30% to 5%). The random baseline samples each turn with the scheduled probability, while UOPD uses teacher confidence to select intervention turns.

Compared with random intervention, UOPD increases unseen ALFWorld success from 83.8% to 88.8%, and WebShop score from 77.3 to 82.4. These results highlight the contribution of intervention selection: both methods execute teacher actions and learn from them, yet targeting low-confidence turns yields better final performance.

Our additional checkpoint analysis, reported in our responses to Reviewers WENY and aeT7, directly examines why low-confidence selection is beneficial. At steps 25, 50, 75, and 100, we compare single-turn teacher corrections at low- and high-confidence turns, followed by frozen-student continuation. Low-confidence corrections yield larger average gains at every tested checkpoint on both benchmarks. These findings indicate that teacher confidence helps prioritize states where corrections provide greater downstream benefit, complementing the random-intervention training baseline.

Q1: How much of UOPD's performance comes from reasoning tokens versus the executed action? Could the gains be due to reasoning ability or response length?

We add an action-only SFT ablation that masks the loss on teacher reasoning tokens while retaining them as context for predicting action tokens. The intervention gate, teacher execution, generation budget, and overall teacher-supervision weight are unchanged. Results are averaged over three independent training runs.

Method / teacher SFT supervisionALFWorld seen SR (%)ALFWorld unseen SR (%)WebShop scoreWebShop SR (%)
OPD87.785.077.366.2
UOPD: action only88.587.379.468.4
UOPD89.688.681.369.5

Action-only SFT already outperforms OPD on both benchmarks, increasing unseen ALFWorld success from 85.0% to 87.3% and WebShop score from 77.3 to 79.4. This shows that supervising the teacher's corrective actions alone provides an effective learning signal.

Reasoning is also important because our agents generate reasoning before producing an action. In the action-only variant, the teacher still reasons before generating its action, and the student is trained to imitate that action. Supervising both reasoning and action achieves the best results, further increasing unseen ALFWorld success to 88.6% and WebShop score to 81.3. Learning both therefore teaches the student which action to take and the reasoning that leads to that action.

Besides the action-only SFT ablation above, our response to Reviewer WENY reports the action-only confidence gate and an analysis of correction gains. Our response to Reviewer aeT7 presents comparisons with SAGE-OPD and results from three independent training runs.

Reviewer_qD4LOpenReview ↗

Review

Thanks to the authors, this answers my concerns and I maintain my score

Thank you for the thoughtful feedback, and we are glad that our response has addressed your concerns.

Reviewer_WENYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UOPD selectively substitutes teacher actions at states judged uncertain and trains on the resulting mixed trajectories. It combines on-policy distillation with supervised learning on teacher-generated actions. The key idea is that intervention can improve both the states encountered during training and the supervision available at difficult decisions.

Critical strengths
  • The treatment of teacher actions is methodologically well motivated. Once actions are sampled from the teacher, simply applying the student-sampled distillation objective is no longer the same update. The separate supervised term addresses this change in sampling distribution.

  • The evidence goes beyond a final-score comparison. The frozen-student intervention study compares targeted and random corrections at the same correction count, while training ablations separate teacher execution from learning on teacher actions. These controls directly support the proposed combination of state-distribution improvement and supervision.

Critical weaknesses
  • The crucial assumption is not merely that the teacher disagrees with the student, but that replacing the student's action at a selected state improves return under subsequent student continuation. Low teacher likelihood can reflect this, but can also reflect differences in reasoning style or alternative valid actions. The theoretical recoverability condition states the relevant requirement; it does not establish that the practical gate identifies such states.

  • The frozen-student experiment is useful evidence for targeting, but leaves open whether the gate remains aligned with intervention benefit as training changes both the student and its visited states. The confidence-gap alternative and random-intervention training baseline are helpful existing controls. What is still missing is a direct connection between the gate's score and actual correction benefit throughout training, rather than only aggregate performance under different gates.

Questions for authors
  1. At early, middle, and late checkpoints, how does realized teacher-intervention benefit vary across gate-score quantiles? Paired continuations from the same state, using either the student or teacher action followed by student rollout, would directly test the recoverability assumption where the gate is used.

  2. With teacher execution, the supervised objective, and correction count fixed, how does an action-only gate compare with one scoring the full response, alongside the existing confidence-gap gate? This would help distinguish decision-critical uncertainty from disagreement about the reasoning text.

2 comments

We are very grateful for your valuable comments. Below, we summarize your comments in quotes and provide our point-by-point responses.

W1 / W2 / Q1: Low teacher likelihood need not imply a beneficial correction, and the frozen-student study does not show whether low-confidence selection remains useful as training changes the student and its visited states. How does realized correction benefit change across training checkpoints?

We thank the reviewer for highlighting the connection between the theoretical recoverability condition and the practical confidence gate. The condition concerns whether a teacher correction improves downstream return under subsequent student continuation. We therefore directly test whether low teacher confidence identifies states with greater correction benefit at multiple training checkpoints.

Specifically, we evaluate OPD checkpoints at steps 25, 50, 75, and 100 on ALFWorld and WebShop. At each checkpoint, we freeze the student and collect student-only trajectories on 128 training tasks per benchmark. We score each student response under the teacher and select the lowest- and highest-confidence turns from the same trajectory.

At each selected state, we compare two branches with the same environment state and interaction prefix: one executes the original student proposal, while the other executes one teacher correction. Both branches then generate fresh continuations with the same frozen student, without further teacher assistance. We run three paired comparisons per task using different random seeds and average their results. The table reports correction gains relative to each selected state’s student-continuation control.

BenchmarkStepLow-confidence correction gainHigh-confidence correction gain
ALFWorld25+7.8+6.3
ALFWorld50+7.0+2.6
ALFWorld75+3.10.0
ALFWorld100+1.6+0.1
WebShop25+2.7+0.2
WebShop50+7.5+5.8
WebShop75+5.2+3.0
WebShop100+1.7+0.4

Low-confidence corrections yield positive average gains and outperform high-confidence corrections at every tested checkpoint on both benchmarks. This directly links teacher confidence to realized correction benefit, providing empirical support for the practical gate’s ability to identify beneficial intervention states. The smaller correction gains in later training also align with the rationale for our intervention schedule, which gradually reduces the intervention rate as the student becomes more capable.

Q2: With teacher execution, the supervised objective, and correction count fixed, how does an action-only gate compare with full-response scoring and the existing confidence-gap gate?

We agree that scoring the full response can mix uncertainty about the environment decision with disagreement about the reasoning text. We therefore trained an action-only confidence gate with the 3B student on ALFWorld and WebShop. It computes the teacher's mean negative log-likelihood over action tokens, excluding reasoning tokens. Teacher execution, full-response teacher SFT, and the target intervention schedule remain unchanged. For comparison, we include the full-response and confidence-gap results reported in Table 2.

Intervention gateALFWorld seen SR (%)ALFWorld unseen SR (%)WebShop scoreWebShop SR (%)
Full-response teacher confidence90.088.882.469.3
Action-only teacher confidence88.688.379.166.4
Student–teacher confidence gap89.887.877.369.0

Action-only scoring is competitive on ALFWorld, reaching 88.3% unseen success compared with 88.8% for full-response scoring. On WebShop, full-response scoring improves the score from 79.1 to 82.4. These results suggest that uncertainty in the reasoning tokens also provides useful information for intervention selection, supporting our default full-response gate.

Besides the experiments above, we report comparisons with SAGE-OPD and results averaged over three independent training runs on ALFWorld, WebShop, and multi-hop Search in our response to Reviewer aeT7. Our response to Reviewer qD4L also presents an action-only SFT ablation examining the contribution of reasoning-token supervision.

Dear Reviewer,

As the discussion period is nearing its end, we wanted to check whether we have adequately addressed your concerns. If you have any remaining questions or feedback, please let us know. We would be happy to address them.

Thank you again for your time and valuable feedback!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their thoughtful feedback. Several questions concern when teacher intervention is beneficial, the contributions of reasoning and action supervision, and comparisons with existing distillation methods. We clarify these points and summarize our additional experiments below.

Contribution and Scope. UOPD studies on-policy distillation for multi-turn agents using an existing, fixed expert teacher. UOPD intervenes before execution at turns with low teacher confidence, executes a teacher-generated action, and trains the student on the teacher response through supervised learning. At other turns, the student acts and receives the standard OPD objective.

The teacher and student receive the same interaction history; UOPD does not require additional privileged information. Privileged-information distillation could be combined with UOPD by allowing the existing teacher model to use additional privileged information, but this is beyond our current setup.

Additional Results. In response to the reviewers’ questions, we conducted the following analyses:

(1) Intervention benefit across training (Reviewers WENY and aeT7). At OPD checkpoints 25, 50, 75, and 100, we compared executing a student action with executing one teacher correction from the same state, followed by frozen-student continuations. On both ALFWorld and WebShop, low-confidence corrections yielded positive average gains and larger gains than high-confidence corrections at every tested checkpoint. This directly connects the confidence signal to downstream correction benefit.

(2) Comparison with SAGE-OPD (Reviewer aeT7). We added comparisons using the same student and environment-specific teacher models. UOPD outperformed OPD and SAGE-OPD across all reported metrics on ALFWorld, WebShop, and multi-hop Search.

(3) Reasoning versus action (Reviewers WENY and qD4L). We separately tested action-only intervention scoring and action-only teacher supervision. Action-only scoring was competitive on ALFWorld, while full-response scoring performed better on WebShop. Masking the loss on teacher reasoning tokens still produced improvements over OPD, while supervising both reasoning and action yielded further gains.

(4) Training-seed robustness and intervention controls (Reviewers qD4L and aeT7). We completed three independent training runs for OPD and UOPD, and results show the superiority of UOPD. We also clarified the existing random-turn baseline, which retains teacher execution and supervision under the same target intervention schedule. UOPD’s stronger performance supports the benefit of confidence-based selection.

We thank the reviewers for their thoughtful feedback and Reviewer qD4L for confirming that our response addressed their concerns. We believe our clarifications and additional experiments also address the main concerns raised by the other reviewers. Please let us know if you have any remaining concerns, and we would be happy to address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.