- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work addresses the problem that early errors in multi-turn agent OPD can affect subsequent interaction trajectories. It proposes UOPD, which decides whether to intervene before execution based on the teacher's confidence in the student's candidate action. At an intervention step, the teacher's action is executed and learned through SFT, with the corresponding objective being forward KL. At a non-intervention step, the student's action is executed and the standard OPD loss is applied. UOPD consistently outperforms standard OPD on ALFWorld, WebShop, and multi-hop Search, while requiring fewer interaction rounds.
Critical strengths
-
Teacher intervention before action execution not only changes the training objective at the current step, but also affects subsequent rollouts. Compared with methods such as SAGE-OPD, which adjust supervision signals based on existing trajectories, this design can directly change subsequent interaction states.
-
UOPD outperforms standard OPD on ALFWorld, WebShop, and multi-hop Search. Table 2 further presents ablation studies on the intervention signal, teacher action execution, and SFT weight to analyze the impact of different components on performance.
Critical weaknesses
-
The related work section discusses how SAGE-OPD uses teacher judgments and confidence to adjust turn-level supervision, but the main experiments do not directly compare against it. Although The work compares UOPD with related methods such as FTB-OPD, a comparison with SAGE-OPD under the same settings would still help determine to what extent the gains from UOPD come from teacher action replacement before execution, rather than selective supervision alone.
-
The conclusions rely on a key assumption: the average advantage of the teacher's action over the student's action is positive at selected intervention steps. The experiments in Section 3 and Table 2 provide some evidence, but do not directly verify this theoretical assumption or sufficiently establish whether confidence can consistently identify steps with greater intervention gains across different tasks.
Questions for authors
-
Could UOPD be directly compared with SAGE-OPD under the same settings to further clarify its independent contribution?
-
Do the key experimental findings remain stable across multiple independent training seeds, especially the performance gains on multi-hop Search?
2 comments
Dear Reviewer,
As the discussion period is nearing its end, we wanted to check whether we have adequately addressed your concerns. If you have any remaining questions or feedback, please let us know. We would be happy to address them.
Thank you again for your time and valuable feedback!
We are very grateful for your valuable comments. Below, we summarize your comments in quotes and provide our point-by-point responses.
We thank the reviewer for this suggestion. We have added comparisons with SAGE-OPD and completed three independent training runs for each method on ALFWorld, WebShop, and multi-hop Search, using the same student and environment-specific teacher models. The table reports averages over the three training runs. ALFWorld and WebShop use 3B students, and Search uses the paper's 1.5B base student.
UOPD outperforms both baselines across all reported metrics. Compared with SAGE-OPD, it improves unseen ALFWorld success by 6.0 percentage points, WebShop score by 9.3 points, and multi-hop Search EM by 5.6 percentage points.
SAGE-OPD underperforms OPD in our setup. A practical challenge is its dependence on rubric-based teacher judgments. The teacher must label each student response as SKIP, WEAK, or STRONG. These labels determine whether and how strongly the turn receives distillation supervision. Our teachers are RL policies specialized for environment interaction. A teacher that acts effectively is not necessarily a reliable judge of another agent’s actions.
For example, in one SAGE-OPD training run on ALFWorld, the mean SKIP rate rose from 29.0% in the first 20 rollout batches to 68.7% in the last 20. Since SKIP assigns zero distillation weight, this reduces teacher supervision and may help explain SAGE-OPD’s weaker performance than OPD in our setting.
SAGE-OPD asks the teacher to explicitly judge whether a student turn needs supervision under a predefined rubric. UOPD instead uses the teacher's likelihood of the student response to select intervention turns and samples corrective actions directly from the teacher. This avoids requiring specialized expert policies to also perform rubric-based evaluation.
We agree that aggregate final performance does not directly verify this assumption. The condition concerns whether a teacher correction improves downstream return under subsequent student continuation. We therefore directly test whether low teacher confidence identifies states with greater correction benefit at multiple training checkpoints.
Specifically, we evaluate OPD checkpoints at steps 25, 50, 75, and 100 on ALFWorld and WebShop. At each checkpoint, we freeze the student and collect student-only trajectories on 128 training tasks per benchmark. We score each student response under the teacher and select the lowest- and highest-confidence turns from the same trajectory.
At each selected state, we compare two branches with the same environment state and interaction prefix: one executes the original student proposal, while the other executes one teacher correction. Both branches then generate fresh continuations with the same frozen student, without further teacher assistance. We run three paired comparisons per task using different random seeds and average their results. The table reports correction gains relative to each selected state’s student-continuation control.
Low-confidence corrections yield positive average gains and outperform high-confidence corrections at every tested checkpoint on both benchmarks. This directly links teacher confidence to realized correction benefit, providing empirical support for the practical gate’s ability to identify beneficial intervention states. The smaller correction gains in later training also align with the rationale for our intervention schedule, which gradually reduces the intervention rate as the student becomes more capable.
Besides these replications and baseline comparisons, our response to Reviewer qD4L describes the action-only SFT ablation, and our response to Reviewer WENY describes the action-only uncertainty gate. These experiments address different questions about what the teacher supplies, how the correction is learned, and where intervention is selected.