- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper finds that existing OPD methods mainly provide supervision on trajectories generated by the student itself, attempting to help the student make fewer mistakes, but they cannot enable the student to directly learn to recover after making mistakes and complete the task. Therefore, this paper proposes PivotOPD, which first uses the teacher to locate key error steps in the trajectory and provide the correct actions. It then trains the student to avoid these mistakes, while retaining the error state and having the teacher guide subsequent recovery actions. It distills recovery behaviors that the student would otherwise have difficulty exploring. The experiments on three main tasks (and one additional dataset) show that PivotOPD significantly improves the Agent’s task success rate and error recovery ability.
Critical strengths
- The authors propose the concept of “pivotal mistake” and find that pivotal mistakes usually occur early. Correcting these mistakes can substantially improve the task success rate. This provides some inspiration for future work.
- PivotOPD is effective. Compared with 13 other baselines, PivotOPD achieves the best performance on most subtasks under the paper’s experimental settings.
- The paper additionally conducts experiments on error recovery, self-distillation, and extensions to more complex tasks.
Critical weaknesses
- The paper mainly compares with other OPD baselines, but comparisons with existing error recovery methods are insufficient. There are already many papers that enable students to directly learn to recover after making mistakes and complete tasks, including corresponding improvements based on SFT and RL. This paper can be viewed as applying a similar idea to improve OPD. Therefore, it is difficult to judge whether this work makes an incremental contribution over existing recovery training paradigms or merely transfers an idea.
- The experiments on SWE-Bench are incomplete and lack validation of the error recovery mechanism.
Questions for authors
- Could comparisons with existing error recovery methods be added?
- Could experiments with the recovery mechanism enabled be added on SWE-Bench or other complex tasks?
- In steps such as error localization and the teacher’s action hints, how can the correctness of the teacher’s outputs be guaranteed? If the teacher is wrong, how should this be addressed?
5 comments
Response to Reviewer wb3s (Part 2)
Q3. In steps such as error localization and the teacher's action hints, how can the correctness of the teacher's outputs be guaranteed? If the teacher is wrong, how should this be addressed?
We measure how often the teacher is right, filter what it names before any update reaches the student, and bound what a wrong hint can do inside the objective, so a wrong hint costs one bounded, RL-corrected update rather than a hard imitation target. The evidence for each part follows.
How often the teacher is right (Appendix E.6, Tables 14 and 15). The teacher reads the whole trajectory and its outcome in hindsight before naming pivotal turns and gold actions. Against the symbolic ALFWorld oracle, which training never uses, its detected pivotal turns hit the exact first oracle turn in 54.6% and 68.3% of failed trajectories (3.8 to 5.7 times random), and the named gold action agrees with an oracle-optimal action 79.4% and 88.6% of the time on those turns.
| Teacher → student | Exact-turn hit | Random | Within ±1 turn | Gold action is oracle-optimal | Precision / recall (exact) |
|---|---|---|---|---|---|
| Qwen3-30B-A3B → Qwen3-1.7B | 54.6% | 14.2% | 71.1% | 79.4% | 46.1% / 60.9% |
| Qwen3.5-122B-A10B → Qwen3-8B | 68.3% | 11.9% | 84.4% | 88.6% | 58.1% / 72.6% |
Most remaining false positives are redundant but harmless actions, such as an extra look, flagged where the teacher prefers a more direct step. The method also degrades gracefully with teacher quality. With Qwen3-8B as its own teacher (Figure 3), PivotOPD still beats every baseline trained with the same self-teacher on all three benchmarks, and it still recovers from 61.5%, 59.2%, and 47.6% of pivotal mistakes on ALFWorld, WebShop, and Search-based QA against 20.3%, 16.8%, and 23.5% for standard OPD with the same self-teacher (Table 13).
What is filtered before any update (Appendix B, Algorithm 1).
- A named action must resolve onto the admissible set in ALFWorld and WebShop (normalized exact match, then article-stripped match, then unique containment, then token overlap with a strict winner), or pass the same schema check as student actions and execute in the tool or container interface in Search-based QA and SWE-Bench. Unresolved replies are dropped.
- A candidate turn counts as pivotal only if the student's committed action differs from the gold action.
- The teacher never writes the training target. Its action enters as a hint to the frozen self-teacher, any recovery response that commits to no valid action or acknowledges the hint is discarded, and an assertion keeps hinted text out of every training prompt.
- Recovery turns beyond the first replay the recorded prefix and abort on any observation mismatch.
What bounds the damage when the teacher is still wrong (Section 3.3, Appendix G).
- A flagged but reasonable alternative action costs little, because (Table 5), and recovery distillation at that turn still trains toward the teacher's next action at a state the student actually visited.
- Targets come from the hinted self-teacher, so an implausible hint moves the distribution little relative to the unhinted student and yields a small distillation advantage (Eq. 4). The clip on the recovery advantage (Eq. 5) and the PPO ratio clip (Eq. 7) bound any single update.
- Distillation is optimized jointly with the group-relative outcome advantage (Eqs. 3 and 6) and a low-variance KL penalty (Appendix D.1), so a hint that lowers episode success is penalized by the reward.
The revision adds cross-references from Appendix B to the exact-turn precision and recall analysis of Appendix E.6 (Tables 14 and 15) and to Appendix G, which now states these safeguards and extensions.
I thank the authors for the additional experiments and analysis. While the experiments in response to Q1 do show that no method significantly outperforms PivotOPD on any metric, PivotOPD does not demonstrate a particularly clear advantage over some error-recovery methods or even Matched-data SFT.
Response to Reviewer wb3s (Part 1)
We thank Reviewer wb3s for the constructive review. We addressed all three requests with new experiments and analysis.
Yes. We added a controlled comparison with Agent-R, SCoRe-style RL, and a matched-data SFT control on ALFWorld at equal data and cost (Appendix E.7, Table 16).
All four methods train the same Qwen3-1.7B student with Qwen3-30B-A3B as teacher on a 256-task subset of ALFWorld, which isolates the recovery objective at a fraction of the Table 1 budget. Agent-R runs two rounds of revision-trajectory SFT, and SCoRe-style RL learns from post-error states. Matched-data SFT trains on exactly the recovery samples PivotOPD uses, and PivotOPD enters as distillation only (no outcome RL). Recovery is measured without teacher assistance on the 72 ALFWorld pivotal mistakes of Appendix E.2, plus success on the held-out valid seen and valid unseen splits (means ± 95% t-intervals across seeds).
PivotOPD has the highest valid seen success, the narrowest recovery interval, and the lowest cost, and no method beats it significantly on any metric. Against SCoRe-style RL it has the higher mean on every metric at 2.7 times lower cost and a 3.7 times narrower recovery interval (±3.3 vs. ±12.1). Agent-R's higher mean recovery comes with a ±12.3 interval, lower valid seen success, and a paired 95% cluster-bootstrap interval against PivotOPD that includes zero (+4.6, [-1.4, 10.7]), so 3.7 times the compute buys no significant gain. On valid unseen, all four methods tie within noise. On the same samples at the same cost, forward-KL recovery distillation is ahead of matched-data SFT by +2.2 on recovery and +1.3 on valid seen and cuts the recovery interval by more than half (±8.3 to ±3.3), with a paired interval ([-0.5, 5.0]) that includes zero.
At the full budget, the same RL-free PivotOPD objective reaches 69.8% (1.7B) and 90.2% (8B) ALFWorld success (Table 11), and the full method recovers from 72.7% of the same 72 pivotal mistakes against 20.3% for standard OPD (Figure 5b, Table 13).
The revision adds this comparison as Appendix E.7.
Yes. We ran the full PivotOPD, with earlier-turn pivot detection and recovery distillation, on SWE-Bench Verified, Terminal-Bench 2.1, and SWE-Bench Multilingual. Recovery distillation more than doubles the gain of prevention alone (+4.7 vs. +2.1 macro-average), most of all on Terminal-Bench.
Our original configuration audited only the final action with K=0 to avoid container replay, but the first recovery turn starts from the recorded post-mistake state ct+1 and needs no replay (Section 3.3, Appendix E.1). The full configuration audits up to m=3 candidate turns before the final action, uses K=1, samples the recovery response from Nemotron-3-Super hinted with the recovery action, and applies it at wrec=0.05, over three seeds.
On SWE-Bench Verified, recovery distillation adds +2.1 on top of prevention and takes the teacher gap closed from 31% to 52%. On Terminal-Bench, prevention alone adds +1.8 and recovery raises the gain to +5.9, since early command-line mistakes leave states that the agent must repair rather than avoid. On SWE-Bench Multilingual, +1.4 becomes +3.0 across nine languages. The recovery mechanism validated on ALFWorld (Figure 5b, 45.8% vs. 72.7% post-mistake recovery) therefore transfers to a new model family and to containerized environments.
The revision reports all three benchmarks in Sections 4.1 and 4.2 and Figure 4 and the full configuration in Appendix D.6.