acceptodds
Under review as a conference paper at ICLR 2027

PivotOPD: Learning to Recover from Pivotal Mistakes in Multi-Turn Agents

Abstract

On-policy distillation (OPD) is a promising approach for training language agents, providing dense teacher supervision on student-generated trajectories. However, in multi-turn interaction, an incorrect action changes the states the student encounters later, so errors compound across turns. In our preliminary experiments across three Qwen3 models (8B–235B), we find that more than half of the failed rollouts contain a *pivotal mistake*, an action that moves the agent farther from completing the task, and this mistake typically occurs early. These pivotal mistakes often remain recoverable: guiding the model for only a few turns after the pivotal turn can restore task success. We therefore propose **PIVOTOPD**, an on-policy distillation framework that jointly trains the student to prevent pivotal mistakes and to recover from the states they create. At each pivotal mistake, a teacher model provides a gold action and then names a recovery action at each of the next few turns. ***Preventive distillation*** uses the gold action with reverse KL to steer the student away from the pivotal mistake, while ***recovery distillation*** uses the recovery actions with forward KL to transfer recovery behaviors that the student rarely samples. Against 13 baselines on ALFWorld, WebShop, and Search-based QA, PIVOTOPD achieves the strongest average performance for both Qwen3-1.7B and Qwen3-8B students, improving over the strongest baseline on ALFWorld by +5.5% with the 1.7B student. The gains also transfer to another model family on the software engineering domain, where PIVOTOPD raises the resolve rate of a Nemotron-3.5 student on SWE-Bench Verified by +3.2%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
pivotal turns, error accumulation, multi-turn agents, on-policy distillation, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wb3sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper finds that existing OPD methods mainly provide supervision on trajectories generated by the student itself, attempting to help the student make fewer mistakes, but they cannot enable the student to directly learn to recover after making mistakes and complete the task. Therefore, this paper proposes PivotOPD, which first uses the teacher to locate key error steps in the trajectory and provide the correct actions. It then trains the student to avoid these mistakes, while retaining the error state and having the teacher guide subsequent recovery actions. It distills recovery behaviors that the student would otherwise have difficulty exploring. The experiments on three main tasks (and one additional dataset) show that PivotOPD significantly improves the Agent’s task success rate and error recovery ability.

Critical strengths
  1. The authors propose the concept of “pivotal mistake” and find that pivotal mistakes usually occur early. Correcting these mistakes can substantially improve the task success rate. This provides some inspiration for future work.
  2. PivotOPD is effective. Compared with 13 other baselines, PivotOPD achieves the best performance on most subtasks under the paper’s experimental settings.
  3. The paper additionally conducts experiments on error recovery, self-distillation, and extensions to more complex tasks.
Critical weaknesses
  1. The paper mainly compares with other OPD baselines, but comparisons with existing error recovery methods are insufficient. There are already many papers that enable students to directly learn to recover after making mistakes and complete tasks, including corresponding improvements based on SFT and RL. This paper can be viewed as applying a similar idea to improve OPD. Therefore, it is difficult to judge whether this work makes an incremental contribution over existing recovery training paradigms or merely transfers an idea.
  2. The experiments on SWE-Bench are incomplete and lack validation of the error recovery mechanism.
Questions for authors
  1. Could comparisons with existing error recovery methods be added?
  2. Could experiments with the recovery mechanism enabled be added on SWE-Bench or other complex tasks?
  3. In steps such as error localization and the teacher’s action hints, how can the correctness of the teacher’s outputs be guaranteed? If the teacher is wrong, how should this be addressed?
5 comments
AuthorsOpenReview ↗

Response to Reviewer wb3s (Part 1)

We thank Reviewer wb3s for the constructive review. We addressed all three requests with new experiments and analysis.

Q1. Could comparisons with existing error-recovery methods be added?

Yes. We added a controlled comparison with Agent-R, SCoRe-style RL, and a matched-data SFT control on ALFWorld at equal data and cost (Appendix E.7, Table 16).

All four methods train the same Qwen3-1.7B student with Qwen3-30B-A3B as teacher on a 256-task subset of ALFWorld, which isolates the recovery objective at a fraction of the Table 1 budget. Agent-R runs two rounds of revision-trajectory SFT, and SCoRe-style RL learns from post-error states. Matched-data SFT trains on exactly the recovery samples PivotOPD uses, and PivotOPD enters as distillation only (no outcome RL). Recovery is measured without teacher assistance on the 72 ALFWorld pivotal mistakes of Appendix E.2, plus success on the held-out valid seen and valid unseen splits (means ± 95% t-intervals across seeds).

MethodA100-h/seedRecovery (%)valid seen (%)valid unseen (%)
Agent-R (2 rounds)1.85 (3.7×)12.6 ± 12.330.5 ± 10.923.3 ± 20.7
SCoRe-style RL1.35 (2.7×)7.6 ± 12.120.7 ± 13.421.0 ± 10.2
Matched-data SFT0.51 (1.0×)5.7 ± 8.332.2 ± 4.423.0 ± 13.3
PivotOPD (distillation only)0.50 (1.0×)7.9 ± 3.333.5 ± 8.122.4 ± 13.1

PivotOPD has the highest valid seen success, the narrowest recovery interval, and the lowest cost, and no method beats it significantly on any metric. Against SCoRe-style RL it has the higher mean on every metric at 2.7 times lower cost and a 3.7 times narrower recovery interval (±3.3 vs. ±12.1). Agent-R's higher mean recovery comes with a ±12.3 interval, lower valid seen success, and a paired 95% cluster-bootstrap interval against PivotOPD that includes zero (+4.6, [-1.4, 10.7]), so 3.7 times the compute buys no significant gain. On valid unseen, all four methods tie within noise. On the same samples at the same cost, forward-KL recovery distillation is ahead of matched-data SFT by +2.2 on recovery and +1.3 on valid seen and cuts the recovery interval by more than half (±8.3 to ±3.3), with a paired interval ([-0.5, 5.0]) that includes zero.

At the full budget, the same RL-free PivotOPD objective reaches 69.8% (1.7B) and 90.2% (8B) ALFWorld success (Table 11), and the full method recovers from 72.7% of the same 72 pivotal mistakes against 20.3% for standard OPD (Figure 5b, Table 13).

The revision adds this comparison as Appendix E.7.

Q2. Could experiments with the recovery mechanism enabled be added on SWE-Bench or other complex tasks?

Yes. We ran the full PivotOPD, with earlier-turn pivot detection and recovery distillation, on SWE-Bench Verified, Terminal-Bench 2.1, and SWE-Bench Multilingual. Recovery distillation more than doubles the gain of prevention alone (+4.7 vs. +2.1 macro-average), most of all on Terminal-Bench.

Our original configuration audited only the final action with K=0K=0 to avoid container replay, but the first recovery turn starts from the recorded post-mistake state ct+1c_{t+1} and needs no replay (Section 3.3, Appendix E.1). The full configuration audits up to m=3m=3 candidate turns before the final action, uses K=1K=1, samples the recovery response from Nemotron-3-Super hinted with the recovery action, and applies it at wrec=0.05w_\text{rec}=0.05, over three seeds.

Nemotron-3.5-SFT, 3 seedsTerminal-Bench 2.1, 89 tasks (%)SWE-Bench Multilingual, 300 tasks (%)SWE-Bench Verified, 500 tasks (%)Macro-avg. (%)Teacher gap closed
Student37.5 ± 0.955.0 ± 0.862.8 ± 0.651.8 ± 0.50%
PivotOPD, preventive only (K=0K=0)39.3 ± 1.0 (+1.8)56.4 ± 0.9 (+1.4)66.0 ± 0.8 (+3.2)53.9 ± 0.6 (+2.1)11.8%
PivotOPD, full (with recovery)43.4 ± 0.9 (+5.9)58.0 ± 0.8 (+3.0)68.1 ± 0.7 (+5.3)56.5 ± 0.5 (+4.7)26.2%
Nemotron-3-Super teacher65.9 ± 0.770.5 ± 0.673.0 ± 0.569.8 ± 0.4100%

On SWE-Bench Verified, recovery distillation adds +2.1 on top of prevention and takes the teacher gap closed from 31% to 52%. On Terminal-Bench, prevention alone adds +1.8 and recovery raises the gain to +5.9, since early command-line mistakes leave states that the agent must repair rather than avoid. On SWE-Bench Multilingual, +1.4 becomes +3.0 across nine languages. The recovery mechanism validated on ALFWorld (Figure 5b, 45.8% vs. 72.7% post-mistake recovery) therefore transfers to a new model family and to containerized environments.

The revision reports all three benchmarks in Sections 4.1 and 4.2 and Figure 4 and the full configuration in Appendix D.6.

AuthorsOpenReview ↗

Response to Reviewer wb3s (Part 2)

Q3. In steps such as error localization and the teacher's action hints, how can the correctness of the teacher's outputs be guaranteed? If the teacher is wrong, how should this be addressed?

We measure how often the teacher is right, filter what it names before any update reaches the student, and bound what a wrong hint can do inside the objective, so a wrong hint costs one bounded, RL-corrected update rather than a hard imitation target. The evidence for each part follows.

How often the teacher is right (Appendix E.6, Tables 14 and 15). The teacher reads the whole trajectory and its outcome in hindsight before naming pivotal turns and gold actions. Against the symbolic ALFWorld oracle, which training never uses, its detected pivotal turns hit the exact first oracle turn in 54.6% and 68.3% of failed trajectories (3.8 to 5.7 times random), and the named gold action agrees with an oracle-optimal action 79.4% and 88.6% of the time on those turns.

Teacher → studentExact-turn hitRandomWithin ±1 turnGold action is oracle-optimalPrecision / recall (exact)
Qwen3-30B-A3B → Qwen3-1.7B54.6%14.2%71.1%79.4%46.1% / 60.9%
Qwen3.5-122B-A10B → Qwen3-8B68.3%11.9%84.4%88.6%58.1% / 72.6%

Most remaining false positives are redundant but harmless actions, such as an extra look, flagged where the teacher prefers a more direct step. The method also degrades gracefully with teacher quality. With Qwen3-8B as its own teacher (Figure 3), PivotOPD still beats every baseline trained with the same self-teacher on all three benchmarks, and it still recovers from 61.5%, 59.2%, and 47.6% of pivotal mistakes on ALFWorld, WebShop, and Search-based QA against 20.3%, 16.8%, and 23.5% for standard OPD with the same self-teacher (Table 13).

What is filtered before any update (Appendix B, Algorithm 1).

  • A named action must resolve onto the admissible set in ALFWorld and WebShop (normalized exact match, then article-stripped match, then unique containment, then token overlap with a strict winner), or pass the same schema check as student actions and execute in the tool or container interface in Search-based QA and SWE-Bench. Unresolved replies are dropped.
  • A candidate turn counts as pivotal only if the student's committed action differs from the gold action.
  • The teacher never writes the training target. Its action enters as a hint to the frozen self-teacher, any recovery response that commits to no valid action or acknowledges the hint is discarded, and an assertion keeps hinted text out of every training prompt.
  • Recovery turns beyond the first replay the recorded prefix and abort on any observation mismatch.

What bounds the damage when the teacher is still wrong (Section 3.3, Appendix G).

  1. A flagged but reasonable alternative action costs little, because wprev=0.001w_\text{prev}=0.001 (Table 5), and recovery distillation at that turn still trains toward the teacher's next action at a state the student actually visited.
  2. Targets come from the hinted self-teacher, so an implausible hint moves the distribution little relative to the unhinted student and yields a small distillation advantage (Eq. 4). The clip on the recovery advantage (Eq. 5) and the PPO ratio clip (Eq. 7) bound any single update.
  3. Distillation is optimized jointly with the group-relative outcome advantage ARLA^{\mathrm{RL}} (Eqs. 3 and 6) and a low-variance KL penalty (Appendix D.1), so a hint that lowers episode success is penalized by the reward.

The revision adds cross-references from Appendix B to the exact-turn precision and recall analysis of Appendix E.6 (Tables 14 and 15) and to Appendix G, which now states these safeguards and extensions.

I thank the authors for the additional experiments and analysis. While the experiments in response to Q1 do show that no method significantly outperforms PivotOPD on any metric, PivotOPD does not demonstrate a particularly clear advantage over some error-recovery methods or even Matched-data SFT.

Reviewer_qZ37OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper first presents an analysis on ALFWorld using a symbolic oracle, showing that more than half of the failed rollouts from Qwen3 models on this benchmark include "pivotal" mistakes which move the agent away from solving the task. Then, the authors argue that On-policy Distillation (OPD), or more precisely OPSD, fails to repair these mistakes. They propose PivotOPD, which adds preventive and recovery distillation losses to the PPO objective with the goal of reducing those mistakes. They evaluate PivotOPD on a few agentic tasks and compare it to other post-training methods and report improvements in average performance.

Critical strengths

1- The paper is well-motivated, surfacing an issue with OPSD method on multi-turn tasks: While this is only shown for ALFWorld, it is important for the community to understand why OPD methods struggle with multi-turn tasks and how to fox it.

2- Experiments are extensive across three benchmarks with 13 baselines, two student sizes, a self-teacher variant. Ablation studies are also useful to understand the role of the proposed algorithm's components.

Critical weaknesses

1- The source of PivotOPD's improvements are not clear and the comparison to the baselines are not fair:

  • Amount of privileged information the teacher has access to for PivotOPD is not comparable to other teacher-based methods.

  • Compute is not matched between PivotOPD and other baselines. For example, PivotOPD costs substantially more per training step than GRPO (up to +112.3% with K=3).

  • Most of the improvements seems to come from RL + preventive distillation. It would be useful to report a variant of PivotOPD without any RL to understand how much improvements is really because of only preventive and recovery distillation.

2- The claim that OPD does not repair the pivotal mistakes and PivotOPD does is not supported by enough evidence:

  • The authors test this claim with OPSD, which uses a self-teacher. The correct test is to use same teacher PivotOPD use for pivot detection and providing the gold action.

  • The analysis is only on ALFWorld. Even though, I understand that this is due to having the symbolic oracle only for this benchmark, but one benchmark is still not enough to support this claim.

  • The paper counts a detection as correct if at least one teacher-detected pivotal turn falls within -/+ 1 turn of the oracle pivot turn and even then the agreement is around 70-80%. PivotOPD modifies the action at the first pivotal turn so why not reporting if there is exact match on the first pivotal turn between the teacher and oracle?

Questions for authors
  • Teacher access in table 1: Which methods use the teacher, a skill bank, or neither? Do the baseline teachers get the same full-trajectory and outcome hindsight

  • Can you report standard deviations or CIs across seeds for Tables 1, 2 and 6 and Figs. 3 and 4?

  • What is the exact-turn hit rate for the first oracle pivotal turn, and the precision and recall of detected pivotal turns?

5 comments
AuthorsOpenReview ↗

Response to Reviewer qZ37 (Part 1)

We thank Reviewer qZ37 for the detailed review. Please see our responses below.

Q1. The source of the gains is unclear and the comparison unfair. The teacher's privileged information is not comparable, compute is not matched (up to +112.3% per step with K = 3), and most of the gain may come from RL plus preventive distillation. Report a variant without RL.

Privileged information. PivotOPD never uses the ALFWorld oracle or hidden state in training (Section 3.2, Appendix B). For pivot detection the teacher sees the student trajectory τ\tau and its outcome R(τ)R(\tau), the same signals that the teacher-based baselines receive, and at recovery turns only the student-facing post-mistake observation. We then gave the strongest baselines PivotOPD's own teacher report (episode lesson and gold actions) as hints to their self-teachers, and ran PivotOPD without the outcome flag (Qwen3-8B, three seeds, Table 10).

Method (Qwen3-8B), Table 1 / + teacher hintsALFWorld (%)Search-QA (%)WebShop scoreWebShop succ. (%)
StepOPSD79.3 / 85.442.8 / 44.680.7 / 82.572.7 / 75.4
SDAR83.8 / 87.941.7 / 44.385.5 / 86.179.7 / 80.2
RLSD90.8 / 91.234.1 / 41.884.6 / 85.478.9 / 79.5
PivotOPD w/o outcome R(τ)R(\tau)92.146.987.681.1
PivotOPD93.047.488.281.9

With the same gold actions as hints, every baseline still trails PivotOPD on every benchmark, and dropping the outcome flag costs PivotOPD at most 0.9. Hinting a self-teacher helps only with forward-KL recovery distillation at post-mistake states (Proposition 1), since reverse KL on student-sampled responses cannot move mass onto actions that the student samples with probability below 10−210^{-2}.

Compute. The +112.3% figure is K=3K=3 from the budget ablation (Table 8). Default PivotOPD uses K=1K=1 on WebShop and Search-based QA (no replay, +12.4% per step on the student node) and K=2K=2 on ALFWorld. We measured end-to-end H100 GPU-hours on both the student node and the teacher endpoint, including pivot detection, recovery queries, and replay (221.1 / 106.4 / 133.5 hours for PivotOPD), and retrained the strongest baselines until each consumed 100 to 101% of that total (Table 9, three seeds, same checkpoint selection).

Method (Qwen3-8B), compute-matchedSteps (ALF / WS / QA)ALFWorld (%)Search-QA (%)WebShop scoreWebShop succ. (%)
OPID296 / 182 / 19385.446.282.175.0
RLSD311 / 191 / 20291.535.485.379.8
SDAR311 / 191 / 20285.642.686.280.5
PivotOPD, K=1K=1 replay-free275 / 160 / 16092.147.488.281.9
PivotOPD, default (K=2/1/1K=2/1/1)160 / 160 / 16093.047.488.281.9

Compute parity does not close the gap, because extra steps plateau once the non-pivotal failures are gone. Even with replay removed (K=1K=1 on ALFWorld, about +18% total overhead over GRPO including all teacher calls) and the saved hours reinvested in 275 steps, PivotOPD still beats every compute-matched baseline (92.1 vs. 91.5 for RLSD).

Without RL. The paper already showed that removing recovery distillation (preventive only, K=0K=0, even with a 100 times larger wprev=0.1w_\text{prev}=0.1) drops Pick2 success by 15.1 points (Table 7) and the WebShop validation score by 21.5 points (Figure 7), and cuts post-mistake recovery from 72.7% to 45.8% (Figure 5b). We now also train PivotOPD w/o RL with LRL=0L^\text{RL}=0 (ARL=0A^\text{RL}=0 in Eqs. 3 and 6), so no outcome reward enters the update (Table 11, three seeds).

MethodALFWorld (%)Search-QA (%)WebShop scoreWebShop succ. (%)
1.7B GRPO (pure RL)42.737.774.557.0
1.7B OPSD (standard OPD)12.436.849.210.1
1.7B PivotOPD w/o RL69.842.479.673.2
1.7B PivotOPD73.744.584.476.6
8B GRPO (pure RL)35.337.583.575.0
8B OPSD (standard OPD)46.743.347.218.2
8B PivotOPD w/o RL90.245.885.178.9
8B PivotOPD93.047.488.281.9

The two distillation terms alone account for roughly 90% of the full method's gain on both students. PivotOPD w/o RL beats pure RL by +27.1 (1.7B) and +54.9 (8B) on ALFWorld and RL-free OPSD by +57.4 and +43.5, and it surpasses every baseline of Table 1 on Search-based QA and, with 1.7B, on ALFWorld (with 8B only RLSD stays ahead there, 90.8 vs. 90.2). Keeping ARLA^\text{RL} adds +1.6 to +4.8 more, from trajectory-level efficiency on non-pivotal turns.

The revision adds the compute-matched comparison as Appendix E.4 (Table 9) and the information controls and the RL-free ablation as Appendix E.5 (Tables 10 and 11).

AuthorsOpenReview ↗

Response to Reviewer qZ37 (Part 2)

Q2. The claim that OPD does not repair pivotal mistakes and PivotOPD does is under-supported. The test uses OPSD with a self-teacher, the right test is standard OPD with the same teacher and gold actions, the analysis is ALFWorld only, agreement is counted within ±1 turn and is only 70 to 80%, and PivotOPD modifies the first pivotal turn, so why not report the exact match?

Standard OPD with the same Qwen3.5-122B-A10B teacher, even given the gold actions as hints, still fails to repair pivotal mistakes (Table 12). We trained Qwen3-8B for 120 steps under the protocol of Appendix A with full-trajectory reverse-KL distillation from Qwen3.5-122B-A10B, without hints (Teacher-OPD) and conditioned on the same teacher report and gold actions that PivotOPD uses (Teacher-OPD + Gold Hints), and measured failures on the 140 held-out ALFWorld tasks.

Method (Qwen3-8B, step 120)TeacherTotal fail (%)Fail after a pivotal turn (%)Fail without a pivotal turn (%)
Base Model (step 0)none79.351.427.9
OPSD (Figure 1c)Qwen3-8B (self)55.749.36.4
Teacher-OPDQwen3.5-122B-A10B47.143.63.6
Teacher-OPD + Gold HintsQwen3.5-122B-A10B42.939.33.6
PivotOPDQwen3.5-122B-A10B13.610.72.9

The stronger teacher removes the non-pivotal failures (3.6%) but leaves 39.3% to 43.6% of the tasks failing after a pivotal mistake, because reverse KL on student-generated trajectories gives almost no signal at post-mistake states, where the student assigns probability below 10−210^{-2} to the recovery action (Proposition 1). PivotOPD cuts these failures from 51.4% to 10.7%. Teacher capacity is not the explanation either. With Qwen3-8B as its own teacher (Figure 3), PivotOPD beats OPSD by +33.5 on ALFWorld (82.8 vs. 49.3), +59.0 on WebShop success (77.2 vs. 18.2), and +3.7 on Search-based QA (46.6 vs. 42.9).

The recovery bottleneck holds on all three benchmarks (Table 13). WebShop and Search-based QA replay exactly (Appendix G), so we extended the counterfactual diagnosis of Appendix A and the replay evaluation of Appendix E.2 (8 replays per mistake) to all 500 held-out WebShop instructions and 725 Search-based QA questions, with the teacher's counterfactual action in place of the oracle (the earliest turn at which substituting it restores success while continuing from ata_t fails). Pivotal mistakes account for 64.9% of base-model failures on ALFWorld (72/111), 61.3% on WebShop (285/465), and 54.8% on Search-based QA (324/591).

Policy (Qwen3-8B)TeacherALFWorld (72)WebShop (285)Search-QA (324)
Base Modelnone8.3%9.4%11.2%
OPSD (standard OPD)Qwen3-8B (self)20.3%16.8%23.5%
Teacher-OPDQwen3.5-122B-A10B24.8%22.1%26.4%
Teacher-OPD + Gold HintsQwen3.5-122B-A10B31.4%27.9%29.8%
PivotOPD (self-teacher)Qwen3-8B (self)61.5%59.2%47.6%
PivotOPDQwen3.5-122B-A10B72.7%68.4%51.9%

Standard OPD recovers from fewer than 32% of the pivotal mistakes on every benchmark, even with the 122B teacher and gold hints, whereas PivotOPD more than doubles the OPSD rate everywhere and reaches 47.6% to 61.5% with no external teacher at all.

Exact-turn agreement. PivotOPD does not act only at the first pivotal turn. It evaluates all up to mm candidate turns that the teacher returns and treats every turn with at≠at∗a_t \neq a^*_t as pivotal (Appendix B), since a trajectory can contain several pivotal mistakes. Table 3 used a ±1 window because agents often take a benign observation action, such as look, right next to the irreversible mistake. Under strict ±0 matching (Table 14), the detected pivotal turns hit the exact first oracle turn in 54.6% (Qwen3-30B-A3B) and 68.3% (Qwen3.5-122B-A10B) of failed trajectories, 3.8 to 5.7 times random, the primary detected pivot alone hits it in 44.8% and 56.2% (3.9 to 5.9 times random), and the named gold action is oracle-optimal on 79.4% and 88.6% of the matched turns. Our answer to Q5 gives the full table with precision and recall.

The revision adds the external-teacher controls, the multi-benchmark recovery analysis, and the exact-turn agreement to Appendix E.6 (Tables 12 to 15).

AuthorsOpenReview ↗

Response to Reviewer qZ37 (Part 3)

Q3. Teacher access in Table 1. Which methods use the teacher, a skill bank, or neither, and do the baseline teachers get the same full-trajectory and outcome hindsight?

Every method with a teacher queries the same one (Qwen3-30B-A3B for the 1.7B student, Qwen3.5-122B-A10B for the 8B student), and all methods share the training tasks and rollout budget (160 steps, G=8G=8). Across the 14 other rows of Table 1, two use neither a teacher nor a skill bank (Base Model, GRPO), one uses only the static skill bank (Skill-GRPO, which injects the top-6 retrieved entries of a bank with 47 / 200 / 75 entries into the student prompts), one conditions its self-teacher on a successful rollout from the same group (StepOPSD), four distill from a privileged self-teacher conditioned on retrieved skills, so the teacher model enters them only through the skill bank distilled offline from earlier student rollouts (OPSD, Skill-SD, RLSD, SDAR), and six give the teacher the full trajectory and outcome (τ,R(τ))(\tau, R(\tau)) with no skill bank (TurnOPD, TCOD, SOD, AgentOPSD, PivotRL, OPID). PivotOPD passes (τ,R(τ))(\tau, R(\tau)) to the teacher for pivot detection and only the student-facing post-mistake observation at recovery turns. Please see the table below.

MethodTeacher modelSkill bankFull trajectory and outcome hindsightHow the signal is applied
Base Modelnononountrained checkpoint
GRPOnonono (outcome to RL only)group-relative outcome advantage
Skill-GRPOnoyes (top-6)offline (skill bank)retrieved skills in student prompts
StepOPSDno (self-teacher)noyes (successful group rollout)self-teacher conditioned on a successful group trajectory
OPSDoffline (skill bank)yesoffline (skill bank)reverse KL on student rollouts
Skill-SDoffline (skill bank)yesoffline (skill bank)auxiliary reverse KL on student rollouts
RLSDoffline (skill bank)yesoffline (skill bank)teacher-guided advantage rescaling on student rollouts
SDARoffline (skill bank)yesoffline (skill bank)gated distillation loss on student rollouts
TurnOPDyesnoyes, full (τ,R(τ))(\tau, R(\tau))turn-aware reverse KL on student rollouts
TCODyesnoyes, full (τ,R(τ))(\tau, R(\tau))temporal curriculum distillation on student rollouts
SODyesnoyes, full (τ,R(τ))(\tau, R(\tau))step-wise divergence reweighting on student rollouts
AgentOPSDyesnoyes, full (τ,R(τ))(\tau, R(\tau))recursive self-distillation turning R(τ)R(\tau) into turn credit
PivotRLyesnoyes, full (τ,R(τ))(\tau, R(\tau))RL updates concentrated on teacher-identified turns
OPIDyesno (online skills)yes, full (τ,R(τ))(\tau, R(\tau))extracted trajectory and turn skills, reverse KL on student rollouts
PivotOPDyesnoyes (detection), no (recovery)reverse-KL prevention at tt and forward-KL recovery at t+kt+k

The six baselines with the same hindsight still trail PivotOPD. On Qwen3-8B, PivotOPD beats the strongest of them by +9.3 on ALFWorld and +1.8 on Search-based QA (OPID in both cases) and by +4.0 in WebShop score (TurnOPD). Also, giving the skill-bank baselines PivotOPD's own hindsight report does not close the gap. When the skill bank of RLSD, SDAR, and StepOPSD is replaced by the episode lesson and per-turn gold actions of PivotOPD, they still trail on every benchmark (91.2 vs. 93.0 on ALFWorld, 44.6 vs. 47.4 on Search-based QA, 86.1 vs. 88.2 in WebShop score, Appendix E.5, Table 10), while removing R(τ)R(\tau) from PivotOPD's teacher prompt changes its results by at most 0.9. The decisive factor is not hindsight access but how the signal is applied. The baselines apply reverse KL or advantage reweighting only to actions that the student has sampled, whereas PivotOPD applies forward-KL recovery distillation to self-teacher responses at post-mistake states (Proposition 1).

The revision adds this part as Table 6 in Appendix D.3, with the two conclusions above.

Reviewer_UkNSOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

PivotOPD addresses mistakes that redirect a multi-turn agent into states from which it rarely recovers. A teacher examines student rollouts, identifies candidate pivotal turns, and names a preferred action at each turn and subsequent recovery actions. The method combines preventive distillation on recorded student responses with recovery distillation on responses sampled from an action-hinted model, alongside group-based reinforcement learning. An ALFWorld oracle analysis motivates the approach. Experiments with Qwen3 students compare against 13 baselines on ALFWorld, Search-based QA, and WebShop; replay experiments and ablations examine recovery. A modified preventive-only configuration is also tested with Nemotron models on SWE-Bench Verified.

Critical strengths
  1. The recovery claim has direct behavioral evidence. On the same 72 oracle-identified ALFWorld mistakes, the reported final recovery rates are 20.3% for standard OPD, 45.8% for preventive-only training, and 72.7% for PivotOPD. The fixed-prefix design makes this more informative than task success alone.

  2. The method performs consistently across the three main benchmarks. PivotOPD has the highest reported benchmark averages for both Qwen3 student sizes. Ablations of recovery, pivot placement, and action hints give useful evidence that the improvement depends on the proposed supervision.

Critical weaknesses
  1. The main comparisons do not establish compute efficiency. ALFWorld selects two recovery turns, for which the reported per-step overhead is 94.2% over GRPO; this figure excludes pivot detection. The extra teacher queries and environment replays may account for a substantial part of the advantage over baselines trained for the same number of steps. This matters to the practical value of the method, especially where the 8B margins over the strongest baseline are modest.

  2. The SWE-Bench experiment tests a different, narrower configuration. Appendix D.6 sets the recovery budget to zero, audits only the final action, and uses the external teacher’s hinted distribution instead of the privileged self-teacher used in the main method. Its +3.2-point result supports a form of preventive intervention on another model family, but does not demonstrate transfer of the paper’s recovery mechanism.

Questions for authors
  1. At matched total training cost, including pivot detection, teacher calls, and environment replay, how does PivotOPD compare with the strongest baselines? This could materially change the assessment of its practical gain.

  2. Can the authors quantify uncertainty for the relatively small 8B margins using paired task and seed comparisons? Are those advantages consistent across seeds?

  3. Can the SWE-Bench claim be limited to preventive intervention, or can the authors provide an experiment with recovery distillation enabled? The answer affects how broadly the central mechanism has been validated.

3 comments
AuthorsOpenReview ↗

Response to Reviewer UkNS (Part 1)

We thank Reviewer UkNS for the careful reading and for two concrete suggestions which are compute-matched comparisons and recovery distillation on SWE-Bench. Both of them are now in the revised paper, and both strengthen our main claims.

W1 / Q1. Compute efficiency is not established. The reported +94.2% per-step overhead excludes pivot detection, and the extra teacher queries and environment replays may explain much of the advantage over baselines trained for the same number of steps. At matched total training cost, how does PivotOPD compare with the strongest baselines?

At equal end-to-end GPU-hours, counting pivot detection, every teacher call, and environment replay, every baseline still trails PivotOPD on every benchmark (new Appendix E.4, Table 9).

The +12.4% (K=1K=1) and +94.2% (K=2K=2) figures in Appendix E.1 (Table 8) are student-node overheads with the Qwen3-1.7B student. For a strict accounting we measured wall-clock H100 GPU-hours for the Qwen3-8B student over the 160 training steps on ALFWorld / WebShop / Search-based QA, across both the 4$\times$H100 student node (vLLM, TP=1, plus FSDP) and the Qwen3.5-122B-A10B teacher endpoint (TP=8):

  1. student rollouts and FSDP PPO updates: 108.8 / 85.3 / 101.2 GPU-hours;
  2. privileged self-teacher forward passes: 5.1 / 3.9 / 4.7;
  3. full-batch teacher pivot detection: 10.6 / 8.6 / 20.5;
  4. teacher recovery-action queries: 5.0 / 2.1 / 1.8;
  5. environment replay and hinted recovery rollouts: 91.6 / 6.5 / 5.3.

The first recovery turn starts from the recorded post-mistake state ct+1c_{t+1}, so WebShop and Search-based QA (K=1K=1) need no environment replay at all. There, the total cost of PivotOPD is 1.19 to 1.26 times that of the self-distillation baselines (RLSD, SDAR, StepOPSD). ALFWorld (K=2K=2) is the expensive case at 1.94 times (221.1 GPU-hours against 113.9 for those baselines at 160 steps), and 91.6 of those hours are environment replay.

We then retrained the strongest baselines of Table 1 for more steps until each consumed 100 to 101% of PivotOPD's total GPU-hours, under the same checkpoint selection (validation every 10 steps, three seeds). We also trained a replay-free PivotOPD (K=1K=1) for 275 steps on ALFWorld at the same total cost.

Method (Qwen3-8B, 3 seeds)Steps (ALF / WS / QA)Total H100 GPU-hours (ALF / WS / QA)ALFWorld Avg. (%)Search-QA Avg. (%)WebShop ScoreWebShop Succ. (%)
GRPO (Table 1)160 / 160 / 160108.8 / 85.3 / 101.235.337.583.575.0
GRPO (compute-matched)325 / 200 / 212221.0 / 106.6 / 134.138.138.684.476.2
StepOPSD (compute-matched)311 / 191 / 202221.4 / 106.5 / 133.781.243.581.674.1
Skill-SD (compute-matched)296 / 182 / 193221.1 / 106.5 / 133.689.841.082.674.0
OPID (compute-matched)296 / 182 / 193221.1 / 106.5 / 133.685.446.282.175.0
RLSD (compute-matched)311 / 191 / 202221.4 / 106.5 / 133.791.535.485.379.8
SDAR (compute-matched)311 / 191 / 202221.4 / 106.5 / 133.785.642.686.280.5
PivotOPD (K=1K=1, replay-free)275 / 160 / 160221.0 / 106.4 / 133.592.147.488.281.9
PivotOPD (default, K=2/1/1K=2/1/1)160 / 160 / 160221.1 / 106.4 / 133.593.047.488.281.9

Compute parity does not close the gap: 91.5 vs. 93.0 on ALFWorld, 46.2 vs. 47.4 on Search-based QA, 86.2 vs. 88.2 in WebShop score, 80.5 vs. 81.9 in WebShop success. The best compute-matched baseline also changes from benchmark to benchmark (RLSD, OPID, SDAR), whereas PivotOPD is first on all of them.

Extra steps do not help for the reason given by Proposition 1 and Figure 1c. Basically, methods that update only on student-sampled trajectories plateau once the non-pivotal failures are gone, because the student assigns probability below 10−210^{-2} to the recovery action after a pivotal mistake, so more rollouts still rarely sample one. Also important to note that replay here is a cost knob, not the source of the gain. With replay removed (K=1K=1, total overhead over GRPO about +18% with all teacher calls included) and the saved hours reinvested into 275 steps, PivotOPD still beats every compute-matched baseline (92.1 vs. 91.5 for RLSD). K=1K=1 is therefore a practical replay-free operating point, and K=2K=2 buys the last point on ALFWorld (93.0).

Revision: Appendix E.4 (Table 9) reports the GPU-hour breakdown and the compute-matched comparison.

AuthorsOpenReview ↗

Response to Reviewer UkNS (Part 2)

W2 / Q3. The SWE-Bench experiment uses a narrower configuration (recovery budget zero, only the final action audited, the external teacher's hinted distribution), so it shows preventive intervention on another model family but not transfer of the recovery mechanism. Can the authors enable recovery distillation?

We ran the full PivotOPD, with earlier-turn pivot detection and recovery distillation, on SWE-Bench Verified, Terminal-Bench 2.1, and SWE-Bench Multilingual. Recovery distillation more than doubles the gain of prevention alone (+4.7 vs. +2.1 macro-average), most of all on Terminal-Bench.

The reviewer's description was accurate, as we had set K=0K=0 to avoid container replay. But as Section 3.3 and Appendix E.1 note, the first recovery turn starts from the recorded post-mistake state ct+1c_{t+1} and needs no replay. The full configuration therefore lets the teacher audit up to m=3m=3 candidate turns before the final committed action, uses K=1K=1, samples the recovery response from Nemotron-3-Super hinted with the recovery action (as the preventive term already did here), and applies it at wrec=0.05w_\text{rec}=0.05.

Nemotron-3.5-SFT student, 3 seedsTerminal-Bench 2.1, 89 tasks (%)SWE-Bench Multilingual, 300 tasks (%)SWE-Bench Verified, 500 tasks (%)Macro-avg. (%)Teacher gap closed
Student37.5 ± 0.955.0 ± 0.862.8 ± 0.651.8 ± 0.50%
PivotOPD, preventive only (K=0K=0)39.3 ± 1.0 (+1.8)56.4 ± 0.9 (+1.4)66.0 ± 0.8 (+3.2)53.9 ± 0.6 (+2.1)11.8%
PivotOPD, full (with recovery)43.4 ± 0.9 (+5.9)58.0 ± 0.8 (+3.0)68.1 ± 0.7 (+5.3)56.5 ± 0.5 (+4.7)26.2%
Nemotron-3-Super teacher65.9 ± 0.770.5 ± 0.673.0 ± 0.569.8 ± 0.4100%

On SWE-Bench Verified, recovery distillation adds +2.1 on top of prevention and takes the share of the teacher gap closed from 31% to 52%. On Terminal-Bench, prevention alone adds +1.8 and recovery distillation raises the gain to +5.9, since early command-line mistakes leave states that the agent has to repair rather than avoid. On SWE-Bench Multilingual, +1.4 becomes +3.0 across nine languages. The recovery mechanism validated on ALFWorld (Figure 5b, 45.8% post-mistake recovery for preventive only against 72.7% for PivotOPD) therefore transfers to another model family and to containerized environments.

In the uploaded revised manuscript, Sections 4.1 and 4.2 and Figure 4 now cover all three benchmarks, and Appendix D.6 describes the full configuration.

Q2. Can the authors quantify uncertainty for the relatively small 8B margins using paired task and seed comparisons? Are those advantages consistent across seeds?

Yes. PivotOPD beats the strongest baseline on 3 of 3 seeds on every benchmark, and every paired task-level 95% bootstrap confidence interval lies above zero (p≤0.023p \le 0.023).

All methods in Table 1 and Figure 3 share the same three seeds and the same held-out tasks (274 ALFWorld tasks, 725 Search-based QA questions, 500 WebShop instructions), so we pair the comparison per seed and per task (4,000 bootstrap draws, as in Appendix A).

Benchmark (Qwen3-8B)PivotOPD (mean ± SD)Strongest baseline (mean ± SD)Per-seed diffs (S0 / S1 / S2)Paired diff (mean ± SD)Seed winsTask-level 95% CIPaired pp
ALFWorld (274)93.0 ± 0.9RLSD, 90.8 ± 1.0+2.3 / +2.2 / +2.1+2.2 ± 0.13 / 3[+0.6, +3.9]0.008
Search-QA (725)47.4 ± 1.0OPID, 45.6 ± 1.1+1.9 / +1.8 / +1.7+1.8 ± 0.13 / 3[+0.4, +3.2]0.011
WebShop score (500)88.2 ± 1.8SDAR, 85.5 ± 1.4+2.3 / +2.7 / +3.1+2.7 ± 0.43 / 3[+0.9, +4.5]0.004
WebShop succ. (500)81.9 ± 3.4SDAR, 79.7 ± 3.2+2.0 / +2.2 / +2.4+2.2 ± 0.23 / 3[+0.3, +4.1]0.023

The paired differences are much tighter than the marginal SDs because a seed shifts the training progress of every method in the same direction. Marginal SDs on Qwen3-8B are 0.9 to 1.8 (3.4 on WebShop success), whereas the seed-level SD of the paired difference is 0.1 to 0.4.

Without an external teacher (Figure 3), PivotOPD also wins 3 of 3 seeds against the strongest baseline on each benchmark, with paired differences of +1.5 ± 0.2 on ALFWorld (vs. RLSD, 82.8 ± 1.0 vs. 81.3 ± 1.1, CI [+0.2, +2.9]), +2.2 ± 1.2 on WebShop success (vs. GRPO, 77.2 ± 2.1 vs. 75.0 ± 3.3, CI [+0.4, +4.1]), and +1.7 ± 0.2 on Search-based QA (vs. OPID, 46.6 ± 0.9 vs. 44.9 ± 1.1, CI [+0.3, +3.1]).

In the uploaded revised manuscript, Appendix E.8 (Table 17) adds this paired analysis, including the second-best baselines, and Tables 1, 2, and 7 and Figures 3 and 4 now report mean ± SD over the three seeds.

AuthorsOpenReview ↗

Request for feedback on our responses to Reviewer UkNS

Dear Reviewer UkNS,

With less than 24 hours left in the discussion period, we would be grateful if you could take a look at our responses and the revised manuscript. Both experiments you requested are now in the paper, the compute-matched comparison with the strongest baselines (Appendix E.4, Table 9), where PivotOPD still leads on every benchmark at equal GPU-hours, and the full recovery mechanism on SWE-Bench Verified, Terminal-Bench 2.1, and SWE-Bench Multilingual (Figure 4, Appendix D.6), where recovery distillation more than doubles the gain of prevention alone. The paired seed and task-level analysis you asked about is in Appendix E.8.

If anything remains unclear, we would be glad to address it before the deadline. If our responses resolve your concerns, we would appreciate it if you could reflect that in your assessment.

Thank you for your time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear AC and Reviewers,

We thank you for the careful and constructive reviews. We ran every experiment the reviews asked for, and the uploaded revised manuscript contains all of them, marked in red. We summarize the most important additions here and answer each reviewer point by point below.

1. Recovery distillation transfers to another model family and to terminal and software-engineering tasks (Figure 4, Section 4.2, Appendix D.6). We ran the full PivotOPD, with earlier-turn pivot detection (m=3m=3) and recovery distillation (K=1K=1, no container replay), on a Nemotron-3.5-SFT student with Nemotron-3-Super as the teacher, and extended the evaluation from SWE-Bench Verified to Terminal-Bench 2.1 and SWE-Bench Multilingual (three seeds).

Nemotron-3.5-SFT, resolve rate (%)Terminal-Bench 2.1SWE-Bench MultilingualSWE-Bench Verified
Student37.5 ± 0.955.0 ± 0.862.8 ± 0.6
PivotOPD, preventive only (K=0K=0)39.3 ± 1.056.4 ± 0.966.0 ± 0.8
PivotOPD43.4 ± 0.958.0 ± 0.868.1 ± 0.7
Nemotron-3-Super teacher65.9 ± 0.770.5 ± 0.673.0 ± 0.5

PivotOPD raises the resolve rate by +5.9, +3.0, and +5.3 points and closes about half of the gap to the teacher on SWE-Bench Verified. Recovery distillation more than doubles the gain of prevention alone on average (+4.7 vs. +2.1), most of all on Terminal-Bench, where early command-line mistakes leave states that the agent must repair rather than avoid.

2. The baselines do not catch up at equal training cost (Appendix E.4, Table 9). Counting end-to-end GPU-hours on both the student node and the teacher endpoint, including pivot detection, every teacher query, and environment replay, we retrained the strongest baselines until each consumed 100 to 101% of PivotOPD's cost. They still trail on every benchmark (91.5 vs. 93.0 on ALFWorld, 46.2 vs. 47.4 on Search-based QA, 86.2 vs. 88.2 in WebShop score), and a replay-free PivotOPD with only +18% overhead over GRPO still beats every compute-matched baseline.

3. The gains do not come from privileged information or from RL (Appendix E.5, Tables 10 and 11). Baselines given PivotOPD's own teacher report as hints still trail it on every benchmark, and removing the outcome flag from PivotOPD's teacher prompt changes its results by at most 0.9. Without any RL, the two distillation terms alone account for roughly 90% of the gain (69.8% and 90.2% ALFWorld success for the 1.7B and 8B students).

4. Standard OPD fails to repair pivotal mistakes even with the same external teacher and gold-action hints, on all three benchmarks (Appendix E.6, Tables 12 and 13). With Qwen3.5-122B-A10B as its teacher and the same gold actions, standard OPD leaves 39.3% of ALFWorld tasks failing after a pivotal mistake against 10.7% for PivotOPD, and it recovers from fewer than 32% of pivotal mistakes on ALFWorld, WebShop, and Search-based QA against 72.7%, 68.4%, and 51.9% for PivotOPD.

5. Exact-turn agreement with the oracle (Tables 14 and 15). The detected pivotal turns hit the exact first oracle turn in 54.6% and 68.3% of failed trajectories (3.8 to 5.7 times random), the named gold action is oracle-optimal on 79.4% and 88.6% of those turns, and exact-turn precision and recall reach 46.1% / 60.9% and 58.1% / 72.6%.

6. Comparison with existing error-recovery methods (Appendices E.7 and E.9). We added full PivotOPD to the ALFWorld comparison, improving mean valid seen/unseen success over the best competing methods by +8.2/+5.1 points. On the three Nemotron benchmarks, PivotOPD outperforms Agent-R and SCoRe-style RL in all six paired comparisons (95% bootstrap intervals above zero), with macro-average success of 56.5% vs. 53.9%/53.5% and recovery from tool or environment errors of 44.7% vs. 27.4%/25.9%.

7. Standard deviations and paired tests. Tables 1, 2, and 7 and Figures 3 and 4 now report mean ± SD over three seeds, and Appendix E.8 (Table 17) adds paired per-seed and task-level bootstrap intervals. PivotOPD beats the strongest baseline on 3 of 3 seeds on every Qwen3-8B benchmark, and every paired 95% interval excludes zero (p≤0.023p \le 0.023).

We are happy to clarify anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.