acceptodds
Under review as a conference paper at ICLR 2027

From Plausible Repairs to Policy Capability

Abstract

A language model can propose a plausible correction for a failed attempt without knowing whether it will help the agent finish. Even when the intervention works, deciding what the policy should learn from it is a separate problem. We introduce RePair, a research harness that connects failure analysis to policy learning. RePair restores the decision state where a proposed change applies and compares the baseline and modified continuations under matched conditions, with the same frozen policy completing both branches. Only interventions that repeatedly improve task completion become positive training evidence. Using the same verified evidence and action coverage, we then compare action-only supervision with a dual-view alternative that also teaches the subgoal, expected progress, and recovery conditions behind the action. Both train separate descendants of the same parent policy, while deployment retains the original action-only interface. Finally, we remove external failure memory and intervention to test whether any improvement remains in the policy itself. On ALFWorld with Qwen2.5-3B-Instruct, we study the effects of verification and training representation together with the cost of finding useful corrections. RePair makes the path from plausible correction to policy capability experimentally testable by separating what is worth learning from how that evidence is retained in the policy.

Primary area:
reinforcement learning
Keywords:
language agents, agentic reinforcement learning, self-improvement, failure analysis, counterfactual verification, policy distillation, long-horizon planning, ALFWorld

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_25ywOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how an agent can learn from corrections proposed after a failed attempt. A correction may help an externally assisted agent finish a task without teaching the policy to do so independently. RePair tests proposed repairs against baseline continuations from the same state, turns beneficial repairs into action and strategy supervision, and evaluates the trained policy with the assistance removed. In ALFWorld, four successful repair programs provide eight training examples for Qwen2.5-3B. The resulting update leaves selection success unchanged at 3/355 tasks. The paper uses this negative result to examine the gap between verified repairs and learned behavior.

Critical strengths

Useful finding: A verified repair can help an agent finish a task without teaching the policy to reproduce it. The paper documents why this happens: the dual-view update supervises only 4 of the 16 verified action positions, so the later steps remain a generalization demand.

Critical weaknesses

Missing controls: The four repair programs contain 16 action positions, but training covers only their initial actions and strategy descriptions. Without action-only or full-sequence supervision controls, the proposed coverage explanation for failed transfer remains untested.

Unclear local learning: Both policies solve only 3/355 tasks, with most episodes ending after consecutive nonexecuted attempts. Without post-training evaluation from the repaired states, unchanged task-reset success cannot distinguish failure to learn a repair from failure to reach its starting state.

Questions for authors
  1. On the same four sources and training budget, how does the dual-view update compare with action-only training and with supervision covering the complete repair sequences?

  2. Does the trained policy improve action validity or continuation success when started from the verified source states or their intermediate states?

1 comment

The stronger-checkpoint experiment gives a positive answer at the repair decision. Starting from a common Qwen2.5-3B checkpoint that solved 81/355 selection tasks, we matched the update budget and exposure to new action targets for RePair and direct teacher-trajectory SFT. After training, we placed each policy at the same ten observed repair states with memory and the repair program disabled. RePair completed 5/10 and 6/10 across the two training seeds. The parent completed 2/10, and direct teacher SFT also completed 2/10 in both fits. The same three tasks that the parent failed were solved independently by both RePair fits (Table 6, p. 9).

The gain was visible in both action choice and task completion. The first executed action matched the verified repair in 7/10 states in each RePair fit, compared with 2/10 for the parent and 1/10 in each teacher fit. In the run that solved six tasks, one successful continuation began with a different action from the demonstration. Any successful continuation counted. Complete teacher trajectories improved the policy’s task-start score in both runs, while the verified repairs produced more recoveries specifically at the decisions where this policy had failed. At the later verified midpoint, both repair fits scored 7/10 against the parent’s 8/10. Evaluating at each starting state shows exactly where the gain occurred (§3.4, pp. 8–9).

For the four sources in the submission, the requested first-action, full-sequence and strategy controls are in Table 5 (p. 8). All four fits completed 0/4 from the repair state and 2/4 when given the first program action. The first attempt executed on 2/4 tasks for both the parent and each fit. All sixteen reset runs reached the original decision, and none of the fifteen runs with matching public input executed the demonstrated first action. This explains why the original task-start score alone could not locate the failure. The stronger-checkpoint result shows that verified repair training can improve autonomous recovery at those kinds of decisions.

Reviewer_YMDSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a key problem in language-agent self-improvement: a plausible repair for a failed attempt may not improve task success, and even a verified beneficial repair may not translate into capability retained by the policy. To address this, the authors propose RePair, which uses failure analysis and historical experience to generate repairs, conducts paired environment tests from the same source state, and admits only stably beneficial interventions into training. It then applies dual-view supervision to jointly supervise actions and their associated subgoals, progress criteria, and recovery conditions, followed by scaffold-free evaluation. In experiments on ALFWorld with Qwen2.5-3B-Instruct, the later validation identifies four beneficial repairs, but all successful outcomes are achieved by the intervention programs themselves. After training on the eight corresponding examples, the policy still solves 3 of 355 tasks. Through trajectory analysis and supervision audits, the paper illustrates the gap between intervention utility and capability internalization, identifying incomplete supervision coverage as a potential bottleneck.

Critical strengths
  1. RePair conducts paired tests of the baseline and candidate repair under the same source state, policy, and remaining budget, providing reasonable control over factors that affect intervention outcomes. Running a fresh baseline continuation also accounts for the possibility that another attempt may succeed on its own. Selecting training data based on terminal task completion avoids treating local progress as evidence of a beneficial repair and provides a clear basis for converting failure experience into training examples.
  2. Dual-view supervision preserves the action-prediction task while adding supervision for subgoals, expected progress, and recovery conditions, giving the model an opportunity to learn the decision structure behind an action. The strategy view is used only during training, while evaluation retains the original action interface and removes external assistance. This allows a clear assessment of the capability retained by the policy after training.
  3. The authors carefully audit the verification process and the supervision actually used for training, finding that the four successful programs cover 16 action positions, whereas training provides only four initial-action targets. All 20 successful intervention branches also complete without any subsequent policy calls. Together with traces showing missed prerequisites and repeated actions, these analyses clarify the source of successful behavior and provide concrete directions for studying the relationship between supervision coverage and capability transfer.
Critical weaknesses
  1. The paper’s incremental contribution over prior work has not been sufficiently validated, which is my primary concern. CSO already verifies alternative actions through continuation with the current policy and uses the verification results for policy training [1]; Re-ReST uses environment feedback and reflection to improve generated samples for agent self-training [2]; Distilling Step-by-Step uses rationales as auxiliary targets alongside task targets to supervise smaller models [3].
  2. The existing experiments do not establish the main reason for the failure of capability transfer. The reported update uses only four source states, eight examples, and eight optimizer steps, with 51 and 2,546 supervised tokens per pass for the action and strategy views, respectively. Without action-only, full-action-sequence supervision, and comparable-training-budget controls, it is difficult to disentangle the effects of data quantity, action coverage, and supervision objective design. The evidence is therefore insufficient to establish incomplete action coverage as the primary bottleneck.
  3. Basic execution failures may limit the evaluation’s sensitivity to the transfer of strategy knowledge. The parent policy solves only 3 of 355 tasks, with 261 episodes terminating after consecutive nonexecuted attempts and 1,556 out-of-menu actions. The paper does not report scaffold-free execution results for the updated policy starting from the verified source states or intermediate states. It therefore remains unclear whether the policy has learned the local repairs but fails to improve full-task performance because of other execution problems.

[1] Verified Critical Step Optimization for LLM Agents (Li et al., Findings 2026)

[2] Re-ReST: Reflection-Reinforced Self-Training for Language Agents (Dou et al., EMNLP 2024)

[3] Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (Hsieh et al., Findings 2023)

Questions for authors
  1. Compared with the related work discussed in Weakness 1, what is RePair’s central novel mechanism or research finding?
  2. Could the authors provide comparisons among action-only, dual-view, and full-action-sequence supervision using the same parent policy, training sources, and comparable training budgets, to disentangle the effects of data quantity, action coverage, and supervision objective design?
  3. Can the updated policy complete the tasks without external assistance when starting directly from the verified source states and intermediate states? How do the authors distinguish a failure to learn strategy knowledge from limitations in basic execution ability that prevent the policy from applying what it has learned?
1 comment

What does RePair add? A strong teacher can propose a plausible repair that leaves the task unfinished or makes the outcome worse. RePair tests each proposed repair in the task environment against a fresh baseline continuation from the same failure state. Repairs that reliably improve terminal success supply positive supervision. Neutral outcomes show the researcher when a repair was unnecessary or left the failure unresolved. Harmful outcomes become counterexamples for the next round. In the stronger-checkpoint comparison, only ten of nineteen tested repairs met the Benefit criterion. The other nine still supplied evidence for later analysis (§§2.2–2.5 and §3.4).

How does this change the next training round? In the refreshed data-collection setting (R), the retained policy generates new rollouts after each round. The researcher analyzes the failures that policy now produces alongside the previous verification results, proposes new repairs, and tests them from the new failure states. This lets the next round collect supervision for failures the policy still makes or has just started making. Earlier neutral and harmful outcomes inform what the researcher proposes and tests next. This is how verification feeds back into failure analysis and data collection across rounds (§2.6 and Appendix D).

CSO verifies an alternative action by testing whether the current policy can finish after taking it. RePair can test a bounded program that performs several steps, so the property verified for admission is whether the intervention improves the outcome. RePair then tests the trained policy from the original decision with assistance removed. Re-ReST also uses environmental feedback to improve training samples, and Distilling Step-by-Step uses rationales as auxiliary targets. RePair’s action and strategy targets come from repairs that passed the paired test, while the other tested outcomes remain available to guide the researcher’s next proposals (§4).

What did this distinction reveal? At the original four repair states the baseline failed all twenty runs and the programs completed all twenty intervention runs before another policy call (§3.2, pp. 6–7). Following your suggestion, we crossed first-action versus full-sequence supervision with the strategy target on versus off. Each fit used the submitted parent, the same four sources and eight optimizer steps, seven at nonzero learning rates. Extending coverage increased supervised action tokens at nonzero rates from 177 to 693. Adding strategy supervision contributed 8,928 strategy tokens. All four fits solved 0/4 from the repair decision, 2/4 after the first program action, and 0/4 from reset (Table 5, p. 8). Supplying every later action in training still left the first decision unresolved.

Coverage did matter at a later decision in a separate two-source comparison. On one source both fits took the demonstrated first action. At the second decision their behavior diverged, and the first-action fit completed 0/5 continuations while the full-sequence fit completed 5/5. These are five repetitions at one state (Appendix B.5, p. 20). Taken together, the two controls locate where a change in supervision affected execution and where the same explanation fell short.

Could the policy acquire the repair itself? In the stronger-checkpoint experiment, both repair-trained fits matched the verified first action on 7/10 observed sources, compared with 2/10 for the parent and 1/10 for both teacher-trajectory fits. Any successful continuation counted. Autonomous completions rose from the parent’s 2/10 to 5/10 and 6/10, with the same three repaired tasks gained twice (Table 6, p. 9). The ordering against teacher SFT reversed at task reset. The empirical contribution is this behavioral distinction across starting states. Isolating the effect of the filter alone would require holding the candidate repair pool fixed and varying admission to training.

Reviewer_7NzEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes RePair, a harness that uses strong external models (GPT-5.6) to turn a policy’s failed rollouts into training data. The strong models (1) identify decision states in failed rollouts and the failures behind them; (2) propose “repairs” at these states; and (3) propose a strategy description for each repair. Each repair is verified by restoring the source state and comparing it against the policy’s own continuation, with only repairs that reliably convert failure into success being kept. For each verified repair, the policy is fine-tuned with SFT on the repair’s first action and, as an auxiliary target, its strategy description, and the resulting step is accepted only if it improves on held-out selection tasks with all assistance removed. Experiments use ALFWorld with Qwen2.5-3B-Instruct (LoRA). Five development rounds yield one verified repair out of 21 tested states, and a later validation yields four out of five, and in all four cases the repair program completes the task itself. Fine-tuning on the resulting eight examples leaves selection success unchanged at 3/355 tasks.

Critical strengths

None

Critical weaknesses
  • Unclear motivation. Unclear why learning repair steps like this is a good choice given access to a much stronger external model. Why is this better than rejection sampling, or using the external model to generate good traces, or performing RL? No discussion or ablations or baselines. Results in the Appendix support this view: the external model itself solves 81% / 75% of unseen and seen tasks (Tables 8 and 9, “GPT 5.6 sol high”), which makes distilling its full trajectories an obvious baseline. RL methods in the same table reach about 90%+, so RL is a proven alternative on this exact environment.
  • SFT is done with 8 samples: a single training run with 8 optimizer steps, one eval seed, with a change of outcome on only 2 of 355 tasks. The sample size is far too small to draw meaningful conclusions, and it is unclear how to separate the proposed method's effects from noise.
  • A single, relatively easy task (ALFWorld) on a very small and outdated model (Qwen2.5-3B-Instruct). Unclear whether these contributions are meaningful to the community today. Furthermore, with such a model, failures are largely dominated by simple instruction following errors, which makes it hard to identify true bottlenecks in model performance (see Table 4, which document a large number of out-of-menu actions and formatted errors).
  • No ablations on any design choices.
Questions for authors

For me to raise my score, I will need to see:

  • more task/model combinations.
  • proper training runs done using RePair data.
  • proper baselines (SFT, RL) and ablations on design choices.
  • better discussion motivating the experiments (what problem are you trying to solve?), and why this is a good way to use teacher (GPT) compute.
1 comment

You ask why one would spend strong-teacher compute investigating a failed rollout when the teacher can simply demonstrate a solution. We tested those two uses of the teacher from the same parent. Training on verified repairs taught the policy to finish three of its own previously failed repair states under both training seeds. Training on complete teacher trajectories left it at the parent’s 2/10 on those states, even though it improved task-start success in both runs. This is a practical reason to investigate failures when the objective is to teach recovery at identified decisions.

Which proposed repairs actually helped? We fixed 60 training games, ten per ALFWorld family. The parent failed 48. Nineteen failure states reached paired verification. Across five paired continuations per state, fresh baseline branches completed 2/95 and intervention branches completed 50/95. At the state level, ten repairs met the registered Benefit criterion and nine were neutral. The 95 continuations per arm are repeated tests of nineteen states. Verification therefore gave us both the repairs to train on and a record of the proposals that failed to improve the outcome (revised §3.4, p. 8, and accompanying numerical supplement).

How did the alternatives compare? The same pool yielded 37 successful full teacher trajectories and twelve successful student trajectories. We matched initialization, replay, optimizer, 16 nonzero-rate updates and 363 exposures to new action targets in each fit. The collections supplied 63, 363 and 134 distinct targets for repair, teacher and student training, respectively. Table 6 (p. 9) reports the result with memory and intervention off.

From the ten observed repair states, the common parent completed 2/10 tasks. RePair completed 5/10 in the first training run and 6/10 in the second. Direct teacher SFT completed 2/10 in both runs, and SFT on the student's own successes completed 2/10 in its single run.

From the starts of the 355 selection tasks, the parent completed 81/355. With the first training seed, RePair completed 89/355, direct teacher SFT completed 85/355, and student-success SFT completed 62/355. With the second seed, which we prespecified for the RePair and teacher comparison, RePair completed 79/355 and direct teacher SFT completed 86/355. Evaluation seed 17 was fixed throughout. Relative to the parent, RePair gained 31 tasks and lost 23 in the first run, then gained 24 and lost 26 in the second. The local recoveries recurred even as the aggregate ranking reversed.

What does the extra verification buy? Direct teacher SFT teaches how the teacher solves a task from the beginning. RePair starts at a decision where the student failed and tests which proposed correction actually rescues that attempt. This difference showed up after training. On the same ten repair states, RePair completed 5/10 and 6/10 tasks across the two training seeds, while direct teacher SFT completed 2/10 both times. In both runs, RePair recovered the same three tasks the parent had failed. The head-to-head ranking from task reset varied by seed (Table 6, p. 9).

This targeted recovery came with higher collection costs. RePair recorded 9.27M input and 1.52M output teacher tokens and used 22 GPU-minutes for paired verification. Direct teacher collection recorded 0.545M input and 0.077M output tokens. Section 3.5 (p. 9) also reports the records for which token usage was unavailable.

The revision adds the requested SFT baselines and a stronger checkpoint of Qwen2.5-3B. Appendix B.3 (p. 19) also reports a separate, earlier frozen checkpoint pair on the benchmark, 27 versus 21 of 140 seen games and 19 versus 16 of 134 standard unseen games. The latter split had historical exposure. The 355-task results above concern model selection. A matched policy-RL training run, multiple-draw rejection sampling, and replication on a second backbone or environment remain open comparisons (§5, p. 10).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.