acceptodds
Desk rejected at ICLR 2027

Sentry: Learning to Recover from Failures in LLM Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM agents often fail mid-task due to invalid tool calls, repeated actions, or poorly grounded reasoning, and learning from these failures is a natural path to reliability. We find that *how* failure knowledge reaches the agent matters as much as *what* it contains. Failure lessons are conditional: kept in the agent's context, they misfire when their failure is absent, and removing them from an evolving playbook improves performance. Runtime interventions, in contrast, act only when a failure occurs but do not learn from their own repairs. We argue that failure knowledge is conditional knowledge and should be conditionally exposed, and instantiate this principle in Sentry, a failure-management layer that runs alongside the task agent. When Sentry detects a failure, it retrieves matching lessons from an external playbook to guide recovery, verifies without access to task rewards whether the agent recovered, and stores a new lesson only if it did; the full playbook never enters the agent's context. Across multiple agentic benchmarks, Sentry outperforms the strongest runtime-intervention baseline on every benchmark, by 37% on average, and the strongest context-evolution baseline by 39% on the two benchmarks where both are evaluated; combining Sentry with context evolution yields further gains. Learned lessons transfer to held-out tasks, and controlled experiments show that exposing the full playbook to the agent lowers performance even when relevant lessons remain available on demand.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agent reliability, failure detection and recovery, runtime intervention, self-evolving agents, agent memory

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Tr7pOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies LLM agents that fail mid-task through invalid tool calls, loops, or ungrounded reasoning. Its principle is that failure knowledge is conditional: lessons kept in the agent’s context misfire when the failure is absent, and runtime rules do not learn. It proposes Sentry, a failure-management layer that runs beside the task agent. A detector reads the last W=5 reasoning-action-observation cycles. Schema-invalid actions go to hard repair, which asks the agent for a replacement action. Progress and reasoning-grounding failures go to soft repair, which retrieves at most k=5 matching entries from an external playbook. A verifier judges recovery over the next H=10 cycles without reward access, and only verified recoveries become playbook entries. The full playbook never enters the agent’s context. Experiments use Qwen3.5-9B as agent, detector, and verifier, with GPT-OSS-120B on two benchmarks. Benchmarks are WebShop, AppWorld, and Mind2Web Replay (100 tasks each) and SWE-bench Lite (50 tasks). Baselines are four runtime-intervention methods (AgentGuard, AgentFixer, Wink, AgentForesight) and four context-evolution methods (Reflexion, ACE, AWM, ReasoningBank). Sentry beats the best runtime baseline by 37% on average and ACE by 39%. Removing hard repair drops WebShop reward from 0.468 to 0.297. Persistent playbook exposure lowers held-out WebShop reward from 0.364 to 0.296. On 200 paired forks, local recovery rises from 33.5% to 60.5%. Token usage is 1.54x the base agent.

Critical strengths

Strengths

  1. The main claim is simple and can be tested: learn from failures across tasks, show that knowledge only when a failure is detected, and keep a lesson only after the recovery is confirmed.
  2. Two experiments test this claim directly. The same stored playbook scores 0.364 on WebShop when used only on failure, and 0.296 when it is also kept in the agent's context all the time. Removing failure lessons from ACE's own playbook raises ACE from 0.335 to 0.391.
  3. The paper compares against eight baselines, four runtime methods and four memory methods, all on the same 9B model and using the authors' official code.
  4. Every main number is a mean and standard deviation over five runs. Removing any one of the six components lowers the score on both WebShop and AppWorld.
  5. The paper forks 200 trajectories at the point of intervention and runs both branches. Local recovery rises from 33.5% to 60.5%. The intervened branch ends better in 74 cases, the same in 101, and worse in 25.
  6. Humans checked 200 intervention windows without knowing the labels. They agreed with the system 86.0% of the time on whether to intervene and 81.8% on whether recovery happened. Agreement between annotators was high.
  7. The paper reports cost and sensitivity. Sentry uses 1.54 times the base tokens. Random retrieval scores 0.33 and matched retrieval 0.50. Results stay stable across window and horizon settings. All prompts and an example playbook are given.
Critical weaknesses

Weaknesses

  1. The runtime baselines do not get the same chances as Sentry. Sentry checks every step and can regenerate a bad action. Wink checks only every five steps and intervened just 7 times on Mind2Web. AgentForesight was built to predict failures in multi-agent systems, not to repair a single agent. Each baseline should get the same per-step check and the same repair option.
  2. One component, hard repair, gives much of the gain. Without it, WebShop drops from 0.468 to 0.297 and AppWorld from 38.53 to 31.20. No baseline has this option. The paper should report Sentry without hard repair against the baselines.
  3. The task sets are small: 100 tasks per benchmark and 50 on SWE-bench Lite. On SWE-bench the gap of 0.30 vs 0.24 is only 3 tasks, and a spread of ±0.01 on 50 tasks is hard to believe. The memory baselines run on only two benchmarks, and the larger 120B model is a single run on two benchmarks. Report per-run task counts and run every baseline on all four benchmarks.
  4. Mind2Web Replay is the authors' own version of the benchmark. When an action is rejected, the agent gets feedback and stays in place. Any method that retries gains from this. Only the two retry-capable methods, AgentFixer at 0.687 and Sentry at 0.736, are far above the base of 0.420. Report with a fixed retry budget and the official metric.
  5. The same model is the agent, the failure detector, and the recovery judge. The 81.7% recovery rate and the fork results are graded by Sentry's own judge. The human check was done by two authors. Use a separate judge model and outside annotators.
  6. The control for "playbook always in context" adds text to the context, so the drop may come from a longer input and not from the playbook content. Also, the held-out tasks come from the same benchmark as the training tasks. Add a control with the same amount of unrelated text, and test the playbook on a different benchmark.
  7. The "intervene at every step" ablation is a weak comparison. It fires 347 times and scores below the base agent, which only shows that constant interruption hurts. Also, the agent is limited to 384 output tokens while Sentry's own calls get 4096, so cut-off actions become failures that Sentry then fixes. Compare against a simple no-progress trigger, and rerun the base agent with the larger token limit.
Questions for authors

I have no questions

6 comments

We thank the reviewer for the careful assessment and constructive suggestions. In response, we have conducted additional experiments to address baseline fairness, the contribution of hard repair, evaluation coverage, judge independence, and potential confounding factors. Below, we report the new results and clarify the scope and limitations of each evaluation.

1. Equalizing Per-Step Checks and Pre-Execution Action Correction Across Runtime Baselines

We thank the reviewer for highlighting the unequal per-step monitoring and pre-execution action-correction opportunities across runtime baselines. We agree that it is important to compare Sentry against strengthened baselines with per-step checking and action correction. Among the runtime-intervention methods, we strengthen AgentGuard and Wink by adding pre-execution action regeneration to both and increasing Wink's checking frequency from every five steps to every step. AgentFixer and AgentForesight remain unchanged since both already check every step and support action regeneration before execution. Under this strengthened setup, all baselines match Sentry's opportunities for checking and correcting actions.

The results below compare Sentry with these strengthened baselines over 5 seeds across all four benchmarks. Bolded entries indicate newly conducted experiments.

MethodWebShopMind2Web ReplayAppWorldSWE-bench Lite
AgentGuard + action correction0.292 ± 0.0090.501 ± 0.01427.40 ± 0.550.22 ± 0.02
AgentForesight0.168 ± 0.0110.430 ± 0.01624.17 ± 0.910.24 ± 0.01
AgentFixer0.171 ± 0.0170.687 ± 0.02726.69 ± 0.800.22 ± 0.01
Wink + per-step checks + action correction0.280 ± 0.0110.480 ± 0.02228.91 ± 0.720.22 ± 0.02
Sentry0.468 ± 0.0170.736 ± 0.01938.53 ± 0.870.30 ± 0.01

Sentry continues to achieve the highest mean performance across all four benchmarks, even when the baselines are given the same opportunities for per-step checking and pre-execution action correction. This suggests that Sentry's advantage cannot be explained solely by more frequent checking or additional opportunities for action correction.

2. Evaluating Sentry Without Hard Repair Against Runtime Baselines

We thank the reviewer for questioning how much of Sentry's performance gain comes from hard repair. To further assess how much of Sentry's advantage comes from hard repair, we compare Sentry without hard repair against all four runtime-intervention baselines. We disable Sentry's hard-repair component while keeping its remaining mechanisms unchanged, and all baselines retain their original configurations. Notably, AgentFixer and AgentForesight retain their pre-execution action-regeneration capabilities, even though Sentry's hard repair is disabled. We report results across all four benchmarks over five seeds. Bolded entries indicate newly conducted experiments; the remaining results were previously reported in the paper.

MethodWebShopMind2Web ReplayAppWorldSWE-bench Lite
AgentGuard0.258 ± 0.0190.450 ± 0.01925.09 ± 0.910.22 ± 0.02
AgentForesight0.168 ± 0.0110.430 ± 0.01624.17 ± 0.910.24 ± 0.01
AgentFixer0.171 ± 0.0170.687 ± 0.02726.69 ± 0.800.22 ± 0.01
Wink0.264 ± 0.0150.370 ± 0.02027.56 ± 0.980.18 ± 0.02
Sentry w/o hard repair0.297 ± 0.0160.690 ± 0.02331.20 ± 0.900.28 ± 0.04

Without hard repair, Sentry still achieves higher mean performance than all baselines on WebShop, AppWorld, and SWE-bench Lite, and comparable performance to the strongest baseline on Mind2Web Replay. These results suggest that while hard repair contributes to Sentry's performance, it does not fully account for its advantage over existing baselines.

3. Completing Four-Benchmark Evaluations Across Both Model Sizes

We thank the reviewer for pointing out the limited evaluation coverage, missing context-evolution benchmark comparisons, and single-seed GPT-OSS-120B results. We have now completed the missing benchmark comparisons and expanded the GPT-OSS-120B evaluation from a single seed to five seeds, covering both sets of baselines across all four benchmarks.

3.1. Qwen3.5-9B: Completing Context-Evolution Comparisons on All Four Benchmarks

We have completed the context-evolution comparisons on AppWorld and SWE-bench Lite, using the same evaluation task sets and held-out protocol as in the paper (Table 2, §4.3). In the following table, bold entries indicate the newly added results.

MethodWebShop (avg. reward)Mind2Web Replay (completion)AppWorld (test pass rate, %)SWE-bench Lite (accuracy)
Base agent0.186 ± 0.0140.421 ± 0.02624.92 ± 0.620.20 ± 0.01
Reflexion0.295 ± 0.0250.531 ± 0.02328.75 ± 1.090.22 ± 0.04
ACE0.335 ± 0.0190.562 ± 0.02035.22 ± 0.810.24 ± 0.03
AWM0.270 ± 0.0210.470 ± 0.01831.50 ± 1.120.28 ± 0.02
ReasoningBank0.250 ± 0.0150.483 ± 0.02026.94 ± 0.730.28 ± 0.03
Sentry0.495 ± 0.0170.729 ± 0.01940.17 ± 1.240.32 ± 0.02
Strongest Baseline + SentryACE: 0.555 ± 0.027ACE: 0.789 ± 0.022ACE: 49.56 ± 1.52AWM: 0.36 ± 0.04; ReasoningBank: 0.32 ± 0.02

The expanded results support our original findings: Sentry outperforms the strongest context-evolution baseline by 26% on average across all four benchmarks, including 14% on each new benchmark. The baseline + Sentry combination achieves the highest mean performance on every benchmark.

3.2. GPT-OSS-120B: Five-Seed Comparisons of Both Baseline Families on All Four Benchmarks

The paper reported GPT-OSS-120B results on WebShop and Mind2Web Replay against runtime-intervention baselines using a single seed (§5.4; Table 10). We now evaluate both baseline families on all four benchmarks with five seeds.

We retain the respective evaluation protocols from the paper: online-from-scratch evaluation for runtime-intervention baselines and held-out evaluation for context-evolution baselines. Both tables below present the updated five-seed results.

3.2.1. Runtime-Intervention Baselines

GPT-OSS-120B – Runtime-Intervention Baselines:

MethodWebShop (avg. reward)Mind2Web Replay (completion)AppWorld (test pass rate, %)SWE-bench Lite (accuracy)
Base agent0.201 ± 0.0110.552 ± 0.00347.50 ± 1.170.26 ± 0.01
AgentGuard0.238 ± 0.0320.695 ± 0.07349.79 ± 2.860.30 ± 0.04
AgentForesight0.149 ± 0.0820.618 ± 0.04152.33 ± 1.010.24 ± 0.02
AgentFixer0.262 ± 0.0220.732 ± 0.02954.29 ± 2.150.28 ± 0.06
Wink0.199 ± 0.0150.592 ± 0.01950.01 ± 3.540.28 ± 0.06
Sentry0.549 ± 0.0940.812 ± 0.04563.73 ± 3.200.38 ± 0.04

3.2.2. Context-Evolution Baselines

GPT-OSS-120B – Context-Evolution Baselines:

MethodWebShop (avg. reward)Mind2Web Replay (completion)AppWorld (test pass rate, %)SWE-bench Lite (accuracy)
Base agent0.204 ± 0.0130.550 ± 0.00548.32 ± 0.800.26 ± 0.01
Reflexion0.283 ± 0.0560.653 ± 0.05248.49 ± 1.470.32 ± 0.04
ACE0.385 ± 0.0350.679 ± 0.03156.35 ± 3.200.32 ± 0.02
AWM0.372 ± 0.0470.613 ± 0.02255.25 ± 2.290.36 ± 0.02
ReasoningBank0.386 ± 0.0880.628 ± 0.04353.40 ± 3.790.28 ± 0.04
Sentry0.560 ± 0.0730.817 ± 0.03265.90 ± 4.210.42 ± 0.02

Sentry achieves the highest mean performance on every benchmark against both sets of baselines, with average relative improvements of 41% over the strongest runtime-intervention baseline and 25% over the strongest context-evolution baseline. These expanded experiments support the performance pattern reported in the paper across both model sizes and baseline families.

4. Evaluating Mind2Web Replay With Fixed Submission Budgets and Clarifying the Official Metric

We thank the reviewer for raising the point about retry opportunities in Mind2Web Replay. We have now evaluated all methods under the same fixed per-step budget of R=1,3 environment submissions. If a reference step is not accepted within R submissions, the episode terminates. Thus, R=1 permits a single submission with no retry after rejection, while R=3 permits up to two retries. Agents are not informed of the cap.

Only submissions to the environment count toward this budget. Pre-execution corrections, including Sentry's hard repair and action regeneration in certain baselines, do not count toward R.

MethodR = 1R = 3
Base agent0.269 ± 0.0350.297 ± 0.030
AgentGuard0.284 ± 0.0300.342 ± 0.035
AgentForesight0.301 ± 0.0120.301 ± 0.021
AgentFixer0.354 ± 0.0390.471 ± 0.047
Wink0.261 ± 0.0240.302 ± 0.050
Sentry0.432 ± 0.0470.659 ± 0.024

Effect of retry constraints. The lower absolute scores compared with our original results arise from the stricter evaluation protocol. The retry constraint reduces the opportunity to recover from rejected actions, lowering absolute completion scores across methods (e.g., the base agent decreases from 0.420 to 0.297 at R=3). Importantly, Sentry maintains the highest mean performance under both R=1 and R=3, indicating that its advantage is not solely attributable to unrestricted environment retries.

Regarding the official metric, as noted in §4.1 and Appendix A.3.1, the official Mind2Web protocol evaluates each step independently: even if an agent fails at step k, it is given the correct reference history and webpage state for step k+1. Thus, failures do not propagate across steps, and the protocol cannot evaluate an agent's ability to recover from its mistakes during execution. We therefore adapt Mind2Web for sequential interaction with feedback, while preserving the original tasks, recorded states, and action correctness criteria. Mind2Web Replay measures normalized trajectory completion rather than the official step-wise metric.

5. Re-Evaluating Sentry's Judgments With Independent Judge GPT-6 Astra

We thank the reviewer for highlighting potential self-judging bias from using Qwen3.5-9B as both the task agent and the failure/recovery evaluator. We first note that the main results (Tables 1–2) use benchmark-defined metrics, and that action-validity recovery (92.6%) is decided by whether the environment accepts the corrected action; neither depends on Sentry's judge. For the remaining LLM-judged results, we re-evaluate Sentry's judgments using GPT-6 Astra, a judge model separate from the original Qwen3.5-9B. Astra receives the same recorded trajectories, evaluation prompts, and context as Qwen's original detector and verifier, without access to Qwen's verdicts. We conduct three evaluations:

5.1. Comparing Intervention Decisions Across Judges (200 Windows)

We evaluate whether GPT-6 Astra agrees with Qwen3.5-9B on when an agent needs intervention. Following §5.2, we use the same 200 trajectory windows, including 113 triggered windows and 87 non-triggered windows. For each window, GPT-6 Astra receives the task objective, current action, and the W = 5 most recent reasoning–action–observation cycles, and independently determines whether intervention is needed.

Results: GPT-6 Astra and Qwen3.5-9B agree on 93.0% of intervention decisions. Against the human annotations, Qwen agrees on 86.0% and Astra on 90.0%.

5.2. Comparing Recovery Judgments Across Judges (99 Windows)

We evaluate whether GPT-6 Astra agrees with Qwen3.5-9B on whether an intervention successfully resolves the detected failure. Using the same 99 human-annotated soft-repair windows from §5.2, Astra receives the diagnosed failure and the subsequent H = 10 reasoning–action–observation cycles, and independently determines whether the agent has recovered.

Results: GPT-6 Astra and Qwen3.5-9B agree on 85.9% of recovery judgments. Against the human annotations, Qwen agrees on 81.8% and Astra on 89.9%.

Reviewer_HYt2OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Sentry, a failure-management layer that provide selective failure context to task agent. Sentry outperforms run-time intervention baselines and context-eovlution baselines across benchmarks. The paper also unveils that exposing full playbook can lower perfomance.

Overall the paper provides a sounding solution to balance the context length for task agent, and shows superior performance.

Critical strengths

The paper presents a clear hypothesis, and validates with extensive ablations. It also demonstrates SOTA performance on task completion, meanwhile the token usage and latency is well controlled.

Extensive ablations address the potential confounding variables.

Critical weaknesses

The significant token usage (+54%) is a weak point for this framework. It would be better if authors can benchmark other methods under a similar cost.

Questions for authors

If we hypothesize that the failure playbook is hurting the model performance due to the extra context length, will a bigger model helps to close the gap - e.g. testing context-evolution baseline with GPT-OSS-120B would provide the answer.

4 comments

We thank the reviewer for the kind words and helpful feedback! We address each question and concern below.

1. Comparing Methods Under a Shared Token Budget

We thank the reviewer for raising this concern about token usage, and we agree that this is an important factor in evaluating runtime-intervention methods. We therefore compare all methods under a shared per-task intervention budget, which caps the number of tokens each method can spend on intervention. Exactly matching the cost of every method is not feasible without changing the methods themselves, since they intervene at different frequencies and have different intervention costs. This setting asks which method makes the best use of the same limited budget to help agents recover from failures. Cheap runtime-intervention methods are not penalized, since they can spend less.

For each task, we set the intervention budget to 40% of the base agent's average token usage on that task across five runs. This gives longer tasks larger budgets and shorter tasks smaller budgets. Once a method exhausts its intervention budget on a task, we disable further interventions for that task, and the base agent continues independently. This budget lies below Sentry's own original overhead (+54%; Table 5), so it constrains Sentry and other methods like AgentForesight and AgentFixer.

The table below shows results with Qwen3.5-9B. Bolded entries indicate newly conducted experiments; the remaining results are reused from the paper, as no tasks exceeded the budget for those method–benchmark pairs. We see that Sentry remains the best method on all four benchmarks, outperforming the strongest baseline under the same budget by 30% on average, compared with 37% without the budget, so its advantage persists when intervention cost is limited.

MethodWebShop (avg. reward)Mind2Web Replay (completion)AppWorld (test pass rate, %)SWE-bench Lite (accuracy)
Base agent0.170 ± 0.0160.420 ± 0.03025.59 ± 0.880.20 ± 0.01
AgentGuard0.258 ± 0.0190.450 ± 0.01925.09 ± 0.910.22 ± 0.02
AgentForesight0.162 ± 0.0270.392 ± 0.02626.25 ± 0.700.16 ± 0.02
AgentFixer0.169 ± 0.0140.645 ± 0.02326.04 ± 1.110.20 ± 0.04
Wink0.264 ± 0.0150.370 ± 0.02027.56 ± 0.980.18 ± 0.02
Sentry0.402 ± 0.0130.698 ± 0.01236.07 ± 0.590.28 ± 0.02

2. Model Size and Context Length

Thank you for this insightful question! It has two parts, which we address in turn: whether a larger model can close the performance gap because it is better at handling long context, and whether the harm comes from the extra context length or from the failure-related lessons themselves, which can distract the agent.

2.1. Does a Larger Model Close the Gap?

We evaluated Sentry and all context-evolution baselines using GPT-OSS-120B across all four benchmarks, with five seeds per method on each benchmark. We find that Sentry still outperforms the strongest context-evolution baseline by 25% on average, compared with 26% with Qwen3.5-9B. Sentry's advantage is therefore not specific to the smaller model.

MethodWebShop (avg. reward)Mind2Web Replay (completion)AppWorld (test pass rate, %)SWE-bench Lite (accuracy)
Base agent0.204 ± 0.0130.550 ± 0.00548.32 ± 0.800.26 ± 0.01
Reflexion0.283 ± 0.0560.653 ± 0.05248.49 ± 1.470.32 ± 0.04
ACE0.385 ± 0.0350.679 ± 0.03156.35 ± 3.200.32 ± 0.02
AWM0.372 ± 0.0470.613 ± 0.02255.25 ± 2.290.36 ± 0.02
ReasoningBank0.386 ± 0.0880.628 ± 0.04353.40 ± 3.790.28 ± 0.04
Sentry0.560 ± 0.0730.817 ± 0.03265.90 ± 4.210.42 ± 0.02

2.2. Controlling Context Length

To examine whether context length, rather than actual exposure to failure knowledge, is hurting the task agent's performance, we conducted the following controlled experiments using Qwen3.5-9B on WebShop and Mind2Web Replay.

2.2.1. ACE: In our paper, we compared standard ACE with a variant that removes failure-specific lessons. To test whether the improvement is simply due to reduced context length, we conducted an additional experiment in which we padded the pruned playbook with unrelated Wikipedia text about history to match the length of the original ACE playbook. The full ACE playbook contains 1,532 tokens for WebShop and 1,099 for Mind2Web Replay, of which 659 and 424 tokens, respectively, are failure-specific lessons. Results show that removing failure-specific lessons improves performance even when context length is padded back to the length of standard ACE, suggesting that the gains are not simply due to shorter context.

VariantWebShopMind2Web Replay
ACE0.335 ± 0.0190.562 ± 0.020
ACE w/o failure lessons0.391 ± 0.0090.643 ± 0.021
ACE w/o failure lessons (length-matched to standard ACE)0.387 ± 0.0240.635 ± 0.017

2.2.2. Sentry: Following Table 4 of our paper, we initialize a playbook for Sentry using 50 tasks and evaluate it on 50 disjoint held-out tasks. The frozen playbook contains 848 tokens for WebShop and 974 tokens for Mind2Web Replay. Keeping this playbook and Sentry's mechanism unchanged, we compare three variants, each adding the same number of tokens (848 or 974, respectively) to the task agent's persistent context:

  • Unrelated text: Add length-matched text from an unrelated Wikipedia article.
  • Cross-domain playbook: Add content from a Sentry SWE-bench playbook, truncated to the same length.
  • Persistent exposure: Add Sentry's complete frozen playbook, as in Table 4.

Results show that adding Sentry's full playbook to the task agent's context causes a larger performance drop than adding unrelated text or cross-domain knowledge of the same length, suggesting that the performance degradation is due less to context length than to exposure to failure-specific knowledge.

VariantWebShopMind2Web Replay
Frozen playbook0.364 ± 0.0190.651 ± 0.023
+ unrelated text (playbook length)0.349 ± 0.0250.632 ± 0.019
+ cross-domain playbook (playbook length)0.332 ± 0.0290.635 ± 0.024
+ persistent exposure (playbook length)0.296 ± 0.0170.599 ± 0.018

Taken together, length-matched unrelated text lowers performance by at most 0.02 in both experiments, far less than the effect of failure-specific lessons (0.05–0.08).

Finally, we really appreciate your insightful suggestions, which have helped us strengthen the analysis of the work. We have compared all runtime-intervention methods under a shared token budget, evaluated all context-evolution baselines with GPT-OSS-120B on all four benchmarks, and added length-matched controls for context length. We are committed to incorporating all these additions into the revised version to make the paper as valuable as possible to the community.

Thank you for the additional evidence. My questions are well addressed.

Thank you!

Reviewer_AmQdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies an important research problem about how to recover from LLM agent's mistake. Due to the large amount of hallucination, agent failure recovery is a key component in building a reliable agent system. The authors studied previous approaches: Context evolution and Runtime intervention, and got an interesting finding where simply exposing all failure lessons can even worse the performance. Therefore, the authors designed a Sentry method, a failure-management layer which is in between the Context evolution and Runtime intervention. Sentry monitors the agent failures and retrieves at most k matching lessons from an external playbook instead of exposing all lessons in the context. Then, it checks whether the agent is recovered and adds the successful lesson into the playbook. Experiments are conducted four agent benchmarks WebShop, Mind2Web Replay, SWE-bench Lite, and AppWorld across various domains.

Critical strengths
  1. The authors find an interesting phenomena where exposing all history agent failure experience and lessons might not be a good way for agent itself. Removing failure lessons from playbook recovers performance for both WebShop and Mind2Web Replay.
  2. The proposed Sentry method is a reasonable approach to solve the problem of ACE approaches by only providing k matched lessons to the agent. Experiments show that the proposed Sentry method achieves significantly improvements across different agent benchmarks over the previous ACE or Runtime intervention methods.
  3. The ablation experiments are comprehensive showing both analysis of each sub module as well as the token usage.
Critical weaknesses
  1. One concern is about the result comparison. The comparison with context-evolution baselines only tested two benchmarks, while the comparison with runtime intervention baselines shows all four benchmarks. What caused this disparity of result showing?
  2. Only Qwen3.5-9B was tested with the agent failure recovery methods. It would be stronger if the conclusion works for different models and sizes.
  3. Another variable might need to be studied: context length. The authors claimed that adding full agent playbook is harmful. This could also because adding full playbook increase the conext length and therefore degrade the model performance. The conclusion would be stronger if there is a controlled experiments under similar context length.
Questions for authors

N/A

3 comments

We thank the reviewer for the kind words and helpful feedback! We address each question and concern below.

1. Context-Evolution Baselines on All Four Benchmarks

We thank the reviewer for pointing out that the context-evolution comparison covered only two of the four benchmarks. We have now completed it on AppWorld and SWE-bench Lite, using the same evaluation task sets and held-out protocol as in the paper (Table 2, §4.3); bolded entries are new results. The results are consistent with our findings in the paper: Sentry outperforms the strongest context-evolution baseline on all four benchmarks, by 26% on average and by 14% on each of the two new benchmarks. Moreover, combining Sentry with the strongest baseline gives the best result on every benchmark.

MethodWebShop (avg. reward)Mind2Web Replay (completion)AppWorld (test pass rate, %)SWE-bench Lite (accuracy)
Base agent0.186 ± 0.0140.421 ± 0.02624.92 ± 0.620.20 ± 0.01
Reflexion0.295 ± 0.0250.531 ± 0.02328.75 ± 1.090.22 ± 0.04
ACE0.335 ± 0.0190.562 ± 0.02035.22 ± 0.810.24 ± 0.03
AWM0.270 ± 0.0210.470 ± 0.01831.50 ± 1.120.28 ± 0.02
ReasoningBank0.250 ± 0.0150.483 ± 0.02026.94 ± 0.730.28 ± 0.03
Sentry0.495 ± 0.0170.729 ± 0.01940.17 ± 1.240.32 ± 0.02
Strongest Baseline + SentryACE: 0.555 ± 0.027ACE: 0.789 ± 0.022ACE: 49.56 ± 1.52AWM: 0.36 ± 0.04; ReasoningBank: 0.32 ± 0.02

2. Generalization to a Different Model and Size (GPT-OSS-120B)

We thank the reviewer for suggesting evaluation beyond Qwen3.5-9B. The paper reported GPT-OSS-120B only on WebShop and Mind2Web Replay against runtime-intervention baselines, with a single seed (§5.4; Table 10). We now cover all four benchmarks and both baseline families with five seeds. Task sets and protocols match the paper (online-from-scratch for runtime intervention as in Table 1, held-out for context evolution as in Table 2), hence the slightly different Base agent and Sentry numbers across the two tables. The updated results are presented below. Sentry remains the best method on every benchmark against both baseline families, outperforming the strongest baseline by 41% (runtime intervention) and 25% (context evolution) on average.

For runtime-intervention baselines:

MethodWebShop (avg. reward)Mind2Web Replay (completion)AppWorld (test pass rate, %)SWE-bench Lite (accuracy)
Base agent0.201 ± 0.0110.552 ± 0.00347.50 ± 1.170.26 ± 0.01
AgentGuard0.238 ± 0.0320.695 ± 0.07349.79 ± 2.860.30 ± 0.04
AgentForesight0.149 ± 0.0820.618 ± 0.04152.33 ± 1.010.24 ± 0.02
AgentFixer0.262 ± 0.0220.732 ± 0.02954.29 ± 2.150.28 ± 0.06
Wink0.199 ± 0.0150.592 ± 0.01950.01 ± 3.540.28 ± 0.06
Sentry0.549 ± 0.0940.812 ± 0.04563.73 ± 3.200.38 ± 0.04

For context-evolution baselines:

MethodWebShop (avg. reward)Mind2Web Replay (completion)AppWorld (test pass rate, %)SWE-bench Lite (accuracy)
Base agent0.204 ± 0.0130.550 ± 0.00548.32 ± 0.800.26 ± 0.01
Reflexion0.283 ± 0.0560.653 ± 0.05248.49 ± 1.470.32 ± 0.04
ACE0.385 ± 0.0350.679 ± 0.03156.35 ± 3.200.32 ± 0.02
AWM0.372 ± 0.0470.613 ± 0.02255.25 ± 2.290.36 ± 0.02
ReasoningBank0.386 ± 0.0880.628 ± 0.04353.40 ± 3.790.28 ± 0.04
Sentry0.560 ± 0.0730.817 ± 0.03265.90 ± 4.210.42 ± 0.02

3. Controlling for Context Length

We thank the reviewer for raising context length as a possible confounding factor. We conducted two additional length-matched controls with Qwen3.5-9B and found that the harm comes mainly from exposing failure-specific lessons rather than from the extra context length:

3.1. ACE: In our paper, we compared standard ACE with a variant that removes failure-specific lessons. To test whether the improvement is simply due to reduced context length, we conducted an additional experiment in which we padded the pruned playbook with unrelated Wikipedia text about history to match the length of the original ACE playbook. The full ACE playbook contains 1,532 tokens for WebShop and 1,099 for Mind2Web Replay, of which 659 and 424 tokens, respectively, are failure-specific lessons. Results show that removing failure-specific lessons improves performance even when context length is padded back to the length of standard ACE, suggesting that the gains are not simply due to shorter context.

VariantWebShopMind2Web Replay
ACE0.335 ± 0.0190.562 ± 0.020
ACE w/o failure lessons0.391 ± 0.0090.643 ± 0.021
ACE w/o failure lessons (length-matched to standard ACE)0.387 ± 0.0240.635 ± 0.017

3.2. Sentry: Following Table 4 of our paper, we initialize a playbook for Sentry using 50 tasks and evaluate it on 50 disjoint held-out tasks. The frozen playbook contains 848 tokens for WebShop and 974 tokens for Mind2Web Replay. Keeping this playbook and Sentry's mechanism unchanged, we compare three variants, each adding the same number of tokens (848 or 974, respectively) to the task agent's persistent context:

  • Unrelated text: Add length-matched text from an unrelated Wikipedia article.
  • Cross-domain playbook: Add content from a SWE-bench playbook, truncated to the same length.
  • Persistent exposure: Add Sentry's complete frozen playbook, as in Table 4.

Results show that adding Sentry's full playbook to the task agent's context causes a larger performance drop than adding unrelated text or cross-domain knowledge of the same length, suggesting that the degradation is due less to context length than to exposure to failure-specific knowledge.

VariantWebShopMind2Web Replay
Frozen playbook0.364 ± 0.0190.651 ± 0.023
+ unrelated text (playbook length)0.349 ± 0.0250.632 ± 0.019
+ cross-domain playbook (playbook length)0.332 ± 0.0290.635 ± 0.024
+ persistent exposure (playbook length)0.296 ± 0.0170.599 ± 0.018

Taken together, length-matched unrelated text lowers performance by at most 0.02 in both experiments, far less than the effect of failure-specific lessons (0.05–0.08).

Finally, we really appreciate your insightful suggestions, which have already helped us strengthen both the analysis and presentation of the work. We have also extended the context-evolution comparison with Qwen3.5-9B and the GPT-OSS-120B evaluation to all benchmarks and methods. If the response is helpful, we would be very grateful if you could consider adjusting your score accordingly. We are committed to incorporating all these additions in the camera-ready version to make the paper as valuable as possible to the community.

Hi Reviewer AmQd, as the discussion window is coming to a close, we wanted to kindly follow up on our rebuttal. We have included additional experiments to address your concerns, including expanded evaluations across all four benchmarks, experiments with a larger model, and context-length controls to isolate the effects of failure-lesson exposure. We hope these results help address your main concerns. Please let us know if further clarification would be helpful before the deadline.

Thank you again for your time and thoughtful feedback!

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We would like to briefly summarize the progress made during the rebuttal and discussion period.

All three reviewers recommended acceptance (HYt2: clear acceptance; AmQd and Tr7p: weak acceptance). Reviewer HYt2 replied that their questions were well addressed; Reviewers AmQd and Tr7p have not yet responded to our rebuttal. The main updates are summarized below.

  1. Extending context-evolution comparisons (AmQd, Tr7p). We now compare against context-evolution baselines on all four benchmarks (previously two). Sentry outperforms the strongest context-evolution baseline by 26% on average (relative), and combining Sentry with this baseline gives the best performance on every benchmark.
  2. Expanding the GPT-OSS-120B evaluation (AmQd, HYt2, Tr7p). We expanded the GPT-OSS-120B experiments from one to five seeds, covering all four benchmarks and both baseline families. Sentry leads on every benchmark, outperforming the strongest runtime-intervention and context-evolution baselines by 41% and 25% on average, respectively.
  3. Controlling for context length (AmQd, HYt2, Tr7p). To test whether the performance drop from persistent exposure to failure lessons is simply due to longer contexts, we padded the playbooks to matched lengths. Keeping failure lessons in the agent’s context reduces performance by 0.05–0.08, compared with at most 0.02 for the padding controls.
  4. Controlling for budget and baseline strength (HYt2, Tr7p). Under a shared token budget for interventions, Sentry retains a 30% average advantage over the strongest runtime baseline. We also gave runtime baselines the same opportunities for per-step checks and pre-execution action correction, and applied identical per-step submission caps to all methods on Mind2Web Replay. Sentry remained best in each of these settings.
  5. Validating with an independent judge (Tr7p). We used GPT-6 Astra to re-evaluate when Sentry should intervene, whether its repairs succeeded, and the outcomes of paired runs with and without Sentry. Astra agrees with the original judge on 93.0% of intervention decisions and 85.9% of recovery judgments, and finds comparable gains from Sentry in the paired comparisons.
  6. Ablating the failure detector (Tr7p). Replacing Sentry’s LLM-based failure detector with a rule-based no-progress trigger reduces performance (e.g., 0.468 → 0.326 on WebShop).

We also clarified the Mind2Web Replay evaluation protocol and showed that the task agent’s token limit has negligible impact on action truncation (Tr7p).

We sincerely thank the reviewers for their detailed feedback, which has helped us strengthen the paper. We will incorporate these additional experiments and clarifications into the final version.

Thank you for your time and consideration.

Sincerely,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.