- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies LLM agents that fail mid-task through invalid tool calls, loops, or ungrounded reasoning. Its principle is that failure knowledge is conditional: lessons kept in the agent’s context misfire when the failure is absent, and runtime rules do not learn. It proposes Sentry, a failure-management layer that runs beside the task agent. A detector reads the last W=5 reasoning-action-observation cycles. Schema-invalid actions go to hard repair, which asks the agent for a replacement action. Progress and reasoning-grounding failures go to soft repair, which retrieves at most k=5 matching entries from an external playbook. A verifier judges recovery over the next H=10 cycles without reward access, and only verified recoveries become playbook entries. The full playbook never enters the agent’s context. Experiments use Qwen3.5-9B as agent, detector, and verifier, with GPT-OSS-120B on two benchmarks. Benchmarks are WebShop, AppWorld, and Mind2Web Replay (100 tasks each) and SWE-bench Lite (50 tasks). Baselines are four runtime-intervention methods (AgentGuard, AgentFixer, Wink, AgentForesight) and four context-evolution methods (Reflexion, ACE, AWM, ReasoningBank). Sentry beats the best runtime baseline by 37% on average and ACE by 39%. Removing hard repair drops WebShop reward from 0.468 to 0.297. Persistent playbook exposure lowers held-out WebShop reward from 0.364 to 0.296. On 200 paired forks, local recovery rises from 33.5% to 60.5%. Token usage is 1.54x the base agent.
Critical strengths
Strengths
- The main claim is simple and can be tested: learn from failures across tasks, show that knowledge only when a failure is detected, and keep a lesson only after the recovery is confirmed.
- Two experiments test this claim directly. The same stored playbook scores 0.364 on WebShop when used only on failure, and 0.296 when it is also kept in the agent's context all the time. Removing failure lessons from ACE's own playbook raises ACE from 0.335 to 0.391.
- The paper compares against eight baselines, four runtime methods and four memory methods, all on the same 9B model and using the authors' official code.
- Every main number is a mean and standard deviation over five runs. Removing any one of the six components lowers the score on both WebShop and AppWorld.
- The paper forks 200 trajectories at the point of intervention and runs both branches. Local recovery rises from 33.5% to 60.5%. The intervened branch ends better in 74 cases, the same in 101, and worse in 25.
- Humans checked 200 intervention windows without knowing the labels. They agreed with the system 86.0% of the time on whether to intervene and 81.8% on whether recovery happened. Agreement between annotators was high.
- The paper reports cost and sensitivity. Sentry uses 1.54 times the base tokens. Random retrieval scores 0.33 and matched retrieval 0.50. Results stay stable across window and horizon settings. All prompts and an example playbook are given.
Critical weaknesses
Weaknesses
- The runtime baselines do not get the same chances as Sentry. Sentry checks every step and can regenerate a bad action. Wink checks only every five steps and intervened just 7 times on Mind2Web. AgentForesight was built to predict failures in multi-agent systems, not to repair a single agent. Each baseline should get the same per-step check and the same repair option.
- One component, hard repair, gives much of the gain. Without it, WebShop drops from 0.468 to 0.297 and AppWorld from 38.53 to 31.20. No baseline has this option. The paper should report Sentry without hard repair against the baselines.
- The task sets are small: 100 tasks per benchmark and 50 on SWE-bench Lite. On SWE-bench the gap of 0.30 vs 0.24 is only 3 tasks, and a spread of ±0.01 on 50 tasks is hard to believe. The memory baselines run on only two benchmarks, and the larger 120B model is a single run on two benchmarks. Report per-run task counts and run every baseline on all four benchmarks.
- Mind2Web Replay is the authors' own version of the benchmark. When an action is rejected, the agent gets feedback and stays in place. Any method that retries gains from this. Only the two retry-capable methods, AgentFixer at 0.687 and Sentry at 0.736, are far above the base of 0.420. Report with a fixed retry budget and the official metric.
- The same model is the agent, the failure detector, and the recovery judge. The 81.7% recovery rate and the fork results are graded by Sentry's own judge. The human check was done by two authors. Use a separate judge model and outside annotators.
- The control for "playbook always in context" adds text to the context, so the drop may come from a longer input and not from the playbook content. Also, the held-out tasks come from the same benchmark as the training tasks. Add a control with the same amount of unrelated text, and test the playbook on a different benchmark.
- The "intervene at every step" ablation is a weak comparison. It fires 347 times and scores below the base agent, which only shows that constant interruption hurts. Also, the agent is limited to 384 output tokens while Sentry's own calls get 4096, so cut-off actions become failures that Sentry then fixes. Compare against a simple no-progress trigger, and rerun the base agent with the larger token limit.
Questions for authors
I have no questions
6 comments
3. Completing Four-Benchmark Evaluations Across Both Model Sizes
We thank the reviewer for pointing out the limited evaluation coverage, missing context-evolution benchmark comparisons, and single-seed GPT-OSS-120B results. We have now completed the missing benchmark comparisons and expanded the GPT-OSS-120B evaluation from a single seed to five seeds, covering both sets of baselines across all four benchmarks.
3.1. Qwen3.5-9B: Completing Context-Evolution Comparisons on All Four Benchmarks
We have completed the context-evolution comparisons on AppWorld and SWE-bench Lite, using the same evaluation task sets and held-out protocol as in the paper (Table 2, §4.3). In the following table, bold entries indicate the newly added results.
| Method | WebShop (avg. reward) | Mind2Web Replay (completion) | AppWorld (test pass rate, %) | SWE-bench Lite (accuracy) |
|---|---|---|---|---|
| Base agent | 0.186 ± 0.014 | 0.421 ± 0.026 | 24.92 ± 0.62 | 0.20 ± 0.01 |
| Reflexion | 0.295 ± 0.025 | 0.531 ± 0.023 | 28.75 ± 1.09 | 0.22 ± 0.04 |
| ACE | 0.335 ± 0.019 | 0.562 ± 0.020 | 35.22 ± 0.81 | 0.24 ± 0.03 |
| AWM | 0.270 ± 0.021 | 0.470 ± 0.018 | 31.50 ± 1.12 | 0.28 ± 0.02 |
| ReasoningBank | 0.250 ± 0.015 | 0.483 ± 0.020 | 26.94 ± 0.73 | 0.28 ± 0.03 |
| Sentry | 0.495 ± 0.017 | 0.729 ± 0.019 | 40.17 ± 1.24 | 0.32 ± 0.02 |
| Strongest Baseline + Sentry | ACE: 0.555 ± 0.027 | ACE: 0.789 ± 0.022 | ACE: 49.56 ± 1.52 | AWM: 0.36 ± 0.04; ReasoningBank: 0.32 ± 0.02 |
The expanded results support our original findings: Sentry outperforms the strongest context-evolution baseline by 26% on average across all four benchmarks, including 14% on each new benchmark. The baseline + Sentry combination achieves the highest mean performance on every benchmark.
3.2. GPT-OSS-120B: Five-Seed Comparisons of Both Baseline Families on All Four Benchmarks
The paper reported GPT-OSS-120B results on WebShop and Mind2Web Replay against runtime-intervention baselines using a single seed (§5.4; Table 10). We now evaluate both baseline families on all four benchmarks with five seeds.
We retain the respective evaluation protocols from the paper: online-from-scratch evaluation for runtime-intervention baselines and held-out evaluation for context-evolution baselines. Both tables below present the updated five-seed results.
3.2.1. Runtime-Intervention Baselines
GPT-OSS-120B – Runtime-Intervention Baselines:
| Method | WebShop (avg. reward) | Mind2Web Replay (completion) | AppWorld (test pass rate, %) | SWE-bench Lite (accuracy) |
|---|---|---|---|---|
| Base agent | 0.201 ± 0.011 | 0.552 ± 0.003 | 47.50 ± 1.17 | 0.26 ± 0.01 |
| AgentGuard | 0.238 ± 0.032 | 0.695 ± 0.073 | 49.79 ± 2.86 | 0.30 ± 0.04 |
| AgentForesight | 0.149 ± 0.082 | 0.618 ± 0.041 | 52.33 ± 1.01 | 0.24 ± 0.02 |
| AgentFixer | 0.262 ± 0.022 | 0.732 ± 0.029 | 54.29 ± 2.15 | 0.28 ± 0.06 |
| Wink | 0.199 ± 0.015 | 0.592 ± 0.019 | 50.01 ± 3.54 | 0.28 ± 0.06 |
| Sentry | 0.549 ± 0.094 | 0.812 ± 0.045 | 63.73 ± 3.20 | 0.38 ± 0.04 |
3.2.2. Context-Evolution Baselines
GPT-OSS-120B – Context-Evolution Baselines:
| Method | WebShop (avg. reward) | Mind2Web Replay (completion) | AppWorld (test pass rate, %) | SWE-bench Lite (accuracy) |
|---|---|---|---|---|
| Base agent | 0.204 ± 0.013 | 0.550 ± 0.005 | 48.32 ± 0.80 | 0.26 ± 0.01 |
| Reflexion | 0.283 ± 0.056 | 0.653 ± 0.052 | 48.49 ± 1.47 | 0.32 ± 0.04 |
| ACE | 0.385 ± 0.035 | 0.679 ± 0.031 | 56.35 ± 3.20 | 0.32 ± 0.02 |
| AWM | 0.372 ± 0.047 | 0.613 ± 0.022 | 55.25 ± 2.29 | 0.36 ± 0.02 |
| ReasoningBank | 0.386 ± 0.088 | 0.628 ± 0.043 | 53.40 ± 3.79 | 0.28 ± 0.04 |
| Sentry | 0.560 ± 0.073 | 0.817 ± 0.032 | 65.90 ± 4.21 | 0.42 ± 0.02 |
Sentry achieves the highest mean performance on every benchmark against both sets of baselines, with average relative improvements of 41% over the strongest runtime-intervention baseline and 25% over the strongest context-evolution baseline. These expanded experiments support the performance pattern reported in the paper across both model sizes and baseline families.
4. Evaluating Mind2Web Replay With Fixed Submission Budgets and Clarifying the Official Metric
We thank the reviewer for raising the point about retry opportunities in Mind2Web Replay. We have now evaluated all methods under the same fixed per-step budget of R=1,3 environment submissions. If a reference step is not accepted within R submissions, the episode terminates. Thus, R=1 permits a single submission with no retry after rejection, while R=3 permits up to two retries. Agents are not informed of the cap.
Only submissions to the environment count toward this budget. Pre-execution corrections, including Sentry's hard repair and action regeneration in certain baselines, do not count toward R.
| Method | R = 1 | R = 3 |
|---|---|---|
| Base agent | 0.269 ± 0.035 | 0.297 ± 0.030 |
| AgentGuard | 0.284 ± 0.030 | 0.342 ± 0.035 |
| AgentForesight | 0.301 ± 0.012 | 0.301 ± 0.021 |
| AgentFixer | 0.354 ± 0.039 | 0.471 ± 0.047 |
| Wink | 0.261 ± 0.024 | 0.302 ± 0.050 |
| Sentry | 0.432 ± 0.047 | 0.659 ± 0.024 |
Effect of retry constraints. The lower absolute scores compared with our original results arise from the stricter evaluation protocol. The retry constraint reduces the opportunity to recover from rejected actions, lowering absolute completion scores across methods (e.g., the base agent decreases from 0.420 to 0.297 at R=3). Importantly, Sentry maintains the highest mean performance under both R=1 and R=3, indicating that its advantage is not solely attributable to unrestricted environment retries.
Regarding the official metric, as noted in §4.1 and Appendix A.3.1, the official Mind2Web protocol evaluates each step independently: even if an agent fails at step k, it is given the correct reference history and webpage state for step k+1. Thus, failures do not propagate across steps, and the protocol cannot evaluate an agent's ability to recover from its mistakes during execution. We therefore adapt Mind2Web for sequential interaction with feedback, while preserving the original tasks, recorded states, and action correctness criteria. Mind2Web Replay measures normalized trajectory completion rather than the official step-wise metric.
5. Re-Evaluating Sentry's Judgments With Independent Judge GPT-6 Astra
We thank the reviewer for highlighting potential self-judging bias from using Qwen3.5-9B as both the task agent and the failure/recovery evaluator. We first note that the main results (Tables 1–2) use benchmark-defined metrics, and that action-validity recovery (92.6%) is decided by whether the environment accepts the corrected action; neither depends on Sentry's judge. For the remaining LLM-judged results, we re-evaluate Sentry's judgments using GPT-6 Astra, a judge model separate from the original Qwen3.5-9B. Astra receives the same recorded trajectories, evaluation prompts, and context as Qwen's original detector and verifier, without access to Qwen's verdicts. We conduct three evaluations:
5.1. Comparing Intervention Decisions Across Judges (200 Windows)
We evaluate whether GPT-6 Astra agrees with Qwen3.5-9B on when an agent needs intervention. Following §5.2, we use the same 200 trajectory windows, including 113 triggered windows and 87 non-triggered windows. For each window, GPT-6 Astra receives the task objective, current action, and the W = 5 most recent reasoning–action–observation cycles, and independently determines whether intervention is needed.
Results: GPT-6 Astra and Qwen3.5-9B agree on 93.0% of intervention decisions. Against the human annotations, Qwen agrees on 86.0% and Astra on 90.0%.
5.2. Comparing Recovery Judgments Across Judges (99 Windows)
We evaluate whether GPT-6 Astra agrees with Qwen3.5-9B on whether an intervention successfully resolves the detected failure. Using the same 99 human-annotated soft-repair windows from §5.2, Astra receives the diagnosed failure and the subsequent H = 10 reasoning–action–observation cycles, and independently determines whether the agent has recovered.
Results: GPT-6 Astra and Qwen3.5-9B agree on 85.9% of recovery judgments. Against the human annotations, Qwen agrees on 81.8% and Astra on 89.9%.
We thank the reviewer for the careful assessment and constructive suggestions. In response, we have conducted additional experiments to address baseline fairness, the contribution of hard repair, evaluation coverage, judge independence, and potential confounding factors. Below, we report the new results and clarify the scope and limitations of each evaluation.
1. Equalizing Per-Step Checks and Pre-Execution Action Correction Across Runtime Baselines
We thank the reviewer for highlighting the unequal per-step monitoring and pre-execution action-correction opportunities across runtime baselines. We agree that it is important to compare Sentry against strengthened baselines with per-step checking and action correction. Among the runtime-intervention methods, we strengthen AgentGuard and Wink by adding pre-execution action regeneration to both and increasing Wink's checking frequency from every five steps to every step. AgentFixer and AgentForesight remain unchanged since both already check every step and support action regeneration before execution. Under this strengthened setup, all baselines match Sentry's opportunities for checking and correcting actions.
The results below compare Sentry with these strengthened baselines over 5 seeds across all four benchmarks. Bolded entries indicate newly conducted experiments.
Sentry continues to achieve the highest mean performance across all four benchmarks, even when the baselines are given the same opportunities for per-step checking and pre-execution action correction. This suggests that Sentry's advantage cannot be explained solely by more frequent checking or additional opportunities for action correction.
2. Evaluating Sentry Without Hard Repair Against Runtime Baselines
We thank the reviewer for questioning how much of Sentry's performance gain comes from hard repair. To further assess how much of Sentry's advantage comes from hard repair, we compare Sentry without hard repair against all four runtime-intervention baselines. We disable Sentry's hard-repair component while keeping its remaining mechanisms unchanged, and all baselines retain their original configurations. Notably, AgentFixer and AgentForesight retain their pre-execution action-regeneration capabilities, even though Sentry's hard repair is disabled. We report results across all four benchmarks over five seeds. Bolded entries indicate newly conducted experiments; the remaining results were previously reported in the paper.
Without hard repair, Sentry still achieves higher mean performance than all baselines on WebShop, AppWorld, and SWE-bench Lite, and comparable performance to the strongest baseline on Mind2Web Replay. These results suggest that while hard repair contributes to Sentry's performance, it does not fully account for its advantage over existing baselines.