acceptodds
Under review as a conference paper at ICLR 2027

CatchBench: When Can an Agent Failure Be Caught?

Abstract

When can an agent failure be caught? A weak audit score alone cannot identify whether the record or the method is limiting. CatchBench therefore puts one auditor's question to three information states: the declared configuration before a run (PRE), a growing prefix of its trace (LIVE), and the finished trace (POST). Prior benchmarks fix one of these states or vary the telemetry; to our knowledge none scores all three under one task-method interface. Each state admits different questions, so seven task contracts carry their own labels and metrics rather than one leaderboard. Four are evidential; three are Gold-derived mechanism diagnostics. The release scores 72 entrants, from rule scanners and structural models to eleven LLM judges across nine model families (GPT, Claude, Gemini, Gemma, Llama, Qwen, DeepSeek, Mistral, Nova), over 1187 declared configurations and 1162 recorded runs. Every recorded comparison is published as a measured difference with its interval, uncorrected, and no board declares a winner it cannot show. The three sharpest results cut against our own data. One rule reads declaration order alone and reaches a perfect F1 on one of six configuration sources, so a score there measures how the corpus was built. Our admissibility bar then rejected one injected substrate and withheld evidential status from the other. A published structural gain also turns on which size reference it is measured against. A benchmark number is therefore not interpretable until the process behind its labels is published and tested for the shortcut it may leave. We report all three, and regenerate every ordering from released predictions with no model call.

Primary area:
datasets and benchmarks
Keywords:
agent auditing, benchmarks, failure detection, failure localization, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_ZGnhOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a benchmark called CatchBench for evaluating AI-agent auditing methods before (PRE), during (LIVE), or after (POST) execution. This benchmark considers seven tasks over six scenarios combined from existing agent benchmarks such as SWE-Gym, tau-bench, and Who&When, along with newly constructed datasets. The main findings are concerning the sensitivity of feature improvements to baselines, dataset-construction artifacts, and the limitations of existing auditing methods. The authors evaluate their framework through baseline comparisons and statistical tests, and also acknowledge that several constructed evaluation components fail their own acceptance criteria.

Critical strengths
  • The authors make an effort to quantify uncertainty rigorously through statistical tests and sensitivity analysis; however, I do not consider this sufficient to support acceptance.
Critical weaknesses
  • Limited scientific novelty and significance overall. The PRE/LIVE/POST framework largely combines tasks that all have different datasets, labels, and metrics without establishing a compelling scientific reason to do so. The paper also does not demonstrate how this aggregation of tasks and datasets addresses the motivating research question. Finally, several of the constructed evaluation components suffer from validity issues as acknowledged by the authors.
  • The paper has poor presentation and severe clarity issues. It is extremely difficult to comprehend due to its convoluted prose, unclear figures and captions, and excessive reliance on the appendix for essential details. Even with several passes on the abstract and introduction with careful reading, the connection between the research question, methodology, and scientific conclusions remained unclear.
Questions for authors
  • What does the proposed PRE/LIVE/POST framework enable that cannot be obtained by evaluating the tasks independently?
  • What are the main scientific conclusions that readers should understand from this work?
2 comments
AuthorsOpenReview ↗

Response to Reviewer ZGnh

Thank you for pointing out that our opening made it hard to see why we study these tasks together. We revised the abstract to clarify which comparison our design supports. In the revised paper, we also tested how well detectors generalize to unseen tasks and added a first human check on disputed tool labels.

Q1. What do we learn by studying PRE, LIVE, and POST together?

In our view, understanding why an agent failed is hard enough even when we can see the whole run. Looking at its life cycle in stages gives us a more manageable question: what could an auditor have seen before the run, partway through it, or only after it ended? For LIVE and POST, we can explore that question on the same runs.

For instance, on SWE-Gym, we evaluate the same detector on the first quarter of each run (LIVE) and on the complete run (POST). The runs, failure labels, and metric stay fixed. We can therefore ask whether the dependencies between steps help detect failure at either point once we account for run size.

At the first quarter, dependency features appear to add ten ROC-AUC points over a linear size baseline. When we let the baseline fit a curve, the gain falls below one point and its uncertainty interval includes zero. Run length explains much of the apparent early warning signal. We could not check that explanation by comparing scores from benchmarks with different runs and labels.

PRE evaluates a configuration before a run begins. It uses different data and labels, so comparing PRE with LIVE or POST cannot isolate the effect of seeing more of a run. What carries across all three settings is a way to interpret the scores. We check how labels were made, what a simple rule can achieve, and whether a difference between methods is large enough to distinguish from noise.

Q2. What are the main scientific findings?

  1. A benchmark can reward a shortcut. On one PRE source, injecagent, declaration order alone identified every excess capability without a false positive. A method could score perfectly there without reading what any tool does. Pooling sources hides that shortcut.

  2. The choice of baseline can change the conclusion. On complete SWE-Gym traces, as at the first quarter, a flexible size baseline accounts for the apparent benefit of dependency features. On complete tau-bench traces, the gain over a flexible baseline persists even when we hold out entire tasks. The gain over a linear baseline is uncertain under both splits.

  3. Injected failures need validation. Neither injected dataset met all five criteria we set for drawing conclusions about auditors. We use those datasets only to study how auditors respond to the particular failures we injected.

  4. Auditors can locate some failures, although their ranking is uncertain. On Who&When, the strongest language-model judge identified the decisive step in 45% of failed runs, about twice as often as the strongest structural method. The results do not distinguish the top four judges. In the separate unlabeled detection comparison, an agent-specific detector and a standard tabular detector performed about equally well.

Findings 1, 2, and 4 use configuration data or naturally occurring failures. All four findings are independent of the disputed crewai and n8n labels. In the final version, we will put these answers on the first page and define each term where readers first need it.

We also see this paper as an invitation for others to try their hand at auditing agent failures. Our main contribution is the benchmark and what we learned from testing it. We built on existing datasets and adapted them for auditing, including by injecting failures to study how auditors respond.

Thank you again for pressing us on the motivation. Your questions helped us explain more clearly what we set out to do :)

Thank you to the authors for their careful response. Although I appreciate the clarifications and your effort, however my primary concerns remain. Additionally, there are unresolved issues in several components. As a result, I do not believe the revisions address my concerns, and therefore maintain my original rating.

Reviewer_v2wDOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

CatchBench organizes external agent auditing around PRE (declared harness), LIVE (trace prefix), and POST (complete trace), with seven task contracts, nine scored boards and 72 entrants over 1,187 configurations and 1,162 runs. Its main contributions are explicit within-task comparison contracts and checks showing how apparent success can reflect label construction or model specification: an order-only rule perfectly solves one PRE source, SWE-Gym dependency gains become unresolved under flexible size control, and neither Gold injection substrate clears its admissibility bar. Tau-bench's dependency increments remain positive as point estimates under both controls. In natural failure localization, the leading frozen judge scores 0.452 Top-1, but its contrasts with the next three do not establish their ordering. LIVE early warning evaluates eventual-length prefixes rather than deployable clock-time alarms; only this within-state sweep holds corpus, labels, metric and entrants fixed. Cross-state scores do not isolate information alone. (pp. 1–4; p. 6 sec. 4.3; pp. 6–8 sec. 5.1–5.4.)

Critical strengths
  1. The study tests explanations for apparent gains and reports when its own data fail. The reported SWE-Gym increment changes from +0.147 under linear control to −0.005 with marginal 95% interval [−0.026, +0.015] under matched splines; the order-only rule perfectly solves injecagent; and the admissibility bar demotes both Gold substrates. These checks provide actionable lessons about benchmark construction and interpreting structural models, which is the strongest case for the paper's scientific value. The spline result remains exploratory, conditional on the saved fits. (p. 2 Figure 1/contributions; p. 6 sec. 4.3; pp. 6–8 sec. 5.1/5.4.)

  2. The task contracts make heterogeneous audit evidence explicit. The shared interface separates PRE/LIVE/POST questions, source label processes, metrics and trivial floors, with localized natural-failure, detection and prefix tasks rather than an aggregate ranking. This gives a useful framework for adding and comparing auditors within a stated contract. The described cached scoring could support reproducibility. (pp. 3–5 sec. 3.1–4.2/Table 2; p. 10 reproducibility statement.)

Critical weaknesses
  1. PRE's surviving method comparisons have limited validation of the target itself. The six sources encode constructed rosters, synthetic grants, observed-use proxies and cross-vendor model agreement; unused does not establish unneeded. MCP's margin depends on one large server, while crewai/n8n have the lowest inter-judge agreement among judged sources and a human audit is identified as future work. Passing construction checks therefore does not establish correct least-privilege decisions. This limits the new corpus's demonstrated validity for actual capability need, while leaving its provenance and shortcut findings valuable. (p. 5 sec. 4.2; pp. 6–7 sec. 5.1; p. 9 sec. 6.)
Questions for authors
  1. For crewai/n8n, what independent evidence, if any, supports the model-derived key as a measure of actual capability need, beyond model agreement and construction checks? Please distinguish existing validation from the human-audit future work and state which method conclusions remain conditional on that key. This could change how much benchmark validity I credit to the new PRE corpus. (p. 5 sec. 4.2; pp. 6–7 sec. 5.1; p. 9 sec. 6.)
2 comments
AuthorsOpenReview ↗

Response to Reviewer v2wD

Thank you for this insightful question. We agree that whether an agent actually needs a tool is worth checking directly. In the revised paper, we report a first independent human check on crewai and show how the method comparisons on crewai and n8n change when we change the labeling rule (Tables 26 to 28).

1. What independent evidence supports the labels?

At submission, neither source had human validation. The two labeling models agreed moderately, we checked for shortcuts, and we evaluated a judge from a third model family. Those checks leave open whether an agent actually needs a particular tool.

During discussion, one annotator outside the team checked 123 crewai library-tool assignments without seeing the model votes or predictions. Of 83 disputed assignments, the annotator judged 45 tools unnecessary and 10 necessary, and left 28 undecided. Where the models agreed, the annotator confirmed 34 of 40 labels. This check covers documented library tools on crewai. For the final version, at least two outside annotators will examine project-defined crewai tools and the n8n templates under a shared decision rule.

2. Which method rankings depend on the labeling rule?

On crewai, the held-out judge scores higher than the scanner under the released rule, which calls a tool excess only if both labeling models judge it unnecessary. Under GPT labels alone, the scanner scores higher and the judge is level with flagging every tool. Counting a tool as excess if either model flags it also puts the judge near that baseline. On n8n, the scanner leads under the released rule and the judge leads under GPT labels alone.

Table 26 compares all four rules, including Claude labels alone, using the same predictions. Six of the eight judge-scanner differences are inconclusive. On crewai, all four are inconclusive when we resample whole repositories. We cannot claim a stable ranking within either source.

3. Which conclusions hold across the rules?

Under every rule, at least one method beats the flag-everything baseline on both crewai and n8n. When we pool all six sources, the judge leads the scanner by roughly three to five F1 points under every rule. Both patterns held in 10,000 random assignments of disputed labels. When we searched for unfavorable assignments, the pooled lead and the crewai lead over baseline each fell to about zero (Table 27). These tests show how the results depend on resolving disagreements between the same two labelers. They do not independently establish whether a tool is needed.

The human check illustrates the distinction (Table 28). After we substituted the decided answers, the judge still beat the baseline on crewai, while its lead over the scanner was uncertain under repository resampling. Counting every undecided case as excess reduced both leads. Only the lead over the baseline was still clear. In a further scenario, added after seeing the answers, we also counted the 65 unchecked disputes as excess. Both leads then disappeared.

4. What changed in the paper?

Section 5.1 now makes the method comparisons on crewai and n8n conditional on how the labels were assigned. Section 4.2 explains why the released labels are close to the Claude labeler's decisions. In most disagreements, only the GPT labeler calls the tool unnecessary, while the released rule requires both labelers to agree. The declaration-order shortcut on injecagent and the findings on synthetic and sweagent use separate label sources.

Thank you again for raising this point. It prompted us to have someone outside the team check those judgments and to show how the choice of labeling rule can change the ranking. We appreciate the care you took in drawing that distinction!

Thank you for the blinded crewai audit and sensitivity analysis. They ve answered my question and show why the judge–scanner ranking depends on the labeling rule. Validation is still limited to documented crewai tools, so I maintain my initial rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to Reviewer sNK2 (1/2): Q1

Thank you for asking these questions and helping us improve CatchBench!

We agree that a researcher needs to know both what the judges saw and how well the labels reflect which tools an agent needs. We added a prompt-reconstruction script to the supplement and report a first independent human check on crewai in the revised paper (Table 28).

Q1a. Can a new auditor see the same inputs as the cached judges?

The released records alone are insufficient. They omit task and role descriptions because some contain contact details and some sources have restrictive or unstated licenses. For most records, another researcher can recover the exact input to the held-out judge.

Please use the updated script in the revised supplement (rebuttal/pre_refetch_verify.py). It fetches the upstream text or rebuilds synthetic records with the released generator. It reconstructs each prompt and checks it against the SHA-256 hash stored with the prediction. On November 11, running from the supplement alone, we obtained an exact match for 1,165 of 1,187 records (98%). The other 22 depended on material that had changed, disappeared, or become private. The supplement includes the prompt builder, model identifier, and decoding settings.

Separately, using our unreleased copies, we verified every cached prompt for the held-out judge and for the labeling model that saved hashes. The other labeling model did not save hashes, so its inputs cannot be verified this way.

Q1b. What independent evidence supports the labels?

At submission we had no human validation. During discussion, one annotator outside the author team checked 123 crewai library-tool assignments. The annotator saw the agent's role, the tool, and its documented function. We hid the model votes, labels, and predictions.

The check covered 83 of the 148 assignments where the labeling models disagreed. The annotator judged 45 tools unnecessary and 10 necessary, and left 28 undecided. Among the decided disputes, 45 judgments matched the GPT labeler and 10 matched the Claude labeler. Of 40 cases where the models agreed, the annotator confirmed 34 labels, disagreed with one, and left five undecided.

Q1c. Do the human judgments change the comparison on crewai?

The released rule marks a tool as excess only when both models judge it unnecessary. We kept predictions fixed and substituted the human answers wherever the annotator reached a decision. The held-out judge still beat the baseline that flags every tool. Its smaller lead over the scanner was inconclusive when we resampled whole repositories.

Counting every undecided case as excess still left the judge ahead of the baseline, with no clear lead over the scanner. If we also counted the remaining 65 unchecked disputes as excess, both leads disappeared (Table 28). Those disputes mostly involve project-defined tools. We added this last scenario after seeing the answers.

Under four labeling rules and 10,000 random assignments of disputed labels, at least one method beat the baseline on both crewai and n8n, and the judge led the scanner when we pooled the six sources. A search for unfavorable assignments brought the pooled lead and the crewai lead over baseline each to about zero.

The human check covers documented library tools on crewai. With one annotator we cannot estimate agreement between people, and n8n still lacks human validation. For the final version, at least two outside annotators will apply a shared decision rule while examining the upstream code for project-defined crewai tools and the n8n templates.

Thank you again for suggesting the human check. In our second reply, we turn to your questions about held-out tasks (Q2) and what a detector can tell during a run (Q3) :)

AuthorsOpenReview ↗

Response to Reviewer sNK2 (2/2): Q2 and Q3

Following up on Q1, thank you for asking how the results change on unseen tasks and what a detector could know during a run. The revised paper reports tests on held-out tau-bench tasks (Table 20) and on runs observed after fixed numbers of steps (Table 15). Section 3 now defines LIVE as a retrospective test.

Q2. Does the gain on tau-bench hold for unseen tasks?

On complete traces, dependency features still helped beyond what the flexible size baseline could explain, even when every test task was new to the detector. We kept all four attempts at each task in the same fold and changed only the split. Every detector scored lower on held-out tasks, by 0.2 to 2.4 ROC-AUC points. The table shows how much adding dependency features improved ROC-AUC on complete traces, in points, pooling the held-out predictions.

Size baselineAttempts split individuallyEntire tasks held out
Linear+4.6+3.2
Spline+4.5+4.1

The linear comparison was inconclusive under both splits. Table 20 gives the intervals obtained by resampling whole tasks. For complete traces, the intervals for the spline and boosted-tree comparisons stayed above zero under both splits, narrowly so on held-out tasks. Averaging scores by fold also gave positive point estimates for all three comparisons (Section 5.4). For partial traces, the gains were inconclusive under both splits. The same four models generated attempts in every fold. Holding out a generating model in a future evaluation would let us ask whether this gain also carries over to runs produced by a new model.

Q3a. Could the reported LIVE score have triggered an alarm during a run?

We can identify the first quarter of a run only after it ends, so the reported score could not have triggered an alarm at that point. Even the number of steps visible at that point gives away information about final length. On SWE-Gym, a flexible model using size alone reached 0.79 ROC-AUC at the first quarter.

The revised text reports the first prefix where ROC-AUC exceeds 0.70. That value describes how well a detector distinguishes failed from successful runs. We have not evaluated a per-run alarm on natural runs. Four generated tables retain the old alarm wording pending an update to the generator.

Q3b. What if we count steps as they arrive?

In the new exploratory test, we scored only runs still active at each step count. On SWE-Gym, size alone performed near chance or worse. Dependency features helped at steps six and eight, while almost all runs were still active. The signal coincided with runs about to end. All 79 runs that ended by step eleven failed. From step twelve onward, no detector exceeded 0.52 ROC-AUC on the remaining runs.

On tau-bench, the dependency features were constant over the first eight steps. Scores stayed near chance through step 24 and improved at step 32 among the runs still active. The gain from adding dependencies was inconclusive at every step tested. A next step would be to choose an alarm threshold on separate runs, then replay held-out runs one step at a time. We could measure how often the alarm fires on successful runs and, for failures it catches, how many steps remain before the run ends.

Thank you again for the care you took with these questions. Your suggestion to keep all attempts at a task together gave us a fairer test of generalization. Asking what the detector could actually see during a run prompted the fixed-step experiment and helped us explain much more clearly what LIVE measures :)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.