- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces a benchmark called CatchBench for evaluating AI-agent auditing methods before (PRE), during (LIVE), or after (POST) execution. This benchmark considers seven tasks over six scenarios combined from existing agent benchmarks such as SWE-Gym, tau-bench, and Who&When, along with newly constructed datasets. The main findings are concerning the sensitivity of feature improvements to baselines, dataset-construction artifacts, and the limitations of existing auditing methods. The authors evaluate their framework through baseline comparisons and statistical tests, and also acknowledge that several constructed evaluation components fail their own acceptance criteria.
Critical strengths
- The authors make an effort to quantify uncertainty rigorously through statistical tests and sensitivity analysis; however, I do not consider this sufficient to support acceptance.
Critical weaknesses
- Limited scientific novelty and significance overall. The PRE/LIVE/POST framework largely combines tasks that all have different datasets, labels, and metrics without establishing a compelling scientific reason to do so. The paper also does not demonstrate how this aggregation of tasks and datasets addresses the motivating research question. Finally, several of the constructed evaluation components suffer from validity issues as acknowledged by the authors.
- The paper has poor presentation and severe clarity issues. It is extremely difficult to comprehend due to its convoluted prose, unclear figures and captions, and excessive reliance on the appendix for essential details. Even with several passes on the abstract and introduction with careful reading, the connection between the research question, methodology, and scientific conclusions remained unclear.
Questions for authors
- What does the proposed PRE/LIVE/POST framework enable that cannot be obtained by evaluating the tasks independently?
- What are the main scientific conclusions that readers should understand from this work?
2 comments
Thank you to the authors for their careful response. Although I appreciate the clarifications and your effort, however my primary concerns remain. Additionally, there are unresolved issues in several components. As a result, I do not believe the revisions address my concerns, and therefore maintain my original rating.
Response to Reviewer ZGnh
Thank you for pointing out that our opening made it hard to see why we study these tasks together. We revised the abstract to clarify which comparison our design supports. In the revised paper, we also tested how well detectors generalize to unseen tasks and added a first human check on disputed tool labels.
Q1. What do we learn by studying PRE, LIVE, and POST together?
In our view, understanding why an agent failed is hard enough even when we can see the whole run. Looking at its life cycle in stages gives us a more manageable question: what could an auditor have seen before the run, partway through it, or only after it ended? For LIVE and POST, we can explore that question on the same runs.
For instance, on SWE-Gym, we evaluate the same detector on the first quarter of each run (LIVE) and on the complete run (POST). The runs, failure labels, and metric stay fixed. We can therefore ask whether the dependencies between steps help detect failure at either point once we account for run size.
At the first quarter, dependency features appear to add ten ROC-AUC points over a linear size baseline. When we let the baseline fit a curve, the gain falls below one point and its uncertainty interval includes zero. Run length explains much of the apparent early warning signal. We could not check that explanation by comparing scores from benchmarks with different runs and labels.
PRE evaluates a configuration before a run begins. It uses different data and labels, so comparing PRE with LIVE or POST cannot isolate the effect of seeing more of a run. What carries across all three settings is a way to interpret the scores. We check how labels were made, what a simple rule can achieve, and whether a difference between methods is large enough to distinguish from noise.
Q2. What are the main scientific findings?
A benchmark can reward a shortcut. On one PRE source, injecagent, declaration order alone identified every excess capability without a false positive. A method could score perfectly there without reading what any tool does. Pooling sources hides that shortcut.
The choice of baseline can change the conclusion. On complete SWE-Gym traces, as at the first quarter, a flexible size baseline accounts for the apparent benefit of dependency features. On complete tau-bench traces, the gain over a flexible baseline persists even when we hold out entire tasks. The gain over a linear baseline is uncertain under both splits.
Injected failures need validation. Neither injected dataset met all five criteria we set for drawing conclusions about auditors. We use those datasets only to study how auditors respond to the particular failures we injected.
Auditors can locate some failures, although their ranking is uncertain. On Who&When, the strongest language-model judge identified the decisive step in 45% of failed runs, about twice as often as the strongest structural method. The results do not distinguish the top four judges. In the separate unlabeled detection comparison, an agent-specific detector and a standard tabular detector performed about equally well.
Findings 1, 2, and 4 use configuration data or naturally occurring failures. All four findings are independent of the disputed crewai and n8n labels. In the final version, we will put these answers on the first page and define each term where readers first need it.
We also see this paper as an invitation for others to try their hand at auditing agent failures. Our main contribution is the benchmark and what we learned from testing it. We built on existing datasets and adapted them for auditing, including by injecting failures to study how auditors respond.
Thank you again for pressing us on the motivation. Your questions helped us explain more clearly what we set out to do :)