- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents Auto Benchmark Audit (ABA), a pipeline using AI agents to collect benchmark artifacts and review task instructions, environments, evaluation logic, and agent trajectories. After reviewing and auditing, ABA generates an audit record containing the category, severity, justification, evidence, and suggested fixes for each finding. Across 34,285 tasks from 168 benchmarks, ABA flags 25.7% as having major issues and 15.1% as having minor issues.
Critical strengths
ABA has a broad coverage of benchmark validity issues and outputs with actionable findings. The validation against externally identified defects provides evidence that ABA can detect real problems.
Critical weaknesses
The paper does not provide enough information to determine whether the auditing rubrics in the audit prompts (Appendix C.4) were developed or refined against the external validation set in Table 2. Not clarifying this can compromise the validity and generalizability of ABA to other benchmarks.
Questions for authors
-
How are the auditing rubrics developed? Can you provide more details about the development and validation isolation of ABA?
-
How stable are the ABA's findings across repeated runs on a same set of tasks?
-
Since simply removing difficult tasks can also increase scores and change rankings, leaderboard changes do not establish that filtering improves benchmark validity. Do the performance and ranking effects persist after repairing confirmed defects? How do they compare with removing equally difficult tasks?
4 comments
Response to Reviewer mGMj (2/2)
3. Hard tasks or broken tasks (Q3)
We appreciate this question, because it is the main threat to the leaderboard results. Flagged tasks have low pass rates, so a naive comparison cannot tell broken tasks from hard ones. We test it in three ways.
(a) Difficulty-matched random removal. We match on difficulty labels set by the benchmark authors. SWE-bench Verified uses the time estimate; Terminal-Bench 2 has a difficulty field and DABstep has levels. We remove the same number of unflagged tasks with the same label mix 2000 times and estimate the matched lift. Lift is the mean change in model score, in percentage points.
| Benchmark (static Major) | Removed | Pass rate of removed tasks: ABA / matched | ABA lift | Matched lift |
|---|---|---|---|---|
| SWE-bench Verified | 21 | 11.2% / 44.9% | +1.8 | +0.3 [−0.1, 0.9] |
| Terminal-Bench 2 | 5 | 14.6% / 44.1% | +1.8 | +0.1 [−1.1, 1.2] |
| DABstep | 120 | 8.5% / 23.0% | +5.3 | +0.0 [−0.6, 0.7] |
Flagged tasks fail far more often than tasks their own authors rated equally hard, so difficulty alone does not explain the lift. We use static flags here because the static auditor never sees agent outcomes. Trajectory flags give the same picture, and we will add both to the manuscript.
(b) Confirmed defects only. Removing the 18 Terminal-Bench 2 tasks that the maintainers fixed in PR #53 raises the mean score by 4.6 pp, against −0.4 pp for matched removal.
(c) Repair instead of removal. Maintainers fix broken tasks rather than delete them, so we also look at what happens after a repair. Terminal-Bench 2.1 repairs these 18 tasks and leaves 61 tasks unchanged. We compare its 21 leaderboard entries with Terminal-Bench 2.0 entries that score the same on the unchanged tasks. On the repaired tasks, the 2.1 entries pass 25% more often, and all 21 sit above the 2.0 trend. The same comparison with 18 random unchanged tasks gives −8% to +10%. So once the issues are fixed, models solve these tasks. This is what we expect if they were broken rather than hard.
Regarding your question on rankings after repair. Across all model pairs, ABA removal reorders no more pairs than matched random removal. The top of the leaderboard behaves differently. When we drop every task with a Major or Minor trajectory flag, a top-15 model moves 2.5 places on average on SWE-bench Verified and 3.5 on Terminal-Bench 2. Matched random removal moves it 1.4 and 1.0 places (App. E.3). We will revise the abstract and Section 5 to state these results.
Revised PDF uploaded
Dear Reviewer mGMj,
Thank you again for your review. We have uploaded a revised PDF that adds how the rubric was built (App. A.2), the repeat-run results with Opus 4.7 (App. D.5), and the difficulty-matched controls (App. E.4). We would be grateful to hear whether our response addresses your concerns, and we are happy to clarify anything before the discussion closes.
Thank you for the additional data and clarification. I have no further questions at this point and would like to maintain my positive rating.
Response to Reviewer mGMj (1/2)
Thank you for recognizing ABA's broad coverage of validity issues, its actionable findings, and its validation against externally identified defects. We agree that the paper should say how the rubric was built, and that the leaderboard results need a control for task difficulty. We address each question below and will add the new results to the appendix.
1. How the rubric was developed (Q1)
Thanks for raising this. The rubric was not built or tuned on any Table 2 set. We built it in three steps before running any Table 2 comparison. First, we collected candidate criteria from the Agentic Benchmark Checklist (Zhu et al., 2025) and BetterBench (Reuel et al., 2024). Second, we kept only the criteria that an auditor can check on one task given its own files. We dropped criteria about reporting or about the benchmark as a whole. Third, we grouped the kept criteria by the three stages every agentic task has: the instruction, the environment, and the evaluation. A broken task fails at one of these stages, so each finding tells the benchmark author which part to fix (App. A.2).
We also iterated on the severity scale. It started with four levels, following OpenAI's SWE-bench Verified annotation. The two middle levels were hard to tell apart and almost always led to the same fix, so we merged them into three. Severity now rests on one test: could most experienced developers in the domain pass the tests from the prompt alone?
The SWE-bench Verified case also shows that the rubric did not need the answers. The auditor flags both tasks from the OpenAI post as Major, and its logs show it never searched the web, read git history, or fetched the related PR. We will expand App. A.2 to list which criteria we kept and why.
2. Consistency across repeated runs (Q2)
We appreciate the reviewer raising this concern. App. D.5 reports a repeat-run consistency check, in which we audited the ScienceAgentBench and BixBench gold subsets several times with the Codex auditor. We additionally ran the same check with Opus 4.7 and extended it to the two frontier leaderboard benchmarks. Each task set was audited three times with the corpus rubric:
More than nine in ten Terminal-Bench 2 and SWE-bench Verified tasks keep the same Major label in all three runs, and Opus 4.7 is far more consistent than Codex. On the App. D.5 metric, Opus 4.7 reaches κ 0.85 on ScienceAgentBench and 0.55 on BixBench, against 0.65 and 0.13 for Codex. A few single-task labels do change between runs, so the corpus-level rates, which average over many tasks, are the most reliable view of the results. We will add these runs to App. D.5 and point to them from the main text.