acceptodds
Under review as a conference paper at ICLR 2027

Automated Benchmark Auditing for AI Agents and Large Language Models

Abstract

Modern AI benchmarks operate at a complexity that outpaces traditional verification methods. Tasks authored by domain experts often contain implicit assumptions, incomplete environment specifications, and brittle evaluation logic that human annotation cannot reliably catch. We introduce Auto Benchmark Audit (ABA), an agentic framework that systematically audits individual benchmark tasks, surfacing issues such as hidden environment dependencies, specification gaps, and limited grading logic. We run ABA on a collection of frontier LLM benchmarks and previous NeurIPS publications, totaling 168 benchmarks across nine domains. Across this corpus, ABA identifies issues on over 25.7% of the tasks, and we further validate ABA quality by expert review and independent third-party reports such as upstream PRs. Filtering out these problematic tasks alters model rankings on prominent leaderboards, increasing average performance on SWE-bench Verified and Terminal-Bench 2 by 9.9% and 9.6%, respectively. We release the agentic tool and all task annotations to support the future development of frontier benchmarks.

Primary area:
datasets and benchmarks
Keywords:
Benchmark Audit, Automatic Evaluation, Agents, Benchmark, Evaluation Methods

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over one review, above 85% of the others.
Reviewer_mGMjOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Auto Benchmark Audit (ABA), a pipeline using AI agents to collect benchmark artifacts and review task instructions, environments, evaluation logic, and agent trajectories. After reviewing and auditing, ABA generates an audit record containing the category, severity, justification, evidence, and suggested fixes for each finding. Across 34,285 tasks from 168 benchmarks, ABA flags 25.7% as having major issues and 15.1% as having minor issues.

Critical strengths

ABA has a broad coverage of benchmark validity issues and outputs with actionable findings. The validation against externally identified defects provides evidence that ABA can detect real problems.

Critical weaknesses

The paper does not provide enough information to determine whether the auditing rubrics in the audit prompts (Appendix C.4) were developed or refined against the external validation set in Table 2. Not clarifying this can compromise the validity and generalizability of ABA to other benchmarks.

Questions for authors
  1. How are the auditing rubrics developed? Can you provide more details about the development and validation isolation of ABA?

  2. How stable are the ABA's findings across repeated runs on a same set of tasks?

  3. Since simply removing difficult tasks can also increase scores and change rankings, leaderboard changes do not establish that filtering improves benchmark validity. Do the performance and ranking effects persist after repairing confirmed defects? How do they compare with removing equally difficult tasks?

4 comments
AuthorsOpenReview ↗

Response to Reviewer mGMj (1/2)

Thank you for recognizing ABA's broad coverage of validity issues, its actionable findings, and its validation against externally identified defects. We agree that the paper should say how the rubric was built, and that the leaderboard results need a control for task difficulty. We address each question below and will add the new results to the appendix.

1. How the rubric was developed (Q1)

Thanks for raising this. The rubric was not built or tuned on any Table 2 set. We built it in three steps before running any Table 2 comparison. First, we collected candidate criteria from the Agentic Benchmark Checklist (Zhu et al., 2025) and BetterBench (Reuel et al., 2024). Second, we kept only the criteria that an auditor can check on one task given its own files. We dropped criteria about reporting or about the benchmark as a whole. Third, we grouped the kept criteria by the three stages every agentic task has: the instruction, the environment, and the evaluation. A broken task fails at one of these stages, so each finding tells the benchmark author which part to fix (App. A.2).

We also iterated on the severity scale. It started with four levels, following OpenAI's SWE-bench Verified annotation. The two middle levels were hard to tell apart and almost always led to the same fix, so we merged them into three. Severity now rests on one test: could most experienced developers in the domain pass the tests from the prompt alone?

The SWE-bench Verified case also shows that the rubric did not need the answers. The auditor flags both tasks from the OpenAI post as Major, and its logs show it never searched the web, read git history, or fetched the related PR. We will expand App. A.2 to list which criteria we kept and why.

2. Consistency across repeated runs (Q2)

We appreciate the reviewer raising this concern. App. D.5 reports a repeat-run consistency check, in which we audited the ScienceAgentBench and BixBench gold subsets several times with the Codex auditor. We additionally ran the same check with Opus 4.7 and extended it to the two frontier leaderboard benchmarks. Each task set was audited three times with the corpus rubric:

Task setTasksSame Major label in all 3 runsFleiss' κ (Major label)
Terminal-Bench 289 (all)93%0.55
SWE-bench Verified100 (random subset)93%0.83
BixBench gold1776%0.68
ScienceAgentBench gold1267%0.54
All task sets21890%0.76

More than nine in ten Terminal-Bench 2 and SWE-bench Verified tasks keep the same Major label in all three runs, and Opus 4.7 is far more consistent than Codex. On the App. D.5 metric, Opus 4.7 reaches κ 0.85 on ScienceAgentBench and 0.55 on BixBench, against 0.65 and 0.13 for Codex. A few single-task labels do change between runs, so the corpus-level rates, which average over many tasks, are the most reliable view of the results. We will add these runs to App. D.5 and point to them from the main text.

AuthorsOpenReview ↗

Response to Reviewer mGMj (2/2)

3. Hard tasks or broken tasks (Q3)

We appreciate this question, because it is the main threat to the leaderboard results. Flagged tasks have low pass rates, so a naive comparison cannot tell broken tasks from hard ones. We test it in three ways.

(a) Difficulty-matched random removal. We match on difficulty labels set by the benchmark authors. SWE-bench Verified uses the time estimate; Terminal-Bench 2 has a difficulty field and DABstep has levels. We remove the same number of unflagged tasks with the same label mix 2000 times and estimate the matched lift. Lift is the mean change in model score, in percentage points.

Benchmark (static Major)RemovedPass rate of removed tasks: ABA / matchedABA liftMatched lift
SWE-bench Verified2111.2% / 44.9%+1.8+0.3 [−0.1, 0.9]
Terminal-Bench 2514.6% / 44.1%+1.8+0.1 [−1.1, 1.2]
DABstep1208.5% / 23.0%+5.3+0.0 [−0.6, 0.7]

Flagged tasks fail far more often than tasks their own authors rated equally hard, so difficulty alone does not explain the lift. We use static flags here because the static auditor never sees agent outcomes. Trajectory flags give the same picture, and we will add both to the manuscript.

(b) Confirmed defects only. Removing the 18 Terminal-Bench 2 tasks that the maintainers fixed in PR #53 raises the mean score by 4.6 pp, against −0.4 pp for matched removal.

(c) Repair instead of removal. Maintainers fix broken tasks rather than delete them, so we also look at what happens after a repair. Terminal-Bench 2.1 repairs these 18 tasks and leaves 61 tasks unchanged. We compare its 21 leaderboard entries with Terminal-Bench 2.0 entries that score the same on the unchanged tasks. On the repaired tasks, the 2.1 entries pass 25% more often, and all 21 sit above the 2.0 trend. The same comparison with 18 random unchanged tasks gives −8% to +10%. So once the issues are fixed, models solve these tasks. This is what we expect if they were broken rather than hard.

Regarding your question on rankings after repair. Across all model pairs, ABA removal reorders no more pairs than matched random removal. The top of the leaderboard behaves differently. When we drop every task with a Major or Minor trajectory flag, a top-15 model moves 2.5 places on average on SWE-bench Verified and 3.5 on Terminal-Bench 2. Matched random removal moves it 1.4 and 1.0 places (App. E.3). We will revise the abstract and Section 5 to state these results.

AuthorsOpenReview ↗

Revised PDF uploaded

Dear Reviewer mGMj,

Thank you again for your review. We have uploaded a revised PDF that adds how the rubric was built (App. A.2), the repeat-run results with Opus 4.7 (App. D.5), and the difficulty-matched controls (App. E.4). We would be grateful to hear whether our response addresses your concerns, and we are happy to clarify anything before the discussion closes.

Thank you for the additional data and clarification. I have no further questions at this point and would like to maintain my positive rating.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Reviewer ZNh6 (1/2)

Thank you for recognizing ABA as a practically useful auditing resource, and for noting that the maintainer-fix and BenchGuard comparisons show it finds real defects. We address each concern below.

1. Human validation and the precision multiplication (W1, Q1)

Thank you for raising the point about coverage of human validation, especially on medical and safety/alignment. These two domains have the highest Major rates (41.5% and 42.2%). To bridge this gap, we labeled a new random sample of 25 tasks and a total of 45 findings. One author labeled each task.

FindingsnStrictPartial
Major, paper5472%91%
Minor, paper3063%83%
Major, new sample3168%90%
Minor, new sample1464%79%

We sampled 5 tasks from medical and safety each, and 15 from the rest of 7 domains. Precision on the new sample is close to the paper's for both tiers. Medical and safety/alignment findings, which the paper's sample did not cover, are 88% strict and 94% partial.

Regarding how the sample was drawn, we sampled flagged tasks at random within each of the 12 frontier benchmarks, with a fixed quota per benchmark and severity. The annotators are experts (PhD level) in the field of LLM, math and sciences. They are not blinded to model identities and outcomes because there is only one model the annotators are judging and their goal is to verify the ABA auditor’s findings. 15 of the findings are labeled twice, with 80% raw agreement and Krippendorff's α of 0.66.

We also agree that multiplying finding-level precision by the task-level flag rate is not well founded, because one task can carry several findings. So we recompute precision per task. A task counts as a true Major task if at least one of its Major findings is confirmed. The 54 Major findings sit on 43 tasks.

PrecisionPer finding (paper)Per task (new)
Strict72%86% [77, 93]
Partial91%95% [88, 100]
Band (× 25.7% flag rate)18.5–23.4%22.1–24.5%

Brackets are 95% bootstrap intervals over tasks, stratified by benchmark. Precision per task is higher because a task with one wrong finding often has another finding that is right. We will report the task-level band and call it a rough estimate for the covered domains, not a prevalence estimate.

2. Validity or difficulty (W2, Q2)

We appreciate this question, because it is the main threat to the leaderboard results. Flagged tasks have low pass rates, so a naive comparison cannot tell broken tasks from hard ones. We test it in three ways.

(a) Difficulty-matched random removal. We match on difficulty labels set by the benchmark authors. SWE-bench Verified uses the time estimate; Terminal-Bench 2 has a difficulty field and DABstep has levels. We remove the same number of unflagged tasks with the same label mix 2000 times and estimate the matched lift. Lift is the mean change in model score, in percentage points.

Benchmark (static Major)RemovedPass rate of removed tasks: ABA / matchedABA liftMatched lift
SWE-bench Verified2111.2% / 44.9%+1.8+0.3 [−0.1, 0.9]
Terminal-Bench 2514.6% / 44.1%+1.8+0.1 [−1.1, 1.2]
DABstep1208.5% / 23.0%+5.3+0.0 [−0.6, 0.7]

Flagged tasks fail far more often than tasks their own authors rated equally hard, so difficulty alone does not explain the lift. We use static flags here because the static auditor never sees agent outcomes. Trajectory flags give the same picture, and we will add both to the manuscript.

(b) Confirmed defects only. Removing the 18 Terminal-Bench 2 tasks that the maintainers fixed in PR #53 raises the mean score by 4.6 pp, against −0.4 pp for matched removal.

(c) Repair instead of removal. Maintainers fix broken tasks rather than delete them, so we also look at what happens after a repair. Terminal-Bench 2.1 repairs these 18 tasks and leaves 61 tasks unchanged. We compare its 21 leaderboard entries with Terminal-Bench 2.0 entries that score the same on the unchanged tasks. On the repaired tasks, the 2.1 entries pass 25% more often, and all 21 sit above the 2.0 trend. The same comparison with 18 random unchanged tasks gives −8% to +10%. So once the issues are fixed, models solve these tasks. This is what we expect if they were broken rather than hard.

Regarding your question on rankings after repair. Across all model pairs, ABA removal reorders no more pairs than matched random removal. The top of the leaderboard behaves differently. When we drop every task with a Major or Minor trajectory flag, a top-15 model moves 2.5 places on average on SWE-bench Verified and 3.5 on Terminal-Bench 2. Matched random removal moves it 1.4 and 1.0 places (App. E.3). We will revise the abstract and Section 5 to state these results.

AuthorsOpenReview ↗

Response to Reviewer ZNh6 (2/2)

3. Auditor stability and the supplied trace (Q3)

We appreciate the reviewer raising this concern. In addition to the repeat-run consistency check in App. D.5, we audited each task set below three times with Opus 4.7, the auditor behind the corpus, and the corpus rubric. The SWE-bench Verified set is all 21 corpus Major tasks plus 79 random tasks.

Task setSame Major label in all 3 runsFleiss' κTasks that enter or leave the retained subset
Terminal-Bench 293%0.556 of 89
SWE-bench Verified93%0.837 of 100
BixBench gold76%0.684 of 17
ScienceAgentBench gold67%0.544 of 12
All task sets90%0.7621 of 218

Most labels are consistent across runs, and the retained subsets change by only a few tasks. We also recompute the leaderboard lift from section 2, the mean gain in model score after removing the tasks a run flags as Major. On Terminal-Bench 2, two of the three runs reproduce the corpus lift closely (1.8 and 1.9 pp, against 1.8 pp). The third gives a smaller but still positive lift of 0.4 pp. We will add these runs to App. D.5 and report the lift as a range across runs.

Regarding your question on which trace the auditor sees: all of the trajectory audits give the auditor every run of the task. That includes both passing and failing runs. Each SWE-bench Verified task has three runs and each Terminal-Bench 2 task has five. To test whether the runs shown change the result, we re-audited the 59 tasks that have both passing and failing runs in three ways: with all runs, with passing runs only, and with failing runs only. The table counts Major tasks among these 59: 35 on Terminal-Bench 2 and 24 on SWE-bench Verified.

SettingTerminal-Bench 2SWE-bench Verified
Major tasks, passing runs only1 of 350 of 24
Major tasks, all runs7 of 352 of 24
Major tasks, failing runs only10 of 356 of 24

With failing runs only, the auditor flags more tasks as Major. With passing runs only it flags almost none because a passing run is evidence that the task can be solved. This is why we default to show every run. An audit with all runs gives the same Major label as the paper’s result on 89% of these Terminal-Bench 2 tasks and 96% of these SWE-bench Verified tasks. We will add this comparison to App. E.2 and state in the main text that trajectory audits read all runs.

AuthorsOpenReview ↗

Revised PDF uploaded

Dear Reviewer ZNh6,

Thank you again for your review. We have uploaded a revised PDF that adds the new human-labeled sample and the task-level precision (Section 4.3), the repeat-run and trace comparisons (App. D.5 and E.2), and the difficulty-matched controls (App. E.4). We would be grateful to hear whether our response addresses your concerns, and we are happy to clarify anything before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.