acceptodds
Under review as a conference paper at ICLR 2027

Benchmarks as Software: Toward Continuous Benchmarks from Terminal-Bench

Abstract

Coding agents are among the most economically consequential deployments of LLMs to date, making reliable evaluation increasingly important. Yet even benchmarks that are carefully validated at release can become stale. As models improve, they can discover valid solutions that existing verifiers were not designed to recognize, while changes to evaluation infrastructure can introduce new sources of error. We call the resulting accumulation of defects benchmark maintenance debt: a growing gap between benchmark scores and the capabilities they intend to measure. We introduce a Continuous-Validation Workflow for agentic benchmarks that combines community reports, task artifacts, agent traces, automated oracle checks, and human-gated repairs. We apply this workflow to Terminal-Bench 2.0, a benchmark that underwent extensive human and LM-assisted review and red-teaming before release. Our audit identifies issues in roughly 30% of the benchmark, and we release the resulting repairs as Terminal-Bench 2.1. These repairs change evaluation results: across agents, accuracy shifts by as much as 12 percentage points and leaderboard positions move by up to three ranks, even as the overall ranking remains highly correlated (Spearman ). We also show the workflow generalizes to tasks in other agentic benchmarks. Beyond accuracy, we evaluate how resource allocation shapes benchmark conclusions. We show that fixed wall-clock and output-token budgets can materially change apparent agent performance: some large harness gaps at the default timeout shrink substantially when agents are given more time. These results show that benchmark maintenance is not only about repairing task artifacts; resource budgets are part of the measurement specification and must be validated alongside the tasks themselves. More broadly, benchmark maintenance debt is not just accumulated implementation bugs: our results show that as models improve, both evaluation assumptions and resource settings may need to be revalidated.

Primary area:
datasets and benchmarks
Keywords:
benchmark, evaluations, agentic systems

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eJjdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper frames post-release defects in executable agent benchmarks as "debt" and proposes a continuous-validation workflow: an evidence-collector agent (task files, prior traces, community reports), an auditor agent that applies a fixed four-label rubric (no defect / instruction-verification mismatch / task drift / environment issue), a verifier agent plus a deterministic oracle check that rebuilds the task and reruns the reference solution, and human-gated repairs.

Applied to Terminal-Bench 2.0 (89 tasks), the workflow consolidates 16 already-public repairs (11 from Z.ai's release, 5 from GitHub issues) with 14 newly identified ones into 28 repaired tasks, released as Terminal-Bench 2.1. Re-evaluating 19 agent-model pairs (13 models across terminus-2, Claude Code, Codex CLI and Gemini CLI) on both versions gives Spearman rho = 0.958, median per-agent shift 5.3%, maximum about 12%, with five verifier repairs accounting for 62.7% of the absolute pass-rate change.

The workflow is evaluated by inter-auditor agreement (kappa = 0.80 between Claude Code and Codex on the primary label), by precision 57.1% / recall 71.4% against the TB2.1 release, and by transfer to 34 Harbor-Index tasks.

Critical strengths

The Terminal-Bench 2.1 artifact and the rank-stability measurement are useful to practitioners. Terminal-Bench 2.0 is reported in model cards from OpenAI, Anthropic, Google, Qwen and GLM, so a consolidated, evidence-backed repair set and a measurement of what it does to 19 agents (rho = 0.958, shifts up to ~12%, rank moves up to 3%; Sec. 6.1, Table 7) has immediate practical value. The rank-stability protocol after Recht et al. is appropriate, and the 62.7% attribution to five verifier fixes is consistent with Table 7. However, the paper on its own is a maintenance contribution rather than a research one.

The resource-budget analysis is the most informative result. Fig. 3b/Fig. 7 show the timeouts dominate failures for weak pairs; harness-dependent timeout rates from 18% to 68% and the within-vendor swap in Fig. 6 support the conclusion that budgets are part of the measurement specification. This seems useful for benchmark maintainers, and is evalauted on six pairs and two OpenAI models.

Critical weaknesses

Thecontribution is limited; the paper is largely a maintenance report on one benchmark. The "continuous validation workflow" is the existing ABC rubric of Zhu et al. applied by LLM auditors, plus a reference-solution rerun and a GitHub Actions trigger; none of these pieces are new and no new auditing contribution is introduced. The artifact is mostly consolidation: 16 of the 28 repairs were already public through Z.ai's verified release and GitHub issues, and the reference annotation for Z.ai notes that a 5.5% shift from environment fixes alone had already been reported. The main result of contribution three is that rank order is preserved, i.e. that repairs change little.

For the different LLM auditors, agreement only shows the auditors are consistent. Two auditors can agree on a wrong label, so this says nothing about whether the flags are correct. The computed precision 57% means the workflow flagged 35 tasks and 20 of them were real, so 15 were false alarms. The computed recall 71% means it found 20 of the 28 real defects. But 14 of those 28 defects were discovered by the workflow in the first place, so the answer key was partly written by the thing being graded which inflates recall.

Drift always breaks the reference solution, so a plain script that reruns the reference solution catches every drift defect automatically, with no LLM involved. That means the LLM-auditor part of the workflow, may only account for a handful of the discoveries. These two are never detangled.

The paper says repairs move individual agents but it never says how many runs per agent per task went into those pass rates. There are no confidence intervals per agent, only one for the overall rank correlation. Before analysis, 27% of all runs were thrown out as infrastructure failures, and the paper never checks whether that discard rate differs by harness or model.

Smaller stuff

  • The 1.5x oracle-slack convention raises time/memory/CPU on a dozen tasks, which lowers difficulty; Table 7 therefore mixes "verifier now accepts valid solutions" with "task is now easier". Splitting Table 7 by repair component would clarify.
  • The 32x budget sweep covers six pairs, five of them terminus-2, and the harness-gap claim (Fig. 6) is made on two OpenAI models although Table 6 shows Claude Code/terminus-2 and Gemini CLI/terminus-2 pairs were run.
  • Fig. 3a percentages exceed 100% because tasks carry multiple labels, which the caption should state.
Questions for authors

For each of the 28 repaired tasks: was it (i) already public, (ii) detectable by oracle alone, or (iii) found only by the LLM auditor? What would an oracle-only or issues-only baseline score on the precision/recall metric?

What does Terminal-Bench 2.1 change relative to Z.ai's Terminal-Bench 2.0 Verified beyond the 14 newly identified tasks, and how do the per-agent shifts compare with the 5.5% shift Z.ai reported?

How many trials per agent-task pair were run for Terminal-Bench 2.0 and Terminal-Bench 2.1 evaluation data? Were both versions run in the same time window and harness versions?

Please report per-cell discard rates alongside Fig. 7.

Does the Fig. 6 harness-gap result hold at 32x for Claude Code vs terminus-2 (Opus/Sonnet 4.6) and Gemini CLI vs terminus-2 (Pro 3.1 / Flash 3)?

5 comments

We thank the reviewer for finding our work useful and informative. Below we answer the concerns and questions, with results we add to the revised paper.

W1. Contribution is limited.

(the reviewer mentions: "The artifact is mostly consolidation:")

We would like to clarify our contribution, which has three parts: (1) a reproducible process for auditing agent benchmarks; (2) the release of Terminal-Bench 2.1; and (3) an analysis of how the broken tasks affected TB2.0’s measurements (scores, rankings, and harness and time-budget comparisons) and what this implies for TB2.1.

(1) A reproducible audit design and process for continuous benchmarks: This workflow has been adopted and built upon by at least two open source frameworks:

TB2.0 Merrill et al., 2026, App. B.5), scored and passed the benchmark against the ABC checklist before release. Our audit builds on ABC, runs continuously rather than once, and adds checks ABC does not cover:

  • Hidden constraints: ABC checks that answer formats are stated and tasks are solvable. Our work also checks that the verifier accepts other valid solution the instruction allows via evidence aggregation.
  • Evidence aggregation: the workflow pools recorded run of the task across agents and models with community issues and pull requests. This surfaces both kinds of grading error: valid solutions the verifier rejected (false negatives) and wrong ones it accepted (false positives).
  • Resource limits: ABC doesn't emphasize resources. TB2.0 contributors set budgets from human resource use. The workflow set a feasibility floor guideline which reduces the cost of ablations and can change as models evolve in performance.

(2) Terminal-Bench 2.1, a repaired release in which half had not been raised by any public report (Section 5), and which changes model rankings (Section 6).

  • TB 2.0 verified's release covers half of the broken tasks but unfortunately not a reproducible or generalizable process.
  • Where both releases repaired the same task, they differ in approach. Verified edits the instruction to state exactly what the test checks, which in many cases reduces task difficulty (e.g., by hinting that an agent should use a specific algorithm or package). TB2.1 more often edits the test to accept all correct solutions. In other words, stating the grader’s expectation tells the agent part of the solution, while widening what the grader accepts keeps the task as hard as its contributor intended.
  • Verified adapts the task images to specific harness needs, including a note on five tasks telling the agent not to use the Read tool on images or PDFs. TB2.1 adds no harness-specific changes, so every agent is evaluated on the same task.

(3) Measurements of what the repairs change and its implications: across 19 agents run on both versions (Section 6). By defect type and repaired component, and per agent, where 8 of 20 agents shift by up to +12.0 pp, and rankings move by up to three positions (Section 6.1). Evaluating 19 agents lets us separate the effect of the repairs from the effect of any single agent or harness.

For these reasons we disagree would politely push back on the assertion that this work is "mostly consolidation"

AuthorsOpenReview ↗

Response p2

W2. Auditor Agreement versus correctness. (the reviewer mentioned: "Two auditors can agree on a wrong label … the answer key was partly written by the thing being graded")

We agree that agreement alone does not show correctness, which is why Section 4 scores the workflow against human labels. A human reviewer checked all 89 tasks, flagged or not. Furthermore, every repair proposal was independently reviewed: the review was fully open, and each proposed repair was discussed on GitHub by 10 participants, including community contributors outside our team. Open contributors had every opportunity to surface any issues. In the table above we report the results on the entire benchmark and for the held out set.

W3. Runs and per-agent statistics.

  • Unless otherwise stated we followed the setup of Terminal Bench 2.0 and have added these details to the appendix. In the paper we report independent runs, with error bars are 1 standard error of task-mean accuracy. For task adjudication experiments we use 3 trials per task for every agent–model pair.
  • Per-agent uncertainty: we add the leaderboard table to the appendix; it reports each agent's accuracy on TB2.0 and TB2.1 with cross-task standard error.
  • Infrastructure errors: every result uses the same whitelist filter: a trial is kept only if it ends in a pass, a wrong answer, or a timeout; every other exception is dropped and the trial rerun. The 27% in Appendix C.2 is the share of upstream trials this filter removed. We report it because it is a real cost of running agent benchmarks.

Smaller points

  • Split Table 7 results per component. We agree and add the following table:
Repair touchedTasksShare of total pass-rate changeMean change
Instruction1157%+24.2 pp
Verifier (Table 3 Tests)1150%+20.4 pp
Resources (time / memory / CPUs)932%+16.0 pp
  • The 32x budget sweep covers six pairs: we add Claude Code and Gemini CLI (see below). For Gemini, the two harnesses are already within 2 pp at 1× so there is no harness gap for a larger budget to close, and we do not extend them to 32×. For Claude Sonnet 4.6, Claude Code leads Terminus-2 by 7.1 pp at 1× and by 2.3 pp at 32×: as with the OpenAI models, more time closes most of the gap.

Accuracy (%) on TB2.1, native harness vs Terminus-2, at 1× and 32× time budget

ModelHarness1×32×
GPT 5.4Codex / Terminus-274.2 / 53.974.9 / 75.3
GPT 5.4 miniCodex / Terminus-267.0 / 37.168.2 / 62.9
Claude Sonnet 4.6Claude Code / Terminus-259.2 / 52.166.7 / 64.4
Gemini Pro 3.1Gemini CLI / Terminus-271.2 / 70.0–
Gemini Flash 3Gemini CLI / Terminus-255.4 / 53.6–
  • Figure 3a: Thank you, the caption will state that tasks can carry several labels, so percentages sum to more than 100%.
AuthorsOpenReview ↗

Response p3

Questions

  • For each of the 28 repaired tasks: was it already public, detectable by oracle alone, or found only by the LLM auditor? What would an oracle-only or issues-only baseline score?

The Section 4 results reported on all TB 2.0 tasks. Below we break down the evaluation under a subset of 41 tasks not used to tune the auditor. Its F1 on those 41 is 0.62 vs 0.63 in Section 4 as shown below. The precision/recall are also consistent.

Table 1. Held-out test tasks from Terminal Bench 2.0: 41 tasks, 12 broken, 29 clean.

DetectorPrecision (TP ÷ all flagged)Recall (TP ÷ all broken)F1
Workflow61%64%0.62
Z.ai TB2.0 Verified + community PRs50%58%0.54
Z.ai TB2.0 Verified83%42%0.56
Community PRs before Z.ai release22%17%0.19
All-positive baseline (flag every task)29%100%0.45
Oracle rerun (reference solution fails)57%33%0.42

Table 2. All tasks: 89 tasks, 28 broken, 61 clean.

DetectorPrecision (TP ÷ all flagged)Recall (TP ÷ all broken)F1
Workflow57%71%0.63
Z.ai TB2.0 Verified + community PRs56%54%0.55
Z.ai TB2.0 Verified91%36%0.51
All-positive baseline (flag every task)31%100%0.48
Oracle rerun (reference solution fails)62%29%0.39
  • What does Terminal-Bench 2.1 change relative to Z.ai's Terminal-Bench 2.0 Verified beyond the 14 newly identified tasks?

As shown in W1, sources differ both in how many defects they find (table above) and in how they repair them (Verified edits instructions where TB2.1 fixes the test), which supports our argument that benchmark repair is not yet a standardized or automatic process.

  • How many trials per agent-task pair were run? Were both versions run in the same time window and harness versions?

Yes, we describe this in W3. Yes, we run both version within a two week timeline and using the same harness version.

  • Please report per-cell discard rates alongside Fig. 7. Discards follow a fixed whitelist of exceptions treated as infrastructure failures, applied identically to TB2.0 and TB2.1. We break down the issues encountered while running the benchmark below:
Reviewer_WpZJOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that agentic benchmarks are software artifacts that accumulate maintenance debt after release. Therefore, continuous validation and versioned repair are required rather than one-time validation.

The authors propose an auditing workflow that combines community reports, task artifacts, agent traces, automated oracle checks, and human review. Applied to Terminal-Bench 2.0, it newly identified 14 defective tasks. Combined with prior audits, 28 of the 89 tasks are revised. The results are released as Terminal-Bench 2.1.

Across 19 agents evaluated on both TB 2.0 and TB 2.1, rankings remain highly correlated (Spearman ρ= 0.958). However, individual accuracy shifts by up to 12 pp and ranks move by up to three positions. The paper also shows that resource allocation affects conclusions. As an external-validity check on 34 Harbor-Index tasks, 12 of 14 flagged tasks are confirmed as defective.

The main contributions are the continuous validation workflow, the TB 2.1 release, and the evidence that benchmark repairs and resource settings affect evaluation.

Critical strengths
  1. The paper makes a useful practical contribution to benchmark maintenance. The auditing workflow identifies 14 new defective tasks. Together with community findings, this leads to the TB 2.1 release, which revises 28 of the 89 TB 2.0 tasks.

  2. The experiments provide useful evidence about how evaluation settings affect results. Scores change by up to about 12 pp after the repairs. Giving agents more time also reduces some large performance gaps between harnesses. These findings show why benchmark versions and resource settings matter when interpreting agent performance.

Critical weaknesses
  1. Several important experimental details are unclear. The paper does not clearly state how many trials were run for each task, what value of k was used for coverage, or how the budget curves and error bars were calculated. Appendix C.2 excludes about 27% of upstream trials from the failure analysis because of infrastructure problems, but it is unclear whether the same exclusions were applied to the main accuracy results.

  2. The evidence that the repairs improve grading accuracy is limited. The paper describes reference-solution checks and human review, but does not report how the repaired graders perform on a set of independently checked correct and incorrect solutions. For example, the OmniMath grading prompt in Harbor-Index is revised, but Appendix C.3 does not report how accurately the updated judge grades answers.

Questions for authors
  1. How did you independently check for problems in tasks that the workflow did not flag?

  2. Were the fixed tests also checked using other valid solutions and deliberately incorrect outputs?

  3. Were the roughly 27% of infrastructure-failure trials excluded in Appendix C.2 also excluded from any main accuracy comparisons? Were these trials rerun?

  4. Can authors clarify the attribution of the largest score gains to verifier repairs? How did the instruction and resource changes in Table 3 contribute to the improvements in Table 7?

3 comments

We thank the reviewer for their comments and highlighting the practical impact of our work. Below we answer the two concerns (experimental details and grading accuracy of the repairs) and the four questions, with results we add to the revised paper.

W1. Experimental details. (the reviewer mentioned: "how many trials were run for each task, what value of k was used for coverage, or how the budget curves and error bars were calculated")

  • Unless otherwise stated we followed the setup of Terminal Bench 2.0 and have added these details to the appendix. In the paper we report independent runs, with error bars are 1 standard error of task-mean accuracy. For task adjudication experiments we use 3 trials per task for every agent–model pair.
  • Coverage (Figure 5) is the fraction of tasks solved by at least one trial at that budget.
  • Infrastructure errors. Yes. Every result uses the same whitelist filter: a trial is kept only if it ends in a pass, a wrong answer, or a timeout; every other exception is dropped and the trial rerun. The 27% in Appendix C.2 is the share of upstream trials this filter removed. We report it because it is a real cost of running agent benchmarks.

W2. Evidence that the repairs improve grading accuracy. (the reviewer mentioned "the paper .. does not report how the repaired graders perform on a set of independently checked correct and incorrect solutions")

Every repair proposal was independently reviewed before release: the review was fully open, and each proposed repair was discussed on GitHub by 10 participants, including community contributors outside our team. To measure grading on individual solutions, we compare TB2.0 and TB2.1 pass rates on the tasks whose repair changed the verifier (39% of repaired tasks; Table 3).

  • All verifier repairs: 55% raise pass rates, where the old verifier had rejected valid solutions, and 36% lower them, consistent with removing credit for invalid solutions (Table 7). The largest effects come from fixing verifiers that rejected valid solutions: on three tasks, the pass rate rose from 5.0% to 66.7–80.0%.
  • Held-out verifier repairs, on tasks never used to tune the auditor: the workflow flagged all of them, and their mean pass rate rose from 33.9% to 59.0% (Table 7).

We will consolidate these results next to Table 7 in App. C.1.

Questions

  • How did you independently check for problems in tasks that the workflow did not flag? A human reviewer checked all 89 tasks, flagged or not, and every task was reviewed publicly where independent contributors could openly voice any concerns.
  • Were the fixed tests also checked using other valid solutions and deliberately incorrect outputs?

Other valid solutions: yes, by using data from agents’ own solutions; repaired verifiers that had rejected valid solutions now accept them (Table 7). Deliberately incorrect outputs: by using no-op tests. We did not delivery construct incorrect outputs, the tasks used to tune the auditor already had incorrect outputs.

  • Were the roughly 27% of infrastructure-failure trials excluded in Appendix C.2 also excluded from any main accuracy comparisons? Were these trials rerun?

Yes, see response above.

  • How did the instruction and resource changes in Table 3 contribute to the improvements in Table 7?

We break the changes per section where the repair touched before. The results echo that instruction verification changes drive most of the performance changes. We will add these results to the camera ready version of the paper.

Repair touchedTasksShare of total pass-rate changeMean change
Instruction1157%+24.2 pp
Verifier (Table 3 Tests)1150%+20.4 pp
Resources (time / memory / CPUs)932%+16.0 pp

We hope the clarifications and additional details address the reviewer's concerns. Should any important concern remain, we would be grateful if the reviewer could bring it to our attention and we would be happy to address it.

Area_Chair_4yQNOpenReview ↗

Reminder from AC

Dear Reviewer WpZJ,

As the ICLR discussion period is closing, please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.

Thank you for your time, your thoughtful reviews, and your service to the ICLR community.

Best Regards, AC

After checking the rebuttal and other reviewers' comments, I would like to maintain my current rating.

Reviewer_G57SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces “benchmark maintenance debt”: executable agent benchmarks become less valid through instruction–verifier mismatches, external dependency drift, and resource configuration problems. It proposes a continuous-validation workflow combining community reports, execution traces, agent auditing, reference-solution checks, and human approval. Applying it repairs 28 of 89 Terminal-Bench 2.0 tasks, producing Terminal-Bench 2.1. Fresh evaluation of 19 agents shows accuracy changes of up to approximately 12 percentage points and rank movements of up to three positions, while overall rankings remain highly correlated (Spearman ρ = 0.958). Budget experiments and an audit of 34 Harbor-Index tasks provide additional evidence of resource sensitivity and workflow transfer.

Critical strengths
  1. A repair artifact with demonstrated practical impact. The work preserves and repairs tasks rather than relying solely on filtering. Across 19 agents, the median score change is 5.3 percentage points despite broadly stable rankings (§6.1, §8). This establishes that maintenance materially affects performance reporting and makes TB2.1 a useful research artifact.
  2. Direct evidence that resource budgets affect comparisons. In Figure 6, the approximately 20-point default-budget gap between Codex and Terminus-2 for GPT 5.4 largely disappears after extending timeouts. This provides compelling evidence that resource allocation belongs in the evaluation specification.
Critical weaknesses
  1. Independent validation of workflow correctness remains limited. The reported 57.1% precision and 71.4% recall use the final repair set as reference, although the workflow itself first surfaced 14 of those 28 repairs; this dependence is disclosed. The Harbor-Index study supports transfer but misses important task-specific and shared-grader defects. Systematic regression evidence covering diverse valid and invalid solutions is also insufficiently reported. Consequently, the evidence supports maintainer-assisted triage more strongly than comprehensive validation.
  2. Important experimental details are missing, limiting reproducibility and interpretation. The paper does not clearly specify per-task repeat counts, pass@k settings, or whether Figure 5 uses independent budget runs or retrospective truncation. Figure 6’s error bars are undefined. Appendix C.2 excludes approximately 27% of upstream trials from its failure decomposition, while the treatment of infrastructure errors in the main experiments remains unclear. This does not establish bias, but prevents a confident assessment of score changes and local rank movements.
  3. The workflow largely integrates existing auditing rubrics, reference-solution checks, and human review, which looks solid to me. However, empirical comparisons in against other auditing procedures or work are missing, making its methodological contribution difficult to assess.
Questions for authors

Which auditor configuration produces the reported precision and recall? Were both flagged and unflagged tasks independently adjudicated? For verifier repairs, what regression results are available for valid solutions, invalid solutions, and alternative valid approaches?

3 comments

We thank the reviewer for highlighting Terminal Bench 2.1 as an artifact with demonstrated practical impact.

Below we address the reviewer's three concerns (validation, experimental details, and comparison with other auditing procedures) and three questions, with additional experiments and results that we include in the revised paper.

W1. Independent validation of the workflow is limited.

The correctness of the benchmark auditing tool is validated in the following ways:

  • creating ground truth labels: a human reviewer examined all terminal bench tasks, including every unflagged task. Every repair was adjudicated openly in github, with participation from 10 independent community authors.
  • Validating on a held-out set. The Section 4 prompts were calibrated on a task pilot and reported on all 89 tasks. Below we break down the evaluation under a subset of 41 tasks not used to tune the auditor. Its F1 on those 41 is 0.62 vs 0.63 in Section 4 as shown below.

Table 1. Held-out test tasks from Terminal Bench 2.0: 41 tasks, 12 broken, 29 clean.

DetectorPrecision (TP ÷ all flagged)Recall (TP ÷ all broken)F1
Workflow61%64%0.62
Z.ai TB2.0 Verified + community PRs50%58%0.54
Z.ai TB2.0 Verified83%42%0.56
Community PRs before Z.ai release22%17%0.19
All-positive baseline (flag every task)29%100%0.45
Oracle rerun (reference solution fails)57%33%0.42

Table 2. All tasks: 89 tasks, 28 broken, 61 clean.

DetectorPrecision (TP ÷ all flagged)Recall (TP ÷ all broken)F1
Workflow57%71%0.63
Z.ai TB2.0 Verified + community PRs56%54%0.55
Z.ai TB2.0 Verified91%36%0.51
All-positive baseline (flag every task)31%100%0.48
Oracle rerun (reference solution fails)62%29%0.39

These results show the auditing tool has better F1 scores than public sources combined. We also note that other community contributors did not disclose reproducible or generalizable processes to identify issues.

  • Transfer to other benchmarks. We ran the workflow on 34 Harbor-Index tasks drawn from 7 source benchmarks; a human reviewer confirmed 12 of its 14 flags (86% precision; Section 7, Table 8), showing that the workflow finds real defects beyond Terminal-Bench. We agree with the reviewer that It is designed for maintainer-in-the-loop triage (as noted in Sections 1, 4 and 7). We will state this more prominently.
  • New defects. Our audit newly identified 14 defects that public reports had not raised. Our results show how public reporting alone would have missed half of the 28 repairs. This is the workflow’s role: it finds defects public reporting may miss, and maintainers decide what to fix.
AuthorsOpenReview ↗

Response p2

W2. Experimental details.

  • Unless otherwise stated we followed the setup of Terminal Bench 2.0 and have added these details to the appendix. In the paper we report independent runs, with error bars are 1 standard error of task-mean accuracy. For task adjudication experiments we use 3 trials per task for every agent–model pair.
  • Coverage (Figure 5) is the fraction of tasks solved by at least one trial at that budget.
  • Infrastructure errors. Yes. Every result uses the same whitelist filter: a trial is kept only if it ends in a pass, a wrong answer, or a timeout; every other exception is dropped and the trial rerun. The 27% in Appendix C.2 is the share of upstream trials this filter removed. We report it because it is a real cost of running agent benchmarks.

W3. Comparison with other auditing procedures.

We add comparisons of the workflow with Z.ai’s review (TB2.0 Verified), community proposals and the oracle rerun (tables above). Terminal-Bench 2.0’s release review checked every task with an oracle run, a no-op agent, LLM checks and human review, and its authors scored the benchmark as a whole against the Agentic Benchmark Checklist (Merrill et al., 2026, Sec. 2.3 and App. B.5). All 28 tasks later repaired in TB2.1 went through this review and shipped in TB2.0. Here we list what these checks do not cover:

  • Hidden constraints: ABC checks that answer formats are stated and tasks are solvable. The workflow also checks that the verifier accepts other valid solution the instruction allows via evidence aggregation.
  • Evidence aggregation: the workflow pools every recorded run of the task across agents and models with community issues and pull requests. This surfaces both kinds of grading error: valid solutions the verifier rejected (false negatives) and wrong ones it accepted (false positives).
  • Resource limits: ABC doesn't emphasize resources. TB2.0 contributors set budgets from human resource use. The workflow checks every budget against a feasibility floor which changes as models evolve in performance.

Questions

  • Which auditor configuration produces the reported precision and recall?

Claude Code (Opus 4.6) and Codex (GPT-5.4); a task counts as flagged only if both flag it. Appendix A.2 reports agreement between multiple auditor pairs; below we add precision and recall with one agent and with two for these agents.

All 89 tasks (28 broken), the Section 4 run:

ConfigurationPrecisionRecallF1
One agent45%82%0.58
Two agents57%71%0.63

41 held-out tasks (12 broken), new runs:

ConfigurationPrecisionRecallF1
Claude Code54%81%0.64
Codex36%83%0.50
Two agents61%64%0.62
  • Were both flagged and unflagged tasks independently adjudicated? Yes. A human reviewer checked all 89 tasks, flagged or not, and every repair was reviewed publicly where 10 participants engaged.
  • What regression results are available for valid solutions, invalid solutions, and alternative valid approaches?

We compare TB2.0 and TB2.1 on the repairs that changed the verifier (39%; Table 3). Of these, 55% raise pass rates, where the old verifier had rejected valid solutions, and 36% lower them, consistent with removing credit for invalid solutions (Table 7). The repairs with the largest effect fixed verifiers that rejected valid solutions: on three of the six, the pass rate rose from 5.0% to 66.7–80.0%. Every repaired task also passes its reference solution and rejects a no-op agent. We agree these results belong together and will consolidate them in one paragraph next to Table 7 in App. C.1.

We hope the clarifications and additional details address the reviewer's concerns. Should any important concern remain, we would be grateful if the reviewer could bring it to our attention and we would be happy to address it.

Area_Chair_4yQNOpenReview ↗

Reminder from AC

Dear Reviewer G57S,

As the ICLR discussion period is closing, please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.

Thank you for your time, your thoughtful reviews, and your service to the ICLR community.

Best Regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.