- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper frames post-release defects in executable agent benchmarks as "debt" and proposes a continuous-validation workflow: an evidence-collector agent (task files, prior traces, community reports), an auditor agent that applies a fixed four-label rubric (no defect / instruction-verification mismatch / task drift / environment issue), a verifier agent plus a deterministic oracle check that rebuilds the task and reruns the reference solution, and human-gated repairs.
Applied to Terminal-Bench 2.0 (89 tasks), the workflow consolidates 16 already-public repairs (11 from Z.ai's release, 5 from GitHub issues) with 14 newly identified ones into 28 repaired tasks, released as Terminal-Bench 2.1. Re-evaluating 19 agent-model pairs (13 models across terminus-2, Claude Code, Codex CLI and Gemini CLI) on both versions gives Spearman rho = 0.958, median per-agent shift 5.3%, maximum about 12%, with five verifier repairs accounting for 62.7% of the absolute pass-rate change.
The workflow is evaluated by inter-auditor agreement (kappa = 0.80 between Claude Code and Codex on the primary label), by precision 57.1% / recall 71.4% against the TB2.1 release, and by transfer to 34 Harbor-Index tasks.
Critical strengths
The Terminal-Bench 2.1 artifact and the rank-stability measurement are useful to practitioners. Terminal-Bench 2.0 is reported in model cards from OpenAI, Anthropic, Google, Qwen and GLM, so a consolidated, evidence-backed repair set and a measurement of what it does to 19 agents (rho = 0.958, shifts up to ~12%, rank moves up to 3%; Sec. 6.1, Table 7) has immediate practical value. The rank-stability protocol after Recht et al. is appropriate, and the 62.7% attribution to five verifier fixes is consistent with Table 7. However, the paper on its own is a maintenance contribution rather than a research one.
The resource-budget analysis is the most informative result. Fig. 3b/Fig. 7 show the timeouts dominate failures for weak pairs; harness-dependent timeout rates from 18% to 68% and the within-vendor swap in Fig. 6 support the conclusion that budgets are part of the measurement specification. This seems useful for benchmark maintainers, and is evalauted on six pairs and two OpenAI models.
Critical weaknesses
Thecontribution is limited; the paper is largely a maintenance report on one benchmark. The "continuous validation workflow" is the existing ABC rubric of Zhu et al. applied by LLM auditors, plus a reference-solution rerun and a GitHub Actions trigger; none of these pieces are new and no new auditing contribution is introduced. The artifact is mostly consolidation: 16 of the 28 repairs were already public through Z.ai's verified release and GitHub issues, and the reference annotation for Z.ai notes that a 5.5% shift from environment fixes alone had already been reported. The main result of contribution three is that rank order is preserved, i.e. that repairs change little.
For the different LLM auditors, agreement only shows the auditors are consistent. Two auditors can agree on a wrong label, so this says nothing about whether the flags are correct. The computed precision 57% means the workflow flagged 35 tasks and 20 of them were real, so 15 were false alarms. The computed recall 71% means it found 20 of the 28 real defects. But 14 of those 28 defects were discovered by the workflow in the first place, so the answer key was partly written by the thing being graded which inflates recall.
Drift always breaks the reference solution, so a plain script that reruns the reference solution catches every drift defect automatically, with no LLM involved. That means the LLM-auditor part of the workflow, may only account for a handful of the discoveries. These two are never detangled.
The paper says repairs move individual agents but it never says how many runs per agent per task went into those pass rates. There are no confidence intervals per agent, only one for the overall rank correlation. Before analysis, 27% of all runs were thrown out as infrastructure failures, and the paper never checks whether that discard rate differs by harness or model.
Smaller stuff
- The 1.5x oracle-slack convention raises time/memory/CPU on a dozen tasks, which lowers difficulty; Table 7 therefore mixes "verifier now accepts valid solutions" with "task is now easier". Splitting Table 7 by repair component would clarify.
- The 32x budget sweep covers six pairs, five of them terminus-2, and the harness-gap claim (Fig. 6) is made on two OpenAI models although Table 6 shows Claude Code/terminus-2 and Gemini CLI/terminus-2 pairs were run.
- Fig. 3a percentages exceed 100% because tasks carry multiple labels, which the caption should state.
Questions for authors
For each of the 28 repaired tasks: was it (i) already public, (ii) detectable by oracle alone, or (iii) found only by the LLM auditor? What would an oracle-only or issues-only baseline score on the precision/recall metric?
What does Terminal-Bench 2.1 change relative to Z.ai's Terminal-Bench 2.0 Verified beyond the 14 newly identified tasks, and how do the per-agent shifts compare with the 5.5% shift Z.ai reported?
How many trials per agent-task pair were run for Terminal-Bench 2.0 and Terminal-Bench 2.1 evaluation data? Were both versions run in the same time window and harness versions?
Please report per-cell discard rates alongside Fig. 7.
Does the Fig. 6 harness-gap result hold at 32x for Claude Code vs terminus-2 (Opus/Sonnet 4.6) and Gemini CLI vs terminus-2 (Pro 3.1 / Flash 3)?
5 comments
Response p2
W2. Auditor Agreement versus correctness. (the reviewer mentioned: "Two auditors can agree on a wrong label … the answer key was partly written by the thing being graded")
We agree that agreement alone does not show correctness, which is why Section 4 scores the workflow against human labels. A human reviewer checked all 89 tasks, flagged or not. Furthermore, every repair proposal was independently reviewed: the review was fully open, and each proposed repair was discussed on GitHub by 10 participants, including community contributors outside our team. Open contributors had every opportunity to surface any issues. In the table above we report the results on the entire benchmark and for the held out set.
W3. Runs and per-agent statistics.
- Unless otherwise stated we followed the setup of Terminal Bench 2.0 and have added these details to the appendix. In the paper we report independent runs, with error bars are 1 standard error of task-mean accuracy. For task adjudication experiments we use 3 trials per task for every agent–model pair.
- Per-agent uncertainty: we add the leaderboard table to the appendix; it reports each agent's accuracy on TB2.0 and TB2.1 with cross-task standard error.
- Infrastructure errors: every result uses the same whitelist filter: a trial is kept only if it ends in a pass, a wrong answer, or a timeout; every other exception is dropped and the trial rerun. The 27% in Appendix C.2 is the share of upstream trials this filter removed. We report it because it is a real cost of running agent benchmarks.
Smaller points
- Split Table 7 results per component. We agree and add the following table:
| Repair touched | Tasks | Share of total pass-rate change | Mean change |
|---|---|---|---|
| Instruction | 11 | 57% | +24.2 pp |
| Verifier (Table 3 Tests) | 11 | 50% | +20.4 pp |
| Resources (time / memory / CPUs) | 9 | 32% | +16.0 pp |
- The 32x budget sweep covers six pairs: we add Claude Code and Gemini CLI (see below). For Gemini, the two harnesses are already within 2 pp at 1× so there is no harness gap for a larger budget to close, and we do not extend them to 32×. For Claude Sonnet 4.6, Claude Code leads Terminus-2 by 7.1 pp at 1× and by 2.3 pp at 32×: as with the OpenAI models, more time closes most of the gap.
Accuracy (%) on TB2.1, native harness vs Terminus-2, at 1× and 32× time budget
Model Harness 1× 32× GPT 5.4 Codex / Terminus-2 74.2 / 53.9 74.9 / 75.3 GPT 5.4 mini Codex / Terminus-2 67.0 / 37.1 68.2 / 62.9 Claude Sonnet 4.6 Claude Code / Terminus-2 59.2 / 52.1 66.7 / 64.4 Gemini Pro 3.1 Gemini CLI / Terminus-2 71.2 / 70.0 – Gemini Flash 3 Gemini CLI / Terminus-2 55.4 / 53.6 –
- Figure 3a: Thank you, the caption will state that tasks can carry several labels, so percentages sum to more than 100%.
Response p3
Questions
- For each of the 28 repaired tasks: was it already public, detectable by oracle alone, or found only by the LLM auditor? What would an oracle-only or issues-only baseline score?
The Section 4 results reported on all TB 2.0 tasks. Below we break down the evaluation under a subset of 41 tasks not used to tune the auditor. Its F1 on those 41 is 0.62 vs 0.63 in Section 4 as shown below. The precision/recall are also consistent.
Table 1. Held-out test tasks from Terminal Bench 2.0: 41 tasks, 12 broken, 29 clean.
| Detector | Precision (TP ÷ all flagged) | Recall (TP ÷ all broken) | F1 |
|---|---|---|---|
| Workflow | 61% | 64% | 0.62 |
| Z.ai TB2.0 Verified + community PRs | 50% | 58% | 0.54 |
| Z.ai TB2.0 Verified | 83% | 42% | 0.56 |
| Community PRs before Z.ai release | 22% | 17% | 0.19 |
| All-positive baseline (flag every task) | 29% | 100% | 0.45 |
| Oracle rerun (reference solution fails) | 57% | 33% | 0.42 |
Table 2. All tasks: 89 tasks, 28 broken, 61 clean.
| Detector | Precision (TP ÷ all flagged) | Recall (TP ÷ all broken) | F1 |
|---|---|---|---|
| Workflow | 57% | 71% | 0.63 |
| Z.ai TB2.0 Verified + community PRs | 56% | 54% | 0.55 |
| Z.ai TB2.0 Verified | 91% | 36% | 0.51 |
| All-positive baseline (flag every task) | 31% | 100% | 0.48 |
| Oracle rerun (reference solution fails) | 62% | 29% | 0.39 |
- What does Terminal-Bench 2.1 change relative to Z.ai's Terminal-Bench 2.0 Verified beyond the 14 newly identified tasks?
As shown in W1, sources differ both in how many defects they find (table above) and in how they repair them (Verified edits instructions where TB2.1 fixes the test), which supports our argument that benchmark repair is not yet a standardized or automatic process.
- How many trials per agent-task pair were run? Were both versions run in the same time window and harness versions?
Yes, we describe this in W3. Yes, we run both version within a two week timeline and using the same harness version.
- Please report per-cell discard rates alongside Fig. 7. Discards follow a fixed whitelist of exceptions treated as infrastructure failures, applied identically to TB2.0 and TB2.1. We break down the issues encountered while running the benchmark below:
Response
We thank the reviewer for finding our work useful and informative. Below we answer the concerns and questions, with results we add to the revised paper.
W1. Contribution is limited.
(the reviewer mentions: "The artifact is mostly consolidation:")
We would like to clarify our contribution, which has three parts: (1) a reproducible process for auditing agent benchmarks; (2) the release of Terminal-Bench 2.1; and (3) an analysis of how the broken tasks affected TB2.0’s measurements (scores, rankings, and harness and time-budget comparisons) and what this implies for TB2.1.
(1) A reproducible audit design and process for continuous benchmarks: This workflow has been adopted and built upon by at least two open source frameworks:
TB2.0 Merrill et al., 2026, App. B.5), scored and passed the benchmark against the ABC checklist before release. Our audit builds on ABC, runs continuously rather than once, and adds checks ABC does not cover:
(2) Terminal-Bench 2.1, a repaired release in which half had not been raised by any public report (Section 5), and which changes model rankings (Section 6).
(3) Measurements of what the repairs change and its implications: across 19 agents run on both versions (Section 6). By defect type and repaired component, and per agent, where 8 of 20 agents shift by up to +12.0 pp, and rankings move by up to three positions (Section 6.1). Evaluating 19 agents lets us separate the effect of the repairs from the effect of any single agent or harness.
For these reasons we disagree would politely push back on the assertion that this work is "mostly consolidation"