- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Agents-Eval is an open-source platform that runs agent benchmarks, harnesses, and sandboxed environments through one control plane, which handles scheduling, execution, native scoring, fault handling, and cleanup. It also proposes a capability translation matrix, in which an LLM (GPT-5.6-sol) rates each task from 0 to 5 on six capability dimensions. The experiments profile six benchmarks, compare harnesses across demand levels on SWE-bench Verified, Terminal-Bench 2.1, and ALE-CLI, and categorize recorded execution failures.
Critical strengths
-
The problem is real. Agent results are hard to compare because harnesses, environments, and failure handling differ, and a maintained open-source platform that keeps each benchmark's native scoring is useful.
-
Treating evaluation failures separately from task failures, and making fault handling part of the protocol (Table 2, Fig. 7), is good practice that many papers ignore.
Critical weaknesses
-
It is unclear what the platform adds over HAL, AgentCompass, Harbor (which it integrates as a runner), and Inspect. More importantly, the central reproducibility claim is never measured: there is no run-to-run variance, no reproduction of official scores, and no before/after comparison of the fault handling. Failure handling even differs across benchmarks (failures are excluded on ALE-CLI and ResearchClawBench but counted on SWE-bench and Terminal-Bench), which works against the goal of consistent conditions.
-
The capability analysis rests on labels from a single LLM that sees only the task description, with no agreement check against human experts and no stability test. Fig. 4 pools four benchmarks with different metrics, adds an artificial zero-success point at level 20, and extrapolates past the observed levels 1–5, so demand level is confounded with benchmark identity.
Questions for authors
- Can you show a measurable advantage over HAL, AgentCompass, or plain Harbor, such as fewer failures, lower run-to-run variance, or closer reproduction of official scores? Absolute counts behind Fig. 7 would help here.
- Appendix A.3 excludes execution failures on ALE-CLI (except timeouts) and ResearchClawBench but counts them as failures on SWE-bench Verified and Terminal-Bench. Do harness rankings change under a single rule?
- How well do the LLM capability labels agree with expert labels on a random sample, and does the Fig. 4 trend hold within individual benchmarks?
4 comments
Response to W1 & Q2: Failure Accounting and Ranking Sensitivity
Thank you for the careful examination. Failure handling follows each benchmark’s official protocol: for example, ResearchClawBench excludes failed evaluations from score aggregation, while some other benchmarks assign zero. We apologize for this oversight of ALE-CLI. Nevertheless, we acknowledge that excluding failures can produce different evaluation subsets across harnesses and bias comparisons.
Across the nine ALE-CLI evaluation runs, an average of 3.33 task executions per run were excluded from scoring. Holding the latest labels fixed, the leading harness remains unchanged in 44 of 48 nonempty capability–demand groups. All DeepSeek group rankings remain unchanged. We will update Figure 5 and its interpretation, and specify the task pools, denominators, and failure policies in Appendix A.3.
Response to W2 & Q3: Annotation Reliability and Within-Benchmark Trends
Thank you for this constructive suggestion. We expanded annotation to three independent LLM judges, and expanded the expert panel to three. Across 6,246 task–dimension pairs, 76.14% received identical ratings from all three judges, and 95.95% had a maximum rating difference of one level. Our acceptance rule requires at least two identical ratings and a maximum difference of two levels; 97.58% satisfy this criterion. The remaining 2.42% were resolved through expert adjudication.
Removing the artificial anchor preserves the overall decreasing trends with the latest annotations. All six dimensions retain negative fitted slopes for both models. EISS, RLPE, and SRV become flatter, while ICA, TC, and CIE change little.
The overall decreasing trend is also supported in some within-benchmark analyses, particularly TC and CIE for Qwen on Terminal-Bench. Other results remain inconclusive or not estimable. This uncertainty can be partly attributed to concentrated capability-demand levels within individual benchmarks, together with limited sample sizes or sparse outcomes in some subsets. For example, TC and CIE cover only levels 2–3 in SWE-bench subsets, restricting the variation available for trend estimation. We will add the benchmark-specific results to the appendix and clarify these limitations in the capability analysis.
We thank the reviewer again for the constructive feedback. We hope our responses and additional evidence address your concerns, and would appreciate reconsideration of your assessment and score if these concerns have been satisfactorily resolved. We will incorporate all additional results, methodological details, and clarifications into the revised manuscript.
Thanks for the new results. The fault-handling experiments show fewer execution failures, and the ALE-CLI rerun leaves the top harness unchanged in 44 of 48 groups. I’m keeping my rating at 2. The experts only reviewed labels the models disagreed on, so I’m still unsure how well the labels hold up on a random sample.
Thank you for the follow-up. We are glad the fault-handling experiments and the ALE-CLI rerun helped address your concerns about execution failures and harness rankings. To address your remaining concern, we randomly sample task–dimension labels on which all three LLM judges agree, balancing the sample across six benchmarks and six capability dimensions. For 81.4% of the evaluable labels, the LLM rating was within one level of the expert reference. In particular, agreement within one level reach over 90% for ICA and 100% for both SRV and RLPE. Agreement was lower for TC and CIE. One possible explanation is that, during a brief annotation session, experts may find it difficult to anticipate which tools an agent will use or how difficult the implementation will be. We expect agreement to increase if experts carry out the tasks themselves. We see implementation planning as a strength of LLMs, and the experts generally agree with the LLMs’ proposed level after reviewing them. We hope these results resolve your remaining concern.
We thank the reviewer for recognizing the practical value of a maintained evaluation platform, preserving benchmark-native scoring, and distinguishing evaluation failures from task failures.
Response to W1 & Q1: Platform Contribution and Measured Reliability Gains
Thank you for this careful assessment. Relative to frameworks such as HAL, AgentCompass, Harbor, and Inspect, our contribution centers on providing shared, fully configured and prevalidated evaluation environments, together with concrete reference results under documented conditions. Harbor is integrated as an execution component. This becomes particularly valuable in complex environments such as ALE-CLI [1] and Toolathlon [2]: for observational evaluation, researchers can launch runs through the interface without repeating environment configuration, diagnosis, and alignment with reference performance. We will clarify this positioning in the introduction and related work.
We quantify reliability through both a fault-handling ablation and a direct framework comparison. Enabling our mitigations reduces failure rates by 4.13, 7.51, and 19.37 percentage points on SWE-bench Verified, Terminal-Bench 2.1, and ALE-CLI, respectively, weighting the nine model–harness configurations within each benchmark equally.
We also compare Agents-Eval, AgentCompass [3], and Harbor [4] under matched APIs, computing resources, environment configurations, and evaluation settings. The table reports Terminal-Bench 2.1 results with Claude Code with score / failure rate . Officially reported scores are provided separately as references.
Agents-Eval achieves scores that are among the closest to the officially reported results and lower or equal failure rates across these configurations. We restrict this comparison to benchmarks supported by all the evaluated framework versions. Benchmarks such as ALE-CLI are not compared because neither framework supports them, and porting them would require substantial engineering beyond the scope of this comparison.
[1] Sun, Y., Han, X., Zhang, W., Pang, Y., Wang, T., Cao, Y., Huang, Y., Duroiu, C., Zhang, H., Lin, J., Zhang, W., Zeng, T., Yan, Y., Liu, B., Wen, H., Xu, M., Liu, X., Chen, Z., Shi, W., . . . Jiang, Z. 2026. Agents’ last exam. arXiv.
[2] Li, J., Zhao, W., Zhao, J., Zeng, W., Wu, H., Wang, X., Ge, R., Cao, Y., Huang, Y., Liu, W., Liu, J., Su, Z., Guo, Y., Zhou, F., Zhang, L., Michelini, J., Wang, X., Yue, X., Zhou, S., . . . He, J. 2025. The Tool Decathlon: Benchmarking language agents for diverse, realistic, and long-horizon task execution. arXiv.
[3] Chen, K., Ding, Z., Ge, J., Jiang, S., Li, M., Li, Q., Li, Z., Li, Z., Liang, T., Liu, S., Ma, Z., Shang, Z., Tian, W., Wang, Z., Wu, L., Wu, Z., Xu, J., Yang, B., Yuan, D., . . . Zhu, D. 2026. AgentCompass: A unified evaluation infrastructure for agent capabilities. arXiv.
[4] Harbor Framework Team. 2026. Harbor: A framework for evaluating and optimizing agents and models in container environments.
[5] DeepSeek-AI. n\.d\.. DeepSeek-V4.1-Flash [Model card]. Hugging Face.
[6] Z.ai. n\.d\.. GLM-5.3: Terminal-Bench 2.1 evaluation results [Evaluation record]. Hugging Face. Retrieved November 17, 2026, from the official evaluation record.