acceptodds
Under review as a conference paper at ICLR 2027

Agents-Eval: A Lifecycle Platform for End-to-End Agent Evaluation

Abstract

Language-model agents are increasingly used for complex tasks in software engineering, information seeking, and scientific research. Evaluating these agents is essential for reliable comparison and identifying where further improvement is needed. Yet comparing agents requires aligning benchmark protocols, agent harnesses, and execution environments. Researchers often repeat this benchmark-specific integration and configuration, incurring substantial engineering effort while leaving evaluation conditions inconsistent across studies. Existing evaluations also rely largely on aggregate scores and broad task categories, offering limited insight into which capabilities tasks require and at what demand levels. We introduce Agents-Eval, an open-source platform that addresses these two challenges. First, the platform provides reusable benchmark integrations and a unified evaluation workflow, standardizing execution conditions and lifecycle management while preserving benchmark-native interaction and scoring semantics. This allows researchers to reuse maintained evaluation setups across models and harnesses. Second, it introduces a capability translation matrix that decomposes heterogeneous tasks into six shared capability dimensions and assigns an explicit capability demand level to each. These annotations describe task requirements independently of agent responses and support comparisons across demand levels, models, and harnesses. Our experiments reveal shared capability demands across benchmark domains, harness gains that vary with model choice and task requirements, and recurring environment, service, and verification failures. Agents-Eval combines reusable evaluation workflows with fine-grained capability analysis to guide improvements in models, harnesses, and evaluation infrastructure.

Primary area:
datasets and benchmarks
Keywords:
agent, evaluation, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TbL3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Agents-Eval is an open-source platform that runs agent benchmarks, harnesses, and sandboxed environments through one control plane, which handles scheduling, execution, native scoring, fault handling, and cleanup. It also proposes a capability translation matrix, in which an LLM (GPT-5.6-sol) rates each task from 0 to 5 on six capability dimensions. The experiments profile six benchmarks, compare harnesses across demand levels on SWE-bench Verified, Terminal-Bench 2.1, and ALE-CLI, and categorize recorded execution failures.

Critical strengths
  1. The problem is real. Agent results are hard to compare because harnesses, environments, and failure handling differ, and a maintained open-source platform that keeps each benchmark's native scoring is useful.

  2. Treating evaluation failures separately from task failures, and making fault handling part of the protocol (Table 2, Fig. 7), is good practice that many papers ignore.

Critical weaknesses
  1. It is unclear what the platform adds over HAL, AgentCompass, Harbor (which it integrates as a runner), and Inspect. More importantly, the central reproducibility claim is never measured: there is no run-to-run variance, no reproduction of official scores, and no before/after comparison of the fault handling. Failure handling even differs across benchmarks (failures are excluded on ALE-CLI and ResearchClawBench but counted on SWE-bench and Terminal-Bench), which works against the goal of consistent conditions.

  2. The capability analysis rests on labels from a single LLM that sees only the task description, with no agreement check against human experts and no stability test. Fig. 4 pools four benchmarks with different metrics, adds an artificial zero-success point at level 20, and extrapolates past the observed levels 1–5, so demand level is confounded with benchmark identity.

Questions for authors
  1. Can you show a measurable advantage over HAL, AgentCompass, or plain Harbor, such as fewer failures, lower run-to-run variance, or closer reproduction of official scores? Absolute counts behind Fig. 7 would help here.
  2. Appendix A.3 excludes execution failures on ALE-CLI (except timeouts) and ResearchClawBench but counts them as failures on SWE-bench Verified and Terminal-Bench. Do harness rankings change under a single rule?
  3. How well do the LLM capability labels agree with expert labels on a random sample, and does the Fig. 4 trend hold within individual benchmarks?
4 comments

We thank the reviewer for recognizing the practical value of a maintained evaluation platform, preserving benchmark-native scoring, and distinguishing evaluation failures from task failures.

Response to W1 & Q1: Platform Contribution and Measured Reliability Gains

Thank you for this careful assessment. Relative to frameworks such as HAL, AgentCompass, Harbor, and Inspect, our contribution centers on providing shared, fully configured and prevalidated evaluation environments, together with concrete reference results under documented conditions. Harbor is integrated as an execution component. This becomes particularly valuable in complex environments such as ALE-CLI [1] and Toolathlon [2]: for observational evaluation, researchers can launch runs through the interface without repeating environment configuration, diagnosis, and alignment with reference performance. We will clarify this positioning in the introduction and related work.

We quantify reliability through both a fault-handling ablation and a direct framework comparison. Enabling our mitigations reduces failure rates by 4.13, 7.51, and 19.37 percentage points on SWE-bench Verified, Terminal-Bench 2.1, and ALE-CLI, respectively, weighting the nine model–harness configurations within each benchmark equally.

We also compare Agents-Eval, AgentCompass [3], and Harbor [4] under matched APIs, computing resources, environment configurations, and evaluation settings. The table reports Terminal-Bench 2.1 results with Claude Code with score / failure rate %. Officially reported scores are provided separately as references.

ModelAgents-EvalAgentCompassHarborOfficial reference score
Qwen-3.8-27B70.79 / 7.9068.50 / 18.5065.50 / 16.20—
DeepSeek-V4.1-Flash91.00 / 1.1075.70 / 10.7085.40 / 1.1088.00[5]
GLM 5.386.52 / 2.2068.50 / 15.7067.00 / 15.7088.20[6]

Agents-Eval achieves scores that are among the closest to the officially reported results and lower or equal failure rates across these configurations. We restrict this comparison to benchmarks supported by all the evaluated framework versions. Benchmarks such as ALE-CLI are not compared because neither framework supports them, and porting them would require substantial engineering beyond the scope of this comparison.

[1] Sun, Y., Han, X., Zhang, W., Pang, Y., Wang, T., Cao, Y., Huang, Y., Duroiu, C., Zhang, H., Lin, J., Zhang, W., Zeng, T., Yan, Y., Liu, B., Wen, H., Xu, M., Liu, X., Chen, Z., Shi, W., . . . Jiang, Z. 20262026. Agents’ last exam. arXiv.

[2] Li, J., Zhao, W., Zhao, J., Zeng, W., Wu, H., Wang, X., Ge, R., Cao, Y., Huang, Y., Liu, W., Liu, J., Su, Z., Guo, Y., Zhou, F., Zhang, L., Michelini, J., Wang, X., Yue, X., Zhou, S., . . . He, J. 20252025. The Tool Decathlon: Benchmarking language agents for diverse, realistic, and long-horizon task execution. arXiv.

[3] Chen, K., Ding, Z., Ge, J., Jiang, S., Li, M., Li, Q., Li, Z., Li, Z., Liang, T., Liu, S., Ma, Z., Shang, Z., Tian, W., Wang, Z., Wu, L., Wu, Z., Xu, J., Yang, B., Yuan, D., . . . Zhu, D. 20262026. AgentCompass: A unified evaluation infrastructure for agent capabilities. arXiv.

[4] Harbor Framework Team. 20262026. Harbor: A framework for evaluating and optimizing agents and models in container environments.

[5] DeepSeek-AI. n\.d\.. DeepSeek-V4.1-Flash [Model card]. Hugging Face.

[6] Z.ai. n\.d\.. GLM-5.3: Terminal-Bench 2.1 evaluation results [Evaluation record]. Hugging Face. Retrieved November 17, 2026, from the official evaluation record.

Response to W1 & Q2: Failure Accounting and Ranking Sensitivity

Thank you for the careful examination. Failure handling follows each benchmark’s official protocol: for example, ResearchClawBench excludes failed evaluations from score aggregation, while some other benchmarks assign zero. We apologize for this oversight of ALE-CLI. Nevertheless, we acknowledge that excluding failures can produce different evaluation subsets across harnesses and bias comparisons.

Across the nine ALE-CLI evaluation runs, an average of 3.33 task executions per run were excluded from scoring. Holding the latest labels fixed, the leading harness remains unchanged in 44 of 48 nonempty capability–demand groups. All DeepSeek group rankings remain unchanged. We will update Figure 5 and its interpretation, and specify the task pools, denominators, and failure policies in Appendix A.3.

Response to W2 & Q3: Annotation Reliability and Within-Benchmark Trends

Thank you for this constructive suggestion. We expanded annotation to three independent LLM judgesGPT−61˙−sol,DeepSeek−V41˙−Flash−1017,andKimi−K31˙−ProGPT-6\.1-sol, DeepSeek-V4\.1-Flash-1017, and Kimi-K3\.1-Pro, and expanded the expert panel to three. Across 6,246 task–dimension pairs, 76.14% received identical ratings from all three judges, and 95.95% had a maximum rating difference of one level. Our acceptance rule requires at least two identical ratings and a maximum difference of two levels; 97.58% satisfy this criterion. The remaining 2.42% were resolved through expert adjudication.

Removing the artificial 20,020, 0 anchor preserves the overall decreasing trends with the latest annotations. All six dimensions retain negative fitted slopes for both models. EISS, RLPE, and SRV become flatter, while ICA, TC, and CIE change little.

The overall decreasing trend is also supported in some within-benchmark analyses, particularly TC and CIE for Qwen on Terminal-Bench. Other results remain inconclusive or not estimable. This uncertainty can be partly attributed to concentrated capability-demand levels within individual benchmarks, together with limited sample sizes or sparse outcomes in some subsets. For example, TC and CIE cover only levels 2–3 in SWE-bench subsets, restricting the variation available for trend estimation. We will add the benchmark-specific results to the appendix and clarify these limitations in the capability analysis.

We thank the reviewer again for the constructive feedback. We hope our responses and additional evidence address your concerns, and would appreciate reconsideration of your assessment and score if these concerns have been satisfactorily resolved. We will incorporate all additional results, methodological details, and clarifications into the revised manuscript.

Thanks for the new results. The fault-handling experiments show fewer execution failures, and the ALE-CLI rerun leaves the top harness unchanged in 44 of 48 groups. I’m keeping my rating at 2. The experts only reviewed labels the models disagreed on, so I’m still unsure how well the labels hold up on a random sample.

Thank you for the follow-up. We are glad the fault-handling experiments and the ALE-CLI rerun helped address your concerns about execution failures and harness rankings. To address your remaining concern, we randomly sample task–dimension labels on which all three LLM judges agree, balancing the sample across six benchmarks and six capability dimensions. For 81.4% of the evaluable labels, the LLM rating was within one level of the expert reference. In particular, agreement within one level reach over 90% for ICA and 100% for both SRV and RLPE. Agreement was lower for TC and CIE. One possible explanation is that, during a brief annotation session, experts may find it difficult to anticipate which tools an agent will use or how difficult the implementation will be. We expect agreement to increase if experts carry out the tasks themselves. We see implementation planning as a strength of LLMs, and the experts generally agree with the LLMs’ proposed level after reviewing them. We hope these results resolve your remaining concern.

Reviewer_41R8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Agents-Eval, a unified platform for evaluating LLM agents across different benchmarks and harnesses, with lifecycle management for execution, environments, and failure handling. It also introduces a capability translation matrix, using LLM-based annotations to describe task requirements along six capability dimensions. Experiments across multiple benchmarks analyze capability demands, model-harness interactions, and execution failures.

Critical strengths
  1. The platform addresses a practical problem in agent evaluation. Managing environments, harnesses, execution failures, and evaluation records is indeed painful. The integration of these components, especially keeping execution evidence for diagnosis, is useful for researchers running large-scale evaluations.

  2. Annotating task requirements independently of agent outputs is a reasonable idea. It provides a common view across heterogeneous benchmarks, and the observed model-harness differences are interesting. Same harness can behave quite differently with different models.

Critical weaknesses
  1. I have some doubt about whether the capability translation really provides meaningful diagnosis. The six dimensions are annotated by an LLM, with human calibration but no independent validation of annotation reliability. More importantly, these capabilities are highly correlated in real tasks. A task requiring complex tools usually also involves difficult coding or longer workflows. So the performance differences across demand levels may simply reflect overall task difficulty. Figure 4 also introduces an artificial zero-success anchor at demand level 20, while actual annotations only range from 0 to 5. This may influence the fitted trends. I don't quite see how these results can reliably tell which specific capability should be improved.

  2. The reproducibility claim is also not fully supported. Appendix A.3 shows different treatment of failed evaluations across benchmarks. In ALE-CLI, some execution failures are excluded from scoring, meaning different harnesses may effectively be evaluated on different subsets of successful executions. This could bias the harness comparison. Also, Figure 7 mainly reports failure distributions. It does not demonstrate how much the proposed lifecycle management actually improves reliability compared with existing evaluation systems.

Questions for authors
  1. How consistent are the capability annotations across independent human annotators or different LLM judges? Especially for overlapping dimensions like TC and CIE.

  2. For Figure 4, do the observed trends remain if removing the artificial demand-level-20 anchor and controlling for benchmark identity and correlated capability demands?

  3. For ALE-CLI, how many runs are excluded for each model-harness configuration? Would the harness comparison in Figure 5 change if all attempted tasks were included in the denominator?

2 comments

We thank the reviewer for recognizing the platform's practical value, the usefulness of retained execution evidence, and the benefits of annotating task requirements independently of agent outputs.

Response to W1 & Q1: Annotation Reliability and Diagnostic Value

Thank you for this constructive suggestion. We expanded annotation to three independent LLM judgesGPT−61˙−sol,DeepSeek−V41˙−Flash−1017,andKimi−K31˙−ProGPT-6\.1-sol, DeepSeek-V4\.1-Flash-1017, and Kimi-K3\.1-Pro and expanded the expert panel to three. Before adjudication, agreement was:

DimensionAll judges agree exactlyMaximum rating difference ≤1
ICA73.87%95.29%
TC43.04%89.24%
CIE66.57%96.93%
EISS86.84%98.66%
SRV92.51%97.69%
RLPE94.04%97.89%
Overall76.14%95.95%

We accept a label when at least two models agree and the maximum rating difference is ≤2, a criterion satisfied by 97.58% of task–dimension units. The remaining 2.42% are assigned to three experts for blind review, with majority voting and discussion for three-way disagreements. To clarify their operational distinction, TC concerns selecting and coordinating tools, whereas CIE concerns implementing and executing code. For example, ALE-CLI's internal HR/IT assistant task coordinates knowledge-base search, web search, email, and JIRA while tracking confirmations and cancellations. All three judges assigned TC=3, while CIE ratings were 1, 0, and 0, yielding CIE=0. The task uses provided tool interfaces and local fallbacks without requiring substantive code implementation, illustrating how diverse, stateful tool interactions can impose substantial TC demand while requiring little coding.

The diagnostic value lies in guiding targeted improvement: evidence of instruction-following weaknesses motivates constraint-adherence training, while coding weaknesses motivate implementation and debugging improvements. We agree that correlated demands complicate attribution. These profiles identify candidate capability gaps for focused follow-up testing, without establishing that a single capability independently causes the observed failures.

Response to W1 & Q2: Anchor Removal

The overall decreasing trends remain after removing the artificial 20,020, 0 anchor using the lateset annotations. With task selection, weighting, and fitting procedure held fixed, all six dimensions retain negative fitted slopes for both models. EISS, RLPE, and SRV become flatter, whereas ICA, TC, and CIE change little. Thus, the anchor affects some slope magnitudes but does not determine their direction. We refitted Figure 4 using the original data, holding task selection, weighting, and the fitting procedure fixed. All six dimensions retain negative fitted slopes for both evaluated models. ICA, EISS, and SRV become flatter, while TC, CIE, and RLPE change less. The ablation therefore preserves the qualitative direction while revealing some sensitivity in slope magnitude. Our interpretation is limited to the annotated demand range of 0–5.

Response to W2 & Q3: Failure Accounting and Reliability Improvements

Thank you for the careful examination. We apologize for the unclear reporting. Failure handling follows each benchmark’s official protocol: for example, ResearchClawBench excludes unscorable executions. Nevertheless, we agree that excluding ALE-CLI failures can bias harness comparisons.

We recomputed Figure 5 over ALE-CLI tasks for every configuration, assigning zero to failed or missing executions. The numbers previously excluded within this task pool were:

ModelClaude CodeALE-ClawCodex
Qwen-3.8-27B122
DeepSeek-v4.1323
GPT-6-Luna1052

Holding the latest labels fixed, the leading harness remains unchanged in 44 of 48 nonempty capability–demand groups. All DeepSeek group rankings remain unchanged. We will update Figure 5 and its interpretation, and specify the task pools, denominators, and failure policies in Appendix A.3.

Regarding reliability, our mitigations reduced failure rates by 4.13, 7.51, and 19.37 percentage points on SWE-bench Verified, Terminal-Bench 2.1, and ALE-CLI, respectively. In order to confirm reliability, we also compare our system with Harbor and AgentCompass on shared benchmarks under the official benchmark settings, reporting the score and failure rates respectively. The table reports Terminal-Bench 2.1 results with Claude Code with score / failure rate %. Officially reported scores are provided separately as references.

ModelAgents-EvalAgentCompassHarborOfficial reference score
Qwen-3.8-27B70.79 / 7.9068.50 / 18.5065.50 / 16.20—
DeepSeek-V4.1-Flash91.00 / 1.1075.70 / 10.7085.40 / 1.1088.00[1]
GLM 5.386.52 / 2.2068.50 / 15.7067.00 / 15.7088.20[2]

Agents-Eval achieves scores that are among the closest to the officially reported results and lower or equal failure rates across these configurations. We restrict this comparison to benchmarks supported by all the evaluated framework. Benchmarks such as ALE-CLI are not compared because neither framework supports them, and porting them would require substantial engineering beyond the scope of this comparison.

[1] DeepSeek-AI. n\.d\.. DeepSeek-V4.1-Flash [Model card]. Hugging Face.

[2] Z.ai. n\.d\.. GLM-5.3: Terminal-Bench 2.1 evaluation results [Evaluation record]. Hugging Face. Retrieved November 17, 2026, from the official evaluation record.

We thank the reviewer again for the constructive feedback. We hope our responses and additional evidence address your concerns, and would appreciate reconsideration of your assessment and score if these concerns have been satisfactorily resolved. We will incorporate all additional results, methodological details, and clarifications into the revised manuscript.

Reviewer_Tcd4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an open-source platform for agent evaluation with two parts. The first is a managed lifecycle, built on runners such as Harbor. The second is a “capability translation matrix”: an LLM rates each task 0–5 on six dimensions based on rubrics. Experiments show different benchmark capability profiles, gains that vary across models and tasks, and recurring infrastructure failures.

Critical strengths
  1. Standardizing agent evaluation is an important problem.
  2. The LLMaaJ evaluation is detailed and informative.
  3. The artifacts are potentially useful.
Critical weaknesses
  1. Key background works are not cited or compared. Specifically, Exgentic, Cube, or Hrabor are designed to address the same gap. A few other works tackle agent failure attribution, and general assessment, like agentic Clear, Trail, and more. More broadly, agent evaluation surveys, and the paper "Agentic systems should be general". As the latter only demonstrate the lack of proper contextualization of the work, the first points towards a gap in comparison with current methods that already aim to provide a remedy to the first issue, which is more engineering-heavy. The second part also needs better differentiation from what is currently supported, although it is not the main contribution.
  2. Design choices: 1. Evaluation is done with one judge, with no held-out set or human calibration. 2. Control demand is mostly data-dependent. 3. Harness comparison uses small test pools; high-demand ones are especially small.‏
Questions for authors
  1. Can the harness comparisons be redone on the intersection of valid tasks, with seeds and CIs?
  2. What is the failure rate over all runs, how were failures labeled, and did the mitigations measurably reduce them?
  3. What are the human–LLM and cross-LLM agreement for demand labels?
2 comments

We thank the reviewer for recognizing the importance of standardized agent evaluation, the informative evaluation analysis, and the potential usefulness of our artifacts.

Response to W1: Comparison with Related Work and Our Contributions

We agree that the related-work discussion needs strengthening. Exgentic, CUBE, and Harbor standardize benchmark integration and execution, while our contribution is a unified evaluation paradigm that makes fully configured and prevalidated task environments directly reusable across harnesses. Building on these efforts including Harbor as an integrated runner, our contribution is a shared, fully configured and validated evaluation environment, accompanied by concrete reproductions of reported results and controlled comparisons across models and harnesses.

This becomes particularly useful in complex environments such as ALE-CLI [1] and Toolathlon [2] which only Agents-Eval provides integrated support, where dependencies, persistent state, and tool interactions increase setup and debugging demands. Especially for observational evaluation, researchers can launch runs through the interface with one click without manually validating score reproduction against official references or repeatedly configuring and diagnosing complex environments, and focus on comparing agent behavior and performance.

To quantify the practical benefit, we compare Agents-Eval, AgentCompass [3], and Harbor [4] under matched APIs, computing resources, environment configurations, and evaluation settings. The table reports Terminal-Bench 2.1 results with Claude Code with score / failure rate %. Officially reported scores are provided separately as references.

ModelAgents-EvalAgentCompassHarborOfficial reference score
Qwen-3.8-27B70.79 / 7.9068.50 / 18.5065.50 / 16.20—
DeepSeek-V4.1-Flash91.00 / 1.1075.70 / 10.7085.40 / 1.1088.00[5]
GLM 5.386.52 / 2.2068.50 / 15.7067.00 / 15.7088.20[6]

Agents-Eval achieves scores that are among the closest to the officially reported results and lower or equal failure rates across these configurations. We restrict this comparison to benchmarks supported by all the evaluated framework versions. Benchmarks such as ALE-CLI are not compared because neither framework supports them, and porting them would require substantial engineering beyond the scope of this comparison.

[1] Sun, Y., Han, X., Zhang, W., Pang, Y., Wang, T., Cao, Y., Huang, Y., Duroiu, C., Zhang, H., Lin, J., Zhang, W., Zeng, T., Yan, Y., Liu, B., Wen, H., Xu, M., Liu, X., Chen, Z., Shi, W., . . . Jiang, Z. 20262026. Agents’ last exam. arXiv.

[2] Li, J., Zhao, W., Zhao, J., Zeng, W., Wu, H., Wang, X., Ge, R., Cao, Y., Huang, Y., Liu, W., Liu, J., Su, Z., Guo, Y., Zhou, F., Zhang, L., Michelini, J., Wang, X., Yue, X., Zhou, S., . . . He, J. 20252025. The Tool Decathlon: Benchmarking language agents for diverse, realistic, and long-horizon task execution. arXiv.

[3] Chen, K., Ding, Z., Ge, J., Jiang, S., Li, M., Li, Q., Li, Z., Li, Z., Liang, T., Liu, S., Ma, Z., Shang, Z., Tian, W., Wang, Z., Wu, L., Wu, Z., Xu, J., Yang, B., Yuan, D., . . . Zhu, D. 20262026. AgentCompass: A unified evaluation infrastructure for agent capabilities. arXiv.

[4] Harbor Framework Team. 20262026. Harbor: A framework for evaluating and optimizing agents and models in container environments.

[5] DeepSeek-AI. DeepSeek-V4.1-Flash [Model card]. Hugging Face.

[6] Z.ai. GLM-5.3: Terminal-Bench 2.1 evaluation results [Evaluation record]. Hugging Face. Retrieved November 17, 2026, from the official evaluation record.

Response to W2 & Q3: Annotation Reliability and Evaluation Design

Thank you for highlighting these limitations. We conducted additional validation with three LLM annotators GPT−61˙−sol,DeepSeek−V41˙−Flash−1017,andKimi−K31˙−ProGPT-6\.1-sol, DeepSeek-V4\.1-Flash-1017, and Kimi-K3\.1-Pro and expanded the expert panel to three.

Three LLMs independently annotated 1,041 tasks across six benchmarks. Across 6,246 task–dimension pairs, exact three-model agreement is 76.14%. We accept a label when at least two models agree and the maximum rating difference is ≤2, a criterion satisfied by 97.58% of task–dimension units. The remaining 2.42% are assigned to three experts for blind review, with majority voting and discussion for three-way disagreements. We will incorporate this protocol and agreement results into the manuscript, clarify the task-dependent nature of demand, and qualify conclusions from small high-demand subsets.

Response to Q1: Harness Comparisons and Statistical Uncertainty

Our evaluation focuses on how reliably each benchmark’s officially recommended harness executes its intended tasks, without requiring every harness to support all benchmarks. Variability intrinsic to benchmark design can persist under standardized execution and cannot be eliminated by the evaluation framework alone. We therefore emphasize evaluation stability as an important consideration in benchmark construction.

We also appreciate the reviewer’s constructive suggestion and will explore more general cross-harness comparisons on shared valid tasks.

Response to Q2: Failure Handling and Reliability Improvements

Failure handling follows each benchmark’s official protocol. For example, ResearchClawBench excludes failed runs from score aggregation. Failures are automatically identified from harness execution status and error logs. Our fault-tolerance mechanisms reduce failure rates by 4.13, 7.51, and 19.37 percentage points on SWE-bench Verified, Terminal-Bench 2.1, and ALE-CLI, respectively. Across all configurations, the equally weighted mean failure rate decreases from 15.36% to 5.02%, a 10.34-percentage-point reduction.

We will incorporate these comparisons and mitigation results into the experimental section and Figure 7 discussion, and clarify failure labeling, scoring rules, and aggregation methods in Appendix A.3.

We thank the reviewer again for the constructive feedback. We hope our responses and additional evidence address your concerns, and would appreciate reconsideration of your assessment and score if these concerns have been satisfactorily resolved. We will incorporate all additional results, methodological details, and clarifications into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.