- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors mean to investigate agentic harnesses.
To this effort, they construct a custom 300 task dataset, using user logs, public workflows, and public resources. The tasks go through human review. The construction of a custom dataset was motivated by wanting to create a judgeless task with stable starting conditions and specific delivery requirements. Though the authors promise to open source all evaluations and trajectories, and code, it is unclear if the dataset will be open sourced.
Then the authors test a matrix of Models x Harnesses on this 300 task dataset. By doing so, the authors find some insights, such as no single harness is the best, but it depends on the model. This goes for cost as well.
Critical strengths
The full model x harness gauntlet was potentially a very costly experiment that reports many different model-harness performances.
Critical weaknesses
No contribution over existing professional work / workflow benchmarks have been established.
Incremental contribution in the model-harness evaluation space is also insufficiently established. The analysis and experiments, though probably quite expensive as api costs go, primarily reports configuration-dependent performance differences with limited insights on harness mechanisms to explain those differences.
Questions for authors
n/a
2 comments
W2. Findings and explanations beyond configuration-dependent performance
We agree that the general observation that harness performance depends on the model is already supported by prior work. Our analysis provides more specific findings about the conditions under which harness choices change and the execution problems that accompany incomplete delivery.
1. Delivery objectives and task requirements change harness preferences.
In Appendix C.1, Table 10, optimizing model-specific harness selection for full-credit attainment improves held-out R₁ by 2.53–2.73 percentage points over selecting one shared harness. When selection instead optimizes mean score, the corresponding gains range from −0.14 to +0.14 score points. Thus, the practical benefit of model-specific selection depends on whether the application prioritizes average partial completion or complete delivery.
Task requirements also matter within the same model. Table 14 shows that Gemma’s OpenCode–Goose R₁ difference changes from +8.45 percentage points on tasks carrying an error-recovery demand tag to −7.39 percentage points on the remaining tasks. These results connect harness selection to both the delivery objective and the requirements of the work.
2. Trajectory annotation reveals distinct error and recovery patterns.
We respectfully disagree that the analysis is primarily limited to reporting configuration-dependent scores. Our Luna-assisted trajectory analysis examines execution errors, whether subsequent actions recover from them, and how these observations relate to final delivery. The annotations are grounded in task requirements, execution events, tool feedback, and artifact evidence, with human validation described in Appendix D.
To make this contribution clearer, we supplement this response with aggregate statistics already reported in Appendix D. The general-task base sample contains 5,600 trajectories. Table 22 shows markedly different recovery patterns across error categories:
| Error category | Annotated error observations | Followed by complete recovery | Recovery rate |
|---|---|---|---|
| Tool invocation | 729 | 655 | 89.8% |
| State and artifacts | 401 | 26 | 6.5% |
| Verification | 962 | 7 | 0.7% |
These counts concern individual error observations, and a trajectory can contain multiple labels. They describe the score-stratified sample rather than benchmark-wide prevalence.
The distinction between an execution error and an unsuccessful outcome is also important: 403 of the 1,400 full-credit runs in the general-task base sample contain an annotated error. An error count alone is therefore insufficient to characterize harness effectiveness; whether subsequent execution resolves the error matters.
Together, these findings distinguish frequently recovered tool errors from state, artifact, and verification errors that often remain unresolved in the recorded execution. They motivate examining recovery beyond successful tool invocation: whether workspace changes persist, whether the required artifacts are produced, and whether final verification covers the original task requirements.
3. Paired executions connect these patterns to concrete delivery failures.
For example, the materials-collection case in Figure 4 shows that Hermes omits a still-valid declaration, leading to an incorrect register entry and an unnecessary reminder. Its subsequent checks of the selected files and hashes do not expose the omission, whereas Goose retains the declaration. This illustrates a specific verification failure: checking the consistency of produced outputs does not establish that all source requirements have been covered.
The aggregate annotations establish that these error categories recur across executions, while paired cases explain how particular errors arise and affect delivery. These are diagnostic findings grounded in execution evidence; we do not claim that they isolate the causal effect of individual harness components.
We will foreground these results and connect the aggregate statistics, paired cases, and implications for harness design more explicitly in the main text. We ask that the benchmark dataset, reusable execution corpus, and diagnostic findings be considered together when assessing the contribution.
Thank you again for your feedback.
Thank you for your review. We would like to clarify the benchmark’s data contribution, the findings supported by the execution records, and the availability of the task dataset.
W1. Contribution beyond existing workflow benchmarks
We respectfully disagree with the assessment that no contribution beyond existing professional-work and workflow benchmarks has been established. Harness-Bench contributes a reusable dataset and evaluation protocol for connecting explicit workflow requirements, module-level delivery outcomes, execution behavior, and resource use across model–harness combinations.
We recognize that prior work, including HAL and DuMateBench, has established the importance of evaluating agent systems across models and execution frameworks. Our contribution builds on this foundation by examining how task requirements and delivery criteria affect harness selection, and by linking incomplete delivery to evidence from the corresponding executions. The resulting task dataset and execution corpus also provide resources for further research beyond the particular findings reported in this paper.
Dataset availability. The task definitions were already included in the submitted supplementary materials. The benchmark dataset is also included in our release, with task instructions, initial resources, environment setup, and task-specific evaluators. We will make this scope explicit in the reproducibility statement to remove the ambiguity identified in the review.
Value of the evaluation investment. The evaluation produces more than 20,000 execution records across the 300 tasks. These records support multiple analyses of the same configurations: average performance, complete delivery, cost–quality trade-offs, execution errors, and recovery behavior. Releasing the trajectories and evaluation records enables researchers to revisit these questions without repeating the original evaluation matrix. We therefore believe that the reusable dataset and execution evidence should be considered alongside the individual empirical findings when assessing the contribution.