- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TraceTrial-Bench, a benchmark for evaluating agent harness reliability separately from end-to-end agent performance. It constructs 95 mechanism-oriented tasks across seven reliability dimensions and proposes a framework combining trajectory tracing, multi-source evidence collection, two-layer evaluation, and failure attribution. Experiments across four harnesses and nine models (3,420 runs) reveal substantial differences in harness reliability and underlying model behaviors.
Critical strengths
S1. Important Problem of Separating Agent and Harness Evaluation
The paper addresses an important and timely problem of evaluating Agent performance and Harness reliability separately, rather than relying solely on end-to-end task success. This distinction is valuable for understanding agent failures and identifying potential harness-related limitations.
S2. Extensive Evaluation Across Models and Harnesses
The paper conducts an extensive evaluation involving four representative agent harnesses, nine models, and 95 tasks, covering 3,420 model–harness–task executions. The comprehensive model–harness evaluation grid provides a valuable basis for comparing harness behaviors across different models and studying their interactions.
Critical weaknesses
W1. Insufficient Methodological Justification for Separating Model and Harness Effects
A central claim of this work is the separate evaluation of Agent performance and Harness reliability. However, it remains unclear how the proposed benchmark and task design actually disentangles model-dependent behavior from harness-specific effects, beyond introducing two sets of evaluation criteria. While the complete model–harness evaluation grid enables matched comparisons, the paper does not sufficiently explain the key challenges in constructing such a benchmark or demonstrate how its design addresses potential confounding factors, such as differences in model behavior. This raises concerns about whether the resulting Harness scores reliably reflect harness-specific limitations.
W2. Limited Experimental Analysis and Insufficient Evidence for the Reported Findings
Despite the extensive evaluation across four harnesses and nine models, the experimental analysis remains largely descriptive, focusing on aggregate scores and rankings without sufficiently investigating the underlying causes of performance differences. For example, Codex consistently obtains lower Harness scores, yet the paper provides limited insights into which specific failure mechanisms explain this gap. Although Appendix H presents several case studies, they offer limited generalizable takeaways. The lack of deeper diagnostic analysis raises concerns about the robustness and interpretability of the reported findings.
Questions for authors
Q1. Benchmark Design for Disentangling Agent and Harness Performance
It remains unclear how the proposed task and benchmark design enables a meaningful separation of Agent performance and Harness reliability, beyond evaluating them with two distinct sets of criteria. What are the key challenges in constructing tasks that can independently assess these aspects, and how does the proposed design address them? Although the complete model–harness evaluation grid provides matched comparisons, how does the benchmark ensure that observed Harness failures reflect harness-specific limitations rather than model behavior?
Q2. Diagnostic Insights from Model–Harness Comparisons
The complete model–harness evaluation grid is a strength, but the analysis provides limited explanations for the observed performance differences. For example, Codex consistently obtains lower Harness scores across all nine models. What specific failure mechanisms account for this gap? Could the authors provide a property-level breakdown or cross-harness trace analysis to identify recurring causes? Overall, I feel deeper analysis on the experimental results are needed, and those takeaways perhaps are the most interesting part of the paper (although I know it is a benchmark paper).
Q3. Utility and Validity of Failure Attribution
What additional diagnostic value does the proposed Failure Attribution module provide beyond the individual Agent/Harness verdicts and direct trace inspection? How should activated links be interpreted or used to evaluate harness reliability?
Q4. Benchmark Coverage and Representativeness
How representative are the 95 tasks of the seven proposed harness reliability dimensions, particularly given that Verification contains only one task and several other dimensions have limited coverage? Could the authors clarify the diversity of task scenarios and provide evidence that the reported harness rankings are robust to the choice or weighting of tasks and dimensions?