acceptodds
Under review as a conference paper at ICLR 2027

Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

Abstract

LLM agents are increasingly used to solve real-world problems, and their effectiveness depends on both the underlying models and the harnesses that support their operation. However, most existing benchmarks focus on model capabilities and overall agent performance, with limited attention to the evaluation of agent harnesses. To address this gap, we introduce **TraceTrial-Bench**, which organizes harness reliability into seven components covering context, execution, governance, lifecycle, observability, tools, and verification. We construct 95 expert-level, mechanism-oriented tasks with executable or state-based ground truth to support semantically equivalent comparisons while preserving the native behavior of each harness. We design a **Trajectory Tracing (TT) module** to capture agent execution traces and introduce **a two-layer evaluation framework** that separately assesses agent performance and harness reliability. In addition, we design a **Multi-source Evidence Acquisition (MEA) module** to provide evidence from the agent context, harness, and execution environment for LLM-based evaluation, and develop a **Failure Attribution (FA) module** that uses explicit mechanism chains to support attribution at the run level. Across 4 harnesses, 9 models, and 95 tasks, our evaluation of 3,420 runs yields macro-averaged harness scores ranging from 59.8% to 72.0%, while 31.1% of runs achieve task success despite failing at least one harness criterion. TraceTrial-Bench provides an auditable and reproducible foundation for comparing agent harnesses, diagnosing their failures, and studying how harness design interacts with model behavior. Code is available at: https://anonymous.4open.science/r/Trace_anoy-F901.

Primary area:
datasets and benchmarks
Keywords:
LLM Agent, Agent Harness, Reliability Evaluation, Trajectory Tracing, Failure Attribution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PCRkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TraceTrial-Bench, a benchmark for evaluating agent harness reliability separately from end-to-end agent performance. It constructs 95 mechanism-oriented tasks across seven reliability dimensions and proposes a framework combining trajectory tracing, multi-source evidence collection, two-layer evaluation, and failure attribution. Experiments across four harnesses and nine models (3,420 runs) reveal substantial differences in harness reliability and underlying model behaviors.

Critical strengths

S1. Important Problem of Separating Agent and Harness Evaluation

The paper addresses an important and timely problem of evaluating Agent performance and Harness reliability separately, rather than relying solely on end-to-end task success. This distinction is valuable for understanding agent failures and identifying potential harness-related limitations.

S2. Extensive Evaluation Across Models and Harnesses

The paper conducts an extensive evaluation involving four representative agent harnesses, nine models, and 95 tasks, covering 3,420 model–harness–task executions. The comprehensive model–harness evaluation grid provides a valuable basis for comparing harness behaviors across different models and studying their interactions.

Critical weaknesses

W1. Insufficient Methodological Justification for Separating Model and Harness Effects

A central claim of this work is the separate evaluation of Agent performance and Harness reliability. However, it remains unclear how the proposed benchmark and task design actually disentangles model-dependent behavior from harness-specific effects, beyond introducing two sets of evaluation criteria. While the complete model–harness evaluation grid enables matched comparisons, the paper does not sufficiently explain the key challenges in constructing such a benchmark or demonstrate how its design addresses potential confounding factors, such as differences in model behavior. This raises concerns about whether the resulting Harness scores reliably reflect harness-specific limitations.

W2. Limited Experimental Analysis and Insufficient Evidence for the Reported Findings

Despite the extensive evaluation across four harnesses and nine models, the experimental analysis remains largely descriptive, focusing on aggregate scores and rankings without sufficiently investigating the underlying causes of performance differences. For example, Codex consistently obtains lower Harness scores, yet the paper provides limited insights into which specific failure mechanisms explain this gap. Although Appendix H presents several case studies, they offer limited generalizable takeaways. The lack of deeper diagnostic analysis raises concerns about the robustness and interpretability of the reported findings.

Questions for authors

Q1. Benchmark Design for Disentangling Agent and Harness Performance

It remains unclear how the proposed task and benchmark design enables a meaningful separation of Agent performance and Harness reliability, beyond evaluating them with two distinct sets of criteria. What are the key challenges in constructing tasks that can independently assess these aspects, and how does the proposed design address them? Although the complete model–harness evaluation grid provides matched comparisons, how does the benchmark ensure that observed Harness failures reflect harness-specific limitations rather than model behavior?

Q2. Diagnostic Insights from Model–Harness Comparisons

The complete model–harness evaluation grid is a strength, but the analysis provides limited explanations for the observed performance differences. For example, Codex consistently obtains lower Harness scores across all nine models. What specific failure mechanisms account for this gap? Could the authors provide a property-level breakdown or cross-harness trace analysis to identify recurring causes? Overall, I feel deeper analysis on the experimental results are needed, and those takeaways perhaps are the most interesting part of the paper (although I know it is a benchmark paper).

Q3. Utility and Validity of Failure Attribution

What additional diagnostic value does the proposed Failure Attribution module provide beyond the individual Agent/Harness verdicts and direct trace inspection? How should activated links be interpreted or used to evaluate harness reliability?

Q4. Benchmark Coverage and Representativeness

How representative are the 95 tasks of the seven proposed harness reliability dimensions, particularly given that Verification contains only one task and several other dimensions have limited coverage? Could the authors clarify the diversity of task scenarios and provide evidence that the reported harness rankings are robust to the choice or weighting of tasks and dimensions?

Reviewer_roAEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes to study the reliability of agent harnesses via the introduction of a new benchmark. It uses 7 different evaluation criteria that assess different harness components and uses an evaluation protocol that separates agent performance from harness performance. Runs on a variety of harness and model combinations show a fair amount of variance in agent performance.

Critical strengths
  • Modern agent systems gain a substantial degree of performance from particular scaffold choices. As a result it seems vital to study how particular changes in the harness influence reliability of the overall agent system.

  • The general design of the benchmark seems sensible to me.

Critical weaknesses
  • Clarity is lacking in many places, likely due to an over-reliance on AI assisted writing. There are many tell-tale signs all over the paper. While the authors say they used AI to help to improve clarity, I would argue that their use of AI tools has substantially harmed clarity in many places.

  • The paper does not motivate the 7 reliability dimensions well. Where do they come from? Why do they matter? Recent work has argued for a different set of reliability dimensions with more grounding in safety-critical disciplines [1] which are arguably also important under scaffold perturbation.

  • Likewise, it is not clear to me what is discussed in the two layer evaluation section. I think this section, and maybe even 4.4, needs to be rewritten with clearer, non-AI generated language.

  • The interpretation of the results appears very shallow to me. A lot of numbers are reported but it is not clear to me what the broader conclusion is despite the fact that harness choice matters and that there will be a performance delta based on which harness is chosen. Which, in turn, does not seem all that surprising.

References:

[1]: Rabanser, Stephan, et al. "Towards a science of AI agent reliability." ICML 2026.

Questions for authors
  • Can you better explain the reliability dimensions?
Reviewer_fi6VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed TraceTrial-Bench. It categorizes harness reliability into 7 components: context, execution, governance, lifecycle, observability, tools, and verification. The authors also introduced a Trajectory Tracing module to capture agent execution traces and a framework to assess agent performance and harness reliability. The evaluation is done on 4 harnesses and 9 models on 95 tasks. Overall, it offers a good toolset to audit and compare harnesses.

Critical strengths
  1. Good engineering work. The framework design is rigorous.
  2. The paper provides a good framework to audit harness, which is important for agent building work.
Critical weaknesses
  1. The data were generated from AI, I'm worried about the quality of the data. Since generating data is also depending on harnesses, could introduce bias.
  2. The authors should compare with Harness-Bench concretely. That benchmark also runs native harnesses across model backends and would be valuable to discuss it.
  3. For the cost comparison, the cache-counter availability differs between different harnesses. For Opus 5, claude code costs are different with other harnesses. So the cost table mixes pricing coverage.
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.