- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper attempts to address the issue of how AI agents are systematically measured and fairly compared in verifiable reasoning tasks. The authors propose the AlphaDiana evaluation system, which supports session isolation and controlled reuse, preserves harness-native execution, and links results to execution evidence, thereby supporting macro comparison of complete agents and micro intervention analysis of tools, skills, memory, and other components. Based on this, the experiments also reveal several counterintuitive empirical findings.
Critical strengths
- Provides a unified Agent evaluation framework that can control reuse conditions while preserving the native execution process of each harness, making cross-model and cross-harness comparisons more fair and reproducible.
- Links answers, execution trajectories, and resource consumption together, and also supports component-level controlled experiments, which can help identify specific failure patterns rather than being limited to final accuracy.
- Reveals a number of counterintuitive empirical findings, such as harnesses sometimes lowering overall accuracy, skills and memory not necessarily improving accuracy, and fewer tokens per attempt sometimes leading to higher token costs per correct answer.
Critical weaknesses
- The component experiments did not completely isolate variables, and when the tools were removed, the prompt instructions also changed accordingly. It is difficult to distinguish whether the performance differences were caused by tool removal or changes to the prompts.
- There are significant differences in computational consumption among different harnesses, and some token records are missing.
- In the paper, the error labels and functional-step labels are mainly generated automatically by GPT-5.6-sol, and human verification only covers a small part.
Questions for authors
- Can the authors provide direct functional or experimental comparisons to demonstrate the core innovative value of AlphaDiana relative to existing evaluation frameworks like Harbor and the Holistic Agent Leaderboard?
- The tool-removal and memory experiments involved modifying the prompt instructions alongside the components themselves. How do the authors distinguish whether the results stem from the components' lack of utility or from the changes made to the prompts?
- Different harnesses inherently involve varying computational costs and operating conditions, and some token logs are incomplete. Might these factors undermine the validity of the efficiency comparisons? Would the conclusions still hold if experimental conditions were strictly aligned?
- Most primary experiments were conducted only once; can the authors provide additional repeated trials or statistical tests to demonstrate that the observed key differences are not merely due to random fluctuation?
- Error classification and functional step labeling relied almost entirely on automated processing via GPT-5.6-sol, with only a small fraction verified manually. How do the authors ensure that annotation biases are not amplified when comparing failure modes across different harnesses?
7 comments
Response to Reviewer Pe8a (part 2 / 4)
W2 & Q3. Efficiency under different execution conditions
There are significant differences in computational consumption among different harnesses, and some token records are missing.
Different harnesses inherently involve varying computational costs and operating conditions, and some token logs are incomplete. Might these factors undermine the validity of the efficiency comparisons? Would the conclusions still hold if experimental conditions were strictly aligned?
Reply: We would like to clarify our efficiency comparisons have a reliable basis. First, the missing records arise from inference interrupted by timeouts or network errors. Excluding these incomplete records avoids underestimating cost and makes the accounting more accurate. Second, even after aligning the task set and token accounting, the core conclusion still holds: lower output per attempt does not necessarily mean lower cost per success. We explain these two points below.
(1) We exclude interrupted records to avoid underestimating cost. Timeouts or network errors can interrupt inference before completion, leaving token records that cover only part of the execution. Including these records would treat unfinished inference as a low-cost attempt. We therefore exclude incomplete records while retaining the token costs of completed but incorrect attempts. The excluded fraction is small, for example, 4 out of 591 tasks (0.68%) for Gemma on HLE-Verifiable.
(2) The conclusion holds with a common task set and consistent token accounting. We recalculated costs on the same 587 tasks with complete records for all systems, using the same definitions of output tokens per attempt and per correct answer. OpenCode generates approximately 0.6% fewer output tokens per attempt than Direct, but uses approximately 16.5% more output tokens per correct answer. Thus, the observation that lower output per attempt does not necessarily mean lower cost per success holds under this common accounting protocol.
Together, excluding incomplete records avoids undercounting interrupted executions, while the shared task set and consistent token accounting make the systems comparable. Under this protocol, lower output per attempt still does not necessarily mean lower cost per success.
W3 & Q5. Reliability of trace annotations
In the paper, the error labels and functional-step labels are mainly generated automatically by GPT-5.6-sol, and human verification only covers a small part.
Error classification and functional step labeling relied almost entirely on automated processing via GPT-5.6-sol, with only a small fraction verified manually. How do the authors ensure that annotation biases are not amplified when comparing failure modes across different harnesses?
Reply: We examined model-human annotation agreement for each harness and found high agreement across all four systems, supporting the reliability of comparisons across harnesses. The audit covers Direct, OpenClaw, ZeroClaw, and OpenCode, with a similar number of trajectories per system, assessed by two human annotators using the same error taxonomy. The results are shown below.
| Harness | Audited trajectories | Model-human agreement |
|---|---|---|
| Direct | 9 | 85.6% |
| OpenClaw | 13 | 86.7% |
| ZeroClaw | 12 | 88.5% |
| OpenCode | 8 | 91.9% |
| Overall | 42 | 88.0% |
Agreement is computed over the 17 error labels for each trajectory and averaged over comparisons with the two human annotators. Model-human agreement is high across all systems, ranging from 85.6% to 91.9%, showing that strong agreement is not confined to a single harness. The numbers of audited trajectories are also relatively similar across systems. Functional-step labels have a separate, independent human audit, with mean model-human pairwise agreement of 76.0% when machine labels are hidden.
In summary, shared annotation rules and trajectory-evidence requirements help reduce differences in how systems are judged. The per-harness human audit and the independent functional-step audit together support the reliability of cross-harness trace analyses.
Response to Reviewer Pe8a (part 3 / 4)
Q1. What AlphaDiana adds to evaluation frameworks
Can the authors provide direct functional or experimental comparisons to demonstrate the core innovative value of AlphaDiana relative to existing evaluation frameworks like Harbor and the Holistic Agent Leaderboard?
Reply: AlphaDiana combines native-harness component interventions with semantic and token-level uncertainty analyses to explain how harnesses change model behavior and task outcomes. HAL, Harbor, and Inspect also support controlled evaluation, native agent execution, and parts of the component interventions. AlphaDiana provides intervention configurations across harnesses together with functional-step labeling, transition analysis, and token-entropy analysis. Conducting these specific analyses on the other platforms requires additional implementation. The table below summarizes these capabilities, followed by details of the component interventions and the two types of analysis.
| Capability | HAL [1] | Harbor [2] | Inspect [3] | AlphaDiana |
|---|---|---|---|---|
| Controlled comparisons across models, agents, and tasks | ✓ | ✓ | ✓ | ✓ |
| Execution of agents using their native control loops | ✓ | ✓ | ✓ | ✓ |
| Tool, skill, and memory interventions within native harnesses | ⍻ | ⍻ | ⍻ | ✓ |
| Semantic analysis of reasoning processes | × | × | × | ✓ |
| Token-level uncertainty analysis | × | × | × | ✓ |
✓ = provided, ⍻ = partially supported, × = requires additional implementation.
(1) Component interventions preserve the native execution workflow. AlphaDiana provides configurations for changing tool exposure, loading skill libraries, and controlling memory retention while retaining each harness's own model calls, tool execution, and feedback handling. Memory settings specify whether memory is cleared after each attempt, retained across attempts on the same task, or accumulated across tasks. Other platforms cover parts of this functionality. Harbor supports native skill loading and tool restrictions, while Inspect provides configurable tools, skills, memory, and external-agent interfaces. AlphaDiana organizes these interventions into component experiments that can be run and compared across the evaluated harnesses.
(2) Semantic analysis of reasoning processes. AlphaDiana labels traces from different harnesses using a shared set of functional steps, then analyzes their distribution and transitions to explain behavioral differences between correct and incorrect executions. For example, we examine whether verification is followed by further reasoning or revision, and how these patterns differ between correct and incorrect attempts. This connects score differences to the execution processes behind them.
(3) Token-level uncertainty analysis. AlphaDiana measures token entropy around tool interactions and compares these patterns between correct and incorrect attempts. This supports the paper's observation that low-entropy generation can accompany incorrect answers and helps characterize changes in generation uncertainty following tool responses.
Conducting these studies in HAL, Harbor, or Inspect requires adding the corresponding component configurations, functional-step labeling, and token-entropy analysis procedures. AlphaDiana implements and connects these experimental controls and analyses, linking harness configuration changes to execution behavior and task outcomes.
Response to Reviewer Pe8a (part 1 / 4)
Thank you for the thoughtful review and constructive feedback. We address your concerns and questions below.
W1 & Q2. What the component interventions measure
Reply: Prompt changes are necessary for these component experiments to suppress tool calls and elicit memory use, respectively. In the tool experiments, models rarely attempt tool calls after the prompt changes, so additionally disabling tool execution has no further effect in the vast majority of trajectories. In the memory experiments, models did not actively use memory under the default configurations, so explicit instructions were needed to engage this capability and study different retention and reuse policies. We explain each case below.
(1) Tools. Our inspection of the execution traces shows that models rarely initiate tool calls after the tool-related prompt instructions are changed. Only 1.73% of ZeroClaw trajectories contain actual tool calls, and no structured tool calls appear in the captured OpenCode responses. Thus, in the vast majority of trajectories, additionally disabling tool execution does not change the actual execution process.
(2) Memory. In our initial experiments, we found that models did not actively use memory without additional prompting. In the default trajectories we inspected for Qwen3.5-27B and Kimi-K2.6 under ZeroClaw and OpenCode, both active memory calls and memory-file reads and writes were zero. We therefore added explicit memory-use instructions to engage this capability and examine the effects of different memory retention and reuse policies.
Thus, the two tool-side changes correspond to a single behavioral outcome in most trajectories: no tool use. For memory, the prompt addition is a necessary step to elicit memory use and study its retention and reuse policies.