acceptodds
Under review as a conference paper at ICLR 2027

AlphaDiana: A Measurement System for Agents in Verifiable Reasoning Tasks

Abstract

Agents pair reasoning models with harnesses that organize model calls, execute tools, load skill instructions, and retain history in memory. Understanding agent performance requires linking execution conditions to outcomes, behavior, and resource use. We introduce AlphaDiana, a measurement system with modular interfaces for benchmarks, agents, environments, and scorers. It preserves harness-native control loops while controlling file reuse and memory retention separately. AlphaDiana supports macro analyses of complete agents and micro interventions on agent components such as tools, skills, and memory under matched conditions. We study three open-source models and three open-source harnesses on five answer-based benchmarks, with two models also tested on two interactive benchmarks. We find that harnesses do not consistently improve the accuracy of reasoning: gains on questions unsolved by the model alone can be outweighed by losses on questions it answers correctly without tools. This inconsistency extends to component interventions: neither skill guidance nor retained history reliably improves accuracy. Behavior and cost must also be interpreted alongside correctness: low token entropy can accompany wrong answers, and less output per execution can mean more output tokens per correct answer when accuracy falls. AlphaDiana links answer correctness and execution status to traces and resource records for comparing agents and diagnosing failures under shared task and resource policies.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Agent, measurement, system, verifiable reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Pe8aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper attempts to address the issue of how AI agents are systematically measured and fairly compared in verifiable reasoning tasks. The authors propose the AlphaDiana evaluation system, which supports session isolation and controlled reuse, preserves harness-native execution, and links results to execution evidence, thereby supporting macro comparison of complete agents and micro intervention analysis of tools, skills, memory, and other components. Based on this, the experiments also reveal several counterintuitive empirical findings.

Critical strengths
  1. Provides a unified Agent evaluation framework that can control reuse conditions while preserving the native execution process of each harness, making cross-model and cross-harness comparisons more fair and reproducible.
  2. Links answers, execution trajectories, and resource consumption together, and also supports component-level controlled experiments, which can help identify specific failure patterns rather than being limited to final accuracy.
  3. Reveals a number of counterintuitive empirical findings, such as harnesses sometimes lowering overall accuracy, skills and memory not necessarily improving accuracy, and fewer tokens per attempt sometimes leading to higher token costs per correct answer.
Critical weaknesses
  1. The component experiments did not completely isolate variables, and when the tools were removed, the prompt instructions also changed accordingly. It is difficult to distinguish whether the performance differences were caused by tool removal or changes to the prompts.
  2. There are significant differences in computational consumption among different harnesses, and some token records are missing.
  3. In the paper, the error labels and functional-step labels are mainly generated automatically by GPT-5.6-sol, and human verification only covers a small part.
Questions for authors
  1. Can the authors provide direct functional or experimental comparisons to demonstrate the core innovative value of AlphaDiana relative to existing evaluation frameworks like Harbor and the Holistic Agent Leaderboard?
  2. The tool-removal and memory experiments involved modifying the prompt instructions alongside the components themselves. How do the authors distinguish whether the results stem from the components' lack of utility or from the changes made to the prompts?
  3. Different harnesses inherently involve varying computational costs and operating conditions, and some token logs are incomplete. Might these factors undermine the validity of the efficiency comparisons? Would the conclusions still hold if experimental conditions were strictly aligned?
  4. Most primary experiments were conducted only once; can the authors provide additional repeated trials or statistical tests to demonstrate that the observed key differences are not merely due to random fluctuation?
  5. Error classification and functional step labeling relied almost entirely on automated processing via GPT-5.6-sol, with only a small fraction verified manually. How do the authors ensure that annotation biases are not amplified when comparing failure modes across different harnesses?
7 comments
AuthorsOpenReview ↗

Response to Reviewer Pe8a (part 1 / 4)

Thank you for the thoughtful review and constructive feedback. We address your concerns and questions below.

W1 & Q2. What the component interventions measure

The component experiments did not completely isolate variables, and when the tools were removed, the prompt instructions also changed accordingly. It is difficult to distinguish whether the performance differences were caused by tool removal or changes to the prompts.

The tool-removal and memory experiments involved modifying the prompt instructions alongside the components themselves. How do the authors distinguish whether the results stem from the components' lack of utility or from the changes made to the prompts?

Reply: Prompt changes are necessary for these component experiments to suppress tool calls and elicit memory use, respectively. In the tool experiments, models rarely attempt tool calls after the prompt changes, so additionally disabling tool execution has no further effect in the vast majority of trajectories. In the memory experiments, models did not actively use memory under the default configurations, so explicit instructions were needed to engage this capability and study different retention and reuse policies. We explain each case below.

(1) Tools. Our inspection of the execution traces shows that models rarely initiate tool calls after the tool-related prompt instructions are changed. Only 1.73% of ZeroClaw trajectories contain actual tool calls, and no structured tool calls appear in the captured OpenCode responses. Thus, in the vast majority of trajectories, additionally disabling tool execution does not change the actual execution process.

(2) Memory. In our initial experiments, we found that models did not actively use memory without additional prompting. In the default trajectories we inspected for Qwen3.5-27B and Kimi-K2.6 under ZeroClaw and OpenCode, both active memory calls and memory-file reads and writes were zero. We therefore added explicit memory-use instructions to engage this capability and examine the effects of different memory retention and reuse policies.

Thus, the two tool-side changes correspond to a single behavioral outcome in most trajectories: no tool use. For memory, the prompt addition is a necessary step to elicit memory use and study its retention and reuse policies.

AuthorsOpenReview ↗

Response to Reviewer Pe8a (part 2 / 4)

W2 & Q3. Efficiency under different execution conditions

There are significant differences in computational consumption among different harnesses, and some token records are missing.

Different harnesses inherently involve varying computational costs and operating conditions, and some token logs are incomplete. Might these factors undermine the validity of the efficiency comparisons? Would the conclusions still hold if experimental conditions were strictly aligned?

Reply: We would like to clarify our efficiency comparisons have a reliable basis. First, the missing records arise from inference interrupted by timeouts or network errors. Excluding these incomplete records avoids underestimating cost and makes the accounting more accurate. Second, even after aligning the task set and token accounting, the core conclusion still holds: lower output per attempt does not necessarily mean lower cost per success. We explain these two points below.

(1) We exclude interrupted records to avoid underestimating cost. Timeouts or network errors can interrupt inference before completion, leaving token records that cover only part of the execution. Including these records would treat unfinished inference as a low-cost attempt. We therefore exclude incomplete records while retaining the token costs of completed but incorrect attempts. The excluded fraction is small, for example, 4 out of 591 tasks (0.68%) for Gemma on HLE-Verifiable.

(2) The conclusion holds with a common task set and consistent token accounting. We recalculated costs on the same 587 tasks with complete records for all systems, using the same definitions of output tokens per attempt and per correct answer. OpenCode generates approximately 0.6% fewer output tokens per attempt than Direct, but uses approximately 16.5% more output tokens per correct answer. Thus, the observation that lower output per attempt does not necessarily mean lower cost per success holds under this common accounting protocol.

Together, excluding incomplete records avoids undercounting interrupted executions, while the shared task set and consistent token accounting make the systems comparable. Under this protocol, lower output per attempt still does not necessarily mean lower cost per success.

W3 & Q5. Reliability of trace annotations

In the paper, the error labels and functional-step labels are mainly generated automatically by GPT-5.6-sol, and human verification only covers a small part.

Error classification and functional step labeling relied almost entirely on automated processing via GPT-5.6-sol, with only a small fraction verified manually. How do the authors ensure that annotation biases are not amplified when comparing failure modes across different harnesses?

Reply: We examined model-human annotation agreement for each harness and found high agreement across all four systems, supporting the reliability of comparisons across harnesses. The audit covers Direct, OpenClaw, ZeroClaw, and OpenCode, with a similar number of trajectories per system, assessed by two human annotators using the same error taxonomy. The results are shown below.

HarnessAudited trajectoriesModel-human agreement
Direct985.6%
OpenClaw1386.7%
ZeroClaw1288.5%
OpenCode891.9%
Overall4288.0%

Agreement is computed over the 17 error labels for each trajectory and averaged over comparisons with the two human annotators. Model-human agreement is high across all systems, ranging from 85.6% to 91.9%, showing that strong agreement is not confined to a single harness. The numbers of audited trajectories are also relatively similar across systems. Functional-step labels have a separate, independent human audit, with mean model-human pairwise agreement of 76.0% when machine labels are hidden.

In summary, shared annotation rules and trajectory-evidence requirements help reduce differences in how systems are judged. The per-harness human audit and the independent functional-step audit together support the reliability of cross-harness trace analyses.

AuthorsOpenReview ↗

Response to Reviewer Pe8a (part 3 / 4)

Q1. What AlphaDiana adds to evaluation frameworks

Can the authors provide direct functional or experimental comparisons to demonstrate the core innovative value of AlphaDiana relative to existing evaluation frameworks like Harbor and the Holistic Agent Leaderboard?

Reply: AlphaDiana combines native-harness component interventions with semantic and token-level uncertainty analyses to explain how harnesses change model behavior and task outcomes. HAL, Harbor, and Inspect also support controlled evaluation, native agent execution, and parts of the component interventions. AlphaDiana provides intervention configurations across harnesses together with functional-step labeling, transition analysis, and token-entropy analysis. Conducting these specific analyses on the other platforms requires additional implementation. The table below summarizes these capabilities, followed by details of the component interventions and the two types of analysis.

CapabilityHAL [1]Harbor [2]Inspect [3]AlphaDiana
Controlled comparisons across models, agents, and tasks✓✓✓✓
Execution of agents using their native control loops✓✓✓✓
Tool, skill, and memory interventions within native harnesses⍻⍻⍻✓
Semantic analysis of reasoning processes×××✓
Token-level uncertainty analysis×××✓

✓ = provided, ⍻ = partially supported, × = requires additional implementation.

(1) Component interventions preserve the native execution workflow. AlphaDiana provides configurations for changing tool exposure, loading skill libraries, and controlling memory retention while retaining each harness's own model calls, tool execution, and feedback handling. Memory settings specify whether memory is cleared after each attempt, retained across attempts on the same task, or accumulated across tasks. Other platforms cover parts of this functionality. Harbor supports native skill loading and tool restrictions, while Inspect provides configurable tools, skills, memory, and external-agent interfaces. AlphaDiana organizes these interventions into component experiments that can be run and compared across the evaluated harnesses.

(2) Semantic analysis of reasoning processes. AlphaDiana labels traces from different harnesses using a shared set of functional steps, then analyzes their distribution and transitions to explain behavioral differences between correct and incorrect executions. For example, we examine whether verification is followed by further reasoning or revision, and how these patterns differ between correct and incorrect attempts. This connects score differences to the execution processes behind them.

(3) Token-level uncertainty analysis. AlphaDiana measures token entropy around tool interactions and compares these patterns between correct and incorrect attempts. This supports the paper's observation that low-entropy generation can accompany incorrect answers and helps characterize changes in generation uncertainty following tool responses.

Conducting these studies in HAL, Harbor, or Inspect requires adding the corresponding component configurations, functional-step labeling, and token-entropy analysis procedures. AlphaDiana implements and connects these experimental controls and analyses, linking harness configuration changes to execution behavior and task outcomes.

Reviewer_j5iKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents AlphaDiana, a modular measurement system for evaluating complete model–harness agents. It has pluggable benchmark, agent, environment, and scorer interfaces; separately controlled session isolation, file reset, and memory retention; and normalized traces linking each attempt’s outcome to behavior and resource use. Using it, the authors pair three open models with three harnesses on five answer-based benchmarks plus two interactive ones. They also run matched interventions on tools, skills, and memory. The main findings are that harnesses often lower accuracy relative to direct inference while solving some new questions, that skills and memory do not reliably help, and that entropy, token, time, and cost-per-success metrics can give conflicting pictures.

Critical strengths
  • Evaluating the full model-harness system rather than the LLM in isolation is a timely topic. Explicitly controlling session isolation, file reuse, and memory persistence is good engineering practice, since cross-episode state leakage is a common headache in agent evaluation.
  • The adapter design keeps each harness's native control loop intact while standardizing I/O and logging. Separating wrong answers, execution crashes, and unscored attempts in the outcome records is also good evaluation hygiene.
  • Running controlled micro-interventions (removing tools, loading skills, varying memory scope) while holding the model, decoding parameters, and budget fixed is a clean way to isolate harness effects.
Critical weaknesses
  • Unclear what AlphaDiana does that existing evaluation frameworks don't already do. Related work is pushed entirely to Appendix E as a generic survey. As a reader, I can't tell what specific measurement or intervention AlphaDiana enables that couldn't easily be done in HAL, Harbor, or Inspect.

  • Many of the ten "Findings" are mathematically obvious or definitional rather than new empirical insights. For example:

  1. Finding 9 basically states that cost per success = (tokens per attempt) / accuracy, which is true by definition.
  2. Finding 6 is just that Avg@4 is not equal to Pass@4.
  3. Finding 3 is simply the difference between macro-averaging (per-trajectory) and micro-averaging (pooled) when trajectory lengths vary.
  • Heavy reliance on LLM-as-a-judge for trace labeling. Error codes and functional steps are labeled by GPT-5.6-sol, with human verification on only 42 trajectories.
Questions for authors
  1. Can you provide a concrete table comparing AlphaDiana against HAL, Harbor, and Inspect, and clarify what analyses in this paper could not be run on those existing platforms?
  2. For Finding 1 on GPQA-Diamond, what is the union accuracy of 4 independent samples from the Direct baseline alone (Direct Pass@4) compared to the 4-harness union (175/198)?
  3. Did you check out-of-the-box tool-calling compatibility for each model-harness pair? What fraction of harness failures come from tool-format/parsing errors versus genuine reasoning mistakes?
  4. Are you releasing the AlphaDiana framework code, harness adapters, and full execution traces?
6 comments
AuthorsOpenReview ↗

Response to Reviewer j5iK (part 1 / 4)

Thank you for the thoughtful review and constructive feedback. We address your concerns and questions below.

W1 & Q1. The contribution beyond existing evaluation frameworks

Unclear what AlphaDiana does that existing evaluation frameworks don't already do. Related work is pushed entirely to Appendix E as a generic survey. As a reader, I can't tell what specific measurement or intervention AlphaDiana enables that couldn't easily be done in HAL, Harbor, or Inspect.

Can you provide a concrete table comparing AlphaDiana against HAL, Harbor, and Inspect, and clarify what analyses in this paper could not be run on those existing platforms?

Reply: AlphaDiana integrates native-harness component interventions with semantic and token-level uncertainty analyses to explain how harnesses change model behavior and task outcomes. HAL, Harbor, and Inspect also support controlled evaluation, native agent execution, and parts of the component interventions. AlphaDiana provides intervention configurations across harnesses together with functional-step labeling, transition analysis, and token-entropy analysis. Conducting these specific analyses on the other platforms requires additional implementation. The table below summarizes these capabilities, followed by details of the component interventions and the two types of analysis.

CapabilityHAL [1]Harbor [2]Inspect [3]AlphaDiana
Controlled comparisons across models, agents, and tasks✓✓✓✓
Execution of agents using their native control loops✓✓✓✓
Tool, skill, and memory interventions within native harnesses⍻⍻⍻✓
Semantic analysis of reasoning processes×××✓
Token-level uncertainty analysis×××✓

✓ = provided, ⍻ = partially supported, × = requires additional implementation.

(1) Component interventions preserve the native execution workflow. AlphaDiana provides configurations for changing tool exposure, loading skill libraries, and controlling memory retention while retaining each harness's own model calls, tool execution, and feedback handling. Memory settings specify whether memory is cleared after each attempt, retained across attempts on the same task, or accumulated across tasks. Other platforms cover parts of this functionality. Harbor supports native skill loading and tool restrictions, while Inspect provides configurable tools, skills, memory, and external-agent interfaces. AlphaDiana organizes these interventions into component experiments that can be run and compared across the evaluated harnesses.

(2) Semantic analysis of reasoning processes. AlphaDiana labels traces from different harnesses using a shared set of functional steps, then analyzes their distribution and transitions to explain behavioral differences between correct and incorrect executions. For example, we examine whether verification is followed by further reasoning or revision, and how these patterns differ between correct and incorrect attempts. This connects score differences to the execution processes behind them.

(3) Token-level uncertainty analysis. AlphaDiana measures token entropy around tool interactions and compares these patterns between correct and incorrect attempts. This supports the paper's observation that low-entropy generation can accompany incorrect answers and helps characterize changes in generation uncertainty following tool responses.

Conducting these studies in HAL, Harbor, or Inspect requires adding the corresponding component configurations, functional-step labeling, and token-entropy analysis procedures. AlphaDiana implements and connects these experimental controls and analyses, linking harness configuration changes to execution behavior and task outcomes.

AuthorsOpenReview ↗

Response to Reviewer j5iK (part 2 / 4)

W2. The empirical contribution of the findings

Many of the ten "Findings" are mathematically obvious or definitional rather than new empirical insights. For example:

Finding 9 basically states that cost per success = (tokens per attempt) / accuracy, which is true by definition. Finding 6 is just that Avg@4 is not equal to Pass@4. Finding 3 is simply the difference between macro-averaging (per-trajectory) and micro-averaging (pooled) when trajectory lengths vary.

Reply: The findings show the behavior of actual model-harness systems and how that behavior affects assessments of component utility and resource efficiency. The mathematical relationships between the metrics explain why these outcomes are possible, while our experiments establish the configurations in which they occur, their magnitude, and the corresponding execution processes.

(1) Finding 3 shows that apparently frequent tool use is concentrated in a single unusually long execution. Among incorrect Gemma/ZeroClaw trajectories on HLE-Verifiable, the pooled Tool Use share is 61.0%, but all tool-use steps come from one trajectory. The mean per-trajectory share is only 1.3%. This finding identifies the behavior behind the aggregate statistic: one unusually long execution dominates the tool-use proportion, making localized activity appear typical of the configuration. A high pooled tool-use share therefore does not mean that the model relies on tools across most tasks.

(2) Finding 6 shows that removing tools increases correct attempts while narrowing task coverage. The definitions of Avg@4 and Pass@4 do not determine how either metric changes after a tool intervention. We observe more correct attempts but fewer distinct tasks solved after tool removal, showing that the increase in successful attempts is concentrated on a smaller set of tasks. Our new SWE Mini experiments also show different effects on these two dimensions: loading Math skills lowers Qwen/ZeroClaw average accuracy while task coverage remains unchanged. These experiments establish how specific component settings affect success frequency and task coverage, distinguishing different aspects of component utility.

(3) Finding 9 shows that a configuration using fewer tokens per attempt can cost more per correct answer. The empirical question is whether a configuration's output savings offset its accuracy loss. On HLE-Verifiable, Gemma/OpenCode uses 0.6% fewer output tokens per attempt than Direct, but 16.5% more per correct answer. The experiment identifies an actual reversal in the efficiency ranking: the configuration that appears cheaper per attempt is more expensive per correct answer. The cost formula explains this result, while the experiment identifies the configuration and measures the change in efficiency.

These findings show how aggregate statistics can be dominated by an unusual execution, how component interventions change success frequency and task coverage, and how accuracy changes affect efficiency assessments. Through controlled experiments and trace analysis, the paper establishes the specific conditions under which these effects occur, their magnitude, and their implications for agent evaluation.

AuthorsOpenReview ↗

Response to Reviewer j5iK (part 3 / 4)

W3. Reliability of error and functional-step labels

Heavy reliance on LLM-as-a-judge for trace labeling. Error codes and functional steps are labeled by GPT-5.6-sol, with human verification on only 42 trajectories.

Reply: We conducted separate human audits of error-code and functional-step annotations, both of which show high agreement with human judgments. The error-label audit covers 42 full execution trajectories, including multi-turn reasoning and tool interactions, with each trajectory assessed against 17 error categories to produce 714 trajectory-label decisions, made independently by each of two human annotators. The functional-step audit separately covers 120 text segments.

(1) Error labels are validated against judgments from two human annotators. Both annotators applied the same error taxonomy to 42 screened trajectories. The audit examines full execution trajectories, including multi-turn reasoning, tool calls, and tool responses. Each trajectory is assessed against 17 error categories, yielding 714 trajectory-label decisions per annotator comparison. Model-human agreement reaches 88.0%, showing high consistency between the error labels and human judgments.

(2) Functional-step labels are validated through an independent segment-level audit. The audit assigned 120 text segments to three annotators under separate conditions with model labels hidden or model suggestions shown. With model labels hidden and annotators judging independently, model-human pairwise agreement reaches 76.0%, also showing high consistency with human judgments.

These two audits provide independent human validation for error labels and functional-step labels separately. Sec. 3 and Appendices C.2.1 and C.3.1 describe the samples, procedures, and results of the two audits.

Q2. Repeated Direct sampling versus the four-system union

For Finding 1 on GPQA-Diamond, what is the union accuracy of 4 independent samples from the Direct baseline alone (Direct Pass@4) compared to the 4-harness union (175/198)?

Reply: We conducted three additional Direct evaluations with Qwen3.5-27B on the same 198 GPQA-Diamond questions, bringing the total to four attempts per question. Four Direct attempts achieve higher overall coverage, but the four-system union still solves some questions that Direct misses in all four attempts.

StrategyAttempts per questionQuestions solvedCoverage
Four independent Direct attempts4183/19892.42%
Direct, OpenClaw, ZeroClaw, and OpenCode once each4175/19888.38%

Comparing the solved sets question by question, both strategies solve 172 questions, and 11 are solved only by four Direct attempts. Another three questions are solved only by the four-system union, with Direct failing on all four attempts for each. Of these three, two are solved by OpenClaw and OpenCode, and the third by ZeroClaw and OpenCode.

Thus, four Direct attempts provide higher overall coverage, while the harnesses still contribute unique solved questions. This supports Finding 1's observation of complementary solved sets and extends the comparison to multiple Direct attempts.

Reviewer_g21fOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces AlphaDiana, an extensible infrastructure layer for systematic model harness evaluation. It covers an empirical study of different model-harness combinations on verifiable reasoning tasks, specifically, 3 open models and 3 open harnesses across 7 benchmarks (two interactive benchmarks use only two of the models). The core contribution lies in the controlled setting in which experiments are conducted, through session isolation, preservation of harness-native execution, and linking execution evidence to results. The paper makes several findings, one of them being that harnesses can solve questions the model misses but still lower accuracy, and an ablation study shows that tool removal can raise average accuracy while solving fewer problems.

Critical strengths
  • The main strength of the paper is the set of specific controls the infrastructure offers. Memory management between shared runs, prevention of leakage between attempts, and file resets are all sound features for harness evaluation. These features address a real source of confounding in agent evaluation, independently of the specific findings. The benefits follows directly from the design, since uncontrolled shared state can carry solutions between attempts.
Critical weaknesses

W1. The claims lack statistical support.

  • Most benchmarks are evaluated with a single sample per question (Avg@1); only AIME 26 uses four samples. The main text reports no confidence intervals or repeated runs for the main comparisons (Fig. 5).
  • Several findings rest on small differences. Finding 6’s contrast between average accuracy and problems solved comes down to 29 vs. 30 AIME problem. Without uncertainty estimates, it is not possible to tell which differences reflect harness effects and which reflect per-question sampling variance. Several findings may therefore not hold up under repeated runs.

W2. The findings are limited in scope and say little specifically about harnesses.

  • Majority of findings rely only on short-horizon, answer-based benchmarks. The two interactive benchmarks, where harnesses are designed to help, report only end-to-end scores for two models.
  • AIME 26 is near ceiling, so Pass@4 separates systems poorly.
  • Several findings are general evaluation cautions, for example about pooled statistics, aggregation, and token accounting, rather than observations specific to harnesses. The paper offers limited insight into how harnesses behave on the long-horizon tasks where harnesses are expected to matter most. The conclusions about tools, skills, and memory may not carry over beyond short-horizon reasoning tasks.
Questions for authors
  1. Can you report variance across seeds or repeated runs for the main results and do the findings still hold?
  2. When a harness scores below the model without a harness, can the existing traces show whether the loss comes from prompt overhead, tool distraction, or context dilution? What are the findings from those traces?
  3. Can the component interventions (tools, skills, memory) be run on SWE Mini or Terminal-Bench 2?
9 comments
AuthorsOpenReview ↗

Response to Reviewer g21f (part 1 / 3)

Thank you for the thoughtful review and constructive feedback. We address your concerns and questions below.

W1 & Q1. Stability of the main comparisons

W1. The claims lack statistical support.

Most benchmarks are evaluated with a single sample per question (Avg@1); only AIME 26 uses four samples. The main text reports no confidence intervals or repeated runs for the main comparisons (Fig. 5). Several findings rest on small differences. Finding 6’s contrast between average accuracy and problems solved comes down to 29 vs. 30 AIME problem. Without uncertainty estimates, it is not possible to tell which differences reflect harness effects and which reflect per-question sampling variance. Several findings may therefore not hold up under repeated runs.

Can you report variance across seeds or repeated runs for the main results and do the findings still hold?

Reply: We conducted additional experiments on GPQA-Diamond and SWE Mini with Qwen3.5-27B and Gemma-4-31B-IT, bringing the total to four runs per task for each model-system combination. These results are consistent with our finding that harness effects vary across model-harness combinations.

Each group combines the original attempt with three additional attempts on the same tasks. Avg@4 measures average correctness across all four attempts, while Pass@4 measures the fraction of tasks solved at least once. SD is the sample standard deviation of the four round-level accuracies, in percentage points. Retries keep the original sample identifier, and unrecovered failures count as incorrect. The additional runs use local serving and APIs with the recorded run configurations.

(1) Qwen's GPQA degradation persists across all four rounds. All three harnesses lower Qwen's accuracy relative to Direct in every round, whereas Gemma shows small average gains. These results support our observation that harness effects depend on the model. The results are shown below.

ModelHarnessAvg@4 ±\pm SD (%)Pass@4 (%)
Qwen3.5-27BDirect84.72 ±\pm 3.5392.42
Qwen3.5-27BOpenClaw73.99 ±\pm 5.3890.40
Qwen3.5-27BZeroClaw76.01 ±\pm 1.5287.88
Qwen3.5-27BOpenCode77.27 ±\pm 3.1987.88
Gemma-4-31B-ITDirect85.48 ±\pm 2.3590.91
Gemma-4-31B-ITOpenClaw85.86 ±\pm 1.0193.94
Gemma-4-31B-ITZeroClaw86.62 ±\pm 1.3492.93
Gemma-4-31B-ITOpenCode87.75 ±\pm 0.9693.43

(2) SWE Mini results also show differences across model-harness combinations. ZeroClaw and OpenCode achieve higher average accuracy than SWE-Agent with Qwen, but lower average accuracy with Gemma. The results are shown below.

ModelHarnessAvg@4 ±\pm SD (%)Pass@4 (%)
Qwen3.5-27BSWE-Agent52.00 ±\pm 3.6566.00
Qwen3.5-27BOpenClaw45.00 ±\pm 1.1566.00
Qwen3.5-27BZeroClaw55.00 ±\pm 5.0366.00
Qwen3.5-27BOpenCode53.50 ±\pm 5.2666.00
Gemma-4-31B-ITSWE-Agent57.50 ±\pm 7.7266.00
Gemma-4-31B-ITOpenClaw42.50 ±\pm 1.0064.00
Gemma-4-31B-ITZeroClaw35.00 ±\pm 6.2260.00
Gemma-4-31B-ITOpenCode52.00 ±\pm 10.9562.00

These additional experiments support our observation that harness effects vary across model-harness combinations. We have updated Sec. 3 and Fig. 5 with Avg@4 and Pass@4 and retained all seven benchmarks in the Macro figure.

AuthorsOpenReview ↗

Response to Reviewer g21f (part 2 / 3)

W2 & Q3. Component effects on interactive tasks

W2. The findings are limited in scope and say little specifically about harnesses.

Majority of findings rely only on short-horizon, answer-based benchmarks. The two interactive benchmarks, where harnesses are designed to help, report only end-to-end scores for two models. AIME 26 is near ceiling, so Pass@4 separates systems poorly. Several findings are general evaluation cautions, for example about pooled statistics, aggregation, and token accounting, rather than observations specific to harnesses. The paper offers limited insight into how harnesses behave on the long-horizon tasks where harnesses are expected to matter most. The conclusions about tools, skills, and memory may not carry over beyond short-horizon reasoning tasks.

Can the component interventions (tools, skills, memory) be run on SWE Mini or Terminal-Bench 2?

Reply: We conducted additional experiments with Qwen3.5-27B on 50 SWE Mini tasks, using four runs per task for each setting in ZeroClaw and OpenCode. We focus on skills because code repair relies on tools and multi-turn history. We compare Default, Math skill, and General skill, retaining their native tools and history handling. The results show that skills can help models identify and correct incomplete fixes, but models may also select skills that do not fit the task. Accuracy gains vary across harnesses and skills.

The results are shown below as Avg@4 / Pass@4 (%).

HarnessDefaultMath skillGeneral skill
ZeroClaw57.0 / 66.052.5 / 66.054.5 / 66.0
OpenCode57.0 / 66.056.5 / 68.058.0 / 68.0

Skill loading lowers ZeroClaw’s Avg@4 while leaving Pass@4 unchanged. For OpenCode, Math skill slightly lowers Avg@4 and General skill slightly raises it, while both increase Pass@4. Examining the execution traces reveals the following findings.

(1) Models can apply verification methods from skills to code repair tasks, helping them identify and correct incomplete fixes.

  • In a website navigation-link repair task, the skill read by the model instructed it to verify results through an alternative route. The model then built the website and inspected the generated HTML, discovered a missing link, and completed the repair before passing evaluation.
  • In a function-parameter display task, the skill instructed the model to check limiting cases. The model then checked empty and single-element tuples, discovered that its initial fix omitted the trailing comma for a single-element tuple, and corrected the patch before passing evaluation.

(2) Models can also select skills that do not fit the task. When repairing a C++ parser in a documentation-generation tool, ZeroClaw selected a document-coauthoring skill. Its instructions concerned organizing documents and collecting reader feedback, rather than modifying parsing logic. This example illustrates the need to select skills based on the operation the task requires.

AuthorsOpenReview ↗

Response to Reviewer g21f (part 3 / 3)

Q2. What the traces reveal about accuracy losses

When a harness scores below the model without a harness, can the existing traces show whether the loss comes from prompt overhead, tool distraction, or context dilution? What are the findings from those traces?

Reply: The traces can help examine whether accuracy losses stem from prompt overhead, tool distraction, or context dilution. Drawing on the error analysis in Fig. 6 and Appendix C.2, we find that more tool calls and verification steps do not necessarily improve reasoning, and performance depends on whether and how effectively the model uses tool responses. We examine these three aspects below through individual execution traces.

(1) Prompt-related execution overhead. Harness prompts encourage models to use tools, verify intermediate steps, and check their final answers. These instructions introduce additional operations as well as additional text. In our previous tool-removal comparison, some tool-enabled ZeroClaw attempts reached the budget limit without submitting an answer, whereas this failure did not occur with tools removed. This shows that additional execution procedures can exhaust the budget before an answer is completed.

(2) Tool distraction. In a MMMU-Pro music question, Kimi/OpenCode is asked to select the correct rewriting of a bar from compound to simple time. It makes three web requests for music-theory material. The requests return an aborted operation, a page whose lesson requires JavaScript, and a certificate error. After these unproductive retrievals, the model ends with “How can I help you today?” instead of selecting an answer. The trajectory shows the model abandoning the requested task after tool use, leaving the question unanswered.

(3) Use of prior context. Our labels for observation integration and decision-action inconsistency identify whether models act on information already available in the trajectory. In a molecular-chirality question, a tool explicitly reports that RDKit is not installed. Kimi/OpenCode nevertheless runs code requiring RDKit in its next action and encounters the same failure. The model receives relevant feedback but fails to use it when selecting its next action.

Together, these traces show that more tool calls and verification steps do not necessarily lead to better reasoning. Their value depends on whether and how effectively the model uses tool responses to revise its assumptions, adjust its actions, and complete the task.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A General Summary by the Authors

Dear Area Chairs and Reviewers,

Thank you for your helpful feedback. Reviewer j5iK raised the rating from 2 to 3 (weak acceptance), and Reviewer g21f from 1 to 2 (weak rejection). Reviewer Pe8a remains at 2 without a reply. Reviewer g21f's technical questions were resolved, and Reviewer j5iK reported a better understanding. We summarize below.

(1) Overview of our work

We propose AlphaDiana, a unified system for harness-aware evaluation of open agents on verifiable reasoning tasks. AlphaDiana standardizes evaluation interfaces, execution protocols, and trajectory logging, enabling controlled comparisons and diagnosis of model-harness systems. Our key contributions are:

  • Controlled evaluation and interventions. AlphaDiana preserves native control loops while separating session isolation, file resets, and memory retention. It compares complete agents and varies tools, skills, and memory under explicit reuse conditions.
  • Analysis of execution processes. A shared taxonomy compares reasoning, verification, revision, and their transitions across correct and incorrect attempts. Token-entropy measurements following tool responses examine how generation uncertainty relates to correctness.
  • Empirical findings. Harnesses can solve questions missed by Direct while lowering average accuracy. Component effects vary across configurations, and low entropy can accompany incorrect answers. Fewer tokens per attempt can also mean more tokens per correct answer.

(2) Strengths recognized by the reviewers

  • Controls against information leakage. Reviewers g21f and j5iK recognize session isolation, file reuse, and memory policies as evaluation controls. Reviewer g21f states that these address "a real source of confounding in agent evaluation."
  • Comparisons within native harnesses. Reviewer j5iK values preserving native control loops while holding the model, decoding parameters, and budget fixed for component interventions. Reviewer Pe8a recognizes the resulting fairer and more reproducible comparisons.
  • Diagnosis beyond final accuracy. Reviewer j5iK highlights distinguishing wrong answers, execution failures, and unscored attempts. Reviewer Pe8a values linking answers, trajectories, and resource use to identify specific failure patterns.

(3) Rebuttal responses and additional evidence

  • Macro evaluations and task coverage. We brought GPQA-Diamond and SWE Mini to four attempts per task for Qwen3.5-27B and Gemma-4-31B-IT, with four systems per benchmark, reporting Avg@4, Pass@4, and round-level variation. All three harnesses remain below Direct in every Qwen/GPQA round, while Gemma shows small average gains. On SWE Mini, ZeroClaw and OpenCode exceed SWE-Agent with Qwen but fall below it with Gemma in average accuracy. Four Direct attempts on Qwen/GPQA cover more questions than the union of one attempt per system, but miss three it solves. (g21f W1/Q1, j5iK Q2, Pe8a Q4)

  • Interactive components and execution behavior. We evaluated Default, Math skill, and General skill on SWE Mini with Qwen in ZeroClaw and OpenCode. Both libraries lower ZeroClaw's average accuracy without expanding coverage, but expand OpenCode's coverage. Traces show verification guidance used for boundary checks and repairs, alongside irrelevant skill selection. Other traces show budget exhaustion, abandonment after failed retrievals, and failure to use tool responses despite additional operations. (g21f W2/Q2/Q3)

  • Trace reliability and token costs. Separate human audits yield 88.0% model-human agreement for error labels and 76.0% for functional-step labels with machine labels hidden. Error-label agreement is high across systems. Format, argument, or tool-name errors affect only 0.28% of recorded tool calls. On a shared Gemma/HLE-Verifiable task set, OpenCode uses fewer output tokens per attempt but more per correct answer than Direct. (j5iK W2/W3/Q3, Pe8a W2/W3/Q3/Q5)

  • Intervention design and framework contribution. We compared AlphaDiana's interventions and trace analyses with HAL, Harbor, and Inspect. After tool-prompt edits, calls are rare in inspected ZeroClaw traces, and no structured tool calls appear in captured OpenCode responses. Inspected default Qwen/Kimi AIME traces in both harnesses show no active memory use, motivating explicit memory instructions. (j5iK W1/Q1, Pe8a W1/Q1/Q2, g21f follow-up)

(4) Reviewer feedback

  • Reviewer j5iK: "I have a better understanding of the paper now, and I think it's above average."
  • Reviewer g21f (Informativeness increased from 1 to 2): The additional experiments "successfully resolved my technical questions," while the reviewer retains a reservation about overall impact.
  • Reviewer Pe8a: No reply to our rebuttal and follow-ups yet.

We hope this summary and our rebuttals clarify the paper's contributions and findings and assist your evaluation.

Thank you again for your time and consideration.

Best regards,
Authors of #35797

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.