acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Verification Design in Agent Harnesses

Abstract

Agent harnesses enable large language models to act through tools, observe the results, and iteratively work toward a task goal. Verification checks and revises agents' work to improve output quality. Yet there is limited guidance on how harnesses should organize this effort. We study verification as an axis of test-time scaling by examining when to intervene, who checks the work, and how checking leads to revision. Our framework spans four layers of the agent lifecycle: initial instructions, mid-turn reminders, completion gates, and independent review. We compare 17 designs across seven frontier coding models on Terminal-Bench (TB) and SWE-bench Verified (SWE), and introduce VisualBench for expert evaluation of visual and interactive artifacts. Initial instructions and execution-time reminders yield modest, model-dependent average gains on the coding benchmarks, without substantially increasing average output token use. Completion gates can yield larger gains but may substantially increase cost. Independent review by a critic offers a more favorable accuracy–cost tradeoff, improving average accuracy by – percentage points on TB and – on SWE at – and – baseline cost, respectively. On VisualBench, independent review achieves net preferences of – points over the baseline across its four designs, exceeding the evaluated L0–L2 designs. Additional review rounds or independent critics further improve average TB accuracy by up to percentage points. Together, these results provide empirical guidance for verification design in agent harnesses and support independent review as a practical axis of test-time scaling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Harnesses, Verification, Test-Time Scaling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JoBROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper divides verification in the agent harness into four layers, initial instructions, reminders during execution, a gate at completion, and independent review, and compares 17 designs. The experiments use seven programming models, report accuracy changes and relative cost on task subsets of Terminal-Bench and SWE-bench Verified, and also include an internal VisualBench of 18 tasks. The average gain from independent review is about 5.9 to 9.5 points on TB and 3.9 to 6.6 points on SWE, at a cost of about 1.00 to 1.25 times the baseline.

Critical strengths

Tables 2 and 3 list accuracy changes for 17 designs and seven models on two programming benchmarks, and negative cells are kept in the tables, for example Kimi on the L0 user setting is −8.3. Cost is priced separately by model and by reviewer, with the reject-style gate at L2 reaching about 1.5 to 1.8 times the baseline and L3 at about 1.00 to 1.25 times. L0 and L1 also have cells where accuracy rises and output tokens do not rise. Costs across the four layers and per-model results can be read directly from the tables.

Critical weaknesses

The tasks used to recommend harness designs were kept after looking at these models' baselines. Design recommendations need to be based on tasks fixed before the runs started. The 31 Terminal-Bench tasks come from an 89-task registry, and the difficulty band used the pilot pass rates of four models in the table. SWE first selected a batch of tasks by repair time, then adjusted them after the baselines were seen, so as to keep both tasks with room to improve and tasks already solved that are used to observe regressions. VisualBench is another 18 internal tasks. The positive averages on the main table are computed on this set of tasks whose baseline behavior had already been seen.

The reviewer is fixed as Qwen3.7-Plus, and the paper does not manually check whether the review comments are reliable. All seven programming models are reviewed by this one model, with new context, sandbox checks, and one rework. The experiments do not include a control that gives the original agent the same extra tokens and the same one rework without introducing a second model, and they do not replace the reviewer model. The human evaluation in the paper evaluates finished VisualBench outputs and does not check whether the review comments point out real defects, whether later edits adopt them, or how this relates to whether the outcome finally passes. The same L3 design is negative on some models, for example on SWE the result for claim-guided changes is −1.1 for Qwen3.7-Max and −3.4 for DeepSeek V4-Flash, and the result for task-first trajectory is −3.4 for GLM-5.2. Coloring is a secondary issue. Figure 2 uses four highly saturated colors to distinguish the four layers and overlays seventeen numbers, and the legend of Figure 3 is blue, orange, green, red, purple, brown, pink, and gray in that order. When reading relative positions, color draws the eye more than the coordinates, and this point does not by itself determine the score.

Questions for authors
  1. Were the TB task list and the SWE task list used in the report frozen before any L0 to L3 run? Did the pilot pass rates of the four models, and the task adjustment after seeing the SWE baselines, take part in deciding which tasks were kept?
  2. Is there a control that gives the original agent the same extra tokens and the same one rework without introducing a second model? Was the reviewer changed from Qwen3.7-Plus to another model?
  3. Please provide a manual check of reviewer reliability. It needs to state whether the review comments point out real defects, whether later edits adopt them, and how this relates to whether the final tests pass. The sample size and the checking method need to be enough to support the average L3 gain. If the paper did not include this check before, please add it now and write the corresponding content into the paper.
Reviewer_DFdCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an empirical study of verification strategies in agent harnesses, investigating when verification should be triggered, who should perform it, and how verification feedback should guide subsequent revisions. The authors organize verification into four layers: initial instructions (L0), execution-time reminders (L1), completion gates (L2), and independent review (L3). The paper evaluates 17 verification configurations across seven coding models on selected subsets of Terminal-Bench and SWE-bench Verified, and introduces VisualBench for evaluating visual and interactive artifacts. The results suggest that initial instructions and reminders provide modest, model-dependent improvements, while completion gates can incur substantial additional costs. Independent review generally achieves a more favorable accuracy–cost tradeoff. The authors further investigate scaling verification through additional review rounds and independent critics. The study aims to provide empirical guidance for verification design in agent harnesses and to establish independent review as a practical approach to test-time scaling.

Critical strengths

S1. Systematic Empirical Comparison of Verification Strategies.

The paper provides a structured investigation of verification across different stages of the agent lifecycle. By comparing 17 configurations across seven coding models and multiple benchmarks, it offers useful empirical evidence on how verification timing, critic context, and review strategies relate to agent performance. This is particularly valuable given the growing importance of harness design in practical coding agents.

S2. Cost-Aware Evaluation and Analysis of Independent Review.

The paper evaluates both task success rates and computational costs, rather than considering accuracy improvements in isolation. The experiments on independent review, including comparisons of critic context and review depth versus width, provide useful insights into allocating test-time computation. The inclusion of VisualBench also extends the evaluation beyond conventional coding benchmarks.

S3. Behavioral Analysis and Illustrative Case Studies.

The paper complements its quantitative results with behavioral diagnostics and detailed execution traces, illustrating how different verification interventions influence agent behavior. The inclusion of both successful and unsuccessful cases provides useful insights into when verification helps identify overlooked errors and when additional checking fails to improve the final outcome. These analyses help contextualize the reported performance differences and provide a more concrete understanding of verification in practice.

Critical weaknesses

W1. Limited Representativeness of Benchmark Subsets and Potential Selection Bias.

The paper evaluates verification strategies on selected subsets of Terminal-Bench and SWE-bench Verified. In particular, Terminal-Bench uses 31 tasks selected from an 89-task registry, with selection guided by baseline success rates. While this approach covers different difficulty levels, it remains unclear whether the selected tasks adequately represent the original benchmark distributions. Moreover, baseline task difficulty does not necessarily reflect verification difficulty, and selecting tasks based on baseline performance may influence the available headroom for improvement. Without stratified analyses across task types, difficulty levels, and verification characteristics, the generalizability of the reported conclusions remains uncertain.

W2. Insufficient Validation of Prompt-Controlled Critic Strategies.

The paper compares Task-first and Claim-guided critic strategies, primarily implemented through different prompting instructions. However, prompting a critic to follow a particular strategy does not guarantee that the intended verification behavior is actually performed. Although the paper provides illustrative execution traces, it lacks systematic evidence showing whether Task-first critics independently derive and execute requirement-driven checks, or whether Claim-guided critics consistently identify and falsify agent assumptions. It is also unclear whether these strategies produce meaningful differences in requirement coverage, test generation, or failure detection. Consequently, the observed performance differences may reflect prompt-specific effects rather than the intended verification strategies.

W3. Insufficient Analysis of Model- and Task-Dependent Verification Effectiveness.

The paper reports substantial variation in the effectiveness of verification strategies across models and benchmarks. Some configurations yield significant improvements for certain models while providing limited benefits or even degrading performance for others. Although these differences are reported, the paper provides limited systematic analysis of the underlying factors that determine when particular verification strategies are effective.

In particular, it remains unclear how verification effectiveness depends on model capabilities, task characteristics, or the types of errors that need to be detected and corrected. Since the paper aims to provide practical guidance for harness verification design, identifying the conditions under which different strategies succeed or fail is important. More detailed analyses across model capabilities, task categories, and verification failure modes would help establish more actionable and generalizable design recommendations.

W4. Limited Exploration of Critic Model, Triggering, and Feedback Design.

Although the paper systematically compares verification strategies across different stages of the agent lifecycle, the design space explored for independent critics remains relatively limited. In particular, all coding experiments use a fixed critic model (Qwen3.7-Plus), while independent review is primarily triggered after the main agent stops and follows a fixed feedback-and-revision protocol. The study mainly varies critic prompting strategies, available context, and review depth or width, without systematically examining how critic model choice, triggering conditions, or feedback mechanisms affect verification effectiveness. These factors may substantially influence both task success and verification cost. For example, different critic models may exhibit different error-detection capabilities, earlier or adaptive triggering may identify problems before they propagate, and alternative feedback formats may affect the main agent's ability to repair identified defects. Without investigating these dimensions, it remains unclear how broadly the reported findings and recommended independent-review configurations generalize to other harness designs.

Questions for authors

Q1. Benchmark Representativeness.

How representative are the selected Terminal-Bench and SWE-bench Verified subsets of the full benchmarks? Can the authors provide stratified results or sensitivity analyses showing that the main conclusions are robust to the task-selection procedure, particularly across different task types and verification difficulties?

Q2. Critic Strategy Adherence.

Have the authors systematically analyzed critic execution trajectories to verify that Task-first and Claim-guided critics actually follow their intended strategies? Do these configurations exhibit measurable differences in test generation, tool execution, requirement coverage, or counterexample discovery?

Q3. Model- and Task-Dependent Effectiveness.

What factors explain the substantial variation in verification effectiveness across models and benchmarks? Can the authors analyze how different verification strategies perform across task categories, model capabilities, or failure types, and identify the conditions under which particular strategies are most effective?

Q4. Critic Design Choices and Generalizability.

How sensitive are the findings on independent review to the choice of critic model, triggering conditions, and feedback mechanisms? In particular, would the conclusions remain consistent when using different critic models or alternative review triggers and feedback formats?

Q5. Baseline Consistency and Experimental Comparability.

In Appendix B, some baseline results appear to differ across verification configurations, particularly for Claim-guided review (Changes). Could the authors clarify the reasons for these differences and explain how comparability across configurations is ensured?

Reviewer_fw5oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares four points where an agent harness can prompt or enforce verification: initial instructions, reminders during execution, completion gates, and a separate critic before acceptance. It evaluates 17 designs across seven coding models on selected Terminal-Bench and SWE-bench Verified tasks, with three trials per model-task pair, and adds an 18-task visual benchmark. The main result is that the tested independent-review setup improves coding accuracy by several points at relatively modest added cost.

Critical strengths
  • This is a real engineering problem, but it is not untouched: OpenAI reported that, as of October 2025, its deployed agentic reviewer handled more than 100,000 external pull requests a day; Claude Code also exposes lifecycle hooks, including at stopping. Verification is already part of frontier-agent practice. These deployments show adoption, not that the design question is solved: public comparative evidence about which approach works best across models, tasks, and budgets remains limited. That is the paper’s useful contribution, rather than introducing verification itself.
  • The four-layer framing is intuitive, and the paper compares a meaningful range of models and verification designs while reporting cost alongside accuracy. The behavioral analyses and failure examples help make the findings useful to harness designers.
Critical weaknesses
  • The headline gains are hard to distinguish from noise. The coding experiments use small, outcome-selected task subsets and only three trials per model-task pair, yet the paper reports no paired confidence intervals or significance analysis. All designs also share a baseline, so a favorable baseline run shifts many comparisons together. VisualBench adds only 18 tasks, and the paper does not report rater agreement or uncertainty. The evidence supports promising effects, but not stable rankings among designs.
  • The strongest claim—that independent review is the best accuracy-cost tradeoff—does not isolate what causes the gain. Every agent is checked by the same separate Qwen critic, which has a fresh context and can use the live sandbox; a critique also forces additional work. The study does not compare this with a matched-cost extra coding attempt, same-model review, or an uninformative continuation prompt. It therefore establishes that this particular critic pipeline can help, not that independence itself is the decisive or most efficient mechanism.
Questions for authors
  1. Problem relevance: Given deployed code review and lifecycle hooks in current frontier products, what important verification failure remains unresolved? Is the paper’s intended contribution the controlled comparison of harness designs, or does it claim a gap in current systems that could be documented with real workflow evidence?
  2. Evaluation request: Were final benchmark tasks independent of the selection pilots, and can you report paired, task-clustered uncertainty plus a matched-cost comparison with an extra coding rollout or same-model critic? This would clarify whether the observed L3 advantage is robust and specific to independent review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.