acceptodds
Under review as a conference paper at ICLR 2027

VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks

Abstract

As LLM agents undertake increasingly complex, long-horizon tasks, verifying their outputs becomes increasingly challenging. We study how verification capability can be strengthened with a fixed base model, without access to reference answers or grading rubrics at test time. Repeated sampling yields multiple rollouts that can contain complementary correct claims, but we need a reliable verification mechanism to determine which claims to trust. We first find that disagreement often exposes correct alternatives, while consensus can conceal errors. These observations motivate VeriHarness, which turns the underlying LLM a generator uses into an agentic verifier by giving it a workspace, evidence tools, and reusable verification skills. A disagreement resolver checks competing claims against environmental evidence, while a consensus challenger tests shared claims and searches for omitted requirements. Their findings guide the selection and revision of the final artifact. Across five long-horizon workspace benchmarks and two frontier models, VeriHarness achieves the highest selection scores among the evaluated baselines. Evidence-backed revision further improves average performance, bringing gains over a single rollout to 6.2 points with Gemini 3.5 Flash and 6.4 points with Claude Opus 4.8. We further show that verification skills can self-improve from failure feedback, demonstrating VeriHarness as a novel and critical approach for scaling long-horizon agentic verification.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agents, verification, harness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_v6YROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies same-model verification of long-horizon agent outputs such as reports, spreadsheets, and patches. Given a pool of N=10 rollouts, the verifier (the same model as the generator) has to select or revise the final artifact without access to reference answers or grading rubrics. A claim-level analysis on APEX-Agents motivates the method: 34% of the claims that all rollouts agree on are wrong, while 74% of disputed claims contain a correct candidate even though the most frequent answer is correct only 47% of the time (Fig. 1). VeriHarness gives the verifier a read-only workspace with all rollouts and the task environment, generic file and shell tools, and a library of 18 human-authored verification skills. A disagreement resolver checks disputed claims against evidence, a consensus challenger tries to refute agreed claims, and an adjudicator selects a base rollout and optionally revises it (Sec. 3, Alg. 1).

The method is evaluated on five workspace benchmarks (APEX-Agents, Workspace-Bench Lite, WorkBuddy Bench, SpreadsheetBench 2, JobBench) with Gemini 3.5 Flash and Claude Opus 4.8 as both generator and verifier. In selection mode it beats majority voting, best-of-N, a pairwise tournament, LLM-as-a-Verifier, and a tool-using agentic verifier in all ten settings, improving over a single rollout by 4.4 (Flash) and 4.1 (Opus) points; with revision, the gains are 6.2 and 6.4 points (Table 1). The paper also reports ablations (Table 4), a cost analysis (App. J), and a self-evolution study in which skills learned from failure feedback on development tasks improve held-out performance (Sec. 5).

Critical strengths
  • The evaluation is broad and controlled. All methods share the same frozen rollout pools and the same model, the benchmarks are recent and realistic, and the selection gains hold in all ten settings (Table 1) and largely carry over when the protocol runs inside Gemini CLI, Claude Code, and Codex (App. F). The cost analysis (App. J) puts selection at $1.44/task with Flash and $3.92/task with Opus. Evidence that a training-free, same-model verifier improves selection at moderate cost is practically useful.
  • The diagnostics explain where same-model verification helps rather than only reporting scores. The selection gains are concentrated in pools that disagree (+0.6 on 154 consensus pools vs. +6.1 on 262 disputed pools, Fig. 4), and Table 7 shows that 70% of the shared errors the challenger misses come from checking an upstream quantity instead of the graded one. This was the most informative part of the paper for me.
Critical weaknesses
  • The contribution of the protocol is confounded with the skill library. Without skills, VeriHarness scores 53.8 (Opus) and 51.3 (Flash) (Table 4), while the agentic verifier + revision baseline, which has the same tools and environment access but neither the protocol nor the skills, scores 53.1 and 50.7 (Table 1). The resolver/challenger protocol on its own therefore adds about 0.6–0.7 points, which is within the reported SDs, and most of the remaining margin comes from the 18 skills that baselines do not receive. App. H states that the library was written from "development failures", Table 1 appears to use the full benchmarks including those development tasks, and some skills describe "how prose is graded" and "how office bundles are graded" (Table 6), which sits uneasily with the no-rubric setting. Without a skills-matched baseline and a clean task split, I cannot tell whether the protocol or prior exposure to the benchmarks drives the headline numbers.
  • The target of the paper is not clear, and the positioning against other agentic verifiers is weak. From the abstract and introduction, it is not clear whether VeriHarness is a test-time compute method (selecting and revising among rollouts at deployment) or a verifier meant to produce a training signal such as RL rewards. The experiments are entirely test-time selection and revision, and App. K says the harness assigns no score to individual rollouts, but the title, the related work (process reward models, agentic verifiers trained with RL), and the conclusion (verification records as training data) point toward training. The two settings call for different related work and baselines: test-time scaling methods at matched compute in the first case, and reward models and agentic graders in the second. Relatedly, the paper claims to be "the first agentic verification harness for long-horizon tasks" (Sec. 1), but agentic verifiers that inspect files and tool state in the task environment already exist, for example Agent-as-a-Judge (Zhuge et al., 2025), which is only cited in passing, and Gandalf (Handshake AI Research, 2026, https://joinhandshake.com/research/ai/gandalf-the-grader/), which runs the verifier agent in the same environment as the rollout agent. None of these are discussed or used as baselines. As far as I can tell, the main differences are that VeriHarness works without rubrics and uses disagreement across rollouts to decide what to check, and this should be made explicit.

Other, less critical points:

  • The uncertainty estimates are limited. The SDs come from three verifier runs on one frozen pool, so they do not capture task or pool sampling, and no paired task-level test is reported. Several "beats every baseline" cells are within noise (e.g., Flash SB-2: 31.5 ± 2.4 vs. 30.9 ± 1.1 for the agentic verifier), JobBench has only 65 tasks, Table 4 has no SDs, and the self-evolution results in Fig. 3 are single runs.
  • Revision delivers "both readings" of unresolved interpretations when the format allows (App. C). With rubric-based LLM graders, this could earn credit by hedging rather than by being correct.
  • The APEX-Agents task counts differ across the paper: 480 in Table 3, 434 in the Fig. 1 analysis (App. A), and 154 + 262 = 416 pools in Fig. 4.
  • nit: on line 54, "Existing test-time verification methods mostly use only the rollouts" is not grammatical; perhaps "Most existing test-time verification methods use only the rollouts".
Questions for authors
  1. (Clarification) Is VeriHarness intended as a test-time compute method or as a verifier for producing training signal (e.g., RL rewards)? Please state this early in the paper and adjust the related work and baselines accordingly. In either case, how does VeriHarness differ from existing agentic verifiers such as Agent-as-a-Judge and Gandalf, and could one of them be included as a baseline?
  2. (Clarification) How was the human-authored skill library (Table 6) built? Which benchmarks' development failures informed it, how were development tasks defined for WSB, WorkBuddy, and JobBench, and are any of those tasks in the Table 1 evaluation? If so, please report Table 1 on tasks the library authors never saw, and explain how skills that describe how deliverables are graded fit the no-rubric setting.
  3. (Suggestion) Please add a skills-matched control: the agentic verifier and agentic verifier + revision baselines given the same 18 skills, next to VeriHarness without skills, and add these baselines to the cost plot (Fig. 7). If the protocol still clearly beats a skills-equipped single-context agent at similar cost, my main concern would be resolved.
  4. (Suggestion) Please report paired, task-level uncertainty, e.g., a bootstrap over tasks of the per-task difference between VeriHarness and the strongest baseline on each benchmark, and SDs or CIs for the Table 4 ablations.
  5. (Clarification) How often does revision deliver both readings of an ambiguous requirement, and how does artifact length change after revision? The largest revision gain is Opus on APEX (41.2 to 47.5, Table 1), so this matters for interpreting the headline numbers.

My score could increase if the authors (i) clarify the target setting and position the work against existing agentic verifiers (Q1), (ii) show that the gains hold on tasks not used to write the skills (Q2), and (iii) show that the protocol beats a skills-matched baseline (Q3).

3 comments
AuthorsOpenReview ↗

Reply to Reviewer v6YR Part 1/2

Thank you for the careful and accurate review, and for singling out the diagnostics as the most informative part. You list three things that would change your assessment. We respond to Q1–Q3 first, then Q4, Q5, and the smaller points. For all details, please refer to the revised PDF file and the general response; we refer to its sections A–F below.

Q1 / W2: regarding the target setting and positioning against Agent-as-a-Judge and Gandalf

VeriHarness focuses on a test-time agent harness for verification. Every experiment is conducted at test time, and nothing is trained. Y

  • Why the paper read otherwise. The abstract said the skills "self-improve", and the conclusion ended on verification records as training data. Both are revised. The abstract now says test-time in its first sentence. The conclusion states that all experiments are at test time, with training as future work outside the paper.
  • Verification belongs in the harness. Verification is a natural component of a modern agent harness, alongside the workspace and tools, not a training-time device. Section 7 now says so with citations. The outputs could later feed training: the selected base expresses a preference among rollouts, and the base with its revision forms a preference pair. We do not do that here.
  • Difference from Agent-as-a-Judge and Gandalf. Both grade one given rollout against a given rubric or requirement list and return a score. VeriHarness sees N rollouts and the environment, takes what to check from their disagreement and consensus, and returns an artifact with its record. Table 9 in Appendix L of the revision places VeriHarness in one table with these two works and the other verification methods the reviews mention. We have removed "the first agentic verification harness for long-horizon tasks".
  • Agent-as-a-Judge as a baseline. We ran it with the authors' released code in our sandbox. For adaptation, we pass the task description in place of the requirement list it expects, since our setting provides none. It scores 48.6 / 50.9 (Flash / Opus) at $1.26 per task. VeriHarness (select) scores 51.6 / 53.7 (section B, Table 2 of the revision).

Q2 / W1: regarding how the library was built and whether Table 1 includes those tasks

No task in Table 1 was seen by the authors of the skills. Section A of the general response gives the split table. In short:

  • 20% of each benchmark was set aside as a development set before any experiment.
  • The library was written from failures on those tasks only, then frozen.
  • Tables 1 and 4 are computed on the remaining 80%.
  • Section 5 splits the evaluation tasks of two benchmarks a second time to obtain an evolve set. The evolution never sees the held-out part.
  • "Full-benchmark pools" in Appendix H meant the pools of all five benchmarks, as opposed to the two benchmarks used for evolution. It did not mean that Table 1 includes development tasks. The revision replaces the phrase.

The revision states this in Section 4.1, Table 3, and Appendix H.

On "how prose is graded" in Table 6: the phrase was ours and it misleads. These skills describe what any reader of a written answer, record, or bundle looks for: stated final values with units, the source named, fields filled. They contain no benchmark's rubric. The entries now read "what is assessed in a written answer / record / bundle". Appendix G states that no skill contains a benchmark's rubric or a task identifier.

Q3 / W1: regarding a skills-matched control

We ran the control you ask for. It is section B of the general response and Table 2 of the revision. Flash / Opus:

  • Agentic verifier + the eighteen skills: 50.2 / 51.8 in selection, 51.0 / 52.6 in revision.
  • VeriHarness: 51.6 / 53.7 in selection, 53.4 / 56.1 in revision. With the skills held fixed, the protocol adds 1.4 / 1.9 and 2.4 / 3.5 points.
  • The 0.6–0.7 you computed from Tables 1 and 4 is the protocol's value when neither side has the skills.
  • Skills given to the single-context agent add at most 0.8. The gain is in the combination, because most skills are written for a step the protocol creates.

On cost: with the same skills, the agentic verifier costs USD 1.21 per task with Flash and USD 3.22 with Opus in selection. VeriHarness (select) costs USD 1.44 and USD 3.92. Revision is about three times selection for both. So for about 20% more verification cost the harness gains 1.4–3.5 points over a skills-equipped single-context agent (Appendix J).

AuthorsOpenReview ↗

Reply to Reviewer v6YR Part 2/2

Q4 / other: regarding paired task-level uncertainty, Table 4 SDs, and single-run Figure 3

The paired task-level intervals are in section C of the general response. The main gains exclude zero. In selection, VeriHarness exceeds the agentic verifier by +2.2 [1.5, 3.2] with Flash and +2.1 [1.0, 3.0] with Opus. In revision, it exceeds agentic verifier + revision by +2.7 [1.3, 3.6] and +3.0 [1.6, 4.4]. The revision gain over selection is +1.8 [1.1, 2.5] / +2.4 [1.6, 3.2] on average and within noise in three of the ten cells (WorkBuddy with Flash; SpreadsheetBench 2 and JobBench with Opus); Section 4.2 now says so.

"Exceeds every baseline in all ten settings" is now a point-estimate statement followed by the intervals. The SD column is described as verifier-run variability on a frozen pool. Table 4 now carries the cross-seed SD of its Avg column; every ablation gap it discusses is at least twice that spread.

Figure 3: a further split means re-evolving conditions C and D, which does not fit in the discussion period. The study is meant to show that the harness can improve its own skills from failure feedback, and to analyse what it learns. It is not meant to establish an effect size. The revision states its single-model, two-benchmark, single-split scope in Section 5.

Q5 / other: regarding "both readings" and artifact length after revision

By hedging we mean delivering two readings of an ambiguous requirement so that one of them matches the grader. Our delivery rule allows this only when the evidence cannot settle the interpretation, and only in written answers; workbooks, patches, and records hold a single value, and revision gains there too. On APEX-Agents with Opus, 11% of the selected base artifacts already present two readings before revision, and 15% after. On the tasks where revision added a reading, scores did not rise, so the gain comes from the other tasks, where revision changes values and adds omitted items on the strength of an evidence record. Artifact length is essentially unchanged. These numbers are in Appendix C of the revision.

Other: regarding the APEX task counts (480 / 434 / 416)

The three numbers are nested. 480 tasks have ten-rollout pools. 434 of them have rubric criteria that name a value readable off an answer, and these enter the Figure 1 analysis. On 416 of those, the extraction produced at least one groupable claim, so the pool entropy of Figure 4 is defined. The chain is now in Appendix A.

Nit: line 54. Fixed with your wording.

We hope the setting statement answers (i), section A answers (ii), and section B answers (iii). We would be glad to discuss anything that remains open.

Dear Reviewer,

As the discussion period closes in about 24 hours, we wanted to check whether our response has addressed your concerns. If anything remains unclear or you have further suggestions, we would be glad to respond in the time left.

Thank you again for your time.

Reviewer_5pa8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VeriHarness, a test-time verification setup where the same model that produced N rollouts acts as an agentic verifier inside the task workspace. One context resolves claims on which rollouts disagree, another challenges claims they all share, and a fresh context adjudicates, picks a base rollout and optionally revises it. A hand-written skill library guides the checks, and the authors also let the model evolve this library from grader feedback on a dev split. Experiments cover five workspace benchmarks with Gemini 3.5 Flash and Claude Opus 4.8, with gains over voting, judge-based selection and an agentic verifier baseline, plus ablations and a cost analysis.

Critical strengths
  1. The setup is clean. All methods share the same frozen rollout pool and the same model, the verifier never sees rubrics, and the selection oracle gives a clear upper bound. Gains point the same way on every model and benchmark pair, so the main selection result is fairly convincing.
  2. Splitting verification into disputed and consensus claims is simple and backed by the data. The gain sits mostly on pools that disagree, and the analysis of why shared errors survive the challenger is candid and useful.
Critical weaknesses
  1. It is not clear where the gain comes from. In Table 4, the harness without skills scores almost the same as the agentic verifier + revision baseline in Table 1, for both models. Most of the margin therefore seems to come from the hand-written skill library. That library was written from development failures on the same benchmarks, and some skills explicitly encode how the benchmarks grade outputs (repair-prose, repair-record). The baselines never get these skills. This makes it hard to credit the protocol itself, and the general-purpose claim is weaker than stated.
  2. In several cells the margin over the best baseline is a point or two, with no significance tests. The self-evolution study uses one model, two benchmarks and one split, which the authors themselves call point estimates.
  3. The numbers in Figure 1 come from Opus extracting claims and matching them to rubric items, with no human check reported, on one benchmark and one model. The design is motivated by these numbers, so the extraction step needs some validation.
  4. Equations (3) and (5) are written as information gain and failure probability, but the model only judges them qualitatively and nothing is computed, so they read as framing.
Questions for authors
  1. What do the agentic verifier baselines score with the same skill library? This affects how much I credit the protocol.
  2. Which tasks did the development failures behind the human-authored library come from, and do they overlap with the tasks in Table 1?
  3. Was the claim extraction behind Figure 1 checked by humans at all?
  4. A compute-matched comparison would help, for example spending the verifier budget on more rollouts plus a judge.
2 comments

Thank you for your careful review. We ran the control you asked for and computed the intervals. Please refer to the general response for the details and to the revised PDF (changes in purple).

W1 / Q1 / Q2: regarding where the gain comes from, the skills-matched baseline, and the provenance of the library

We added the experiment that isolates the protocol from the skills. It is section B of the general response and Table 2 of the revision.

  • Skills-matched baseline (Q1). Agentic verifier + the eighteen skills scores 50.2 / 51.8 (Flash / Opus) in selection and 51.0 / 52.6 in revision. VeriHarness scores 51.6 / 53.7 and 53.4 / 56.1. Skills alone add at most 0.8 to the single-context agent, and the protocol alone at most 0.7. Together they add 2.2 / 2.1 in selection and 2.7 / 3.0 in revision. The gain is in the combination: most skills are written for a step the protocol creates, which a single context does not have.
  • Provenance (Q2). 20% of each benchmark was set aside as a development set before any experiment. The library was written from failures on those tasks only, then frozen. Table 1 reports the remaining 80%. The split table is section A of the general response (Section 4.1, Table 3, Appendix H of the revision).
  • repair-prose and repair-record. They describe what any reader of a written answer or a record looks for: stated final values with units, the source named, fields filled. They contain no benchmark's rubric. "How it is graded" was our wording; the entries now read "what is assessed in". Appendix G states that no skill contains a benchmark's rubric or a task identifier.
  • "General-purpose". The skills are written per deliverable type (workbook, patch, written answer, bundle, record), not per benchmark, and one library runs unchanged across the five benchmarks and three agent CLIs.

W2: regarding significance and the single-split self-evolution study

Paired task-level bootstrap intervals are in section C of the general response. On the five-benchmark average, VeriHarness (select) exceeds the agentic verifier by +2.2 [1.5, 3.2] with Flash and +2.1 [1.0, 3.0] with Opus. The full harness exceeds agentic verifier + revision by +2.7 [1.3, 3.6] and +3.0 [1.6, 4.4]. The revision gain over selection excludes zero on average (+1.8 [1.1, 2.5] / +2.4 [1.6, 3.2]) and in seven of the ten benchmark cells; Section 4.2 now says so instead of "exceeds every baseline in all ten settings".

For the self-evolution study, a further split means re-evolving conditions C and D, which does not fit in the discussion period. The study shows that the harness can improve its own skills from failure feedback and analyses what it learns; it does not establish an effect size. The revision states its single-model, two-benchmark, single-split scope in Section 5.

W3 / Q3: regarding validation of the Figure 1 extraction

Figure 1 is motivation, not part of the pipeline. Correctness there comes from the benchmark's grades, not from Opus. The harness never extracts claims; the resolver and challenger read the rollouts directly. In the offline analysis, the model only records the value each rollout asserts for each rubric criterion. Two annotators with master's degrees in computer science labelled 100 randomly sampled claims (1,000 rollout values), blind to the model's output. The model's values matched theirs for 87% of the values, and its grouping of rollouts into same-value sets matched theirs 84% of the time (section D of the general response, Appendix A of the revision). The analysis uses APEX-Agents alone because it is the only benchmark whose grader records a per-criterion verdict for every rollout; Section 2 now says this.

W4: regarding Equations (3) and (5)

Agreed: they are design principles, not computed quantities. The revision says so next to the equations in Section 3, keeps the Lindley reference, and changes "choose by Eq. (3) / (5)" in Algorithm 1 to "choose following".

Q4: regarding a compute-matched comparison (verifier budget spent on more rollouts plus a judge)

Spending the verifier budget on more rollouts would buy one to three extra rollouts per task, and a judge cannot turn those into a gain. The rollout pools cost over USD 100,000 for about 26,000 rollouts, so a rollout costs about USD 4. Selection with VeriHarness costs USD 1.44 per task with Flash and USD 3.92 with Opus, less than one rollout. Revision costs about three times that. So the compute-matched alternative is eleven to thirteen rollouts plus a judge, against ten rollouts plus the harness. The problem for the judge is not the number of rollouts. With ten rollouts, the selection oracle is already 15 points above the single rollout with Flash and 13 with Opus. Best-of-N with a judge recovers only 0.1 and 1.0 of those points (Table 1). One to three more rollouts would not change that. The comparison is in Appendix J.

Thank you again for the precise questions.

Dear Reviewer,

As the discussion period closes in about 24 hours, we wanted to check whether our response has addressed your concerns. If anything remains unclear or you have further suggestions, we would be glad to respond in the time left.

Thank you again for your time.

Reviewer_GjivOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduced VeriHarness. It turns LLM generator to a verifier with tools and skills. A disagreement resolver will check claims with evidence. A challenger evaluates shared claims and searches for omitted requirements. The authors tested it on 5 workspace benchmarks with 2 models, VeriHarness achieves the best scores.

Critical strengths
  1. It proposed a new harness by incorporating the idea of verification. It demonstrated effectiveness according to experiments, where it improves 6.2 pp with Gemini-3.5-flash and 6.4 pp with Opus 4.8.
  2. the claim-level diagnostics are informative. On apex-agents, it quantify how weak unanimity is as a correctness signal. The authors also showed that disagreement usually exposes a correct alternative. These findings are insightful
Critical weaknesses
  1. The claims is too strong. The authors claim "it's the first agentic verification harness for long-horizon tasks". However, agent-as-a-judge is a close precedent. Another relevant work is agentic aggregation, which describes treating parallel trajectories as an environment and inspect them with tools
  2. Claim extraction is done by opus 4.8. how reliable it is? There's no reported human check. It will be better to provide some audit statistics.
  3. What are the results of the agentic verifier given the same 18 skills? And what's the result of VeriHarness without skills?
Questions for authors

see weaknesses

1 comment

Thank you for the positive assessment, and for finding the claim-level diagnostics insightful. We agree that the weakness of unanimity as a signal is the paper's most useful observation. The new experiments and the revised PDF (changes in purple) are described in the general response; we refer to its sections below.

W1: regarding the "first" claim, Agent-as-a-Judge, and agentic aggregation

We have dropped "first", contrast both works in Section 7, and now run Agent-as-a-Judge as a baseline.

  • The claim was meant for the harness: the reusable layer of workspace, tools, protocol, and skills around a frozen model. It was not meant for agentic verification itself, which Agent-as-a-Judge does. Since the sentence reads more broadly than that, it is removed.
  • Agent-as-a-Judge grades one trajectory against a given requirement list. We work without one and take the targets from disagreement among rollouts. Table 9 in Appendix L of the revision places VeriHarness in one table with Agent-as-a-Judge, Gandalf, agentic aggregation, and the other verification methods. Run with the authors' released code in our setting, it scores 48.6 / 50.9 (Flash / Opus). VeriHarness (select) scores 51.6 / 53.7 (section B, Table 2 of the revision).
  • Agentic aggregation (Lee et al., 2026b, line 992) is the "Aggregation over the pool" row of Table 1, adapted from the authors' released implementation. It stays below the harness on every benchmark (Flash 49.9 vs 53.4, Opus 52.3 vs 56.1).

W2: regarding the reliability of claim extraction

Figure 1 is motivation, not part of the pipeline. Correctness there comes from the benchmark's grades, not from Opus. The harness never extracts claims; the resolver and challenger read the rollouts in the workspace directly. In the offline analysis, the model only records the value each rollout asserts for each rubric criterion. Two annotators with master's degrees in computer science labelled 100 randomly sampled claims (1,000 rollout values), blind to the model's output. For each claim they did the two things the model does: read off the value each of the ten rollouts asserts, and decide which rollouts assert the same value. The model's values matched theirs for 87% of the 1,000 values, and its grouping of rollouts into same-value sets matched theirs 84% of the time. The audit is section D of the general response and is added to Appendix A. Section 2 now states that the harness has no extraction step.

W3: regarding the agentic verifier with the same 18 skills, and VeriHarness without skills

Both numbers are now side by side in section B of the general response (Table 2 of the revision). Flash / Opus, with revision:

  • Agentic verifier + revision with the eighteen skills: 51.0 / 52.6.
  • VeriHarness without skills: 51.3 / 53.8. This one was already in Table 4, Appendix E; Section 4.2 now cites it.
  • VeriHarness: 53.4 / 56.1.

Skills alone and protocol alone each add less than a point. Together they add 2.7 / 3.0, because most skills are written for a step the protocol creates, and a single context has no step at which to apply them.

Thank you again for the review; we hope the clarifications are useful and would be glad to discuss further.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers (1/2)

We thank the reviewers for their careful reading. We have uploaded a revised PDF with all changes in purple. During the discussion period we also ran three new experiments. The points raised by more than one reviewer are answered together here, in labelled sections A–F (A and B in this comment, C–F in the next), and each individual reply points to these sections. Table numbers below follow the submission. The paper revision adds a new Table 2 in Section 4, so the later tables' indices shift by one.

A. Which tasks the skill library was built from, and which tasks Table 1 reports (v6YR Q2, 5pa8 Q2)

  1. Before any experiment, we set aside 20% of the tasks of each benchmark as a development set and froze the split.
  2. The human-authored library was written from general document-handling knowledge and from failures on development tasks only. The authors of the skills never inspected an evaluation task.
  3. Tables 1 and 4 are computed on the evaluation tasks only.
  4. Section 5 needs an evolve set. It therefore splits the evaluation tasks of APEX-Agents and SpreadsheetBench 2 a second time, about 3:1, into an evolve part and a held-out part. The evolution process never sees the held-out part. Figure 3 reports the held-out part whose numbers differ from the VeriHarness row of Table 1: the task subset differs, not the library. "Full-benchmark pools" in Appendix H meant the pools of all five benchmarks, as opposed to the two used for evolution. It did not mean that Table 1 includes development tasks, which we've clarified in the revised paper.
BenchmarkTasksDev (20%)Eval (80%), used in Tables 1 and 4
APEX-Agents48096384
Workspace-Bench Lite1002080
WorkBuddy Bench20040160
SpreadsheetBench 232164257
JobBench651352

The revised paper now states this clearly in Section 4.1, adds the Dev and Eval columns to Table 3, and rewrites the split paragraph of Appendix H.

B. Skill-matched controls and Agent-as-a-Judge (v6YR Q1, Q3; 5pa8 Q1; Gjiv W1, W3)

We ran Agent-as-a-Judge with the authors' released code in our sandbox. For adaptation, we pass the task description in place of the requirement list it expects, since our setting provides none. Then we gave our original baselines the same eighteen skills as our verifier. They are loaded from the same skills directory through the same on-demand mechanism. We also ran VeriHarness without skills, in selection and revision mode.All numbers are five-benchmark averages over three seeds, with the cross-seed standard deviation.

ModeMethodSkillsFlashOpus
SelectionAgent-as-a-Judge, released codenone48.6 ± 0.750.9 ± 0.4
SelectionAgentic verifier (env. access)none49.4 ± 0.651.6 ± 0.3
SelectionAgentic verifier (env. access)18 skills50.2 ± 0.451.8 ± 0.6
SelectionVeriHarness (select)none49.7 ± 0.552.3 ± 0.4
SelectionVeriHarness (select)18 skills51.6 ± 0.553.7 ± 0.5
RevisionAgentic verifier + revisionnone50.7 ± 0.653.1 ± 0.5
RevisionAgentic verifier + revision18 skills51.0 ± 0.652.6 ± 0.8
RevisionVeriHarnessnone51.3 ± 0.753.8 ± 0.4
RevisionVeriHarness18 skills53.4 ± 0.456.1 ± 0.3

What the table shows, Flash / Opus:

  • Skills alone, given to the single-context agentic verifier: +0.8 / +0.2 in selection, +0.3 / −0.5 in revision.
  • Protocol alone, VeriHarness without skills against the agentic verifier: +0.3 / +0.7 in selection, +0.6 / +0.7 in revision.
  • Both together: +2.2 / +2.1 in selection, +2.7 / +3.0 in revision. In every column this is more than the sum of the two separate effects.
  • Why: twelve of the eighteen skills are written for a step the protocol creates. These steps are what decides a disputed claim, how a unanimous pool fails, and how to close an unresolved claim at delivery. A single-context judge reads the same text but has no step at which to apply it.
  • Cost: with the same skills, the agentic verifier costs USD 1.21 / 3.22 per task (Flash / Opus) in selection and USD 3.95 / 9.48 in revision; VeriHarness (select) costs USD 1.44 / 3.92 and revision about three times that. About 20% more verification cost improves results by 1.4–3.5 points (Appendix J).
  • Agent-as-a-Judge lands between best-of-N and the agentic verifier. It costs USD 1.26 per task, against USD 1.44 for VeriHarness (select) with Flash.

This is the new Table 2 and the paragraph "The protocol and the skills are complementary" in Section 4.2 of the uploaded revised paper.

AuthorsOpenReview ↗

General response to all reviewers (2/2):

C. Statistical significance (v6YR Q4, 5pa8 W2)

Paired task-level bootstrap, 95% interval, five-benchmark average unless noted:

ComparisonFlashOpus
VeriHarness (select) − agentic verifier+2.2 [1.5, 3.2]+2.1 [1.0, 3.0]
VeriHarness − agentic verifier + revision+2.7 [1.3, 3.6]+3.0 [1.6, 4.4]
VeriHarness − VeriHarness (select), five-benchmark average+1.8 [1.1, 2.5]+2.4 [1.6, 3.2]

All three exclude zero. Per benchmark, the revision gain over selection excludes zero in seven of the ten cells; it is within noise on WorkBuddy Bench with Flash and on SpreadsheetBench 2 and JobBench with Opus, and Section 4.2 now says so. "Exceeds every baseline in all ten settings" is now stated as a point-estimate result, followed by the intervals (Section 4.2, Appendix D).

D. Human audit of the Figure 1 extraction (5pa8 W3/Q3, Gjiv W2)

  • Figure 1 is motivation, not part of the pipeline. The harness never extracts claims; the resolver and challenger read the rollouts in the workspace directly.
  • In Figure 1 the model only records the value each rollout asserts for each rubric criterion and groups equivalent expressions (Appendix A). Correctness comes from the benchmark's archived per-criterion grades, never from the model.
  • As the revision in Appendix A shows, to check the extraction, we requested two annotators with master's degrees in computer science labelled 100 randomly sampled claims (1,000 rollout values), blind to the model's output. For each claim they did the two things the model does: read off the value each of the ten rollouts asserts, and decide which rollouts assert the same value. The model's values matched theirs for 87% of the 1,000 values, and its grouping of rollouts into same-value sets matched theirs 84% of the time. The model is therefore a reliable reader for this extraction task.

Added to Appendix A and stated in Section 2.

E. Setting, and how VeriHarness differs from Agent-as-a-Judge and Gandalf (v6YR Q1/W2, Gjiv W1)

VeriHarness focuses on a test-time agent harness for verification. Every experiment in the paper is conducted at test time, and nothing is trained. The revision now says so in the abstract, Section 1, Section 7, and the conclusion. The outputs of VeriHarness could later feed training: the selected base expresses a preference among rollouts (GRPO), and the base with its revision forms a preference pair (DPO). That is future work and labelled as such.

Table 9 in Appendix L of the revision conducts a comprehensive comparision of various verification methods: majority voting, best-of-N judging, LLM-as-a-Verifier, Agent-as-a-Judge, Gandalf, and agentic aggregation. Two differences separate VeriHarness from the two closest works. Agent-as-a-Judge and Gandalf need a requirement list or rubric and grade one rollout against it; VeriHarness needs neither and takes its targets from the disagreement and consensus among N rollouts. Agent-as-a-Judge and Gandalf return a score; VeriHarness returns an artifact with its verification record. We have removed "the first agentic verification harness for long-horizon tasks", Section 7 now cites and contrasts both works, and Agent-as-a-Judge is now a baseline in Table 2 (section B).

F. Changes in the revised PDF (all in purple)

  • Abstract, Section 1, conclusion: test-time setting stated up front with "first" removed.
  • Section 2 and Appendix A: Figure 1 is motivation; human audit; 480 / 434 / 416 chain.
  • Section 3 and Algorithm 1: Equations (3) and (5) described as design principles.
  • Section 4.1 and Table 3: development and evaluation split. Section 4.2: new Table 2, bootstrap intervals.
  • Section 5 and Appendix H: second split explained; scope of the self-evolution study stated.
  • Section 7: contrast with Agent-as-a-Judge and Gandalf. Appendix L: Table 9, verification methods compared.
  • Appendix C: hedging statistics. Appendix D: Agent-as-a-Judge and skill-matched baselines; bootstrap procedure. Appendix G: skill entries renamed. Appendix J: cost of the skill-matched baselines; verifier budget against more rollouts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.