- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies same-model verification of long-horizon agent outputs such as reports, spreadsheets, and patches. Given a pool of N=10 rollouts, the verifier (the same model as the generator) has to select or revise the final artifact without access to reference answers or grading rubrics. A claim-level analysis on APEX-Agents motivates the method: 34% of the claims that all rollouts agree on are wrong, while 74% of disputed claims contain a correct candidate even though the most frequent answer is correct only 47% of the time (Fig. 1). VeriHarness gives the verifier a read-only workspace with all rollouts and the task environment, generic file and shell tools, and a library of 18 human-authored verification skills. A disagreement resolver checks disputed claims against evidence, a consensus challenger tries to refute agreed claims, and an adjudicator selects a base rollout and optionally revises it (Sec. 3, Alg. 1).
The method is evaluated on five workspace benchmarks (APEX-Agents, Workspace-Bench Lite, WorkBuddy Bench, SpreadsheetBench 2, JobBench) with Gemini 3.5 Flash and Claude Opus 4.8 as both generator and verifier. In selection mode it beats majority voting, best-of-N, a pairwise tournament, LLM-as-a-Verifier, and a tool-using agentic verifier in all ten settings, improving over a single rollout by 4.4 (Flash) and 4.1 (Opus) points; with revision, the gains are 6.2 and 6.4 points (Table 1). The paper also reports ablations (Table 4), a cost analysis (App. J), and a self-evolution study in which skills learned from failure feedback on development tasks improve held-out performance (Sec. 5).
Critical strengths
- The evaluation is broad and controlled. All methods share the same frozen rollout pools and the same model, the benchmarks are recent and realistic, and the selection gains hold in all ten settings (Table 1) and largely carry over when the protocol runs inside Gemini CLI, Claude Code, and Codex (App. F). The cost analysis (App. J) puts selection at $1.44/task with Flash and $3.92/task with Opus. Evidence that a training-free, same-model verifier improves selection at moderate cost is practically useful.
- The diagnostics explain where same-model verification helps rather than only reporting scores. The selection gains are concentrated in pools that disagree (+0.6 on 154 consensus pools vs. +6.1 on 262 disputed pools, Fig. 4), and Table 7 shows that 70% of the shared errors the challenger misses come from checking an upstream quantity instead of the graded one. This was the most informative part of the paper for me.
Critical weaknesses
- The contribution of the protocol is confounded with the skill library. Without skills, VeriHarness scores 53.8 (Opus) and 51.3 (Flash) (Table 4), while the agentic verifier + revision baseline, which has the same tools and environment access but neither the protocol nor the skills, scores 53.1 and 50.7 (Table 1). The resolver/challenger protocol on its own therefore adds about 0.6–0.7 points, which is within the reported SDs, and most of the remaining margin comes from the 18 skills that baselines do not receive. App. H states that the library was written from "development failures", Table 1 appears to use the full benchmarks including those development tasks, and some skills describe "how prose is graded" and "how office bundles are graded" (Table 6), which sits uneasily with the no-rubric setting. Without a skills-matched baseline and a clean task split, I cannot tell whether the protocol or prior exposure to the benchmarks drives the headline numbers.
- The target of the paper is not clear, and the positioning against other agentic verifiers is weak. From the abstract and introduction, it is not clear whether VeriHarness is a test-time compute method (selecting and revising among rollouts at deployment) or a verifier meant to produce a training signal such as RL rewards. The experiments are entirely test-time selection and revision, and App. K says the harness assigns no score to individual rollouts, but the title, the related work (process reward models, agentic verifiers trained with RL), and the conclusion (verification records as training data) point toward training. The two settings call for different related work and baselines: test-time scaling methods at matched compute in the first case, and reward models and agentic graders in the second. Relatedly, the paper claims to be "the first agentic verification harness for long-horizon tasks" (Sec. 1), but agentic verifiers that inspect files and tool state in the task environment already exist, for example Agent-as-a-Judge (Zhuge et al., 2025), which is only cited in passing, and Gandalf (Handshake AI Research, 2026, https://joinhandshake.com/research/ai/gandalf-the-grader/), which runs the verifier agent in the same environment as the rollout agent. None of these are discussed or used as baselines. As far as I can tell, the main differences are that VeriHarness works without rubrics and uses disagreement across rollouts to decide what to check, and this should be made explicit.
Other, less critical points:
- The uncertainty estimates are limited. The SDs come from three verifier runs on one frozen pool, so they do not capture task or pool sampling, and no paired task-level test is reported. Several "beats every baseline" cells are within noise (e.g., Flash SB-2: 31.5 ± 2.4 vs. 30.9 ± 1.1 for the agentic verifier), JobBench has only 65 tasks, Table 4 has no SDs, and the self-evolution results in Fig. 3 are single runs.
- Revision delivers "both readings" of unresolved interpretations when the format allows (App. C). With rubric-based LLM graders, this could earn credit by hedging rather than by being correct.
- The APEX-Agents task counts differ across the paper: 480 in Table 3, 434 in the Fig. 1 analysis (App. A), and 154 + 262 = 416 pools in Fig. 4.
- nit: on line 54, "Existing test-time verification methods mostly use only the rollouts" is not grammatical; perhaps "Most existing test-time verification methods use only the rollouts".
Questions for authors
- (Clarification) Is VeriHarness intended as a test-time compute method or as a verifier for producing training signal (e.g., RL rewards)? Please state this early in the paper and adjust the related work and baselines accordingly. In either case, how does VeriHarness differ from existing agentic verifiers such as Agent-as-a-Judge and Gandalf, and could one of them be included as a baseline?
- (Clarification) How was the human-authored skill library (Table 6) built? Which benchmarks' development failures informed it, how were development tasks defined for WSB, WorkBuddy, and JobBench, and are any of those tasks in the Table 1 evaluation? If so, please report Table 1 on tasks the library authors never saw, and explain how skills that describe how deliverables are graded fit the no-rubric setting.
- (Suggestion) Please add a skills-matched control: the agentic verifier and agentic verifier + revision baselines given the same 18 skills, next to VeriHarness without skills, and add these baselines to the cost plot (Fig. 7). If the protocol still clearly beats a skills-equipped single-context agent at similar cost, my main concern would be resolved.
- (Suggestion) Please report paired, task-level uncertainty, e.g., a bootstrap over tasks of the per-task difference between VeriHarness and the strongest baseline on each benchmark, and SDs or CIs for the Table 4 ablations.
- (Clarification) How often does revision deliver both readings of an ambiguous requirement, and how does artifact length change after revision? The largest revision gain is Opus on APEX (41.2 to 47.5, Table 1), so this matters for interpreting the headline numbers.
My score could increase if the authors (i) clarify the target setting and position the work against existing agentic verifiers (Q1), (ii) show that the gains hold on tasks not used to write the skills (Q2), and (iii) show that the protocol beats a skills-matched baseline (Q3).
3 comments
Reply to Reviewer v6YR Part 2/2
Q4 / other: regarding paired task-level uncertainty, Table 4 SDs, and single-run Figure 3
The paired task-level intervals are in section C of the general response. The main gains exclude zero. In selection, VeriHarness exceeds the agentic verifier by +2.2 [1.5, 3.2] with Flash and +2.1 [1.0, 3.0] with Opus. In revision, it exceeds agentic verifier + revision by +2.7 [1.3, 3.6] and +3.0 [1.6, 4.4]. The revision gain over selection is +1.8 [1.1, 2.5] / +2.4 [1.6, 3.2] on average and within noise in three of the ten cells (WorkBuddy with Flash; SpreadsheetBench 2 and JobBench with Opus); Section 4.2 now says so.
"Exceeds every baseline in all ten settings" is now a point-estimate statement followed by the intervals. The SD column is described as verifier-run variability on a frozen pool. Table 4 now carries the cross-seed SD of its Avg column; every ablation gap it discusses is at least twice that spread.
Figure 3: a further split means re-evolving conditions C and D, which does not fit in the discussion period. The study is meant to show that the harness can improve its own skills from failure feedback, and to analyse what it learns. It is not meant to establish an effect size. The revision states its single-model, two-benchmark, single-split scope in Section 5.
Q5 / other: regarding "both readings" and artifact length after revision
By hedging we mean delivering two readings of an ambiguous requirement so that one of them matches the grader. Our delivery rule allows this only when the evidence cannot settle the interpretation, and only in written answers; workbooks, patches, and records hold a single value, and revision gains there too. On APEX-Agents with Opus, 11% of the selected base artifacts already present two readings before revision, and 15% after. On the tasks where revision added a reading, scores did not rise, so the gain comes from the other tasks, where revision changes values and adds omitted items on the strength of an evidence record. Artifact length is essentially unchanged. These numbers are in Appendix C of the revision.
Other: regarding the APEX task counts (480 / 434 / 416)
The three numbers are nested. 480 tasks have ten-rollout pools. 434 of them have rubric criteria that name a value readable off an answer, and these enter the Figure 1 analysis. On 416 of those, the extraction produced at least one groupable claim, so the pool entropy of Figure 4 is defined. The chain is now in Appendix A.
Nit: line 54. Fixed with your wording.
We hope the setting statement answers (i), section A answers (ii), and section B answers (iii). We would be glad to discuss anything that remains open.
Dear Reviewer,
As the discussion period closes in about 24 hours, we wanted to check whether our response has addressed your concerns. If anything remains unclear or you have further suggestions, we would be glad to respond in the time left.
Thank you again for your time.
Reply to Reviewer v6YR Part 1/2
Thank you for the careful and accurate review, and for singling out the diagnostics as the most informative part. You list three things that would change your assessment. We respond to Q1–Q3 first, then Q4, Q5, and the smaller points. For all details, please refer to the revised PDF file and the general response; we refer to its sections A–F below.
VeriHarness focuses on a test-time agent harness for verification. Every experiment is conducted at test time, and nothing is trained. Y
No task in Table 1 was seen by the authors of the skills. Section A of the general response gives the split table. In short:
The revision states this in Section 4.1, Table 3, and Appendix H.
On "how prose is graded" in Table 6: the phrase was ours and it misleads. These skills describe what any reader of a written answer, record, or bundle looks for: stated final values with units, the source named, fields filled. They contain no benchmark's rubric. The entries now read "what is assessed in a written answer / record / bundle". Appendix G states that no skill contains a benchmark's rubric or a task identifier.
We ran the control you ask for. It is section B of the general response and Table 2 of the revision. Flash / Opus:
On cost: with the same skills, the agentic verifier costs USD 1.21 per task with Flash and USD 3.22 with Opus in selection. VeriHarness (select) costs USD 1.44 and USD 3.92. Revision is about three times selection for both. So for about 20% more verification cost the harness gains 1.4–3.5 points over a skills-equipped single-context agent (Appendix J).