- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper formulates agent verification as a budget aware sequential policy over evidence gathering, tool execution, model delegation, and stopping. It introduces AgentVerificationBench, containing 541 coding and 189 scientific verification instances, and optimizes verifier harnesses with fixed model weights. Experiments show substantial reductions in verification cost and improved budget adherence, alongside lower held out science accuracy.
Critical strengths
-- A relevant problem and a potentially useful benchmark. Evaluating verification accuracy together with realized cost addresses a practical limitation of agent systems. AVB’s fixed candidate sets, executable labels, and source-task-disjoint splits provide a useful experimental foundation.
-- Useful evidence on resource efficiency. Harness optimization substantially reduces verification costs and improves budget adherence. The paper appropriately acknowledges the accompanying accuracy decrease and the limits of its conclusions.
Critical weaknesses
-- The methodological increment is insufficiently established. Agent-as-a-Judge, VerifiAgent, and Meta-Harness already cover important components of the approach. The experiments do not clearly isolate the additional value of adaptive verification: execution modes are distinguished only through prompts, while the main science comparison changes penalty instructions and concurrency. The optimized harness also adopts a largely fixed, inexpensive verification routine (§3.1; Appendices C–D).
-- The benchmark contribution needs stronger substantiation and reproducibility support. AVB contains 730 verification instances derived from 262 source questions, making curation and coverage central to its value. However, the main coding comparison uses only 98 of 270 test instances, and the selection rule for the “hard” subset is insufficiently specified. I could not locate an accessible code/data artifact in the supplied manuscript to independently check construction, filtering, labels, or environment reconstruction (Appendices A–B).
Questions for authors
-- Novelty clarification: What is the distinction from existing adaptive agentic verifiers equipped with the same tools, model pool, and budget information?
-- Validation of the central mechanism: Do the gains persist against the initial harness and a simple fixed Luna-panel verifier under identical penalty, concurrency, and turn limits? Can the replanning comparison establish actual differences in planning behavior at matched realized cost?
-- Benchmark clarification and reproducibility: Can the authors provide the construction code, frozen split/cohort manifests, grader/environment versions, and optimized harness? What determined the 98-instance coding subset, including all exclusions from the original 100-instance subset?
6 comments
Author Response (Part 2/3)
Q2: Validation of the central mechanism: Do the gains persist against the initial harness and a simple fixed Luna-panel verifier under identical penalty, concurrency, and turn limits? Can the replanning comparison establish actual differences in planning behavior at matched realized cost?
Thank you for requesting matched controls and a test of replanning.
Initial, fixed-panel, and optimized harnesses. At a nominal 16 USD budget, the table combines submitted harness results with additional fixed-panel experiments on 98 Code instances and additional initial–panel experiments on 78 Science instances over 30 source questions. Figure 8 (Appendix D) illustrates the final selected Code harness procedure; Appendix E.3 provides its complete frozen prompt.
| Domain | Metric | Initial () | Fixed Luna panel | Optimized () |
|---|---|---|---|---|
| Code (98) | Correct () | 61/98 (62.24%) | 50/98 (51.02%) | 73/98 (74.49%) |
| Mean cost (USD) () | 44.06 USD | 2.80 USD | 9.84 USD | |
| Within 16 USD () | 24/98 (24.49%) | 98/98 (100.00%) | 82/98 (83.67%) | |
| Mean utility () () | −1.249 | 0.510 | 0.684 | |
| Science (78) | Correct () | 40/78 (51.28%) | 36/78 (46.15%) | 43/78 (55.13%) |
| Correct and within 16 USD () | 34/78 (43.59%) | 36/78 (46.15%) | 42/78 (53.85%) | |
| Mean cost (USD) () | 9.13 USD | 1.05 USD | 5.72 USD | |
| Within 16 USD () | 67/78 (85.90%) | 78/78 (100.00%) | 77/78 (98.72%) | |
| Mean utility () () | 0.379 | 0.462 | 0.542 |
Panel procedure and accounting. The scripted panels use predetermined calls and aggregation. New analysis of the original Code records establishes complete costs including all failed and retried attempts. All 13 invalid Code panel decisions count wrong.
Author Response (Part 3/3)
W2: The benchmark contribution needs stronger substantiation and reproducibility support. AVB contains 730 verification instances derived from 262 source questions, making curation and coverage central to its value. However, the main coding comparison uses only 98 of 270 test instances, and the selection rule for the “hard” subset is insufficiently specified. I could not locate an accessible code/data artifact in the supplied manuscript to independently check construction, filtering, labels, or environment reconstruction (Appendices A–B).
Q3: Benchmark clarification and reproducibility: Can the authors provide the construction code, frozen split/cohort manifests, grader/environment versions, and optimized harness? What determined the 98-instance coding subset, including all exclusions from the original 100-instance subset?
Thank you for raising this reproducibility concern; we agree that cohort selection and access to construction artifacts must be explicit.
Cohort selection. The Code test cohort is a stratified sample; "hard subset" was an inaccurate description, which we correct. Source questions are split 40.00%/10.00%/50.00% into train/validation/test, giving 270 test instances over 100 questions. Difficulty is the number of the 16 candidates that pass the source benchmark's executable grader; no LLM assigns correctness. To control cost, we sample 100 test instances proportionally across domain × generator × difficulty cells:
| Difficulty | Passing candidates | Full test | Sample |
|---|---|---|---|
| Hard | 1–4 | 99 | 36 |
| Medium | 5–11 | 102 | 39 |
| Easy | 12–15 | 69 | 25 |
| Total | 1–15 | 270 | 100 |
Exclusions and sensitivity. After freezing the 270-instance Code test split and the stratified 100-instance sample, we excluded break-filter-js-from-html/gpt_5_4_mini for execution issues and regex-log/luna over leakage concerns, leaving 98 instances for the shared comparison. For the two exclusions, new worst-case arithmetic uses the observed 73/98 versus 61/98 correct (Sol, 16 USD, soft-overrun). Even if both restored cases were wrong for the optimized harness and correct for the initial harness, the totals would be 73/100 versus 63/100: the ordering cannot reverse.
Benchmark construction. An instance is a question–generator pair. Code has 541 instances over 192 questions, each with 16 trajectories; Science has 189 instances over 70 questions, totaling 730 instances over 262 questions. Science uses same-generator triples with zero, one or two correct candidates; NONE is correct only when all three fail. From Terminal-Bench 2.1 and DeepSWE v1.1, we keep 541 fully labeled Code instances that contain both passing and failing candidates, after removing observed hidden-test or reference-solution leakage. The full splits and evaluated test cohorts are:
| Benchmark | Train | Validation | Test | Evaluated test |
|---|---|---|---|---|
| Code | 217 | 54 | 270 | 98 |
| Science | 69 | 42 | 78 | 78 |
Anonymous materials. Benchmark code, frozen split/cohort manifests, and optimized harnesses are available in our anonymous repository.
Paper revisions (in blue)
All changes are marked in blue in the revised manuscript:
-
Related Work. We clarified the distinctions from Agent-as-a-Judge and VerifiAgent and credited Meta-Harness for the optimizer.
-
§4.3 and Appendix G. We added the Code fixed-panel results with complete costs and invalid decisions counted wrong, and distinguish the separate runtimes.
-
§4.3 and Appendix H. We report exactly 42/78 optimized Science decisions correct within budget, eight more than the matched H0 run.
-
Conclusion and Limitations. We aligned the Science summary with the matched H0 comparison and distinguished the limitations of the comparison with the submitted H0 run.
-
Figure 8 (Appendix D) and Appendix E.3. We added a schematic and the complete frozen prompt of the final selected Code harness.
-
Appendix B. We specified stratified sampling, exclusions, cohort bounds, the released package’s contents, and its direct anonymous link.
Thank you again for the constructive feedback. We welcome follow-up questions about these results and the released materials.
Author Response (Part 1/3)
We thank the reviewer for recognizing (i) AVB's fixed candidates, executable labels, and task-disjoint splits and (ii) cost savings and improved budget adherence from harness optimization. We address each concern below.
(W1/Q1) Novelty. We study verification of agent trajectories as a budget-conditioned sequential policy and optimize its reusable harness with fixed weights.
(Q2) Initial-harness and fixed-panel comparisons. The fixed-cohort results are shown below the correction. The optimized harnesses are more accurate than the fixed Luna panels in both domains, at higher cost.
(W2/Q3) Benchmark and reproducibility. We specify the stratified sampling and the two exclusions behind the fixed 98-instance Code cohort. The anonymous package is now released.
(Correction to the Science baseline): The submitted Science H0 run used a weaker overrun penalty than the optimized harness: 0.0625 utility per dollar over the 16 USD budget (normalized λ = 1) instead of 0.2 per dollar (λ = 3.2). It also used a different concurrency limit and model pool. We now compare against a matched H0 run that uses the optimized harness’s settings: 0.2 per dollar, Luna-only delegation, at most three concurrent calls, and a 28-turn cap. Even with these settings, on 78 test instances H0 → H* increases accuracy, budget adherence, and utility from 51.28%, 85.90%, and 0.379 to 55.13%, 98.72%, and 0.542, and lowers mean cost from 9.13 to 5.72 USD. The comparison with the submitted H0 run (47/78 correct, at least 23.20 USD mean cost) is retained in Appendix G.
Here H0 is the initial harness and H* the validation-selected harness for each domain. The Science H0 entries use the matched H0 run.
Thank you for asking us to clarify the increment over existing adaptive verifiers.
Contribution relative to prior work. With the same tools, model pool, and budget information, our empirical contribution is optimizing the reusable verification harness with fixed model weights, and evaluating it on AVB. Agent-as-a-Judge [1] and VerifiAgent [2] use one backbone with fixed modules or stages and report cost afterwards, whereas a Meta-Verifier picks a model per call from a priced pool and conditions on its remaining budget. We build on Meta-Harness [3] for the optimizer. We therefore locate our contribution in the trajectory-verification setting, budget-aware evaluation, and verification-specific harness optimization.
Sequential formulation (§3.1). For task q, candidate trajectories t=(τ1,…,τN), resources R, model weights θ and harness H, step k selects ak∼πθ,H(⋅∣q,t,R,hk,bk). Here hk is the prior action–observation history and bk=B−C<k the remaining nominal budget, with C<k the total cost so far. Actions include inspection, executable checks, delegation, comparison, and terminal selection or rejection. This describes the induced agent behavior; no separate policy network is trained.
Harness optimization (§3.3). For instance i, correctness yi(H), total cost Ci(H) and nominal budget Bi define utility uiλ(H)=yi(H)−λmax(0,Ci(H)/Bi−1), with normalized overrun penalty λ≥0. The optimizer revises H from training traces, correctness and cost feedback with θ fixed. We select by mean utility on task-disjoint validation data and freeze before testing.
[1] Zhuge et al. Agent-as-a-Judge: Evaluate Agents with Agents. arXiv:2410.10934, 2024.
[2] Han et al. VerifiAgent: a Unified Verification Agent in Language Model Reasoning. Findings of EMNLP, 2025.
[3] Lee et al. Meta-Harness: End-to-End Optimization of Model Harnesses. arXiv:2603.28052, 2026.