acceptodds
Under review as a conference paper at ICLR 2027

Verification as a Policy: Meta-Verification for AI Agents

Abstract

Verifying whether an AI agent has successfully completed a task could require inspecting its execution trajectory, running executable checks, or consulting other models. Which verification action to perform next depends on the evidence collected so far and the resources remaining. We study Meta-Verification, formulating verification as a sequential decision-making problem in which an agent adaptively selects verification actions and decides when to stop under a resource budget. Rather than following a prescribed verification procedure, we propose Meta-Verification, an approach which provides an agent the ability to construct an instance-specific verification procedure and delegate compute for executing it as evidence accumulates. We introduce AgentVerificationBench, comprising 541 best-of-16 coding tasks and 189 best-of-3 scientific tasks, to evaluate verification accuracy, realized cost, and budget adherence. We find that frontier agents combine diverse verification mechanisms, but the benefits of online replanning vary across models, and unmodified agents frequently exceed their nominal budgets. We therefore investigate harness optimization as a means of improving Meta-Verification through experience while keeping model weights fixed. On a held-out science test set, harness optimization reduces mean verification cost from $23.20 to $5.72 and increases budget adherence from 42.3% to 98.7%, while selection accuracy changes from 60.3% to 55.1%. These findings establish sequential verification-policy construction as a measurable agent capability and demonstrate that its resource efficiency can be substantially improved through harness optimization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Verification, Agents, Self-Improvement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NfveOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates agent verification as a budget aware sequential policy over evidence gathering, tool execution, model delegation, and stopping. It introduces AgentVerificationBench, containing 541 coding and 189 scientific verification instances, and optimizes verifier harnesses with fixed model weights. Experiments show substantial reductions in verification cost and improved budget adherence, alongside lower held out science accuracy.

Critical strengths

-- A relevant problem and a potentially useful benchmark. Evaluating verification accuracy together with realized cost addresses a practical limitation of agent systems. AVB’s fixed candidate sets, executable labels, and source-task-disjoint splits provide a useful experimental foundation.

-- Useful evidence on resource efficiency. Harness optimization substantially reduces verification costs and improves budget adherence. The paper appropriately acknowledges the accompanying accuracy decrease and the limits of its conclusions.

Critical weaknesses

-- The methodological increment is insufficiently established. Agent-as-a-Judge, VerifiAgent, and Meta-Harness already cover important components of the approach. The experiments do not clearly isolate the additional value of adaptive verification: execution modes are distinguished only through prompts, while the main science comparison changes penalty instructions and concurrency. The optimized harness also adopts a largely fixed, inexpensive verification routine (§3.1; Appendices C–D).

-- The benchmark contribution needs stronger substantiation and reproducibility support. AVB contains 730 verification instances derived from 262 source questions, making curation and coverage central to its value. However, the main coding comparison uses only 98 of 270 test instances, and the selection rule for the “hard” subset is insufficiently specified. I could not locate an accessible code/data artifact in the supplied manuscript to independently check construction, filtering, labels, or environment reconstruction (Appendices A–B).

Questions for authors

-- Novelty clarification: What is the distinction from existing adaptive agentic verifiers equipped with the same tools, model pool, and budget information?

-- Validation of the central mechanism: Do the gains persist against the initial harness and a simple fixed Luna-panel verifier under identical penalty, concurrency, and turn limits? Can the replanning comparison establish actual differences in planning behavior at matched realized cost?

-- Benchmark clarification and reproducibility: Can the authors provide the construction code, frozen split/cohort manifests, grader/environment versions, and optimized harness? What determined the 98-instance coding subset, including all exclusions from the original 100-instance subset?

6 comments
AuthorsOpenReview ↗

Author Response (Part 1/3)

We thank the reviewer for recognizing (i) AVB's fixed candidates, executable labels, and task-disjoint splits and (ii) cost savings and improved budget adherence from harness optimization. We address each concern below.

Summary

(W1/Q1) Novelty. We study verification of agent trajectories as a budget-conditioned sequential policy and optimize its reusable harness with fixed weights.

(Q2) Initial-harness and fixed-panel comparisons. The fixed-cohort results are shown below the correction. The optimized harnesses are more accurate than the fixed Luna panels in both domains, at higher cost.

(W2/Q3) Benchmark and reproducibility. We specify the stratified sampling and the two exclusions behind the fixed 98-instance Code cohort. The anonymous package is now released.

(Correction to the Science baseline): The submitted Science H0 run used a weaker overrun penalty than the optimized harness: 0.0625 utility per dollar over the 16 USD budget (normalized λ = 1) instead of 0.2 per dollar (λ = 3.2). It also used a different concurrency limit and model pool. We now compare against a matched H0 run that uses the optimized harness’s settings: 0.2 per dollar, Luna-only delegation, at most three concurrent calls, and a 28-turn cap. Even with these settings, on 78 test instances H0 → H* increases accuracy, budget adherence, and utility from 51.28%, 85.90%, and 0.379 to 55.13%, 98.72%, and 0.542, and lowers mean cost from 9.13 to 5.72 USD. The comparison with the submitted H0 run (47/78 correct, at least 23.20 USD mean cost) is retained in Appendix G.

DomainMetricInitial (H0H_0)Fixed Luna panelOptimized (H⋆H^\star)
Code (98)Correct (↑\uparrow)61/98 (62.24%)50/98 (51.02%)73/98 (74.49%)
Mean cost (USD) (↓\downarrow)44.06 USD2.80 USD9.84 USD
Science (78)Correct (↑\uparrow)40/78 (51.28%)36/78 (46.15%)43/78 (55.13%)
Mean cost (USD) (↓\downarrow)9.13 USD1.05 USD5.72 USD

Here H0 is the initial harness and H* the validation-selected harness for each domain. The Science H0 entries use the matched H0 run.


W1: The methodological increment is insufficiently established. Agent-as-a-Judge, VerifiAgent, and Meta-Harness already cover important components of the approach. The experiments do not clearly isolate the additional value of adaptive verification: execution modes are distinguished only through prompts, while the main science comparison changes penalty instructions and concurrency. The optimized harness also adopts a largely fixed, inexpensive verification routine (§3.1; Appendices C–D).

Q1: Novelty clarification: What is the distinction from existing adaptive agentic verifiers equipped with the same tools, model pool, and budget information?

Thank you for asking us to clarify the increment over existing adaptive verifiers.

Contribution relative to prior work. With the same tools, model pool, and budget information, our empirical contribution is optimizing the reusable verification harness with fixed model weights, and evaluating it on AVB. Agent-as-a-Judge [1] and VerifiAgent [2] use one backbone with fixed modules or stages and report cost afterwards, whereas a Meta-Verifier picks a model per call from a priced pool and conditions on its remaining budget. We build on Meta-Harness [3] for the optimizer. We therefore locate our contribution in the trajectory-verification setting, budget-aware evaluation, and verification-specific harness optimization.

Sequential formulation (§3.1). For task qq, candidate trajectories t=(τ1,…,τN)t=(\tau_1,\ldots,\tau_N), resources R\mathcal{R}, model weights θ\theta and harness HH, step kk selects ak∼πθ,H(⋅∣q,t,R,hk,bk)a_k\sim\pi_{\theta,H}(\cdot\mid q,t,\mathcal{R},h_k,b_k). Here hkh_k is the prior action–observation history and bk=B−C<kb_k=B-C_{<k} the remaining nominal budget, with C<kC_{<k} the total cost so far. Actions include inspection, executable checks, delegation, comparison, and terminal selection or rejection. This describes the induced agent behavior; no separate policy network is trained.

Harness optimization (§3.3). For instance ii, correctness yi(H)y_i(H), total cost Ci(H)C_i(H) and nominal budget BiB_i define utility uiλ(H)=yi(H)−λmax⁡(0,Ci(H)/Bi−1)u_i^\lambda(H)=y_i(H)-\lambda\max(0,C_i(H)/B_i-1), with normalized overrun penalty λ≥0\lambda\geq0. The optimizer revises HH from training traces, correctness and cost feedback with θ\theta fixed. We select by mean utility on task-disjoint validation data and freeze before testing.


References

[1] Zhuge et al. Agent-as-a-Judge: Evaluate Agents with Agents. arXiv:2410.10934, 2024.

[2] Han et al. VerifiAgent: a Unified Verification Agent in Language Model Reasoning. Findings of EMNLP, 2025.

[3] Lee et al. Meta-Harness: End-to-End Optimization of Model Harnesses. arXiv:2603.28052, 2026.

AuthorsOpenReview ↗

Author Response (Part 2/3)

Q2: Validation of the central mechanism: Do the gains persist against the initial harness and a simple fixed Luna-panel verifier under identical penalty, concurrency, and turn limits? Can the replanning comparison establish actual differences in planning behavior at matched realized cost?

Thank you for requesting matched controls and a test of replanning.

Initial, fixed-panel, and optimized harnesses. At a nominal 16 USD budget, the table combines submitted harness results with additional fixed-panel experiments on 98 Code instances and additional initial–panel experiments on 78 Science instances over 30 source questions. Figure 8 (Appendix D) illustrates the final selected Code harness procedure; Appendix E.3 provides its complete frozen prompt.

DomainMetricInitial (H0H_0)Fixed Luna panelOptimized (H⋆H^\star)
Code (98)Correct (↑\uparrow)61/98 (62.24%)50/98 (51.02%)73/98 (74.49%)
Mean cost (USD) (↓\downarrow)44.06 USD2.80 USD9.84 USD
Within 16 USD (↑\uparrow)24/98 (24.49%)98/98 (100.00%)82/98 (83.67%)
Mean utility (λ=1\lambda=1) (↑\uparrow)−1.2490.5100.684
Science (78)Correct (↑\uparrow)40/78 (51.28%)36/78 (46.15%)43/78 (55.13%)
Correct and within 16 USD (↑\uparrow)34/78 (43.59%)36/78 (46.15%)42/78 (53.85%)
Mean cost (USD) (↓\downarrow)9.13 USD1.05 USD5.72 USD
Within 16 USD (↑\uparrow)67/78 (85.90%)78/78 (100.00%)77/78 (98.72%)
Mean utility (λ=3.2\lambda=3.2) (↑\uparrow)0.3790.4620.542

Panel procedure and accounting. The scripted panels use predetermined calls and aggregation. New analysis of the original Code records establishes complete costs including all failed and retried attempts. All 13 invalid Code panel decisions count wrong.

AuthorsOpenReview ↗

Author Response (Part 3/3)

W2: The benchmark contribution needs stronger substantiation and reproducibility support. AVB contains 730 verification instances derived from 262 source questions, making curation and coverage central to its value. However, the main coding comparison uses only 98 of 270 test instances, and the selection rule for the “hard” subset is insufficiently specified. I could not locate an accessible code/data artifact in the supplied manuscript to independently check construction, filtering, labels, or environment reconstruction (Appendices A–B).

Q3: Benchmark clarification and reproducibility: Can the authors provide the construction code, frozen split/cohort manifests, grader/environment versions, and optimized harness? What determined the 98-instance coding subset, including all exclusions from the original 100-instance subset?

Thank you for raising this reproducibility concern; we agree that cohort selection and access to construction artifacts must be explicit.

Cohort selection. The Code test cohort is a stratified sample; "hard subset" was an inaccurate description, which we correct. Source questions are split 40.00%/10.00%/50.00% into train/validation/test, giving 270 test instances over 100 questions. Difficulty is the number of the 16 candidates that pass the source benchmark's executable grader; no LLM assigns correctness. To control cost, we sample 100 test instances proportionally across domain × generator × difficulty cells:

DifficultyPassing candidatesFull testSample
Hard1–49936
Medium5–1110239
Easy12–156925
Total1–15270100

Exclusions and sensitivity. After freezing the 270-instance Code test split and the stratified 100-instance sample, we excluded break-filter-js-from-html/gpt_5_4_mini for execution issues and regex-log/luna over leakage concerns, leaving 98 instances for the shared comparison. For the two exclusions, new worst-case arithmetic uses the observed 73/98 versus 61/98 correct (Sol, 16 USD, soft-overrun). Even if both restored cases were wrong for the optimized harness and correct for the initial harness, the totals would be 73/100 versus 63/100: the ordering cannot reverse.

Benchmark construction. An instance is a question–generator pair. Code has 541 instances over 192 questions, each with 16 trajectories; Science has 189 instances over 70 questions, totaling 730 instances over 262 questions. Science uses same-generator triples with zero, one or two correct candidates; NONE is correct only when all three fail. From Terminal-Bench 2.1 and DeepSWE v1.1, we keep 541 fully labeled Code instances that contain both passing and failing candidates, after removing observed hidden-test or reference-solution leakage. The full splits and evaluated test cohorts are:

BenchmarkTrainValidationTestEvaluated test
Code2175427098
Science69427878

Anonymous materials. Benchmark code, frozen split/cohort manifests, and optimized harnesses are available in our anonymous repository.


Paper revisions (in blue)

All changes are marked in blue in the revised manuscript:

  • Related Work. We clarified the distinctions from Agent-as-a-Judge and VerifiAgent and credited Meta-Harness for the optimizer.

  • §4.3 and Appendix G. We added the Code fixed-panel results with complete costs and invalid decisions counted wrong, and distinguish the separate runtimes.

  • §4.3 and Appendix H. We report exactly 42/78 optimized Science decisions correct within budget, eight more than the matched H0 run.

  • Conclusion and Limitations. We aligned the Science summary with the matched H0 comparison and distinguished the limitations of the comparison with the submitted H0 run.

  • Figure 8 (Appendix D) and Appendix E.3. We added a schematic and the complete frozen prompt of the final selected Code harness.

  • Appendix B. We specified stratified sampling, exclusions, cohort bounds, the released package’s contents, and its direct anonymous link.

Thank you again for the constructive feedback. We welcome follow-up questions about these results and the released materials.

Reviewer_5DtYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks a trained model to judge whether an agent has finished a task. Checking has a limited budget, so the model must decide what to inspect next and when to stop. The authors build AgentVerificationBench from existing coding and science tasks. The model has to pick a successful trajectory and, on the science tasks, reject every candidate when all of them fail. The weights stay fixed, and only the prompts and steps used for checking are changed. On the held-out tasks, this change costs less and stays within budget more often.

Critical strengths

There are two contributions. The first is AgentVerificationBench. It turns existing coding and science tasks into verification problems: select a successful trajectory, and reject every candidate when all of them fail. The evaluation records whether the choice is correct, what it costs, and whether it stays within budget.

The second is a method that leaves the model weights unchanged and edits only the prompts and steps used for checking. Its advantage is that both the science tasks and the coding tasks cost less and stay within budget more often.

Critical weaknesses

The saving comes with a drop in accuracy. The experiments do not separate a better decision about the next check from a cheaper fixed routine.

Questions for authors
  1. Once a procedure stays within budget, why not choose the one with higher accuracy? The present score penalizes overspending so heavily that a cheaper and less accurate procedure wins.

  2. This optimization only edits the checking procedure, and it should still be possible to combine it with updating the model weights. Do the authors have any ideas or plans for that?

  3. Have the authors considered the time taken to reach a decision, not only the dollar cost?

6 comments
AuthorsOpenReview ↗

Author Response (Part 1/4)

We thank the reviewer for recognizing (i) AVB's evaluation of accuracy, cost, and budget adherence and (ii) lower cost and better budget adherence from fixed-weight harness optimization. We address each concern below.

Summary

(W/Q1) Accuracy and cost. The optimized harnesses have higher observed accuracy and lower cost than the initial harnesses in both domains. They are more accurate than the fixed Luna panels at higher cost, as shown below the correction.

(Q2) Weight updates. We propose alternating weight and harness updates using the same utility [1, 2].

(Q3) Decision time. On 54 Code validation instances, median episode time falls from 72.8 to 21.4 minutes.

(Correction to the Science baseline): The submitted Science H0 run used a weaker overrun penalty than the optimized harness: 0.0625 utility per dollar over the 16 USD budget (normalized λ = 1) instead of 0.2 per dollar (λ = 3.2). It also used a different concurrency limit and model pool. We now compare against a matched H0 run that uses the optimized harness’s settings: 0.2 per dollar, Luna-only delegation, at most three concurrent calls, and a 28-turn cap. Even with these settings, on 78 test instances H0 → H* increases accuracy, budget adherence, and utility from 51.28%, 85.90%, and 0.379 to 55.13%, 98.72%, and 0.542, and lowers mean cost from 9.13 to 5.72 USD. The comparison with the submitted H0 run (47/78 correct, at least 23.20 USD mean cost) is retained in Appendix G.

DomainMetricInitial (H0H_0)Fixed Luna panelOptimized (H⋆H^\star)
Code (98)Correct (↑\uparrow)61/98 (62.24%)50/98 (51.02%)73/98 (74.49%)
Mean cost (USD) (↓\downarrow)44.06 USD2.80 USD9.84 USD
Science (78)Correct (↑\uparrow)40/78 (51.28%)36/78 (46.15%)43/78 (55.13%)
Mean cost (USD) (↓\downarrow)9.13 USD1.05 USD5.72 USD

Here H0 is the initial harness and H* the validation-selected harness for each domain. The Science H0 entries use the matched H0 run.

Anonymous materials. The released benchmark code, frozen cohorts, and optimized harnesses are available in the anonymous package, also linked in Appendix B.


W: The saving comes with a drop in accuracy. The experiments do not separate a better decision about the next check from a cheaper fixed routine.

Thank you for raising the accuracy–cost trade-off and the need to separate adaptive checking from fixed routines.

Observed accuracy and cost. At a nominal 16 USD budget, the optimized Sol harnesses have higher observed accuracy and lower cost than H0H_0 in both domains. The table adds the fixed Luna panels on the same 98 Code and 78 Science test instances. It combines the submitted harness results with additional control experiments and analysis.

DomainMetricInitial (H0H_0)Fixed Luna panelOptimized (H⋆H^\star)
Code (98)Correct (↑\uparrow)61/98 (62.24%)50/98 (51.02%)73/98 (74.49%)
Mean cost (USD) (↓\downarrow)44.06 USD2.80 USD9.84 USD
Within 16 USD (↑\uparrow)24/98 (24.49%)98/98 (100.00%)82/98 (83.67%)
Mean utility (λ=1\lambda=1) (↑\uparrow)−1.2490.5100.684
Science (78)Correct (↑\uparrow)40/78 (51.28%)36/78 (46.15%)43/78 (55.13%)
Correct and within 16 USD (↑\uparrow)34/78 (43.59%)36/78 (46.15%)42/78 (53.85%)
Mean cost (USD) (↓\downarrow)9.13 USD1.05 USD5.72 USD
Within 16 USD (↑\uparrow)67/78 (85.90%)78/78 (100.00%)77/78 (98.72%)
Mean utility (λ=3.2\lambda=3.2) (↑\uparrow)0.3790.4620.542

Fixed-panel comparison. The panels use predetermined calls and aggregation. They provide cheaper operating points, while the optimized harnesses have higher observed accuracy in both domains. New analysis establishes complete Code panel costs, including every failed and retried attempt; all 13 invalid Code panel decisions count wrong.

Science trade-offs. Relative to H0H_0, H⋆H^\star adds 3.85 percentage points in accuracy, lowers mean cost by 37.4%, and makes eight more correct decisions within budget. Its only over-budget decision was correct, giving exactly 43 − 1 = 42 correct decisions within budget. The panel is cheaper than H0H_0 on all 78 instances (panel − H0H_0 mean-cost difference: 95% source-question cluster-bootstrap percentile CI [−9.99, −6.23] USD; 30 clusters, 20,000 resamples).

Final Code harness. Figure 8 (Appendix D) illustrates the final selected Code harness procedure, including task-specific probes, disconfirmation, and evidence-dependent stopping; Appendix E.3 provides its complete frozen prompt.

AuthorsOpenReview ↗

Author Response (Part 2/4)

Q1: Once a procedure stays within budget, why not choose the one with higher accuracy? The present score penalizes overspending so heavily that a cheaper and less accurate procedure wins.

Thank you for asking how the objective ranks procedures within budget.

Within-budget preference. Yes: among procedures that stay within budget on every episode, our objective prefers the more accurate one. Section 3.3 defines uiλ(H)=yi(H)−λmax⁡(0,Ci(H)/Bi−1)u_i^\lambda(H)=y_i(H)-\lambda\max(0,C_i(H)/B_i-1). Here ii indexes instances, HH is the harness, yi(H)y_i(H) indicates a valid correct decision, Ci(H)C_i(H) is total cost, BiB_i is the nominal budget, and λ≥0\lambda\geq0 is the normalized overrun penalty. Spending within budget incurs no penalty, and spending less earns no extra reward.

Fixed-panel utility. The panels stay within budget on all 98 Code and 78 Science instances, so their utilities equal their accuracies: 0.510 and 0.462. The optimized harnesses attain higher utilities, 0.684 and 0.542, respectively, despite their higher mean costs.

Science within-budget example. In a separate new Science validation experiment, both H0H_0 and a cheaper candidate stay within 16 USD on all 42 instances. The objective selects the more accurate H0H_0: 24/42 (57.14%) correct at 7.80 USD mean cost, versus 19/42 (45.24%) at 7.19 USD; their utilities are 0.571 and 0.452, respectively.

Science validation selection. New rescoring of the seven submitted candidates on the common 38-instance validation cohort selects the same H⋆H^\star by accuracy or utility. H0→H⋆H_0 \to H^\star: 19/38 (50.00%) → 21/38 (55.26%) correct, 29.04 → 5.56 USD mean cost, and −2.337 → 0.553 utility.

Science test utility. The optimized harness has the highest evaluated utility on this split: 0.542, versus 0.428 for initial AAV and 0.290/0.261 for its one/two continuations.

Code objective sensitivity. New analysis of the three archived searches on the same 99 Code test instances (Appendix H.1) at a nominal 16 USD budget gives the results below.

Search objectiveCorrect (↑\uparrow)Mean cost (USD) (↓\downarrow)Within 16 USD (↑\uparrow)
Soft, normalized λ=0.2\lambda=0.270/99 (70.71%)17.64 USD49/99 (49.49%)
Soft, normalized λ=1\lambda=161/99 (61.62%)11.30 USD81/99 (81.82%)
Strict66/99 (66.67%)10.57 USD80/99 (80.81%)
AuthorsOpenReview ↗

Author Response (Part 3/4)

Q2: This optimization only edits the checking procedure, and it should still be possible to combine it with updating the model weights. Do the authors have any ideas or plans for that?

Thank you for suggesting joint harness and weight optimization.

Joint optimization. Yes, we have some ideas and plans for joint optimization, starting with alternating weight updates and harness optimization [1, 2] for a trainable Meta-Verifier. Our policy πθ,H\pi_{\theta,H} depends on both weights θ\theta and harness HH.

Joint objective and schedule. For joint updates, expose weight dependence in the same utility as uiλ(θ,H)u_i^\lambda(\theta,H). Its mean over instance set DD is Jλ(θ,H;D)=∣D∣−1∑i∈Duiλ(θ,H)J_\lambda(\theta,H;D)=|D|^{-1}\sum_{i\in D}u_i^\lambda(\theta,H), reducing to §3.3's Jλ(H;D)J_\lambda(H;D) at fixed θ\theta. From initial pair (θ0,H0)(\theta_0,H_0), use fixed budgets per alternating cycle kk and phase:

1. Update weights with HkH_k fixed. Initialize θ←θk\theta\leftarrow\theta_k and collect fresh TRAIN records Ek\mathcal{E}_k under πθ,Hk\pi_{\theta,H_k} with complete costs. Let Ek+\mathcal{E}_k^+ contain records with valid correct decisions and positive recorded utility. Apply utility-filtered rejection-sampling fine-tuning [2], minimizing Lk(θ)=−∑ei∈Ek+∑ℓlog⁡πθ,Hk(ai,ℓ∣qi,ti,R,hi,ℓ,bi,ℓ)∑ei∈Ek+∑ℓ∣ai,ℓ∣\mathcal{L}_k(\theta)=-\frac{\sum_{e_i\in\mathcal{E}_k^+}\sum_\ell\log\pi_{\theta,H_k}(a_{i,\ell}\mid q_i,t_i,\mathcal{R},h_{i,\ell},b_{i,\ell})}{\sum_{e_i\in\mathcal{E}_k^+}\sum_\ell |a_{i,\ell}|}. Record eie_i supplies task qiq_i, candidates tit_i, action ai,ℓa_{i,\ell}, prior history hi,ℓh_{i,\ell} and remaining budget bi,ℓb_{i,\ell} at step ℓ\ell, with resources R\mathcal{R}. Token count ∣ai,ℓ∣|a_{i,\ell}| normalizes loss Lk\mathcal{L}_k over parent-generated text; user/tool text is masked. Refresh rollouts under updated weights, skip empty accepted batches, and save θk+1\theta_{k+1}.

2. Update the harness with θk+1\theta_{k+1} fixed. Starting from HkH_k, use the harness optimizer A\mathcal{A} to propose revisions and evaluate the seed and candidates on common TRAIN task–budget pairs under the updated weights. Retain the training-JλJ_\lambda maximizer as Hk+1H_{k+1}.

3. Select and freeze. Evaluate the initial and saved pairs on validation set DvalD_{\mathrm{val}} and select (θ⋆,H⋆)(\theta^\star,H^\star) by maximum Jλ(θ,H;Dval)J_\lambda(\theta,H;D_{\mathrm{val}}). Freeze both before TEST. Both updates use TRAIN feedback only, and delegate weights stay fixed.

Reviewer_61SqOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines meta-verification. Specifically, the authors treat verification itself as a sequential decision rule, where the agent has autonomy to request specific actions to be verified. The agent in this scenario has a resource (e.g., token, financial cost) budget. Secondly, the authors study how to improve the verifier itself through harness (e.g., prompt, tools, etc.) optimization.

Critical strengths

S1. The paper discusses a timely and important problem.

S2. The benchmark results are notable, there are large cost savings on the real tasks the authors used.

Critical weaknesses

W1. Fig 1/2 are quite unclear: Figure 1: t1, t2, t3 not introduced, what are the nodes? Figure 2: Unclear what each of these steps means: what is “screen/audit” vs “comparison” vs “adversarial” etc

W2. In general, paper is hard to read and unclear figures.

Questions for authors

Can the agent choose different verifier models for each action? Are there any assumptions on the verifiers?

2 comments
AuthorsOpenReview ↗

Author Response (Part 1/2)

We thank the reviewer for recognizing (i) the timeliness and importance of agent verification and (ii) the substantial cost savings on real tasks. We address each concern below.

Summary

(Q1) Per-call verifier choice. Yes. The Meta-Verifier chooses a model for each delegation and may override its judgment.

(W1) Figures 1 and 2. We redrew both figures with labeled rows, legends for every glyph, and self-contained captions.

(W2) Readability. We define notation at first use, standardize model names, and summarize the evaluation cohorts in the main text.

(Correction to the Science baseline): The submitted Science H0 run used a weaker overrun penalty than the optimized harness: 0.0625 utility per dollar over the 16 USD budget (normalized λ = 1) instead of 0.2 per dollar (λ = 3.2). It also used a different concurrency limit and model pool. We now compare against a matched H0 run that uses the optimized harness’s settings: 0.2 per dollar, Luna-only delegation, at most three concurrent calls, and a 28-turn cap. Even with these settings, on 78 test instances H0 → H* increases accuracy, budget adherence, and utility from 51.28%, 85.90%, and 0.379 to 55.13%, 98.72%, and 0.542, and lowers mean cost from 9.13 to 5.72 USD. The comparison with the submitted H0 run (47/78 correct, at least 23.20 USD mean cost) is retained in Appendix G.

DomainMetricInitial (H0H_0)Fixed Luna panelOptimized (H⋆H^\star)
Code (98)Correct (↑\uparrow)61/98 (62.24%)50/98 (51.02%)73/98 (74.49%)
Mean cost (USD) (↓\downarrow)44.06 USD2.80 USD9.84 USD
Science (78)Correct (↑\uparrow)40/78 (51.28%)36/78 (46.15%)43/78 (55.13%)
Mean cost (USD) (↓\downarrow)9.13 USD1.05 USD5.72 USD

Here H0 is the initial harness and H* the validation-selected harness for each domain. The Science H0 entries use the matched H0 run.

Anonymous materials. The released benchmark code, frozen cohorts, and optimized harnesses are available in the anonymous package, also linked in Appendix B.


Q1: Can the agent choose different verifier models for each action? Are there any assumptions on the verifiers?

Thank you for asking about per-call model choice and verifier assumptions.

Per-call model choice. Yes. As specified in the submitted §3.1 and Appendix E, each delegation selects a model from the configured pool, the candidates to inspect, and the check objective. The next call can use a different model, including one stronger than the Meta-Verifier, based on prior evidence and remaining budget. Both parent and delegate costs count toward the budget.

Verifier assumptions and inductive biases. Within the task, resource, and access constraints, verifiers may take any action. We leave the checking procedure open to minimize hand-designed heuristic inductive biases. We do not assume correctness, calibration, or independence, and the Meta-Verifier may override their judgments. Correctness labels and private grading information remain hidden. Appendix E lists available models, prices, and context limits.

Fixed-panel reference. Our scripted baselines fix delegation to Luna and use predetermined calls and aggregation. The summary table reports their accuracy and cost in both domains alongside the initial and optimized Meta-Verifiers.

Observed model choice. With the initial harness, the Meta-Verifier called more than one verifier model in 28 of 31 Code training episodes (90.3%; Appendix D.1). As illustrated in Figure 2, the Meta-Verifier can combine different models and verification strategies, including screening, comparisons, adversarial checks, and its own executable checks.

AuthorsOpenReview ↗

Author Response (Part 2/2)

W1: Fig 1/2 are quite unclear: Figure 1: t1, t2, t3 not introduced, what are the nodes? Figure 2: Unclear what each of these steps means: what is “screen/audit” vs “comparison” vs “adversarial” etc

Thank you for pointing out the unclear figure labels and verification roles.

Figure 1 revision. We replaced the task labels with "Task 1–3" and defined each glyph in the legend below. Panels (a) and (b) are schematic procedures: panel (a) shows a prescribed verifier that runs the same procedure on every task, and panel (b) a Meta-Verifier that builds a different procedure per task. Panel (c) optimizes the reusable harness across training tasks with model weights fixed, freezes it after validation selection, then applies it to new tasks.

GlyphMeaning
DotsThe task's candidate trajectories
SquareExecutable check (e.g., running a candidate's code)
CircleModel call, such as a judgment or a delegated check; larger circles cost more
Dashed arrowThe procedure is revised after seeing evidence

Figure 2 revision. We identify the parent, verifier models, and call roles in each run on the same Science test task, with roles defined below. Colour gives the verifier model. The roles describe observed calls rather than mandatory algorithm stages. A shared cost axis shows the optimized run selecting a correct candidate at 7.56 USD.

RoleShapeWhat the delegated call does
Screen/auditCircleChecks one candidate against the task requirements
ComparisonDiamondJudges several candidates against each other
Adversarial checkTriangleLooks for a failure in one candidate

W2: In general, paper is hard to read and unclear figures.

Thank you for the constructive comments on readability.

Readability revisions. We revised the text and figures for readability (marked in blue):

Notation at first use. The captions of Figures 1–3 now define H0H_0 (initial harness), H⋆H^\star (optimized harness), BB (budget), the parent Meta-Verifier, and candidate labels; plot axes use plain labels (selection accuracy, mean cost in USD); harness search uses one term (epoch) throughout.

Consistent names. The two harnesses are called initial and optimized throughout the text and plots, models are named by version (e.g., Sol 5.6), and benchmark versions are listed (DeepSWE v1.1).

Evaluation cohorts. A new paragraph in §4.1 summarizes every cohort (Code: 270 → stratified 100 → common 98, and 99 for execution modes; Science: 17 training tasks per candidate, 38 validation tasks, 78 test tasks).

Accuracy–cost presentation. The summary table compares the initial harness, fixed Luna panel, and optimized harness on 98 Code and 78 Science test instances at a nominal 16 USD budget. It combines submitted harness results with additional control experiments and analysis of their costs. In both domains, the panel is cheaper and less accurate than the optimized harness.

Final Code harness. Figure 8 (Appendix D) illustrates the final selected Code harness procedure, and Appendix E.3 provides its complete frozen prompt.

Historical comparisons. The submitted H0 run remains separate, with its baseline plot in Appendix G and labeled trace/resource analyses.

Paper revisions (in blue)

All changes are marked in blue in the revised manuscript:

  • Figures 1 and 2. We redrew the existing figures with labeled rows, glyph and role legends, and self-contained captions.

  • §3.1. We clarified per-call model choice and verifier assumptions.

  • Figure 8 (Appendix D) and Appendix E.3. We added a schematic and the complete frozen prompt of the final selected Code harness.

  • Figures 1–3 captions, §3.3, §4.1, Table 1, Figures 4–6. We defined notation, standardized names, summarized cohorts, and specified the updated Science comparison while distinguishing the original plots and trace analyses.

  • §4.3 and Appendix G. We added the Code fixed-panel results with complete costs and invalid decisions counted wrong, and distinguish the separate runtimes.

  • §4.3 and Appendix H. We report exactly 42/78 optimized Science decisions correct within budget, eight more than the matched H0 run.

  • Conclusion and Limitations. We aligned the Science summary with the matched H0 comparison and distinguished the limitations of the comparison with the submitted H0 run.

We thank the reviewer again for the constructive feedback. We hope these changes address the points raised, and we would be grateful if the reviewer would take them into account in the final assessment. We are happy to provide further details during the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers and AC,

We sincerely appreciate your valuable time and effort spent reviewing our manuscript.

As reviewers highlighted, our work has the following strengths.

  • Resource efficiency. Lower verification costs and improved budget adherence through harness optimization with fixed model weights (5DtY, Nfve).
  • Executable benchmark. Fixed candidate sets, executable correctness labels, and source-task-disjoint splits (Nfve).
  • Budgeted verification. A timely study of how agents select verification actions under resource constraints (61Sq).

Your feedback guided our revisions to the manuscript, including the additional analyses and experimental results listed below.

  • Science comparison against a matched initial run showing that the optimized harness is more accurate, cheaper, and within budget more often (§4.3, Appendix G)
  • Fixed Luna-panel comparisons in both Code and Science domains showing higher optimized-harness accuracy and the most correct decisions within budget (Appendices G and H)
  • Lower-budget optimized Code harness with higher accuracy and lower cost than the fixed Luna panel (Figure 4, Appendix G)
  • Highest Science test utility at every penalty weight, including accuracy only (Appendix G)
  • Code penalty sensitivity across separate searches and lower median Code validation episode time (Appendix H)
  • Same penalty, verifier pool, and budget for the Science matched initial and fixed-panel runs (Appendix G)
  • Per-call verifier model choice (Appendix D.1) and the final selected Code harness with its complete prompt (Figure 8, Appendix E.3)
  • Redrawn Figures 1 and 2, updated Figure 4, and a new Figure 8
  • Stratified Code test sample, exclusions, and anonymous benchmark code, frozen cohorts, and optimized harnesses (package, Appendix B)

To help reviewers follow the revisions, we temporarily mark all changes in blue\textcolor{blue}{\text{blue}}. We hope these revisions provide stronger support for our contributions and address the concerns raised in the reviews.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.