acceptodds
Under review as a conference paper at ICLR 2027

Trust Is a Process, Not Just a Score: Agentic Verification for Test-Time Scaling

12 views

Abstract

While test-time scaling improves large language models (LLMs) by generating multiple candidates, verification is often treated as a terminal scoring problem, separate from the acquisition and maintenance of evidence. In this paper, we explore verification as an interleaved process of gathering observations and updating a validity-aware state: execution probes, knowledge-base traversals, and model adjudication distinguish candidates, while the state tracks dependence, supersession, disclosure, and repair. We instantiate this approach, named GEVA (Governed Evidence Verification and Audit), as a deterministic state–action–observation loop over a typed claim–evidence graph, with certificate-based audits governing evidence producers. Across seven regimes spanning code, text-to-SQL, knowledge-graph question answering, and multiple-choice reasoning, GEVA improves over greedy decoding throughout. On MetaQA-3hop, it reaches , points above common-pool Fixed V; matched-evidence controls show that this recovery comes from executed retrieval rather than graph storage alone. On a disjoint held-out split, producer audit restores accuracy points by withdrawing a harmful judge. Event replay further reproduces full recomputation with nearly fewer claim assessments. These results demonstrate the benefits of jointly acquiring, maintaining, and governing verification evidence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic Verification, Test-Time Scaling, Provenance Graphs, Verifier Auditing, Independence-Aware Verification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8qppOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that when large language models (LLMs) are used for test time scaling, "verification" should not merely consist of assigning a score to a candidate answer at the end; rather, it should be an ongoing process of gathering evidence while continuously maintaining the validity of that evidence. To address this, the authors propose the GEVA (Governed Evidence Verification and Audit) framework. This framework employs a deterministic controller that operates on a "claim-evidence graph," iteratively invoking mechanisms such as execution probes, knowledge base traversal, model-based adjudication, and error correction. Throughout this process, it tracks the provenance, dependencies, and validity of the evidence, while using label-agnostic auditing to prune harmful sources of evidence. Experiments across seven task categories demonstrate that GEVA consistently outperforms greedy decoding. Notably, on the MetaQA-3hop task, GEVA achieves an improvement of 11.10 percentage points over the "Fixed V" baseline (using the same candidate pool). Furthermore, removing harmful adjudicator models yields a recovery of 4.38 percentage points in accuracy. Finally, the results from incremental replay are consistent with those from full re-computation, while requiring approximately eight times fewer evaluations.

Critical strengths
  1. This paper proposes structuring the verification process as a workflow that simultaneously gathers evidence and ensures its ongoing validity, this approach I find both novel and practically significant.
  2. The author has conducted extensive testing across a variety of benchmarks.
  3. The author formalizes the verification process as a multi-stage evidence review. While this adds complexity to the overall verification procedure, it does indeed result in performance improvements.
Critical weaknesses
  1. Certain terms in the paper, such as "Fixed V," are not sufficiently explained.
  2. There are formatting issues; for instance, the LaTeX rendering of parameters around line 336 appears to be problematic. The author should carefully review the manuscript.
  3. The author has conflated parts of the main text with content that should instead be placed in an appendix—for example, the first and second paragraphs of Section 4.5.
  4. The models and methods used for comparison in Section 4.4 are somewhat outdated. I suggest that the author actively seek out and include comparisons with architectures similar to the one proposed in this paper—such as the work related to "tool feedback" mentioned in the "Related Work" section.
Questions for authors
  1. Could the authors provide a further explanation of how "Fixed V" functions?
  2. What is the significance of the two different colors used for the columns in Table 3? This does not appear to be addressed in the main text.
  3. In the section detailing the first experiment, the content consists primarily of a presentation of experimental data, with very few conclusions drawn. Could the authors revise this section to provide a summary that includes concrete conclusions?
3 comments
AuthorsOpenReview ↗

How `Fixed V` Functions & Visual Key for Table 3 (W1, Q1, Q2)

We greatly value your encouraging feedback on our work—in particular your recognition of structuring verification as a validity-governed workflow across diverse benchmarks and formal stages. To address each of your points thoroughly while keeping manuscript edits surgical, we organize our reply to you into three focused parts: (1/3) W1, Q1 & Q2, (2/3) W2, W3 & Q3, and (3/3) W4.

1. How Fixed V Functions Across All Seven Regimes (W1, Q1; §1, §4.1, Table 24 in Appendix O)

You noted that Fixed V was insufficiently explained (W1) and asked for a clear explanation of how it functions (Q1). In §1 and §4.1 (with full summary table and code-level mappings in Table 24, Appendix O), we now define Fixed V explicitly upfront: on the same ordered initial candidate pool evaluated by Majority and GEVA—a nonempty greedy output plus 88 temperature-0.80.8 draws (9 candidates, or 8 on the 12 MathQA-Geo cases with empty greedy output; Appendix O)—Fixed V scores each candidate once using only the task's initial public-input verifier Vx(ci)V _ x(c _ i) and selects the earliest candidate achieving the maximum score (j⋆=min⁡arg⁡max⁡iVx(ci)j^\star = \min\arg\max _ i V _ x(c _ i)), performing zero post-pool probes, database queries, exact-hop traversals, judge calls, or repairs:

RegimePublic-Input Verifier Vx(ci)V _ x(c _ i) Used by Fixed V (§4.1; Table 24, Appendix O)
MBPP+ / HumanEval+Pass fraction on visible prompt base tests (base_pass / base_total; 0 if none)
SpiderMean of 4 pre-execution SQL checks: syntax parse, grounded tables, schema bare columns, direct FK links
MetaQA-1hop / -3hopKeyword-matched global-KB reachability (1.0 typed kk-hop reachable, 0.6 untyped reachable, 0.1 KB node, 0 else)
KQA ProDeterministic record-local Wikidata subgraph reachability (1.0 typed neighborhood, 0.3 subgraph node, 0 else)
MathQA-Geo0.51[executes]+0.31[matches declared option]+0.21[valid numeric option]0.5\mathbf{1}[\text{executes}] + 0.3\mathbf{1}[\text{matches declared option}] + 0.2\mathbf{1}[\text{valid numeric option}]

In addition, we applied the deterministic Wikidata subgraph scorer uniformly to KQA Pro's greedy decode in Table 1, completing all 13 evaluated configurations (7 on Qwen3.5-9B and 6 on Qwen3.6-27B; 27B MathQA-Geo was not evaluated) with zero missing -- entries (+5.50 pp on 9B, p=2.2×10−7p{=}2.2\times 10^{-7}; +3.50 pp on 27B, p=2.2×10−4p{=}2.2\times 10^{-4}).

2. Meaning of the Column Colors and Boldface in Table 3 (Q2; §4.5, Table 3)

Because we preserved the manuscript's original float layout, Table 3 in your review remains Table 3 in §4.5. Since its visual encoding was previously left implicit, we updated Table 3's caption to state all three visual cues explicitly:

  • Blue column shading highlights the three columns belonging to our method (GEVA at N=8,16,32N{=}8, 16, 32).
  • Gray text marks the sealed-outcome Oracle pool ceiling (whether any candidate in greedy + SNS _ N is correct—an upper bound, not a deployable selector).
  • Boldface marks the row-wise highest GEVA accuracy across N in {8, 16, 32} (indicating within-row pool-scaling direction rather than statistical significance).
MetaQA-3hop Scaling (Table 3, §4.5)N=8N=8N=16N=16N=32N=32Scaling Trend (N=8→32N=8\to 32)
Fixed V (samples / greedy-incl.)28.60% / 31.40%27.20% / 30.20%27.00% / 29.80%Degrades (-1.60 pp greedy-incl., p=0.0090p=0.0090)
GEVA (blue columns; row-max bold)41.40%44.10%48.30%Climbs +6.90 pp (+18.50 pp vs Fixed V at N=32N{=}32)
Oracle-in-pool (gray text ceiling)46.90%51.90%57.00%Pool ceiling expands +10.10 pp

As Table 3 shows, drawing more candidates increases false-positive relation ties that hurt static Fixed V, whereas GEVA's executed exact-hop path verification converts pool diversity into steady gains (+5.30 / +6.90 / +2.40 pp from N=8→32N{=}8\to 32 on MetaQA-1hop / MetaQA-3hop / KQA Pro).

AuthorsOpenReview ↗

LaTeX Formatting Fixes, Streamlining §4.5 & Concrete Takeaways in §4.2 (W2, W3, Q3)

Continuing from Part 1, we are very thankful for your careful line-by-line reading, which caught formatting and structural friction in §3.4, §4.2, and §4.5 (W2, W3, Q3) and helped us noticeably improve the flow of our revised paper.

1. LaTeX Parameter Formatting & Full Compilation Audit (W2; §3.4 & §§3–4)

Around line 336 (W2), bare leading-dot decimals ($.803$, $.983$, $.90$, $\rho=.35$) rendered awkwardly next to surrounding punctuation. We audited the entire manuscript and appendices, normalized all parameter values and thresholds to explicit leading-zero math formatting ($0.803$, $0.983$, $0.90$, $\rho=0.35$, $C _ {\min}=0.60$, etc.), and verified clean compilation across all 41 pages (10 main-body pages, 0 LaTeX errors, 0 undefined references, and 0 overfull boxes).

2. Streamlining §4.5 and Moving Granular Archive Details to the Appendix (W3; §4.5, Appendices L–N)

Following your suggestion on the first two paragraphs of §4.5 (W3), we removed inline archive-history caveats from §4.5 and moved granular artifact notes, CPU subphase breakdowns (Table 14), and full perturbation tables (Tables 16–17) into Appendices L–N. §4.5 now focuses cleanly on four empirical questions: nested-pool scaling (Table 3), adaptive versus full-revelation budgets (Table 4), end-to-end single-H20 closed-loop runtime and token trade-offs (Table 15), and post-repair lifecycle validity.

3. Explicit Cross-Regime Conclusions Added to §4.2 (Q3; Table 1)

You noted that §4.2 previously read mainly as a listing of numbers without clear high-level takeaways (Q3). We revised §4.2 to synthesize Table 1 around three explicit conclusions:

  1. Substantial gains under compositional multi-hop ambiguity: When initial reachability heuristics suffer frequent ties or inversions across multi-hop relations (Figure 2), active evidence acquisition yields large improvements—lifting MetaQA-3hop by +11.10 pp on Qwen3.5-9B (42.70% vs. 31.60%, p=6.5×10−26p{=}6.5\times 10^{-26}), +7.30 pp on Qwen3.6-27B (44.10% vs. 36.80%, p=1.4×10−18p{=}1.4\times 10^{-18}), and +4.50 pp on Mistral-7B (Table 13), while lifting KQA Pro by +5.50 pp on 9B (p=2.2×10−7p{=}2.2\times 10^{-7}) and +3.50 pp on 27B (p=2.2×10−4p{=}2.2\times 10^{-4}).
  2. Meaningful gains from execution probes and counterexample repair on 9B code and SQL: Where public prompt assertions under-specify edge cases or SQL join semantics, GEVA gains +2.91 pp on 9B MBPP+ (p=0.00342p{=}0.00342: +1.32 pp from private probes pre-repair plus +1.59 pp from versioned counterexample repair, p=0.031p{=}0.031, six improved, none harmed; Table 19) and +1.74 pp on 9B Spider (p=0.0114p{=}0.0114) via database execution checks and selective gating.
  3. Small or unresolved differences on near-ceiling pools vs. certificate-free MathQA-Geo: Differences relative to Fixed V are small where stronger 27B pools or visible prompt tests already rank candidates near ceiling (27B MBPP+/Spider/HumanEval+ +0.79/+0.68/0.00+0.79/+0.68/0.00 pp; 9B HumanEval+ +0.61+0.61 pp; MetaQA-1hop −0.10-0.10 pp). Distinctly, on certificate-free MathQA-Geo (86.92% vs. 87.80%, −0.89-0.89 pp, p=0.541p{=}0.541), faulty symbolic derivations can still produce the right multiple-choice letter (0.958 negative precision, Table 18); because sound wrongness certificates are unavailable, GEVA retains graded evidence without operational vetoes, avoiding uncertified rejections.
AuthorsOpenReview ↗

Comparisons with Modern CRITIC-Style Tool-Feedback & Neural Selectors (W4)

Rounding out our reply (Part 3), we want to thank you for W4: your suggestion to go beyond the tabular selectors in §4.4 (Scalar, LR, GBDT) and compare head-to-head against modern iterative tool-feedback architectures discussed in §2 Related Work inspired the centerpiece new experiment of this revision. Below we detail both the neural selector baselines in Appendix E and our new end-to-end single-H20 tool-feedback benchmark in §4.5 (Table 15; Appendix L.1).

1. Neural Set Selectors Under Zero-Label Transfer (§4.4, Table 7 in Appendix E)

Alongside Figure 3's target-label learning curves for the three tabular selectors, Table 7 (Appendix E) evaluates five supervised flat-ledger selectors including permutation-invariant DeepSets and multi-head Set-Transformer. While fitted models are competitive when given in-domain target labels, under realistic leave-one-regime-out transfer (zero target-domain labels, matching GEVA's operating regime), all five supervised selectors trail zero-label GEVA on Spider (63.93%–68.86% vs. 70.21%), MetaQA-3hop (35.10% Set-Transformer and 37.80% DeepSets vs. 42.70%), and KQA Pro (54.70%–61.60% vs. 61.90%).

2. New Closed-Loop Comparison vs. CRITIC-Style Tool-Feedback on One H20 GPU (§4.5, Table 15; Appendix L.1)

To compare directly against modern iterative tool-feedback workflows on Qwen3.5-9B and Qwen3.6-27B, we implemented TF+cache (CRITIC-style) (following Gou et al., 2024: an LLM controller inspects candidates and tool outputs and iteratively invokes sandboxes, SQL execution, KB path probes, external adjudication, or code repair) and FO+cache (fixed-order bench/property -> KB path -> model probe -> adjudication escalation, with shared operational-veto repair eligibility and charged revalidation), alongside wider static sampling (Fixed V at N=12N{=}12, consuming 2.2%–13.1% more actual tokens than GEVA). Every method receives GEVA's exact lifecycle cache and is billed end-to-end on one H20 GPU for Stage-0 generation, tools, controller prompts, and repairs under shared wall-clock and token caps:

H20 Closed-Loop Benchmark (Table 15)Fixed V (N=12N{=}12)TF+cache (CRITIC-style)FO+cacheGEVA
A. MBPP+ (378; Audit=Off)76.72% (12.6s / 6.29k)78.04% (18.9s / 10.86k)78.57% (15.1s / 7.48k)79.10% (12.8s / 6.14k)
A. Spider (1,034; Audit=Off)68.18% (8.1s / 6.62k)69.05% (15.3s / 13.18k)69.44% (9.8s / 7.91k)69.73% (8.2s / 6.48k)
A. MetaQA-3hop (1,000; Audit=Off)30.60% (3.8s / 2.24k)41.10% (6.8s / 3.96k)41.30% (4.1s / 2.19k)42.50% (3.5s / 1.98k)
B. MetaQA-3hop + 27B Judge (800; Audit=On)— (200/800 split)39.88% (8.46s / 4.64k)39.13% (7.92s / 4.19k)43.50% (4.62s / 2.61k)

Two conclusions highlight what GEVA's structured controller adds over unconstrained tool-feedback loops:

  1. Substantially lower latency and token cost without sacrificing accuracy (Panel A): Because GEVA routes tools via analytical marginal utility per unit cost (Eq. 4) rather than multi-turn LLM orchestration prompts, it reduces end-to-end latency by 32.3%–48.5% and total tokens by 43.5%–50.8% relative to TF+cache (CRITIC-style), while achieving +1.40 pp higher accuracy on MetaQA-3hop (p=0.020p{=}0.020; MBPP+ and Spider accuracy CIs span zero at 32.3%–46.4% lower latency) and beating wider static sampling Fixed V (N=12N{=}12) across all three benchmarks (+2.38 / +1.55 / +11.90 pp, p≤0.0225p\le 0.0225).
  2. Resilience to misaligned external evaluators via label-free auditing (Panel B): When an external Qwen3.6-27B judge is available on MetaQA-3hop, unaudited FO+cache and TF+cache (CRITIC-style) repeatedly invoke the biased judge and drop to 39.13% and 39.88%. Even after charging GEVA the full 200-case unlabeled calibration pass amortized across the 800 held-out queries (Ccal/800=+1.14 s,+630 tok/caseC _ {\text{cal}}/800 = +1.14\text{ s}, +630\text{ tok/case}), GEVA's certificate audit (Eq. 5) withdraws the harmful judge automatically, gaining +4.38 pp over FO+cache (p=5.1×10−6p{=}5.1\times 10^{-6}, -41.7% latency) and +3.63 pp over TF+cache (CRITIC-style) (p=1.1×10−4p{=}1.1\times 10^{-4}). We are glad to discuss any of these results further during the discussion period.
Reviewer_nAsFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes GEVA, a verification method for best-of-N selection. A frozen model generates nine candidates per question, and a deterministic controller chooses which candidate to output and whether to accept it. Instead of scoring candidates once, the controller can run additional checks: hidden test probes, knowledge-base traversals, judge-model adjudication, or counterexample-driven repair. It records the results in a graph that tracks each piece of evidence's source, correlation with other evidence, and whether it has been superseded or disclosed to the model. It also includes a label-free procedure for removing unreliable evidence sources.

Experiments cover code generation, text-to-SQL, knowledge-graph QA and multiple-choice geometry. The paper reports a large gain on MetaQA-3hop (+11.10 points over the task verifier) and small or no gains elsewhere.

Critical strengths

The problem is legitimate and is useful: verifiers should account for correlated evidence and for evidence that becomes stale after a candidate is revised. The paper also keeps gold labels strictly out of the verification pipeline, which is good evaluation practice. However, the presentation prevents me from judging how much of the claimed contribution is substantiated beyond the MetaQA-3hop result.

Critical weaknesses

The paper is quite difficult to follow for me, and the presentation honestly weakens the contribution. The text relies on dense phrases that are often abstract nouns ("validity-aware state", "certificate-based audits governing evidence producers", "matched-evidence controls") without any concrete examples. I reached the end of the abstract and introduction without a clear picture of the task, the inputs, or what the method does differently from the existing works. There are sentences like a YouTube script and less like scientific text:

“Acquisition expands the available evidence; management tracks its applicability, dependence and permitted reuse, informing both selection and acceptance. “

The clearest motivating case, MetaQA's keyword-based relation guess being replaced by an executed KB traversal, appears only in Appendix M; it should appear in the introduction.

Several key elements are used before they are defined or are defined only in the appendices, including the Fixed V baseline, the components of Eq. 4, and Proposition 4. The manuscript also reads as a revision history: current results appear alongside "historical", "archived" and "pre-correction" ones.

So it is hard to tell which numbers support the claims, and the density of disclaimers obscures the findings. I strongly recommend restructuring the paper and rewriting to make it accessible to reviewers. At the very least

  1. defining all baselines and symbols in the main text, and introducing the concepts before they are references or used. Avoid buzzwords and abstract nouns or use them wisely.
  2. separate superseded results, and consolidating caveats into a single limitations section.
Questions for authors

What exactly is the Fixed V baseline for each task, and why is it the right comparison?

I don’t think I have any more questions because I wasn’t able to understand much

3 comments
AuthorsOpenReview ↗

Targeted Clarity Revisions, Plain-Language Rollouts & Motivating Walkthrough

Your candid feedback on presentation clarity helped us look at our manuscript with fresh eyes, and we are glad you found the underlying problem—accounting for correlated evidence and invalidating stale checks after candidate revision while keeping gold labels strictly outside verification—legitimate and practically useful. You pointed out clearly that abstract noun phrases, late definitions (Fixed V, Eq. 4, Proposition 4), and revision-history disclaimers made our draft difficult to follow. We took every part of your feedback to heart and organize our reply to you into two focused parts: (1/2) our targeted readability revisions and motivating MetaQA walkthrough, and (2/2) the exact Fixed V definitions across all seven regimes and why Fixed V is the right comparison (Q1).

1. Surgical Readability Improvements Without Disrupting Manuscript Structure

Rather than rewriting the entire paper from scratch, we made targeted edits in Abstract, §1, §3, §4, and §5 while preserving the original section layout, main-text Tables 1–4, and Figures 1–3:

  • Concrete prose before formalism (Abstract & §1): We removed abstract buzzword sentences (including the sentence in §1 that you quoted) and opened §1 with the concrete task inputs, candidate pool structure, and an explicit real MetaQA-3hop trace before any formal notation.
  • Upfront and inline definitions (§1, §3.2, §3.5, §4.1): §1 and §4.1 define single-pass Fixed V upfront across all seven regimes, paired with a complete summary table and executable code mappings in Table 24 (Appendix O). In §3.5, the main text explains Eq. 4's main terms (GraphCut, CorrPenalty, Δ^\widehat{\Delta}) inline alongside β=0.25,γ=0.30\beta{=}0.25, \gamma{=}0.30, with full router constants in Appendix C. In §3.2, the main text summarizes our incremental equivalence result in plain prose, with the formal statement and proof of Proposition 4 in Appendix N.1.
  • Separating superseded snapshots & consolidating caveats (§3, §4, §5): We removed revision-history remarks ("legacy policy", "revised retention rule") from the main text, confined superseded pre-correction snapshots to the appendix, presented all current evaluations with their respective protocols, and consolidated scope boundaries in §5.

2. Plain-Language Contrast: Generation vs. Verification Rollouts (§1, §3.1, Figure 1)

To make the setup transparent from page 1, §1 contrasts two processes over a frozen generator in plain English:

  • Generation rollout: Given public task input xx (a coding prompt, database schema, or KG question), the frozen LLM samples an initial candidate pool CxC _ x (a nonempty greedy output plus NN stochastic draws).
  • Verification rollout: Starting from that identical frozen pool, GEVA executes a step-by-step verification process (Figure 1)—invoking code/SQL sandboxes, KB path traversals, judges, or counterexample repairs, revoking stale support when a candidate is revised, and returning both a top-ranked answer and a gate-based ACCEPT / UNACCEPTED decision without modifying model weights.

3. Motivating Recorded MetaQA Walkthrough in §1 & Appendix O (Table 25)

Following your recommendation, we brought recorded instance metaqa_official_3hop_10001 ("the movies that share actors with the movie [Luther] were in which genres") directly into §1 (full trace in Table 25, Appendix O):

Candidate (Draws)Initial VVExact-Hop PathsPath StrengthFinal SSFinal RiskRollout Outcome
Action (3, incl. greedy)0.630.4440.5600.787Fixed V pick (wrong; starts with written_by)
Crime (1)0.630.4440.5300.799Rejected
Drama (4, gold)0.681.0000.7680.620GEVA rank 1 (UNACCEPTED, 0.620>0.350.620 > 0.35)
Biography (1)0.640.3060.4730.835Rejected (0.5x substring discount)

Because all nine candidates lie within three KB hops of Luther, shallow reachability scores every candidate V=0.6V{=}0.6, leaving Fixed V tied on the wrong greedy answer Action. GEVA invokes ACQUIRE-KB-PATH, supersedes the relation guess, and verifies 8 exact 3-hop film -> actor -> film -> genre paths for Drama (path strength = 1.000, S=0.768S{=}0.768), ranking Drama first while keeping it UNACCEPTED because its evidence-conditioned risk score (0.620) exceeds ρ=0.35\rho{=}0.35.

AuthorsOpenReview ↗

Exact `Fixed V` Definitions & Why It Is the Right Comparison (Q1)

Following up on Part 1, we appreciate your asking directly in Q1: "What exactly is the Fixed V baseline for each task, and why is it the right comparison?" Below, we lay out the exact executable rule for every task (§1, §4.1; Table 24, Appendix O) and explain why Fixed V pairs naturally with our closed-loop baselines under shared caps.

1. What Fixed V Computes on Each Task (§1, §4.1; Table 24, Appendix O)

For every task input xx, all common-pool methods in Table 1 (Majority, Fixed V, and GEVA) share the same ordered initial candidate pool CxC _ x—a nonempty greedy output followed by 8 sampled draws, preserving draw multiplicity (9 candidates, or 8 on the 12 MathQA-Geo instances with empty greedy output; Appendix O). Fixed V is single-pass best-of-NN selection: it evaluates each candidate once using only the task's initial public-input verifier Vx(ci)V _ x(c _ i) and returns the earliest candidate maximizing Vx(ci)V _ x(c _ i) (j⋆=min⁡arg⁡max⁡iVx(ci)j^\star = \min\arg\max _ i V _ x(c _ i)), executing zero post-pool probes, database queries, exact-hop path checks, judge calls, or repairs:

RegimeInitial Public-Input Verifier Vx(ci)V _ x(c _ i) Used by Fixed V (§4.1; Table 24, Appendix O)
MBPP+ / HumanEval+Pass fraction on visible prompt base assertions (base_pass / base_total; 0 if none)
SpiderMean of 4 pre-execution SQL checks: syntax parse, grounded tables, schema bare columns, direct FK links
MetaQA-1hop / -3hopKeyword-matched global-KB reachability (1.0 typed kk-hop reachable, 0.6 untyped reachable, 0.1 KB node, 0 else)
KQA ProDeterministic record-local Wikidata subgraph reachability (1.0 typed neighborhood, 0.3 subgraph node, 0 else)
MathQA-Geo0.51[executes]+0.31[matches declared option]+0.21[valid numeric option]0.5\mathbf{1}[\text{executes}] + 0.3\mathbf{1}[\text{matches declared option}] + 0.2\mathbf{1}[\text{valid numeric option}]

In Table 1, we also rescored KQA Pro's greedy candidate uniformly with the deterministic Wikidata subgraph verifier above, filling the two previously missing -- cells (56.40% on 9B vs. 61.90% GEVA, +5.50 pp, p=2.2×10−7p{=}2.2\times 10^{-7}; 63.60% on 27B vs. 67.10% GEVA, +3.50 pp, p=2.2×10−4p{=}2.2\times 10^{-4}) across all 13 evaluated configurations.

2. Why Fixed V Is the Right Primary Baseline—and How Closed-Loop Baselines Complement It

  1. Cleanly isolates multi-step verification from initial candidate quality and initial scoring: Because GEVA initializes G0G _ 0 with the exact same candidate pool CxC _ x and initial Vx(ci)V _ x(c _ i) checks as Fixed V, the difference GEVA minus Fixed V in Table 1 isolates the net contribution of active evidence acquisition, provenance-aware aggregation, selective gating, and versioned repair over standard static best-of-NN reranking.
  2. Reveals when static scoring breaks under larger candidate pools: In Table 3 (N in {8, 16, 32}), Fixed V tests whether simply drawing more candidates is enough for a static scorer. On MetaQA-3hop, expanding the pool adds false-positive relation matches that cause Fixed V to degrade (31.40% → 29.80% greedy-inclusive, p=0.0090p{=}0.0090), whereas GEVA's executed path verification rises +6.90 pp (41.40% → 48.30%).
  3. Paired with closed-loop baselines under shared caps and approximately matched total tokens: Because GEVA spends extra post-pool verification compute beyond Fixed V at fixed N=8N{=}8, we also added an end-to-end single-H20 closed-loop benchmark (§4.5, Table 15; Appendix L.1) against wider static sampling (Fixed V at N=12N{=}12, which uses 2.2%–13.1% more actual tokens than GEVA) and iterative tool-feedback loops (TF+cache (CRITIC-style), FO+cache). Even with extra sampling tokens, Fixed V (N=12N{=}12) trails GEVA by +2.38 / +1.55 / +11.90 pp on MBPP+, Spider, and MetaQA-3hop (p≤0.0225p\le 0.0225), confirming that active verification outperforms allocating comparable or slightly larger total token budgets to wider static sampling. We hope these two replies make our setup and baselines completely transparent, and we welcome any follow-up thoughts.
AuthorsOpenReview ↗

Brief Check-In on Our Clarity Revisions & Fixed V

With the discussion window closing in a few hours, we wanted to thank you again for your candid feedback on our draft's readability—it genuinely helped us make our paper much clearer. Following every item in your suggestions, we rewrote §1 in plain language with the motivating Luther walkthrough on page 1, defined Fixed V upfront across all seven tasks (§4.1; Table 24), and explained Eq. 4 inline (§3.5), as summarized in our two replies (1/2 and 2/2) above. Please let us know if anything still feels unclear—we would be very glad to clarify before the window closes.

Reviewer_iEu3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GEVA, a system that gathers and updates evidence to check LLM answers. It tracks evidence dependencies, rechecks repaired answers, and audits evidence producers. Experiments across seven settings show task-dependent improvements, with the largest gains on multi-hop knowledge-graph questions.

Critical strengths
  • Explicit evidence tracking helps prevent outdated or disclosed checks from being reused as independent support.

  • Matched-evidence controls help separate gains from new evidence, answer selection, and repair.

Critical weaknesses
  • Improvements over fixed verification are small or absent on several tasks, limiting evidence of broad benefits.

  • Reported acquisition costs exclude several pipeline stages, leaving the overall accuracy–cost tradeoff unclear.

Questions for authors
  • How does GEVA compare with strong verification baselines under the same total runtime and token budget?

  • When does evidence management improve accuracy beyond a simpler system with the same evidence and version tracking?

  • How sensitive are the results to the fixed evidence weights, correlation values, and acceptance threshold?

5 comments
AuthorsOpenReview ↗

End-to-End Runtime & Token Budget vs. Strong Baselines (W2, Q1)

We really appreciate your incisive review—especially your pointing out how explicit dependency tracking prevents outdated or disclosed checks from being reused as independent support, alongside your positive note on our matched-evidence controls. To address every point thoroughly, we organize our reply to you into three focused parts covering (1/3) W2 & Q1, (2/3) Q2, and (3/3) W1 & Q3.

1. Full-Pipeline Closed-Loop Evaluation on One H20 GPU (§4.5, Table 15; Appendix L.1)

You rightly pointed out that Table 4 originally tracked only pre-repair nominal action units (W2), and asked how GEVA compares against strong baselines under the same total runtime and token budget (Q1). In §4.5 (Table 15; full protocol in Appendix L.1), we ran an end-to-end closed-loop benchmark on a single NVIDIA H20 GPU across MBPP+ (n=378n{=}378, cap 60s / 18k tok), Spider (n=1,034n{=}1{,}034, cap 60s / 24k tok), and MetaQA-3hop (n=1,000n{=}1{,}000, cap 30s / 8k tok), plus an 800-instance held-out MetaQA-3hop split with an external Qwen3.6-27B judge. Every policy shares GEVA's lifecycle cache and is billed in full for Stage-0 generation, initial scoring, tool runs, controller inference, and repairs (rotating order, prefix caching disabled):

  • Fixed V (N=12N{=}12, tok-matched+): Reallocates post-pool budget into wider static sampling (11 greedy + 1212 draws, zero post-pool actions), using 2.2%–13.1% more actual mean tokens than GEVA.
  • TF+cache (CRITIC-style): Iterative LLM tool-feedback loop invoking probes, SQL/KB checks, adjudication, or repairs up to the cap.
  • FO+cache: Fixed-order producer-group escalation (bench/property -> KB path -> model probe -> adjudication) under GEVA's identical gate (ρ=0.35,Cmin⁡=0.60\rho{=}0.35, C _ {\min}{=}0.60); candidate repair is triggered separately by shared operational-veto eligibility with mandatory charged revalidation. | Benchmark (Table 15) | Fixed V (N=12N{=}12) | TF+cache (CRITIC-style) | FO+cache | GEVA | |---|---|---|---|---| | A. MBPP+ (378; Audit=Off) | 76.72% (12.6s / 6.29k) | 78.04% (18.9s / 10.86k) | 78.57% (15.1s / 7.48k) | 79.10% (12.8s / 6.14k) | | A. Spider (1,034; Audit=Off) | 68.18% (8.1s / 6.62k) | 69.05% (15.3s / 13.18k) | 69.44% (9.8s / 7.91k) | 69.73% (8.2s / 6.48k) | | A. MetaQA-3hop (1,000; Audit=Off) | 30.60% (3.8s / 2.24k) | 41.10% (6.8s / 3.96k) | 41.30% (4.1s / 2.19k) | 42.50% (3.5s / 1.98k) | | B. MetaQA-3hop + 27B Judge (800; Audit=On) | — (200/800 split) | 39.88% (8.46s / 4.64k) | 39.13% (7.92s / 4.19k) | 43.50% (4.62s / 2.61k) |

2. What the Closed-Loop Results & Complementary Controls Show

  1. Targeted verification beats wider static sampling: Even though Fixed V (N=12N{=}12) spends 2.2%–13.1% more tokens on extra draws, GEVA improves accuracy by +2.38 pp on MBPP+ (p=0.0225p{=}0.0225), +1.55 pp on Spider (p=0.0195p{=}0.0195), and +11.90 pp on MetaQA-3hop (p=4.6×10−25p{=}4.6\times 10^{-25}).
  2. Utility routing cuts cost sharply over LLM loops and fixed schedules: In Panel A, GEVA cuts latency by 32.3%–48.5% and tokens by 43.5%–50.8% vs. TF+cache (CRITIC-style), resolving +1.40 pp higher accuracy on MetaQA-3hop (p=0.020p{=}0.020, while MBPP+/Spider accuracy CIs span zero at 32.3%–46.4% lower latency). GEVA is also faster and cheaper than FO+cache because our router (Eq. 4) scores actions by marginal utility per unit cost: at=arg⁡max⁡aΔ^(Gt,a)+β(1−GraphCut(Gt))−γCorrPenalty(Gt,a)Cost(a)a _ t = \arg\max _ {a} \frac{\widehat{\Delta}(G _ t, a) + \beta(1 - \mathrm{GraphCut}(G _ t)) - \gamma \mathrm{CorrPenalty}(G _ t, a)}{\mathrm{Cost}(a)} halting once family concentration drives marginal gain to zero instead of querying LLM probes whenever the gate (ρ=0.35,Cmin⁡=0.60\rho{=}0.35, C _ {\min}{=}0.60) remains unmet. In Panel B—including 200-case calibration amortized over M=800M{=}800 held-out queries (Ccal/800=+1.14 s,+630 tok/caseC _ {\text{cal}}/800 = +1.14\text{ s}, +630\text{ tok/case})—label-free judge auditing (Eq. 5) gains +4.38 pp over FO+cache (p=5.1×10−6p{=}5.1\times 10^{-6}, -41.7% s) and +3.63 pp over TF+cache (CRITIC-style) (p=1.1×10−4p{=}1.1\times 10^{-4}).
  3. Complementary frozen-pool & transfer controls: Separate controls confirm value beyond raw tool access: GEVA beats a deterministic selector on identical final MetaQA-3hop paths (+1.40 pp, p=0.0125p{=}0.0125, §4.3, Table 19), drops 2.13 pp without Spider risk/coverage gates (p=9.4×10−4p{=}9.4\times 10^{-4}, Table 20), and beats all five supervised selectors (Scalar, LR, GBDT, DeepSets, Set-Transformer) under leave-one-regime-out transfer on Spider, MetaQA-3hop, and KQA Pro (§4.4, Table 7).
AuthorsOpenReview ↗

When Lifecycle Governance Outperforms Simpler Version Caching (Q2)

Continuing from Part 1, we are grateful to you for raising such a sharp architectural question in Q2: when does GEVA's evidence management improve accuracy or safety beyond a simpler system with the same arriving evidence and basic version tracking? Our controlled experiments in §4.3, §4.5, and Appendices N.1–N.3 (Tables 21–23) establish the exact boundary between where simple version caching suffices and where explicit dependency and disclosure governance becomes essential:

Controlled EvaluationPolicy / Cache DesignAccuracyInvalid / Stale SupportKey Outcome & Computational Finding
Mistral-7B Repair Pilot (24 cases, 30 repairs; Table 23)Stale reuse37.50%16.67% invalidReuses pre-repair or disclosed prompt assertions
Mistral-7B Repair Pilot (single-candidate rows; Table 23)Version cache (hashes + disclosure)37.50%0.00%Matches GEVA when no cross-candidate rows are active
Mistral-7B Repair Pilot (single-candidate rows; Table 23)GEVA lifecycle37.50%0.00%Changes 11/24 final picks vs. stale reuse
MBPP+ Repair Replay (144 episodes, 36 clusters; Table 22)Stale reuse18.75%90.28% staleRetains pre-repair peer-consensus & old checks
MBPP+ Repair Replay (144 episodes, 36 clusters; Table 22)Producer-local overwrite15.97%83.33% stale3/10 incorrect intermediate acceptances; 4 regressions
MBPP+ Repair Replay (144 episodes, 36 clusters; Table 22)GEVA lifecycle / Flat-contract18.75%0.00%+2.78 pp vs. Overwrite (0.00% final acceptance rate for all policies)
Live KB Event Replay (100 cases, 3,108 events; Table 21)Full recompute vs. GEVA localIdentical— (state/decision equivalence)3,708 vs. 28,872 claim folds (-87.16%, 3.0x faster)

1. When Simpler Version Caching Suffices (Appendix N.3, Table 23)

In our prospective 24-problem Mistral-7B sequential-repair pilot, a version cache checking code hashes and disclosure eligibility matches GEVA when only single-candidate assertion rows are active (Table 23): both eliminate contract-invalid reuse (16.67% -> 0.00%) and reach 37.50% accuracy (with GEVA changing 11/24 selections relative to Stale reuse). Thus, when every check depends only on a single candidate's code and recorded disclosure flag, hash-and-disclosure keying already captures the needed boundary.

2. When Explicit Dependency & Disclosure Governance Is Essential (Appendix N.2, Table 22)

Simpler caches break down once disclosed counterexamples (consumes edges, remaining active as regression vetoes while forfeiting fresh-support credit) or cross-candidate peer-consensus observations (behaviour_consensus, depending jointly on participating versions) create multi-candidate or transitive dependencies across revisions. Across 144 MBPP+ repair episodes in 36 clusters (Table 22), holding candidates and arriving tool outputs identical:

  • Stale reuse and Producer-local overwrite leave 90.28% and 83.33% of selected claims with contract-invalid stale support.
  • Immediately after the new public gate, Producer-local overwrite triggers 3/10 incorrect intermediate acceptances because replacing only the repaired producer's local row leaves stale cross-candidate consensus intact; it also incurs four episode regressions (15.97% accuracy). Once post-repair consensus is invalidated at the final endpoint, the final acceptance rate is 0.00% for all policies.
  • GEVA lifecycle eliminates stale support completely (0.00%), blocks all 3/10 premature wrong acceptances, matches Stale reuse in final accuracy (18.75%), and recovers +2.78 pp (95% CI [0.00, +8.33], four improved, zero harmed) over Producer-local overwrite.

3. Contracts vs. Graph Layout & Computational Payoff (Proposition 4; Tables 14, 21)

Crucially, an independent flat cache enforcing GEVA's exact version, dependency, and disclosure contracts reproduces GEVA's active sets and decisions at every checkpoint (Table 22). This confirms that behavioral safety comes from formal invalidation contracts rather than graph storage per se. What explicit dependency edges buy computationally is local propagation (Proposition 4, Appendix N.1): across 100 live KB traversals (3,108 events, 1,056 invalidations), GEVA reproduces full recomputation exactly while cutting per-event claim assessments by 87.16% (3,708 vs. 28,872, 3.0x faster replay in Table 21), with only 1.7 ms/case for graph construction and migration (Table 14).

AuthorsOpenReview ↗

Task-Dependent Gains Across Regimes & Hyperparameter Sensitivity (W1, Q3)

In this final part, we thank you for prompting us to clarify why gains over static verification vary across tasks (W1) and how sensitive GEVA is to fixed weights, correlation priors, and threshold ρ\rho (Q3).

1. Why Gains Vary Across the 13 Configurations (W1; §4.2, Table 1 & §4.5, Table 3)

With uniform KQA Pro greedy rescoring, Table 1 compares Greedy, Majority, Fixed V, and GEVA across all 13 configurations (7 on Qwen3.5-9B, 6 on Qwen3.6-27B; 27B MathQA-Geo not run) with zero missing cells, organized around three patterns (§4.2):

  1. Substantial gains under compositional multi-hop ambiguity: Where shallow reachability ties or inverts (Figure 2), active path and adjudication checks yield +11.10 / +7.30 pp on 9B/27B MetaQA-3hop (p=6.5×10−26p{=}6.5\times 10^{-26} / 1.4×10−181.4\times 10^{-18}; +4.50 pp on Mistral-7B, Table 13) and +5.50 / +3.50 pp on 9B/27B KQA Pro (p=2.2×10−7p{=}2.2\times 10^{-7} / 2.2×10−42.2\times 10^{-4}). Expanding N=8→32N{=}8\to 32 (Table 3) degrades MetaQA-3hop Fixed V (31.40% → 29.80% greedy-incl., p=0.0090p{=}0.0090) under extra false-positive ties, whereas GEVA rises +6.90 pp (41.40% → 48.30%, +18.50 pp over Fixed V at N=32N{=}32).
  2. Meaningful gains from probes & repair on 9B code/SQL: On 9B MBPP+, GEVA gains +2.91 pp over Fixed V (79.37% vs. 76.46%, p=0.00342p{=}0.00342: +1.32 pp from hidden probes pre-repair plus +1.59 pp from counterexample repair, p=0.031p{=}0.031, 6 improved, 0 harmed; Table 19). On 9B Spider, execution probes and gates add +1.74 pp (p=0.0114p{=}0.0114).
  3. Near-ceiling pools vs. certificate-free MathQA-Geo: Headroom is naturally small when 27B pools or visible base tests rank candidates near ceiling (27B MBPP+/Spider/HumanEval+ +0.79/+0.68/0.00+0.79/+0.68/0.00 pp; 9B HumanEval+ +0.61+0.61 pp; MetaQA-1hop −0.10-0.10 pp). Separately, on certificate-free MathQA-Geo (−0.89-0.89 pp, p=0.541p{=}0.541), faulty symbolic steps can still yield the right option letter (0.958 negative precision, Table 18); lacking sound wrongness certificates, GEVA withholds operational vetoes to avoid uncertified rejections.

2. Sensitivity to Weights, Correlation Priors & Threshold ρ\rho (Q3; §4.5, Appendix L.2)

Appendix L.2 (Tables 16–17) evaluates 25 settings per regime (1 default + 24 perturbations across 7 regimes = 5,027 cases, 120,648 perturbed decisions), sweeping weights wfw _ f, priors FamCorr\mathrm{FamCorr}, ρ∈[0.20,0.50]\rho\in[0.20, 0.50], and joint (wf,ρ)(w _ f, \rho) shifts:

Regime (Table 16)DefaultWeight PerturbationsCorrelation Shiftsρ\rho Sweep (0.20→0.500.20\to 0.50)Joint (wf,ρ)(w _ f, \rho)Max Abs. Δ vs. Default
MBPP+ (378)79.37%79.37–79.37%79.37–79.37%79.37–79.37%79.37–79.37%0.00 pp
HumanEval+ (164)87.80%87.80–87.80%87.80–87.80%87.80–87.80%87.80–87.80%0.00 pp
Spider (1,034)69.83%69.83–69.92%69.83–69.83%69.83–70.02%69.83–70.12%0.29 pp
MetaQA-1hop (1,000)33.70%33.70–33.70%33.70–33.70%33.70–33.70%33.70–33.70%0.00 pp
MetaQA-3hop (1,000)42.70%42.50–43.30%42.70–42.70%42.70–42.70%42.60–43.20%0.60 pp
KQA Pro (1,000)61.90%61.60–62.10%61.90–61.90%61.90–61.90%61.60–62.10%0.30 pp
MathQA-Geo (451)86.92%86.47–86.92%86.92–86.92%86.92–86.92%86.47–86.92%0.44 pp

(Ranges match Table 16, including the default; Max Abs. Δ vs. Default uses raw unrounded counts—at most 0.60 pp from default, with full min–max span 0.80 pp on MetaQA-3hop and 0.50 pp on KQA Pro.)

Two mechanisms drive this stability:

  • Provenance overlap dominates correlation priors: Eq. 1 takes max⁡(FamCorr(fi,fj),Jac(anci,ancj))\max(\mathrm{FamCorr}(f _ i, f _ j), \mathrm{Jac}(\mathrm{anc} _ i, \mathrm{anc} _ j)), so measured ancestor overlap governs real verification graphs.
  • Role of ρ\rho & producer audit: Varying ρ\rho trades acceptance coverage against false-trust as intended (e.g., MetaQA-3hop stays 42.70% across ρ∈[0.20,0.50]\rho\in[0.20,0.50] while acceptance spans 0%–30.60%), and Eq. 5's label-free audit withdraws misaligned judges (+4.38 pp, Table 2). We welcome any follow-up discussion.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

With the discussion window closing tomorrow, we wanted to send a brief, polite note to thank you again for your thoughtful feedback, which guided our revised manuscript and new single-H20 closed-loop experiments (§4.5, Table 15; Appendix L.1) as well as our upfront baseline and walkthrough clarifications (§1, §4.1; Appendix O).

Please let us know if any part of our responses or revisions remains unclear or if any quick clarification from us would be helpful before the discussion period ends—we remain available and happy to help.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.