- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that when large language models (LLMs) are used for test time scaling, "verification" should not merely consist of assigning a score to a candidate answer at the end; rather, it should be an ongoing process of gathering evidence while continuously maintaining the validity of that evidence. To address this, the authors propose the GEVA (Governed Evidence Verification and Audit) framework. This framework employs a deterministic controller that operates on a "claim-evidence graph," iteratively invoking mechanisms such as execution probes, knowledge base traversal, model-based adjudication, and error correction. Throughout this process, it tracks the provenance, dependencies, and validity of the evidence, while using label-agnostic auditing to prune harmful sources of evidence. Experiments across seven task categories demonstrate that GEVA consistently outperforms greedy decoding. Notably, on the MetaQA-3hop task, GEVA achieves an improvement of 11.10 percentage points over the "Fixed V" baseline (using the same candidate pool). Furthermore, removing harmful adjudicator models yields a recovery of 4.38 percentage points in accuracy. Finally, the results from incremental replay are consistent with those from full re-computation, while requiring approximately eight times fewer evaluations.
Critical strengths
- This paper proposes structuring the verification process as a workflow that simultaneously gathers evidence and ensures its ongoing validity, this approach I find both novel and practically significant.
- The author has conducted extensive testing across a variety of benchmarks.
- The author formalizes the verification process as a multi-stage evidence review. While this adds complexity to the overall verification procedure, it does indeed result in performance improvements.
Critical weaknesses
- Certain terms in the paper, such as "Fixed V," are not sufficiently explained.
- There are formatting issues; for instance, the LaTeX rendering of parameters around line 336 appears to be problematic. The author should carefully review the manuscript.
- The author has conflated parts of the main text with content that should instead be placed in an appendix—for example, the first and second paragraphs of Section 4.5.
- The models and methods used for comparison in Section 4.4 are somewhat outdated. I suggest that the author actively seek out and include comparisons with architectures similar to the one proposed in this paper—such as the work related to "tool feedback" mentioned in the "Related Work" section.
Questions for authors
- Could the authors provide a further explanation of how "Fixed V" functions?
- What is the significance of the two different colors used for the columns in Table 3? This does not appear to be addressed in the main text.
- In the section detailing the first experiment, the content consists primarily of a presentation of experimental data, with very few conclusions drawn. Could the authors revise this section to provide a summary that includes concrete conclusions?
3 comments
LaTeX Formatting Fixes, Streamlining §4.5 & Concrete Takeaways in §4.2 (W2, W3, Q3)
Continuing from Part 1, we are very thankful for your careful line-by-line reading, which caught formatting and structural friction in §3.4, §4.2, and §4.5 (W2, W3, Q3) and helped us noticeably improve the flow of our revised paper.
1. LaTeX Parameter Formatting & Full Compilation Audit (W2; §3.4 & §§3–4)
Around line 336 (W2), bare leading-dot decimals ($.803$, $.983$, $.90$, $\rho=.35$) rendered awkwardly next to surrounding punctuation. We audited the entire manuscript and appendices, normalized all parameter values and thresholds to explicit leading-zero math formatting ($0.803$, $0.983$, $0.90$, $\rho=0.35$, $C _ {\min}=0.60$, etc.), and verified clean compilation across all 41 pages (10 main-body pages, 0 LaTeX errors, 0 undefined references, and 0 overfull boxes).
2. Streamlining §4.5 and Moving Granular Archive Details to the Appendix (W3; §4.5, Appendices L–N)
Following your suggestion on the first two paragraphs of §4.5 (W3), we removed inline archive-history caveats from §4.5 and moved granular artifact notes, CPU subphase breakdowns (Table 14), and full perturbation tables (Tables 16–17) into Appendices L–N. §4.5 now focuses cleanly on four empirical questions: nested-pool scaling (Table 3), adaptive versus full-revelation budgets (Table 4), end-to-end single-H20 closed-loop runtime and token trade-offs (Table 15), and post-repair lifecycle validity.
3. Explicit Cross-Regime Conclusions Added to §4.2 (Q3; Table 1)
You noted that §4.2 previously read mainly as a listing of numbers without clear high-level takeaways (Q3). We revised §4.2 to synthesize Table 1 around three explicit conclusions:
- Substantial gains under compositional multi-hop ambiguity: When initial reachability heuristics suffer frequent ties or inversions across multi-hop relations (Figure 2), active evidence acquisition yields large improvements—lifting MetaQA-3hop by +11.10 pp on Qwen3.5-9B (
42.70%vs.31.60%, ), +7.30 pp on Qwen3.6-27B (44.10%vs.36.80%, ), and +4.50 pp on Mistral-7B (Table 13), while lifting KQA Pro by +5.50 pp on 9B () and +3.50 pp on 27B (). - Meaningful gains from execution probes and counterexample repair on 9B code and SQL: Where public prompt assertions under-specify edge cases or SQL join semantics, GEVA gains +2.91 pp on 9B MBPP+ (:
+1.32 ppfrom private probes pre-repair plus +1.59 pp from versioned counterexample repair, , six improved, none harmed; Table 19) and +1.74 pp on 9B Spider () via database execution checks and selective gating. - Small or unresolved differences on near-ceiling pools vs. certificate-free MathQA-Geo: Differences relative to
Fixed Vare small where stronger 27B pools or visible prompt tests already rank candidates near ceiling (27B MBPP+/Spider/HumanEval+ pp; 9B HumanEval+ pp; MetaQA-1hop pp). Distinctly, on certificate-free MathQA-Geo (86.92%vs.87.80%, pp, ), faulty symbolic derivations can still produce the right multiple-choice letter (0.958negative precision, Table 18); because sound wrongness certificates are unavailable, GEVA retains graded evidence without operational vetoes, avoiding uncertified rejections.
Comparisons with Modern CRITIC-Style Tool-Feedback & Neural Selectors (W4)
Rounding out our reply (Part 3), we want to thank you for W4: your suggestion to go beyond the tabular selectors in §4.4 (Scalar, LR, GBDT) and compare head-to-head against modern iterative tool-feedback architectures discussed in §2 Related Work inspired the centerpiece new experiment of this revision. Below we detail both the neural selector baselines in Appendix E and our new end-to-end single-H20 tool-feedback benchmark in §4.5 (Table 15; Appendix L.1).
1. Neural Set Selectors Under Zero-Label Transfer (§4.4, Table 7 in Appendix E)
Alongside Figure 3's target-label learning curves for the three tabular selectors, Table 7 (Appendix E) evaluates five supervised flat-ledger selectors including permutation-invariant DeepSets and multi-head Set-Transformer. While fitted models are competitive when given in-domain target labels, under realistic leave-one-regime-out transfer (zero target-domain labels, matching GEVA's operating regime), all five supervised selectors trail zero-label GEVA on Spider (63.93%–68.86% vs. 70.21%), MetaQA-3hop (35.10% Set-Transformer and 37.80% DeepSets vs. 42.70%), and KQA Pro (54.70%–61.60% vs. 61.90%).
2. New Closed-Loop Comparison vs. CRITIC-Style Tool-Feedback on One H20 GPU (§4.5, Table 15; Appendix L.1)
To compare directly against modern iterative tool-feedback workflows on Qwen3.5-9B and Qwen3.6-27B, we implemented TF+cache (CRITIC-style) (following Gou et al., 2024: an LLM controller inspects candidates and tool outputs and iteratively invokes sandboxes, SQL execution, KB path probes, external adjudication, or code repair) and FO+cache (fixed-order bench/property -> KB path -> model probe -> adjudication escalation, with shared operational-veto repair eligibility and charged revalidation), alongside wider static sampling (Fixed V at , consuming 2.2%–13.1% more actual tokens than GEVA). Every method receives GEVA's exact lifecycle cache and is billed end-to-end on one H20 GPU for Stage-0 generation, tools, controller prompts, and repairs under shared wall-clock and token caps:
| H20 Closed-Loop Benchmark (Table 15) | Fixed V () | TF+cache (CRITIC-style) | FO+cache | GEVA |
|---|---|---|---|---|
| A. MBPP+ (378; Audit=Off) | 76.72% (12.6s / 6.29k) | 78.04% (18.9s / 10.86k) | 78.57% (15.1s / 7.48k) | 79.10% (12.8s / 6.14k) |
| A. Spider (1,034; Audit=Off) | 68.18% (8.1s / 6.62k) | 69.05% (15.3s / 13.18k) | 69.44% (9.8s / 7.91k) | 69.73% (8.2s / 6.48k) |
| A. MetaQA-3hop (1,000; Audit=Off) | 30.60% (3.8s / 2.24k) | 41.10% (6.8s / 3.96k) | 41.30% (4.1s / 2.19k) | 42.50% (3.5s / 1.98k) |
| B. MetaQA-3hop + 27B Judge (800; Audit=On) | — (200/800 split) | 39.88% (8.46s / 4.64k) | 39.13% (7.92s / 4.19k) | 43.50% (4.62s / 2.61k) |
Two conclusions highlight what GEVA's structured controller adds over unconstrained tool-feedback loops:
- Substantially lower latency and token cost without sacrificing accuracy (Panel A): Because GEVA routes tools via analytical marginal utility per unit cost (Eq. 4) rather than multi-turn LLM orchestration prompts, it reduces end-to-end latency by 32.3%–48.5% and total tokens by 43.5%–50.8% relative to
TF+cache (CRITIC-style), while achieving +1.40 pp higher accuracy on MetaQA-3hop (; MBPP+ and Spider accuracy CIs span zero at 32.3%–46.4% lower latency) and beating wider static samplingFixed V() across all three benchmarks (+2.38 / +1.55 / +11.90 pp, ). - Resilience to misaligned external evaluators via label-free auditing (Panel B): When an external Qwen3.6-27B judge is available on MetaQA-3hop, unaudited
FO+cacheandTF+cache (CRITIC-style)repeatedly invoke the biased judge and drop to39.13%and39.88%. Even after charging GEVA the full 200-case unlabeled calibration pass amortized across the 800 held-out queries (), GEVA's certificate audit (Eq. 5) withdraws the harmful judge automatically, gaining +4.38 pp overFO+cache(, -41.7% latency) and +3.63 pp overTF+cache (CRITIC-style)(). We are glad to discuss any of these results further during the discussion period.
How `Fixed V` Functions & Visual Key for Table 3 (W1, Q1, Q2)
We greatly value your encouraging feedback on our work—in particular your recognition of structuring verification as a validity-governed workflow across diverse benchmarks and formal stages. To address each of your points thoroughly while keeping manuscript edits surgical, we organize our reply to you into three focused parts: (1/3) W1, Q1 & Q2, (2/3) W2, W3 & Q3, and (3/3) W4.
1. How
Fixed VFunctions Across All Seven Regimes (W1, Q1; §1, §4.1, Table 24 in Appendix O)You noted that
Fixed Vwas insufficiently explained (W1) and asked for a clear explanation of how it functions (Q1). In §1 and §4.1 (with full summary table and code-level mappings in Table 24, Appendix O), we now defineFixed Vexplicitly upfront: on the same ordered initial candidate pool evaluated byMajorityandGEVA—a nonempty greedy output plus 8 temperature-0.8 draws (9candidates, or8on the12MathQA-Geo cases with empty greedy output; Appendix O)—Fixed Vscores each candidate once using only the task's initial public-input verifier Vx(ci) and selects the earliest candidate achieving the maximum score (j⋆=minargmaxiVx(ci)), performing zero post-pool probes, database queries, exact-hop traversals, judge calls, or repairs:Fixed V(§4.1; Table 24, Appendix O)base_pass / base_total;0if none)1.0typed k-hop reachable,0.6untyped reachable,0.1KB node,0else)1.0typed neighborhood,0.3subgraph node,0else)In addition, we applied the deterministic Wikidata subgraph scorer uniformly to KQA Pro's greedy decode in Table 1, completing all 13 evaluated configurations (
7on Qwen3.5-9B and6on Qwen3.6-27B; 27B MathQA-Geo was not evaluated) with zero missing--entries (+5.50 pp on 9B, p=2.2×10−7; +3.50 pp on 27B, p=2.2×10−4).2. Meaning of the Column Colors and Boldface in Table 3 (Q2; §4.5, Table 3)
Because we preserved the manuscript's original float layout, Table 3 in your review remains Table 3 in §4.5. Since its visual encoding was previously left implicit, we updated Table 3's caption to state all three visual cues explicitly:
GEVAat N=8,16,32).Oraclepool ceiling (whether any candidate in greedy + SN is correct—an upper bound, not a deployable selector).GEVAaccuracy acrossN in {8, 16, 32}(indicating within-row pool-scaling direction rather than statistical significance).Fixed V(samples / greedy-incl.)GEVA(blue columns; row-max bold)Oracle-in-pool (gray text ceiling)As Table 3 shows, drawing more candidates increases false-positive relation ties that hurt static
Fixed V, whereas GEVA's executed exact-hop path verification converts pool diversity into steady gains (+5.30 / +6.90 / +2.40 pp from N=8→32 on MetaQA-1hop / MetaQA-3hop / KQA Pro).