acceptodds
Under review as a conference paper at ICLR 2027

What Actually Fixes an LLM Verifier — And Why Nothing Else Does

Abstract

LLM verifiers suffer from a fundamental asymmetry: while they reliably confirm correct answers (87.9% True Positive Rate), they fail to catch errors, catching only 46.1% of incorrect candidates overall and plunging to 26.4% in mathematics. Across 64,800 controlled verifications spanning four models and three domains, we investigate why popular evaluator interventions fail to break this specificity collapse (i.e., low True Negative Rate and indiscriminate error approval). Holding the exact error fixed across 6,969 strata, models approve their own wrong answers 11.9 percentage points more often than other models approve those same errors. Five targeted diagnostic probes reveal that this self-preference is not driven by stylistic fingerprints or active self-recognition, but by belief persistence—verifiers defending the same flawed reasoning priors that generated the mistake. Consequently, authorship blinding shifts accuracy by less than 1 pp. Furthermore, model scaling, ensemble voting, and multi-agent deliberation fail because ungrounded judges share correlated reasoning errors on incorrect candidates, with ungrounded debate juries catching only 76.7% of bugs despite massive compute inflation. In contrast, environmental execution grounding decorrelates verifier errors and achieves a 93.3% error catch rate. Among the interventions evaluated, model-side prompting, scaling, and debate cannot substitute for an external execution anchor: verifier reliability is bounded by the ground-truth signals provided by the environment, not by the judge reading the output.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM verifiers, LLM-as-a-judge, self-preference, execution grounding, AI agents, multi-agent debate, evaluation, belief persistence

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZYY5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper reports 64,800 verification trials across 4 open-weight models (Qwen2.5-72B, DeepSeek-V3, Llama-3.3-70B, Mistral-Nemo-12B), 3 domains (HumanEval+, MATH, GPQA-Diamond), 3 prompting strategies (Direct, CoT, Rubric), and 3 authorship frames (Self, Other, Neutral). The main claims are:

  1. Verifiers reliably approve correct answers (87.9% TPR) but fail to catch errors (46.1% TNR overall; 26.4% in math).
  2. Holding the exact error fixed across 6,969 strata, models approve their own wrong answers 11.9 pp more often than other models do on the same text, whereas authorship framing shifts accuracy by <1 pp.
  3. Style, self-recognition, and preference probes point to "belief persistence" (shared reasoning priors) rather than self-recognition.
  4. Scaling, ensembling, and ungrounded debate juries fail due to correlated errors (κ=0.35–0.45).
  5. Execution grounding in code reaches a 93.3% catch rate and is presented as the only intervention that works.
Critical strengths
  1. Factorial authorship design: Crossing told authorship (SELF/OTHER/NEUTRAL) with true generator identity cleanly separates framing effects from actual authorship.
  2. Statistical estimation: Item-clustered bootstrap intervals and the error-stratified estimator properly isolate evaluator behavior from item difficulty and generator quality.
  3. Differential fuzzing audit: Using external models to generate adversarial inputs and adjudicate the 1,458 cases where judges overruled passing unit tests is a practical way to separate judge hallucinations (77.2%) from real test-suite gaps. Code and traces are released.
Critical weaknesses
  1. The main execution-grounding result is nearly circular. In code, the ground-truth label is the HumanEval+ test harness result (Table 1). The grounded judge is directly shown the harness output (Status: Passed/Failed, assertions passed, and traceback; Appendix A.1). Showing that a model recovers the ground-truth label when handed that label is not evidence that grounding "decorrelates verifier errors". A 12B model (Mistral) reaching 97.3% accuracy under direct scoring simply reflects copying the status block into JSON. Telling practitioners to run unit tests when available is not a new finding; the hard problem in verification is precisely when no executable oracle exists.
  2. Grounding is completely confounded with domain. As Section 5 notes, "grounding varies across benchmarks rather than within them". Code, math, and science differ in base rates, error types, and task structure. The obvious controls are missing: running the 4-model code evaluation without the execution block, or giving the math/science judges a Python interpreter or reference check. Without within-domain controls, no causal claim about grounding holds.
  3. The jury comparison is underpowered and confounded. Sample size: In Table 4 (n=150 DeepSeek code items), the catch rates (70.00%, 76.67%, 93.33%) and FPRs (30.00%, 23.33%, 6.67%) are all multiples of 1/30, meaning there are only 30 negative items. The comparison is 21/30 and 23/30 vs. 28/30. An unpaired two-sided Fisher's exact test gives p≈0.15 for 28/30 vs. 23/30 (not significant) and p≈0.04 for 28/30 vs. 21/30. No p-values or intervals are reported, yet these numbers drive the abstract.
  4. Confounded models and inputs: The juries use Step-3.7-Flash, Nemotron-340B, and Kimi-K2.7, whereas the grounded baseline uses Qwen2.5-72B. Neither jury was tested with the execution block, and Qwen2.5-72B is not reported without it on those 30 items.
  5. Cost framing: The "28×–457× compute" figure is a dollar-cost ratio across different API providers; actual token inflation in Table 11 is 8.3×–32.2× (5.4K and 21.0K vs. 653 tokens).
  6. The mechanism probes are weaker than claimed. Style transfer (Probe 1): By selecting only cases with 100% baseline FPR, the placebo rewrite alone drops FPR to 85.5% and 91.8% via regression to the mean / rewrite noise, swamping the -0.2 pp and -1.5 pp net effects. Holding formulas and code fixed also leaves intact the exact tokens where model fingerprints are strongest. Only two models were tested.
Questions for authors
  1. Metric reconciliation: What are the exact per-domain confusion matrices? In Section 4.1, math has a 62.8% positive base rate ("always-approve baseline"), 26.4% TNR, and 60.4% balanced accuracy (which implies 94.4% TPR). Yet 0.628(94.4%)+0.372(26.4%)=69.1% raw accuracy, not the reported 64.5%. Why don't these numbers match?

  2. Jury controls: On the 30 wrong code items in Table 4, what is the catch rate of (a) Qwen2.5-72B without the execution block, (b) the individual jury models with and without the execution block, and (c) the juries with the execution block? Per-verifier self-preference: Given the wide spread in Table 3 (+24.8 pp for DeepSeek and +20.3 pp for Qwen, vs. +4.1 pp for Llama and -4.2 pp for Mistral), what is the stratified self-preference estimate with verifier fixed effects and per-verifier 95% CIs?

  3. Probe 3 units: How were the 608 units in Table 10 defined out of 450 items, how many unique items do they span, and what are the item-clustered p-values?

  4. Re-solving and reasoning models: Since the prompt forbids re-solving from scratch and caps CoT at 3–4 sentences (Appendix B.3–B.4), do the ungrounded math/science results hold when judges are allowed to re-solve the problem or when reasoning models (e.g., R1, QwQ) are used?

Reviewer_oxCROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigate the reasons that why LLM verifiers suffer from asymmetry. This paper utilized 648000 controlled verifications in three domains to analyze asymmetry of LLM verifiers. Specifically, five diaganostic probes are designed to support the conclusions for this paper in terms of self-preference, belief persistence etc for models (DeepSeek-V3, Qwen 2.5, Llama-3.3 and Mistral-Nemo).

Critical strengths

There are several strengths for this paper: (1) The paper is dealing with an interesting topic. Instead of just benchmarking the LLMs, it aims to identify the failure mechanism.(2) The analyzing directions are in general great. (3) The experimental design is very interesting.

Critical weaknesses

The following claim could be over-amplified: An LLM could favor an incorrect answer because its internal reasoning favors the same conclusion. That is consistent with belief persistence, but it may also reflect shared training distributions, problem-specific blind spots, familiarity with the derivation, or matching reasoning heuristics.

The abstract claims that environmental execution grounding decorrelates verifier errors. A higher error-detection rate does not necessarily imply lower error correlation. For example, all grounded verifiers might correctly identify more errors while still failing on exactly the same remaining cases

Some conclusions are too strong, and not sufficiently supported by the experimental results. For example, the claim "verifier reliability is bounded by the ground-truth signals provided by the environment, not by the judge reading the output"

There are only 4 benckmarked models, which is not that convincing to fully support the conclusion

Questions for authors

Authors should directly address the question that evidence of havioral patterns are not yet proof of the causal mechanisms. Stronger experimental support are very needed.

Four models are not comprehensive, and more recent models are encouraged to check whether the conclusions are consistent.

Tune-down the over-claims.

Reviewer_UjVsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates why LLM-based verifiers systematically fail to catch errors, finding through 64,800 controlled verifications that verifiers suffer “specificity collapse”, approving correct answers 87.9% of the time but catching only 46.1% of errors overall, plunging to 26.4% in mathematics. The paper’s main finding is that self-preference — where models approve their own wrong answers 11.9 percentage points more often than other models approve those same errors, driven by belief persistence. The paper systematically evaluates popular mitigation strategies and finds them inadequate. Authorship blinding shifts accuracy by less than 1 pp. Correlated reasoning errors across models neutralize ensemble voting. Multi-agent debate juries achieve only 76.7% catch rate at up to 457× the cost of a single verification. Only environmental execution grounding, which provides verifiers with actual runtime signals, achieves a 93.3% error catch rate, decorrelating verifier errors.

Critical strengths

S-1 Unlike many empirical studies that end with ambiguous conclusions, this paper delivers a clear message: “verifier reliability is bounded by the ground-truth signals provided by the environment, not by the sophistication of the judge.” The cost-effectiveness comparison: execution grounding at 93.3% catch rate and 1× cost vs. debate juries at 76.7% and 457× cost, provides practitioners with clear guidance on coding task evaluation.

Critical weaknesses

As an empirical study without a new method or theory, the paper's value rests on the soundness of its experiments. We therefore assess it primarily on experimental design and execution.

W-1 The strongest positive result (93.3% catch rate) holds only for code, where an execution harness exists. Mathematics, where specificity collapse is most severe (26.4% TNR), and other domains have no equivalent, and the paper does not discuss how grounding would be achieved there. In addition, the fuzz-adjusted labels depend on an LLM adjudicator (WizardLM-2-8x22B) whose accuracy is never evaluated. Given the paper's own claim that ungrounded LLM judges are unreliable, relying on one to decide BUG CONFIRMED and REFERENCE BUG verdicts requires validation, for example a human audit of a sample of adjudicated divergences.

W-2 Limited task scope and small evaluation sets. All three benchmarks involve well-defined, objectively verifiable tasks. The paper does not address open-ended generation, summarization, and other tasks where no single-turn evaluation signal is available, and LLM-as-a-judge is most deployed in practice. Moreover, each benchmark has only 150 items, which limits statistical power, and the public benchmarks used cannot rule out training contamination.

W-3 Single-turn evaluation only and no state-of-the-art proprietary models are evaluated. Real agent pipelines involve multi-turn verification loops where verifiers can request clarification or additional evidence. The single-turn design may underestimate the potential of iterative verification strategies. Since proprietary models may employ different RLHF procedures, the generalizability of the findings of belief persistence and correlated reasoning errors to the broader model ecosystem remains uncertain.

Questions for authors

Suggestions for future work:

Execution grounding is available only for code, yet mathematics shows the weakest catch rate (26.4%). Several lines of work combine LLMs with symbolic tools on reasoning tasks, e.g. program-aided language models. Did you consider letting the verifier write and run a symbolic or numerical check (e.g., Python), and if so, how close does it get to the 93.3% catch rate on code?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.