What Actually Fixes an LLM Verifier — And Why Nothing Else Does
Abstract
LLM verifiers suffer from a fundamental asymmetry: while they reliably confirm correct answers (87.9% True Positive Rate), they fail to catch errors, catching only 46.1% of incorrect candidates overall and plunging to 26.4% in mathematics. Across 64,800 controlled verifications spanning four models and three domains, we investigate why popular evaluator interventions fail to break this specificity collapse (i.e., low True Negative Rate and indiscriminate error approval). Holding the exact error fixed across 6,969 strata, models approve their own wrong answers 11.9 percentage points more often than other models approve those same errors. Five targeted diagnostic probes reveal that this self-preference is not driven by stylistic fingerprints or active self-recognition, but by belief persistence—verifiers defending the same flawed reasoning priors that generated the mistake. Consequently, authorship blinding shifts accuracy by less than 1 pp. Furthermore, model scaling, ensemble voting, and multi-agent deliberation fail because ungrounded judges share correlated reasoning errors on incorrect candidates, with ungrounded debate juries catching only 76.7% of bugs despite massive compute inflation. In contrast, environmental execution grounding decorrelates verifier errors and achieves a 93.3% error catch rate. Among the interventions evaluated, model-side prompting, scaling, and debate cannot substitute for an external execution anchor: verifier reliability is bounded by the ground-truth signals provided by the environment, not by the judge reading the output.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.