- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper reports 64,800 verification trials across 4 open-weight models (Qwen2.5-72B, DeepSeek-V3, Llama-3.3-70B, Mistral-Nemo-12B), 3 domains (HumanEval+, MATH, GPQA-Diamond), 3 prompting strategies (Direct, CoT, Rubric), and 3 authorship frames (Self, Other, Neutral). The main claims are:
- Verifiers reliably approve correct answers (87.9% TPR) but fail to catch errors (46.1% TNR overall; 26.4% in math).
- Holding the exact error fixed across 6,969 strata, models approve their own wrong answers 11.9 pp more often than other models do on the same text, whereas authorship framing shifts accuracy by <1 pp.
- Style, self-recognition, and preference probes point to "belief persistence" (shared reasoning priors) rather than self-recognition.
- Scaling, ensembling, and ungrounded debate juries fail due to correlated errors (κ=0.35–0.45).
- Execution grounding in code reaches a 93.3% catch rate and is presented as the only intervention that works.
Critical strengths
- Factorial authorship design: Crossing told authorship (SELF/OTHER/NEUTRAL) with true generator identity cleanly separates framing effects from actual authorship.
- Statistical estimation: Item-clustered bootstrap intervals and the error-stratified estimator properly isolate evaluator behavior from item difficulty and generator quality.
- Differential fuzzing audit: Using external models to generate adversarial inputs and adjudicate the 1,458 cases where judges overruled passing unit tests is a practical way to separate judge hallucinations (77.2%) from real test-suite gaps. Code and traces are released.
Critical weaknesses
- The main execution-grounding result is nearly circular. In code, the ground-truth label is the HumanEval+ test harness result (Table 1). The grounded judge is directly shown the harness output (Status: Passed/Failed, assertions passed, and traceback; Appendix A.1). Showing that a model recovers the ground-truth label when handed that label is not evidence that grounding "decorrelates verifier errors". A 12B model (Mistral) reaching 97.3% accuracy under direct scoring simply reflects copying the status block into JSON. Telling practitioners to run unit tests when available is not a new finding; the hard problem in verification is precisely when no executable oracle exists.
- Grounding is completely confounded with domain. As Section 5 notes, "grounding varies across benchmarks rather than within them". Code, math, and science differ in base rates, error types, and task structure. The obvious controls are missing: running the 4-model code evaluation without the execution block, or giving the math/science judges a Python interpreter or reference check. Without within-domain controls, no causal claim about grounding holds.
- The jury comparison is underpowered and confounded. Sample size: In Table 4 (n=150 DeepSeek code items), the catch rates (70.00%, 76.67%, 93.33%) and FPRs (30.00%, 23.33%, 6.67%) are all multiples of 1/30, meaning there are only 30 negative items. The comparison is 21/30 and 23/30 vs. 28/30. An unpaired two-sided Fisher's exact test gives p≈0.15 for 28/30 vs. 23/30 (not significant) and p≈0.04 for 28/30 vs. 21/30. No p-values or intervals are reported, yet these numbers drive the abstract.
- Confounded models and inputs: The juries use Step-3.7-Flash, Nemotron-340B, and Kimi-K2.7, whereas the grounded baseline uses Qwen2.5-72B. Neither jury was tested with the execution block, and Qwen2.5-72B is not reported without it on those 30 items.
- Cost framing: The "28×–457× compute" figure is a dollar-cost ratio across different API providers; actual token inflation in Table 11 is 8.3×–32.2× (5.4K and 21.0K vs. 653 tokens).
- The mechanism probes are weaker than claimed. Style transfer (Probe 1): By selecting only cases with 100% baseline FPR, the placebo rewrite alone drops FPR to 85.5% and 91.8% via regression to the mean / rewrite noise, swamping the -0.2 pp and -1.5 pp net effects. Holding formulas and code fixed also leaves intact the exact tokens where model fingerprints are strongest. Only two models were tested.
Questions for authors
-
Metric reconciliation: What are the exact per-domain confusion matrices? In Section 4.1, math has a 62.8% positive base rate ("always-approve baseline"), 26.4% TNR, and 60.4% balanced accuracy (which implies 94.4% TPR). Yet 0.628(94.4%)+0.372(26.4%)=69.1% raw accuracy, not the reported 64.5%. Why don't these numbers match?
-
Jury controls: On the 30 wrong code items in Table 4, what is the catch rate of (a) Qwen2.5-72B without the execution block, (b) the individual jury models with and without the execution block, and (c) the juries with the execution block? Per-verifier self-preference: Given the wide spread in Table 3 (+24.8 pp for DeepSeek and +20.3 pp for Qwen, vs. +4.1 pp for Llama and -4.2 pp for Mistral), what is the stratified self-preference estimate with verifier fixed effects and per-verifier 95% CIs?
-
Probe 3 units: How were the 608 units in Table 10 defined out of 450 items, how many unique items do they span, and what are the item-clustered p-values?
-
Re-solving and reasoning models: Since the prompt forbids re-solving from scratch and caps CoT at 3–4 sentences (Appendix B.3–B.4), do the ungrounded math/science results hold when judges are allowed to re-solve the problem or when reasoning models (e.g., R1, QwQ) are used?