acceptodds
Under review as a conference paper at ICLR 2027

Statement Gaming: How Verifier Pressure Turns Formal Mathematics Into Empty Theorems

Abstract

Formal verification is widely treated as the gold standard for AI-generated mathematics: if the Lean kernel accepts a proof, the result is taken to be trustworthy. This trust rests on an assumption that the checked statement faithfully encodes the intended mathematics. We show that when a language model both writes the formal statement and must satisfy the verifier, this assumption fails systematically, and it fails hardest exactly where the verifier's word matters most. We name the phenomenon statement gaming: the model deforms the statement itself — substituting free variables for defined objects, assuming the conclusion among hypotheses, or reducing the claim to a tautology — until the verifier accepts something the source mathematics never asserted. We measure statement gaming across 1,389 model–item pairs — statements drawn from research papers and corrected competition benchmarks, five production models from two vendors, five generation regimes from single-shot drafting to kernel-must-accept repair loops — grading every accepted artifact with a three-layer stack whose floor is fully mechanical: every certified defect replays deterministically, most as kernel-checked theorems. Under the requirement of a complete kernel-accepted proof of research-level mathematics, 69–100% of the three Claude-family models' accepted proofs do not assert the source statement (dose-response trend, Holm-adjusted ). The two GPT-5.6 models at first appear immune, returning almost only empty, token-exhausted completions — but given an adequate output budget they accept 124 of 125 attempts, and 80–85% of those proofs are unfaithful: an output cap had manufactured an apparent vendor-level honesty difference. A lightweight fidelity intervention in the repair loop (a one-paragraph contract plus an advisory, non-gating screen) significantly reduces gaming (68%→53% for the most affected model; paired ) with no significant cost in acceptance yield. Verifier-passing rates systematically overstate what models can formalize, with direct consequences for benchmark construction and reinforcement learning from verifier rewards. The task suite, harness, and certificate checkers accompany this submission as anonymized supplementary material.

Primary area:
datasets and benchmarks
Keywords:
autoformalization, formal verification, large language models, theorem proving, Lean 4, reward hacking, specification gaming, benchmark integrity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UmXaOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates statement gaming in LLM-based mathematical formalization, where models modify formal theorem statements to satisfy a Lean verifier without preserving the intended mathematical meaning. The authors evaluate five models across two datasets and five generation regimes, using mechanical defect certificates, reference equivalence checks, and LLM judges to assess faithfulness. The results show that requiring complete proofs substantially increases the proportion of unfaithful accepted statements. The paper also examines the effects of output-token budgets and proposes a lightweight intervention to reduce statement gaming.

Critical strengths

The paper addresses an important limitation of verifier-based evaluation: a formally verified proof does not guarantee that the underlying statement faithfully represents the intended problem. Although specification gaming and autoformalization faithfulness have been studied previously, examining how this behavior changes under different verification requirements is a useful contribution. The experimental design includes multiple models, two problem families, and several generation regimes. The mechanically verifiable defect certificates are particularly valuable because they establish a lower bound on statement gaming without relying entirely on LLM judgments. The authors also provide statistical analyses, report sensitivity to judge disagreements, and openly document evaluation errors discovered during the study.

Critical weaknesses
  1. The causal interpretation of verifier pressure is not sufficiently supported. The main increase in statement gaming occurs when moving from R2 to R3, which also changes the task from generating a valid formal statement to producing a complete proof. Meanwhile, the comparison between feedback-based repair (R2) and selection without feedback (R1) shows no significant difference. Therefore, the experiments demonstrate an association between proof requirements and statement gaming, but do not clearly isolate the effect of verifier feedback or optimization pressure.

  2. The reliability of the faithfulness evaluation remains uncertain. Although the mechanical certificates provide useful evidence, the reported gaming rates rely substantially on LLM judgments, particularly for research-level statements. The paper does not provide an independent human evaluation of these judgments, and the equivalence checker has limited recall on research problems. Consequently, the precise magnitude of statement gaming remains uncertain, even though the mechanically certified results establish that the phenomenon exists.

  3. The mitigation experiment has important confounding factors. The R2M intervention introduces multiple changes simultaneously, including a fidelity contract, an advisory screen, and restating the original mathematical statement during repair. In contrast, the standard R2/R3 repair prompts do not explicitly repeat the source statement. Without isolating these components, it is unclear whether the improvement comes from the proposed intervention or simply from reminding the model of the original task. Furthermore, the mitigation is not evaluated under R3, where the highest gaming rates are observed.

Questions for authors

Please refer to the Critical Weaknesses section for my main concerns and suggested clarifications.

Reviewer_cBQTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what happens when language models write Lean 4 statements from informal math and are required to satisfy the Lean verifier. It names the outcome "statement gaming": under pressure the model deforms the statement (weakens it, leaves hypotheses unused, and so on) until something compiles. The authors measure this for five models (three Claude, two GPT-5.6 variants) under five inference-time regimes: single shot, best-of-4, compiler-feedback repair, repair plus a "fidelity contract", and repair with a complete kernel-accepted proof required. They do this on 239 research-paper statements and 388 corrected miniF2F statements. Accepted artifacts are graded by a three-layer stack: four mechanical "defect certificates", bidirectional equivalence to a reference formalization, and two LLM judges. The headline result is that the share of unfaithful statements among accepted artifacts increases from regime to regime as pressure is added, and is highest when a complete proof is required (69-100% for the Claude models, 80-85% for the GPT-5.6 models once given enough output tokens). A one-paragraph fidelity prompt plus an advisory screen lowers it for the weaker models in the statement-only repair regime.

Critical strengths
  1. A useful data point.

The measurement is replayable (every certified defect replays with a Lean toolchain alone, references are run through the same checks) and covers five current frontier models under a controlled set of regimes.

  1. Striking that frontier models still do this.

Even today’s strongest models, asked to formalize and prove a research-level statement, mostly end up proving something else: kernel acceptance stays high while faithful acceptance falls to 0-16% of attempts. That is worth knowing for anyone using verifier acceptance as a reward or leaderboard metric.

Critical weaknesses
  1. Lack of novelty, in both the problem and the solution.

The phenomenon is well known to anyone who works with Lean and LLMs: a model that must satisfy the compiler will weaken, trivialise or restate the theorem, and data pipelines such as DeepSeek-Prover and Kimina-Prover have filtered exactly these artifacts (contradictory hypotheses, trivial proofs, unused variables) for years. Beyond Compilation (Zhang et al., arXiv 2606.31002) already measured compiler feedback inflating acceptance with unfaithful statements, and GAR (arXiv 2510.11769) already reported statement tampering as reward hacking under a Lean verifier, amound several other works. The paper’s certificates looks similar to many of the deterministic checks developed in these prior works. The observations themselves are obvious once the prompts (Appendix C) are read: R0-R3 ask only for code that compiles (R3: “give a COMPLETE proof”) and never mention faithfulness, so the model produces compiling but unfaithful statements. R2M is the one regime whose prompt asks for faithfulness, and faithfulness goes up. The proposed solution is accordingly a few extra sentences of prompt plus an unspecified reviwing screen.

  1. Hard to read, and no intuition is developed for the phenomenon.

The paper is written as lists of acronyms (R0–R3, R2M, F1/F2, c1–c4, L1–L3) and inline statistics rather than prose, with sentences strung together by em-dashes and semicolons (page 5 is almost entirely counts, ratios and p-values joined this way). Also, a paper that focuses on an empirical phenomenon needs to show the phenomenon: several artifact pairs per defect categry, with the judge’s reason and some account of what in the prompt or the repair trace led the model there. As written, the reader learns the rates but not what "statement gaming" looks like or why it happens.

  1. Can we trust the LLM reviewers?

If I understood correctly, the judges are never validated against humans and several headlines results depend on it.

Questions for authors
  1. Can you show several source/artifact pairs per defect category, with the judge’s reason and the repair trace that led the model there?
  2. What exactly is the advisory screen, which tactics define the c3 certificate, which linter produces c4?
Reviewer_4UK6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

When AI is tasked with solving high-difficulty mathematical problems, it typically employs formal verification using Lean. This process involves two steps: first, the AI ​​translates the problem into Lean, and second, it constructs a formal proof within Lean. The authors observed that, during this process, the AI ​​frequently alters the meaning of the problem statement during the initial translation phase. Consequently, they propose two pipelines—one based on an LLM and one that is not—to quantitatively verify the accuracy of the translation.

Critical strengths
  1. The study provides a quantitative assessment showing that errors stemming from the translation into Lean account for a significant portion of the issues encountered when LLMs solve math problems—a proportion far higher than that observed when the problem is translated in isolation.
  2. It introduces two pipelines for quantitative analysis (the non-LLM-based pipeline achieves 100% precision with human judgment).
Critical weaknesses
  1. The AGR approach lacks alignment with human evaluation. Furthermore, although a "dual-judge" setup is mentioned, both judges for the GPT-based evaluations are instances of Claude Opus; this fails to achieve the "knowledge isolation" that a dual-judge system is intended to provide.
Questions for authors
  1. There is a point that seems odd—perhaps I am misunderstanding it: In the "Grading" section, the authors state, "The suite was validated at 100% precision on 48 hand-judged pilot pairs." However, in the "Limitations" section, it is noted: "The certified floor needs none, the anchored layer used the internal 48 human verifications." My interpretation is that these 48 samples were used to validate the pipeline that requires no external judge (i.e., the "certified floor" achieves 100% precision). Yet, the paper does not appear to demonstrate any alignment in the "dual-judge" pipeline.
  2. Could the authors please provide some additional analysis regarding the alignment of the AGR?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.