- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates statement gaming in LLM-based mathematical formalization, where models modify formal theorem statements to satisfy a Lean verifier without preserving the intended mathematical meaning. The authors evaluate five models across two datasets and five generation regimes, using mechanical defect certificates, reference equivalence checks, and LLM judges to assess faithfulness. The results show that requiring complete proofs substantially increases the proportion of unfaithful accepted statements. The paper also examines the effects of output-token budgets and proposes a lightweight intervention to reduce statement gaming.
Critical strengths
The paper addresses an important limitation of verifier-based evaluation: a formally verified proof does not guarantee that the underlying statement faithfully represents the intended problem. Although specification gaming and autoformalization faithfulness have been studied previously, examining how this behavior changes under different verification requirements is a useful contribution. The experimental design includes multiple models, two problem families, and several generation regimes. The mechanically verifiable defect certificates are particularly valuable because they establish a lower bound on statement gaming without relying entirely on LLM judgments. The authors also provide statistical analyses, report sensitivity to judge disagreements, and openly document evaluation errors discovered during the study.
Critical weaknesses
-
The causal interpretation of verifier pressure is not sufficiently supported. The main increase in statement gaming occurs when moving from R2 to R3, which also changes the task from generating a valid formal statement to producing a complete proof. Meanwhile, the comparison between feedback-based repair (R2) and selection without feedback (R1) shows no significant difference. Therefore, the experiments demonstrate an association between proof requirements and statement gaming, but do not clearly isolate the effect of verifier feedback or optimization pressure.
-
The reliability of the faithfulness evaluation remains uncertain. Although the mechanical certificates provide useful evidence, the reported gaming rates rely substantially on LLM judgments, particularly for research-level statements. The paper does not provide an independent human evaluation of these judgments, and the equivalence checker has limited recall on research problems. Consequently, the precise magnitude of statement gaming remains uncertain, even though the mechanically certified results establish that the phenomenon exists.
-
The mitigation experiment has important confounding factors. The R2M intervention introduces multiple changes simultaneously, including a fidelity contract, an advisory screen, and restating the original mathematical statement during repair. In contrast, the standard R2/R3 repair prompts do not explicitly repeat the source statement. Without isolating these components, it is unclear whether the improvement comes from the proposed intervention or simply from reminding the model of the original task. Furthermore, the mitigation is not evaluated under R3, where the highest gaming rates are observed.
Questions for authors
Please refer to the Critical Weaknesses section for my main concerns and suggested clarifications.