acceptodds
Under review as a conference paper at ICLR 2027

What Reviewers Catch and Frontier Models Miss: Statement-Level Errors in Autoformalization

Abstract

A formal statement can typecheck and admit a proof while failing to express the intended mathematics. We study such statement-level errors where they naturally occur: in the human review process of a Lean 4 library of formalized open problems. From squash-merged pull-request history we recover 74 natural negatives, erroneous statements corrected during human review. Each is hand-labeled against a 14-class error taxonomy, using the reviewer's fix and comments as the reference. We then evaluate twelve frontier models. When shown the reviewer's correction and comments, the best model distinguishes semantic errors from non-semantic changes at (); without the correction, no model exceeds . In detection, misses concentrate in the classes that depend on Lean semantics: wrong carriers, definitional mismatches, junk-value traps. Under the same detection prompt, the strongest models achieve drastically higher scores on evaluator-injected errors from the same error classes, with reaching . Supplying the file's definitions, forcing a pass through the class list, or saying only that an error is present helps at most modestly; naming the reviewer-assigned error class dramatically improves class-level recall, but wrong hints also elicit substantial label compliance. The corpus of 5,726 reviewed (informal, formal) pairs, the labeled negatives, and the evaluation harness will be released.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
autoformalization, formal mathematics, Lean 4, faithfulness evaluation, error detection, LLM evaluation, benchmarks, natural negatives

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BLiWOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies statement-level errors in autoformalization, i.e., formal statements that typecheck but do not express the informal intent. By collecting 74 such errors from the review history of the formal-conjectures Lean 4 library (PR versions that reviewers corrected before merging),label them with a 14-class taxonomy, and use 286 non-semantic edits from the same period as controls. Eleven frontier models reach F1 0.79 when the reviewer's correction is shown vs. 0.56 when only the docstring and statement are given, and most misses fall in wrong carriers, definitional mismatches and junk values. Adding file context or a per-class checklist changes this little, whereas naming the gold class raises class recall sharply. Under the same detection prompt, the strongest models detect rule-written, model-written and FormalAlign misalignments at F1 0.94–1.00 vs. 0.40–0.52 on natural errors of matched classes, from which the paper concludes that synthetic negatives overestimate detection ability.

Critical strengths

The paper identifies an important problem with how alignment scorers are validated, as candicates are mostly tested on synthetic misalignments. Using PR history to collect natural errors is a nice idea, and each error comes with the reviewer's fix and comments, which could facilitate future research on alignment evaluation. The comparison between natural and synthetic errors is generally careful, as the gap holds across three synthetic sets (rule-written, model-written and FormalAlign's released data) and a class-matched baseline. The distractor-hint control is also worth noticing, since it shows that models echo a wrong class on 27–55 of 70–74 errors.

Critical weaknesses
  1. The paper relies on a single data source, which may not fully reflect the errors that appear in practice. formal-conjectures consists of research-level problems such as open conjectures, whereas most autoformalization work targets high-school competition problems or undergraduate and graduate material, and 11 of the 74 errors come from a project-specific answer() convention. All labels also come from one annotator with no agreement statistic, and Appendix D shows that 59 of 360 labels flipped between the two passes. That's a thin basis for a claim about statement errors in general, and the PR-mining procedure, although presented as a template, is not tried on a second library.
  2. The paper appears to lack comparisons with prior work on semantic alignment. Checking alignment is already a standard step in autoformalization pipelines (LLM judges in Lean Workbook and FormalMATH), and several works tackle this problem directly and evaluate frontier models on misaligned statements (CriticLean, FormalRx). The 14-class taxonomy is also introduced without comparison to existing error categorization works. Without such comparisons, the novelty and positioning of the paper are less clear, and apart from the data source I currently have doubts about the level of its contribution.
Questions for authors

Clarification

  1. Can authors explain how were the 14 classes derived, and is there any evidence that the taxonomy is saturated? Five classes have 3 or fewer examples, whereas other holds 6. Could you list the 6 other errors? Every model finds 0/6 in blind detection (Table 13), and the Legendre-symbol example in Appendix D looks like a wrong-referent error or lack in taxonomy designing.2.
  2. In Appendix F, conjecture2 and conjecture4 fail through def a, which the blind detection prompt does not show. How many of the 74 errors are located outside the detection input, and since Table 5 uses the blind prompt, how does the synthetic gap change with file context?
  3. The per-class intervals in Table 13 overlap for most classes, and only wrong-type-or-domain and definitional-mismatch are clearly lower. Does the Lean-semantics claim still hold without junk-value-trap and wrong-answer-marker?
  4. The abstract mentions that revealing only the presence of an error helps at most modestly, but Table 4 shows Lean-semantics recall rising from 4/57 to 17/57 (Claude Haiku 4.5). Could you reconcile the two?
  5. Have you tried to reproduce and compare the existing methods like LeanScorer or FormalRx on the natural set?

Future work suggestions

  • Since the reviewer's correction is available for every error, I would like to see localization and correction evaluated against it.
  • A Lean REPL loop that evaluates expressions at boundary values seems the natural test for junk-value and carrier errors.

Minor in write-up

  • Muse Spark 1.2 has dashes in every column of Table 2, Table 5 reports F1 on 37 records (GLM 5.3) and 84 records (Gemini 3.1 Pro), and the synthetic rows were not retried. These runs should be completed or removed.
  • Some counts do not match (44–56 of 57 in Section 5 vs. 44/46 and 51/56 in Table 18, and 105 model-written errors vs. 100 in Table 12).
2 comments
AuthorsOpenReview ↗

Response to Reviewer BLiW

Thank you for the detailed review. The revision adds independent annotation of all 360 records, a LeanScorer baseline, and audits of input visibility and synthetic validity. We limit the findings to this repository and position the contribution as a review-derived evaluation set with paired corrections and provenance. The paper now discusses CriticLean, FormalRx, LeanScorer, Lean Workbook, and FormalMATH.

Annotation. A second annotator with greater Lean experience independently labeled all 360 records using the PR evidence, without LLM assistance or access to the first labels or model results. Before discussion, binary agreement was 329/360 (91.4%, κ=.759), and exact class sets agreed on 37/68 jointly positive records. Fable’s detection F1 is .559 under the submitted labels, .581 under the independent labels, and .555/.567 under the two adjudicated versions. Appendix G separates independent agreement, the subsequent AI-assisted decisions, and the 59 changes between the preliminary and submitted passes. Before the submitted pass, the first annotator had received Fable’s help interpreting Lean and had seen neither evaluation predictions nor aggregate scores.

Q1 · Taxonomy and other. We refined the labels while reviewing cases with Fable’s assistance. Many categories overlap prior work. Table 20 maps them to FormalRx’s prioritized scheme, and Appendix G discusses CriticLean and lists all six other cases. We reclassified the three rank cases as non-semantic in a separate sensitivity analysis. The Legendre cases overlap referent or argument errors and retain their submitted tags in the original scores.

Q2 · Omitted definitions and synthetic validity. The AI-assisted audit judged 46 of 74 natural errors inspectable from the detection input, 17 dependent on omitted definitions, and 11 unresolved. The separate audit of 105 model-written mutations supported 83 semantic errors, identified two non-errors, and left 20 unresolved. Five models other than the generator detected 81–83/83 retained synthetic errors versus 17–24/36 inspectable natural errors in overlapping classes, or 11–18/24 after excluding PR #5016. The comparison still differs in available information and difficulty. Synthetic class tags remain unverified, and the requested full-context synthetic comparison remains open.

Q3 · Project conventions and junk values. Across eight models with complete coverage, detection F1 is .268–.513 after excluding answer-marker records and .216–.480 after also excluding junk-value records. Models correctly assign 0–6 of the 31 remaining type/domain and definition labels. We treat these counts as descriptive.

Q4 · Presence hints. We revised the wording to reflect Haiku’s increase from 4/57 to 17/57 correct assignments.

Q5 · Existing-method baseline. We ran LeanScorer’s released prompts at the fixed .60 threshold with DeepSeek V4 Pro. After disclosed formatting recovery and retries, LeanScorer scores .379 F1 on 351 records, versus .441 for the original DeepSeek detector and .559 for Fable on the same records. Its recall is .811 and precision is .247. Table 21 reports each recovery step. FormalRx, localization, repair, and automatic Lean boundary-value checking remain open comparisons.

We also removed the severely incomplete rows from the main comparisons, reported shared denominators in Table 19, and restored the five wrong-referent mutations to the displayed inventory.

Thank you for the response. The independent annotation of all 360 records and the LeanScorer baseline address part of my concerns, and I have raised my score to 2. However, the limited scope remains an issue, since all data are drawn from a single repository, and several key comparisons are still missing.

Reviewer_caarOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks into a practical problem in autoformalization: even when a translated Lean 4 statement compiles and typechecks, it might not match the intended mathematical meaning of the informal text. Instead of relying on synthetic perturbations, the authors mined a few months of pull-request history from the formal-conjectures repository and collected 74 real-world semantic errors ("natural negatives") caught by human reviewers, grouped into 14 categories. Evaluating 12 frontier models, they find that without seeing the reviewer's fix, no model gets past an F1 of 0.56 on blind detection, with most failures happening on errors tied to Lean semantics (like junk values or definitional mismatches). By contrast, using the same prompt on rule-based or LLM-generated synthetic errors pushes F1 close to 1.00. The main takeaway is that models are good at spotting surface artifacts in synthetic data, but struggle with actual human formalization mistakes.

Critical strengths
  1. Realistic angle and good critique of synthetic benchmarks: Mining git review history is a clever way to get real mistakes. The best part of the paper is putting natural errors side-by-side with synthetic ones, and auditing the existing FormalAlign test set (finding commented-out empty statements and truncation bugs in their "aligned" data). It makes a convincing point that high scores on synthetic benchmarks can be misleading.

  2. Checking for model sycophancy: The distractor control in Section 5 is well thought out. Instead of taking the high recall under oracle class hints at face value, the authors fed the models wrong hints and showed that models will happily echo incorrect labels. I appreciate this level of caution.

Critical weaknesses
  1. The prompt setup and batching likely handicap the models: Looking at Appendix E, the evaluation packs 12 problems into a single prompt ("batches of 12") and forces the model to output a single JSON object with "exactly one sentence" for the rationale and "no visible scratchpad". Catching subtle definitional mismatches or type coercions in Lean requires step-by-step unpacking. Bundling 12 math statements together while forbidding chain-of-thought scratchpads artificially caps what the models can do in the blind detection setting. It also caused several models (like Gemini 3.1, Qwen 3.8, and Muse Spark) to hit token limits or drop records, making the cross-model comparisons messy.

  2. Small sample size and unstable single-annotator labels: The main conclusions rely on just 74 positive examples from a single repository. More concerning is the labeling process: only one author labeled the dataset, with zero inter-annotator agreement checks. Appendix D notes that between the preliminary pass and the final blind pass, 45 semantic labels were flipped to non-semantic, and 14 went the other way. Having 59 label flips when you only have 74 positives makes it hard to treat the ground truth as rock-solid.

  3. The natural vs. synthetic comparison isn't apples-to-apples: One of the hardest natural error classes is definitional-mismatch (15 out of 74 cases, where models almost completely fail). Yet the authors excluded definitional-mismatch entirely when building the rule-written and LLM-written synthetic sets. Also, the synthetic edits all happen locally inside the theorem statement line, without requiring the model to cross-reference external definitions. Comparing local, surface-level edits against natural errors that depend on external definitions exaggerates the performance gap. Finally, as an audit paper, it stops at diagnosing the problem without offering a practical fix or neuro-symbolic baseline.

Questions for authors
  1. What happens if you drop the batch-of-12 setup, run 1 record per request on the full 360 examples for the top 2-3 models, and allow unconstrained chain-of-thought reasoning before the JSON output? Does the blind detection F1 stay near 0.56, or does it jump significantly?

  2. Given the 59 label flips between your preliminary and final passes (Appendix D), could you have a second person familiar with Lean 4 independently label the 74 errors and a matching sample of controls during the rebuttal period to report human inter-annotator agreement?

  3. For the LLM-written synthetic arm (Opus 5), why not let the generator modify the auxiliary definitions in the file context to create synthetic definitional-mismatch examples? Also, since those 105 generated records were only typecheck-gated and not manually verified, how do we know they all actually introduced semantic errors?

1 comment
AuthorsOpenReview ↗

Response to reviewer caar

Thank you for the concrete suggestions. Appendix G reports new one-record-per-request runs, independent annotation of all 360 records, an audit of the model-written errors, and a LeanScorer comparison.

Q1 · Batching and reasoning. Fable and Grok each judged all 360 records one at a time with the original prompt and with longer rationales allowed. All 1,440 responses were scoreable.

ModelOriginal batchesOne record, original promptOne record, longer rationale
Fable.559.531.532
Grok.469.478.465

Neither model showed a substantial F1 gain. Fable found more errors with the original prompt one record at a time (43 vs. 38), while also giving more false alarms (45 vs. 24). Both variants retained JSON output and provider reasoning defaults. Free-form reasoning followed by a verdict remains untested, and differences in timing and routing limit attribution to batching alone.

Q2 · Independent annotation. A second annotator with greater Lean experience independently labeled all 360 records using the PR evidence, without LLM assistance or access to the first labels or model results. Before discussion, binary agreement was 329/360 (91.4%, κ=.759). Exact class sets agreed on 37/68 jointly positive records. Fable’s detection F1 is .559 under the submitted labels, .581 under the independent labels, and .555/.567 under the two adjudicated versions. Appendix G reports the 59 changes between preliminary and submitted labels separately from independent agreement and subsequent AI-assisted adjudication.

Q3 · Synthetic validity and auxiliary definitions. The AI-assisted audit of all 105 model-written mutations supported 83 semantic errors, identified two non-errors with targeted Lean checks, and left 20 unresolved. The intended class tags remain unverified. Five models other than the generator detected 81–83/83 retained synthetic errors versus 17–24/36 inspectable natural errors in overlapping classes. Excluding PR #5016 leaves 11–18/24 natural detections. These comparisons still differ in available information and difficulty. An additional arm editing auxiliary definitions would test a failure mode that the current generator omits. That arm remains untested.

Practical comparator. We evaluated LeanScorer’s released prompts with DeepSeek V4 Pro. After formatting recovery and retries, it scores .379 F1 on 351 records, versus .441 for the original DeepSeek detector and .559 for Fable on the same records. Its recall is .811 and precision is .247. Table 21 reports each recovery step. A neuro-symbolic repair baseline remains open.

Reviewer_XVzeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper collects naturally occurring autoformalization errors from the review history of a Lean 4 library. It compares models' ability to detect these errors with their performance on synthetic errors. Detection is much harder on the natural cases (best F1 0.56), despite much stronger results on injected errors. The study uses 74 semantic errors and 286 controls from 502 merged PRs, and tests twelve LLMs with and without the reviewers' corrections.

Critical strengths
  • The natural-versus-synthetic comparison is compelling and relevant to how faithfulness evaluators are tested.
  • The released predictions, prompts, and PR provenance make the experiments unusually easy to audit.
Critical weaknesses
  • Concentration of errors. PR #5016 accounts for 24 of the 74 semantic errors and a large share of several Lean-specific categories. Some errors also depend on auxiliary definitions that are absent from the blind-detection input. This makes the class-level recall results hard to interpret, even if the overall natural-versus-synthetic gap remains convincing.
  • Uncertain gold labels. The labels come from one annotator and changed substantially between passes (including 45 initially positive cases reclassified as non-semantic). Model rankings change with the label version. It also seems the final relabeling was done after the model results were known. Independent annotation is needed before drawing strong model-specific conclusions.
Questions for authors
  1. Do the class-level findings hold after excluding errors located in unseen definitions and grouping repeated error patterns from PR #5016?
  2. Can a second annotator independently review the semantic errors and a sample of controls? Please report agreement and sensitivity to the earlier labels.
  3. How many statement changes were prompted by reviewers rather than initiated by authors or CI?
1 comment
AuthorsOpenReview ↗

Response to Reviewer XVze

Thank you for highlighting the concentration, input visibility, and annotation concerns. Appendix G reports the new analyses, and the revised main text summarizes how they change the interpretation.

Q1 · Unseen definitions and repeated mechanisms. The AI-assisted audit judged 46 of 74 errors inspectable from the detection input, 17 dependent on omitted definitions, and 11 unresolved. PR #5016 contributes 24 records in 12 local mechanism groups. Weighting groups equally changes Fable’s binary recall from .514 to .574. On inspectable cases, Fable detects 32/46 errors, or 22/31 after excluding that PR. After both filters, it assigns the correct class to 3/6 definitional mismatches and 0/5 type or domain errors. Only one junk-value case remains, so we treat the class comparisons as descriptive. The aggregate contrast persists: five models detect 81–83/83 audited synthetic errors versus 17–24/36 inspectable natural errors in overlapping classes, or 11–18/24 after excluding PR #5016.

Q2 · Independent annotation and label sensitivity. A second annotator with greater Lean experience independently labeled all 360 records using the PR evidence, without LLM assistance or access to the first labels or model results. Before discussion, binary agreement was 329/360 (91.4%, κ=.759). Exact class sets agreed on 37/68 jointly positive records. Fable’s detection F1 is .559 under the submitted labels, .581 under the independent labels, and .555/.567 under the two adjudicated versions. The first annotator had received Fable’s help interpreting Lean but had seen no evaluation predictions or aggregate scores before the final pass. The preliminary labels reverse the ordering of Fable and Grok, so we limit claims about model rankings.

Q3 · Who prompted the changes? Among the 74 submitted positives, the available PR evidence attributes 32 corrections to reviewers, 2 to issues authors identified and addressed after reviewer requests, 8 to author-reported repairs, and 2 to upstream merges. Initiation is unknown for 30. We did not retrieve CI logs, so we cannot quantify CI-initiated changes. PR #5016 includes repeated effects of definition repairs, and its reviewer disclosed using AI to summarize personal notes. The revision describes these as errors recovered from the PR process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.