- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies statement-level errors in autoformalization, i.e., formal statements that typecheck but do not express the informal intent. By collecting 74 such errors from the review history of the formal-conjectures Lean 4 library (PR versions that reviewers corrected before merging),label them with a 14-class taxonomy, and use 286 non-semantic edits from the same period as controls. Eleven frontier models reach F1 0.79 when the reviewer's correction is shown vs. 0.56 when only the docstring and statement are given, and most misses fall in wrong carriers, definitional mismatches and junk values. Adding file context or a per-class checklist changes this little, whereas naming the gold class raises class recall sharply. Under the same detection prompt, the strongest models detect rule-written, model-written and FormalAlign misalignments at F1 0.94–1.00 vs. 0.40–0.52 on natural errors of matched classes, from which the paper concludes that synthetic negatives overestimate detection ability.
Critical strengths
The paper identifies an important problem with how alignment scorers are validated, as candicates are mostly tested on synthetic misalignments. Using PR history to collect natural errors is a nice idea, and each error comes with the reviewer's fix and comments, which could facilitate future research on alignment evaluation. The comparison between natural and synthetic errors is generally careful, as the gap holds across three synthetic sets (rule-written, model-written and FormalAlign's released data) and a class-matched baseline. The distractor-hint control is also worth noticing, since it shows that models echo a wrong class on 27–55 of 70–74 errors.
Critical weaknesses
- The paper relies on a single data source, which may not fully reflect the errors that appear in practice. formal-conjectures consists of research-level problems such as open conjectures, whereas most autoformalization work targets high-school competition problems or undergraduate and graduate material, and 11 of the 74 errors come from a project-specific answer() convention. All labels also come from one annotator with no agreement statistic, and Appendix D shows that 59 of 360 labels flipped between the two passes. That's a thin basis for a claim about statement errors in general, and the PR-mining procedure, although presented as a template, is not tried on a second library.
- The paper appears to lack comparisons with prior work on semantic alignment. Checking alignment is already a standard step in autoformalization pipelines (LLM judges in Lean Workbook and FormalMATH), and several works tackle this problem directly and evaluate frontier models on misaligned statements (CriticLean, FormalRx). The 14-class taxonomy is also introduced without comparison to existing error categorization works. Without such comparisons, the novelty and positioning of the paper are less clear, and apart from the data source I currently have doubts about the level of its contribution.
Questions for authors
Clarification
- Can authors explain how were the 14 classes derived, and is there any evidence that the taxonomy is saturated? Five classes have 3 or fewer examples, whereas other holds 6. Could you list the 6 other errors? Every model finds 0/6 in blind detection (Table 13), and the Legendre-symbol example in Appendix D looks like a wrong-referent error or lack in taxonomy designing.2.
- In Appendix F, conjecture2 and conjecture4 fail through def a, which the blind detection prompt does not show. How many of the 74 errors are located outside the detection input, and since Table 5 uses the blind prompt, how does the synthetic gap change with file context?
- The per-class intervals in Table 13 overlap for most classes, and only wrong-type-or-domain and definitional-mismatch are clearly lower. Does the Lean-semantics claim still hold without junk-value-trap and wrong-answer-marker?
- The abstract mentions that revealing only the presence of an error helps at most modestly, but Table 4 shows Lean-semantics recall rising from 4/57 to 17/57 (Claude Haiku 4.5). Could you reconcile the two?
- Have you tried to reproduce and compare the existing methods like LeanScorer or FormalRx on the natural set?
Future work suggestions
- Since the reviewer's correction is available for every error, I would like to see localization and correction evaluated against it.
- A Lean REPL loop that evaluates expressions at boundary values seems the natural test for junk-value and carrier errors.
Minor in write-up
- Muse Spark 1.2 has dashes in every column of Table 2, Table 5 reports F1 on 37 records (GLM 5.3) and 84 records (Gemini 3.1 Pro), and the synthetic rows were not retried. These runs should be completed or removed.
- Some counts do not match (44–56 of 57 in Section 5 vs. 44/46 and 51/56 in Table 18, and 105 model-written errors vs. 100 in Table 12).
2 comments
Thank you for the response. The independent annotation of all 360 records and the LeanScorer baseline address part of my concerns, and I have raised my score to 2. However, the limited scope remains an issue, since all data are drawn from a single repository, and several key comparisons are still missing.
Response to Reviewer BLiW
Thank you for the detailed review. The revision adds independent annotation of all 360 records, a LeanScorer baseline, and audits of input visibility and synthetic validity. We limit the findings to this repository and position the contribution as a review-derived evaluation set with paired corrections and provenance. The paper now discusses CriticLean, FormalRx, LeanScorer, Lean Workbook, and FormalMATH.
Annotation. A second annotator with greater Lean experience independently labeled all 360 records using the PR evidence, without LLM assistance or access to the first labels or model results. Before discussion, binary agreement was 329/360 (91.4%, κ=.759), and exact class sets agreed on 37/68 jointly positive records. Fable’s detection F1 is .559 under the submitted labels, .581 under the independent labels, and .555/.567 under the two adjudicated versions. Appendix G separates independent agreement, the subsequent AI-assisted decisions, and the 59 changes between the preliminary and submitted passes. Before the submitted pass, the first annotator had received Fable’s help interpreting Lean and had seen neither evaluation predictions nor aggregate scores.
Q1 · Taxonomy and
other. We refined the labels while reviewing cases with Fable’s assistance. Many categories overlap prior work. Table 20 maps them to FormalRx’s prioritized scheme, and Appendix G discusses CriticLean and lists all sixothercases. We reclassified the three rank cases as non-semantic in a separate sensitivity analysis. The Legendre cases overlap referent or argument errors and retain their submitted tags in the original scores.Q2 · Omitted definitions and synthetic validity. The AI-assisted audit judged 46 of 74 natural errors inspectable from the detection input, 17 dependent on omitted definitions, and 11 unresolved. The separate audit of 105 model-written mutations supported 83 semantic errors, identified two non-errors, and left 20 unresolved. Five models other than the generator detected 81–83/83 retained synthetic errors versus 17–24/36 inspectable natural errors in overlapping classes, or 11–18/24 after excluding PR #5016. The comparison still differs in available information and difficulty. Synthetic class tags remain unverified, and the requested full-context synthetic comparison remains open.
Q3 · Project conventions and junk values. Across eight models with complete coverage, detection F1 is .268–.513 after excluding answer-marker records and .216–.480 after also excluding junk-value records. Models correctly assign 0–6 of the 31 remaining type/domain and definition labels. We treat these counts as descriptive.
Q4 · Presence hints. We revised the wording to reflect Haiku’s increase from 4/57 to 17/57 correct assignments.
Q5 · Existing-method baseline. We ran LeanScorer’s released prompts at the fixed .60 threshold with DeepSeek V4 Pro. After disclosed formatting recovery and retries, LeanScorer scores .379 F1 on 351 records, versus .441 for the original DeepSeek detector and .559 for Fable on the same records. Its recall is .811 and precision is .247. Table 21 reports each recovery step. FormalRx, localization, repair, and automatic Lean boundary-value checking remain open comparisons.
We also removed the severely incomplete rows from the main comparisons, reported shared denominators in Table 19, and restored the five wrong-referent mutations to the displayed inventory.