acceptodds
Under review as a conference paper at ICLR 2027

Understanding Disagreements in Lean Formalization Evaluation

Abstract

Evaluating the faithfulness of a Lean formalization requires interpreting both its natural-language source and its formal statement. LLM judges, human reviewers, and published labels can disagree because of translation defects, mistaken judgments, or differences in interpretation and grading criteria. We investigate these disagreements using three LLM judges on CriticLeanBench, MA-Align, and 208 formalizations with existing human ratings. An author audit of 75 CriticLeanBench items includes all 51 cases where the judges unanimously contradict the label: the audit supports the judges in 44 cases, the label in 2, and leaves 5 unresolved. Source comparisons and targeted Lean checks reveal missed translation defects, ambiguous conventions, and incorrect model objections. LLM judgments also help identify formalizations on which human reviewers disagree, without using human ratings to select them. Together, these findings show how LLM judgments can guide review, while source comparisons and targeted checks distinguish translation defects from differences in interpretation and grading criteria.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-Judge, Human–AI Evaluation, Statement Autoformalization, Semantic Faithfulness, Noisy Annotators, Dawid–Skene Estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_e3f1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates disagreements among LLM judges, human reviewers, and published benchmark labels when evaluating whether Lean formalizations of natural-language mathematical statements faithfully preserve the mathematical meaning of the source.

To this end, this paper evaluates 208 formalizations from CriticLeanBench and MA-Align for which published labels and prior human ratings are available. Three LLM judges based on GPT, Gemini, and Claude assess the faithfulness of each formalization. Each judge produces three grades on a 0–10 scale, with grades of 9 or above treated as faithful. The paper then compares individual model judgments, majority voting, unanimous acceptance, and Dawid–Skene (DS) aggregation, which estimates observer error rates.

This paper also studies model-split, DS-uncertainty, and grade-margin inspection rankings for determining which items should be prioritized for human review using only LLM outputs. For CriticLeanBench, the authors audit 75 items, including 51 cases in which all three LLM judges disagree with the published label, and investigate the sources of disagreement through source comparison and targeted Lean checks.

The main finding is that, among the 51 unanimous LLM–label disagreements, the author audit supports the LLM judges in 44 cases, supports the published label in 2 cases, and leaves 5 unresolved. The paper also finds that LLM-only rankings select formalizations involving human-reviewer disagreement, or cases rejected during earlier screening but accepted by a human majority, at higher rates than random selection. Meanwhile, DS produces the same binary verdicts as majority voting on the two labeled benchmarks and yields overconfident probabilities relative to the published labels on CriticLeanBench. Simple grade-margin ranking performs similarly to or better than DS in disagreement yield across several review budgets.

Critical strengths

1. Careful analysis of disagreement beyond simple evaluator accuracy

A major strength of this paper is that it does not simply treat disagreements between LLM judges and benchmark labels as LLM evaluation errors. Instead, it investigates which side of the disagreement is actually incorrect through source comparison and targeted Lean checks.

For example, the paper presents a case in which a published faithful label is incorrect because of a missing domain restriction, another in which two of three human reviewers overlook a strengthened hypothesis requiring absolute summability even though the source only requires ordinary convergence, and a case in which an LLM raises an incorrect objection because it misparses the scope of a Lean big operator. These examples concretely demonstrate that none of the benchmark label, human reviewer, or LLM judge can automatically be treated as ground truth.

2. Practical framing of LLM judges as review-prioritization signals

Another strength is that the paper goes beyond using LLM judges as final oracles and instead uses their outputs to prioritize which formalizations should be inspected under a limited human-review budget.

In particular, this paper shows that LLM-derived signals can concentrate reviewer disagreement without using human ratings during ranking. For example, in Panel A, at a 20% review budget, 77.2% of the items selected by grade-margin ranking contain reviewer disagreement, compared with 43.9% under random selection.

Critical weaknesses

1. Limited independent evidence for the paper’s central audit result

One of the paper’s central findings is that, among the 51 CriticLeanBench cases in which all three LLM judges disagree with the published label, the author audit supports the LLM judges in 44 cases. However, these 44 cases are not all validated with the same strength of evidence. The results in Section 4.2 indicate that a substantial fraction of the judge-supported cases are in the direction where the published label is unfaithful, while all three judges and the author audit classify the formalization as faithful. Based on the reported results, at least 28 of the 44 judge-supported cases fall into this category. Unlike an unfaithfulness judgment, which can sometimes be established by identifying a single counterexample or omitted assumption, such cases require establishing that no important semantic mismatch is present. Section 4.2 itself distinguishes these acceptance decisions as “judgments, not proofs.” Therefore, a substantial portion of the headline 44/51 result relies on positive faithfulness judgments rather than direct evidence such as Lean-verified counterexamples or formal implication checks.

Moreover, although the audit was conducted without access to the published labels or judge scores, it remains an author-conducted, AI-assisted audit rather than an independent validation. Consequently, particularly for published-unfaithful / judges-accept cases, it is difficult to distinguish whether the audit has identified a genuine benchmark-label error or simply produced another judgment based on similar interpretations or grading conventions.

2. Review-prioritization results rely on noisy proxy outcomes and exploratory analyses

The effectiveness of the LLM-only ranking is evaluated primarily by how often it selects cases involving human-reviewer disagreement or human-majority acceptance, rather than by how effectively it identifies genuinely incorrect formalizations or incorrect annotations. However, a central premise of the paper is precisely that human judgments can themselves be erroneous. Therefore, successfully prioritizing cases with high reviewer disagreement does not necessarily imply successful identification of semantic faithfulness errors.

Indeed, agreement among the human reviewers is itself relatively low. In Panel A, pairwise Cohen’s (\kappa) ranges from 0.15 to 0.27, and in Panel B it ranges from 0.31 to 0.41. The same reviewer’s acceptance behavior also varies substantially across panels. For example, R1 has the highest acceptance rate among the human reviewers in Panel A but the lowest acceptance rate in Panel B. These results suggest that reviewer disagreement may reflect reviewer-specific interpretations or grading criteria in addition to genuine semantic defects.

Thus, the current experiments show that LLM-based ranking can concentrate cases in which evaluators are likely to disagree, but they do not establish whether using such rankings leads to the discovery of more incorrect annotations or improves final evaluation correctness. The ranking outcomes evaluated in the paper are reviewer disagreement or majority acceptance rather than verified faithfulness, and the paper does not measure whether subsequent review actually identifies and corrects errors in the selected items.

Questions for authors

1. Independent validation of the audit result

Among the 44 cases in which the audit supports the judges out of the 51 unanimous LLM–label disagreements, how many are supported by concrete evidence such as a Lean-verified counterexample, a formal implication check, or direct source comparison? In particular, for cases where the published label is unfaithful but the judges and audit classify the formalization as faithful, positive faithfulness is inherently more difficult to verify directly. Was any separate independent expert re-evaluation conducted for these cases? I believe this distinction is important for assessing how strongly the 44/51 result should be interpreted as evidence of genuine benchmark-label errors.

2. Does review prioritization improve final evaluation correctness?

When items selected by the LLM-only ranking are subjected to additional expert review, does this lead to a higher rate of identifying and correcting incorrect annotations or semantic defects compared with random selection or other baselines? Based on the current results using human disagreement or human-majority acceptance as outcomes, it is difficult to distinguish disagreement detection from actual error detection.

2 comments
AuthorsOpenReview ↗

Response to Reviewer e3f1

Thank you for the accurate and detailed review. Both weaknesses are correct, including your estimate: 30 of the 44 judge-supported cases are positive faithfulness judgments. Changes are marked in blue in the revised PDF; the revision also contains minor wording edits that change no result.

Added to the revised paper

  • Sec. 4.4, Table 4: a new blind audit of the items each LLM-only ranking places first (55 items, with controls).
  • Sec. 4.2, Table 2, App. L: Lean checks for 22 items labeled faithful (3 at submission), in six defect types.
  • App. M: the outcome for all 130 audited items (Table 26), the 16 items whose natural-language statement is itself flawed (Table 27), and every audit record.
  • Supplement: Lean files with compilation logs, and the audit records.

W1 / Q1. Evidence behind the 44; independent re-evaluation.

DirectionAudit sides with judgesEvidence
Label faithful, judges reject1413 have a compiled Lean check (3 at submission); 1 is weaker than its source but not false
Label unfaithful, judges accept30Judgments only
  • Each Lean check proves that the released formal statement is false, vacuous, or true only by a library convention. Examples: Lean's sup metric on ℝ × ℝ where the source means Euclidean distance; a hypothesis α ≠ 2nπ that reuses the number of summands; numerals that all elaborate as natural numbers, so the statement reads 0 = 0.
  • For the 30, no independent expert re-evaluation was conducted, and the audit was AI-assisted. We agree they may be a second judgment under similar readings. The abstract and contributions no longer present them as label errors.

Headline result. Of the 130 audited items, the audit agrees with the label on 45 and finds a checkable defect in 38: 22 Lean-checked, and 16 with a flawed source statement. Another 42 disagreements with the label are judgments without formal evidence, and 5 items are unresolved.

W2 / Q2. Does prioritization find errors, or only disagreement?

For the human panels, only disagreement; Sec. 5 now says so. For the labeled benchmark we ran the experiment you describe (Sec. 4.4). We audited, blind to labels and judge outputs, every unaudited item in the top 10% (43 items) of each ranking, plus random controls: 55 items, with the analysis fixed before review. Each row shows what the audit found in one ranking's first 43 items.

RankingLabel wrongJudge majority wrongBoth wrongNeitherSource flawedUndecided
Model splits16.21.04.010.28.02.0
DS uncertainty16.21.04.010.28.12.0
Grade margin17.51.02.912.17.91.6
  • About half of each list has an error on at least one side. The rows are close because at this budget the three rankings select almost the same items, the split-vote ones. Among 26 controls with a definite verdict, none does.
  • Where judge majority and label conflict, the audit sides against the label on 15 items and against the majority on 1.
  • The corrections are checkable: of the 12 items labeled faithful that this audit rejects, 9 have a compiled Lean check, bringing the total to 22. The other 3 deviate from the source without becoming false and remain judgments.
  • The judges err too: one item labeled unfaithful, accepted by GPT and Gemini, is false in Lean.
  • "Source flawed": the natural-language statement itself is wrong or underspecified, so no verdict on the translation is given. Examples: a recurrence that leaves a2a_2 undefined; a stated minimum that fails without a positivity assumption. Table 27 lists all 16; the benchmark labels 10 of them faithful and 6 unfaithful.

How the audit was done, and its limits. The auditing author is a PhD student in applied mathematics (qualifying exams in applied mathematics and PDE) who reads basic Lean. An assistant model (Claude Sonnet 5.5) first explained each Lean statement without seeing the source, and the author recorded a verdict. The author then discussed source and Lean with the model and recorded a final verdict. The final verdict reversed the initial one on 14 of 55 items, 13 of them to unfaithful; 11 of these 13 are corroborated by a Lean check (6) or by agreement with the label and all three judges (5). No second expert was available, and review time was not measured. For these reasons the defects we claim rest on the Lean checks, not on the verdicts alone.

Thank you for the detailed response and revisions. While the additional analyses address some of my concerns, the lack of independent validation remains a limitation. I therefore maintain my original rating.

Reviewer_ePteOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at disagreements between LLM judges, human reviewers and benchmark labels when evaluating the mathematical correctness of lean formalizations. It finds that LLM judgments can help uncover labelling errors and prioritize examples for human review. Further, sophisticated aggregation methods such as Dawid–Skene often offer little advantage over simple voting or grade-based approaches.

Critical strengths
  1. The paper goes beyond simply measuring agreement between LLM judges and existing labels by investigating disagreements and providing mathematical arguments and targeted Lean checks for specific cases.
  2. The paper demonstrates that LLM judgments can identify examples likely to produce human-reviewer disagreement or warrant reconsideration.
  3. The comparisons against simple voting, grade-based rankings, and Dawid–Skene are valuable, particularly in showing that more sophisticated aggregation does not necessarily improve results.
Critical weaknesses
  1. Nahum et al. (EMNLP 2025) already use LLM ensembles to detect benchmark-label errors, while Poiroux et al. (EMNLP 2025) and Ammanamanchi et al. (ICML 2026) investigate errors in Lean formalization benchmarks. This paper provides new case studies, but no substantially new evaluation or verification method.
  2. Human inter-rater agreement is relatively low (κ = 0.15–0.41). More importantly, removing one reviewer's deductions for stylistic issues reduces Panel B's observed disagreements from 25 to 17.
  3. The claim that the audit supports the LLM judges in 44 of 51 disagreements comes from an author-conducted, AI-assisted audit. Lean checks verify specific errors, but do not formally establish all 44 judgments.
  4. The authors retrospectively rank existing examples and measure how many disagreements appear in the selected sets. They never demonstrate that experts actually correct more errors, achieve higher evaluation accuracy, or save review time using these rankings.
Questions for authors
  1. How does this work substantially advance beyond prior research on LLM-ensemble benchmark auditing, disagreement-based review prioritization, and Lean benchmark error detection?
  2. The headline finding that LLM judges are supported in 44 of 51 unanimous disagreements relies on an author-conducted, AI-assisted audit. How many of these 44 cases are supported by conclusive mathematical evidence versus subjective judgments?
  3. The ranking experiments primarily measure disagreement with benchmark labels or human reviewers, neither of which necessarily indicates mathematical incorrectness. Can the authors evaluate whether their rankings identify more independently verified translation errors than random selection and simple baselines under the same expert-review budget?
2 comments
AuthorsOpenReview ↗

Response to Reviewer ePte

Thank you for the review and for the three references. Changes are marked in blue in the revised PDF; the revision also contains minor wording edits that change no result.

Added to the revised paper

  • Sec. 4.4, Table 4: a new blind audit of the items each LLM-only ranking places first (55 items, with controls).
  • Sec. 4.2, Table 2, App. L: Lean checks for 22 items labeled faithful (3 at submission), in six defect types.
  • App. M: the outcome for all 130 audited items (Table 26), the 16 items whose natural-language statement is itself flawed (Table 27), and every audit record.
  • Sec. 2: a paragraph on errors in benchmarks, citing the three papers you named.
  • Supplement: Lean files with compilation logs, and the audit records.

W1 / Q1. Advance over prior work. We agree: the paper proposes no new method, and "LLMs flag, humans verify" is already in Ammanamanchi et al. The revision claims four things those works do not cover.

  1. Aggregation. Dawid–Skene gives the same verdicts as majority voting on both benchmarks and is overconfident, because the judges' votes stay strongly associated within each label class.
  2. A limit of disagreement-based selection. 51 of 71 majority–label conflicts occur where all judges agree, so rankings built on judge disagreement cannot find them.
  3. Human disagreement. A ranking that uses no human data concentrates the items on which human reviewers split.
  4. A verified catalogue for a faithfulness benchmark. Ammanamanchi et al. mechanically certify defects in theorem-proving benchmarks. We apply the same standard to faithfulness labels: 22 CriticLeanBench items labeled faithful whose formal statement is false, vacuous or trivially true in Lean. They fall into six recurring types, e.g. Lean's sup metric on ℝ × ℝ where the source means Euclidean distance (3 items) and a mis-scoped binder, operator or name (5 items).

W2. Low κ; style deductions. Both are correct and come from our own sensitivity analyses. Sec. 5 now states that the panel outcomes are reviewer disagreement and majority acceptance, not verified faithfulness. Without the style deductions the selected rate is 35.4% against a 19.1% baseline: smaller, still present (App. I).

W3 / Q2. How many of the 44 are conclusive?

  • 14 are items labeled faithful in which the audit names a defect; 13 of these now have a compiled Lean check (3 at submission).
  • 30 are items labeled unfaithful in which the audit found no defect. These are judgments, and the abstract no longer presents them as label errors.

Headline result. Of the 130 audited items, the audit agrees with the label on 45 and finds a checkable defect in 38 (22 Lean-checked, 16 with a flawed source statement). Another 42 disagreements are judgments, and 5 items are unresolved.

W4 / Q3. Verified errors at a fixed budget. We ran this experiment (Sec. 4.4). Blind to labels and judge outputs, we audited every unaudited item in the top 10% (43 items) of each ranking, plus random controls, with the analysis fixed before review. Each row is one ranking's first 43 items.

RankingLabel wrongJudge majority wrongBoth wrongNeitherSource flawedUndecided
Model splits16.21.04.010.28.02.0
DS uncertainty16.21.04.010.28.12.0
Grade margin17.51.02.912.17.91.6
  • About half of each list has an error on at least one side. For random selection we estimate about 7 of 43 from all audited strata. Among 26 controls with a definite verdict, none has an error.
  • The rows are close because at this budget the rankings select almost the same items; the simple model-splits rule matches DS exactly.
  • "Source flawed": the natural-language statement itself is wrong or underspecified, so no verdict on the translation is given. Table 27 lists all 16; the benchmark labels 10 of them faithful and 6 unfaithful.
  • The new defects are checkable: of the 12 items labeled faithful that this audit rejects, 9 have a compiled Lean check. The other 3 deviate from the source without becoming false and remain judgments.

How the audit was done, and its limits. The auditing author is a PhD student in applied mathematics who reads basic Lean. An assistant model (Claude Sonnet 5.5) first explained each Lean statement without seeing the source, and the author recorded a verdict. The author then discussed source and Lean with the model and recorded a final verdict. The final verdict reversed the initial one on 14 of 55 items, 13 of them to unfaithful; 11 of these 13 are corroborated by a Lean check (6) or by agreement with the label and all three judges (5). No second expert was available (stated as a limitation), and review time was not measured. The defects we claim rest on the Lean checks.

Thank you for the additional audit and for clarifying the evidence behind the original 44 cases. The distinction between defects supported by Lean checks and cases where the auditor found no defect addresses my concern about that result.

My main reservation is still the contribution beyond prior work on benchmark auditing. The new cases show that the rankings can surface genuine translation errors. What remains unclear is how many more verified errors an expert would find this way than by reviewing a random sample of the same size, since the controls were drawn from cases where the judges already agreed with the label. I think the revision strengthens the paper, but the advance over prior work still feels modest to me, so I’m keeping my rating at 2 (weak rejection).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to Reviewer kUpY

Thank you for the careful review. We agree with all three weaknesses and have addressed them with new evidence. Changes are marked in blue in the revised PDF; the revision also contains minor wording edits that change no result.

Added to the revised paper

  • Sec. 4.4, Table 4: a new blind audit of the items each LLM-only ranking places first (55 items, with controls).
  • Sec. 4.2, Table 2, App. L: Lean checks for 22 items labeled faithful (3 at submission), in six defect types.
  • App. M: the outcome for all 130 audited items (Table 26), the 16 items whose natural-language statement is itself flawed (Table 27), and every audit record.
  • Supplement: Lean files with compilation logs, and the audit records.

Headline result. Of the 130 audited items, the audit agrees with the label on 45 and finds a checkable defect in 38: 22 Lean-checked, and 16 with a flawed source statement. Another 42 disagreements with the label are judgments without formal evidence, and 5 items are unresolved.

W1. The loop is not closed. It is now (Sec. 4.4). The question was whether the items these rankings put first actually contain errors. We audited them; each row is one ranking's first 43 items (10%).

RankingLabel wrongJudge majority wrongBoth wrongNeitherSource flawedUndecided
Model splits16.21.04.010.28.02.0
DS uncertainty16.21.04.010.28.12.0
Grade margin17.51.02.912.17.91.6
  • About half of each list has an error on at least one side. The rows are close because at this budget the three rankings select almost the same items, the split-vote ones. Among 26 controls with a definite verdict (judges and label agree), none does.
  • "Source flawed": the natural-language statement itself is wrong or underspecified, so no verdict on the translation is given. Examples: a recurrence that leaves a2a_2 undefined; an answer list containing non-solutions. Table 27 lists all 16; the benchmark labels 10 of them faithful and 6 unfaithful.
  • The judges err too: one item labeled unfaithful, accepted by GPT and Gemini, is false in Lean (an out-of-scope atTop is bound as an arbitrary real).
  • Qualifications: one "label wrong" verdict was withdrawn because we could not substantiate it, and three of the four "both wrong" items deviate from the source without becoming false, so they remain judgments.

W2. Labels treated as ground truth. That was not our intent, and the text now says so.

  • "FP/FN" are renamed and described as disagreements with the label.
  • DS and majority voting give identical verdicts under any reference, and DS has the worse Brier score under both. Accuracy (84.7% vs 95.5%) and calibration depend on the reference (App. H.1, Table 16).
  • We confirm your observation and now cite it: the CriticLean paper reports 84.0% for DeepSeek-R1, and our judges reach 82.8–86.8% against the same labels.

W3. The audit behind 44/51 is weak. Agreed. We now report it by evidence.

  • 14 of the 44 are items labeled faithful in which the audit names a defect; 13 of these have a compiled Lean check.
  • 30 are items labeled unfaithful in which the audit found no defect. These are judgments, and the abstract no longer presents them as label errors.
  • The panels are not audited (except B081). Sec. 5 now states that its outcomes are reviewer disagreement and majority acceptance, not verified faithfulness.

Q1. Which model?

  • Audit at submission: the ChatGPT web interface. It displayed only a reasoning setting (“medium”), not a model name, and we did not record the version. To the best of our recollection it was GPT-6 Astra or GPT Sol 6.1, so it may coincide with our GPT judge (GPT-6 Astra).
  • New audit: Claude Sonnet 5.5. The auditing author is a PhD student in applied mathematics (qualifying exams in applied mathematics and PDE) who reads basic Lean. The model first explained each Lean statement without seeing the source, and the author recorded a verdict. The author then discussed source and Lean with the model and recorded a final verdict.
  • The final verdict reversed the initial one on 14 of 55 items, 13 of them to unfaithful. Eleven of these 13 are corroborated independently: 6 by a Lean check, 5 by agreement with the label and all three judges.
  • Because both assistants are related to the judges, the defects we claim rest on the Lean checks, not on the verdicts alone.

Q2. Yields against the audit-corrected reference. With the reference available at submission, each 43-item list contains 16–17 of the 19 known disagreements (1.9 at random), and grade margin's advantage at larger budgets largely disappears. That comparison favors the rankings by construction, because the unaudited items were the ones ranked first. The table under W1 removes this problem.

~Afendulev_Kirill1OpenReview ↗

Thanks for the detailed response. Quick thoughts on each of my points.

W1 (loop not closed) I was worried that the rankings were never tested on real errors. The new blind audit solves this. One small thing left: the controls come from cases where judges and label already agree, so "half vs none" is not fully comparison I had in mind. A random sample would have been cleaner.

W2 (labels as ground truth) My point was that the reference itself is about as good as DeepSeek-R1. The renaming, the second reference in Table 16, and the note about the 85% overlap are enough for me.

W3 (the 44/51 audit is weak) My main worry was that lean-checked defects and judgment calls were mixed in one number. Splitting 14 (13 with Lean) vs 30 and removing the 30 from abstract is the right move. I am still not fully convinced by the new "38 checkable defects " line - the 16 flawed-source items feel like a different thing from translation errors, and this split was not part of the original plan.

Q1 (which model) Thanks for the honest answer. One small point: agreement with the three judges is not really independent support for the reversals, since the concern is exactly that they share readings. Only the 6 Lean checks really stand on their own.

Overall the revision makes the paper stronger. Raising my rating by 0.5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.