- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates disagreements among LLM judges, human reviewers, and published benchmark labels when evaluating whether Lean formalizations of natural-language mathematical statements faithfully preserve the mathematical meaning of the source.
To this end, this paper evaluates 208 formalizations from CriticLeanBench and MA-Align for which published labels and prior human ratings are available. Three LLM judges based on GPT, Gemini, and Claude assess the faithfulness of each formalization. Each judge produces three grades on a 0–10 scale, with grades of 9 or above treated as faithful. The paper then compares individual model judgments, majority voting, unanimous acceptance, and Dawid–Skene (DS) aggregation, which estimates observer error rates.
This paper also studies model-split, DS-uncertainty, and grade-margin inspection rankings for determining which items should be prioritized for human review using only LLM outputs. For CriticLeanBench, the authors audit 75 items, including 51 cases in which all three LLM judges disagree with the published label, and investigate the sources of disagreement through source comparison and targeted Lean checks.
The main finding is that, among the 51 unanimous LLM–label disagreements, the author audit supports the LLM judges in 44 cases, supports the published label in 2 cases, and leaves 5 unresolved. The paper also finds that LLM-only rankings select formalizations involving human-reviewer disagreement, or cases rejected during earlier screening but accepted by a human majority, at higher rates than random selection. Meanwhile, DS produces the same binary verdicts as majority voting on the two labeled benchmarks and yields overconfident probabilities relative to the published labels on CriticLeanBench. Simple grade-margin ranking performs similarly to or better than DS in disagreement yield across several review budgets.
Critical strengths
1. Careful analysis of disagreement beyond simple evaluator accuracy
A major strength of this paper is that it does not simply treat disagreements between LLM judges and benchmark labels as LLM evaluation errors. Instead, it investigates which side of the disagreement is actually incorrect through source comparison and targeted Lean checks.
For example, the paper presents a case in which a published faithful label is incorrect because of a missing domain restriction, another in which two of three human reviewers overlook a strengthened hypothesis requiring absolute summability even though the source only requires ordinary convergence, and a case in which an LLM raises an incorrect objection because it misparses the scope of a Lean big operator. These examples concretely demonstrate that none of the benchmark label, human reviewer, or LLM judge can automatically be treated as ground truth.
2. Practical framing of LLM judges as review-prioritization signals
Another strength is that the paper goes beyond using LLM judges as final oracles and instead uses their outputs to prioritize which formalizations should be inspected under a limited human-review budget.
In particular, this paper shows that LLM-derived signals can concentrate reviewer disagreement without using human ratings during ranking. For example, in Panel A, at a 20% review budget, 77.2% of the items selected by grade-margin ranking contain reviewer disagreement, compared with 43.9% under random selection.
Critical weaknesses
1. Limited independent evidence for the paper’s central audit result
One of the paper’s central findings is that, among the 51 CriticLeanBench cases in which all three LLM judges disagree with the published label, the author audit supports the LLM judges in 44 cases. However, these 44 cases are not all validated with the same strength of evidence. The results in Section 4.2 indicate that a substantial fraction of the judge-supported cases are in the direction where the published label is unfaithful, while all three judges and the author audit classify the formalization as faithful. Based on the reported results, at least 28 of the 44 judge-supported cases fall into this category. Unlike an unfaithfulness judgment, which can sometimes be established by identifying a single counterexample or omitted assumption, such cases require establishing that no important semantic mismatch is present. Section 4.2 itself distinguishes these acceptance decisions as “judgments, not proofs.” Therefore, a substantial portion of the headline 44/51 result relies on positive faithfulness judgments rather than direct evidence such as Lean-verified counterexamples or formal implication checks.
Moreover, although the audit was conducted without access to the published labels or judge scores, it remains an author-conducted, AI-assisted audit rather than an independent validation. Consequently, particularly for published-unfaithful / judges-accept cases, it is difficult to distinguish whether the audit has identified a genuine benchmark-label error or simply produced another judgment based on similar interpretations or grading conventions.
2. Review-prioritization results rely on noisy proxy outcomes and exploratory analyses
The effectiveness of the LLM-only ranking is evaluated primarily by how often it selects cases involving human-reviewer disagreement or human-majority acceptance, rather than by how effectively it identifies genuinely incorrect formalizations or incorrect annotations. However, a central premise of the paper is precisely that human judgments can themselves be erroneous. Therefore, successfully prioritizing cases with high reviewer disagreement does not necessarily imply successful identification of semantic faithfulness errors.
Indeed, agreement among the human reviewers is itself relatively low. In Panel A, pairwise Cohen’s (\kappa) ranges from 0.15 to 0.27, and in Panel B it ranges from 0.31 to 0.41. The same reviewer’s acceptance behavior also varies substantially across panels. For example, R1 has the highest acceptance rate among the human reviewers in Panel A but the lowest acceptance rate in Panel B. These results suggest that reviewer disagreement may reflect reviewer-specific interpretations or grading criteria in addition to genuine semantic defects.
Thus, the current experiments show that LLM-based ranking can concentrate cases in which evaluators are likely to disagree, but they do not establish whether using such rankings leads to the discovery of more incorrect annotations or improves final evaluation correctness. The ranking outcomes evaluated in the paper are reviewer disagreement or majority acceptance rather than verified faithfulness, and the paper does not measure whether subsequent review actually identifies and corrects errors in the selected items.
Questions for authors
1. Independent validation of the audit result
Among the 44 cases in which the audit supports the judges out of the 51 unanimous LLM–label disagreements, how many are supported by concrete evidence such as a Lean-verified counterexample, a formal implication check, or direct source comparison? In particular, for cases where the published label is unfaithful but the judges and audit classify the formalization as faithful, positive faithfulness is inherently more difficult to verify directly. Was any separate independent expert re-evaluation conducted for these cases? I believe this distinction is important for assessing how strongly the 44/51 result should be interpreted as evidence of genuine benchmark-label errors.
2. Does review prioritization improve final evaluation correctness?
When items selected by the LLM-only ranking are subjected to additional expert review, does this lead to a higher rate of identifying and correcting incorrect annotations or semantic defects compared with random selection or other baselines? Based on the current results using human disagreement or human-majority acceptance as outcomes, it is difficult to distinguish disagreement detection from actual error detection.
2 comments
Thank you for the detailed response and revisions. While the additional analyses address some of my concerns, the lack of independent validation remains a limitation. I therefore maintain my original rating.
Response to Reviewer e3f1
Thank you for the accurate and detailed review. Both weaknesses are correct, including your estimate: 30 of the 44 judge-supported cases are positive faithfulness judgments. Changes are marked in blue in the revised PDF; the revision also contains minor wording edits that change no result.
Added to the revised paper
W1 / Q1. Evidence behind the 44; independent re-evaluation.
ℝ × ℝwhere the source means Euclidean distance; a hypothesisα ≠ 2nπthat reuses the number of summands; numerals that all elaborate as natural numbers, so the statement reads0 = 0.Headline result. Of the 130 audited items, the audit agrees with the label on 45 and finds a checkable defect in 38: 22 Lean-checked, and 16 with a flawed source statement. Another 42 disagreements with the label are judgments without formal evidence, and 5 items are unresolved.
W2 / Q2. Does prioritization find errors, or only disagreement?
For the human panels, only disagreement; Sec. 5 now says so. For the labeled benchmark we ran the experiment you describe (Sec. 4.4). We audited, blind to labels and judge outputs, every unaudited item in the top 10% (43 items) of each ranking, plus random controls: 55 items, with the analysis fixed before review. Each row shows what the audit found in one ranking's first 43 items.
How the audit was done, and its limits. The auditing author is a PhD student in applied mathematics (qualifying exams in applied mathematics and PDE) who reads basic Lean. An assistant model (Claude Sonnet 5.5) first explained each Lean statement without seeing the source, and the author recorded a verdict. The author then discussed source and Lean with the model and recorded a final verdict. The final verdict reversed the initial one on 14 of 55 items, 13 of them to unfaithful; 11 of these 13 are corroborated by a Lean check (6) or by agreement with the label and all three judges (5). No second expert was available, and review time was not measured. For these reasons the defects we claim rest on the Lean checks, not on the verdicts alone.