acceptodds
Withdrawn from ICLR 2027

MathAdv: beyond proof accuracy in advanced mathematical reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A verified proof establishes correctness for a formal statement, but aggregate proof accuracy offers limited insight into the capabilities behind model success or failure. We introduce Mathadv to probe these capabilities across advanced mathematics and test whether proof success persists under equivalent reformulations. The benchmark contains 321 undergraduate- and graduate-level problems across 13 domains, including 298 with Lean 4 statements. Alongside theorem proving, suitable problems have up to three auxiliary tasks: multiple-choice questions probing mathematical knowledge, direct-answer questions assessing informal reasoning without Lean proof construction, and expert-crafted reformulations testing robustness to problem presentation. Our evaluations reveal limitations that proof accuracy alone obscures: models can identify promising mathematical approaches yet fail to construct valid Lean proofs, and their performance varies substantially across subjects and equivalent formulations. Natural-language guidance further exposes model-dependent differences, helping general-purpose LLMs but sometimes hindering proof-specialized models. Together, these findings motivate evaluating not only whether models produce valid proofs, but also the mathematical capabilities underlying their performance and the conditions under which they succeed. The dataset and evaluation scripts are available at https://github.com/mathadv912-design/mathadv-iclr.git.

Primary area:
datasets and benchmarks
Keywords:
Theorem proving, Mathematical reasoning, Diagnostic benchmarks, LLMs, Robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_k6s3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper looks maths proofs in Lean but constructs a dataset, MathAdv, with 4 variantes: standard formalised statement to be proved, multiple choice question based on known results needed for the proof, informal reasoning step, and an alternate question formulation. The methodology to generate the 321 question dataset (298 formalised statement) across 13 domains is clearly explained with multiple human verification to ensure the quality of the output. Various models which include specific provers and general-purpose models are evaluated on all of these variations. The paper shows that models often perform better at the direct-answer and multiple choice questions than the Lean proofs. They also show that transformed problems perform worse than the original problems.

Critical strengths
  1. The dataset generated goes through careful evaluation both automatic and human evaluation to ensure that the final questions are of high-quality.
  2. MathAdv is tested on 18 different models which tests a lot of different types of models (proof-step generation models, whole-proof generation models and general-purpose models).
  3. Good examples are shown to illustrate the dataset.
Critical weaknesses
  1. The dataset does not have parallel comparison. For example, 81 direct-answers, 30 transformed problems, 293 Multiple-choice question. This means that the results presented in section 4.3 are not comparable and therefore the analyse is not valid. This raises questions about the soundness of the work.
  2. The dataset presented test different capabilities of the models but some in formal language against natural language. This is firstly unfair as some models are specialised model in Lean proving as mentioned in Line 357/358 "Goedel-Prover remains an exception, consistent with its stronger specialization toward Lean proof search than general answer selection". Moreover, I feel like some of these have already been tested in fairer settings. For example, direct answers is tested by PutnamBench https://proceedings.neurips.cc/paper_files/paper/2024/file/1582eaf9e0cf349e1e5a6ee453100aa1-Paper-Datasets_and_Benchmarks_Track.pdf), CombiBench (https://openreview.net/pdf?id=qeZStJVYnc), ConjectureBench (https://arxiv.org/pdf/2510.11986), but they do so in the formal language setting as opposed to testing it on one hand in natural language and on the other hand in formal language.
  3. Some of the models that are presented in the Table 2 and 3, despite performing the best, are not mentioned in the analysis. I think they have been forgotten because line 322/323 "Goedel-Prover-V2 achieves the best performance at 21.88%, followed by DeepSeek-Prover-V1.5RL + RMaxTS" directly contradicts the results shown in Table 1.
  4. Section 5.1 and 5.2 are entirely written with no statistics or results to show that what is claimed is observed in the data.
Questions for authors

Suggestions and questions:

  • Check the referencing you use in your latex version, \citep{, \citet{, \cite{
Reviewer_QctqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MathAdv is a Lean 4 theorem-proving benchmark with 321 undergraduate- and graduate-level problems across 13 domains (298 formalized in Lean 4 via an LLM-drafted, compiler-checked pipeline with human review). To separate informal reasoning from formal proof writing, problems are paired with multiple-choice questions on relevant theorems or strategies (293 items), natural-language direct-answer tasks (81 items), and expert reformulations (30 items). Evaluating dedicated provers, general LLMs, and agents, the paper finds that overall Lean proof accuracy stays between 1% and 26% (and 0% on topology), even though models score much higher on the multiple-choice and direct-answer variants. Moreover, models frequently fail on reformulated problems, natural-language hints help general LLMs but hurt some specialized provers, and verifier feedback or tool use mainly fixes local errors rather than multi-step reasoning.

Critical strengths
  1. The benchmark pairs Lean 4 statements across 13 advanced domains with multiple-choice, direct-answer, and reformulated tasks, moving beyond the competition-heavy single proof scores of miniF2F and PutnamBench to isolate knowledge gaps, informal reasoning errors, and formalization failures.

  2. The data, prompts, and evaluation code are fully released with item-level problem variants, making the benchmark auditable and the issues raised below fixable in principle.

Critical weaknesses
  1. A third of the sampled Lean statements are unfaithful to the problems, despite the claimed two-expert semantic review (Section 3.2). In a random sample of 30 of the 298 statements in data_4_9/full_problems.jsonl, 10 (33%; 95% Wilson CI 19--51%) are false, vacuous, or unprovable as stated: id 34 uses Mathlib's sup metric on R×R\mathbb{R} \times \mathbb{R}, putting the closest points at x=±(5−1)/2x = \pm(\sqrt{5}-1)/2 rather than ±1/2\pm 1/\sqrt{2}; id 39 equates a Bochner integral over R\mathbb{R} to a function of a free variable xx and arbitrary CC; id 40 assumes ∀x∈Icc 0 π,∑′n,ansin⁡(nx)=1\forall x \in \mathrm{Icc}\ 0\ \pi, \sum' n, a_n \sin(nx) = 1, which gives 0=10=1 at x=0x=0; id 91 states the damped wave equation for arbitrary y,a,by, a, b (false for y=t2y=t^2); id 165 uses opaque predicates; id 179 puts an InnerProductSpace on Fin n→R\mathrm{Fin}\ n \to \mathbb{R}, which has the sup norm and admits no such instance for n≥2n \ge 2; id 204 omits effectiveness in enumeratesAllPartrec, making total indices decidable by parity; id 212 defines IsRingTheoryEncoding with only True, refuted by setting ProvableRing := fun _ => True and χ:=fun ⇒true\chi := \mathrm{fun}\ _ \Rightarrow \mathrm{true}; id 245 never states that BB is a Brownian motion (refuted by B≡0B \equiv 0); and id 265 omits colour/property guards and landing randomization, leaving the hypotheses inconsistent. Four more sampled items give away the solution formula, similarity ansatz, or area integral in the hypotheses (92, 93, 154) or weaken "the best possible bound is 60" to ≤60\le 60 (243). Outside the sample, four items have goal True or ∃…,True\exists \dots, \mathrm{True} (94, 121, 233, 277) and three rely on axiom/opaque declarations (167, 210, 213). Because false items cap accuracy while vacuous or trivial ones inflate it, the scores and rankings in Table 1, Fig. 4, and Tables 7--22 cannot be trusted without fixing the statements and rerunning all models.

  2. The robustness study (Section 4.4, Table 3) is built on broken problem pairs. Only 22 of the 30 reformulations have Lean statements (16 in probability), and 10 of these 22 pairs cannot test presentation sensitivity: the variant is false in four pairs (228 and 229 drop independence or uniformity; 237 evaluates to 33/6733/67, not 32/6532/65; 241 claims E0≤F0E_0 \le F_0 for all p,qp, q even though E0E_0 varies with pp), the original goal is True in one (233), the two versions prove different claims in three (238, 240, and 317, which reduces a group isomorphism with S6S_6 to card(Perm(Fin 6))=720\mathrm{card}(\mathrm{Perm}(\mathrm{Fin}\ 6)) = 720), and the two Lean statements are nearly identical in two (316, and 320, the paper's Example 2). Since a false variant can only yield O-pass/T-fail and a True original can never yield O-fail, these pairs mechanically produce the reported asymmetry; with counts of 0--8 per model and no paired test, Table 3 proves nothing about robustness.

  3. The capability diagnosis (Section 4.3) compares aggregate scores across mismatched item sets rather than cross-tabulating outcomes on the same problems (e.g., passing MC and DA but failing Lean). The 81 direct-answer (DA) items are concentrated in calculus (25) and probability (16), with none in functional analysis, real analysis, or logic, and ask only for a number rather than a proof, whereas Lean accuracy is computed over 320 items including the 22 variants. Thus, DA≫Lean\mathrm{DA} \gg \mathrm{Lean} does not isolate formalization as the bottleneck. Since prior two-stage and variant-based Lean benchmarks already show that informal solving outpaces formalization and that provers are brittle to rephrasing, the benchmark's added value depends on per-item diagnosis that this design fails to deliver.

  4. The model comparisons are confounded by format failures, unequal sampling budgets, and Mathlib version drift. On the MC probe, Goedel-Prover SFT and DPO score 21.2% and 19.5%, below the ≈24%\approx 24\% random-chance level implied by the option counts, so MC measures format-following rather than domain knowledge for specialized provers. Across tables, sampling budgets range from pass@1 for agents to pass@128 for provers, and the evaluation setup pins the verifier to the 2024 Mathlib checkout bundled with DeepSeek-Prover-V1.5, penalizing models trained on newer Mathlib releases via API drift.

Questions for authors

See weaknesses.

Reviewer_giHDOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces MathAdv, a benchmark of 321 undergraduate- and graduate-level problems across 13 domains, 298 of them with Lean 4 statements. Its goal is to look beyond aggregate proof accuracy. Suitable problems are paired with up to three auxiliary tasks: multiple-choice (MC) questions on the relevant theorem or strategy, direct-answer (DA) questions on informal problem solving, and expert-crafted equivalent reformulations. The Lean statements come from an LLM-assisted, compiler-in-the-loop pipeline with dual semantic verification and final expert review. Evaluating step-level provers, whole-proof provers, general-purpose LLMs, and agents, the authors report four main findings: 1) Models answer MC and DA questions far more often than they produce valid Lean proofs. 2) Performance is highly uneven across domains, with every system at 0% on topology. 3) Models solve original problems more often than their reformulations. 4) Natural-language hints help general-purpose LLMs but hurt proof-specialized models.

Critical strengths
  • Diagnostic framing beyond a single proof score. Pairing each problem with knowledge, informal-reasoning, and reformulation probes asks why a prover fails rather than only whether. The headline gap is high MC/DA accuracy alongside low Lean accuracy for general-purpose models. It makes a compelling case that formalization, not mathematical knowledge alone, is a central bottleneck, and the framing is a useful template for future formal benchmarks.
  • Coverage of advanced, underexplored domains. Existing formal benchmarks concentrate on competition-style algebra and number theory. MathAdv draws from standard graduate texts in topology, functional analysis, Fourier analysis, logic, and probability. The resulting domain profiles (strength in number theory and linear algebra, 0% on topology) expose training-distribution skews that aggregate scores hide.
  • Actionable analyses. The paper has provided several findings translate directly into design advice: 1) the gaps between between informal and formal math: informal guidance are not smoothly transformed into formal proof success. 2) verifier feedback fixes local syntax and elaboration errors but not flawed strategies. 3) Agents mostly succeed by finding a library lemma or a simpler formalization rather than building new multi-step arguments.
Critical weaknesses
  • Auxiliary tasks may not measure the intended capabilities. Picking the right theorem from five options is recognition, not the ability to “identify promising approaches” claimed in the abstract. Prover models are also not trained to emit option letters or boxed answers. The anomalies suggest format compliance is a confound: Goedel-Prover sits near chance on MC (19.5–21.2%), and DeepSeek-Prover-V1.5 Base (53.1%) outscores its SFT (32.5%) and RL (33.9%) variants. The same concern applies to the hint ablation, where Goedel-Prover’s drop to 1.56% may reflect an out-of-distribution prompt rather than interference with learned proof patterns.
  • Diagnostic conclusions rest on thin, skewed subsets. Only 30 problems have reformulations (16 in probability), and Table 3 reports raw counts of 0–8 without significance tests, so the robustness gap is not yet established. DA likewise covers only 81 problems, mostly calculus and none in logic, functional analysis, or real analysis. Capabilities are compared through aggregate accuracies on different subsets rather than per-item conditional outcomes, which is what separating knowledge, reasoning, and formalization requires.
  • Uncontrolled budgets and unverified statements limit comparisons. Budgets range from pass@128 to pass@16 to single-attempt agents, so cross-family comparisons are not budget-matched. Only sorry-terminated statements are released, so there is no evidence that every statement is provable as formalized. Domains where all systems score 0%, notably topology, therefore cannot be distinguished from formalization errors. The strongest provers the paper cites, such as DeepSeek-Prover-V2, are also not evaluated, and Aristotle appears in the budget appendix without results.
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.