acceptodds
Under review as a conference paper at ICLR 2027

MLP-Bench: Towards End-to-End Formal Research Assistance in Machine Learning Theory

14 views

Abstract

Large language models (LLMs) have demonstrated strong mathematical reasoning and autonomous research abilities, yet whether they can assist the rigorous construction and verification of machine learning (ML) theory remains underexplored. Since ML theory rests on precise assumptions and derivations, we ground our study in Lean, which mechanically checks every derivation against its stated assumptions. Mirroring how theory is actually developed and refereed, we frame formal ML theory research as two complementary capabilities: *forward construction*, which assists theory derivation, and *backward verification*, which assists reviewing. We instantiate this framework in the **M**achine Learning **L**ean **P**roof **Bench**mark (**MLP-Bench**), a 360-problem benchmark spanning 16 topics. *Forward construction* is realized by two tasks, formalizing natural-language (NL) theorems into Lean statements and proving given Lean statements. *Backward verification* is realized by a third task, judging whether a proof is correct and, if not, localizing the first erroneous step with Lean-checked evidence. Evaluating 11 models, *forward construction* remains challenging. Auto-formalization reaches at most a **27.5%** bidirectional equivalence (BEq) pass rate against gold Lean statements, with omitted implicit assumptions a major failure mode, and specialized provers solve **0.0%** of theorem-proving problems. In *backward verification*, models localize errors more reliably by constructing Lean-checked refutation than by attempting to prove each step. To our knowledge, MLP-Bench is the first Lean benchmark for end-to-end LLM assistance in ML theory research.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Lean4, AI4Math, ML Theory, LLM Agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HP3COpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MLP-Bench is a Lean-based benchmark for assistance with machine-learning theory, covering statement autoformalization, proof generation for fixed formal statements, and error detection with first-error localization in natural-language proofs. It draws on research papers, textbooks, and graduate courses, with 120 theorem targets used for the two forward tasks and 120 proof-verification groups. The paper evaluates single-turn and agentic settings separately and analyzes missing assumptions, equivalence assessment, and proof-based versus refutation-based verification. The authors report gains from agentic settings and higher F1 for FL Refute than FL Prove in the reported closed-source model comparisons.

Critical strengths

The use of authentic academic sources and human semantic checking is a substantive strength. The dataset includes research papers and graduate-level material, and the reported construction process goes beyond Lean compilation and proof checking to include semantic inspection and repair of the NL–formal alignment. A provable statement that changes the source's meaning is not a useful evaluation target. This alignment effort therefore supports the dataset's credibility and provides useful material for studying formalization difficulties in ML theory.

Critical weaknesses

The paper does not clearly distinguish recoverable omissions from ambiguity or missing information in the input, raising a concern that formalization is being mixed with adding assumptions from domain priors. For example, if the context or an explicit convention establishes a : Nat, omitting the type when writing a = 1 can be reasonable. However, given only x² ≥ 0 without a domain or background, a model should not choose the real numbers solely because that interpretation makes the claim true. Formalization should preserve the intended meaning rather than assume truth and supply conditions that make a statement hold. This matters especially for conjectures or claims whose truth is unresolved. My concern is therefore whether the actual model-visible context includes the standing assumptions relevant to the target claim in the source paper or textbook. If it does, missing conditions may reflect a failure to recover supplied information. If it does not, and the model must infer from ML domain knowledge which conditions would make the theorem provable, the task adds reasoning and theorem repair beyond translation. The paper describes context collection and human checking, but does not adequately separate these cases. For example, the forward ch10_lemma_10_1 has an empty context, while its backward counterpart explicitly adds sampling and measurability setup; fully rendered inputs are needed to establish the evaluation-time situation. Until this is clarified, conditions absent relative to the reference should not all be interpreted as model failures to recover assumptions from the input.

The equivalence evaluator lacks sufficient calibration on rejected outputs to support the interpretation of low formalization scores. The BEq+ comparison and inspection of 20 accepted pairs do not estimate how often faithful outputs are rejected by restricted search or the gates. The supplied materials lack a human audit of rejected candidates by model and mode, as well as predictions, directional proofs, and gate outcomes for inspection. If certification recall differs across models or formulations, both absolute scores and relative comparisons may be affected. The current evidence therefore does not sufficiently separate semantic errors from failures to certify equivalence. A direct calibration would construct several human-verified equivalent reference formulations for the same target, varying only hypothesis order, statement structure, or elementary algebraic presentation, and test whether BEq certifies equivalence across them. Rewrites using simple properties such as commutativity could reveal whether restricted search or the gates miss equivalences that are obvious to a human. If such variants are rejected, even entirely faithful outputs could score below 100% because of their formulation. Even if relative model trends remain largely unchanged, absolute pass rates should not be interpreted directly as semantic accuracy without this calibration.

Questions for authors

The following are clarification requests about the evaluated tasks and existing evidence.

  1. Can you provide the fully rendered input for ch10_lemma_10_1 and explain how its sampling, independence, and measurability conditions are recoverable? More generally, how do you distinguish conditions supplied in the input, inherited from other source context, and introduced during formalization? Does proof generation receive a complete informal proof or partial proof hints? These input differences affect the task definition and difficulty.
  2. Have you manually audited a stratified sample of candidates rejected by BEq, distinguishing models, modes, and failed directions or gates? Can you provide representative candidates, directional proofs, and gate decisions to clarify whether low certification rates primarily reflect semantic errors or evaluator limitations?
  3. Can you provide the original and mutated statements for Table 7, their falsity-validation evidence, and the selection procedure for the 120 mutations? Since FL Prove and FL Refute can generate different step propositions, what existing checks beyond the limited diagnostics in D.2 support attributing their performance difference to proof versus refutation mechanisms? These details affect the scope of the conclusions.
  4. Can Table 1 distinguish the statement-autoformalization version of RLM25 from RLMEval, publicly available in October 2025 and already covering complete proof generation and proof autoformalization? With this work included, how should MLP's novelty be specified in terms of the ML domain, dataset construction, or particular three-task combination, rather than a broader first end-to-end claim?
4 comments
AuthorsOpenReview ↗

Response to Review from HP3C (1/2)

Dear Reviewer HP3C,

Thank you for your thoughtful review and constructive suggestions. We especially appreciate the time and care you took to examine the individual cases in our benchmark in such detail. Your comments have helped us strengthen the soundness of our paper. We respond to your concerns as follows:

W1 & Q1: Provenance of assumptions

The input of each task should indeed be fully specified. Our original supplementary material contained the main components of the benchmark, while some additional benchmark data and evaluation scripts were not included. The updated supplementary material now contains the benchmark and evaluation pipeline, including the formalization notes, the fully rendered prompt for ch10_lemma_10_1, and representative model predictions with their directional proofs and gate decisions.

  1. Input of auto-formalization. In all reported runs, the model receives the NL statement, the context, formalization notes, and the Lean file with the target replaced by a hole. The formalization notes record modeling choices the source leaves open and repairs of statements that are false as printed. In the initial version of the paper, the prompt in Appendix F.1 showed this input only as the placeholder {nl}, which stands for the NL statement, the context, and the formalization notes together. The revised Appendix F.1 expands this placeholder. All reported experiments already used this input, so the setup and all numbers are unchanged.

  2. Classification of conditions. We agree with the x² ≥ 0 example. Conditions in a reference statement fall into four kinds, from easiest to hardest to recover:

    1. Stated in the NL statement.
    2. Stated elsewhere in the source, such as standing assumptions or definitions, which we supply as the context.
    3. Field conventions the source leaves implicit, as in the reviewer's a : Nat example. The model must recover these from its own understanding of ML theory and Lean.
    4. Repairs: conditions the printed statement lacks but needs in order to be true, found while building and proving the reference. Since these depart from the literal text, we do not leave them for the model to guess; the notes describe each one and how the reference formalizes it.

    The first, second and fourth kinds are all in the model's input; only the third relies on field knowledge. The model is not asked to repair a theorem or to choose conditions that make it provable. What it must do is read a research-level statement as an expert in the field would, which requires a solid understanding of ML theory and Lean. We regard this as the core difficulty that a research-level benchmark should measure.

  3. Worked example of ch10_lemma_10_1. Appendix E.4 of the revised PDF shows the input for this lemma and where each sampling, independence and measurability condition comes from. Its context is empty because it is the first lemma of its chapter and no earlier definition or result precedes it. The sampling model is provided through the formalization notes instead.

  4. Input of proof generation. The model receives the NL statement and the full Lean statement, but no NL proof.

  5. Ablation on formalization notes. We cleared only the formalization notes and kept every other part of the input verbatim (Claude-Opus-5):

SubsetBEq with notesBEq without notes
Problems with notes (33%)14.5 ± 1.5%2.6 ± 0.0%

When we remove only the notes, BEq on the affected problems drops from 14.5% to 2.6%, which shows the repairs are recovered from the notes rather than guessed.

  1. Revisions. The revised PDF documents the notes and the exact input of each task (Sec. 2.2.1). The benchmark in the updated supplementary material now includes the formalization notes (field nl_notes of Benchmarks/auto-formalization.jsonl), and the fully rendered prompt for ch10_lemma_10_1 is in examples/ch10_lemma_10_1_rendered_prompt.txt.

W2 & Q2: BEq calibration

We followed the reviewer's suggested design and built controlled equivalent rewrites, all of which BEq certified (Common Response 2, part 1). On real model outputs in agent mode, BEq recall is 82.5% for Claude-Opus-5 and 83.3% for GPT-5.6-Sol (Common Response 2, part 2), so low certification rates mainly reflect semantic errors; absolute scores are a conservative lower bound and relative comparisons are unaffected. Of the 13 false negatives, 6 were rejected by the bridge gate, 2 by the tactic whitelist, and for 5 the evaluator found no proof. Representative candidates, directional proofs, and gate decisions are in examples/beq_cases/ of the updated supplementary material, and BEq is implemented in src/mlpbench/autoform/beq.py.

AuthorsOpenReview ↗

Response to Review from HP3C (2/2)

Q3: Table 7 data and attribution

  1. Data. The updated supplementary material provides the original and perturbed statements for Table 7 (examples/table7_perturbations/perturbations.jsonl), each with a gold Lean refutation as falsity evidence (examples/table7_perturbations/gold_refutations/). GPT-5.6-Sol and Claude-Opus-5 each proposed three perturbations per statement; we kept one that compiles and has a gold refutation, balancing six types (strengthened conclusion 20%, changed constant 20%, flipped inequality 19%, dropped hypothesis 18%, changed quantifier 13%, weakened hypothesis 10%). Gold refutations were written with a human in the loop and checked by Lean, so admission does not depend on whether any evaluated model can refute the statement. Models are not told what changed. Appendix C.2 of the revised PDF gives details.

  2. Attribution. Apart from sampling, which the Peer Conflict analysis in Appendix D.2 measures, FL Prove and FL Refute differ only in the decision rule. They share the same formalization prompt, FL Prove only omits the REFUTATION block, and FL Refute refutes the model's own STATEMENT verbatim. Figure 4 shows the effect of the decision rule: FL Prove flags valid steps it fails to prove, so its flags land before the gold step.

Q4: RLMEval and novelty

We thank the reviewer for pointing this out. We agree that RLMEval should be listed separately and that our novelty claim should be stated more precisely. The revised PDF now cites RLMEval [1] and discusses it in Related Work as the first benchmark for research-level theorem proving and proof auto-formalization. Table 1 also lists RLMEval separately from RLM25: RLM25 targets statement auto-formalization, RLMEval theorem proving and proof auto-formalization, and neither covers proof verification or ML theory. We have also replaced the "first end-to-end" claim with a narrower one: the first Lean benchmark combining the three tasks for ML theory under a unified Lean framework.

We are truly grateful for your detailed feedback, which has greatly helped improve the clarity and rigor of our work. We hope the revised paper and the updated supplementary material resolve the concerns you raised. If they do, we would be grateful if you would reconsider your score. We welcome any further questions.

[1] Poiroux, A., Bosselut, A., and Kunčak, V. RLMEval: Evaluating Research-Level Neural Theorem Proving. Findings of EMNLP 2025.

Thank you for the detailed response and the additional experiments. I am raising my rating from 2 to 3. The suggestions below are for clarifying the results and improving future evaluations.

I checked the original MIT lecture notes for Lemma 10.1. The lemma leaves the sampling setup implicit, and its proof uses independence and a common sampling distribution. The rendered prompt clarifies how these conditions were supplied to the model and resolves my concern about this example. One suggestion is to briefly describe how the context and formalization notes were collected and checked for completeness. It would also help to break down the omitted assumptions by whether they were explicitly supplied to the model, appeared elsewhere in the source but were absent from the input, or were left to be inferred from domain conventions.

The audit of actual agent outputs substantially addresses my concern about BEq calibration. For the 90 manually constructed pairs, the changes are mostly local, and whether BEq can handle many of them can be anticipated from its implementation and allowed tactics. My suggestion for future calibration is to construct independently verified equivalent statements whose types, extent, and combinations of changes reflect actual agent outputs. Applying the same BEq protocol to these statements would estimate a practical score ceiling for that output distribution. If an agent were semantically correct on every problem while expressing its answers in forms typical of real agents, what fraction would BEq certify? Comparing each model’s observed score with this estimated ceiling would help readers interpret how far it falls short of that benchmark and how much headroom the evaluator itself leaves below 100%.

I also appreciate the release of the Table 7 perturbations and their Lean refutations. I accept the observed advantage of FL Refute in the full error-localization pipeline. Since FL Prove and FL Refute can produce different Lean statements for the same source step, the current comparison leaves open how much of the gain comes from the refutation rule itself. As a suggestion for future work, applying both rules to the same fixed set of Lean step formalizations under comparable search budgets would help isolate that contribution.

AuthorsOpenReview ↗

A Follow-up Response to Reviewer HP3C

Dear Reviewer HP3C,

We are glad that our response and the additional experiments addressed your concerns, and we sincerely appreciate the update to your score. Thank you as well for the further suggestions. We respond to each of them below.

1. Collection of context and formalization notes; breakdown of omitted assumptions

Collection and completeness. For each problem, an LLM extracted the context from the source. We deliberately over-extracted to reduce omissions, and then manually removed irrelevant material (Sec. 2.1.1, AI Use Statement). The formalization notes were written by the authors, who reviewed every assumption of each reference statement one by one. An assumption goes into the notes if it repairs a statement that is false as printed, or if it admits several reasonable formalizations.

Breakdown. Given the time available, we cannot provide the full statistics across all models. Instead, the revised PDF adds two worked cases in Appendix E.6: one where the omitted assumption is stated in the supplied context, and one where it is left to field convention. Among all omitted assumptions we traced in our audit sample, we found none that appears in the source but is absent from the input.

2. A practical ceiling for BEq

We agree that the controlled rewrites are local. The audit of real agent outputs (Common Response 2, part 2) gives a first estimate of the ceiling you describe. Of the 76 model outputs that are semantically equivalent to the reference, BEq certifies 63, so an agent that is always correct and writes in the style of real agents would score about 83%.

Claude-Opus-5GPT-5.6-Sol
Observed BEq27.5%25.0%
Estimated ceiling82.5%83.3%
Observed / ceiling33.3%30.0%

The evaluator therefore leaves about 17 points of headroom below 100%, and most of the remaining gap comes from the models.

3. Isolating the decision rule

We ran this experiment. We fixed the Lean step statements from each model's original run and gave them to the other model. FL Prove and FL Refute were run under the same budget on 40 problems (20 correct, 20 flawed).

Statements byProverFL Prove F1FL Refute F1
Claude-Opus-5Claude-Opus-545.863.6
Claude-Opus-5GPT-5.6-Sol41.563.6
GPT-5.6-SolClaude-Opus-528.137.0
GPT-5.6-SolGPT-5.6-Sol13.638.5

With the statements fixed, FL Refute still outperforms FL Prove in every setting. Under FL Refute, accuracy is determined mainly by the quality of the formalization: on the same statements, the two provers score almost identically. Under FL Prove, formalization quality matters just as much, but proving ability also plays a role, since on the same statements, Claude-Opus-5 scores higher than GPT-5.6-Sol as the prover.

We are truly grateful for your detailed feedback, which has greatly helped improve the clarity and rigor of our work.

Best Regards,

Submission #19219 Author Team

Reviewer_XHhqOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MLP-Bench, a benchmark designed to evaluate the ability of large language models (LLMs) to assist theoretical machine learning research through Lean-based formal verification. The benchmark consists of 360 problems across 16 topics and covers three tasks: (1) auto-formalization of natural-language theorems into Lean statements, (2) construction of complete Lean proofs, and (3) verification of natural-language proofs by identifying their first substantive error.

The authors evaluate 11 models, comparing single-turn chain-of-thought generation with agentic approaches that iteratively use Lean feedback. Their experiments show that faithful auto-formalization remains challenging, with the best model achieving a bidirectional equivalence (BEq) rate of 27.5%. A major difficulty is recovering implicit assumptions, such as measurability and integrability, that are often omitted in informal mathematical statements. For theorem proving, agentic approaches substantially outperform single-turn generation, while specialized Lean provers fail to solve the benchmark problems. In proof verification, the authors find that identifying errors through Lean-checked refutations is generally more effective than treating proof failures as evidence of incorrectness.

Critical strengths

By incorporating research-level theorems and proofs from the machine learning literature, MLP-Bench addresses important challenges that arise in actual theoretical research, including implicit assumptions, lengthy proofs, and subtle errors in mathematical arguments. Although these challenges are already known, the integration of auto-formalization, theorem proving, and proof verification within a common framework is valuable.

The experiments reveal an interesting asymmetry between the ability to prove mathematical statements and the ability to refute false claims. Although this asymmetry has often been noted, there is value in supporting it with evidence from a systematic benchmark.

Critical weaknesses

(1) The benchmark focuses on formal correctness without addressing mathematical understanding or the explanatory value of proofs.

The paper aims to evaluate end-to-end assistance in theoretical machine learning research. However, its evaluation of theorem proving is almost entirely concerned with whether Lean accepts a generated proof.

In mathematical research, proofs serve purposes beyond establishing the correctness of a theorem. A good proof can explain why a theorem holds, reveal the mathematical structures underlying a result, clarify the role of assumptions, and provide insights that can be transferred to other problems. Moreover, proofs differ significantly in their readability, conceptual organization, and accessibility to human researchers.

These dimensions are not evaluated in MLP-Bench. For example, a long and difficult-to-understand Lean proof may be counted as fully successful, even if it provides little explanatory insight to a human mathematician. Conversely, a concise and conceptually illuminating informal proof may have substantial research value independently of its immediate formalization in Lean.

I do not regard this omission as invalidating the benchmark itself, since formal correctness is an important and well-defined evaluation target. Nevertheless, it limits the extent to which the results can support broader claims about end-to-end assistance in theoretical research.

(2) The evaluation of semantic faithfulness in auto-formalization remains challenging.

A difficulty concerns the relationship between an informal mathematical statement and its formal representation in Lean.

The paper identifies the recovery of implicit assumptions as a major bottleneck. However, it is important to distinguish between making genuinely implicit assumptions explicit and introducing additional assumptions that strengthen the premises and thereby weaken the original mathematical claim. The former may constitute faithful formalization, whereas the latter may change the intended meaning of the theorem.

Although the authors employ human validation and a Lean-based bidirectional equivalence metric, these methods do not entirely resolve the problem of determining whether a formal statement preserves the intended meaning of its natural-language counterpart. In particular, failure to certify equivalence may reflect limitations of the proof-search procedure rather than a genuine semantic difference. Conversely, formal verification of a generated statement does not by itself guarantee that the original statement has been represented faithfully.

This issue is especially important for the proof-verification task. A Lean-checked refutation establishes that a formalized proposition is false, but if the formalization does not faithfully represent the original proof step, the refutation may not establish an error in the original mathematical argument. The additional analysis in Appendix D.2 acknowledges this difficulty, but it remains a fundamental limitation of the proposed framework.

Questions for authors
  1. How do the authors distinguish between assumptions that are genuinely implicit in the original mathematical statement and additional assumptions introduced to make a formalized theorem provable? Are there explicit annotation criteria for determining whether an assumption is implicitly intended by the source text?

  2. The paper evaluates proof construction based on whether the generated proof is accepted by Lean. Do the authors regard mathematical comprehensibility and explanatory value as essential components of theoretical research assistance that fall outside the scope of the present benchmark?

1 comment
AuthorsOpenReview ↗

Response to Review from XHhq

Dear Reviewer XHhq,

Thank you for your thoughtful review. We especially appreciate your perspective on what makes a proof valuable beyond formal correctness, and your careful distinction between recovering implicit assumptions and introducing new ones. Both points have helped us state the scope of our benchmark more precisely. We respond to your concerns as follows:

W1 & Q2: Comprehensibility and explanatory value

We thank the reviewer for this insightful point, which captures what makes a proof valuable in research. We regard comprehensibility and explanatory value as essential to research assistance, but they lie outside the scope of MLP-Bench, which evaluates the layer that can be checked objectively. The revised PDF lists them under Limitations and Future Work (Appendix G). One way to make a Lean proof more interpretable is to view it as a dependency graph, with lemmas as nodes and their uses as edges, which exposes the structure of the argument independently of the length of individual tactic proofs. Appendix E.5 of the revised PDF shows several such graphs.

W2 & Q1: Implicit vs. added assumptions

This distinction is important, and our annotation treats recovering implicit assumptions and introducing new ones differently.

Annotation criteria. We built each reference statement in three steps:

  1. We formalized the conditions stated in the NL statement, then the implicit ones (standing assumptions, definitions and field conventions).
  2. While proving the Lean statement, we added any further condition the proof required.
  3. We reviewed every assumption. If it can be readily inferred from the statement, its context or standard conventions in ML theory, we leave it for the model to recover. If it is hard to infer, or it admits several reasonable formalizations, we record it in the formalization notes.

The formalization notes are a part of the model's input that records modeling choices the source leaves open and repairs of statements that are false as printed. All reported experiments already used them, and the revised PDF documents them in Sec. 2.2.1.

Repairs that strengthen premises. Some conditions are needed because the printed statement is false as written. Since these depart from the literal claim, we do not expect the model to guess them. The formalization notes state each repair and how the reference formalizes it, so the model and the reference target the same, explicitly specified theorem.

Ablation on formalization notes. We cleared only the formalization notes and kept every other part of the input verbatim (Claude-Opus-5):

SubsetBEq with notesBEq without notes
Problems with notes (33%)14.5 ± 1.5%2.6 ± 0.0%

When we remove only the notes, BEq on the affected problems drops from 14.5% to 2.6%, which shows the repairs are recovered from the notes rather than guessed.

BEq calibration. The reviewer raise two failure modes. For BEq failing to certify statements that are in fact equivalent, we measured its recall on real model outputs, which is about 83%, so BEq scores are a conservative lower bound (see Common Response 2). For models adding assumptions of their own, we agree that compiling or type-checking a generated statement does not guarantee faithfulness. This is why BEq does not check the generated statement on its own. Instead, it requires Lean-verified proofs in both directions between the generated statement and the reference statement. Three safeguards make certified pairs trustworthy:

  • Faithful references. Every reference statement passed manual semantic-faithfulness checks against the original natural-language problem (Sec. 2.1.2).
  • Gates against spurious proofs. Three gates reject proofs that do not genuinely derive one statement from the other (Appendix B.3).
  • Manual audit. We inspected 20 randomly sampled certified pairs, covering 40 directional proofs. Every proof was a genuine derivation of one statement from the other (Appendix B.2).

Faithfulness of refutations in proof verification. As you point out, Lean certifies only that the formalized step is false. Our audit finds that 28% of step formalizations are unfaithful, while refutations of annotated errors are mostly on target (80%). Please see Common Response 1 for the full audit.

We are sincerely grateful for your feedback, which has greatly helped improve the completeness of our work. We hope these clarifications address your concerns. If so, we would greatly appreciate it if you would consider raising your score. We are glad to discuss any remaining points.

Reviewer_xucrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MLP-Bench, a benchmark of 360 problems across 16 topics for evaluating LLM assistance in machine learning theory research. It covers three tasks: theorem autoformalization, formal theorem proving, and proof verification with first-error localization. The benchmark includes expert-validated Lean formalizations and proofs, as well as both real and synthetic flawed proofs for verification. Experiments on 11 models reveal substantial difficulties in recovering implicit assumptions and constructing complete proofs, while showing that Lean-checked refutation can be more effective than proof-failure-based error detection.

Critical strengths
  1. MLP-Bench covers autoformalization, theorem proving, and proof verification, including real flawed proofs collected from research submissions. This fills an important gap in existing formal reasoning benchmarks, which mainly focus on competition mathematics or isolated proving tasks.

  2. The experiments identify implicit-assumption omission as a major autoformalization bottleneck and demonstrate the benefits of Lean-checked refutation for error localization. The additional analyses of specialized provers, proof length, and verification failures provide useful directions for future research.

Critical weaknesses
  1. The end-to-end evaluation is incomplete. The three tasks are evaluated independently using gold intermediate inputs. The benchmark does not measure how errors propagate through an actual end-to-end research pipeline.

  2. Semantic faithfulness remains a concern in proof verification. Lean verifies the model-generated formalization rather than the original natural-language proof step. Appendix D.2 identifies cases where models prove flawed steps or produce conflicting formalizations, raising concerns about whether successful refutations always correspond to genuine errors in the source proof.

  3. The advantage of formal refutation is not universal. Although FL Refute substantially outperforms FL Prove, Appendix D.3 shows that it performs worse than the natural-language baseline for all three GPT models. Moreover, FL Prove uses proof failure as a rejection signal, making it a relatively weak baseline for comparison.

  4. The limitations of refutation-based verification need further analysis. A false intermediate claim can be refuted with a counterexample, but an invalid inference may still reach a true conclusion. It remains unclear how well FL Refute handles such errors, which are also important in reviewing mathematical proofs.

Questions for authors
  1. Can the authors clarify whether MLP-Bench supports evaluating a complete pipeline that connects autoformalization, theorem proving, and proof verification? How would error propagation affect the reported performance?

  2. Have the authors manually checked the semantic faithfulness of a representative sample of Lean-refuted steps? How often does a successful refutation actually target the intended natural-language claim rather than an incorrect formalization?

  3. Can the authors provide a breakdown of FL Refute's performance across different error types, particularly invalid inferences that do not produce false conclusions? This would help explain its limitations and its weaker performance than natural-language verification for GPT models.

1 comment
AuthorsOpenReview ↗

Response to Review from xucr

Dear Reviewer xucr,

Thank you for your careful review and constructive suggestions. We especially appreciate your close reading of our proof verification task and your thoughtful questions on how the benchmark should be evaluated and interpreted. They prompted several of the new analyses in this response. We respond to your concerns as follows:

W1 & Q1: End-to-end evaluation

We agree that error propagation across tasks should be measured directly. Proof verification is already a chained task, because for each step the model formalizes the NL claim itself and then proves or refutes it. To measure propagation across tasks as well, we chained auto-formalization and theorem proving. In agent mode on the same 120 problems, a problem counts as solved only if the model's formalization passes BEq and the model proves the reference statement, which is equivalent to proving its own formalization:

ModelClaude-Opus-5Claude-Sonnet-5GPT-5.6-SolGPT-5.6-TerraGPT-5.6-Luna
End-to-end20.8%9.2%11.7%1.7%0.8%

End-to-end success falls below success on either task alone, so per-task scores of auto-formalization and theorem-proving are upper bounds on end-to-end performance. The revised PDF reports this end-to-end setting in Appendix C.3.

W2 & Q2: Faithfulness of refutations

We acknowledge that a Lean refutation certifies only that the formalized step is false. In our audit, when a refuted step is annotated as incorrect, the refutation targets the genuine error in 80% of cases, whereas refutations of steps without errors all come from unfaithful formalization. Since each prediction is scored against the human-annotated error, these false alarms are counted as mistakes. Please see Common Response 1 for the full audit.

W3: FL vs. NL verification

We would like to clarify the purpose of this comparison. MLP-Bench is a benchmark that evaluates how well models reason in Lean. It does not propose FL Refute as a new verification method, and we do not claim that FL verification outperforms NL verification. FL Prove and FL Refute are two Lean-based settings of the same task. Comparing them shows that proof failure is a noisy rejection signal, since a failed proof cannot distinguish a false step from one that is merely hard to prove, which is what motivates refutation. The NL baseline serves only as a reference point (Appendix D.3).

That FL Refute trails the NL baseline for the GPT models is therefore a finding of our analysis, not a weakness of the benchmark. Our audit traces this gap to formalization quality. The refutation step itself is not the cause. GPT-5.6-Sol more often drops a condition and produces a stronger, refutable statement, so verification stops at that step before it reaches the actual error. When formalization is reliable, FL does not lose recall. Claude-Opus-5 matches or exceeds NL in every error category (see W4), and every error it reports comes with a machine-checked counterexample.

W4 & Q3: Per-category breakdown and invalid inferences

Per-category breakdown. We report recall (%) by error category on the 60 flawed problems:

ModelMethodFalse statement or computation (26)Invalid inference (21)Unjustified application (13)
Claude-Opus-5FL855231
NL77388
Claude-Sonnet-5FL503815
NL501415
GPT-5.6-SolFL624338
NL735231
GPT-5.6-TerraFL382931
NL624838
GPT-5.6-LunaFL231923
NL694315

For both Claude models, FL matches or exceeds NL in every category. For the GPT models, NL is higher in most categories, which our audit attributes to formalization quality (see W3).

Invalid inferences (Q3). Refutation does not require the conclusion of a step to be false. Each step is formalized as a standalone proposition that carries the full content of the step as written, so a step that infers b from a is formalized as a → b. If the inference is invalid, this implication is false and has a counterexample, even when b itself holds in context. For example, in ch2_lemma_2_8 the proof asserts |D| ≤ b after establishing only D ≤ b; the step is formalized as D ≤ b → |D| ≤ b and refuted with D = −2. The limitation of FL Refute on such errors comes from unfaithful formalization, which we analyze in Common Response 1.

We hope these responses and the new results address your concerns, and we would be grateful if you would consider raising your score. We are happy to answer any further questions.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Common Response 1: Faithfulness of Lean refutations

We thank Reviewers xucr and XHhq for this insightful point. We agree that in the proof verification task, Lean certifies only that the formalized step is false. We would also like to clarify the role of Lean in proof verification task: it is the instrument the model uses, not the judge of the benchmark. Each prediction is scored against the human-annotated error, so an unfaithful formalization is a model error that lowers the score whenever it leads to a wrong verdict, and the audit below measures how often this happens.

To measure this gap, we ran an audit. We first distilled a detailed rubric from our own earlier manual audits, with explicit criteria for each faithfulness level. Two strong LLM agents (Fable 5.2 and GPT-6-Astra) then independently assessed each sample under this rubric, and humans adjudicated every disagreement.

(1) Formalization faithfulness. We first measure how closely step formalizations follow the original proof. Each step formalization is labeled faithful (it states the same claim as the original step), simplified (it formalizes a simplified version of the step, omitting details irrelevant to the claim), or unfaithful (it states a different claim).

ModelnFaithfulSimplifiedUnfaithful
Claude-Opus-56028%48%23%
GPT-5.6-Sol6033%33%33%
Total12031%41%28%

Most step formalizations are faithful or simplified, but 28% are unfaithful. We agree that this gap is real and should be reported.

(2) Targets of refutations (xucr Q2). We agree that a Lean refutation alone does not show that the original step is wrong, so we checked refuted steps against the source.

Refuted stepsnGenuine source errorUnfaithful formalization
With errors2080%20%
Without errors200%100%

When a refuted step is annotated as incorrect, the refutation usually targets the real error (80%). When a step without errors is refuted, the cause is always an unfaithful formalization.

(3) Causes of missed errors. For each model, we randomly sampled 5 trajectories in which the annotated error was missed and identified the cause.

CauseClaude-Opus-5GPT-5.6-Sol
Error removed by formalization; step proved5/53/5
Error removed by formalization; step undecided0/51/5
Error preserved; no refutation found0/51/5

In nearly all missed cases, the formalization had already corrected the error, so the step that Lean checked was no longer the erroneous one.

(4) Revisions. The revised PDF reports this study in Appendix D.2, discusses this gap in Appendix G, and phrases our claims as "refutation of the formalized step."

We thank the reviewers again for raising this point, which has made our evaluation of proof verification more rigorous.

AuthorsOpenReview ↗

Common Response 2: BEq Calibration

We thank Reviewers HP3C and XHhq for the insightful suggestions. We agree that both the recall and the precision of BEq need to be measured. Below we address them separately.

(1) Controlled rewrites. Following Reviewer HP3C's suggested design, we built equivalent reformulations of the same targets. Each one changes only hypothesis order, propositional structure, or elementary algebraic presentation.

Rewrite typenCertified by BEq
Reordering hypotheses30100%
Propositional structure30100%
Elementary algebra30100%

BEq certified every rewrite, so it does not reject a statement for surface differences of these kinds.

(2) Recall on real model outputs. For every candidate that BEq rejected, an agent (Claude-Fable-5.2 / GPT-6-Astra) first tried to prove the two statements non-equivalent. Humans then audited every case the agent could not settle either way.

Metric (agent mode, n = 120 per model)Claude-Opus-5GPT-5.6-Sol
Certified by BEq27.5%25.0%
Rejected but equivalent (false negatives)5.8%5.0%
BEq recall82.5%83.3%

Of the 13 false negatives, 6 were rejected by the bridge gate, 2 by the tactic whitelist, and for 5 the evaluator found no proof. By direction, 7 failed only from candidate to reference, 4 only from reference to candidate, and 2 in both. Appendix B.4 lists all 13 cases.

BEq misses about 17% of equivalent outputs, so absolute scores are a conservative lower bound. Recall is nearly identical across models, so relative comparisons are unaffected. Representative candidates, directional proofs, and gate decisions are in the updated supplementary material.

(3) Precision on certified outputs. We manually inspected 20 randomly sampled pairs certified by BEq, covering 40 directional proofs. Every proof was a genuine derivation of one statement from the other (Appendix B.2).

(4) Revisions. The revised PDF reports both experiments and lists all 13 false negatives in Appendix B.4. Representative candidates, directional proofs, and gate decisions are in the updated supplementary material.

We thank the reviewers again for these suggestions, which have helped us calibrate BEq more carefully.

AuthorsOpenReview ↗

Summary of Reviewer Feedback and Author Response

Summary of Reviewer Feedback and Author Response

We thank the three reviewers for the time and care they put into our paper. Their comments led to several new experiments and to a clearer statement of what MLP-Bench measures. This note summarizes the strengths they noted, our responses to the main concerns, and the updates to the PDF and the supplementary material.

Strengths. The reviewers found value in bringing auto-formalization, theorem proving, and proof verification into one framework built on research-level ML theory, including real flawed proofs collected from research submissions. They pointed to the authentic academic sources and the human semantic checking behind the dataset. They also valued our empirical findings, in particular that our experiments identify omitted implicit assumptions as a major bottleneck in auto-formalization and provide systematic evidence for the asymmetry between proving statements and refuting false claims.

Main concerns and our responses.

  1. Faithfulness of Lean refutations (xucr, XHhq). We clarify that Lean is the instrument the model uses and that each prediction is scored against the human-annotated error. We also add an audit of formalization faithfulness, refutation targets, and causes of missed errors. Please see Common Response 1.
  2. Calibration of BEq (HP3C, XHhq). We add two experiments. BEq certifies all controlled equivalent rewrites, and its recall on real model outputs is about 83%, so BEq scores are a conservative lower bound. Please see Common Response 2.
  3. End-to-end evaluation (xucr). We chain auto-formalization and theorem proving and report end-to-end success.
  4. Provenance of assumptions (HP3C, XHhq). We document the formalization notes and the exact input of each task, give a worked example, and add an ablation on the notes.
  5. Scope and novelty (XHhq, HP3C). We add a discussion of limitations and future work, cite and discuss RLMEval, and state our novelty claim more precisely.

Revised PDF. The following parts are new or revised, and all changes are marked in blue:

  • Sec. 2.2.1: Formalization notes and the exact input of each task
  • Table 1 and Related Work: RLMEval
  • Appendix B.4: Calibration of BEq
  • Appendix C.2: Construction of perturbed statements
  • Appendix C.3: End-to-end evaluation
  • Appendix D.2: Audit of refutation faithfulness
  • Appendix E.4: Worked example of ch10_lemma_10_1
  • Appendix E.5: Dependency graphs of Lean proofs
  • Appendix F.1: Expanded auto-formalization prompt
  • Appendix G: Limitations and future work

Updated supplementary material. It contains the benchmark with formalization notes, the evaluation pipeline, the fully rendered prompt for ch10_lemma_10_1, representative model predictions with their directional proofs and gate decisions, and the original and perturbed statements for Table 7 with their gold Lean refutations.

We hope this overview makes our responses easier to navigate for the reviewers and the area chair, and we are happy to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.