- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MLP-Bench is a Lean-based benchmark for assistance with machine-learning theory, covering statement autoformalization, proof generation for fixed formal statements, and error detection with first-error localization in natural-language proofs. It draws on research papers, textbooks, and graduate courses, with 120 theorem targets used for the two forward tasks and 120 proof-verification groups. The paper evaluates single-turn and agentic settings separately and analyzes missing assumptions, equivalence assessment, and proof-based versus refutation-based verification. The authors report gains from agentic settings and higher F1 for FL Refute than FL Prove in the reported closed-source model comparisons.
Critical strengths
The use of authentic academic sources and human semantic checking is a substantive strength. The dataset includes research papers and graduate-level material, and the reported construction process goes beyond Lean compilation and proof checking to include semantic inspection and repair of the NL–formal alignment. A provable statement that changes the source's meaning is not a useful evaluation target. This alignment effort therefore supports the dataset's credibility and provides useful material for studying formalization difficulties in ML theory.
Critical weaknesses
The paper does not clearly distinguish recoverable omissions from ambiguity or missing information in the input, raising a concern that formalization is being mixed with adding assumptions from domain priors. For example, if the context or an explicit convention establishes a : Nat, omitting the type when writing a = 1 can be reasonable. However, given only x² ≥ 0 without a domain or background, a model should not choose the real numbers solely because that interpretation makes the claim true. Formalization should preserve the intended meaning rather than assume truth and supply conditions that make a statement hold. This matters especially for conjectures or claims whose truth is unresolved. My concern is therefore whether the actual model-visible context includes the standing assumptions relevant to the target claim in the source paper or textbook. If it does, missing conditions may reflect a failure to recover supplied information. If it does not, and the model must infer from ML domain knowledge which conditions would make the theorem provable, the task adds reasoning and theorem repair beyond translation. The paper describes context collection and human checking, but does not adequately separate these cases. For example, the forward ch10_lemma_10_1 has an empty context, while its backward counterpart explicitly adds sampling and measurability setup; fully rendered inputs are needed to establish the evaluation-time situation. Until this is clarified, conditions absent relative to the reference should not all be interpreted as model failures to recover assumptions from the input.
The equivalence evaluator lacks sufficient calibration on rejected outputs to support the interpretation of low formalization scores. The BEq+ comparison and inspection of 20 accepted pairs do not estimate how often faithful outputs are rejected by restricted search or the gates. The supplied materials lack a human audit of rejected candidates by model and mode, as well as predictions, directional proofs, and gate outcomes for inspection. If certification recall differs across models or formulations, both absolute scores and relative comparisons may be affected. The current evidence therefore does not sufficiently separate semantic errors from failures to certify equivalence. A direct calibration would construct several human-verified equivalent reference formulations for the same target, varying only hypothesis order, statement structure, or elementary algebraic presentation, and test whether BEq certifies equivalence across them. Rewrites using simple properties such as commutativity could reveal whether restricted search or the gates miss equivalences that are obvious to a human. If such variants are rejected, even entirely faithful outputs could score below 100% because of their formulation. Even if relative model trends remain largely unchanged, absolute pass rates should not be interpreted directly as semantic accuracy without this calibration.
Questions for authors
The following are clarification requests about the evaluated tasks and existing evidence.
- Can you provide the fully rendered input for ch10_lemma_10_1 and explain how its sampling, independence, and measurability conditions are recoverable? More generally, how do you distinguish conditions supplied in the input, inherited from other source context, and introduced during formalization? Does proof generation receive a complete informal proof or partial proof hints? These input differences affect the task definition and difficulty.
- Have you manually audited a stratified sample of candidates rejected by BEq, distinguishing models, modes, and failed directions or gates? Can you provide representative candidates, directional proofs, and gate decisions to clarify whether low certification rates primarily reflect semantic errors or evaluator limitations?
- Can you provide the original and mutated statements for Table 7, their falsity-validation evidence, and the selection procedure for the 120 mutations? Since FL Prove and FL Refute can generate different step propositions, what existing checks beyond the limited diagnostics in D.2 support attributing their performance difference to proof versus refutation mechanisms? These details affect the scope of the conclusions.
- Can Table 1 distinguish the statement-autoformalization version of RLM25 from RLMEval, publicly available in October 2025 and already covering complete proof generation and proof autoformalization? With this work included, how should MLP's novelty be specified in terms of the ML domain, dataset construction, or particular three-task combination, rather than a broader first end-to-end claim?
4 comments
Response to Review from HP3C (2/2)
Q3: Table 7 data and attribution
-
Data. The updated supplementary material provides the original and perturbed statements for Table 7 (
examples/table7_perturbations/perturbations.jsonl), each with a gold Lean refutation as falsity evidence (examples/table7_perturbations/gold_refutations/). GPT-5.6-Sol and Claude-Opus-5 each proposed three perturbations per statement; we kept one that compiles and has a gold refutation, balancing six types (strengthened conclusion 20%, changed constant 20%, flipped inequality 19%, dropped hypothesis 18%, changed quantifier 13%, weakened hypothesis 10%). Gold refutations were written with a human in the loop and checked by Lean, so admission does not depend on whether any evaluated model can refute the statement. Models are not told what changed. Appendix C.2 of the revised PDF gives details. -
Attribution. Apart from sampling, which the Peer Conflict analysis in Appendix D.2 measures, FL Prove and FL Refute differ only in the decision rule. They share the same formalization prompt, FL Prove only omits the REFUTATION block, and FL Refute refutes the model's own STATEMENT verbatim. Figure 4 shows the effect of the decision rule: FL Prove flags valid steps it fails to prove, so its flags land before the gold step.
Q4: RLMEval and novelty
We thank the reviewer for pointing this out. We agree that RLMEval should be listed separately and that our novelty claim should be stated more precisely. The revised PDF now cites RLMEval [1] and discusses it in Related Work as the first benchmark for research-level theorem proving and proof auto-formalization. Table 1 also lists RLMEval separately from RLM25: RLM25 targets statement auto-formalization, RLMEval theorem proving and proof auto-formalization, and neither covers proof verification or ML theory. We have also replaced the "first end-to-end" claim with a narrower one: the first Lean benchmark combining the three tasks for ML theory under a unified Lean framework.
We are truly grateful for your detailed feedback, which has greatly helped improve the clarity and rigor of our work. We hope the revised paper and the updated supplementary material resolve the concerns you raised. If they do, we would be grateful if you would reconsider your score. We welcome any further questions.
[1] Poiroux, A., Bosselut, A., and Kunčak, V. RLMEval: Evaluating Research-Level Neural Theorem Proving. Findings of EMNLP 2025.
Thank you for the detailed response and the additional experiments. I am raising my rating from 2 to 3. The suggestions below are for clarifying the results and improving future evaluations.
I checked the original MIT lecture notes for Lemma 10.1. The lemma leaves the sampling setup implicit, and its proof uses independence and a common sampling distribution. The rendered prompt clarifies how these conditions were supplied to the model and resolves my concern about this example. One suggestion is to briefly describe how the context and formalization notes were collected and checked for completeness. It would also help to break down the omitted assumptions by whether they were explicitly supplied to the model, appeared elsewhere in the source but were absent from the input, or were left to be inferred from domain conventions.
The audit of actual agent outputs substantially addresses my concern about BEq calibration. For the 90 manually constructed pairs, the changes are mostly local, and whether BEq can handle many of them can be anticipated from its implementation and allowed tactics. My suggestion for future calibration is to construct independently verified equivalent statements whose types, extent, and combinations of changes reflect actual agent outputs. Applying the same BEq protocol to these statements would estimate a practical score ceiling for that output distribution. If an agent were semantically correct on every problem while expressing its answers in forms typical of real agents, what fraction would BEq certify? Comparing each model’s observed score with this estimated ceiling would help readers interpret how far it falls short of that benchmark and how much headroom the evaluator itself leaves below 100%.
I also appreciate the release of the Table 7 perturbations and their Lean refutations. I accept the observed advantage of FL Refute in the full error-localization pipeline. Since FL Prove and FL Refute can produce different Lean statements for the same source step, the current comparison leaves open how much of the gain comes from the refutation rule itself. As a suggestion for future work, applying both rules to the same fixed set of Lean step formalizations under comparable search budgets would help isolate that contribution.
A Follow-up Response to Reviewer HP3C
Dear Reviewer HP3C,
We are glad that our response and the additional experiments addressed your concerns, and we sincerely appreciate the update to your score. Thank you as well for the further suggestions. We respond to each of them below.
1. Collection of context and formalization notes; breakdown of omitted assumptions
Collection and completeness. For each problem, an LLM extracted the context from the source. We deliberately over-extracted to reduce omissions, and then manually removed irrelevant material (Sec. 2.1.1, AI Use Statement). The formalization notes were written by the authors, who reviewed every assumption of each reference statement one by one. An assumption goes into the notes if it repairs a statement that is false as printed, or if it admits several reasonable formalizations.
Breakdown. Given the time available, we cannot provide the full statistics across all models. Instead, the revised PDF adds two worked cases in Appendix E.6: one where the omitted assumption is stated in the supplied context, and one where it is left to field convention. Among all omitted assumptions we traced in our audit sample, we found none that appears in the source but is absent from the input.
2. A practical ceiling for BEq
We agree that the controlled rewrites are local. The audit of real agent outputs (Common Response 2, part 2) gives a first estimate of the ceiling you describe. Of the 76 model outputs that are semantically equivalent to the reference, BEq certifies 63, so an agent that is always correct and writes in the style of real agents would score about 83%.
| Claude-Opus-5 | GPT-5.6-Sol | |
|---|---|---|
| Observed BEq | 27.5% | 25.0% |
| Estimated ceiling | 82.5% | 83.3% |
| Observed / ceiling | 33.3% | 30.0% |
The evaluator therefore leaves about 17 points of headroom below 100%, and most of the remaining gap comes from the models.
3. Isolating the decision rule
We ran this experiment. We fixed the Lean step statements from each model's original run and gave them to the other model. FL Prove and FL Refute were run under the same budget on 40 problems (20 correct, 20 flawed).
| Statements by | Prover | FL Prove F1 | FL Refute F1 |
|---|---|---|---|
| Claude-Opus-5 | Claude-Opus-5 | 45.8 | 63.6 |
| Claude-Opus-5 | GPT-5.6-Sol | 41.5 | 63.6 |
| GPT-5.6-Sol | Claude-Opus-5 | 28.1 | 37.0 |
| GPT-5.6-Sol | GPT-5.6-Sol | 13.6 | 38.5 |
With the statements fixed, FL Refute still outperforms FL Prove in every setting. Under FL Refute, accuracy is determined mainly by the quality of the formalization: on the same statements, the two provers score almost identically. Under FL Prove, formalization quality matters just as much, but proving ability also plays a role, since on the same statements, Claude-Opus-5 scores higher than GPT-5.6-Sol as the prover.
We are truly grateful for your detailed feedback, which has greatly helped improve the clarity and rigor of our work.
Best Regards,
Submission #19219 Author Team
Response to Review from HP3C (1/2)
Dear Reviewer HP3C,
Thank you for your thoughtful review and constructive suggestions. We especially appreciate the time and care you took to examine the individual cases in our benchmark in such detail. Your comments have helped us strengthen the soundness of our paper. We respond to your concerns as follows:
W1 & Q1: Provenance of assumptions
The input of each task should indeed be fully specified. Our original supplementary material contained the main components of the benchmark, while some additional benchmark data and evaluation scripts were not included. The updated supplementary material now contains the benchmark and evaluation pipeline, including the formalization notes, the fully rendered prompt for
ch10_lemma_10_1, and representative model predictions with their directional proofs and gate decisions.Input of auto-formalization. In all reported runs, the model receives the NL statement, the context, formalization notes, and the Lean file with the target replaced by a hole. The formalization notes record modeling choices the source leaves open and repairs of statements that are false as printed. In the initial version of the paper, the prompt in Appendix F.1 showed this input only as the placeholder
{nl}, which stands for the NL statement, the context, and the formalization notes together. The revised Appendix F.1 expands this placeholder. All reported experiments already used this input, so the setup and all numbers are unchanged.Classification of conditions. We agree with the x² ≥ 0 example. Conditions in a reference statement fall into four kinds, from easiest to hardest to recover:
a : Natexample. The model must recover these from its own understanding of ML theory and Lean.The first, second and fourth kinds are all in the model's input; only the third relies on field knowledge. The model is not asked to repair a theorem or to choose conditions that make it provable. What it must do is read a research-level statement as an expert in the field would, which requires a solid understanding of ML theory and Lean. We regard this as the core difficulty that a research-level benchmark should measure.
Worked example of
ch10_lemma_10_1. Appendix E.4 of the revised PDF shows the input for this lemma and where each sampling, independence and measurability condition comes from. Its context is empty because it is the first lemma of its chapter and no earlier definition or result precedes it. The sampling model is provided through the formalization notes instead.Input of proof generation. The model receives the NL statement and the full Lean statement, but no NL proof.
Ablation on formalization notes. We cleared only the formalization notes and kept every other part of the input verbatim (Claude-Opus-5):
When we remove only the notes, BEq on the affected problems drops from 14.5% to 2.6%, which shows the repairs are recovered from the notes rather than guessed.
nl_notesofBenchmarks/auto-formalization.jsonl), and the fully rendered prompt forch10_lemma_10_1is inexamples/ch10_lemma_10_1_rendered_prompt.txt.W2 & Q2: BEq calibration
We followed the reviewer's suggested design and built controlled equivalent rewrites, all of which BEq certified (Common Response 2, part 1). On real model outputs in agent mode, BEq recall is 82.5% for Claude-Opus-5 and 83.3% for GPT-5.6-Sol (Common Response 2, part 2), so low certification rates mainly reflect semantic errors; absolute scores are a conservative lower bound and relative comparisons are unaffected. Of the 13 false negatives, 6 were rejected by the bridge gate, 2 by the tactic whitelist, and for 5 the evaluator found no proof. Representative candidates, directional proofs, and gate decisions are in
examples/beq_cases/of the updated supplementary material, and BEq is implemented insrc/mlpbench/autoform/beq.py.