acceptodds
Under review as a conference paper at ICLR 2027

LeanPRM: Lean-Grounded Supervision for Natural Language Proof Verification

Abstract

AI systems now write long and difficult mathematical proofs, and checking their validity is becoming a bottleneck. A learned verifier could check proofs cheaply, but training one requires a correctness label on every step, and such labels are hard to obtain for open-ended proofs. Existing methods take them from human annotators, which is costly, or derive them from final answers, which proofs do not have, and formalizing each proof in Lean requires expert effort. We introduce LeanPRM, a framework that uses the Lean proof assistant to create step-level labels for natural-language proofs. A model writes a natural-language proof together with a paired formal proof, without feedback from Lean. Lean then labels every step, and a second model discards failures caused only by the formal syntax, so that the negative labels mark errors in the mathematics. LeanPRM contains 10,634 proofs with 152K labeled steps and no human annotation, and we release a held-out benchmark, LeanPRM-Bench. A Qwen3-4B verifier trained on LeanPRM raises exact step-label accuracy on LeanPRM-Bench from 48.2 to 72.5, as opposed to 59.5 when the same base model is trained on PRM800K. It also selects better proofs in best-of-N selection on IMO-level problems, and leads on 3 of 4 external benchmarks. Training on LeanPRM improves verifiers at every size from 0.6B to 8B parameters on LeanPRM-Bench, Hard2Verify and IMO-GradingBench. Given the labels need no human annotation, LeanPRM can grow with the supply of formal statements, offering a path to step-level feedback on AI-generated proofs that are hard for people to check.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Proof verification, Lean, Process supervision, Synthetic data

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nzx6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Lean-PRM, a framework that applies a Lean proof assistant to create step-level labels for natural-language proofs. The paper applies a closed-source LLM to write NL (natural language) proofs alongside a paired formal proof, then uses Lean and the LLM to filter the data. Obtaining a dataset containing 10,634 proofs with 152K labeled steps, with no human annotation. It also contains a held-out benchmark, LeanPRM-Bench, a 200-problem benchmark for models to locate errors in proofs. The Qwen3-4B verifier, trained on such a task, outperforms the selected small-model baseline across 5 benchmarks.

Critical strengths

Useful NL-Lean-aligned corpus: The released corpus pairs NL proof steps with Lean steps and provides step-level labels at a relatively large scale. It comes with the generation pipeline and LeanPRM-Bench. Such aligned data may benefit the domain beyond verifier training.

Critical weaknesses
  1. Unclear motivation for step-level NL-Lean alignment: The pipeline requires each NL step to match exactly one Lean step. This creates a significant mismatch between the high-level NL proof and low-level Lean. The gap between them is one critical concern with the motivation.
  2. Incomplete experiments: The experiment section for this paper is largely incomplete, at least missing the following components:
    1. Outdated base model: All the verifiers are trained based on Qwen-3, which is relatively outdated compared with newer models such as Qwen-3.5.
    2. No agentic baselines: Current research increasingly employs multi-turn agents that gradually explore and refine proofs rather than one-shot responses. All the trained models and baselines are one-shot and use no tools. Omitting agentic baselines and any discussion of how the data could benefit agentic training therefore significantly limits the paper’s contribution to the community.
  3. Data quality concerns:
    1. No quality guarantee beyond LLM-as-judge: This is my most critical concern about the paper. Apart from the success of Lean compilation, all quality assurance is performed by an LLM in the work, including generating proofs, classifying failures, checking NL-Lean faithfulness, and judging quality audits. In the process, the paper applies Claude for both generation and quality assurance without any human verification. This casts doubt on the foundation of the dataset. At a minimum, the 200-problem benchmark should be verified by humans, record by record, to correct potential mistakes.
    2. The claim of “long and difficult proofs” in the paper: The average proof length has about 14.3 steps, and most of the theorems come from NuminaMath-Lean and Lean-workbook, which are mostly high-school to college-level problems. This is far from the “long and difficult math proofs” used to motivate the paper; thus, it is unclear whether the conclusion applies to that setting.
Questions for authors
  1. What is the motivation for aligning the NL proof with the Lean proof step by step? How do the authors ensure that a failing Lean step reflects an error in the NL step rather than a granularity mismatch?
  2. How does the method perform with a more recent base model such as Qwen3.5? Is the improvement over the PRM800K-trained verifier still significant?
  3. Since large LLMs already verify these proofs well in direct-answer mode, would the task be largely resolved in multi-turn agent mode? What is the performance of Claude with Claude Code and GPT with Codex when tools are enabled?
  4. Can the authors provide human verification of LeanPRM-Bench, or at least of a random sample of the incorrect labels?
  5. What is the performance on genuinely long-horizon proofs, using the latest models with their corresponding harnesses?
4 comments
AuthorsOpenReview ↗

Response to Reviewer nzx6 (1/2)

Thank you for the review. We address each point below and will add these results to the revision.

W1 & Q1: Motivation for step alignment, and granularity

A step-level verifier needs a label on every step, and human step labels are expensive (PRM800K [1] requires about 800,000 human-annotated step labels). Lean failures provide automatic process-level supervision: aligning each natural language sentence with its corresponding Lean step lets us map verification errors back to individual reasoning steps without human annotation. Lean is used only to build the data, and the trained verifier reads natural language only.

We agree that natural language proofs are usually higher level than Lean. The generation prompt fixes the step size: each step is "one Lean tactic or one tightly scoped Lean proof line", the model must not "use one opaque tactic to hide the main argument", and the natural language "must not describe a stronger or more detailed argument than the Lean line actually proves". In the released data the median step has 23 words, and 78.1% of steps correspond to a single Lean line.

To measure the correspondence, we asked Claude Fable 5.1, without our labels, whether each Lean step formalizes its sentence, on 200 random released proofs (2,889 steps).

Table R1: Step correspondence between prose and Lean.

StepsnMatchPartialMismatch
Correct proofs1,40699.6%0.4%0.0%
Incorrect, before the first error95398.4%1.0%0.5%
Incorrect, at the labeled first error7663.2%30.3%6.6%
Incorrect, after the first error45495.2%3.7%1.1%

The mismatch rate is low. 97.6% of all Lean steps match their sentence. Most mismatches are at the error step, where the Lean line makes the same claim but the tactic fails, and we already remove the error classes where this was most common. We align steps to build data for training a process verifier, and the verifier trained on LeanPRM beats the PRM800K-trained one on external benchmarks.

W2 & Q2: Newer base model

All comparisons in the paper fix the base model, recipe and number of updates, and LeanPRM improves Qwen3 at 0.6B, 4B and 8B. Based on your feedack, we have now trained Qwen3.5-4B (newest available Qwen model at 4B scale) with the same recipe on LeanPRM.

Table R2: LeanPRM-Verifier on Qwen3-4B and Qwen3.5-4B, same recipe.

ModelLeanPRM-Bench exactHard2Verify BAProcessBench F1PRMBench first err.IMO-GradingBench BA
Qwen3-4B (initial)48.237.174.342.553.8
LeanPRM-Verifier (Qwen3-4B, 500 updates)74.772.874.947.466.4
Qwen3.5-4B (initial)13.52.443.428.90.2
LeanPRM-Verifier (Qwen3.5-4B, 300 updates)78.579.073.147.270.4

With fewer updates, the Qwen3.5-4B verifier is above the Qwen3-4B verifier on three of the five benchmarks and within 2 points on the other two.

W3 & Q3: Agentic baselines

We use the standard setting for math verifiers and RL reward models, in which the model reads a proof and answers once without tools. Training with tools is outside the scope of the paper, and we will state this in the limitations.

We did test a tool-using frontier model on our LeanPRM-Bench. Claude Fable 5.1 was given a Lean 4 compiler with the Mathlib version used to build LeanPRM (up to 10 calls per proof), and we ran the same harness without the tool as a control.

Table R3: Claude Fable 5.1 on LeanPRM-Bench (mean of three runs).

SettingExactVerdict accuracy
One pass, no tool (paper)78.390.8
Tool harness, compiler disabled79.791.2
Tool harness, Lean compiler enabled83.794.7
LeanPRM-Verifier-4B, one pass74.789.5

The compiler adds 4.0 points (95% CI +0.8 to +7.3). Claude called it on 26% of proofs, and the gain comes almost entirely from those problems. Our 4B verifier, with one pass and no tools, is 9 points below.

The reviewer also asks how the data could benefit agentic training. A step-level verifier can act as a reward model for provers or tell an agent where its first error is. In best-of-N on IMO-GradingBench, ranking candidates with our verifier selects a correct proof for 46.0% of problems at N=16, against 37.3% for the PRM800K verifier and 32.4% at random.

[1] Lightman et al. "Let's Verify Step by Step."

AuthorsOpenReview ↗

Response to Reviewer nzx6 (2/2)

W4 & Q4: Quality beyond LLM judges

The verdict and the first-failure location come from compiling the Lean proof. We use a model call only to decide whether a failure is mathematical or formal and whether the natural language matches the Lean. All 100 correct LeanPRM-Bench proofs compile, and none use sorry, admit or native_decide.

Human audit. Authors with graduate-level expertise graded 79 LeanPRM-Bench proofs by hand (40 we label correct, 39 incorrect), reading the natural language with our label and the GPT-6-Luna answer shown, so this check is not blind. The graders' first-error step agrees with ours on 69 of 79 (kappa 0.83). The graders rejected 6 of the 40 proofs we label correct, in each case agreeing with an error GPT-6-Luna had flagged at the same step, and flagged 38 of the 39 proofs we label incorrect, 35 of them at the same step as our label.

Table R4: Agreement of LLM judges with our labels on LeanPRM-Bench (100 correct, 100 incorrect), judges not shown our labels. Kappa is Cohen's kappa (1 = perfect agreement, 0 = chance).

JudgeCorrect acceptedIncorrect flaggedSame first-error stepKappa (verdict)
Qwen3-4B6682460.48
Claude Haiku 5.57996650.75
Claude Fable 5.19194720.85

On 296 released training proofs, a blind audit with GPT-6-Luna, a model from another provider, found an error in 90.4% of proofs we label incorrect (at our step or earlier in 71.9%) and accepted 84.7% of those we label correct. So a model that had no part in generating the data agrees with our verdict on 259 of the 296 proofs (87.5%).

W5 & Q5: "Long and difficult proofs"

We agree with the reviewer's description of the level and will reword the sentence in the introduction, which describes the motivation and not the dataset. Most statements come from NuminaMath-LEAN (hard tier) and Lean-Workbook, from high-school competition to undergraduate level. Proofs have 14.3 steps on average (median 12), 10.2% have at least 30, and the longest has 95. On Hard2Verify and IMO-GradingBench, whose proofs come from outside our pipeline, LeanPRM-Verifier-4B is 9.7 and 9.4 points above PRM800K-Verifier-4B. We have not evaluated research-level proofs or long agent traces. The pipeline needs no annotators and can be rerun with stronger provers on harder statements.

Thank you so much for your rebuttal. However, my major concerns, the data quality guarantees and the motivation of the paper, remain unanswered. Furthermore, the incorporation of more recent open- and closed-source models also indicates that the proposed benchmark is saturated. Thus, I would like to maintain my rating

Thank you for the follow-up. We respond to the three points below.

On data quality (W4 & Q4 of our response), we label a proof correct only when its Lean proof compiles. We also carried out the human verification requested in Q4 on 79 LeanPRM-Bench proofs. The first-error step of the graders coincides with our label in 69 cases, and they found an error in 38 of the 39 proofs we label incorrect.

Regarding motivation, we pair each sentence with one Lean step because this gives us step labels without human annotation. The gap between natural language and Lean was measured in Table R1, where 97.6% of Lean steps match their sentence.

Regarding saturation, the numbers do not seem to support it. Claude Fable 5.1 with a Lean compiler obtains 83.7% exact accuracy on LeanPRM-Bench (Table R3), and the best model without tools obtains 80.3% (Table 2 of the paper). Besides, most of our evidence is on external benchmarks. There the LeanPRM verifier is ahead of the PRM800K one by 9.7 points on Hard2Verify and 9.4 on IMO-GradingBench.

Reviewer_8PtcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper introduces LeanPRM, a framework for automatically constructing process supervision datasets for use in natural-language mathematical proof verification with Lean. For a given formal theorem statement, an LLM outputs a natural-language proof plus Lean proof code, with no feedback from the theorem prover. Lean is then used to find the first failing formal step, while a second LLM distinguishes mathematical reasoning errors from formalization errors and attempts limited syntax repairs. The labels are then used to train a natural-language proof verifier that does not require Lean at inference time.

The authors construct a dataset of 10,634 proofs and ~152K step labels, along with a held-out benchmark of 200 proofs. They train Qwen3 verifiers using GRPO and evaluate on five benchmarks. The 4B verifier trained on LeanPRM outperforms an otherwise identical PRM800K baseline on four of five benchmarks, including an increase from 59.5% to 72.5% exact step-label accuracy on LeanPRM-Bench and a balanced accuracy increase from 61.3% to 70.8% on Hard2Verify. The authors also investigate model scaling and best-of-N proof selection.

Critical strengths
  1. A practical approach to producing process supervision for open-ended proofs: The proposed framework addresses the important limitation of current PRM training approaches in that it provides for obtaining step-level correctness labels for the mathematical proofs without the need for expensive human annotation or for a checkable numerical final answer. The paired generation of natural-language and formal steps is a good design choice, as it allows the resulting verifier to operate entirely on natural-language input. This is significant, as a successful implementation of such proof verifiers would render the training process of proof verifiers much more scalable.

  2. Improvements across multiple external evaluation benchmarks: The authors test their approach on multiple datasets besides their own benchmark. Under the controlled 300-update training comparison, the LeanPRM verifier performs better than the PRM800K-trained one across four out of the five benchmarks, with especially notable improvements on Hard2Verify and the more recent IMO-GradingBench (Table 2). These results show that the learned verification behavior generalizes to some extent to external evaluations beyond just the authors’ dataset. The model-scale experiments and the solver-trained baseline experiments were a welcome addition to the experimental evaluation.

Critical weaknesses
  1. The reliability and semantics of the generated step labels are not sufficiently established: The main result relies crucially on the assumption that Lean verification can generate reliable correctness labels for natural-language reasoning steps. This rests on two things: first, that the natural language and formal steps are in correspondence with fidelity; and second, that errors are accurately differentiated between mathematical and formalization failures. On both counts, the process relies to a large extent on LLM judgments, not mechanical checks. The audit in Section 2.1 raises a valid concern. In Table 1, the independent judge considers 13 of 30 flagged steps to be mathematically sound in each of the two mathematical classes audited: unjustified assumption and invalid inequality. Although the audit was run on the earlier data pool, the class filter removes only formalization-side classes, so these two classes, which make up roughly half of the final negatives (Figure 2), remain in the final dataset. This raises questions about how noisy the error classifications are, and the paper provides no independent human evaluation of label precision on the final dataset. Moreover, the labeling convention assigns an incorrect label to steps that are locally valid but come after the first error or end an incomplete proof. Figure 7 (Appendix C) shows an example: the proof derives the summed inequality and then ends at step 19 with “Let S denote the sum of all the a_i,” a trivially valid definition, without producing the two distinct zeros. This final step is labeled incorrect only because the proof stops short. This could be a defensible proof-prefix convention, but it does not reflect local mathematical validity. Finally, Appendix B states that Lean does not accept sorry or admit as proofs, but that the dataset contains 605 proofs that include sorry, admit, or native_decide. The standard practice in Lean is to allow sorry declarations but issue warnings. The paper does not sufficiently clarify whether the verification harness outright rejects such proofs or merely warns and whether proofs that successfully use sorryAx are excluded. It is not clear the extent to which the harness ensures the soundness of positive labels. Since the paper's main contribution hinges on the assumption that Lean produces reliable correctness labels, these considerations have direct bearing on the value of the contribution.

  2. The experimental comparison does not isolate the contribution of Lean-grounded supervision: The strongest comparison is to a model trained on PRM800K, but PRM800K and LeanPRM are fundamentally different in their problem distribution, proof organization principles, and annotation conventions. The improvement over PRM800K therefore reflects the combination of training on the authors' particular proof distribution and stronger supervision signals. A more informative experiment would fix the natural-language training proofs and contrast Lean-derived labels with those of an LLM judge. This would allow isolating the contribution of stronger supervision vs. better proof distribution. The latter is important since the paper's main contribution is not the creation of another mathematical proof data set but rather the proposal that formal verification is a superior method of supervision. The current experiments demonstrate the benefit of the combined approach but do not distinguish between the two factors. The best-of-N results are also relatively modest: at N=16, the trained verifier achieves 35.4% correct-proof selection vs. 32.4% random selection (section 3.1). This is a positive result but offers limited evidence of downstream utility.

Questions for authors
  1. Label reliability and formal verification. Could the authors clarify how the checking harness handles sorry, admit, and native_decide, particularly if any rejects declarations that rely on sorryAx? Furthermore, can the authors provide an independent human evaluation of label correctness on a representative sample of the final dataset, including correct proofs, incorrect proofs, and the agreement between the natural-language and Lean steps? This would have a material effect on my perception of supervision quality.

  2. Isolating the benefit of Lean labels. Have the authors considered training a verifier on the same natural-language proofs using labels from a strong LLM judge instead of Lean? Such a comparison would isolate the effect of formal verification versus the effect of proof distribution and training format on the outcome. Evidence that Lean labels reliably outperform LLM-generated labels on the same proofs would materially strengthen my assessment.

  3. Interpretation of step correctness. What proportion of the proofs labeled incorrect receive that label because they are incomplete, rather than because they contain an invalid mathematical inference? In a case as in Figure 7, why should a mathematically valid final step be assigned an incorrect label? Could the authors clarify how frequently this occurs and if it has an effect on the learned verifier's behavior?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8Ptc (1/2)

Thank you for the detailed review. We address each point below with new experimental results, and will add these results to the revision.

W1 & Q1: Label reliability

We audit the pool before filtering, one step at a time, and present the results in Table 1 of the paper. For the released data we ran four further checks.

(1) Human audit. Authors with graduate-level expertise graded 79 LeanPRM-Bench proofs by hand (40 we label correct, 39 incorrect), reading the natural language with our label and the GPT-6-Luna answer shown, so this check is not blind. The graders' first-error step agrees with ours on 69 of 79 (kappa 0.83). The graders rejected 6 of the 40 proofs we label correct, in each case agreeing with an error GPT-6-Luna had flagged at the same step, and flagged 38 of the 39 proofs we label incorrect, 35 of them at the same step as our label.

(2) Blind audit by a cross-provider judge. We gave 296 released proofs to GPT-6-Luna, a model from a different provider than our generator, with only the problem and the natural-language steps.

Table R1: GPT-6-Luna on released proofs, compared with our first-error step.

Our labelProofsError foundSame stepEarlierLaterNo error found
Correct15023---127 (84.7%)
Incorrect146132 (90.4%)82232712 (8.2%)
Unjustified assumption4844281244
Invalid inequality bound262613670
Missing case or side condition252314452
Bad witness or construction232218131
Algebraic identity error19148065
Modular or divisibility error331020

For 2 of the 146 incorrect proofs the judge gave no usable answer. For the two classes the reviewer raises (unjustified assumption and invalid inequality bound), the judge finds an error in 44 of 48 and 26 of 26 proofs, so the noise the reviewer points to is largely absent from the released data. All 23 flagged correct proofs compile in Lean.

(3) Correspondence between natural language and Lean. On 200 random released proofs (2,889 steps) we asked Claude Fable 5.1, without our labels, whether each Lean step formalizes its sentence.

Table R2: Step correspondence.

StepsnMatchPartialMismatch
Correct proofs1,40699.6%0.4%0.0%
Incorrect, before the first error95398.4%1.0%0.5%
Incorrect, at the labeled first error7663.2%30.3%6.6%
Incorrect, after the first error45495.2%3.7%1.1%

Of the 28 error steps (out of 76) that are not a full match, the Lean line states the same claim as the sentence and the tactic fails in 18, states a different claim in 6, and in 4 the prose itself says the step is incomplete. In proofs regenerated by Claude Sonnet 5 and Fable 5.1 for 100 statements, 99.1% and 99.4% of steps match, so the correspondence holds for weaker generators too.

(4) Judges of different strength. In Table R4 under Q2 we show that agreement with our labels rises with judge strength, from Cohen’s kappa 0.48 for Qwen3-4B to 0.85 for Claude Fable 5.1.

W1 & Q1: sorry, admit and native_decide

The reviewer is right, and we will correct the sentence in Appendix B. Lean accepts sorry and admit with a warning, and our harness counts a proof as compiled when Lean reports no error. We instruct the model to not use placeholders in the Stage 1 prompt and in Stage 2 repairs, but we did not filter the Stage 1 output for them. A scan of all 10,634 released proofs:

Table R3: Released proofs containing each construct.

Our labelProofssorryadmitnative_decide
Correct6,3701015
Incorrect4,264571316

The one correct proof with sorry is in the training split, and we will remove it from the release. No other correct proof contains sorry, admit, sorryAx, axiom or unsafe, and no correct proof in the validation split or LeanPRM-Bench uses any of the three constructs. We keep native_decide, which Lean accepts without a warning. In the 574 incorrect proofs with a placeholder, it is at our first-error step in 393, after it in 157 and before it in 24. In those 24 proofs (0.2% of the data), one step is labeled correct without support from Lean. We will remove these proofs from the training data.

AuthorsOpenReview ↗

Response to Reviewer 8Ptc (2/2)

W1 & Q3: Prefix convention and unfinished proofs

Our labels follow a prefix convention, the same one used by PRM800K [1] and ProcessBench [2], and we will state it explicitly in Section 2. A step is labeled correct when the proof is valid up to and including it, and the final step also requires the theorem to be proved. In Figure 7 the proof stops before producing the two zeros, so the final step is labeled incorrect although the sentence is true.

Stage 2 returns one verdict for false, unjustified and unfinished steps, so we cannot separate them exactly. 790 of the 4,264 incorrect proofs (18.5%) have their first error at the final step, which includes every proof that stops short of proving the original theorem. In LeanPRM-Bench the number is 15 of 100. Our external metrics use only the verdict or the first-error position, so labels after the first error do not affect them. On correct LeanPRM-Bench proofs, LeanPRM-Verifier-4B flags only the final step in 5.3 of 100 cases, compared with 3.0 for PRM800K-Verifier-4B, a small side effect we will report.

W2 & Q2: Lean labels vs proof distribution

We have not trained on the same proofs with judge labels and will list this as a limitation. We did measure how far judge labels would differ from ours on LeanPRM-Bench.

Table R4: LLM judges vs our labels on LeanPRM-Bench (100 correct, 100 incorrect). Kappa is Cohen's kappa (1 = perfect agreement, 0 = chance).

JudgeCorrect acceptedIncorrect flaggedSame first-error stepKappa (verdict)Kappa (step)
Qwen3-4B6682460.480.43
Claude Haiku 5.57996650.750.64
Claude Fable 5.19194720.850.75

Judges cheap enough to label 10,634 proofs, such as our base model or Haiku, place the first error differently from Lean in a third to a half of the incorrect proofs. In our pipeline the compiler locates the failure, and the model call in Stage 2 only classifies it as mathematical or formal. In our comparisons the base model, recipe and number of updates are fixed, and NuminaMath-Solver-4B, trained with the same recipe on math problems, stays at the base model's level on LeanPRM-Bench (48.3 vs 48.2).

W2: Best-of-N gain

We agree the gain in the paper is small. The paper picks at random among accepted candidates. We fixed the following rule in advance and did not tune it on IMO-GradingBench. During the rebuttal, based on your feedback, we tried a different rule for choosing a proof among candidates with a verifier model. We score each candidate with the verifier's own label probabilities: with pip_i the probability of "incorrect" at step ii and ff the first step labeled incorrect (the last step if none), the score is max⁡i≤fpi\max_{i \le f} p_i, and we select the lowest. The proof candidates are the same as in the paper.

Table R5: Best-of-N on IMO-GradingBench, % of problems with a correct selected proof (mean of three runs; random is 32.4).

VerifierSelectionN=2N=4N=8N=16
PRM800K-4Brandom among accepted32.933.133.233.2
PRM800K-4Blowest score33.835.035.937.3
LeanPRM-4Brandom among accepted34.635.736.035.9
LeanPRM-4Blowest score36.840.243.246.0

With our new rule, at N=16 the LeanPRM verifier selects a correct proof for 46.0% of problems, against 37.3% for PRM800K. We will use this rule in Figure 4.

Results at 500 updates

In the revision, we report all trained models at 500 updates, the end of each training run, instead of the intermediate checkpoint at 300.

Table R6: Main comparison at 500 updates (mean of three runs).

ModelLeanPRM-Bench exactHard2Verify BAProcessBench F1PRMBench first err.IMO-GradingBench BA
Qwen3-4B (initial)48.237.174.342.553.8
NuminaMath-Solver-4B48.333.671.241.552.4
PRM800K-Verifier-4B60.363.183.043.357.0
LeanPRM-Verifier-4B74.772.874.947.466.4

[1] Lightman et al. "Let's Verify Step by Step." [2] Zheng et al. "ProcessBench: Identifying Process Errors in Mathematical Reasoning."

Thank you for the rebuttal. I am raising my recommendation to weak acceptance. The improvements on external benchmarks and the stronger best-of-N results convince me that the dataset is useful.

The additional audits and clarifications address many of my concerns. I still have some uncertainty about label reliability. However, I believe the demonstrated gains are sufficient to support acceptance. These remaining concerns keep me at weak rather than clear acceptance.

Reviewer_h6fZOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce a mathematical reasoning dataset called LeanPRM which contains several thousand natural language proofs with step-level correctness labels. The proofs are constructed by asking a model to generate a pair of informal and formal (Lean) proofs. When the Lean proof declares an error, we have a signal of where the natural language proof goes wrong. The authors take care to avoid including proofs that only had syntax errors in the Lean code, which is done using an LLM judge. An additional check also filters out proofs where a judge detected that the error comes from the formalization side rather than the natural language proof, like when a theorem is improperly applied in Lean.

A held-out evaluation dataset called LeanPRM-bench is also introduced which is used for evaluation. The authors use GRPO to train models at a variety of sizes on the dataset. A few other similar datasets with step-level labels from different sources are also used to train the same models and used as baselines for evaluation. Models trained on their dataset consistently outperform baselines on several datasets.

Critical strengths

I. Interesting approach to generating proofs with step-level labels.

The methodology that the authors introduce is unique and would be of interest to the community studying mathematical reasoning. The dataset generation method is clearly scalable to bigger models.

II. Care taken to avoid bad examples in dataset.

The set of measures used by the authors to filter out proofs that may be incorrectly labeled are fairly thorough while still retaining scalability. Even though this depends on LLM judges, I believe that these are fairly easy judging tasks that we can trust models on.

Critical weaknesses

I. Some poor quality labels detected by frontier model judges.

The authors report an important audit using a frontier model as a judge which does not agree with all the labels produced by the pipeline. Regardless of this, it seems that GRPO results in gains. I believe that this would become less of a problem when bigger models are used for dataset generation.

Questions for authors
  1. On Line 231, the predicted verdict and the true verdict are not defined. Are these overall proof verdicts or step level verdicts?
  2. Why do you think during the audit, Opus 5 finds that the natural language proofs are correct when the smaller model did not? What is the prompt used for this audit? Does Opus 5 correct the proof while judging?
1 comment
AuthorsOpenReview ↗

Response to Reviewer h6fZ

Thank you for the review. We address each point below and will add these results to the revision.

W1: Labels that frontier judges disagree with

Most of the judge’s disagreement in Section 2.1 came from steps whose math was correct but whose Lean failed for formalization-side reasons (Lean errors caused by misusing Lean, not incorrect math or incorrect syntax). This audit was run before filtering. In this audit, 35.8% of incorrect attempts first failed in one of four formalization-side classes, and the judge found 77% to 93% of the audited steps in them mathematically correct. We removed these classes, so none of those attempts is in the released data.

On the released data we ran three further checks.

(1) Cross-provider judge. GPT-6-Luna, a model from a different provider than our generator, was given 296 released proofs with only the problem and the natural-language steps. It agrees with our label on 132 of the 146 proofs we label incorrect (90.4%) and on 127 of the 150 we label correct (84.7%). All 23 correct proofs it flagged compile in Lean, so the formal proof is valid in every one of them.

(2) Judges of different strength. We also asked three judges of different strength for the first incorrect step of each LeanPRM-Bench proof, without our labels.

Table R1: Agreement with our labels on LeanPRM-Bench (100 correct, 100 incorrect). Kappa is Cohen's kappa (1 = perfect agreement, 0 = chance).

JudgeCorrect acceptedIncorrect flaggedSame first-error stepKappa (verdict)
Qwen3-4B6682460.48
Claude Haiku 5.57996650.75
Claude Fable 5.19194720.85

Agreement increases with judge strength, in line with the reviewer's expectation for larger models.

(3) Human audit. Authors with graduate-level expertise graded 79 LeanPRM-Bench proofs by hand (40 we label correct, 39 incorrect), reading the natural language with our label and the GPT-6-Luna answer shown, so this check is not blind. The graders' first-error step agrees with ours on 69 of 79 (kappa 0.83). The graders rejected 6 of the 40 proofs we label correct, in each case agreeing with an error GPT-6-Luna had flagged at the same step, and flagged 38 of the 39 proofs we label incorrect, 35 of them at the same step as our label.

Q1: Predicted and true verdict on line 231

Both are proof-level verdicts. The verifier outputs a label for each step and one overall verdict. In r=0.7Rstep+0.3⋅1[v^=v]r = 0.7 R_{\mathrm{step}} + 0.3 \cdot \mathbf{1}[\hat{v} = v], v^\hat{v} is that verdict and vv is the label of the proof, which is correct only if every step is correct. Step labels enter through RstepR_{\mathrm{step}}. We will define both symbols at line 231 in the revision.

Q2: Why Opus 5 disagrees, the prompt, and whether it corrects the proof

Our labels come from compiling the Lean proof written with natural language. A Lean step can fail while the sentence is sound, for example when a lemma is cited under a name Mathlib does not have, or an automated tactic such as linarith is too weak to prove a step that is true and justified. Stage 2 is meant to separate these failures from mathematical errors but misses some. Opus 5 sees only the natural language, where such a failure is invisible, and reports no error. This is how we identified the four classes we removed.

We report both prompts for Opus 5 in Appendix I (Prompts). The proof-level prompt asks for "the index of the FIRST step that is mathematically incorrect or unjustified", or 0 if "every step is valid and the proof is complete". The per-class prompt shows the preceding steps and one flagged step and asks whether its claim is "mathematically true AND adequately justified by what precedes it". We will reference the appendix in Section 2.1.

Opus 5 does not correct the proof. It returns a step index and a one-sentence explanation, and we used its answers only to decide which classes to remove.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.