- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Lean-PRM, a framework that applies a Lean proof assistant to create step-level labels for natural-language proofs. The paper applies a closed-source LLM to write NL (natural language) proofs alongside a paired formal proof, then uses Lean and the LLM to filter the data. Obtaining a dataset containing 10,634 proofs with 152K labeled steps, with no human annotation. It also contains a held-out benchmark, LeanPRM-Bench, a 200-problem benchmark for models to locate errors in proofs. The Qwen3-4B verifier, trained on such a task, outperforms the selected small-model baseline across 5 benchmarks.
Critical strengths
Useful NL-Lean-aligned corpus: The released corpus pairs NL proof steps with Lean steps and provides step-level labels at a relatively large scale. It comes with the generation pipeline and LeanPRM-Bench. Such aligned data may benefit the domain beyond verifier training.
Critical weaknesses
- Unclear motivation for step-level NL-Lean alignment: The pipeline requires each NL step to match exactly one Lean step. This creates a significant mismatch between the high-level NL proof and low-level Lean. The gap between them is one critical concern with the motivation.
- Incomplete experiments: The experiment section for this paper is largely incomplete, at least missing the following components:
- Outdated base model: All the verifiers are trained based on Qwen-3, which is relatively outdated compared with newer models such as Qwen-3.5.
- No agentic baselines: Current research increasingly employs multi-turn agents that gradually explore and refine proofs rather than one-shot responses. All the trained models and baselines are one-shot and use no tools. Omitting agentic baselines and any discussion of how the data could benefit agentic training therefore significantly limits the paper’s contribution to the community.
- Data quality concerns:
- No quality guarantee beyond LLM-as-judge: This is my most critical concern about the paper. Apart from the success of Lean compilation, all quality assurance is performed by an LLM in the work, including generating proofs, classifying failures, checking NL-Lean faithfulness, and judging quality audits. In the process, the paper applies Claude for both generation and quality assurance without any human verification. This casts doubt on the foundation of the dataset. At a minimum, the 200-problem benchmark should be verified by humans, record by record, to correct potential mistakes.
- The claim of “long and difficult proofs” in the paper: The average proof length has about 14.3 steps, and most of the theorems come from NuminaMath-Lean and Lean-workbook, which are mostly high-school to college-level problems. This is far from the “long and difficult math proofs” used to motivate the paper; thus, it is unclear whether the conclusion applies to that setting.
Questions for authors
- What is the motivation for aligning the NL proof with the Lean proof step by step? How do the authors ensure that a failing Lean step reflects an error in the NL step rather than a granularity mismatch?
- How does the method perform with a more recent base model such as Qwen3.5? Is the improvement over the PRM800K-trained verifier still significant?
- Since large LLMs already verify these proofs well in direct-answer mode, would the task be largely resolved in multi-turn agent mode? What is the performance of Claude with Claude Code and GPT with Codex when tools are enabled?
- Can the authors provide human verification of LeanPRM-Bench, or at least of a random sample of the incorrect labels?
- What is the performance on genuinely long-horizon proofs, using the latest models with their corresponding harnesses?
4 comments
Response to Reviewer nzx6 (2/2)
W4 & Q4: Quality beyond LLM judges
The verdict and the first-failure location come from compiling the Lean proof. We use a model call only to decide whether a failure is mathematical or formal and whether the natural language matches the Lean. All 100 correct LeanPRM-Bench proofs compile, and none use sorry, admit or native_decide.
Human audit. Authors with graduate-level expertise graded 79 LeanPRM-Bench proofs by hand (40 we label correct, 39 incorrect), reading the natural language with our label and the GPT-6-Luna answer shown, so this check is not blind. The graders' first-error step agrees with ours on 69 of 79 (kappa 0.83). The graders rejected 6 of the 40 proofs we label correct, in each case agreeing with an error GPT-6-Luna had flagged at the same step, and flagged 38 of the 39 proofs we label incorrect, 35 of them at the same step as our label.
Table R4: Agreement of LLM judges with our labels on LeanPRM-Bench (100 correct, 100 incorrect), judges not shown our labels. Kappa is Cohen's kappa (1 = perfect agreement, 0 = chance).
| Judge | Correct accepted | Incorrect flagged | Same first-error step | Kappa (verdict) |
|---|---|---|---|---|
| Qwen3-4B | 66 | 82 | 46 | 0.48 |
| Claude Haiku 5.5 | 79 | 96 | 65 | 0.75 |
| Claude Fable 5.1 | 91 | 94 | 72 | 0.85 |
On 296 released training proofs, a blind audit with GPT-6-Luna, a model from another provider, found an error in 90.4% of proofs we label incorrect (at our step or earlier in 71.9%) and accepted 84.7% of those we label correct. So a model that had no part in generating the data agrees with our verdict on 259 of the 296 proofs (87.5%).
W5 & Q5: "Long and difficult proofs"
We agree with the reviewer's description of the level and will reword the sentence in the introduction, which describes the motivation and not the dataset. Most statements come from NuminaMath-LEAN (hard tier) and Lean-Workbook, from high-school competition to undergraduate level. Proofs have 14.3 steps on average (median 12), 10.2% have at least 30, and the longest has 95. On Hard2Verify and IMO-GradingBench, whose proofs come from outside our pipeline, LeanPRM-Verifier-4B is 9.7 and 9.4 points above PRM800K-Verifier-4B. We have not evaluated research-level proofs or long agent traces. The pipeline needs no annotators and can be rerun with stronger provers on harder statements.
Thank you so much for your rebuttal. However, my major concerns, the data quality guarantees and the motivation of the paper, remain unanswered. Furthermore, the incorporation of more recent open- and closed-source models also indicates that the proposed benchmark is saturated. Thus, I would like to maintain my rating
Thank you for the follow-up. We respond to the three points below.
On data quality (W4 & Q4 of our response), we label a proof correct only when its Lean proof compiles. We also carried out the human verification requested in Q4 on 79 LeanPRM-Bench proofs. The first-error step of the graders coincides with our label in 69 cases, and they found an error in 38 of the 39 proofs we label incorrect.
Regarding motivation, we pair each sentence with one Lean step because this gives us step labels without human annotation. The gap between natural language and Lean was measured in Table R1, where 97.6% of Lean steps match their sentence.
Regarding saturation, the numbers do not seem to support it. Claude Fable 5.1 with a Lean compiler obtains 83.7% exact accuracy on LeanPRM-Bench (Table R3), and the best model without tools obtains 80.3% (Table 2 of the paper). Besides, most of our evidence is on external benchmarks. There the LeanPRM verifier is ahead of the PRM800K one by 9.7 points on Hard2Verify and 9.4 on IMO-GradingBench.
Response to Reviewer nzx6 (1/2)
Thank you for the review. We address each point below and will add these results to the revision.
W1 & Q1: Motivation for step alignment, and granularity
A step-level verifier needs a label on every step, and human step labels are expensive (PRM800K [1] requires about 800,000 human-annotated step labels). Lean failures provide automatic process-level supervision: aligning each natural language sentence with its corresponding Lean step lets us map verification errors back to individual reasoning steps without human annotation. Lean is used only to build the data, and the trained verifier reads natural language only.
We agree that natural language proofs are usually higher level than Lean. The generation prompt fixes the step size: each step is "one Lean tactic or one tightly scoped Lean proof line", the model must not "use one opaque tactic to hide the main argument", and the natural language "must not describe a stronger or more detailed argument than the Lean line actually proves". In the released data the median step has 23 words, and 78.1% of steps correspond to a single Lean line.
To measure the correspondence, we asked Claude Fable 5.1, without our labels, whether each Lean step formalizes its sentence, on 200 random released proofs (2,889 steps).
Table R1: Step correspondence between prose and Lean.
The mismatch rate is low. 97.6% of all Lean steps match their sentence. Most mismatches are at the error step, where the Lean line makes the same claim but the tactic fails, and we already remove the error classes where this was most common. We align steps to build data for training a process verifier, and the verifier trained on LeanPRM beats the PRM800K-trained one on external benchmarks.
W2 & Q2: Newer base model
All comparisons in the paper fix the base model, recipe and number of updates, and LeanPRM improves Qwen3 at 0.6B, 4B and 8B. Based on your feedack, we have now trained Qwen3.5-4B (newest available Qwen model at 4B scale) with the same recipe on LeanPRM.
Table R2: LeanPRM-Verifier on Qwen3-4B and Qwen3.5-4B, same recipe.
With fewer updates, the Qwen3.5-4B verifier is above the Qwen3-4B verifier on three of the five benchmarks and within 2 points on the other two.
W3 & Q3: Agentic baselines
We use the standard setting for math verifiers and RL reward models, in which the model reads a proof and answers once without tools. Training with tools is outside the scope of the paper, and we will state this in the limitations.
We did test a tool-using frontier model on our LeanPRM-Bench. Claude Fable 5.1 was given a Lean 4 compiler with the Mathlib version used to build LeanPRM (up to 10 calls per proof), and we ran the same harness without the tool as a control.
Table R3: Claude Fable 5.1 on LeanPRM-Bench (mean of three runs).
The compiler adds 4.0 points (95% CI +0.8 to +7.3). Claude called it on 26% of proofs, and the gain comes almost entirely from those problems. Our 4B verifier, with one pass and no tools, is 9 points below.
The reviewer also asks how the data could benefit agentic training. A step-level verifier can act as a reward model for provers or tell an agent where its first error is. In best-of-N on IMO-GradingBench, ranking candidates with our verifier selects a correct proof for 46.0% of problems at N=16, against 37.3% for the PRM800K verifier and 32.4% at random.
[1] Lightman et al. "Let's Verify Step by Step."