- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates a long-standing question in the reasoning community: Can formal reasoning capabilities benefit informal reasoning? Although the paper does not employ complex training methods, it provides a convincing answer to this question, supported by extensive experimental designs and insightful analyses of related research questions.
First, the paper develops a method for leveraging Lean-verified formal reasoning trajectories for training, demonstrating that even simple SFT can improve a model's reasoning capabilities. Furthermore, by comparing Lean reasoning trajectories with natural-language solution trajectories, the authors confirm that the improvement comes from the logical structure of formal reasoning, rather than merely from distillation using stronger models or privileged information.
Second, the stability study demonstrates that these findings hold across problems of varying difficulty levels and that the improvements can scale with larger pass@k sampling budgets.
Finally, the paper experimentally validates several interesting findings. For example, the proposed method not only improves model capabilities but can also provide a better initialization for subsequent RL training. The paper also compares the efficiency of using Lean during training versus inference, and the proposed approach, to some extent, addresses the long-standing challenges of high inference costs and low efficiency associated with formal reasoning.
Critical strengths
- The paper is very clearly written and well presented.
- The proposed method is simple yet effective.
- The experimental design and analysis are very well conducted, providing many interesting and insightful findings.
Critical weaknesses
-
The base models used for SFT in this paper are primarily from the Qwen3 series, rather than stronger models such as Qwen3.5 or Qwen3.8. We would be very interested to see whether the proposed method can still effectively improve the reasoning capabilities of state-of-the-art models. If so, this would make the experimental findings even more exciting.
-
The paper only explores using Lean reasoning trajectories for SFT, without considering other training methods. However, I do not believe this limitation undermines the soundness of the experimental design or the completeness of the experimental analysis.
Questions for authors
-
Have the authors tried applying the proposed method to SFT on stronger base models, such as Qwen3.5 or Qwen3.8, which have already undergone substantial post-training? We are curious whether the method can still provide meaningful improvements on these models.
-
Have the authors considered training with OPSD, using Lean proofs as privileged information for the teacher model? How would OPSD compare with the SFT approach proposed in this paper? Which method would perform better?
-
The training data primarily comes from Nemotron-Math-Proofs-v1, where Lean proofs are generated by Goedel-Prover. We wonder whether this dataset exhibits any domain or difficulty biases. As we know, formal provers with Lean verifiers tend to perform well in certain mathematical domains (e.g., linear algebra), while being relatively weaker in others (e.g., geometry and combinatorics). Does the training dataset exhibit such domain biases? Regarding difficulty, does the difficulty distribution of the training problems selected through this process differ from the distribution of problems that the student model itself is capable of solving? If these two types of bias exist, how do they affect model training and downstream reasoning performance?
4 comments
Q3. Domain and Difficulties Bias and Resulting Downstream Behavior
Domain bias and Lean-grounding’s generalization to sparse domains. We drew a random sample of 250 problems uniformly from the 10K training set and labeled the mathematical domains of each sample using Claude Opus 5.
Table 1: Domain distribution of our sampled training data.
| Domain | Percentage |
|---|---|
| Algebra | 64.4% |
| Number theory | 13.6% |
| Geometry | 9.2% |
| Analysis | 9.2% |
| Combinatorics | 2.0% |
| Linear algebra | 1.6% |
We observe that algebra accounts for the largest portion (64.4%), while combinatorics (2.0%) and linear algebra (1.6%) are the smallest. Table 2 splits AIME24--26 performance of the Qwen3-30B-A3B variants by sub-domain. On the domains with a large portion, Lean grounding improves over the base model by +17.1 points on algebra and +37.5 points on number theory, on the sparse domains, by +21.5 points on geometry and +18.8 points on combinatorics, showing that our method also generalizes to domains with few training examples.
Table 2: Pass@1 accuracy (%) by subdomain on AIME24—26 (90 problems) for Qwen3-30B-A3B. Our method generalizes to domains with sparse training data.
| Domain | #problems | Qwen3-30B-A3B | Teacher CoT (Student: Qwen3-30B-A3B) | Teacher Lean (Student: Qwen3-30B-A3B) |
|---|---|---|---|---|
| Algebra | 15 | 41.2 | 51.7 | 58.3 |
| Combinatorics | 30 | 20.2 | 32.5 | 39.0 |
| Geometry | 30 | 17.9 | 38.3 | 39.4 |
| Number theory | 15 | 24.6 | 46.2 | 62.1 |
| All | 90 | 23.7 | 39.9 | 46.2 |
Difficulty bias and consistent gains at every difficulty level.
In our work, we constructed the training set by sampling uniformly from Nemotron-Math-Proofs-v1 after decontamination, with no preference for difficulty. To measure the difficulty distribution, we use Claude Opus 5 to assign a difficulty score from 1 to 10 for 100 problems sampled uniformly from the 10K training set, for all 90 problems of AIME24--26, and for 50 problems sampled uniformly from IMO-AnswerBench. The judge sees only the problem statement.
Table 3: Difficulty distribution of the training data and of the benchmark problems, with Pass@1 accuracy (%) at each level. We also show the comparison between our Lean-grounded method with Teacher CoT and the Base model.
| Difficulty | Training data (%) | Benchmark (%) | Qwen3-30B-A3B (Base) | Qwen3-30B-A3B (Teacher CoT) | Qwen3-30B-A3B (Teacher Lean) |
|---|---|---|---|---|---|
| ≤3 | 15% | 1% | 100.0 | 100.0 | 100.0 |
| 4 | 15% | 13% | 67.7 | 83.7 | 96.5 |
| 5 | 29% | 26% | 24.3 | 37.2 | 47.1 |
| 6 | 34% | 30% | 6.2 | 18.8 | 20.8 |
| ≥7 | 7% | 30% | 17.9 | 19.6 | 21.7 |
| All | 100% | 100% | 23.1 | 32.8 | 38.3 |
We find that the difficulty distribution of sampled problems sits at the boundary of what the student can already solve, and the gains do not come from training on problems harder than the benchmark contains. Teacher Lean has the highest accuracy at every level, improving over the base model by +28.8 points at level 4, +22.8 at level 5, +14.6 at level 6, and +3.8 at level 7 and above. In our paper (Figure 3(b)), we have already found similar pattern of Lean-grounded method improves against all base solve rate on AIME24—26.
I thank the authors for the additional experiments, which provide stronger evidence of the method's effectiveness. Although OPSD does not outperform SFT, it still brings meaningful improvements.
The cross-domain and difficulty-level analyses are particularly insightful. They suggest that learning from Lean solutions benefits reasoning beyond algebra or simpler problems, generalizing across different domains and difficulty levels. I interpret this as evidence that LLMs can acquire a more rigorous, Lean-style reasoning pattern.
Overall, I believe this paper offers valuable insights, and I would like to raise my rating from 3 to 4.
Thanks for recognizing our analyses insightful
Thank you for finding our additional experiments insightful. This is really encouraging for us.
Thanks for your insightful suggestions. We have run our pipeline with a stronger student, added an OPSD variant in which the Lean proof serves as privileged information, and analyzed the domain and difficulty bias of the training data.
W1 & Q1: Results on stronger students
We applied Teacher Lean to Qwen3.5-9B, using the identical 10K Lean-grounded dataset generated by Qwen3.6 and SFT reciepe in our paper. We report Pass@16 for AIME24—26, Pass@8 for IMOAnswerBench on Qwen3.5-9B based models.
Lean grounding still improves the stronger student: +6.7 on AIME25, +10.0 on AIME26, and +2.6 on IMOAnswerBench.
W2&Q2: OPSD with Lean proofs as privileged information
We discussed OPD in Appendix A.4 as future work, and we have now run it. The Qwen3-8B student samples proof on-policy from the natural-language problem, and a frozen copy of the same model, conditioned on the verified Lean proof ρ as privileged information, scores the student’s own completion using soft labels (exactly OPSD), the objective is the exact full-vocabulary reverse KL between the two distributions.
On Qwen3-8B, we observe that SFT outperforms OPSD with Lean proof reference, while OPSD outperforms base models on AIME24, 25, 26, IMO-AnswerBench. A likely reason lies in the teacher: in OPSD, the teacher is a frozen copy of Qwen3-8B itself, whose ability to understand Lean 4 code is limited, whereas the SFT targets are produced by the stronger model (Qwen3.6) in understanding Lean.