acceptodds
Under review as a conference paper at ICLR 2027

Pythagoras-Prover: Advancing Efficient Formal Proving via Augmented Lean Formalisation

15 views

Abstract

LLM-based theorem provers have achieved strong performance on various benchmarks, but still require substantial training and inference compute due to scarce verified proof data and generating long reasoning traces. We introduce Pythagoras-Prover, a compute-efficient, open-source family of Lean theorem provers comprising 4B and 32B autoregressive models and a first diffusion-based proof of concept that iteratively refines proofs at inference time. To improve training efficiency, we apply curriculum supervised fine-tuning to a Lean-verified corpus stratified into easy, medium, and hard problems, progressing from simpler proofs to harder ones. Dynamic proof-reasoning filtering further preserves informative traces within an 8K-token context budget. We then introduce Augmented Lean Formalisation (ALF), which generates structured variants of verified statements and populates them through self-distillation, providing additional training signal without requiring every variant to be formally verified and reducing reliance on statement surface forms. Pythagoras-Prover-4B surpasses DeepSeek-Prover-V2-671B at pass@32 on MiniF2F-Test (86.1% versus 82.4%) with roughly 167× fewer parameters. Pythagoras-Prover-32B achieves state-of-the-art performance among open-source neural theorem provers, reaching 93.0% on MiniF2F-Test and solving 93 of 672 PutnamBench problems. We also release MiniF2F-ALF, a contamination-sensitive benchmark of ALF-generated perturbations. Together, these results demonstrate that strong Lean theorem proving is achievable under practical compute budgets without relying exclusively on frontier-scale models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Foundation Models, Large Language Models, Diffusion Language Models, Theorem Proving, Mathematical Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jHEuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper is well-written in structure and clearly describe the training process and data used. The main contributions are:

  1. Develops 4B and 32B prover models that achieve good performance among open-source models.
  2. Proposes an efficient data augmentation pipeline, ALF.
  3. Introduces the first diffusion-based theorem prover, featuring tactic-level masking.
Critical strengths
  1. Tactic-level masking for the diffusion-based prover is a meaningful methodological innovation.
  2. The paper provides a detailed analysis of model performance.
Critical weaknesses
  1. Limited novelty in the AR training pipeline. The training procedure for the autoregressive models is not particularly novel. Most of the methodological novelty lies in the ALF data augmentation pipeline. However, there is no clear ablation comparing training with ALF-generated data against training with general data under the same training procedure.

  2. Potential reduction in training data difficulty. Using an existing prover to verify formalized problems may reduce the difficulty of the retained training data, potentially preventing the resulting model from surpassing existing provers.

  3. Limited performance improvements. Pythagoras-Prover-32B achieves 89.8%, compared with 88.1% for Goedel-Prover-V2-32B. Furthermore, there is no performance difference in the "+Self Correction" setting.

Questions for authors
  1. Ablation of ALF. As I understand it, the training pipeline consists of SFT → RL → ALF self-distillation. Therefore, the ALF ablation in Table 4 should compare the model after RL with the final model, rather than comparing the model after SFT with the final model. Could the authors clarify this experimental design?

  2. Effectiveness of continued SFT through self-distillation. The continued SFT process is not entirely intuitive to me. Specifically, the post-RL Pythagoras-Prover generates one candidate proof per statement, and these generated proofs are then used for continued SFT. It seems that these generated problems are already within the model's capabilities, yet continued SFT still improves performance. Could the authors provide additional experiments or insights to explain this phenomenon?

  3. Difficulty of the training data. In Section 2.1, the authors mention using Goedel-Prover-v2-32B to generate proofs and filter formalized statements based on proof correctness. Does this process reduce the difficulty of the retained training data?

  4. Ablation of tactic-level masking. For the diffusion-based prover, what are the benefits of tactic-level masking compared with standard token-level masking? Is there an ablation study demonstrating the effectiveness of this design?

5 comments
AuthorsOpenReview ↗

Responses to Reviewer jHEu (Part 1)

We thank Reviewer jHEu for the constructive feedback, as well as for recognizing the strong performance of Pythagoras-Prover 4B and 32B among open-source models and our introduction of the first diffusion-based theorem-proving model.

[W1]. Limited novelty in the AR training pipeline. The training procedure for the autoregressive models is not particularly novel. Most of the methodological novelty lies in the ALF data augmentation pipeline. However, there is no clear ablation comparing training with ALF-generated data against training with general data under the same training procedure.

Our contributions extend beyond ALF, which we address below with the synthetic data-generation pipeline, dynamic proof-reasoning filtering, and the extension of this recipe. Specifically:

  1. A synthetic data-generation pipeline that combines ALF with rubric-guided distillation to construct difficulty tiers for curriculum learning;
  2. Dynamic proof-reasoning filtering, which preserves informative reasoning traces within an 8K-token budget; and
  3. The extension of this recipe to diffusion-based theorem proving. Together, these yield a training corpus of nearly 3M instances (~0.8M Lean-verified seed and ~2M ALF), mitigating the data scarcity that constrains existing provers. Regarding an ablation against general data, to our knowledge no public Lean corpus provides structured statement variants at comparable scale, which is precisely the gap ALF addresses. Tables 4 and 5 isolate ALF’s contribution instead: ALF self-distillation improves over the SFT baseline at both 4B and 32B.
Model variantMiniF2F-Test (Pass@32)
SFT79.10
SFT + fine-tuning on ALF data (new ablation)82.19
SFT + RL81.69
Full pipeline (original)86.07

To isolate the effect of the mutations themselves, we further fine-tuned the ALF dataset directly based on the base model, yielding 82.19% (as shown in Table 5, which we discuss in Appendix C.4). This shows the strength and high quality of our dataset. Thank you again for raising this concern. We have included the clarification in Appendix G.

[W2.] Potential reduction in training data difficulty. Using an existing prover to verify formalized problems may reduce the difficulty of the retained training data, potentially preventing the resulting model from surpassing existing provers.

Thank you for raising this concern. We acknowledge that using an existing prover for proof generation and verification may limit the difficulty of the retained training instances. However, distillation from existing provers is a common practice among open-source theorem provers and LLMs [1-4]. Furthermore, the terms of service of closed-source models often restrict their use for training competing models, while manual formalisation at this scale is prohibitively expensive, making distillation from existing open-source provers a practical approach.

Importantly, our experimental results demonstrate that this approach does not impose an upper bound on the resulting model's performance. As shown in Table 1, Pythagoras-Prover-32B surpasses Goedel-Prover-V2-32B, the very prover used for data generation, achieving 89.8% versus 88.1% at pass@32. At their respective best reported sampling budgets, Pythagoras-Prover-32B further achieves 93.0% at pass@2048, exceeding Goedel-Prover-V2-32B's 92.2% at pass@8192 with only one-quarter of the sampling budget. These results indicate that although the initial data generation is constrained by the existing prover's capabilities, this does not prevent the final model from surpassing its performance.

[1] Lin, Yong, et al. "Goedel-prover-v2: Scaling formal theorem proving with scaffolded data synthesis and self-correction." (2026).

[2] Lee, Sunbowen, et al. "Distillation quantification for large language models." (2025).

[3] Lin, Yong, et al. "Goedel-prover: A frontier model for open-source automated theorem proving." (2025).

[4] Ren, Z. Z., et al. "Deepseek-prover-v2: Advancing formal mathematical reasoning via reinforcement learning for subgoal decomposition." (2025).

AuthorsOpenReview ↗

Responses to Reviewer jHEu (Part 2)

[W3.] Limited performance improvements. Pythagoras-Prover-32B achieves 89.8%, compared with 88.1% for Goedel-Prover-V2-32B. Furthermore, there is no performance difference in the "+Self Correction" setting.

Thank you for bringing this to our attention. We would like to highlight that, as shown in Table 1, Pythagoras-Prover-32B consistently outperforms Goedel-Prover-V2-32B at every shared sampling budget without self-correction, achieving 89.8% versus 88.1% at pass@32.

Furthermore, although Pythagoras-Prover-32B matches the performance of Goedel-Prover-V2-32B with self-correction at pass@1024 (92.6% versus 92.6%), our model achieves this result using standard restart sampling alone, without requiring iterative inference-time repair. At pass@2048, Pythagoras-Prover-32B further reaches 93.0%, surpassing Goedel-Prover-V2-32B's best reported result of 92.2% at pass@8192, while using only one-quarter of the sampling budget. This advantage also extends to PutnamBench, where, at comparable estimated Effective Token Complexity (ETC), Pythagoras-Prover solves 88 problems at pass@1024, compared with 86 for Goedel-Prover-V2 with self-correction at pass@184, as detailed in Appendix F.

Importantly, as discussed in Section 5.2, MiniF2F-Test is approaching saturation, with 83.2% of problems solved by all four evaluated provers. Consequently, the remaining room for improvement is relatively limited, making these performance gains meaningful, particularly given that our model achieves them without additional inference-time self-correction.

[Q1.] Ablation of ALF. As I understand it, the training pipeline consists of SFT → RL → ALF self-distillation. Therefore, the ALF ablation in Table 4 should compare the model after RL with the final model, rather than comparing the model after SFT with the final model. Could the authors clarify this experimental design?

To our knowledge, no existing Lean corpus provides structured statement variants at ALF’s scale (~2M instances), hence no external dataset allows a direct comparison. Self-distillation on unchanged statements cannot match ALF’s data volume by construction. Unchanged statements are bounded by the Lean-verified seed corpus (~0.8M), and the additional instances exist only because ALF mutates each statement into formally distinct variants. Increasing volume without mutation would instead require repeatedly sampling proofs for the same statements, which the model has already been trained on during SFT. Hence, to address your concern, we ran the matched control in which we finetuned the base 4B model on the ~2M datapoints of the ALF dataset.

The Table below (similar to the table attached on addressing W1) compares the performance of SFT+finetuning on ALF data with that achieved at each stage of our original 4B training pipeline on the original MiniF2F-Test, whose statements are not mutated (thus they do not depend on ALF-style evaluation).

Model variantMiniF2F-Test (Pass@32)
SFT79.10
SFT + fine-tuning on ALF data (new ablation)82.19
SFT + RL81.69
Full pipeline (original)86.07

As we can see, ALF contributes +4.38 points over the post-RL checkpoint, showing the strength of applying ALF after the conventional post-RL methodology and +3.09 over the SFT checkpoint without RL, yielding a better result than the post-RL checkpoint. Thus, the diversity introduced by ALF improves the model both with and without RL, and the two stages are complementary, with the full training pipeline gaining +6.97 over SFT, and thus showing that the diversity introduced by the dataset helps in further improving the model.

We have included these results in Table 5 and discuss them in Appendix C.4, marked in blue.

AuthorsOpenReview ↗

Responses to Reviewer jHEu (Part 3)

[Q2.] Effectiveness of continued SFT through self-distillation. The continued SFT process is not entirely intuitive to me. Specifically, the post-RL Pythagoras-Prover generates one candidate proof per statement, and these generated proofs are then used for continued SFT. It seems that these generated problems are already within the model's capabilities, yet continued SFT still improves performance. Could the authors provide additional experiments or insights to explain this phenomenon?

Thank you for raising this question. We would like to clarify that the final training stage is self-distillation rather than conventional SFT, where the model is trained on the ALF-augmented dataset instead of reusing the original RL training data.

As demonstrated by Zhang et al. (2026) [5], self-distillation can improve model performance even when the generated samples fall within the model's existing capabilities. This is because the improvement primarily comes from reshaping the model's output distribution rather than acquiring new knowledge. Specifically, self-distillation encourages the model to assign greater confidence to precise outputs, such as tactic syntax and lemma names, while preserving diversity where multiple proof strategies remain viable. Consequently, the model learns to express its existing proof-solving capabilities more reliably during inference, explaining why continued SFT can improve performance despite the absence of additional external supervision.

We have included the explanation in the Appendix C.5 of the revised manuscript.

[5] Zhang, Ruixiang, et al. "Embarrassingly simple self-distillation improves code generation." arXiv preprint arXiv:2604.01193 (2026).

[Q3.] Difficulty of the training data. In Section 2.1, the authors mention using Goedel-Prover-v2-32B to generate proofs and filter formalized statements based on proof correctness. Does this process reduce the difficulty of the retained training data?

We acknowledge that any generate-and-verify pipeline inherently retains only those instances that fall within the generating model's capabilities. However, this limitation applies equally to existing open-source theorem provers and approaches that distill training data from frontier models.

To mitigate this limitation, our data construction incorporates three complementary strategies:

  • Hard-problem sourcing: We construct the hard tier from competition-level mathematical problems, which require more challenging and non-routine reasoning.
  • Increased sampling budget: We increase the sampling budget to (n=4) for hard-tier problems, compared with (n=2) for easy and medium problems, to improve the coverage of difficult instances. We also reserve the original Big-Math hard problems for RL, rather than including them in SFT.
  • ALF augmentation: We further expand the training distribution through ALF-generated statement variants, introducing additional structural diversity beyond the original Lean-verified corpus.

Importantly, as shown in Table 1, Pythagoras-Prover-32B surpasses Goedel-Prover-V2-32B, the model used for proof generation and filtering. This demonstrates that, although the initial data filtering is constrained by the generator's capabilities, it does not impose a corresponding upper bound on the performance of the final trained prover.

[Q4.] Ablation of tactic-level masking. For the diffusion-based prover, what are the benefits of tactic-level masking compared with standard token-level masking? Is there an ablation study demonstrating the effectiveness of this design?

Thank you for raising this question. The key distinction is that standard token-level masking allows masked tokens to be predicted from visible tokens within the same tactic, whereas our tactic-level masking requires the model to reconstruct complete proof steps. This aligns the denoising objective with the complete Lean tactics committed during inference, as described in Section 2.4.

Initial Observation: In our experiments, we observed that the diffusion model fails to generate Lean-valid proofs without tactic-level masking. As further discussed in Appendix C.8, diffusion-based theorem proving faces three primary challenges: (i) Sparse supervision under conventional random masking, (ii) Limited stable training-context lengths, (iii) Scarcity of diffusion-suitable Lean training data. Therefore, tactic-level masking is a necessary component for successful Lean proof generation in our setting, rather than merely providing an incremental performance improvement.

Our approach addresses the first challenge through tactic-level masking and the third through ALF self-distillation. Without these interventions, the diffusion model fails to produce valid Lean proofs in our experiments, while the context-length limitation remains an open challenge.

We are happy to address any further concerns. Thank you again for the insightful feedback.

Reviewer_xwRkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents 4B and 32B Lean provers trained through curriculum SFT, GRPO, and self-distillation. Its main contribution, Augmented Lean Formalisation (ALF), generates statement variants and trains on generated proofs without exhaustive verification. An audit reports that 87.8% of 2,000 sampled pairs compile. The 4B model reaches 86.1% on MiniF2F-Test at pass@32. The 32B model reaches 93.0% at pass@2048 and solves 93 PutnamBench problems. The paper also introduces MiniF2F-ALF and a diffusion prover using tactic-level masking.

Critical strengths

Competitive performance at small model sizes. The 4B model exceeds the reported Goedel-Prover-V2-8B result at pass@32: 86.1% versus 84.6% (Table 1). The 32B model also performs strongly on PutnamBench. These results suggest practical value for users who face model memory constraints.

The diffusion experiment provides a useful feasibility result. The diffusion prover reaches approximately 63.2% at pass@32. Under matched training and evaluation context limits, the autoregressive counterpart reaches 74.6% (Figure 4). This comparison provides an informative starting point for diffusion-based formal proving.

Critical weaknesses

The experiments do not isolate the central training contributions, and their descriptions conflict. Table 4 compares SFT with SFT plus ALF self-distillation. It does not compare ALF against additional training or self-distillation without mutations under a matched token budget. It also omits a separate post-RL result, despite the stated SFT→RL→ALF pipeline. Thus, the reported gains cannot be assigned specifically to ALF. The proof-reasoning filter has two different definitions. Section 2.2 describes length-based retention, reasoning removal, and example rejection. Appendix C.5 instead describes rejecting traces whose reasoning conflicts with the proof. Table 5 therefore does not clearly validate the method described in the main text. These issues obstruct both attribution and reproduction.

The compute comparisons do not establish the claimed efficiency advantage. On PutnamBench, the headline comparison uses 2,048 attempts versus 184 self-correction attempts. Appendix F addresses this difference through an attention-work proxy, but that proxy does not measure total inference cost. Its accounting also needs reconciliation. Table 6’s three Goedel output lengths sum to approximately 63,842 tokens. Appendix F charges these lengths to every attempt. However, Goedel’s evaluation section specifies 40,000 total tokens for self-correction. Different protocols or conditional averaging could explain this discrepancy, but the comparison must establish which applies. Goedel-Prover-V2, §3.2 Similarly, Section 5.3’s accuracy × tokens/second metric does not establish solved problems per GPU-second without comparable output lengths. The paper acknowledges this condition but does not demonstrate it. Total data-generation and training costs are also absent, despite the emphasis on compute efficiency.

Questions for authors
  1. Which checkpoints appear in Table 4, and where does RL enter each comparison? Does ALF outperform self-distillation without mutations under matched training tokens?
  2. Which proof-reasoning filter produced the reported models: the length-based rule in Section 2.2 or the consistency filter in Appendix C.5? Please reconcile the 4B SFT baseline in Table 4 with the filtered result in Table 5.
  3. How do Table 6’s Goedel token counts relate to the published 40K total budget? Were later-round means conditional on reaching those rounds? Can you provide total generation and verification costs for the compared runs?
  4. Does every MiniF2F-ALF statement have a complete verified reference proof? Appendix B.3 describes statement well-formedness checks, which do not establish provability. How did you control difficulty changes when selecting the two most semantically divergent mutations? This determines whether accuracy drops support conclusions about memorization or reflect changed mathematical tasks.
8 comments
AuthorsOpenReview ↗

Responses to Reviewer xwRk (Part 1)

We thank Reviewer xwRk for the constructive feedback, as well as for recognizing that our 4B model exceeds the reported baselines and that the new diffusion experiment provides a useful feasibility result.

[W1.a] The experiments do not isolate the central training contributions, and their descriptions conflict. Table 4 compares SFT with SFT plus ALF self-distillation. It does not compare ALF against additional training or self-distillation without mutations under a matched token budget. It also omits a separate post-RL result, despite the stated SFT→RL→ALF pipeline. Thus, the reported gains cannot be assigned specifically to ALF.

[Q1] Which checkpoints appear in Table 4, and where does RL enter each comparison? Does ALF outperform self-distillation without mutations under matched training tokens?

We thank the reviewer for raising this point. We have added the post-RL result and a new ablation of ALF without RL in Table 5, discussed in Appendix C.4 (marked in blue).

Model variantMiniF2F-Test (Pass@32)
SFT79.10
SFT + fine-tuning on ALF data (new ablation)82.19
SFT + RL81.69
Full pipeline (original)86.07
  • Regarding which checkpoints appear in Table 4, we have clarified that the final row for each model size is the full SFT→RL→ALF model.
  • Regarding the post-RL result, we report it in the Table below (SFT+RL), where we can see that RL contributes 2.59 wrt the SFT baseline, and ALF manages to contribute +4.38 points over the post-RL checkpoint.
  • Regarding the comparison wrt additional training or self-distillation without mutations under a matched token budget, there exists no open Lean corpus which compares with ALF’s scale (~2M instances), hence no external dataset allows a direct comparison. On the other hand, increasing volume without mutation would instead require repeatedly sampling proofs for the same statements, which the model has already been trained on during SFT. To overcome such limitations and still show the effectiveness of distilling using our ALF dataset we directly finetuned our base 4B model on the ~2M datapoints of the ALF dataset without doing RL first. This is again reported in the Table below (row: SFT+finetuning on ALF data (new ablation)), which shows that ALF again improves the model, this time by +3.09 points, thus yielding a better result than the post-RL checkpoint.

[W1.b] The proof-reasoning filter has two different definitions. Section 2.2 describes length-based retention, reasoning removal, and example rejection. Appendix C.5 instead describes rejecting traces whose reasoning conflicts with the proof. Table 5 therefore does not clearly validate the method described in the main text. These issues obstruct both attribution and reproduction.

Thank you for pointing out this inconsistency! We would like to clarify that the correct proof-reasoning filtering procedure is the length-based dynamic filter defined in Section 2.2 and Figure 1, which operates under an 8,192-token context budget.

Specifically, the filtering procedure follows three conditions:

  1. When both the reasoning trace and proof fit within the context budget, both are retained.
  2. When the combined sequence exceeds the budget but the proof alone fits, only the proof is retained.
  3. When the proof alone exceeds the budget, the instance is discarded.

We have clarified the definition in Section 2.2 and aligned the description in what is now Appendix C.7 (Appendix C.5 in the original submission) accordingly, removing the inconsistency to ensure consistency and reproducibility throughout the script. Thank you again for pointing the inconsistency

AuthorsOpenReview ↗

Responses to Reviewer xwRk (Part 2)

[W2.a] The compute comparisons do not establish the claimed efficiency advantage. On PutnamBench, the headline comparison uses 2,048 attempts versus 184 self-correction attempts. Appendix F addresses this difference through an attention-work proxy, but that proxy does not measure total inference cost. Its accounting also needs reconciliation. Table 6’s three Goedel output lengths sum to approximately 63,842 tokens. Appendix F charges these lengths to every attempt. However, Goedel’s evaluation section specifies 40,000 total tokens for self-correction. Different protocols or conditional averaging could explain this discrepancy, but the comparison must establish which applies.

[Q3] How do Table 6’s Goedel token counts relate to the published 40K total budget? Were later-round means conditional on reaching those rounds? Can you provide total generation and verification costs for the compared runs?

Thank you for bringing this question. We would like to clarify the following two points:

  1. Compute comparison at matched ETC. We acknowledge that our headline PutnamBench result uses different sampling budgets (pass@2048 versus pass@184). To address this, Appendix F additionally provides a comparison at approximately matched Effective Token Complexity (ETC). Specifically, Pythagoras-Prover at pass@1024 (ETC ≈ 1.85 × 10¹¹) solves 88 problems, compared with 86 for Goedel-Prover-V2 at pass@184 (ETC ≈ 1.95 × 10¹¹). We have revised Appendix F to further clarify this (all changes are marked in blue). We acknowledge that ETC is an idealised attention-work proxy rather than a direct measurement of total inference cost.

  2. Clarification of the 40K-token budget. The 40,000-token limit reported by Goedel-Prover-V2 [1] applies to each generation round rather than the entire self-correction process, as the rounds are evaluated separately. Table 7 therefore reports the average generated tokens for each round. Importantly:

Each round remains within the 40K-token limit (e.g., approximately 8.3K input + 23.8K output tokens in round 3). The approximately 63.8K generated tokens represent the cumulative output across three rounds, rather than a single generation exceeding the budget.

We have further clarified the per-round token accounting and the limitations of ETC in Appendix F (Table 7).

[1] Lin, Yong, et al. "Goedel-prover-v2: Scaling formal theorem proving with scaffolded data synthesis and self-correction." International Conference on Learning Representations. Vol. 2026. 2026.

W2.b Section 5.3’s accuracy × tokens/second metric does not establish solved problems per GPU-second without comparable output lengths.

We would like to clarify our comparison controls for the training context, evaluation budget, and hardware configuration. Specifically, both models are evaluated under the same 8,192-token output budget on an identical 8 H100 node with batch size 4, using the same ALF self-distillation corpus. The autoregressive model is also retrained at the diffusion model's 4,096-token training context to ensure a fair comparison.

Under this setting, the accuracy gap narrows from 22.8 to 11.4 percentage points (74.6% versus 63.2%), while diffusion achieves 2.58x higher decoding throughput (10.56 versus 4.10 output tokens per second).

To account for the accuracy-throughput trade-off, we estimate the cost of generating an 8,192-token attempt:

  1. Generation time: Diffusion requires approximately 776 s per attempt, compared with 1,998 s for autoregressive decoding.
  2. Accuracy-adjusted throughput: Diffusion achieves an estimated 2.18x advantage (6.68 versus 3.06), despite its lower pass@32 accuracy. The advantage remains 1.89x when compared against the full-context autoregressive model.

We acknowledge that these estimates assume equal realised output lengths and exclude Lean-verification overhead. Therefore, we present them as a throughput-based efficiency estimate rather than a direct measurement of solved problems per GPU-second. We have clarified these assumptions in Section 5.3.

AuthorsOpenReview ↗

Responses to Reviewer xwRk (Part 3)

[W2.c] The paper acknowledges this condition but does not demonstrate it. Total data-generation and training costs are also absent, despite the emphasis on compute efficiency.

Thank you for bringing this to our attention! Our efficiency claims concern training and inference time. Our aim is to release a family of models that the wider community can use for research, including groups with limited compute. The competitive 4B model and the diffusion-based prover reflect this aim: Pythagoras-Prover--4B surpasses DeepSeek-Prover-V2-671B at matched pass@32 with roughly 167x fewer parameters.

Model efficiency is separate from data-generation efficiency. The data pipeline is a one-off cost: once generated, the corpus can be reused by others without incurring it again. For transparency, we detail the data-generation cost below.

  1. Data generation. The pipeline relies predominantly on open models of at most 32B parameters: Goedel-Prover-V2-32B for proof generation, and Qwen3.6-27B and Qwen3-30B for variant generation. Qwen3-235B is the only larger component.

  2. Training. Training cost is limited by LoRA-based SFT within an 8K-token context.

  3. Compute. The full training of all components uses 16 GPUs.

Last but not least, we will be releasing all generated data upon acceptance. We have added these details in the new Section G.

[Q2.] Which proof-reasoning filter produced the reported models: the length-based rule in Section 2.2 or the consistency filter in Appendix C.5? Please reconcile the 4B SFT baseline in Table 4 with the filtered result in Table 5.

Thank you for raising this concern. We would like to clarify that all reported models are trained using the dynamic proof-reasoning filter described in Section 2.2 and Figure 1, which operates under an 8,192-token context budget.

Specifically, the filtering procedure follows three conditions: When both the reasoning trace and formal proof fit within the 8,192-token budget, both are retained. When the combined sequence exceeds the budget but the proof alone fits, only the proof is retained. When the proof alone exceeds the budget, the instance is discarded.

Appendix C.7 (Appendix C.5 in the original submission) is intended to evaluate this same filtering procedure. We acknowledge that its description of consistency-based filtering introduces ambiguity, and have clarified the wording to align with the length-based procedure in Section 2.2 to clarify all the experiments are conducted with dynamic proof-reasoning filters unless specified.

Regarding the discrepancy between Tables 4 and 6 (note that Table 5 from the original submission is now Table 6), the SFT-only baselines in Table 4 correspond to the dynamically filtered setting in Table 6. The 32B result is consistent at 84.02%, whereas the 4B baseline requires correction from the unfiltered value of 79.10% to the filtered value of 80.70%, assuming Table 6 reports the intended result. We hope this addresses the concern, and thank you for raising this out!

Reviewer_92jvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Pythagoras-Prover trains Lean provers using a verified corpus, curriculum learning, and self-distillation on mutated statements. It also tests tactic-level diffusion, which generates tokens faster but proves fewer problems.

Critical strengths

The 4B model reaches 86.1% MiniF2F pass@32, a strong result for its size. The data pipeline is described in detail, including decontamination and an audit of unverified augmented examples. Controlling training-context length in the diffusion comparison is also helpful. I also give weight to the decision to report the diffusion accuracy loss rather than presenting throughput in isolation. The prover results and documented data construction are the main reasons I lean toward acceptance, even with reservations about the efficiency framing.

Critical weaknesses

I am not convinced by the compute-efficiency claim. Appendix F measures an idealized attention proxy. Multiplying pass@32 by token throughput also leaves out attempt lengths and verification costs. The diffusion model generates tokens faster, but the accuracy loss could outweigh that saving in actual proof search.

The ALF ablation compares the full recipe with SFT. It does not show whether mutating statements helps more than ordinary self-distillation with the same amount of data. This is particularly relevant because the perturbed evaluation also uses ALF-style mutations. The appendix does acknowledge limitations of the compute proxy. My concern is with using it to support the broader efficiency claim. A controlled wall-time comparison could change that assessment without requiring a new model or a larger benchmark.

Questions for authors

How many problems does each method solve per measured GPU-hour, including verification, under the same stopping rule? How does ALF compare with self-distillation on unchanged statements at matched data and training budgets? For the timing comparison, please state how unsuccessful attempts and early stopping are counted.

3 comments
AuthorsOpenReview ↗

Responses to Reviewer 92jv (Part 1)

We sincerely thank Reviewer 92jv for their positive assessment of our work and their recommendation toward acceptance. We appreciate the recognition of Pythagoras-4B’s strong performance given its compact size, as well as the acknowledgment of our detailed pipeline description. Below, we offer clarifications and additional results that we hope will address the remaining concerns.

[W1] I am not convinced by the compute-efficiency claim. Appendix F measures an idealized attention proxy. Multiplying pass@32 by token throughput also leaves out attempt lengths and verification costs.

Thank you bringing this to our attention. Our efficiency claim does not rest only on Appendix F. Effective Token Complexity (ETC) only shows that restart sampling and self-correction operate at comparable attention cost. The claim itself mostly relies on directly measurable quantities:

  1. Pythagoras-Prover-4B surpasses DeepSeek-Prover-V2-671B at matched pass@32 with roughly 167× fewer parameters, and
  2. Pythagoras-Prover-32B exceeds Goedel-Prover-V2-32B’s pass@8192 result at pass@2048.

We have clarified this further in Appendix G. Thank you again for bringing this to our attention.

[W2] The diffusion model generates tokens faster, but the accuracy loss could outweigh that saving in actual proof search.

Thank you for raising this concern. Please notice that attempt length and verification cost are controlled in the diffusion comparison. Indeed, both the autoregressive models and the diffusion-based one generate under the same 8,192-token budget, and perform the same number of Lean verifications (32 per problem).

From the measured throughput on the same 8× H100 node (10.56 vs. 4.10 output tokens per second for Pythagoras-ProverDiffusion and Pythagoras-Prover, respectively), a full-length 8,192-token attempt takes approximately 776 s for Pythagoras-Prover-Diffusion versus 1,998 s for Pythagoras-Prover. Diffusion therefore completes about 82 attempts in the time autoregressive decoding completes 32. Per unit of GPU time, diffusion thus solves 2.18× more problems despite its 11.4-point lower pass@32 accuracy.

At a fixed sampling budget, autoregressive proving remains more accurate, as shown in Figure 4 (Experimental Results) and Section 5.3. We have clarified this further at Section 5.3 and marked the new explanation in blue.

[W3] The ALF ablation compares the full recipe with SFT. It does not show whether mutating statements helps more than ordinary self-distillation with the same amount of data. This is particularly relevant because the perturbed evaluation also uses ALF-style mutations. The appendix does acknowledge limitations of the compute proxy.

[Q2] How does ALF compare with self-distillation on unchanged statements at matched data and training budgets?

To our knowledge, no existing Lean corpus provides structured statement variants at ALF’s scale (~2M instances), hence no external dataset allows a direct comparison. Self-distillation on unchanged statements cannot match ALF’s data volume by construction. Unchanged statements are bounded by the Lean-verified seed corpus (~0.8M), and the additional instances exist only because ALF mutates each statement into formally distinct variants. Increasing volume without mutation would instead require repeatedly sampling proofs for the same statements, which the model has already been trained on during SFT. Hence, to address your concern, we ran the matched control in which we finetuned the base 4B model on the ~2M datapoints of the ALF dataset.

The Table below compares the performance of SFT+finetuning on ALF data with that achieved at each stage of our original 4B training pipeline on the original MiniF2F-Test, whose statements are not mutated (thus they do not depend on ALF-style evaluation).

Model variantMiniF2F-Test (Pass@32)
SFT79.10
SFT + fine-tuning on ALF data (new ablation)82.19
SFT + RL81.69
Full pipeline (original)86.07

As we can see, ALF contributes +4.38 points over the post-RL checkpoint, showing the strength of applying ALF after the conventional post-RL methodology and +3.09 over the SFT checkpoint without RL, yielding a better result than the post-RL checkpoint. Thus, the diversity introduced by ALF improves the model both with and without RL, and the two stages are complementary, with the full training pipeline gaining +6.97 over SFT. We have included these results in Table 5 and discuss them in Appendix C.4, marked in blue.

AuthorsOpenReview ↗

Responses to Reviewer 92jv (Part 2)

[W4] My concern is with using it to support the broader efficiency claim. A controlled wall-time comparison could change that assessment without requiring a new model or a larger benchmark.

[Q1] How many problems does each method solve per measured GPU-hour, including verification, under the same stopping rule? For the timing comparison, please state how unsuccessful attempts and early stopping are counted.

Thank you for raising this concern! We have evaluated all the checkpoints of our model in identical hardware (8× H100) with vLLM inference. As they share the same model size, architecture and maximum token of 40,000, differences in computational cost between the variants are negligible. We have included this detail at the bottom of Appendix C.4.

Due to the time constraint, we evaluated only Pythagoras-Prover-4B, and we will include in the final version of the paper in case of acceptance the evaluation for the other variants. Pythagoras-Prover-4B solves 182 of 244 MiniF2F-Test problems (74.6%) in 4 GPU-hours. The verification of all the problems took 0.5 hours of wall-clock time, corresponding to roughly 32 CPU-hours.The number of tokens was capped at the maximum token length of the Qwen3 series (40k). All 32 attempts per problem are generated and verified even after a successful proof is found. Unsuccessful attempts, including those that reach the maximum generation length without producing a complete proof, are counted at their full generated length. The same rule applies to both methods, so neither is credited for terminating once a proof is found. We have clarified this further at Appendix B.2.

We hope our responses address your concerns. If there are any further questions or suggestions, please don't hesitate to let us know!

AuthorsOpenReview ↗

Following up on our response to your concerns

Dear Reviewer 92jv,

We understand the discussion timeline may be inconvenient for your busy schedule, but we would love to continue our discussion before the discussion period closes.

We hope we’ve been able to address your questions and concerns so far! Please don’t hesitate to let us know if there’s anything else we can help clarify or discuss.

Thank you again for recognizing the importance of our work, for leaning towards acceptance, and for your feedback and comments. They were really helpful!

Warm regards, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank all reviewers for their valuable feedback. We are truly grateful for their constructive comments and their recognition of our Pythagoras-Prover Model.

Main contributions of the paper

  1. We introduce Pythagoras-Prover, a family of theorem-proving models at 4B and 32B scales (92jv, xwRk, jHEu), with Pythagoras-Prover 4B achieving 86.1% on MiniF2F at pass@32 (92jv, xwRk), outperforming DeepSeek-Prover-671B despite being 167× smaller.
  2. Pythagoras-Prover’s strength lies in its synthetic data generation, followed by curriculum-learning SFT, GRPO, and self-distillation through Augmented Lean Formalisation (ALF) on mutated statements (jHEu, 92jv, xwRk).
  3. We further introduce Pythagoras-Prover-Diffusion, the first diffusion-based theorem prover to date, trained with tactic-level diffusion. It generates tokens faster but proves fewer problems (jHEu, 92jv, xwRk).
  4. Finally, we introduce MiniF2F-ALF (xwRk), a perturbed and human-annotated version of MiniF2F that reduces saturation and contamination while ensuring the correctness of the dataset.

Main Results, Analysis and Presentation

  1. The key strength of Pythagoras-Prover is its strong performance at a small model size (92jv). We further report the accuracy cost of Pythagoras-Prover-Diffusion alongside its throughput rather than presenting throughput in isolation, noted by Reviewer 92jv.
  2. Empirically, Pythagoras-Prover-4B exceeds the current state-of-the-art result reported by Goedel-Prover-V2-8B at pass@32 (86.1% vs. 84.6%, Table 1), as well as DeepSeek-Prover-671B (xwRk), a model with 167x smaller. The 32B model also performs strongly on PutnamBench (xwRk).
  3. The data pipeline is also well-documented (92jv), including decontamination and an audit of unverified ALF data, together with thorough performance analysis (jHEu).
  4. Lastly, we included a thorough analysis on the diffusion model’s performance and efficiency under matched training and evaluation context limits (xwRk).

Changes in the revision

Following the reviewers’ feedback, we have incorporated the additional experimental results and clarifications from the rebuttal into the revised paper.

  1. In response to Reviewers 92jv, xwRk, and jHEu, we added a more thorough stage-wise decomposition (SFT, SFT+RL, SFT on ALF data only, full pipeline), showing the improvement contributed by each stage. (Appendix C.4)
  2. In response to Reviewers 92jv, xwRk, and jHEu, we added a comparison and analysis under identical training and inference configurations, with measured solves per GPU-hour including Lean verification and an explicit counting protocol (Section 5.3, Appendix B). We also add an ablation of tactic-level versus token-level masking (jHEu). (Section 5.3, Appendix B).
  3. In response to Reviewers jHEu, 92jv, and xwRk, we added a new Appendix Section G that clarifies the training novelty and efficiency claims of Pythagoras-Prover.
  4. In response to Reviewer xwRk, we further clarified the robustness of MiniF2F-ALF, including the LLM-based semantic verification and manual verification used to ensure its correctness. (Appendix B.3)
  5. We have also addressed the additional clarifications towards the concerns raised by Reviewers 92jv, xwRk, and jHEu in the main manuscript.

Most concerns were shared across reviewers and have been acknowledged by Reviewer jHEu with a score raised. We sincerely thank the reviewers again for their recognition and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.