- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces daVinci-Prover, a framework for interactive theorem proving that combines a 33.7B-token mid-training corpus with approximately 36K agent trajectories for supervised fine-tuning. The corpus covers proof decomposition, search, revision, self-verification, and tool interactions. It also introduces a Lean-based agent harness and reports consistent improvements over SFT-only baselines across multiple models and theorem-proving benchmarks.
Critical strengths
-
A well-motivated training recipe for interactive theorem proving. The paper combines mathematical skill supervision—decomposition, search, revision, and self-verification—with environment-native agent trajectories. This design directly addresses the gap between learning from static proofs and interacting with a verifier during proof construction.
-
Broad empirical evaluation with consistent downstream gains. The approach is evaluated across multiple model backbones, model sizes, and theorem-proving benchmarks. Improvements over the corresponding SFT-only baselines, together with component ablations and data-scaling experiments, provide evidence for the practical utility of the overall training recipe.
Critical weaknesses
-
The contribution of the synthesis strategy is confounded with teacher-model capability. The proposed pipeline relies on strong teacher LLMs to generate training data, but the experiments do not adequately distinguish the benefit of the synthesis strategy from that of distilling the teacher’s capabilities. The observed gains could therefore largely reflect the quality of teacher-generated supervision rather than the specific data construction method. To isolate the methodological contribution, the authors should compare alternative synthesis strategies using the same teacher, source problems, and comparable generation and training budgets. Evaluating the proposed pipeline with teachers of different capability levels would further clarify its robustness and dependence on teacher quality.
-
Limited comparative evaluation of the agent harness. The paper presents the daVinci-Prover harness as a contribution, but does not directly compare it with alternative harnesses. Table 3 primarily evaluates the effect of adding retrieval within the proposed harness, which does not establish the effectiveness of the harness design relative to existing alternatives. A comparison with another Lean-oriented harness, or general-purpose coding-agent harnesses such as Codex or Claude Code equipped with Lean verification, would strengthen the evaluation. These comparisons should use the same underlying model where feasible, comparable tool access, and matched inference budgets to isolate the contribution of the harness itself.
-
Insufficient reporting of data synthesis costs. Although data construction is a central contribution, the paper does not report the wall-clock time or computational and monetary costs required to construct the corpus. The teacher models and final corpus size are specified, but the total generation budget, including discarded samples, failed attempts, and Lean verification overhead, remains unclear. Moreover, the reported training hardware does not characterize the cost of data synthesis. A breakdown of these costs by data component would help assess the practical reproducibility and cost-effectiveness of the proposed pipeline, particularly when comparing it with alternative synthesis methods.
Questions for authors
- Directly compare with a data synthesis baseline.
- Change a weak teacher model to show the effectiveness of the proposed method.
3 comments
Response to the rebuttal.
Thank the author's response to my concerns. Based on the rebuttal results, the improvements of the proposed agent and skill are minor compared with the base. I will keep my score unchanged.
We respectfully disagree that the improvements are minor, particularly given that your original review acknowledged our "broad empirical evaluation with consistent downstream gains."
1. Our midtraining data recipe combines skill and agent data.
As you acknowledged in Strength 1, our approach combines mathematical skill supervision with agent trajectories. We do not claim that each component individually yields substantial gains. Instead, our ablations show that the skill–agent mixture outperforms either component alone, including skill-only training at the same token budget. Furthermore, Section 6.2 shows that skill data also influence agent behavior.
2. The improvements are not minor.
For Qwen3-8B, our complete training recipe achieves an average improvement of 13.7 percentage points across three benchmarks over the SFT-only baseline (The SFT data are collected using the harness we developed).
| Training configuration | MT Tokens | MiniF2F | FATE-M | ProofNet |
|---|---|---|---|---|
| Base + SFT (5 epochs) | 0B | 73.4 | 68.0 | 28.4 |
| Base + SFT (7 epochs) | 0B | 80.9 | 72.7 | 32.6 |
| Agent + SFT (5 epochs) | 7.1B | 84.2 | 79.3 | 37.1 |
| Agent + Skill + SFT (5 epochs) | 33.7B | 85.3 | 85.3 | 40.4 |
| Δ vs. 5-epoch SFT | +11.9 | +17.3 | +12.0 |
Even compared with the stronger 7-epoch SFT baseline, the gains remain +4.4, +12.6, and +7.8 points, respectively.
Moreover, our recipe also improves the already strong Qwen3.5-9B backbone, despite its extensive prior agentic training. We include DeepSeek-Flash-0731 as a higher-performing reference model.
| Model | Size | MiniF2F | FATE-M | ProofNet |
|---|---|---|---|---|
| Qwen3.5-9B-Base + SFT | 9B | 87.4 | 88.4 | 42.9 |
| Qwen3.5-9B-Midtrain + SFT | 9B | 90.1 | 93.3 | 44.5 |
| Δ | +2.7 | +4.9 | +1.6 | |
| DeepSeek-Flash-0731 (reference) | 286B-A13B | 92.7 | 98.1 | 53.4 |
We believe these results demonstrate meaningful and consistent improvements, rather than minor gains, and respectfully encourage you to reconsider this assessment.
Response to Reviewer BHWc
Thank you for acknowledging our well-motivated data recipe and broad empirical evaluation.
Our central empirical finding is that incorporating agentic trajectories during mid-training improves subsequent adaptation to interactive theorem proving. We compare skill-only, agent-only, and mixed-data configurations for Qwen3.5-9B under the same downstream SFT, with skill-only training serving as a baseline for assessing the contribution of agentic trajectories. Our skill and agent data are generated using the same teacher.
Conclusion:
In addition, Section 6.5 compares our mid-trained Qwen3-8B with OProver-8B-Base (mid-trained on 65B tokens) under identical downstream SFT settings, further supporting the effectiveness of our mid-training recipe. Another relevant work is Leanstral, which reports 6.5B tokens of Lean data but provides limited details on data collection.
We compare four harnesses using the same DeepSeek-V4.1-1012 checkpoint and sampling parameters, with a maximum of 200 model calls per problem, averaged over three runs. For Claude Code, Codex, and the DeepSeek harness, we use Docker containers to provide a consistent, isolated, and internet-free evaluation environment.
Conclusion: Our harness is competitive: it ties on FATE-H, scores highest on FATE-X, and trails DeepSeek on Putnam-200.
The shared synthesis service used 48 H200 GPUs for approximately 20 days: 23,040 allocated GPU-hours. Lean verification used approximately 160 CPU cores over the same period: 76,800 allocated CPU core-hours. We have added these details to Appendix F, with the additions highlighted in blue.
Our pipeline can also produce Lean-verified outputs with weak teachers. We conducted two targeted synthesis evaluations, each involving 1,000 examples.
(1) Decomposition synthesis. Our pipeline starts from existing verified proofs, prompts the teacher to rewrite them into lemma-based proofs, and verifies the resulting proofs with Lean.
(2) Agent-trajectory collection. We evaluate three teacher models using our agent rollout harness.
These results demonstrate that verifier-guided rejection sampling enables the weak teacher to generate Lean-verified data. Although its success rate remains below that of the stronger teacher, this gap can be mitigated by scaling up the sampling budget.
However, for the weak model like Qwen3.6-35B-A3B that exhibits repetitive and circuitous interactions, this may require additional rubric-based filtering beyond Lean-guided rejection sampling and increase the sampling cost per retained trajectory.
These experiments demonstrate the feasibility of collecting Lean-verified data with smaller teachers.
We hope to engage in further constructive discussion to help clarify our work and its contributions.