- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies how supervised fine-tuning objectives can account for finite-budget search in automated theorem proving. It introduces strategy-specific Compute-Aligned Training (CAT) objectives and a search-agnostic Uniform-Allocation (UA) objective, both implemented through weighted tactic-level cross-entropy gradients. The theoretical analysis examines how alternative proofs, off-trace exploration costs, and recoverability affect approximation error. Experiments evaluate Qwen2.5-Math-7B-Instruct on 458 LeanDojo theorems across six search strategies and multiple computational budgets.
Critical strengths
The paper connects training objectives with proof discovery under finite computational budgets, extending compute-aligned training to tree search and providing a unified framework for addressing the mismatch between supervised training and deployment-time search.
By introducing both strategy-specific and shared objectives and observing improvements across multiple search methods, the paper demonstrates the feasibility of improving deployment-time search through changes to offline supervised training objectives.
Critical weaknesses
The trace-supported objectives omit aspects of deployed search. Non-demonstrated actions are collapsed into a single deviation event, while the best-first search surrogate replaces frontier competition with repeated sampling at a single node. Demonstrations alone cannot determine alternative proof opportunities, recovery costs, or irreversible deviations, so the theoretical conclusions depend on explicit modeling assumptions.
The experiments use one model and theorems whose reference proofs contain only two to five steps. The budget sweep trains separate adapters for each strategy and budget, and the completed evaluation subsets differ across budgets. Consequently, cross-budget comparisons involve changes in both adapters and evaluated theorem sets.
Questions for authors
What motivates using (N/4) to approximate the rollout budget for Pass@N training, and how does this conversion account for actual proof lengths and early rollout failures?
How large is the common theorem subset evaluated at every budget in Table 2, and what are the gains over CE at 16 and 256 expansions on that shared subset?
3 comments
Thank you for your responses and clarifications. I maintain my overall positive assessment of the work, while retaining views on the potential issues.
Thank you for your positive perspective on our work. We appreciate the reply and the broader review.
We would like to thank the reviewer for their positive perspective on our work and the careful review.
Q1
Kindly refer to our response to reviewer amMw on Q3.
Q2
The common completed subsets contain 408 theorems for Pass@N and 407 for BFS. Recomputing the paired results on these subsets, the respective alignment gains at budgets 16 and 256 are +1.96 versus +4.17 percentage points for Pass@N, and +3.93 versus +6.88 points for BFS.
These comparisons eliminate changes in theorem composition across budgets. As noted in our response to Reviewer amMw in their Q2, the point-estimate increases remain positive.