- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces LEVER, an LLM-based theorem-proving system that guides proof construction using computational cost and user-selected proof metrics. It represents alternative proof decompositions as an AND/OR graph, combines measured contributions with estimates for unfinished subgoals, and can revisit goals to seek better decompositions. Completed proofs are checked in Lean. On 80 selected PutnamBench problems, LEVER reports a 96.3% solve rate versus 80.0% for a single-conversation baseline, with average cost falling from $1.44 to $0.95 under standardized cache pricing. On 20 problems, it also explores proof-length and topical-impurity objectives. The results demonstrate improvements in these measured objectives, although the comparison supporting superior impurity reduction over refactoring needs clarification. Appendix D establishes the additive backup equations’ exactness under its stated assumptions, rather than an optimality guarantee for the implemented adaptive search.
Critical strengths
-
I liked the explicit treatment of proof characteristics during construction. Sections 3–4 describe a concrete mechanism for balancing unfinished work against measured proof properties, and Table 2 shows that changing the objective produces meaningfully different outcomes. This makes the system useful beyond simply finding another correct proof.
-
I found the reported cost and solve-rate improvements substantial within the evaluated setting. Table 1 shows improvements on both measures, and Appendix C.3 reports that the advantage remains under actual cache behavior: 96.3% versus 86.3% solved, at $0.90 versus $1.07 per problem. Including planning and value-function calls in the reported cost strengthens this evidence.
Critical weaknesses
-
I would appreciate clarification on how the refactoring baseline handles rewrites that worsen the original proof. Section 5.2 gives an example where impurity increases from 29 to 132, although the original valid proof could be retained. If this outcome contributes to the reported average, keeping the better proof could substantially change the comparison. LEVER’s lower cost would remain a strength, but I am currently unsure whether its advantage in impurity reduction would hold under this adjustment.
-
I found the overall efficiency gains encouraging, but I had some difficulty determining how much comes specifically from adaptive search. Section 5.1 compares systems that also differ in planning, decomposition, and context management, and the paper notes that early rejection of expensive decompositions is rare. A focused comparison retaining these components while disabling adaptive redraws and prior updates would help me assess the contribution of the proposed search mechanism. I would reconsider this concern if such evidence shows a meaningful benefit from adaptation.
Questions for authors
-
I would appreciate clarification on whether the impurity increase from 29 to 132 is included in Figure 5’s average. Could you report the reduction and variability when the refactoring baseline retains whichever proof has lower impurity, with the computation charges unchanged? This would help me assess whether the reported quality advantage remains. Clarification / Additional evidence/analysis.
-
I would like to understand how LEVER performs when adaptive redraws and prior updates are disabled, while keeping the planner, local contexts, tools, and direct-proof routing unchanged. Is there evidence showing how these adaptive components affect solve rate and cost? Additional evidence/analysis.
These concerns are provisional, and I would be open to revising my recommendation based on further clarification or evidence addressing them.
2 comments
Discussion reminder
Dear reviewer,
Thank you again for your detailed review. As the discussion period draws to a close, we would be happy to discuss any remaining questions. In addition to our responses above, our follow-up to Reviewer Yvqh isolates the effect of objective-guided search: with adaptive redraws disabled, LEVER's proofs are no purer than the baseline's, while at λ = 0.1 they are purer on 19 of the 20 problems. If our responses have addressed your concerns, we would appreciate it if you could re-evaluate your assessment and update your score accordingly.
Response to Reviewer kRGb
N.B. a mix of USD and $ notation is used below due to a LaTeX rendering bug.
Yes,
putnam_2007_a2is included in the reported average. When the refactoring baseline keeps whichever of NearAI's original proof and the refactored version is purer, with charges unchanged, the mean impurity reduction rises from 33% (SD 101) to 53% (SD 38), at USD 2.88 per problem. LEVER at λ = 0.1 reduces impurity by 42% (SD 32) at $1.85. LEVER's advantage is cost, about two-thirds of the refactoring pipeline's. We will report both versions of the baseline and correct the claims in the relevant sections in the paper.The
putnam_2007_a2rewrite replaced one linear-algebra step (a linear map scales area by its determinant) with a direct computation by shears and scalings. It traded NearAI's 11 linear-algebra references for 131 references to measurability, which lies outside the subjects read off the statement. Whether that trade counts as a loss depends on whether measurability is part of this theorem's subject, and in this case it was not per our implementation (see "Topical impurity" in the general response).With adaptation disabled, cost and solve rate barely move: 96.3% at USD 0.95 against 95.0% at $0.97, a cost ratio of 0.98. Adaptive redraws were 8% of LEVER's attempts, and the routing those prices drive is cost-neutral on its own.
We take your point: on this benchmark, adaptation as implemented does not show a meaningful benefit. The general response explains why with measurements. Alternatives differ in cost on about a third of problems. In a pre-registered test on 10 harder problems, the value function's choice among three root decompositions was 12% cheaper than a random choice, but the two extra draws cost more than that, so drawing three and choosing did not beat drawing one. LEVER's search commits to its first root and redraws only when the value function flags, so it rarely gets to choose. We will revise the paper to state this and report these measurements. The natural next step is to draw an alternative only when its expected saving exceeds its cost.