- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes BudgetPV, a budget-aware framework that allocates proof attempts across unresolved subgoals and heterogeneous LLM solvers. It uses a cross-solver predictor to rank actions by estimated success relative to generation cost, updates estimates after failed attempts, and switches between two predictors trained for different stages of proving. Experiments on four datasets report lower proof-generation costs than uniform allocation at matched subgoal-solving targets in both live and replay settings.
Critical strengths
-
The paper studies computation allocation jointly across proof subgoals and heterogeneous solvers, addressing a meaningful efficiency challenge when models differ in capability and inference cost.
-
The paper evaluates the method through both live proof generation and controlled replay across four datasets. Using shared proof-attempt pools in replay helps isolate the effect of allocation decisions from variation in proof generation.
Critical weaknesses
-
The evaluation does not establish theorem-level proving effectiveness. The main results measure generation cost at fixed subgoal-solving targets, rather than the fraction of complete theorems proved under a fixed budget. These targets are defined relative to the subgoals solved by Goedel-Prover-V2-32B, so the reported “90%” target should not be interpreted as a benchmark-level theorem success rate. Since completing a theorem requires solving every necessary subgoal, prioritizing inexpensive subgoals may improve the reported metric without increasing the number of completed proofs. The paper should report complete-theorem success rates across matched computation budgets, with final proof assembly and verification.
-
The reported efficiency gains exclude the predictor’s inference cost. The evaluation counts only proof-generation GPU-seconds, although BudgetPV relies on an 8B predictor with two stage-specific variants. The paper does not quantify this overhead or clearly specify the inference and caching protocol, including what is recomputed when switching predictors. Consequently, the reported generation savings do not establish a net inference-cost advantage. A cost breakdown including prediction, proof generation, and verification is needed, particularly where the advantage over Uniform is small. Meanwhile, the BudgetPV should require 4 models (one predictor, and 3 solver) in the environment, much larger than the baseline.
-
The cost of constructing the predictor’s training data is not quantified. Although the paper describes the basic labeling procedure, the three-solver configuration requires eight proof attempts per solver for each of 3,488 training subgoals—83,712 proof-generation attempts, together with verification. The paper does not report the total data-collection and fine-tuning costs, or sufficiently explain the upstream subgoal-construction process. Reporting these costs separately and estimating how much deployment usage is needed to amortize them would clarify the method’s practical value.
-
The comparisons do not isolate the benefit of the proposed allocation design. Uniform is the only practical allocation baseline, while the Statistics Oracle uses unavailable outcome information. Comparisons with a cheap-to-expensive cascade and a static learned router would help establish whether the proposed adaptive mechanism is necessary. Moreover, the predictor analyses in Appendix B do not substitute for end-to-end comparisons of weighted-CE-only, plain-CE-only, and the proposed two-stage policy. Such comparisons are needed to substantiate the claimed benefit of switching objectives during proving.
Questions for authors
Please refer to my weaknesses.