- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes segment-level supervision, which splits verified Lean 4 proof trajectories whenever the number of open goals changes and uses the resulting multi-tactic segments as supervision targets. They report that a Qwen2.5-Math-7B policy trained this way on NuminaMath-LEAN reaches 66.31% on miniF2F, compared with 63.11% for step-level and 55.90% for whole-proof supervision on the same data. The paper also proposes goal-aware rollout, which reuses the same signal at inference time and lets an existing step-level prover continue along the current branch for a few steps whenever a tactic changes the goal count. Without retraining, this raises BFS-Prover-V2-7B from 68.77% to 70.74% on miniF2F and from 45.60% to 50.67% on FATE-M.
Critical strengths
- The comparison of supervision granularities is clean, with step-level, whole-proof and segment-level targets are built from the same verified trajectories and trained on the same base model. Segment-level training acheives the best in all setting (66.31% vs. 63.11% and 55.90% on miniF2F with NuminaMath-LEAN).
- Goal-aware rollout, by only changing the search loop, an easy-adopt method, improves the step provers on both benchmarks while using fewer tokens and less time (on FATE-M, BFS-Prover-V2-7B goes from 45.60% to 50.67% with about 25% fewer tokens),
Critical weaknesses
- The gains are consistent in direction but small, and the evaluation cannot show that they are significant.
- Most gains in Tables 1 and 3 are under 2%. Only NuminaMath-LEAN on miniF2F (+3.2%) and the two rollout results on FATE-M (+5.1% and +2.7%) are larger, and with Qwen3-14B (Appendix A.3) the gain over step-level falls below the standard deviation (53.36% vs. 52.46% on miniF2F, 17.87% vs. 17.20% on FATE-M).
- The reported standard deviation comes from five beam-search runs of one checkpoint, so it does not cover training randomness. With differences of 1–2 points and a single training seed, the ranking between granularities is not reliable, and the +0.74 gain for InternLM2.5-StepProver on miniF2F is within the run-to-run noise.
- The step-level baselines in Table 3 are also weaker than in their own papers. BFS-Prover-V2-7B gets 68.77% on miniF2F, below the 70.8% its paper reports for the older V1-7B, and InternLM2.5-StepProver gets 59.59%, which matches its no-critic result (59.4%) rather than the 65.9% with the critic. Both are run with beam search instead of their original decoding and on Lean 4.11 instead of 4.10.0 and 4.7.0, so small gains on these baselines may not hold under the original settings.
- The paper credits the gains to the goal-count boundaries, but none of the experiments separates this effect from simply taking several steps per expansion.
- Table 3 compares goal-aware rollout only with plain best-first search. A rollout triggered at random positions with the same frequency could conceivably give a similar gain, and this would be easy to test.
- I would also like to see the contributions of training and inference separated. The authors' own step-level policy is never combined with goal-aware rollout, so the extra value of segment-level training on top of rollout is unknown.
Questions for authors
- In Table 3, BFS-Prover-V2-7B and InternLM2.5-StepProver both score below the performance in their own papers. Have you tried their original inference settings (sampling at temperature 1.3 for BFS-Prover-V2, and the critic for InternLM2.5-StepProver)? I would like to see the results of goal-aware rollout under these settings.
- Did you try goal-aware rollout on your own step-level model? A rollout triggered at random positions with the same frequency would also be a useful control for the goal-change trigger.
- If I understand correctly, DeepSeek-Prover-V1.5 already uses a truncate-and-resume search, which generates several tactics at once, cuts the output at the first error and continues the search from the last valid state. Whole-proof-seg in Table 1 follows the same procedure, but the paper cites V1.5 only as a whole-proof method. Apart from the stopping rule, I do not see a clear difference between the two. Could you clarify this?
- I notice the paper only reveals a per-theorem timeout of 1800 seconds. Is there also a per-tactic timeout or heartbeat limit, and how much CPU and memory does each Lean process get? Are the final proofs also recompiled as complete files, with sorry, admit, native_decide and new axioms rejected? In my experience older Lean and LeanDojo setups let a proof pass while still relying on such shortcuts, and this matters a lot once the pipeline is used for RL. Since time cost is one of the main metrics, the reported time also depends directly on this environment.
Suggestions for future work
- It would be interesting to see segment-level supervision on top of a strong prover or how well it bridges the performance via RL.
- If the goal count is later used as a step boundary or reward in RL, the policy might add constructor or have steps just to change the number of goal count it self to hack, which is worth a short note in the limitations.
10 comments
Response to W1-1 && W1-2 (Part 2)
W1-1 && W1-2 (Part 2)
The tests show that all training-side comparisons on miniF2F are statistically significant, whereas some comparisons on FATE-M do not reach significance. Among the four inference-side GAR comparisons, three are significant after Holm correction. For the nonsignificant result of InternLM2.5-StepProver on miniF2F, the GAR trigger rate is only 3.65%; this relatively low trigger frequency may be one reason for the limited improvement. Although some comparisons are not statistically significant, Segment-level achieves the highest mean accuracy across all six training-set and test-set combinations under the additional training seed, with overall improvements in accuracy.
We also agree that on formal theorem-proving benchmarks of limited size, such as miniF2F and FATE-M, an accuracy improvement of approximately one percentage point corresponds to only a small number of additional successfully proved theorems. However, each additional success represents the discovery of a valid path through a vast proof search space, culminating in a complete proof script that passes Lean verification. These improvements therefore remain practically meaningful. More importantly, Segment-level consistently improves performance across different training-data sources and both evaluation benchmarks. This overall pattern across data sources and evaluation benchmarks further supports the effectiveness of segment-level supervision.
References [1] S. H. Jung. Stratified Fisher's exact test and its sample size calculation. Biometrical Journal, 56(1):129–140, 2014. [2] S. Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70, 1979.
Response to W1-3 && Q1
W1-3 && Q1
We thank the reviewer for the suggestion. To further validate the effectiveness of goal-aware rollout (GAR) under the original inference configurations, we add controlled experiments using the official inference settings of the two models. Specifically, for BFS-Prover-V2-7B, we use the official prompt template, sampling with temperature = 1.3, and path-length-normalized scoring (). For InternLM2.5-StepProver, we use the official prompt template, temperature = 0.7, and the original 1.8B critic model for state scoring. We retain the original numbers of generated candidates for the two models (3 and 32, respectively) and set the GAR rollout horizon to for both.
For a fair comparison, Base and GAR use the same model configurations and search budgets, with at most 600 regular expansions and a search timeout of 600 seconds per theorem. The additional generation and execution costs of GAR are also included in the shared budget. We conduct three independent evaluations on miniF2F (244 theorems) and FATE-M (150 theorems), respectively, with the following results:
Table M: Results of Goal-aware Rollout under the Original Inference Configurations (Mean ± Std over Three Independent Runs)
| Model | Dataset | Base Pass@1 | + GAR Pass@1 | Gain |
|---|---|---|---|---|
| BFS-Prover-V2-7B | miniF2F | 61.48 ± 1.08 | 62.43 ± 1.03 | +0.96 |
| BFS-Prover-V2-7B | FATE-M | 22.22 ± 1.92 | 26.22 ± 0.77 | +4.00 |
| InternLM2.5-StepProver | miniF2F | 51.64 ± 1.48 | 53.83 ± 0.24 | +2.19 |
| InternLM2.5-StepProver | FATE-M | 30.00 ± 2.40 | 31.33 ± 3.71 | +1.33 |
The results show that, with the original sampling strategies and scoring mechanisms, GAR still improves performance in all four settings involving the two models and two datasets. Specifically, BFS-Prover-V2-7B improves by 0.96 and 4.00 percentage points on miniF2F and FATE-M, respectively, while InternLM2.5-StepProver improves by 2.19 and 1.33 percentage points, respectively. These results further demonstrate that the effectiveness of GAR is not limited to the unified inference configuration in Table 3; it also improves proving performance under different sampling strategies and critic-guided search.
We thank the reviewer again for the suggestion. We will include these experimental results in the revised manuscript and further clarify the evaluation settings under different inference configurations to provide a more comprehensive account of the applicability of GAR to existing step-level provers.
Response to W1-1 && W1-2 (Part 1)
W1-1 && W1-2 (Part 1)
We thank the reviewer for their attention to the statistical reliability of the experiments. First, we clarify that the five independent runs reported in the original paper were all inference runs using the same fixed model checkpoint. Therefore, the reported mean ± std reflects randomness during inference rather than randomness during training. To further examine the effect of training randomness, we retrained the models using a different PyTorch random seed (seed 123, whereas the original experiments used seed 42) and conducted three independent inference evaluations on each of miniF2F and FATE-M. As shown in Table F, across all six combinations of the three training datasets and two evaluation benchmarks, Segment-level achieves the highest mean proof success rate, consistent with the original experimental results.
Table F: Results with an Additional Training Seed (PyTorch Seed 123; Mean ± Std over Three Independent Runs)
In addition, we added statistical significance tests. For the training-side comparisons, we combine the five inference results from the original experiments with the three additional inference results under seed 123, for a total of eight evaluations. For the inference-side GAR comparisons, we use the five inference results from the original experiments. miniF2F and FATE-M contain 244 and 150 theorems, respectively. Because the different methods are evaluated on the same test set, we use a two-sided theorem-stratified exact conditional randomization test, preserving theorem-level pairing and testing differences in mean Pass@1 between methods [1]. We also apply Holm correction separately to the multiple training-side and inference-side comparisons [2]. The results are shown in Tables G and H.
Table G: Statistical Significance of Segment-level Supervision (Training-side Comparisons; Holm-adjusted p-values)
Table H: Statistical Significance of Goal-aware Rollout (Inference-side Comparisons; Holm-adjusted p-values)