- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper trains Qwen3 4B and 8B provers to generate tactics with predicted Lean states and diagnostics, allowing repair within one response before final verification. Training combines annotated proof and repair data, SFT, and GRPO with feedback supervision and replay. The 8B model reaches 75.61% versus 71.31% pass@32 on combined miniF2F validation/test problems, and 16.44% versus 13.21% on ProofNet.
Critical strengths
- Predicted diagnostics and within-response repair extend proof-state prediction in a useful direction. The auxiliary feedback loss addresses a genuine mismatch: a valid final proof does not guarantee accurate predicted states or errors.
- The contribution includes substantial data construction and model training, supported by model-size comparisons, repair experiments, and feedback-accuracy measurements. The integrated training recipe is worthwhile.
Critical weaknesses
- The baseline omits both replay and feedback supervision, while the models use different self-distillation data and filtered RL pools. This prevents attribution of the gain specifically to feedback prediction. Section 2 discusses DeepSeek-Prover-V1.5, but the relation to ProofOptimizer and Spark-Prover-X1 also needs clarification. Novelty lies in the diagnostic-prediction and repair combination, not state prediction alone.
- Table 3 shows 6,153 versus 1,095 generated tokens per miniF2F response and 6,727 versus 897 on ProofNet. Both WM8 and S8 avoid intermediate Lean calls, so their comparison does not establish a speedup from replacing those calls. Token-matched and wall-clock comparisons against direct generation and actual-feedback repair are missing, as are finalized uncertainty estimates.
Questions for authors
- Does the gain persist with matched replay and RL data, and over a state-prediction-only model? Please compare the training and inference mechanisms with ProofOptimizer, Appendix A.1, and Spark-Prover-X1, Section 3.1.2.
- Under equal token or time budgets, how does predicted-feedback repair compare with direct generation and repair using actual Lean feedback?
- Please provide the counts and evaluation cohorts for Table 15. Table 3 implies 61 solved ProofNet problems for WM8, but 7.69% is not a rounded integer share of 61.
5 comments
Post 8: Response to Reviewer erWN (2/3)
Q2: Relation to ProofOptimizer and Spark-Prover-X1
Reviewer comment/question. Please clarify the relationship between the proposed training and inference mechanisms and ProofOptimizer (Appendix A.1) and Spark-Prover-X1 (Section 3.1.2). The novelty appears to lie in combining diagnostic prediction with repair rather than in state prediction alone.
Author response. We thank the reviewer for pointing out these connections. Both works use state prediction only as training data. Our prover goes further in three ways that neither work shares:
- It predicts Lean's errors, not only states. WM8 predicts the proof state after each block of tactics, and Lean's error message when a tactic fails, inline in its own response.
- It repairs from its own predictions. A predicted error starts a new attempt in the same response, without calling Lean. ProofOptimizer's repair scheme instead hands Lean's actual error messages to a separate, larger model, and Spark-Prover-X1 generates whole proofs without any repair from predicted errors.
- It keeps learning these predictions during RL. The auxiliary feedback loss trains WM8 on Lean's annotations of its own sampled proofs throughout GRPO.
Our experiments show that these choices pay off. From initial SFT to the end of GRPO, WM8's miniF2F pass@32 rises from 74.66 to 75.54, while the standard prover's falls from 72.00 to 70.36 (means of three evaluations). At the same GPU budget, on all 859 problems together, WM8 beats the standard prover by 2.2 points (95% CI [+0.9, +3.6]) and WM8-NR, a feedback-predicting prover trained without repair examples, by 1.5 points (95% CI [+0.3, +2.7]).
In ProofOptimizer (Appendix A.1), tactic and proof-state prediction is one of five tasks used to fine-tune its general Lean base model: given the state before a tactic, the model predicts the tactic and the resulting state, within a mixture of about one billion tokens. At inference, ProofOptimizer shortens existing proofs iteratively, and its separate repair scheme passes Lean's error messages to Goedel-Prover-V2-32B (Section 3.2). Spark-Prover-X1 (Section 3.1.2) extracts "state-before, tactic, state-after" triples, has an LLM predict the next state with a natural-language explanation, keeps the predictions that match the ground-truth states, and uses them as pre-training data for state prediction. We will add both works to the related work and state these differences explicitly.
Post 9: Response to Reviewer erWN (3/3)
Q3: Equal-token or equal-time evaluation
Reviewer comment/question. Under equal token or wall-clock budgets, how does predicted-feedback repair compare with direct generation and with repair using actual Lean feedback?
Author response. We compare provers at the same GPU time, measured as wall-clock time on one H100 per problem, with the same n-gram speculative decoding for every prover. We use GPU time rather than tokens because tokens differ in cost: with speculative decoding, WM8's feedback tokens, which restate hypotheses and goals already in the context, are cheaper than its tactic tokens. The note at the end of our general response explains how we measure GPU time and why we use speculative decoding.
WM8 gets 32 samples per problem, and each prover in Table R9 gets as many samples as fit into the same average GPU time per problem. WM8 beats the standard prover S8, which writes proofs directly without feedback, although S8 gets 2.3 to 6 times as many samples. On all 859 problems together, WM8 leads S8 after GRPO by 2.2 points and S8 before GRPO by 1.4 points, both with 95% confidence intervals above zero. Self-repair also beats spending the same compute on more independent samples: WM8-NR predicts feedback but sees no repair examples in training, gets 41 samples to WM8's 32, and WM8 still leads by 1.5 points, again with a 95% confidence interval above zero. This budget is lenient toward S8, since it does not charge for verifying S8's extra candidates.
Table R9. WM8 after GRPO (32 samples per problem) against each prover at the same GPU budget. Samples: that prover's samples per problem on miniF2F / ProofNet. Other columns: WM8 minus that prover, in pass-rate points, with paired bootstrap 95% confidence intervals over problems. miniF2F + ProofNet: all 859 problems as one set.
| Prover | Samples | WM8 − prover [95% CI]: miniF2F | WM8 − prover [95% CI]: ProofNet | WM8 − prover [95% CI]: miniF2F + ProofNet |
|---|---|---|---|---|
| S8 after GRPO | 193 / 133 | +2.7 [+0.8, +4.7] | +1.6 [−0.5, +3.8] | +2.2 [+0.9, +3.6] |
| S8 before GRPO | 73 / 122 | +0.6 [−1.1, +2.4] | +2.5 [+0.6, +4.6] | +1.4 [+0.1, +2.8] |
| WM8-NR after GRPO | 41 / 41 | +1.8 [+0.2, +3.5] | +1.1 [−0.2, +3.0] | +1.5 [+0.3, +2.7] |
Repair with actual Lean feedback costs far more for little gain. We measured it directly: giving WM8 Lean's actual feedback after every tactic, instead of its own predictions, makes inference take about three times as long on our 64-core machine and improves miniF2F pass@32 by only about one percentage point over WM8. In this run, time spent in Lean dominates inference. Each sample waits for Lean after every tactic. Informal reasoning gives WM8 a larger gain (75.61% to 77.05%, Table 3). WM8 instead predicts the feedback itself, repairs within one response, and sends only its final candidate to Lean.
Q4: Table 15 counts and evaluation cohorts
Reviewer comment/question. Please provide the counts and evaluation cohorts used for Table 15. Table 3 appears to imply that WM8 solves 61 ProofNet problems, but 7.69% is not a rounded integer share of 61. Please reconcile these quantities.
Author response. We thank the reviewer for checking these numbers so carefully, which helps us make the paper clearer. Table 3 and Table 15 come from different evaluation runs. Table 3 is the main pass@32 evaluation, in which WM8 solves 61 of the 371 ProofNet problems. The repair statistics of Table 15 come from a separate run with 8 samples per problem (pass@8), in which WM8 solves 52 ProofNet problems. Four of these are solved only through repair, which gives the 7.69% (4 of 52). The revised paper states that Table 15 comes from a separate run in the caption of the new main-text table that summarizes it, and we will add the sample count and these counts to both captions.
We thank the reviewer again for the careful reading and helpful suggestions. We hope our answers address the reviewer's concerns, and we are happy to answer any further questions.
Post 7: Response to Reviewer erWN (1/3)
Q1: Matched replay, RL data, and a state-prediction-only model
Author response. WM8 keeps its advantage when S8 also uses replay, and over a model without repair supervision.
Replay. We add the supervised replay of WM8's GRPO stage to S8's GRPO stage. Replay has little effect on S8: over three evaluations, its pass@32 goes from 70.36 ± 0.66 to 71.45 ± 0.31 on miniF2F and stays at 13.93 on ProofNet, while WM8 reaches 75.54 ± 0.43 and 16.35 ± 0.41. WM8 keeps a lead of 4.1 and 2.4 points over S8 with replay. We will report this control in Section 5.2 of the revision.
RL data. Every prover selects its GRPO problems with the same rule, keeping the problems on which its own success rate is above zero and at most 0.75. This follows the (0, 0.75] range of Goedel-Prover-V2 and the moderate-success-rate selection of DeepSeek-Prover-V1.5 (Appendix A.3). The pools differ only because each prover applies this rule to its own success rates, so that it trains on problems at its own level of difficulty.
State prediction only. WM8-NR is this model. It keeps feedback prediction and both auxiliary losses but sees no repair examples. In initial SFT and self-distillation, it therefore learns to predict proof states without any error targets, and before GRPO it never repairs (Section 5.2). During GRPO, the auxiliary feedback loss adds error targets from its own failed attempts. Comparing it with WM8 and S8 shows two things (Table R8):
Table R8. Pass-rate differences in points, each row at the same GPU budget, with paired bootstrap 95% confidence intervals over problems. Sample counts are per problem (for S8, on miniF2F and ProofNet).