- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether learned models can reduce the cost of evaluating Lean proof candidates during reinforcement learning. Using 4.84 million compiler-labelled programs, the authors train validity classifiers and a model that generates predicted goal states and compiler diagnostics. Their main evaluator combines an 8B classifier with a 4B feedback model, accepting a candidate only when both predict validity. The default cascade achieves 87.8% verdict accuracy on 68,932 candidates from three public provers. Speculative decoding accelerates feedback generation, and the authors report approximately 3.5 candidates per second on one H100. For GRPO training, Lean and the cascade run concurrently; completed Lean verdicts take precedence, and predictions supply the remaining rewards. In the reported one-epoch comparison, this reduces approximate step time by 31%, with similar final miniF2F pass@1 to standard Lean-only training and a higher observed pass@32. These results demonstrate the trade-off for the evaluated checkpoints and configuration.
Critical strengths
-
I liked that the paper evaluates the proposed reward system through actual prover training, including a useful control for simply shortening verification. In Table 7, hybrid training reduces approximate step time from 1,050 to 720 seconds while reaching 62.2% pass@1, compared with 62.4% for standard Lean-only training. The 13-second Lean-only condition uses approximately the same step time as hybrid training but reaches 59.8%. Because final candidates are checked independently with Lean, this provides meaningful evidence that learned rewards can save training time while retaining useful performance.
-
I also liked the substantive evaluation of the learned evaluator itself. The default cascade reaches 87.8% accuracy, compared with 85.1% for feedback generation alone and 83.1% for the calibrated one-token classifier (Table 3), and improves accuracy across all three prover families (Table 4). The feedback analysis also exposes important failure modes instead of relying only on aggregate verdict accuracy. Together, these results make the proposed evaluator more convincing and help explain its practical limitations.
Critical weaknesses
-
I think the reproducibility of the central training result remains uncertain. Section 6.2 and Appendix E report one final trained checkpoint per reward condition; the standard deviations in Table 7 come from repeated decoding and evaluation of those fixed checkpoints. They therefore do not measure variation across training runs. The observed near-equality in pass@1 and 1.1-point pass@32 advantage are encouraging, but their stability is unclear. An independent repeat of the matched comparison would strengthen this conclusion; otherwise, the quality-preservation claim should remain explicitly limited to the reported runs.
-
I had questions about the evidence for reward quality on the cases where predictions are most needed. The 92.5% hybrid agreement excludes the original timeouts, while the extended-budget audit reports 80.1% recall for the standalone feedback model, rather than the cascade used during training (6.2; Appendix E). It also does not report how often the cascade accepts the 334 recovered invalid proofs. This leaves an important uncertainty about the balance between recovered correct rewards and incorrect positive rewards on difficult checks. A cascade confusion matrix for the resolved timeout cases, together with results for candidates receiving model-based rewards at the cutoff, would address this gap.
Questions for authors
-
I would like to know whether the matched GRPO comparison has been repeated with independent training randomness. If additional runs are available, do they preserve the observed pass@1 similarity and pass@32 ordering? Additional evidence/analysis; relates to weakness 1. This would help distinguish a reproducible training trade-off from differences between individual checkpoints.
-
I would like to see the deployed cascade’s confusion matrix on the 797 resolved timeout cases, including its recall on the 463 valid proofs and false-positive rate on the 334 invalid proofs. Could you also report these rates for the labelled candidates whose rewards actually come from predictions at the 90-second cutoff? Additional evidence/analysis; relates to weakness 2. These results would establish reward reliability on the relevant subset more directly than full-cohort agreement.
2 comments
## Post 13: Response to Reviewer S2bK (2/2): Q2
Q2: How reliable is the cascade where its predictions supply the reward?
Reviewer comment/question. The 92.5% hybrid agreement excludes the original timeouts, and the timeout audit does not report how often the cascade used during training accepts the 334 recovered invalid proofs. Please give the deployed cascade's confusion matrix on the 797 resolved timeouts, including its recall on the 463 valid proofs and its false-positive rate on the 334 invalid proofs. Please also report these rates for the labelled candidates whose rewards come from predictions at the 90 s cutoff.
Author response. We thank the reviewer for this request. For both sets of rollouts, Table R11 gives the confusion matrices of the cascade used in training, which accepts a proof only if both its 8B classifier and its 4B feedback model accept it. Both come from the replay of Section 6.2, which applies the hybrid rule to 24,576 saved training rollouts. In training, the cutoff is the moment the cascade finishes, after about 90 s on average, and the replay fixes it at 90 s. Every rollout without a Lean verdict at the cutoff gets the cascade's verdict as its reward.
Table R11. Cascade verdicts against Lean labels in the replay of Section 6.2. Resolved timeouts: proofs that exceed Lean's 60 s limit per proof, labelled afterwards by a 600 s recheck. Scored by the cascade: rollouts whose original Lean verdict arrives after the 90 s cutoff.
| Rollouts | Lean label | Cascade: valid | Cascade: invalid | Total |
|---|---|---|---|---|
| Resolved timeouts | Valid | 371 | 92 | 463 |
| Invalid | 56 | 278 | 334 | |
| Scored by the cascade | Valid | 6,180 | 1,198 | 7,378 |
| Invalid | 564 | 3,863 | 4,427 |
On resolved timeouts, the cascade accepts 371 of the 463 valid proofs (80.1% recall) and 56 of the 334 invalid ones (16.8% false-positive rate). Lean returns no verdict for these proofs within the 60 s limit of GRPO training. Under hybrid rewards, the cascade's verdict becomes their reward. Under Lean-only training, none of the 463 valid proofs gets a positive reward. Hybrid rewards thus recover 371 deserved positive rewards and add 56 undeserved ones.
On the labelled rollouts whose reward comes from the cascade at the cutoff, its recall is 83.8% and its false-positive rate 12.7%. On all 23,489 labelled rollouts, these rates are 85.1% and 11.6% (Section 6.2). Rollouts whose Lean verdict arrives after the cutoff are therefore barely harder for the cascade.
Most of the cascade's acceptances are valid: 371 of 427 on resolved timeouts (86.9%) and 6,180 of 6,744 at the cutoff (91.6%).
In the replay, the hybrid reward accepts only 6.7% of invalid labelled rollouts, against 11.6% for the cascade's own verdicts. Every labelled rollout not scored by the cascade gets its reward from Lean, and that reward matches Lean's label. The cascade's errors at the cutoff are therefore the only reward errors among the 23,489 labelled rollouts. The hybrid reward accepts 564 of the 8,480 invalid rollouts (6.7%) and rejects 1,198 of the 15,009 valid ones (8.0%). These 1,762 errors give the 92.5% agreement of Section 6.2.
We thank the reviewer again for the careful reading and helpful questions. We hope our answers address the reviewer's concerns, and we are happy to answer any further questions.
## Post 12: Response to Reviewer S2bK (1/2): Q1
Response to Reviewer S2bK
Q1: Does the matched comparison hold with independent training randomness?
Author response. The second seed preserves both the pass@1 similarity and the pass@32 ordering. We repeat the hybrid run, the only condition whose rewards depend on model predictions, with a new random seed and otherwise identical settings. We compare both hybrid runs with the same Lean-only controls (Table R10).
Table R10. miniF2F pass@1 and pass@32 (%) on all 488 problems after one epoch of GRPO. Mean ± standard deviation over five evaluations (pass@1) and three (pass@32) of each checkpoint. Rows marked † repeat Table 7 of the paper.
Both hybrid runs are within 0.3 points of the 60 s Lean-only run in pass@1 (62.2% and 62.7%, against 62.4%) and above it in pass@32 (72.4% and 72.1%, against 71.3%). Both also beat the 13 s Lean-only control on both metrics (59.8% and 69.7%). In the paper, this control and the first hybrid run both take about 90 s on average to score a batch (Section 6.1).
The two hybrid runs differ by only 0.5 points in pass@1 and 0.3 points in pass@32, far less than their lead of at least 2.4 points over the 13 s control. The miniF2F comparison of Table 7 is therefore reproducible across independent hybrid training runs.