acceptodds
Under review as a conference paper at ICLR 2027

Learning Lean Feedback for Efficient Proof-Candidate Evaluation on GPUs

12 views

Abstract

Training language models for formal theorem proving requires large numbers of verifier calls. Slow checks and timeouts often leave GPUs waiting and withhold rewards from potentially correct proofs. We train neural evaluators to predict Lean 4 verdicts and compiler feedback on GPUs using 4.84M labelled programs. We compare two validity classifiers with a feedback model that annotates proof attempts with predicted goal states and error messages. A two-stage evaluator that screens candidates with a classifier and scores its positive predictions with the feedback model reaches 87.8% accuracy in predicting whether Lean accepts a candidate proof, evaluated on a 68,932-program benchmark collected from public provers. Speculative decoding triples feedback-generation throughput, enabling the combined evaluator to process 3.5 candidates per second on one H100, 4.2 the throughput of our 64-worker Lean baseline with environment reuse. Finally, we combine our evaluator with Lean verdicts to obtain a hybrid reward system that shortens GRPO training steps by approximately 31%. After one GRPO epoch with hybrid rewards, the prover achieves nearly the same miniF2F pass@1 as with Lean-only rewards (62.2% versus 62.4%) and a pass@32 of 72.4% versus 71.3%.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
automated theorem proving, Lean, formal verification, compiler feedback prediction, learned proof evaluators, reinforcement learning, GRPO

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_S2bKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether learned models can reduce the cost of evaluating Lean proof candidates during reinforcement learning. Using 4.84 million compiler-labelled programs, the authors train validity classifiers and a model that generates predicted goal states and compiler diagnostics. Their main evaluator combines an 8B classifier with a 4B feedback model, accepting a candidate only when both predict validity. The default cascade achieves 87.8% verdict accuracy on 68,932 candidates from three public provers. Speculative decoding accelerates feedback generation, and the authors report approximately 3.5 candidates per second on one H100. For GRPO training, Lean and the cascade run concurrently; completed Lean verdicts take precedence, and predictions supply the remaining rewards. In the reported one-epoch comparison, this reduces approximate step time by 31%, with similar final miniF2F pass@1 to standard Lean-only training and a higher observed pass@32. These results demonstrate the trade-off for the evaluated checkpoints and configuration.

Critical strengths
  1. I liked that the paper evaluates the proposed reward system through actual prover training, including a useful control for simply shortening verification. In Table 7, hybrid training reduces approximate step time from 1,050 to 720 seconds while reaching 62.2% pass@1, compared with 62.4% for standard Lean-only training. The 13-second Lean-only condition uses approximately the same step time as hybrid training but reaches 59.8%. Because final candidates are checked independently with Lean, this provides meaningful evidence that learned rewards can save training time while retaining useful performance.

  2. I also liked the substantive evaluation of the learned evaluator itself. The default cascade reaches 87.8% accuracy, compared with 85.1% for feedback generation alone and 83.1% for the calibrated one-token classifier (Table 3), and improves accuracy across all three prover families (Table 4). The feedback analysis also exposes important failure modes instead of relying only on aggregate verdict accuracy. Together, these results make the proposed evaluator more convincing and help explain its practical limitations.

Critical weaknesses
  1. I think the reproducibility of the central training result remains uncertain. Section 6.2 and Appendix E report one final trained checkpoint per reward condition; the standard deviations in Table 7 come from repeated decoding and evaluation of those fixed checkpoints. They therefore do not measure variation across training runs. The observed near-equality in pass@1 and 1.1-point pass@32 advantage are encouraging, but their stability is unclear. An independent repeat of the matched comparison would strengthen this conclusion; otherwise, the quality-preservation claim should remain explicitly limited to the reported runs.

  2. I had questions about the evidence for reward quality on the cases where predictions are most needed. The 92.5% hybrid agreement excludes the original timeouts, while the extended-budget audit reports 80.1% recall for the standalone feedback model, rather than the cascade used during training (6.2; Appendix E). It also does not report how often the cascade accepts the 334 recovered invalid proofs. This leaves an important uncertainty about the balance between recovered correct rewards and incorrect positive rewards on difficult checks. A cascade confusion matrix for the resolved timeout cases, together with results for candidates receiving model-based rewards at the cutoff, would address this gap.

Questions for authors
  1. I would like to know whether the matched GRPO comparison has been repeated with independent training randomness. If additional runs are available, do they preserve the observed pass@1 similarity and pass@32 ordering? Additional evidence/analysis; relates to weakness 1. This would help distinguish a reproducible training trade-off from differences between individual checkpoints.

  2. I would like to see the deployed cascade’s confusion matrix on the 797 resolved timeout cases, including its recall on the 463 valid proofs and false-positive rate on the 334 invalid proofs. Could you also report these rates for the labelled candidates whose rewards actually come from predictions at the 90-second cutoff? Additional evidence/analysis; relates to weakness 2. These results would establish reward reliability on the relevant subset more directly than full-cohort agreement.

2 comments
AuthorsOpenReview ↗

## Post 12: Response to Reviewer S2bK (1/2): Q1

Response to Reviewer S2bK

Q1: Does the matched comparison hold with independent training randomness?

Reviewer comment/question. Section 6.2 and Appendix E report one final trained checkpoint per reward condition, and the standard deviations in Table 7 come from repeated decoding and evaluation of those fixed checkpoints. Has the matched GRPO comparison been repeated with independent training randomness? If so, do the additional runs preserve the observed pass@1 similarity and pass@32 ordering?

Author response. The second seed preserves both the pass@1 similarity and the pass@32 ordering. We repeat the hybrid run, the only condition whose rewards depend on model predictions, with a new random seed and otherwise identical settings. We compare both hybrid runs with the same Lean-only controls (Table R10).

Table R10. miniF2F pass@1 and pass@32 (%) on all 488 problems after one epoch of GRPO. Mean ± standard deviation over five evaluations (pass@1) and three (pass@32) of each checkpoint. Rows marked † repeat Table 7 of the paper.

Reward sourcePass@1 (%)Pass@32 (%)
Lean only, 60 s timeout†62.4 ± 0.671.3 ± 0.8
Lean only, 13 s timeout†59.8 ± 0.669.7 ± 1.1
Hybrid, cascade, first seed†62.2 ± 0.872.4 ± 0.4
Hybrid, cascade, second seed62.7 ± 0.772.1 ± 0.6

Both hybrid runs are within 0.3 points of the 60 s Lean-only run in pass@1 (62.2% and 62.7%, against 62.4%) and above it in pass@32 (72.4% and 72.1%, against 71.3%). Both also beat the 13 s Lean-only control on both metrics (59.8% and 69.7%). In the paper, this control and the first hybrid run both take about 90 s on average to score a batch (Section 6.1).

The two hybrid runs differ by only 0.5 points in pass@1 and 0.3 points in pass@32, far less than their lead of at least 2.4 points over the 13 s control. The miniF2F comparison of Table 7 is therefore reproducible across independent hybrid training runs.

AuthorsOpenReview ↗

## Post 13: Response to Reviewer S2bK (2/2): Q2

Q2: How reliable is the cascade where its predictions supply the reward?

Reviewer comment/question. The 92.5% hybrid agreement excludes the original timeouts, and the timeout audit does not report how often the cascade used during training accepts the 334 recovered invalid proofs. Please give the deployed cascade's confusion matrix on the 797 resolved timeouts, including its recall on the 463 valid proofs and its false-positive rate on the 334 invalid proofs. Please also report these rates for the labelled candidates whose rewards come from predictions at the 90 s cutoff.

Author response. We thank the reviewer for this request. For both sets of rollouts, Table R11 gives the confusion matrices of the cascade used in training, which accepts a proof only if both its 8B classifier and its 4B feedback model accept it. Both come from the replay of Section 6.2, which applies the hybrid rule to 24,576 saved training rollouts. In training, the cutoff is the moment the cascade finishes, after about 90 s on average, and the replay fixes it at 90 s. Every rollout without a Lean verdict at the cutoff gets the cascade's verdict as its reward.

Table R11. Cascade verdicts against Lean labels in the replay of Section 6.2. Resolved timeouts: proofs that exceed Lean's 60 s limit per proof, labelled afterwards by a 600 s recheck. Scored by the cascade: rollouts whose original Lean verdict arrives after the 90 s cutoff.

RolloutsLean labelCascade: validCascade: invalidTotal
Resolved timeoutsValid37192463
Invalid56278334
Scored by the cascadeValid6,1801,1987,378
Invalid5643,8634,427

On resolved timeouts, the cascade accepts 371 of the 463 valid proofs (80.1% recall) and 56 of the 334 invalid ones (16.8% false-positive rate). Lean returns no verdict for these proofs within the 60 s limit of GRPO training. Under hybrid rewards, the cascade's verdict becomes their reward. Under Lean-only training, none of the 463 valid proofs gets a positive reward. Hybrid rewards thus recover 371 deserved positive rewards and add 56 undeserved ones.

On the labelled rollouts whose reward comes from the cascade at the cutoff, its recall is 83.8% and its false-positive rate 12.7%. On all 23,489 labelled rollouts, these rates are 85.1% and 11.6% (Section 6.2). Rollouts whose Lean verdict arrives after the cutoff are therefore barely harder for the cascade.

Most of the cascade's acceptances are valid: 371 of 427 on resolved timeouts (86.9%) and 6,180 of 6,744 at the cutoff (91.6%).

In the replay, the hybrid reward accepts only 6.7% of invalid labelled rollouts, against 11.6% for the cascade's own verdicts. Every labelled rollout not scored by the cascade gets its reward from Lean, and that reward matches Lean's label. The cascade's errors at the cutoff are therefore the only reward errors among the 23,489 labelled rollouts. The hybrid reward accepts 564 of the 8,480 invalid rollouts (6.7%) and rejects 1,198 of the 15,009 valid ones (8.0%). These 1,762 errors give the 92.5% agreement of Section 6.2.

We thank the reviewer again for the careful reading and helpful questions. We hope our answers address the reviewer's concerns, and we are happy to answer any further questions.

Reviewer_H36JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a methodology for training a neural surrogate that predicts lean verification outcomes and feedback. The motivation is to reduce the time spent validating generated proofs using lean during RL training. Specifically, the authors propose a 2 stage neural surrogate. The first stage predicts correctness, and the second generates feedback traces. Additionally, the authors propose a hybrid reward mechanism that combines completed oracle lean outputs with surrogate predictions for checks still unfinished given a cut-off runtime. They show that training with this hybrid reward system (using their surrogate) leads to a prover that achieves almost the same miniF2F pass@1 performance as training with oracle lean rewards.

Critical strengths
  • The motivation is clear and well articulated throughout the paper. Verification can be slow and delay reward computation during RL training.
  • The authors construct a 4.84M training corpus and an evaluation benchmark that could benefit others working in this direction.
  • I appreciate that the authors test the neural surrogate using the hybrid reward system on the motivating problem.
Critical weaknesses
  • Design choices: Although the paper motivates the two-model design, I would like to understand how it compares with fine-tuning an existing theorem prover or a stronger reasoning model. i.e. what advantages does a two stage design provides vs fine-tuning a stronger model on the same larger corpus ? Similarly, it would be useful to evaluate how stronger models (either open-source or frontier models/reference) perform on this task off-the-shelf, either as baselines or starting points.
  • Additional usages of the neural surrogate. Could the learned surrogate also be evaluated end to end as a way to make test-time scaling cheaper? The motivation could be similar evaluating more proof candidates, allowing a larger K in pass@K. I feel this could strengthen the paper.
  • No discussion on reward hacking. How susceptible is the neural surrogate to reward hacking? I imagine the hybrid reward helps mitigate this, but it would be interesting to discuss whether it leaves the underlying reward-hacking problem “unresolved”. Similarly, what would happen if GRPO used only the neural surrogate? Would the prover learn to reward hack? How sensitive is this to the balance between Lean and surrogate rewards?
  • Only one 8B prover is tested in the GRPO experiments. What would be the impact wrt to above point if a stronger prover is used?
Questions for authors

see above

4 comments
AuthorsOpenReview ↗

## Post 8: Response to Reviewer H36J (1/4): Q1

Response to Reviewer H36J

Q1: How does the two-stage design compare with stronger models, off the shelf or fine-tuned?

Reviewer comment/question. Although the paper motivates the two-model design, how does it compare with fine-tuning an existing theorem prover or a stronger reasoning model? What advantages does a two-stage design provide over fine-tuning a stronger model on the same large corpus? It would also be useful to evaluate how stronger models, either open-source or frontier models, perform on this task off the shelf, either as baselines or as starting points.

Author response. Off the shelf, the two strong open models we test are far less accurate and slower than our two-stage cascade. In hybrid GRPO, the evaluator supplies the rewards for proofs that Lean has not finished checking, and each training step waits for the evaluator to finish. A good evaluator must therefore be both accurate and fast.

Off-the-shelf models. We thank the reviewer for this suggestion. We run two strong open models zero-shot, without any fine-tuning: Goedel-Prover-V2-8B, a strong Lean prover, and Qwen3-32B with thinking, a general reasoning model. Each model reads a Lean program and answers whether Lean accepts it. We use the 2,004 programs that Section 5.2 samples from the outputs of three public provers to assess feedback quality. Half are valid and half invalid, so random guessing gives 50% accuracy. We score our own models on the same programs (Table R6).

Table R6. Accept/reject predictions on the 2,004 programs of Section 5.2 (1,002 valid, 1,002 invalid). The paper measures feedback quality on these programs. Here we score accept/reject predictions on the same programs for all four evaluators. Accuracy: share of programs with the correct prediction. Recall: share of valid programs accepted. FPR (false-positive rate): share of invalid programs accepted. An output without a prediction counts as an error in accuracy but not as an acceptance in FPR. Among the 1,002 invalid programs, Goedel-Prover-V2-8B gives no prediction for 315 and correctly rejects 149. Qwen3-32B gives no prediction for 56 and correctly rejects 742. Our feedback model predicts Lean's goal states and error messages, and the cascade screens candidates with our 8B classifier first. On these programs, the cascade's accuracy happens to equal its 87.8% on the full benchmark (Table 3). Throughputs come from the serving runs in Table 6.

EvaluatorAccuracy (%)Recall (%)FPR (%)Candidates per second
Goedel-Prover-V2-8B, zero-shot50.987.053.71.15 on one H100
Qwen3-32B with thinking, zero-shot69.464.820.40.90 on two H100s
Our 4B feedback model86.098.025.93.27 on one H100
Our cascade87.896.721.13.48 on one H100

Goedel-Prover-V2-8B is at chance level: it accepts 87.0% of valid programs but correctly rejects only 149 of the 1,002 invalid ones. Qwen3-32B is more careful but accepts only 64.8% of valid programs. Our cascade is 18.4 points more accurate than Qwen3-32B and accepts 96.7% of valid programs at a similar false-positive rate (21.1% against 20.4%). It also processes 3.48 candidates per second on one H100, against 0.90 for Qwen3-32B on two.

Why two stages. In the cascade, the one-token 8B classifier first screens each candidate in one forward pass. Only the candidates it accepts go on to the slower 4B feedback model. The cascade accepts a candidate only if both models accept it. This design is more accurate than either model alone. On the paper's 68,932-program benchmark, the cascade reaches 87.8% accuracy, against 85.1% for the 4B feedback model and 83.1% for the larger 8B classifier with its threshold tuned on validation data (Table 3). On the 2,004 programs of Table R6, requiring both models to accept cuts the false-positive rate from 25.9% for the feedback model alone to 21.1%. The design is also faster than generating feedback for every candidate. In the serving run of Section 5.3, only 64.8% of candidates reach the feedback model. The cascade processes 3.48 candidates per second on one H100, against 3.27 for the feedback model alone (Table R6). After one epoch of hybrid GRPO, the cascade also trains a better prover than either of our classifiers. With the cascade, the prover reaches 62.2% miniF2F pass@1 (Table 7). In new runs with the same GRPO settings, the prover reaches 57.6% with the 8B classifier and 48.4% with ModernBERT, a 395M-parameter encoder.

AuthorsOpenReview ↗

## Post 9: Response to Reviewer H36J (2/4): Q1 continued, and Q2

Starting from a prover. We also train the feedback model from a theorem prover: Qwen3-4B-Base after the same continued pretraining, first trained as a prover on the valid proofs that Section 3.1 uses for its sampler, then trained to predict feedback like our feedback model. On the 2,004 programs of Table R6, it matches our feedback model exactly: 86.0% accuracy, 98.0% recall and a 25.9% false-positive rate. On these programs, the extra prover training adds nothing on top of our compiler-labelled corpus.

Q2: Can the evaluator make test-time scaling cheaper?

Reviewer comment/question. Could the learned surrogate also be evaluated end to end as a way to make test-time scaling cheaper? The motivation would be similar: evaluating more proof candidates, allowing a larger K in pass@K.

Author response. The paper already tests this use on saved prover outputs (Section 5.1, "Ranking candidates for verification", and Appendix F). At test time, a prover samples many candidate proofs per problem, and checking all of them with Lean is costly. The cascade can rank the candidates so that Lean checks only the most promising few. In the paper, we rank the saved candidates of three public provers, about 28 per problem, and count how often the top 1, 2 or 4 include a valid proof (Table R7).

Table R7. Share of problems (%) for which a valid proof is found when Lean checks only the top-ranked candidates, among the problems with at least one valid candidate: 1,125 on miniF2F and 173 on ProofNet, each paired with one prover (Appendix F). Random order: expected share when Lean checks the same number of candidates in random order.

BenchmarkLean checks per problemCascade ranking (%)Random order (%)
ProofNet157.247.9
ProofNet272.363.2
miniF2F183.979.3
miniF2F495.292.2

With the same number of Lean checks, cascade ranking finds a valid proof more often than random order. On ProofNet, checking only the top-ranked candidate finds a valid proof for 57.2% of the problems that have one, against 47.9% in random order, while using 3.7% of the Lean calls needed to check every candidate. The cascade is also fast: it processes 3.48 candidates per second on one H100, 4.2× the throughput of our 64-worker Lean pool (Table 6). Running this live, with more sampled candidates per problem and the cascade choosing which ones Lean checks, is a natural next step.

AuthorsOpenReview ↗

## Post 10: Response to Reviewer H36J (3/4): Q3

Q3: How susceptible is the evaluator to reward hacking, with and without Lean?

Reviewer comment/question. How susceptible is the neural surrogate to reward hacking? The hybrid reward may help mitigate this, but does it leave the underlying reward-hacking problem unresolved? What would happen if GRPO used only the neural surrogate, and would the prover learn to reward hack? How sensitive is this to the balance between Lean and surrogate rewards?

Author response. The cascade resists reward hacking far better than the classifiers, even without Lean. We thank the reviewer for these questions, which make the paper more complete. We answer all four in turn.

  1. In hybrid training, reward hacking against the cascade plateaus from step 125 to step 250, beyond one epoch (Table R8).
  2. The hybrid reward limits what the policy can exploit: by step 250, only 17.0% of invalid rollouts receive a positive reward, and the policy does not learn to escape the Lean checker (Table R8).
  3. Trained on the cascade alone, the prover exploits it more than in hybrid training, but less than either classifier with Lean (Table R8). Its miniF2F pass@1 still rises from 53.8% to 59.0% after one epoch (Table R9).
  4. Keeping Lean's completed verdicts lifts pass@1 from 59.0% to the Lean-only level (Table R9).

In hybrid training, Lean and the evaluator score each batch in parallel. With the cascade, the evaluator finishes after about 90 s on average. We call this moment the cutoff. Every rollout that Lean has checked before the cutoff gets Lean's verdict, and the evaluator scores the rest.

Measuring reward hacking. A policy that hacks the reward learns to write invalid proofs that the evaluator accepts, so the evaluator's false-positive rate on the policy's rollouts rises. We continue one hybrid run per evaluator to step 250, beyond the one-epoch checkpoint of about 170 steps. We also train one run on the cascade alone for one epoch. After training, we check every rollout of the listed steps with Lean. We then compare both the evaluator's verdicts and the training reward with Lean's verdicts (Table R8).

Table R8. Reward hacking during GRPO, measured on all rollouts of the given step with Lean labels obtained after training. One training run per row group. Accuracy describes the accuracy of the model's predictions with respect to Lean's verdicts. FPR (false-positive rate) describes the share of invalid rollouts that the model accepts. Lean share: share of rollouts whose Lean verdict arrives before the cutoff, so they get Lean's verdict. Reward FPR: share of invalid rollouts that receive a positive training reward. All values are percentages.

Training rewardStepAccuracyFPRLean shareReward FPR
Hybrid, cascade2587.917.842.810.7
12581.925.742.815.4
25081.728.943.717.0
Cascade only, no Lean2586.419.80.019.8
12576.834.60.034.6
17074.938.10.038.1
Hybrid, one-token 8B classifier2581.137.041.922.4
12568.056.940.235.2
25059.674.140.145.5
Hybrid, ModernBERT classifier2558.287.636.755.8
12529.099.328.570.5
25033.799.633.465.6
AuthorsOpenReview ↗

## Post 11: Response to Reviewer H36J (4/4): Q3, continued, and Q4

In hybrid training, reward hacking against the cascade plateaus. The cascade's accuracy drops from 87.9% at step 25 to 81.9% at step 125, then holds at 81.7% at step 250. Its FPR rises by 7.9 points over the 100 steps from step 25 to 125, then by only 3.2 points over the next 125 steps, and stays below 30%. By step 250, the one-token 8B classifier accepts 74.1% of invalid rollouts and ModernBERT accepts 99.6%.

In the hybrid cascade run, the Lean share stays stable, so the policy does not learn to escape the Lean checker. One way to exploit the hybrid reward is to write proofs that Lean cannot finish before the cutoff, so that only the cascade judges them. This would lower the Lean share, which instead stays between 42.8% and 43.7% at steps 25, 125 and 250.

Without Lean, the prover exploits the cascade more, but far less than the classifiers. The cascade's FPR rises from 19.8% at step 25 to 34.6% at step 125 and 38.1% at step 170. At step 125, the cascade's FPR in hybrid training is 25.7%, while the classifiers with Lean reach 56.9% (one-token 8B) and 99.3% (ModernBERT).

The hybrid protocol limits what the policy can exploit. Every Lean verdict that arrives before the cutoff overrides the cascade's prediction. As a result, only 10.7%, 15.4% and 17.0% of invalid rollouts receive a positive reward at steps 25, 125 and 250 of hybrid training with the cascade (Reward FPR, Table R8). Without Lean, this share is 34.6% at step 125, more than twice the 15.4% of hybrid training. The protocol also allows further safeguards, which we leave to future work:

  • A later cutoff gives Lean more time and raises the Lean share.
  • Lean checks can continue after the cutoff, and a later Lean rejection of a proof that the cascade accepted can trigger an extra penalty.
  • The evaluator can be retrained periodically on Lean-labelled rollouts.
  • A random sample of the rollouts scored by the cascade can be rechecked with a longer Lean budget.

Balance between Lean and the surrogate. Table R9 compares training with no Lean rewards, with hybrid rewards and with Lean rewards only.

Table R9. miniF2F pass@1 and pass@32 (%) on all 488 problems, before and after one epoch of GRPO. Mean ± standard deviation over five evaluations (pass@1) and three (pass@32) of each checkpoint. Rows marked † repeat Table 7 of the paper.

Training rewardPass@1 (%)Pass@32 (%)
Initial checkpoint, before GRPO†53.8 ± 0.672.8 ± 0.3
Cascade only, no Lean59.0 ± 0.670.7 ± 0.1
Hybrid, cascade, first seed†62.2 ± 0.872.4 ± 0.4
Hybrid, cascade, second seed62.7 ± 0.772.1 ± 0.6
Lean only, 60 s timeout†62.4 ± 0.671.3 ± 0.8

Keeping Lean's completed verdicts lifts pass@1 to the Lean-only level. In the hybrid cascade run, Lean supplies about 43% of the rewards (Lean share, Table R8). Hybrid training reaches 62.2% and 62.7% pass@1 over two seeds, against 59.0% without Lean and 62.4% with Lean only. The cutoff sets this balance: waiting longer for Lean can only raise the Lean share, at the cost of a longer step.

Q4: Would a stronger prover exploit the evaluator more?

Reviewer comment/question. Only one 8B prover is tested in the GRPO experiments. What would be the impact on reward hacking if a stronger prover were used?

Author response. "Stronger" can mean a more capable prover or a larger one.

A more capable prover. The cascade is most accurate on the prover outputs that are most often valid. Table 4 evaluates it on 68,932 candidates from three public provers. Its accuracy is 86.5% on Pythagoras-Prover-4B (32.8% valid), 87.8% on DeepSeek-Prover-V2-7B (40.8% valid) and 89.3% on Goedel-Prover-V2-8B (53.0% valid). Our own prover also becomes more capable during GRPO, from 53.8% to 62.2% miniF2F pass@1, while reward hacking against the cascade levels off (our answer to Q3).

A larger prover. Whether a much larger prover, such as a 32B or 100B model, could learn where the evaluator is weak is a fair question. Training such a prover with GRPO needs more GPUs than we had available, and we leave it to future work. Larger runs would also help the hybrid protocol. The evaluator runs on the same GPUs as training, so more GPUs would let the cascade score each batch faster, or leave room for a larger evaluator.

We thank the reviewer again for these helpful questions. We hope our answers address the reviewer's concerns, and we are happy to answer any further questions.

Reviewer_EwAxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a learned Lean feedback evaluator to reduce the verification bottleneck in reinforcement learning for formal theorem proving. The authors train validity classifiers and a generative feedback model on 4.84 million Lean-labeled programs. A two-stage cascade first screens candidates using a classifier and then predicts compiler feedback for the remaining candidates. The resulting evaluator achieves 87.8% verdict accuracy on a benchmark of 68,932 proof candidates. Speculative decoding accelerates feedback generation, and a hybrid reward protocol combines completed Lean verdicts with neural predictions for unfinished checks. In GRPO training, the hybrid approach reduces training step time by approximately 31% while achieving similar miniF2F pass@1 to standard Lean-only rewards. Additional experiments investigate feedback quality, inference throughput, candidate ranking, and the trade-off between verification time and reward accuracy.

Critical strengths

1. A practically relevant problem.

The paper addresses a concrete bottleneck in reinforcement learning for formal theorem proving, where expensive Lean verification delays reward computation. The proposed hybrid evaluation pipeline offers a practical way to trade some reward accuracy for reduced training latency.

2. Meaningful end-to-end evaluation.

The authors evaluate both neural feedback prediction and its effect on GRPO training. In particular, comparing hybrid rewards with a shorter-timeout Lean-only baseline provides useful evidence that learned feedback can preserve training quality under a reduced reward-computation budget.

Critical weaknesses

1. Limited methodological novelty and insufficient component isolation.

The proposed framework mainly combines existing techniques, including learned verifier prediction, classifier cascades, and speculative decoding. Its contribution is primarily the integration of these techniques into a hybrid reward pipeline rather than a fundamentally new evaluation or learning method. Moreover, although the feedback model generates detailed Lean diagnostics, the GRPO pipeline uses only its binary validity predictions. The paper does not compare hybrid GRPO training using the classifier alone, leaving the necessity of the more expensive feedback-generation stage insufficiently established.

2. Insufficient evaluation of reward reliability and training stability.

The hybrid framework replaces unfinished Lean checks with neural predictions, introducing potentially incorrect rewards into GRPO training. Such errors can distort group-relative advantages, reinforce invalid proofs, or suppress valid ones. More importantly, the neural evaluator is preferentially applied to slow or difficult candidates whose distribution may differ substantially from the overall evaluation benchmark. The reported hybrid agreement of 92.5% includes candidates already verified by Lean and therefore does not establish the evaluator's reliability on this critical subset. With only one GRPO training run per condition, the paper also provides limited evidence that reward errors do not accumulate or become exploitable during longer training.

Questions for authors

Q1. Is the generative feedback model necessary for GRPO acceleration?

Since the GRPO pipeline ultimately uses binary validity rewards, could the authors compare hybrid training with classifier-only predictions against the proposed cascade? This would help establish whether generating detailed Lean feedback provides an independent benefit beyond direct validity prediction.

Q2. What is the overall computational benefit after accounting for evaluator training?

Training the evaluators requires substantial GPU resources, while the demonstrated GRPO speedup is measured after these models have been trained. How many downstream training runs are needed to amortize the upfront cost? Could the authors provide an end-to-end compute comparison, including evaluator training and data collection?

Q3. How do neural reward errors affect GRPO training?

Since the evaluator is primarily used for candidates whose Lean checks remain unfinished, could the authors report false-positive and false-negative rates specifically on this subset? More importantly, how often do predicted rewards change the ordering or signs of group-relative advantages compared with Lean rewards? Could the authors investigate whether such errors accumulate or induce reward hacking during longer GRPO training?

Q4. How robust are the downstream results across training seeds and longer training?

The reported miniF2F results are based on one GRPO training run per reward condition, with repeated evaluations of the resulting fixed checkpoints. Could the authors provide independent training runs or longer training experiments to determine whether the observed speed-quality trade-off remains stable?

7 comments
AuthorsOpenReview ↗

## Post 3: Response to Reviewer EwAx (1/5): Q1

Response to Reviewer EwAx

Q1: Is predicting Lean feedback new, and does GRPO need it?

Reviewer comment/question. The framework mainly integrates existing techniques: learned verifier prediction, classifier cascades and speculative decoding. GRPO uses only the binary verdicts, not the generated diagnostics. Could the authors compare hybrid training with classifier-only predictions, to show whether generating Lean feedback gives an independent benefit beyond direct validity prediction?

Author response. Yes, generating Lean feedback gives an independent benefit beyond direct validity prediction, both in GRPO training (Table R3) and on the paper's evaluation benchmark (Table 3). Our central contribution is to predict Lean's compiler feedback, its goal states and error messages, rather than only the verdict. We read the verdict off this predicted feedback (Section 4.2). To our knowledge, predicting compiler feedback to check proof correctness is new. The naive approach to train a simple classifier gives much weaker results.

Our second main contribution is the hybrid reward protocol (Section 6). Lean and the evaluator score each GRPO batch in parallel, and every Lean verdict that arrives before the evaluator finishes replaces the evaluator's prediction. This cuts the GRPO step time by about 31%, from 1,050 s to 720 s, with nearly unchanged miniF2F pass@1 (62.2% against 62.4%, Table 7). We also contribute a 4.84M-program compiler-labelled corpus and a 68,932-program benchmark built from the outputs of three public provers.

GRPO receives only the binary verdict from the evaluator, our cascade. The cascade accepts a proof only if both the one-token 8B classifier and the 4B feedback model accept it. The question is whether predicting the feedback gives a better verdict for training than predicting validity directly. To test this, we train with each of our two classifiers in place of the cascade, with the same GRPO settings (Table R3). Both predict validity directly. The one-token 8B classifier is trained to output a single true or false token, and ModernBERT is an encoder.

Table R3. miniF2F pass@1 and pass@32 (%) on all 488 problems, before and after one epoch of GRPO. Mean ± standard deviation over five evaluations (pass@1) and three (pass@32) of each checkpoint. Rows marked † repeat Table 7 of the paper.

Reward sourcePass@1 (%)Pass@32 (%)
Initial checkpoint, before GRPO†53.8 ± 0.672.8 ± 0.3
Lean only, 60 s timeout†62.4 ± 0.671.3 ± 0.8
Hybrid, cascade†62.2 ± 0.872.4 ± 0.4
Hybrid, one-token 8B classifier (threshold 0.86)57.6 ± 0.870.3 ± 0.4
Hybrid, ModernBERT classifier (threshold 0.5)48.4 ± 0.663.1 ± 0.4
Cascade only, no Lean59.0 ± 0.670.7 ± 0.1

The feedback cascade clearly beats both classifiers in GRPO. The one-token 8B classifier is the cascade's first stage. Used alone, at its calibrated threshold chosen on validation data, it reaches 57.6% pass@1 in hybrid training, against 62.2% for the full cascade. This 4.6-point gap is much larger than the 0.5-point difference between two hybrid cascade runs with different seeds (answer to Q4). Hybrid training with ModernBERT ends at 48.4% pass@1, below the initial checkpoint (53.8%). Even with no Lean verdicts at all, the cascade trains a better prover (59.0%) than either classifier with Lean.

Our reward-hacking analysis in the answer to Q3 also shows that, during training, the policy exploits both classifiers far more than the cascade. Predicting feedback also gives a better verdict outside training. On the 68,932-program benchmark, our 4B feedback model alone is more accurate than the larger 8B classifier with a calibrated threshold (85.1% against 83.1%), and the cascade reaches 87.8% (Table 3).

AuthorsOpenReview ↗

## Post 4: Response to Reviewer EwAx (2/5): Q2

Q2: When does evaluator training pay for itself?

Reviewer comment/question. Training the evaluators requires substantial GPU resources, while the demonstrated GRPO speedup is measured after these models have been trained. How many downstream training runs are needed to amortize the upfront cost? Could the authors provide an end-to-end compute comparison, including evaluator training and data collection?

Author response. Against Lean-only training with the 60 s timeout, evaluator training pays for itself within six one-epoch GRPO runs. Data collection and evaluator training are one-time costs. We will release the corpus with its Lean feedback and the trained evaluators, so others can use them without paying either cost. The full cost comparison is as follows.

  • Data, once. The corpus (Section 3.1) combines public Lean proofs with about 2.8M proof attempts that we generated. Generating them took about 3,400 H100 GPU-hours. Checking all 4.84M programs with Lean took about 25,000 CPU core-hours.
  • Evaluator training, once. Training all three evaluators, including continued pretraining, takes 740 H100 GPU-hours (Appendix B): 12 for ModernBERT, 192 for the one-token 8B classifier and 536 for the 4B feedback model.
  • Each GRPO run. One epoch of about 170 steps on eight H100s takes about 400 H100 GPU-hours with Lean-only rewards and 270 with hybrid rewards, including the evaluator's inference on the same GPUs (Appendix E). Each run saves about 130 H100 GPU-hours.
  • Break-even. Six runs save about 780 H100 GPU-hours, more than the 740 for evaluator training. Including data generation, about 32 runs recover the 4,140 H100 GPU-hours spent on data generation and evaluator training.

These costs are paid once, and our evaluator is not tied to our prover. The paper shows that it works on other provers without retraining: on the outputs of three public provers, the cascade reaches 86.5% to 89.3% accuracy (Table 4). The saving also grows with run length, since each training step is about 330 s shorter on average (1,050 s against 720 s, Table 7). At that rate, two three-epoch runs would cover evaluator training.

AuthorsOpenReview ↗

## Post 5: Response to Reviewer EwAx (3/5): Q3

Q3: How reliable are predicted rewards, and do errors accumulate?

Reviewer comment/question. Neural reward errors can distort group-relative advantages. Predicted rewards mostly replace slow or difficult checks, and the 92.5% hybrid agreement includes candidates already verified by Lean. Could the authors report false-positive and false-negative rates on this subset, how often predicted rewards change the signs or ordering of group-relative advantages, and whether errors accumulate or induce reward hacking in longer GRPO training?

Author response. We answer the three parts of the question in turn.

  1. On the labelled rollouts whose reward comes from the cascade, its false-positive and false-negative rates are within 1.3 points of those on all labelled rollouts. Among resolved timeouts, it accepts 80.1% of the valid proofs, which Lean-only training never rewards (Table R4).
  2. Compared with exact Lean rewards, hybrid rewards keep the sign of the GRPO advantage for 90.2% of labelled rollouts and flip it for 6.8%.
  3. Up to step 250, beyond one epoch, reward hacking against the cascade plateaus after step 125, and the training reward accepts only 17.0% of invalid rollouts at step 250 (Table R5). In a new step-250 evaluation, the prover keeps its miniF2F scores.

False-positive and false-negative rates where predictions supply the reward. In hybrid GRPO, Lean and the cascade score each batch in parallel. The cascade finishes about 90 s after scoring starts, on average. At this cutoff, each rollout that Lean has already checked gets Lean's verdict. The rest, including timeouts, get the cascade's verdict. Section 6.2 replays this rule on 24,576 saved training rollouts, using the Lean verdicts and completion times recorded during training and a cutoff fixed at 90 s. Table R4 gives the cascade's false-positive and false-negative rates on the rollouts the cascade scores, next to its rates on all labelled rollouts.

Table R4. Cascade error rates against Lean in the replay of Section 6.2. All labelled: the 23,489 rollouts with an original Lean verdict. Scored by the cascade: those whose Lean verdict arrives after the 90 s cutoff, so the cascade's verdict is their reward. Timeouts: proofs that exceed Lean's 60 s limit per proof, labelled afterwards by a 600 s recheck. FNR (false-negative rate): share of valid rollouts that the cascade rejects. FPR (false-positive rate): share of invalid rollouts that it accepts. When the cascade supplies the reward, a false negative loses a deserved reward and a false positive gets an undeserved one.

RolloutsValidInvalidFNR (%)FPR (%)
All labelled15,0098,48014.911.6
Scored by the cascade7,3784,42716.212.7
Timeouts resolved at 600 s46333419.916.8

Rollouts whose Lean verdict arrives after the cutoff are barely harder for the cascade. On them, its FNR is 16.2% and its FPR 12.7%, against 14.9% and 11.6% on all labelled rollouts (Table R4).

The training reward is wrong much less often than the cascade's own verdicts. Among labelled rollouts, every reward that the cascade does not supply comes from Lean itself and matches Lean's label. The only reward errors are therefore the cascade's errors on the rollouts whose reward it supplies. Over all 23,489 labelled rollouts, the hybrid reward thus rejects 8.0% of valid rollouts (1,198 of 15,009) and accepts 6.7% of invalid ones (564 of 8,480), against 14.9% and 11.6% for the cascade's own verdicts (Table R4). These errors give the 92.5% hybrid agreement that the reviewer cites (Section 6.2).

On timeouts, the cascade recovers rewards that Lean-only training loses. Lean-only training gives no positive reward to a proof that times out, even when it is valid. The 600 s recheck finds 463 valid proofs among the timeouts, and the cascade accepts 371 of them (80.1%, Table R4). Most of the resolved timeouts it accepts are valid: 371 of 427 (86.9%).

Comments on the paper

2 comments
AuthorsOpenReview ↗

## Post 1: General Response to All Reviewers (1/2)

General Response to All Reviewers

We thank the reviewers for their careful reading and constructive questions. Below, we report four new GRPO training runs (Table R1) and an analysis of reward hacking (Table R2).

1. New GRPO runs. We repeat hybrid training with a second seed. We also replace the cascade with each of our two classifiers, the one-token 8B classifier and ModernBERT, which predict validity directly without generating Lean feedback. Finally, we train with the cascade alone, without any Lean verdicts.

Table R1. miniF2F pass@1 and pass@32 (%) on all 488 problems, before and after one epoch of GRPO. Mean ± standard deviation over five evaluations (pass@1) and three (pass@32) of each checkpoint. Rows marked † repeat Table 7 of the paper. Each GRPO row is one training run.

Reward sourcePass@1 (%)Pass@32 (%)
Initial checkpoint, before GRPO†53.8 ± 0.672.8 ± 0.3
Lean only, 60 s timeout†62.4 ± 0.671.3 ± 0.8
Hybrid, cascade, first seed†62.2 ± 0.872.4 ± 0.4
Hybrid, cascade, second seed62.7 ± 0.772.1 ± 0.6
Hybrid, one-token 8B classifier57.6 ± 0.870.3 ± 0.4
Hybrid, ModernBERT classifier48.4 ± 0.663.1 ± 0.4
Cascade only, no Lean59.0 ± 0.670.7 ± 0.1

A second seed reproduces the result. It reaches 62.7% pass@1 and 72.1% pass@32, against 62.2% and 72.4% for the first seed and 62.4% and 71.3% for Lean-only training.

Predicting Lean feedback matters for training. With a classifier in place of the cascade, hybrid training reaches only 57.6% pass@1 (one-token 8B) and 48.4% (ModernBERT), against 62.2% with the cascade. With ModernBERT, the prover even ends below the initial checkpoint (53.8%).

Even without Lean, the cascade trains a better prover than either classifier with Lean. Training on cascade verdicts alone reaches 59.0% pass@1.

AuthorsOpenReview ↗

## Post 2: General Response to All Reviewers (2/2)

2. Reward hacking. A learned reward can be exploited: the policy may learn to write invalid proofs that the evaluator accepts. To measure this, we train one hybrid run per evaluator to step 250, and the cascade-only run for one epoch, to step 170. After training, we check every rollout of the listed steps with Lean. We then compare both the evaluator's verdicts and the training reward with Lean's verdicts (Table R2).

Table R2. Reward hacking during GRPO, measured on all rollouts of the given step with Lean labels obtained after training. One training run per row group. Accuracy describes the accuracy of the model's predictions with respect to Lean's verdicts. FPR (false-positive rate) describes the share of invalid rollouts that the model accepts. Lean share: share of rollouts whose Lean verdict arrives before the evaluator finishes, so they get Lean's verdict. Reward FPR: share of invalid rollouts that receive a positive training reward. All values are percentages.

Training rewardStepAccuracyFPRLean shareReward FPR
Hybrid, cascade2587.917.842.810.7
12581.925.742.815.4
25081.728.943.717.0
Cascade only, no Lean2586.419.80.019.8
12576.834.60.034.6
17074.938.10.038.1
Hybrid, one-token 8B classifier2581.137.041.922.4
12568.056.940.235.2
25059.674.140.145.5
Hybrid, ModernBERT classifier2558.287.636.755.8
12529.099.328.570.5
25033.799.633.465.6

With the cascade, the training reward stays hard to exploit. In hybrid training with the cascade, only 10.7% of invalid rollouts receive a positive reward at step 25, 15.4% at step 125 and 17.0% at step 250. With the classifiers, this share reaches 45.5% (one-token 8B) and 65.6% (ModernBERT) by step 250.

Reward hacking against the cascade plateaus. The cascade's own FPR rises from 17.8% at step 25 to 25.7% at step 125, then only to 28.9% at step 250. Its accuracy holds at about 82% from step 125 on. By step 250, the classifiers accept 74.1% and 99.6% of invalid rollouts.

Without Lean, the cascade rewards more than twice the share of invalid rollouts. At step 125, 34.6% of invalid rollouts receive a positive reward when the cascade alone supplies the reward, against 15.4% in hybrid training with the cascade.

In the cascade run, the Lean share stays stable, so the policy does not learn to escape the Lean checker. One way to exploit the hybrid reward is to write proofs that Lean cannot finish before the evaluator does, so that only the evaluator judges them. This would lower the Lean share. Instead, it stays at 42.8–43.7% across steps 25, 125 and 250.

The hybrid protocol also allows further safeguards, which we leave to future work: giving Lean more time before the evaluator's verdicts are used, an extra penalty for proofs that the evaluator accepts and Lean later rejects, periodic retraining of the evaluator on Lean-labelled rollouts, and random rechecks of evaluator-scored rollouts with a longer Lean budget.

We thank all reviewers again for their constructive questions. We are happy to answer any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.