- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SketchForge, a reinforcement learning method for training the sketcher in agentic theorem proving in Lean, which is the model that breaks a theorem into lemmas and writes the main proof on top of them. Instead of rubric-based or AST-based rewards, SketchForge rewards a sketch only when a frozen prover proves all of its lemmas and the assembled proof passes Lean. The authors use a moderately capable prover (Qwen3.5-9B), arguing that a strong prover also rewards sketches that barely simplify the problem, and expose the verification as a tool so that the sketcher learns to revise its sketch from lemma-level feedback. After RL training on about 11K hard Lean statements autoformalized from public math problems, SketchForge-27B raises Pass@32 over Qwen3.5-27B in the same agent by 4.7% on average across five benchmarks (ProofNet, ProverBench, PutnamBench, MathOlympiad and MiniF2F).
Critical strengths
The paper tackles an important problem in training agentic provers with RL, which is how to reward intermediate steps such as lemma decomposition in a reliable way. The motivation is well presented, with concrete examples of rubric-based and AST-based rewards accepting false or trivial lemmas, and letting a prover and Lean verify every lemma is a simple and convincing fix. I liked the care in the verification step, where the final proof is rechecked without admitted lemmas and the statement and axioms are checked as well. The improvement over the base model is consistent across all five benchmarks and holds even with one attempt per lemma, so it is not only a matter of a larger test-time budget. I also found it interesting that a stronger verification prover can lead to a worse sketcher, which contradicts the usual intuition that a stronger verifier leads to a better reward.
Critical weaknesses
- The choice of the training prover seems to matter the most, yet the mechanism behind it does not seem to be measured in detail. I notice in Table 3 that switching from the 9B to the 27B prover drops PutnamBench from 12.9 to 7.4 and below the untrained base model, which means that a wrong choice removes most of the training gain. The paper explains this through sketch effectiveness (Definition 4) but never estimates the expected difficulty reduction or describes how the ineffective sketches in Figure 1a are identified. The two theorems do not help much here because Theorem 1 follows from the coverage assumption and Theorem 2 is a general probability bound, so it is not really clear from them how effectiveness relates to the prover choice or why a moderate prover should work best. The result could also be understood as the 9B prover simply producing more GRPO groups with mixed rewards. If the problem is that a strong prover rewards sketches that restate the theorem, the most direct control would be the strong prover plus a filter on restated lemmas, which the existing BEq check could already provide. Without it the prover choice looks more like a setting tuned for this sketcher, and it is unclear how to choose the prover for other sketchers or datasets.
- The comparison with LongCat-Flash-Prover in Table 2 is incomplete. Section 5.3 mentions that the method gets the best Pass@32 on ProofNet, ProverBench and PutnamBench among agentic systems including LongCat-Flash-Prover, whose two whole-proof rows in Table 2 are taken from OProver. When I checked the LongCat-Flash-Prover paper I found that it also reports a sketch-proof mode at Pass@32 that seems to follow the same lemma-then-prove idea and reaches 47.3, 66.5 and 28.9 on these three benchmarks (vs. 38.7, 61.2 and 12.9). I am not fully sure the settings match and the method does not need to beat this row considering the differences in model size and budget, but it would be good to list it and discuss the gap. The gap is also hard to judge without the cost of one Pass@32 sample, which may include up to 15 rounds with several prover calls per lemma, so reporting tokens or prover calls per problem would help a lot.
Questions for authors
- Did you try the strong prover with a filter that rejects lemmas equivalent to the main theorem, or a reward that takes the difficulty of each lemma into account? This question matters a lot for my evaluation, and I would be glad to raise my score if such a control shows that the gain comes from rewarding effective decompositions. It would also help if you could explain how the ineffective sketches in Figure 1a are counted.
- I would like to know which of the three components contributes the most, and which one you see as the weakest part that most needs improvement (even seems not a purely clear ablation, I can treat the gain as a compositional effect, which works for me, and I'm not asking/expect a fully clean separation). But two of the current ablations still make this hard to read. In Figure 4a the two runs already differ by about 19 points at step 0 because the rubric baseline stops refinement once the judge accepts the sketch, and the single-round row in Table 4 matches the first round in Figure 5, so it seems to measure the number of test-time rounds rather than training with refinement. Is it possible to evaluate the rubric-trained checkpoint with the same protocol as SketchForge, so that only the training reward differs?
- I'm a bit confused about how much of the gain comes from the sketches themselves. Each trained model also proves its own lemmas at evaluation, and SketchForge-27B is also a better whole-proof prover than the base model (Table 6), so the sketcher and the prover change together in Table 2 and Table 3. Is there a result with the SketchForge sketcher and the base model as the prover, or with all rows of Table 3 evaluated with the same prover as in Figure 3?
Future work suggestions
- Since a wrong prover choice can remove most of the gain, adapting the prover or its budget during training, for example based on the share of groups with mixed rewards, would make the method easier to apply to new sketchers and datasets.
Minor in write-up
- I could not find which checkpoint is evaluated, the number of samples behind Pass@32, or the Mathlib version, and all results seem to come from a single training run.
- When comparing training data with OProver, it would be fair to mention that the 11K statements come from a Qwen3.5-Plus run over about 400K autoformalized problems.
- I notice some earlier work also trains lemma decomposition with RL, and some of it already discards lemmas that directly imply the theorem, so the related work could cover this line more comprehensively. There is also a small inconsistency between the Introduction, which describes LongCat-Flash-Prover and Goedel-Code-Prover as unifying decomposition and refinement in one model, and Section 2, where they are described as treating lemma decomposition as a standalone training process.
Dear Authors,
we have not yet received a response and would appreciate an update during the rebuttal period.
Thank you.