acceptodds
Under review as a conference paper at ICLR 2027

SketchForge: Prover-Guided Reinforcement Learning for Sketch Generation in Agentic Theorem Proving

Abstract

Sketch generation is a key capability of LLM-based theorem-proving agents. However, training sketch generation models remains hindered by unreliable correctness assessments, rewards misaligned with difficulty reduction, and limited lemma-level provability feedback for sketch refinement. To this end, we introduce SketchForge, an agentic reinforcement learning method for sketch generation. SketchForge comprises three components: (1) a prover-guided sketch verification framework; (2) sketch effectiveness discrimination through prover selection; and (3) a model-as-a-tool system for learning iterative sketch refinement from lemma-level feedback. Trained with our method, SketchForge-27B improves Pass@32 over its base model by an average of 4.7 percentage points across five benchmarks, matching or surpassing OProver-32B, the previous SOTA open-weight agentic prover of comparable size, with substantially less training data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning, Automated Theorem Proving, Formal Math

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GLA9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SketchForge, a reinforcement learning method for training the sketcher in agentic theorem proving in Lean, which is the model that breaks a theorem into lemmas and writes the main proof on top of them. Instead of rubric-based or AST-based rewards, SketchForge rewards a sketch only when a frozen prover proves all of its lemmas and the assembled proof passes Lean. The authors use a moderately capable prover (Qwen3.5-9B), arguing that a strong prover also rewards sketches that barely simplify the problem, and expose the verification as a tool so that the sketcher learns to revise its sketch from lemma-level feedback. After RL training on about 11K hard Lean statements autoformalized from public math problems, SketchForge-27B raises Pass@32 over Qwen3.5-27B in the same agent by 4.7% on average across five benchmarks (ProofNet, ProverBench, PutnamBench, MathOlympiad and MiniF2F).

Critical strengths

The paper tackles an important problem in training agentic provers with RL, which is how to reward intermediate steps such as lemma decomposition in a reliable way. The motivation is well presented, with concrete examples of rubric-based and AST-based rewards accepting false or trivial lemmas, and letting a prover and Lean verify every lemma is a simple and convincing fix. I liked the care in the verification step, where the final proof is rechecked without admitted lemmas and the statement and axioms are checked as well. The improvement over the base model is consistent across all five benchmarks and holds even with one attempt per lemma, so it is not only a matter of a larger test-time budget. I also found it interesting that a stronger verification prover can lead to a worse sketcher, which contradicts the usual intuition that a stronger verifier leads to a better reward.

Critical weaknesses
  1. The choice of the training prover seems to matter the most, yet the mechanism behind it does not seem to be measured in detail. I notice in Table 3 that switching from the 9B to the 27B prover drops PutnamBench from 12.9 to 7.4 and below the untrained base model, which means that a wrong choice removes most of the training gain. The paper explains this through sketch effectiveness (Definition 4) but never estimates the expected difficulty reduction or describes how the ineffective sketches in Figure 1a are identified. The two theorems do not help much here because Theorem 1 follows from the coverage assumption and Theorem 2 is a general probability bound, so it is not really clear from them how effectiveness relates to the prover choice or why a moderate prover should work best. The result could also be understood as the 9B prover simply producing more GRPO groups with mixed rewards. If the problem is that a strong prover rewards sketches that restate the theorem, the most direct control would be the strong prover plus a filter on restated lemmas, which the existing BEq check could already provide. Without it the prover choice looks more like a setting tuned for this sketcher, and it is unclear how to choose the prover for other sketchers or datasets.
  2. The comparison with LongCat-Flash-Prover in Table 2 is incomplete. Section 5.3 mentions that the method gets the best Pass@32 on ProofNet, ProverBench and PutnamBench among agentic systems including LongCat-Flash-Prover, whose two whole-proof rows in Table 2 are taken from OProver. When I checked the LongCat-Flash-Prover paper I found that it also reports a sketch-proof mode at Pass@32 that seems to follow the same lemma-then-prove idea and reaches 47.3, 66.5 and 28.9 on these three benchmarks (vs. 38.7, 61.2 and 12.9). I am not fully sure the settings match and the method does not need to beat this row considering the differences in model size and budget, but it would be good to list it and discuss the gap. The gap is also hard to judge without the cost of one Pass@32 sample, which may include up to 15 rounds with several prover calls per lemma, so reporting tokens or prover calls per problem would help a lot.
Questions for authors
  1. Did you try the strong prover with a filter that rejects lemmas equivalent to the main theorem, or a reward that takes the difficulty of each lemma into account? This question matters a lot for my evaluation, and I would be glad to raise my score if such a control shows that the gain comes from rewarding effective decompositions. It would also help if you could explain how the ineffective sketches in Figure 1a are counted.
  2. I would like to know which of the three components contributes the most, and which one you see as the weakest part that most needs improvement (even seems not a purely clear ablation, I can treat the gain as a compositional effect, which works for me, and I'm not asking/expect a fully clean separation). But two of the current ablations still make this hard to read. In Figure 4a the two runs already differ by about 19 points at step 0 because the rubric baseline stops refinement once the judge accepts the sketch, and the single-round row in Table 4 matches the first round in Figure 5, so it seems to measure the number of test-time rounds rather than training with refinement. Is it possible to evaluate the rubric-trained checkpoint with the same protocol as SketchForge, so that only the training reward differs?
  3. I'm a bit confused about how much of the gain comes from the sketches themselves. Each trained model also proves its own lemmas at evaluation, and SketchForge-27B is also a better whole-proof prover than the base model (Table 6), so the sketcher and the prover change together in Table 2 and Table 3. Is there a result with the SketchForge sketcher and the base model as the prover, or with all rows of Table 3 evaluated with the same prover as in Figure 3?

Future work suggestions

  • Since a wrong prover choice can remove most of the gain, adapting the prover or its budget during training, for example based on the share of groups with mixed rewards, would make the method easier to apply to new sketchers and datasets.

Minor in write-up

  • I could not find which checkpoint is evaluated, the number of samples behind Pass@32, or the Mathlib version, and all results seem to come from a single training run.
  • When comparing training data with OProver, it would be fair to mention that the 11K statements come from a Qwen3.5-Plus run over about 400K autoformalized problems.
  • I notice some earlier work also trains lemma decomposition with RL, and some of it already discards lemmas that directly imply the theorem, so the related work could cover this line more comprehensively. There is also a small inconsistency between the Introduction, which describes LongCat-Flash-Prover and Goedel-Code-Prover as unifying decomposition and refinement in one model, and Section 2, where they are described as treating lemma decomposition as a standalone training process.
Reviewer_yNvrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SketchForge, where a sketcher generates auxiliary lemmas and constructs proofs of the master theorem based on these lemmas. Symbolic tactics and a frozen prover then attempt to prove each lemma. Lean 4 then checks the complete proof, the original theorem statement, and any newly added axioms. The selection of the prover used for training aims to balance two issues: excessively sparse rewards and rewards even for invalid decompositions. Through the model-as-a-tool interface, the sketcher can revise its output based on compilation errors and unproven lemmas.

Critical strengths
  • The authors investigated the problem of reward sparsity and low-quality factorization (e.g., repetitive primal theorems).

  • The proposed method outperforms Qwen3.5-27B on all five benchmarks.

Critical weaknesses
  • The results for other systems in Table 2 are referenced from OProver; therefore, the same Pass@32 does not necessarily mean the same cost. Comparisons should be made under similar budgets, and total token consumption, prover calls, and runtime should be reported.

  • In Section 5.4, the rubric baseline stops revision once the judge approves the sketch, while SketchForge can continue revision even after a lemma proof fails.

  • Definition 4 defines the reduction in proof difficulty based on a fixed prover distribution. Figure 1 does not adequately explain how invalid sketches are identified.

Questions for authors
  • The main experiments use SketchForge-27B as both the sketcher and the prover, but Appendix D checks whole-proof ability only on MiniF2F?
1 comment

Dear Authors,

we have not yet received a response and would appreciate an update during the rebuttal period.

Thank you.

Reviewer_zfDgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The authors develop an agentic RL method for generating and refining proof sketches in theorem proving. They identify limitations in prior work, including difficulties with reward assignment and a lack of refinement of sketches, and develop their method around a moderately capable prover tasked with proving the lemmas proposed by the sketcher, providing a reward based on proof verification in Lean. Feedback identifying the lemmas that the prover failed to prove allows the sketcher to refine its sketch. The authors report performance gains over the base model on five benchmarks.

Critical strengths
  • The motivation in relation to prior work, particularly the use of rubric-based LLM-as-a-judge evaluation, is well explained and makes the reward hacking issue clear.
  • Using a moderately capable model to assign rewards, thereby pushing the sketch model to learn to generate more accessible sketches, is an interesting idea.
Critical weaknesses
  • The presentation is unclear and hard to follow. The authors use heavy formalism for simple definitions that do not seem to require so much notation, such as Definitions 1-4.
  • Comparing systems such as Kimina-Prover with SketchForge at pass@32 does not seem entirely fair to me, since SketchForge includes multiple rounds of sketch refinement. The authors state that "evaluation allows up to 81,920 tokens (excluding prover tokens), 15 sketch-refinement rounds and 8 parallel proof attempts per lemma". Evaluating these systems under equal inference budgets would provide a fairer and more informative comparison. This particularly limits the interpretation of the second claim in Section 5.3: "SketchForge-27B surpasses larger open-weight theorem proving systems on three challenging benchmarks".
Questions for authors

Could you provide comparative evaluation results under equal inference budgets? Without this, it is difficult to assess the performance improvements reported by the authors relative to other systems.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.