acceptodds
Under review as a conference paper at ICLR 2027

Pushing the Math Reasoning Frontier by Making Beyond-Capability Problems Learnable

Abstract

Large language models (LLMs) have made substantial progress in mathematical reasoning, which is largely driven by reinforcement learning (RL). However, further pushing the mathematical reasoning frontier of LLMs through RL remains challenging, as models rarely generate successful rollouts on problems beyond their current capability frontier, resulting in a scarcity of effective learning signals. To address this challenge, we propose **LeanTree**, which decomposes beyond-capability problems into learnable subproblems. The major challenges in developing LeanTree are twofold. (1) LLM-based decompositions rely on stronger models and still introduce logical hallucinations. (2) The resulting subproblems lack a difficulty hierarchy, leaving the training order unclear. LeanTree leverages the intermediate reasoning structure of existing formal proofs (machine-verifiable representations of mathematical reasoning) to generate *reliable* and *hierarchical* subproblems. By training on these hierarchical subproblems, the model progressively acquires the reasoning skills needed to solve the corresponding beyond-capability problems. Extensive experiments on mathematical reasoning benchmarks demonstrate that LeanTree provides more effective learning signals and improves overall reasoning accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mathematical Reasoning, Large Language Models, Reinforcement Learning, Formal Proofs, Problem Decomposition, Curriculum Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cZo8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LeanTree extracts intermediate goals from existing Lean proofs and converts them into natural-language problems with verifiable answers. It uses proof-tree depth as a difficulty proxy and trains models through a curriculum progressing from deeper subproblems to original problems. Experiments across two model backbones and multiple benchmarks show improvements over direct RL and data-synthesis baselines. Curriculum ablations and held-out-root evaluations provide additional support for the approach.

Critical strengths
  1. Useful supervision from formal proof structure. Existing proofs provide explicit assumptions, intermediate targets, and dependencies, offering a principled basis for constructing auxiliary training tasks.
  2. Supportive empirical evaluation. Improvements across the evaluated configurations, together with same-data curriculum ablations and held-out-root evaluation, support the practical value of the method.
Critical weaknesses
  1. Restricted coverage of proof-derived skills. The pipeline retains tasks convertible into concrete answer-based questions and excludes pure proof tasks. It is unclear how much important intermediate reasoning is lost through this filtering and whether the retained tasks adequately cover the prerequisites of the original problems.
  2. Curriculum implementation and justification need clarification. The paper specifies the depth order but does not clearly explain training-step allocation, sampling, and transitions between stages. Furthermore, comparison with random ordering does not establish whether proof depth is preferable to model-dependent difficulty measures such as empirical success rate.
Questions for authors
  1. What proportion of intermediate goals is excluded because they cannot be converted into answer-based tasks? Could this remove important prerequisite skills?
  2. How are the 100 training steps allocated across curriculum stages, and what determines stage transitions and sampling within each stage?
  3. Have you compared depth-based scheduling with scheduling based on empirical success rates? Such a comparison would help establish the additional value of proof-tree depth.
3 comments
AuthorsOpenReview ↗

Response to Reviewer cZo8 Part (1/2)

Dear Reviewer cZo8,

Thank you for your valuable review and for recognizing how the assumptions, targets, and dependencies in a formal proof can supply useful intermediate supervision. We especially appreciate your observation that the same-data ablation and the held-out evaluation address different questions about the curriculum. Your questions give us an opportunity to clarify which reasoning skills survive conversion and how the learner encounters them during training.

We genuinely welcome your feedback throughout the discussion. If these clarifications address your concerns, we would be grateful if you would consider raising your score. If anything remains unclear, we would truly appreciate your further guidance.


W1 & Q1: Coverage of proof-derived skills. What proportion of intermediate goals is excluded because they cannot be converted into answer-based tasks? Could this remove important prerequisite skills?

Thank you for this valuable question.

TL;DR: Although answer-based filtering inevitably excludes some formal proof goals, the retained subproblems provide informative learning signals and help the model improve on the original difficult problems, as evidenced by Figure 2.

How much is filtered? We extract non-root nodes as candidate subproblems and apply rule-based and LLM-based pruning to remove trivial transformations, bookkeeping, and goals that cannot be faithfully converted into self-contained questions with concrete, verifiable answers. In our construction, we retain 4,772 quality-filtered subproblems. Approximately 19.7% are excluded during this process.

Does filtering remove important prerequisite skills? We acknowledge that some proof goals, particularly pure proof tasks or goals without uniquely determined concrete answers, cannot be directly converted into our answer-based RL format. However, retaining every formal goal is not necessarily beneficial: some represent proof-engineering steps or trivial transformations rather than transferable mathematical skills.

More importantly, Figure 2 provides process-level evidence that the retained subproblems supply useful prerequisite supervision. Under matched rollout budgets, LeanTree maintains higher reward informativeness than direct RL over most of training (Figure 2a), indicating that the synthesized tasks provide more informative reward contrasts for GRPO. When training returns to the original difficult problems at steps 600–700, LeanTree's mean reward recovers rapidly and surpasses direct RL during the latter part of this phase (Figure 2b). This demonstrates that the retained intermediate tasks provide useful preparation for subsequent learning on the original problems.

The held-out evaluation provides complementary evidence: LeanTree improves accuracy on original root problems from 24.02% to 28.03% over direct RL, and outperforms the size-matched Prompt LLM baseline at every proof depth (Table 4). Together, these results support the practical usefulness of the retained proof-derived subproblems, even though they do not imply that every prerequisite skill in the original proof is preserved.

AuthorsOpenReview ↗

Response to Reviewer cZo8 Part (2/2)

W2 & Q2: Stage allocation, sampling, and transitions. How are the 100 training steps allocated across curriculum stages, and what determines stage transitions and sampling within each stage?

Thank you for this valuable question.

TL;DR: We sample in descending order of proof depth. The six groups are ≥5 → 4 → 3 → 2 → 1 → 0, with all original problems assigned depth 0 (Section 3.3).

For the 100-step main run, the six groups account for 6, 3, 4, 4, 2, and 81 updates, respectively. We group retained subproblems by their depth in the reconstructed proof tree and pool all depths of 5 or greater in the first group. Stage transitions occur at the boundaries between consecutive depth groups in the sampling order. They require no reward or success-rate threshold. Within a group, examples share the same depth priority and are shuffled before batching. The standard Qwen/GRPO run uses a global batch of 256 prompts and generates eight responses per prompt (Appendix B.4).

Figure 2 uses a separate 700-step diagnostic run with a batch size of eight prompts. There, steps 600–700 put both methods on the difficult original problems so that we can compare their rewards on the same tasks (Appendix C.3).

W2 & Q3: Proof depth versus empirical success rate. Have you compared depth-based scheduling with scheduling based on empirical success rates? Such a comparison would help establish the additional value of proof-tree depth.

Thank you for this valuable question.

TL;DR: Proof-depth ordering provides both a difficulty signal and a structural learning hierarchy without additional rollout-based calibration. Unlike independent accuracy estimates, proof-tree depth also preserves the dependencies among intermediate reasoning steps.

Thank you for this valuable question. An accuracy-based curriculum requires evaluating each candidate subproblem to estimate its difficulty. For our 4,772 synthesized subproblems, even a single calibration pass with 8 rollouts per question would require 38,176 additional generations. In contrast, LeanTree obtains the depth of every subproblem directly during proof-tree reconstruction, requiring no additional rollouts for difficulty estimation.

More importantly, proof depth provides structural information that independent accuracy estimates cannot directly capture. A proof tree organizes intermediate goals hierarchically, reflecting how local reasoning steps contribute to higher-level goals. This allows LeanTree to organize training from foundational intermediate tasks toward the original problem, rather than merely sorting questions by their individual success rates.

Although we were unable to complete a direct comparison with accuracy-based scheduling within the rebuttal period, our existing experiments provide complementary evidence for the effectiveness of this design. Table 4 shows that initial-model accuracy increases monotonically from 21.61% on original root problems to 70.14% on subproblems at depth ≥5. Table 2 further shows that depth-based scheduling improves average accuracy from 68.91% to 71.14% compared with random ordering on exactly the same training data. Finally, Figure 2 shows that the model trained with LeanTree obtains more informative reward signals and adapts more effectively when returning to the original difficult problems.


Thank you again for your thoughtful review and for encouraging us to make the construction and training details more explicit.

AuthorsOpenReview ↗

Gentle Reminder: Author–Reviewer Discussion Ending Soon

Dear Reviewer cZo8,

Thank you again for your valuable review and constructive feedback. As the Author–Reviewer discussion period is coming to an end soon, we would like to gently remind you that we have posted our point-by-point responses to all your comments.

We would greatly appreciate it if you could review our responses before the deadline. If our clarifications have adequately resolved your concerns, we would be deeply grateful if you would consider raising your score. Should anything remain unclear, please let us know—we remain available and eager to engage in further discussion.

Best regards,

Authors

Reviewer_TvkgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LeanTree, a data synthesis and curriculum-learning framework for learning from mathematical problems that are beyond the current capability of an LLM. The paper is motivated by a limitation of group-relative RL algorithms such as GRPO: when every rollout for a difficult problem is incorrect, the group has no reward contrast and therefore provides little useful policy-gradient signal. LeanTree uses existing Lean proofs to address this issue. It reconstructs a tree of intermediate goal states from each formal proof, applies structural transformations and filtering to remove proof-execution artifacts and trivial goals, and translates the retained formal goals into self-contained natural-language problems with verifiable answers. The resulting problems are organized into a curriculum according to proof-tree depth, proceeding from deeper, more local goals toward the original root problems. The method is evaluated on Qwen2.5-7B-Instruct and DeepSeek-R1-Distill-Llama-8B, using GRPO and, for Qwen, BAPO. Across GSM8K, MATH500, AMC 2023, and AIME 2025, LeanTree improves average accuracy over direct RL, Prompt LLM decomposition, and SwS. Additional analyses suggest that deeper proof nodes are empirically easier, that the synthesized tasks provide more informative reward distributions, and that depth-based ordering outperforms randomly shuffling the same data.

Critical strengths

The paper addresses an important and well-motivated RLVR failure mode with a conceptually appropriate source of supervision. When all rollouts for a difficult problem fail, group-relative RL receives no reward contrast. LeanTree turns such problems into intermediate tasks that are more likely to produce mixed rollout outcomes. Using formal proof structure is especially appealing because the assumptions, targets, and dependencies of intermediate goals are more reliable than those produced by unconstrained natural-language decomposition.

Critical weaknesses

Limited empirical gains. LeanTree improves over direct RL by only 1.99–2.66 average points and over SwS by 1.15–1.98 points, despite requiring a complex pipeline involving Lean proof extraction, LLM filtering, translation, and curriculum construction. Moreover, the paper reports evaluation-sampling variance rather than variance across independent RL training runs, so it is unclear whether these modest gains are statistically robust.

Limited methodological novelty. The method largely combines existing ideas from DeepSeek-Prover-V2 (Lean subgoal decomposition and curriculum learning), SwS (failure-driven problem synthesis), TD-Grokking (hierarchical decomposition of zero-reward problems), and the broader neuro-symbolic paradigm exemplified by AlphaGeometry. The current ablations do not isolate whether formal proof structure provides benefits beyond generic decomposition and easy-to-hard training.

Outdated model evaluation. The experiments rely on Qwen2.5-7B and DeepSeek-R1-Distill-Llama-8B. Since LeanTree’s usefulness depends on the learner’s capability frontier, its gains may not transfer to stronger current models. Experiments on a modern matched-scale backbone, such as Qwen3-8B, and ideally a stronger model such as Qwen3.8-27B, are needed to establish current relevance and generality.

Questions for authors

None

4 comments
AuthorsOpenReview ↗

Response to Reviewer Tvkg Part (1/3)

Dear Reviewer Tvkg,

Thank you for your valuable review and for recognizing why formal proofs are an appealing source of supervision when all rollouts on a difficult problem fail. We especially appreciate your observation that a useful intermediate task needs to preserve the assumptions, target, and dependencies that connect it to the original solution. Your questions about the cost of obtaining this supervision and the evidence for its contribution helped us sharpen both points.

We genuinely welcome your feedback throughout the discussion. If the clarifications and additional comparison below address your concerns, we would be grateful if you would consider raising your score. If anything remains unclear, we would truly appreciate your further guidance.


W1: Limited empirical gains. LeanTree improves over direct RL by only 1.99–2.66 average points and over SwS by 1.15–1.98 points, despite requiring a complex pipeline involving Lean proof extraction, LLM filtering, translation, and curriculum construction. Moreover, the paper reports evaluation-sampling variance rather than variance across independent RL training runs, so it is unclear whether these modest gains are statistically robust.

Thank you for this insightful comment.

TL;DR: LeanTree achieves the highest average accuracy in all three settings despite limited proof coverage. Our additional experiment shows that broader source coverage improves solution-guided decomposition, suggesting that expanding LeanTree’s proof corpus could further improve its performance.

Proof availability limits which failures LeanTree can turn into training tasks. As discussed in Section 6, a hard problem can supply intermediate supervision only if it has a usable Lean proof. In the data construction, the initial model answered 10,186 problems incorrectly, but only 2,844 of those had usable proofs. These eligible roots represent 12.4% of the 22,993 original training problems and yielded 4,772 quality-filtered subproblems. The remaining 7,342 failures therefore contributed no proof-derived exercises. Even at this coverage, LeanTree outperforms direct RL, Prompt LLM, and SwS on average in every configuration (Table 1).

The additional experiment gives us a concrete reason to pursue broader coverage. Expanding Prompt LLM from 433 to the same 2,844 roots used by LeanTree improved its average accuracy in all three settings. LeanTree continued to lead on those same roots, as shown under W2 below. Pairing more of the learner’s remaining failures with usable proofs would allow LeanTree to teach intermediate skills from problems its curriculum currently cannot reach.

We agree that independent training runs would provide a direct estimate of training-seed variability. Appendix B.5 reports repeated evaluations of fixed checkpoints, and during rebuttal we prioritized the source-coverage comparison. The paper also provides complementary evidence for the learning effect you identified: under matched rollout budgets, LeanTree yields more informative reward contrasts over most of Qwen/GRPO training and higher mean reward later in the shared root-training phase (Figure 2). On held-out original problems, accuracy rises from 24.02% with direct RL to 28.03% with LeanTree (Table 4).

AuthorsOpenReview ↗

Response to Reviewer Tvkg Part (2/3)

W2: Limited methodological novelty. The method largely combines existing ideas from DeepSeek-Prover-V2 (Lean subgoal decomposition and curriculum learning), SwS (failure-driven problem synthesis), TD-Grokking (hierarchical decomposition of zero-reward problems), and the broader neuro-symbolic paradigm exemplified by AlphaGeometry. The current ablations do not isolate whether formal proof structure provides benefits beyond generic decomposition and easy-to-hard training.

Thank you for this insightful comment.

TL;DR: LeanTree turns an existing Lean proof into a curriculum for natural-language mathematical reasoning. Its checked goal states anchor each exercise in the assumptions of the original proof, reducing logical hallucinations. Its tree structure supplies a difficulty hierarchy without additional calibration rollouts.

First, LeanTree makes subproblem generation more reliable by grounding the decomposition in a machine-verifiable proof. Directly asking an LLM to decompose a problem, even with a worked solution, can introduce logical hallucinations. In Section 2.1, for example, inferring x=0x=0 from x2+y2=0x^2+y^2=0 relies on xx and yy being real. If a generated question drops that condition, x=1, y=ix=1,\ y=i is a counterexample. An answer checker could still reward “0” for this ill-posed question. LeanTree extracts the variable types, hypotheses, and target together from the Lean goal state. The synthesis model therefore has an explicit mathematical specification to preserve when writing the natural-language question.

This distinction matters for the comparison with SwS and TD-Grokking. SwS identifies concepts implicated in the learner’s failures and generates new problems about those concepts. LeanTree can identify a specific intermediate result used in the proof of the very problem the learner failed. TD-Grokking also produces answer-verifiable subproblems, but an LLM derives them from informal solution guides. In LeanTree, the proof already supplies the decomposition before the LLM translates it.

Second, LeanTree obtains a difficulty hierarchy directly from dependencies within each proof. In the same example, the branch proving x=0x=0 first establishes x2≤0x^2\leq 0. An accuracy estimate can tell us how often the learner solves either task. The proof also tells us how one result is used to obtain the other. LeanTree uses depth to schedule local prerequisites before returning to the harder root. TD-Grokking uses policy rollouts to calibrate which subproblems are learnable and which need further decomposition. LeanTree reads depth directly from the reconstructed tree, requiring no additional rollouts to estimate subproblem difficulty.

LeanTree uses formal proofs to teach a natural-language reasoner. DeepSeek-Prover-V2 decomposes a theorem into subgoals to construct a formal Lean proof. AlphaGeometry proposes auxiliary constructions during symbolic search for a geometry proof. For LeanTree, the formal proof is an offline input. We translate its retained goals into self-contained natural-language questions with automatically checkable answers. The learner trains on those questions with an ordinary final-answer reward and solves natural-language math problems at test time. It never needs to generate a Lean proof or invoke Lean during rollouts or inference.

Like LeanTree, Prompt LLM begins RL on synthesized subproblems before returning to the original problems. In our additional experiment, we also gave Prompt LLM the same 2,844 source problems as LeanTree, together with their worked solutions and final answers. Matching the source problems and this broad training order lets us assess what formal proof structure adds beyond solution-guided decomposition and subproblem-first training. LeanTree outperforms Prompt LLM in all three settings, as shown by the average accuracies (%) below.

Backbone / optimizerOriginal Prompt LLMPrompt LLM on the same 2,844 rootsLeanTree
Qwen2.5-7B-Instruct / GRPO68.8969.3271.14
Qwen2.5-7B-Instruct / BAPO69.6269.9371.88
DeepSeek-R1-Distill-Llama-8B / GRPO71.4772.0974.45

In a separate comparison that matches total training-set size, LeanTree also outperforms Prompt LLM on held-out problems at every proof depth, including the original roots (Table 4). This comparison partially isolates the contribution of formal proofs. Because Prompt LLM’s subproblems have no intrinsic proof-tree depth, the observed advantage reflects both the proof-derived tasks and their ordering within the curriculum.

AuthorsOpenReview ↗

Response to Reviewer Tvkg Part (3/3)

W3: Outdated model evaluation. The experiments rely on Qwen2.5-7B and DeepSeek-R1-Distill-Llama-8B. Since LeanTree’s usefulness depends on the learner’s capability frontier, its gains may not transfer to stronger current models. Experiments on a modern matched-scale backbone, such as Qwen3-8B, and ideally a stronger model such as Qwen3.8-27B, are needed to establish current relevance and generality.

Thank you for suggesting this comparison. We agree that evaluating LeanTree on a newer backbone would help clarify how its benefits change as the learner becomes stronger. Given the limited rebuttal window, we were unable to complete the additional RL runs on the suggested models. As discussed in Section 6, LeanTree draws its intermediate goals from existing Lean proofs, so the synthesis model can turn them into natural-language questions with automatically checkable answers without needing to discover the decomposition itself. This is particularly relevant near the reasoning frontier, where a reliably stronger teacher may be unavailable. For a stronger backbone, we would build the curriculum from problems that learner still fails to solve under a fixed rollout budget.

Our current experiments show how learning from these intermediate goals helps on the original hard problems. In Figure 2a, LeanTree produces more informative within-group rewards than direct RL over most of Qwen/GRPO training. When both methods train exclusively on the difficult original problems (steps 600–700), LeanTree’s mean reward recovers quickly after the transition and exceeds direct RL later in that phase (Figure 2b). Table 1 also shows that LeanTree achieves the highest average accuracy on both backbone families we tested, including 74.45% on DeepSeek-R1-Distill-Llama-8B, compared with 71.92% for direct RL. These results motivate testing the same approach on problems at a stronger learner’s own capability frontier.


Thank you again for your thoughtful review. We hope these results clarify what existing proofs contribute to informal RL: mathematically grounded intermediate tasks and a way to order them so the learner can return to the original hard problems.

AuthorsOpenReview ↗

Gentle Reminder: Author–Reviewer Discussion Ending Soon

Dear Reviewer Tvkg,

Thank you again for your valuable review and constructive feedback. As the Author–Reviewer discussion period is coming to an end soon, we would like to gently remind you that we have posted our point-by-point responses to all your comments.

We would greatly appreciate it if you could review our responses before the deadline. If our clarifications and have adequately resolved your concerns, we would be deeply grateful if you would consider raising your score. Should anything remain unclear, please let us know—we remain available and eager to engage in further discussion.

Best regards,

Authors

Reviewer_2zeWOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses sparse rewards in mathematical reinforcement learning by converting intermediate goals from existing Lean proofs into self-contained natural-language questions and reference answers. LeanTree reconstructs and prunes proof trees, uses an LLM for conversion, and performs training from deeper goals toward original problems. Experiments with 7B/8B models report average gains of 1.15% to 1.98% over Self-aware Weakness-driven problem Synthesis (SwS), together with a curriculum ablation and a held-out original-problem diagnostic.

Critical strengths

The approach provides a concrete way to reuse formal-proof resources for informal training, with explicit assumptions and targets supplied to the translation model.

Critical weaknesses
  • The incremental novelty is limited and insufficiently established. Decomposition-based training and curricula are established ideas: Goedel-Prover-V2 extracts proof goals to augment formal training, while TD-Grokking (training-time decomposition) applies decomposition to informal answer-based RL. LeanTree's narrower contribution is its use of existing proof goals and their depth. The Prompt LLM baseline is not identified as a TD-Grokking implementation, and its source coverage differs: it decomposes 433 roots, whereas LeanTree starts from 2,844 proof-eligible roots (Appendices B.1 and B.3). A controlled replacement of proof-derived subproblems with solution-guided informal subproblems is needed to isolate the proof source. Although depth correlates with empirical difficulty (Table 4), its advantage over a conventional difficulty-based curriculum is also unestablished.

  • The demonstrated gains are too modest and insufficiently characterized to establish substantial significance. Improvements over SwS average 1.15% to 1.98%, with AIME gains of only 0–1.25% (Table 1). For Qwen/GRPO, random-order training on the augmented pool scores 68.91%, compared with 69.15% for direct RL and 71.14% for the full schedule (Tables 1–2). This supports scheduling relative to random mixing, but does not demonstrate an average benefit from the augmented pool alone; the small deficit does not establish harm. The held-out root result is encouraging, but exceeds size-matched Prompt LLM by only 1.43% (Table 4). Reported standard deviations measure evaluation sampling, not independent training-run variation (Appendix B.5), leaving robustness unresolved.

Questions for authors
  • Could the authors report accuracy differences between LeanTree and SwS across independent training runs, with confidence intervals, particularly on AIME 2025?
  • Could the authors compare proof-derived and TD-Grokking-style solution-guided subproblems on the same source problems, matching the synthesis model, task counts, and RL budgets, under a shared accuracy-based curriculum?
  • Does proof-depth ordering outperform an accuracy-based curriculum?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 2zeW Part (1/5)

Dear Reviewer 2zeW,

Thank you for your valuable review and for recognizing the value of giving the translation model explicit assumptions and targets from a formal proof. We especially appreciate your careful reading of Appendices B and C. Your observation that Prompt LLM covers fewer source problems identified a difference that the subproblem counts alone obscure, and it prompted us to run an additional comparison.

Throughout the discussion phase, we genuinely look forward to your feedback and are fully committed to addressing your concerns and strengthening our work. We hope that our responses and additional experimental results will demonstrate the novelty and effectiveness of LeanTree. If our clarifications have addressed your concerns, we would be deeply grateful if you would consider raising your score. If anything remains unclear, we would truly appreciate your further guidance.


W1: Novelty of LeanTree

The incremental novelty is limited and insufficiently established. Decomposition-based training and curricula are established ideas: Goedel-Prover-V2 extracts proof goals to augment formal training, while TD-Grokking (training-time decomposition) applies decomposition to informal answer-based RL. LeanTree's narrower contribution is its use of existing proof goals and their depth.

Thank you for this insightful comment.

TL;DR: LeanTree addresses two fundamental limitations of existing LLM-based decomposition: logical hallucinations in generated subproblems and the lack of a principled difficulty hierarchy. By leveraging machine-verifiable Lean proofs, LeanTree constructs reliable intermediate tasks and derives a natural, computationally lightweight curriculum directly from the proof-tree structure.

First, LeanTree generates substantially more reliable subproblems by grounding decomposition in machine-verifiable formal proofs. Directly asking an LLM to decompose a mathematical problem, even with a reference solution, can introduce logical hallucinations, such as omitting necessary assumptions or generating intermediate questions that are not logically supported by the original problem. Such errors can directly corrupt the training signals in RLVR. In contrast, LeanTree extracts intermediate goals from existing Lean proofs, where every goal state is validated by Lean's type checker and retains its explicit assumptions and target. This provides a fundamentally more reliable foundation for decomposition.

For example, consider the Lean theorem in Section 2.1, where (x2+y2=0x^2+y^2=0) implies (x=0x=0). This conclusion relies critically on (x,y∈R)x,y\in\mathbb{R}). If an LLM-generated subproblem omits this domain assumption, the conclusion no longer holds over the complex numbers (e.g., (x=1,y=ix=1,y=i)). LeanTree preserves such essential assumptions directly from the formal proof, substantially reducing the risk of logical hallucinations during decomposition.

Second, LeanTree obtains a natural difficulty hierarchy without requiring additional computationally expensive difficulty estimation. Conventional accuracy-based curricula need to evaluate candidate subproblems using multiple rollouts to estimate their difficulty, introducing substantial additional generation costs. In contrast, LeanTree directly inherits the hierarchical structure of the proof tree and uses node depth as a difficulty proxy, requiring no additional rollouts or model-based difficulty estimation. More importantly, proof depth provides structural information about how intermediate reasoning steps contribute to higher-level goals, rather than merely assigning independent difficulty scores to individual questions. This enables a bottom-up curriculum that progressively builds prerequisite reasoning skills before returning to the original difficult problems.

Relationship to existing work. We acknowledge that related works have explored formal subgoals and decomposition-based RL, but LeanTree differs in its use of these ideas and the learning setting it targets:

  • Goedel-Prover-V2 uses extracted proof goals to support formal theorem proving. LeanTree develops this formal proof resource into answer-based, natural-language RL supervision and demonstrates its transfer to informal mathematical reasoning, a fundamentally different learning setting.
  • TD-Grokking also constructs self-contained, verifiable subproblems, but obtains them through solution-guided informal decomposition and employs policy-dependent difficulty calibration to determine which subproblems are learnable and when to recursively expand them. LeanTree instead derives its decomposition and learning hierarchy directly from existing formal proof structures, avoiding additional rollout-based difficulty calibration while retaining the logical structure and dependencies of the original proof.
AuthorsOpenReview ↗

Response to Reviewer 2zeW Part (2/5)

W1 & Q2: Prompt LLM baseline and source coverage

The Prompt LLM baseline is not identified as a TD-Grokking implementation, and its source coverage differs: it decomposes 433 roots, whereas LeanTree starts from 2,844 proof-eligible roots (Appendices B.1 and B.3). A controlled replacement of proof-derived subproblems with solution-guided informal subproblems is needed to isolate the proof source. Could the authors compare proof-derived and TD-Grokking-style solution-guided subproblems on the same source problems, matching the synthesis model, task counts, and RL budgets, under a shared accuracy-based curriculum?

Thank you for this constructive suggestion.

TL;DR: LeanTree outperforms Prompt LLM when both methods decompose the same 2,844 source problems. Prompt LLM’s improvement with broader coverage suggests that expanding LeanTree’s proof corpus could further improve reasoning performance.

First, Prompt LLM is an independently constructed baseline, not a reproduction of TD-Grokking. Our work was developed during the same period, and we were not aware of TD-Grokking when designing this baseline. As shown in Appendix B.3, Prompt LLM receives the original problem, its step-by-step solution, and its final answer, and is instructed to generate self-contained, answer-verifiable subproblems ordered from easier to harder. It therefore provides a solution-guided informal decomposition baseline. TD-Grokking additionally employs staged validation, policy-dependent difficulty calibration, recursive expansion, and dynamic active-set selection. TD-Grokking, Sections 3.2–3.4.

Second, we conducted an additional source-matched comparison following your suggestion. In the held-out evaluation reported in Table 4, LeanTree outperforms the size-matched Prompt LLM baseline at every proof depth, including on the original root problems. To further examine the effect of source coverage, we expanded Prompt LLM to decompose the same 2,844 source problems used by LeanTree and evaluated it under the experimental settings in the paper.

Backbone / optimizerOriginal Prompt LLMPrompt LLM on the same 2,844 rootsLeanTree
Qwen2.5-7B-Instruct / GRPO68.8969.3271.14
Qwen2.5-7B-Instruct / BAPO69.6269.9371.88
DeepSeek-R1-Distill-Llama-8B / GRPO71.4772.0974.45

Entries report average accuracy (%).

Notably, LeanTree consistently achieves the best performance across all three configurations, outperforming the expanded Prompt LLM baseline on both model backbones and both RL optimizers. This provides further evidence for the effectiveness of leveraging formal proof structures to construct intermediate supervision, beyond conventional solution-guided decomposition.

Finally, the additional results also highlight the substantial potential for further improving LeanTree through broader proof coverage. This bears directly on the limitation we identify in Section 6. In our data construction, the original Base Data contains 22,993 answer-verifiable mathematical problems, but only 2,844 (12.4%) of them are both beyond the initial model's capability and supported by usable Lean proofs. Consequently, the current LeanTree pipeline can exploit only a small fraction of the original training problems to construct proof-grounded intermediate supervision.

The expanded Prompt LLM experiment provides an encouraging observation: broadening the source problems from which subproblems are synthesized leads to better overall reasoning performance. This is particularly relevant to LeanTree because its current supervision is fundamentally constrained by the availability of formal proofs, rather than by the potential to extract more useful intermediate reasoning tasks. If more original mathematical problems can be paired with usable Lean proofs, LeanTree can construct substantially more proof-grounded subproblems and extend its curriculum to a much broader range of beyond-capability problems. We therefore believe that the current results represent only an initial demonstration of the potential of proof-grounded supervision, with considerable room for further improvement as formal-proof coverage expands.

AuthorsOpenReview ↗

Response to Reviewer 2zeW Part (3/5)

W1 & Q3: Proof-depth curriculum versus accuracy-based curriculum

Although depth correlates with empirical difficulty (Table 4), its advantage over a conventional difficulty-based curriculum is also unestablished. Does proof-depth ordering outperform an accuracy-based curriculum?

Thank you for this valuable question.

TL;DR: A key advantage of proof-depth ordering is that it provides a difficulty signal and a structural learning hierarchy simultaneously, without requiring additional rollout-based calibration. The proof tree identifies which intermediate goals support a harder problem, allowing the learner to practice prerequisite skills before combining them.

An accuracy-based curriculum requires evaluating each candidate subproblem to estimate its difficulty. For our 4,772 synthesized subproblems, even a single calibration pass with 8 rollouts per question would require 38,176 additional generations. More reliable estimates would require additional sampling. LeanTree obtains the depth of every subproblem directly during proof-tree reconstruction, requiring no additional rollouts for difficulty estimation.

More importantly, proof depth provides structural information that independent accuracy estimates cannot directly capture. A proof tree organizes intermediate goals hierarchically, reflecting how local reasoning steps contribute to higher-level goals. This allows LeanTree to organize training from foundational intermediate tasks toward the original problem, rather than merely sorting questions by their individual success rates.

Our existing experiments provide strong evidence for this design.

  • Depth reflects empirical difficulty. As shown in Table 4, the initial model’s accuracy increases monotonically from 21.61% on original root problems to 70.14% on subproblems at depth ≥5. Depth-based ordering also improves average accuracy over random ordering on the same training data (Table 2).
  • The curriculum provides more informative rewards and supports rapid adaptation to hard problems. Under matched rollout budgets, LeanTree maintains higher reward informativeness than direct RL over most of training, giving GRPO stronger within-group reward contrast (Figure 2a). At step 600, LeanTree transitions to the difficult original root problems, and both methods train exclusively on these roots through step 700. After a brief dip at the transition, LeanTree’s mean reward recovers quickly and consistently exceeds the direct-RL baseline during the latter part of this shared phase (Figure 2b). This pattern suggests that practice on intermediate goals prepares the learner to tackle the original hard problems.
  • The benefit extends to every proof-depth group. In Table 4, LeanTree outperforms the size-matched Prompt LLM baseline at every depth, including the original root problems.

Together, these results support proof depth as an effective, computationally lightweight signal for organizing hierarchical mathematical learning. Because depth is available directly from each proof tree, the same curriculum can be applied to larger collections of synthesized problems without an additional round of difficulty calibration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.