acceptodds
Under review as a conference paper at ICLR 2027

Learning State Transitions Improves Downstream Policy Learning in Geometry Theorem Proving

Abstract

Geometry theorem proving can be formulated as sequential action selection over an evolving proof state. Standard policy supervised fine-tuning (\policysft) teaches a model to imitate reference actions, but does not directly supervise how those actions change the state. We study whether explicitly learning these state transitions before policy training improves downstream theorem proving. We build a stateful agent environment around the open-source GenesisGeo symbolic engine. Within this environment, we first apply world-model supervised fine-tuning (\wmsft) to predict auxiliary line construction feedback, whose newly derived geometric relations define the action-induced state delta. We then perform on actions from engine-generated trajectories using the same model. Across three runs, followed by achieves 72.9% mean proof success, compared with 65.9% for alone. Additionally, after subsequent reinforcement learning, followed by and RL reaches up to 76.8%, compared with 71.2% for followed by RL. is used only during training, all policies follow the same action-selection procedure at inference, with no additional prediction stage, model call, or planning. These results show that supervised learning of action-induced state transitions can provide an effective initialization for training mathematical-agent policies.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
mathematical agents, world models, reinforcement learning, geometric reasoning, state-transition learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fg2gOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes world-model supervised fine-tuning (WM-SFT) to teach a model to predict changes in the proof state after auxiliary geometric constructions. Applying WM-SFT before policy supervised fine-tuning (Policy-SFT) improves mean proof success from 65.9% to 72.9%. WM-SFT also provides an effective initialization for reinforcement learning: the sequence of WM-SFT, Policy-SFT, and RL achieves a proof success rate of up to 76.8%, compared with 71.2% for Policy-SFT followed by RL. However, the main policy-learning experiments are limited to a single backbone model, leaving generalization across models unexplored.

Critical strengths
  1. The paper motivates improving policy learning by explicitly training the model to predict environment feedback following auxiliary geometric constructions.

  2. The results show that WM-SFT improves subsequent supervised policy learning and provides an effective initialization for reinforcement learning.

Critical weaknesses
  1. The main policy-learning experiments use a single backbone model. Could the authors evaluate additional models to demonstrate the robustness and generalizability of the method?

  2. Could the proposed method extend to other settings, such as formal theorem proving in Lean 4? What adaptations would be needed to define the states, actions, and feedback?

  3. Could the authors explain why Policy-SFT → GRPO achieves slightly higher proof success than WM-SFT (r=32r=32) → GRPO (71.2% versus 70.4%)? Further analysis could clarify the respective contributions of action imitation and state-transition prediction to subsequent RL performance.

Questions for authors

Please refer to the weaknesses discussed above.

2 comments

We thank the reviewer for the encouraging feedback and thoughtful suggestions. We are glad that the reviewer finds the motivation and the improvements from transition supervision meaningful.


TL;DR

Our main finding is that learning action-induced state changes provides a useful initialization for downstream policy learning, without requiring a learned simulator at inference time. We agree that broader model and environment evaluations are valuable next steps. We also clarify that Policy-SFT directly teaches action selection, which helps explain its slightly stronger performance than WM-SFT when each is used alone before RL.


1. Generalization across backbone models (W1)

We agree that evaluating additional backbone models would strengthen the generality of our findings.

Our main policy-learning experiments use Qwen3.5-27B. We also conducted experiments with a smaller 4B model to study transition supervision from different data sources (Table 2). These results show that transition prediction can also be learned by a smaller model, although they do not yet establish the same downstream policy-learning gains across model sizes.

We appreciate this suggestion and plan to further investigate different model sizes and architectures, including models outside the Qwen family.

2. Extension to Lean 4 (W2)

Thank you for this suggestion. We agree that Lean 4 would be an interesting setting for testing whether the proposed learning objective generalizes beyond geometry.

Our method only requires a structured state, an action, and verifiable feedback describing how the state changes. In Lean 4, the current proof goals and local hypotheses could define the state, a tactic could serve as the action, and the updated proof goals and hypotheses returned by Lean could provide transition supervision.

The model could first learn to predict these action-induced changes and then be fine-tuned for tactic selection or further optimized with RL. Importantly, the actual Lean environment would still execute and verify tactics during inference, without requiring the model to simulate their outcomes.

We see this as a natural extension of our approach and appreciate the suggestion.

3. Why does Policy-SFT -> GRPO slightly outperform WM-SFT -> GRPO? (W3)

We believe this difference is consistent with the different objectives of the two training stages.

Policy-SFT directly teaches the model which construction actions to select to solve a problem. WM-SFT instead teaches what geometric relations may result from a supplied construction, without directly teaching which action is useful for the current proof goal. Therefore, Policy-SFT provides more direct supervision for the downstream policy task.

We also note that the difference is small: 71.2% versus 70.4% on 250 evaluation problems, corresponding to only two problems. Since these RL results are from single runs, we do not interpret this difference as a meaningful performance gap.

More importantly, the two objectives appear complementary. WM-SFT -> Policy-SFT -> GRPO achieves 76.4–76.8%, outperforming either Policy-SFT -> GRPO (71.2%) or WM-SFT -> GRPO (70.4%). This supports our main finding that transition supervision can provide a useful initialization, while subsequent action imitation and RL teach the model how to use that initialization for problem solving.


We thank the reviewer again for these suggestions, which will help us clarify the scope and broader applicability of the work.

Reviewer_fg2gOpenReview ↗

Official Comment by Reviewer fg2g

Thank you for the authors' response. However, it would be better if the authors could provide additional experimental results to support their claims.

Reviewer_e7T9OpenReview ↗
Rating
1(−1)
Confidence
3(−1)
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of neural-assisted auxiliary construction in geometry theorem proving and proposes World Model Supervised Fine-Tuning (WM-SFT) as a pretraining task before formal policy training. Inspired by world models, the paper treats the new facts derived through solver reasoning after a given auxiliary construction as a transition of the proof state. The model is trained to predict this increment given the proof state and auxiliary construction, with the expectation that this task can improve the model's performance in downstream policy learning. The authors conduct experiments on a subset of data from GenesisGeo, demonstrating the effectiveness of WM-SFT as a pretraining task: after this pretraining, policy learning performance is effectively improved under both SFT-based and reinforcement learning-based approaches.

Critical strengths

This paper designs a pretraining method for auxiliary construction prediction in geometry theorem proving, allowing the model to learn to predict newly derived facts from proof states and actions. The motivation is relatively reasonable, and the experiments also demonstrate the effectiveness of this pretraining method.

Critical weaknesses

The main issue with this paper is its limited contribution and novelty. The paper only proposes an auxiliary task as a pretraining method before policy training and verifies its effectiveness. However, using information generated during the reasoning process for pretraining is not a particularly new idea, and various motivations and approaches already exist. For example, in AlphaGeometry, the authors use prediction of the deduction process as a pretraining task. This paper merely changes the prediction target to improve downstream task performance, which provides relatively limited insight to the community. If the paper could investigate in depth how to design and select the pretraining objectives most beneficial for this task, its contribution would be greatly increased.

The paper does not compare its method with the pretraining task proposed in existing work (AlphaGeometry). This makes it unclear to me whether the pretraining task designed in this paper is superior to existing designs. Furthermore, the experiments are conducted only on a subset of GenesisGeo, and generalization across datasets remains unknown.

The methodological details are insufficiently described. From the "Engine execution and feedback" paragraph, it seems that the paper selects at most 8 newly derived facts as prediction targets for pretraining. The specific rules and rationale for selecting at most 8 facts are not explained in detail.

Some details in the paper's writing are overly verbose. For example, Section 2 uses a large amount of text to describe the execution process of the symbolic environment. Adding pseudocode or examples would make the description clearer and more concise.

Questions for authors

Compared with the pretraining task of predicting the deduction process proposed in AlphaGeometry, how do the upper bound on downstream task accuracy and the training cost of the proposed method compare?

Please explain the specific rules for selecting at most 8 facts from the newly derived facts for prediction, as well as the motivation for doing so, or clarify any possible misunderstanding on my part as a reviewer. Would selecting different numbers of facts significantly affect the experimental conclusions?

4 comments

We sincerely thank the reviewer for the careful assessment and constructive suggestions. We appreciate the concern about novelty and agree that understanding which pretraining objectives are most useful is an important research question.


TL;DR

Our work studies how models can benefit from learning environment transitions without requiring an accurate learned world model during inference. This is motivated by practical settings where reliable predictive models are difficult to obtain, but real environment feedback is available through tool execution.

We show that learning action-conditioned transitions provides a useful initialization for subsequent Policy-SFT and RL. We agree that a broader comparison of pretraining objectives, especially with AlphaGeometry-style deduction learning, would further strengthen our understanding of why this approach works.


1. Novelty and motivation (Weakness paragraph 1, WP1)

We agree that using synthetic geometric deductions for pretraining is not new. AlphaGeometry is an example, and we have discussed its relationship to our approach in the Introduction. Our motivation comes from a broader challenge in tool-using agents: in many practical environments, an accurate learned world model is difficult to obtain. Using an imperfect model to predict future states during planning can introduce errors, especially over multiple steps.

However, even without an accurate predictive model, we can often collect reliable state transitions by executing actions in the real environment. For example, a CAD agent may not have an accurate model for predicting the consequences of complex editing operations, but it can execute those operations through APIs and observe their actual results.

This motivates our central question: Can learning from real state transitions improve downstream policy learning without requiring the model to predict those transitions during inference?

Our approach uses transition prediction only as a training objective. The resulting policy is optimized through action imitation and model-free RL. During inference, it selects actions and receives actual environment feedback, without learned state simulation or additional planning calls. We believe this is a useful direction for agents operating in environments where learned dynamics are not sufficiently reliable for planning.

2. AlphaGeometry and our contribution (WP2)

We recognize that AlphaGeometry already uses synthetic geometric proofs and deduction traces to support learning useful constructions. Our work builds on this broader direction rather than claiming that learning from geometric deductions is entirely new.

One difference in training setup is that AlphaGeometry trains its language model from scratch, while we start from an existing pretrained LLM and use WM-SFT as an intermediate fine-tuning stage. More importantly, the two approaches are potentially complementary. WM-SFT is a training objective rather than a new model architecture, so it could, in principle, also be incorporated into an AlphaGeometry-style pipeline to provide additional supervision on the consequences of auxiliary constructions. We have not tested this combination, but see it as a valuable direction for future work.

Our focus is on isolating action-conditioned transition prediction as a pretraining objective and studying its effect on subsequent policy learning. Specifically, we separate two objectives:

  • WM-SFT: Given the current state and a supplied construction, predict the resulting environment feedback and newly derived geometric relations.
  • Policy-SFT: Given the current proof context and goal, predict the next useful action.

AlphaGeometry pretrains on synthetic proofs, where the model learns to generate proof steps given the geometric premises and a target conclusion. Our WM-SFT focuses on a different task: given the current geometry and a supplied construction, the model predicts the resulting state changes. We explicitly remove the proof goal from the input, so the model learns what happens after an action, rather than which actions or deductions help reach a goal. We then study whether learning these transitions helps the model learn a better goal-directed policy through SFT and RL, without requiring transition prediction during inference.

We additionally find that training order matters. Reversing the order to Policy-SFT -> WM-SFT produces higher final transition prediction accuracy but lower proof success (58.8%). This suggests that transition supervision is most useful as preparation for policy learning, rather than as a standalone prediction objective.

We do not claim that AlphaGeometry requires a learned world model at inference, or that our transition objective is universally better than its deduction-based pretraining. Our contribution is a focused empirical study of how action-conditioned transition learning benefits downstream policies.

...(Continued in the next post, post 1/2)

...(Continued from the previous post, post 2/2)

3. Comparison with other pretraining objectives (QP1)

We appreciate the reviewer's suggestion to compare against AlphaGeometry-style deduction prediction. We agree that such a comparison would help explain which forms of geometric supervision are most effective. Our current experiments control the source problems and training budget. WM-SFT and Policy-SFT use the same source trajectories, and the Policy-SFT-only baseline receives the same total training budget.

However, we recognize that these comparisons do not fully isolate the benefit of transition prediction from other possible forms of geometric supervision. We also do not currently have a controlled comparison establishing a higher performance ceiling or lower training cost than AlphaGeometry-style pretraining.

We have started exploring this broader question in a separate study using Asymptote-based geometric constructions. We compare SFT on geometric construction code with action-conditioned transition prediction. Under matched training conditions, SFT on geometric code representations alone did not improve performance on Geometry3K, whereas action-conditioned transition supervision showed positive gains.

Although this is not a direct comparison with AlphaGeometry, it provides preliminary evidence that learning the consequences of actions may offer benefits beyond simply learning an alternative geometric representation.

4. Selection of at most eight newly derived facts (WP3, QP2)

Thank you for pointing out that this design choice needs a clearer explanation.

The environment reports at most eight canonical new facts after each accepted construction, prioritizing facts involving the newly constructed point. This provides a compact and consistent observation while limiting the length of feedback and training targets.

Our current distribution analysis shows that 96.6% of expert WM-SFT examples reach this eight-fact reporting limit. We are therefore examining the original, uncapped distribution to better understand how many additional facts are omitted and whether a larger limit changes downstream performance.

We agree that the effect of this limit is worth investigating and that different choices may influence the amount of useful supervision and the training cost.


We appreciate the reviewer's suggestions and believe they will help us communicate the contribution more clearly.

Thank you very much to the authors for their response. Unfortunately, I do not think the authors' response has addressed my concerns.

Regarding my Q1, the authors acknowledge that they have not compared their method with the pretraining method used in AlphaGeometry, and state that they have begun exploring this issue in a separate study. However, we are reviewing this paper, not other studies. The authors have not provided any quantitative experimental results.

I appreciate the explanations provided by the authors during the rebuttal, and I understand that lowering the score may be disappointing. However, based on the evidence presented in the current paper and the rebuttal, I believe that this paper has not yet met the acceptance standards, and therefore I have decided to lower my score.

Thank you for taking the time to read our response and clarify your concerns. We understand your point that a direct comparison with AlphaGeometry-style deduction pretraining would provide stronger evidence for the choice of our training objective.

Our intention was to clarify the scope of our contribution: studying whether action-conditioned transition supervision improves downstream policy learning, rather than claiming it is superior to existing deduction-based pretraining.

We agree that this distinction does not replace the need for a controlled comparison of training objectives. We appreciate your feedback and will make this limitation clearer in the paper.

Reviewer_kWANOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether pretraining on environment state transitions benefits policy learning in geometry theorem proving. The authors construct a stateful multi-turn environment around GenesisGeo that returns verified geometric relation updates after each auxiliary construction. The proposed training pipeline operates in two stages: first, World-Model SFT (WM-SFT) trains the model to predict the newly derived geometric relations given a candidate construction; second, Policy-SFT fine-tunes the model on reference actions, optionally followed by GRPO. At test time, the transition predictor is discarded, and the policy interacts directly with the symbolic engine. On 250 held-out problems, WM-SFT + Policy-SFT improves average Pass@1 from 65.9% to 72.9% over Policy-SFT alone across three seeds, with subsequent RL further boosting performance to 76.8%. Notably, the authors observe that explicit transition prediction accuracy drops from 60.1% to 13.4% F1 after Policy-SFT, despite the downstream gains.

Critical strengths

Practical and Well-Motivated Problem: Each auxiliary construction changes the proof state that later decisions depend on. Pretraining the policy on these state transitions before action imitation is a natural setup for this domain.

Thorough Diagnostic Analysis: The paper provides solid diagnostics, notably decomposing policy gains into reachability versus submission (S = R - U) and showing that policy rollouts yield far weaker transition supervision than engine data.

Critical weaknesses

Unfair baseline: The baseline uses the same compute, but the proposed method gets extra training examples on the same problems. The improvement could simply come from seeing these geometry problems twice. To show that transition learning actually matters, the authors need a control that uses the same amount of data.

The model does not keep its transition ability: The paper argues that predicting state transitions helps the policy, but transition accuracy drops sharply right after policy training. In RL experiments, the model with high transition accuracy performs poorly on theorem proving, while models with low accuracy perform much better. Since solving problems does not track prediction accuracy, the main claim is not convincing.

Questions for authors
  1. How does the proposed method compare with a baseline trained on the same amount of geometry data without transition prediction?

  2. Since transition prediction accuracy drops so quickly after policy training, what evidence shows that the model is actually using this transition knowledge during proof search?

2 comments

We thank the reviewer for the thoughtful feedback and for recognizing the value of our diagnostic analysis. We appreciate both concerns, as they touch on important questions about what transition supervision actually contributes to policy learning.


TL;DR

Our method does not introduce additional geometry problems. WM-SFT and Policy-SFT reuse the same source trajectories, and the Policy-SFT-only baseline receives the same total training budget. However, we agree that transition supervision introduces additional learning targets, and a matched-supervision control would help isolate its benefit.

We also agree that transition prediction accuracy declines after policy training. Our claim is not that the final policy can accurately simulate environment transitions, but that learning transitions provides a useful initialization for downstream policy learning. The policy uses actual engine feedback during inference, without needing to generate future states.


1. Fairness of the training-data comparison (W1)

We thank the reviewer for raising this important concern. We agree that WM-SFT -> GRPO receives additional offline supervision compared with Base → GRPO. This is why we also include a more controlled comparison between WM-SFT -> Policy-SFT and Policy-SFT alone.

Both methods use trajectories from the same 1,240 training problems, and the Policy-SFT-only baseline receives the same total supervised training budget as WM-SFT and Policy-SFT combined. The main difference is how that budget is used: our method first learns the consequences of construction actions before learning to select actions, while the baseline focuses on action imitation.

Under this setting, WM-SFT -> Policy-SFT improves mean proof success from 65.9% to 72.9% across three runs. After subsequent GRPO, the corresponding policies achieve 71.2% and 76.4%, respectively. These results suggest that using part of the supervised training budget for transition prediction provides benefits that carry over to RL.

We also recognize that WM-SFT introduces different supervision targets, even though they come from the same problems. We agree that comparing against other forms of additional supervision would help further isolate the contribution of transition learning, and we appreciate this suggestion.

2. Why does transition prediction accuracy drop after policy training? (W2)

We agree that the decline in transition prediction accuracy is an important observation. In fact, we explicitly report and discuss it in Section 4.4. Our intended claim is not that the final policy preserves a highly accurate transition simulator or uses predicted transitions during proof search. Instead, we investigate whether learning action-induced state changes before policy optimization provides a better initialization for downstream learning. The results support this distinction.

For example, WM-SFT (r=32) reaches 60.1% new-fact F1, which drops to 13.4% after Policy-SFT. However, the resulting policy achieves 72.9% mean proof success, compared with 65.9% for Policy-SFT alone. Similarly, WM-SFT -> GRPO reaches 70.4% proof success, compared with 59.2% for Base -> GRPO.

Importantly, all policies use the same inference procedure. They select actions based on the current proof context and receive verified feedback from GenesisGeo. They do not predict future states or use a learned simulator during inference. Therefore, high explicit transition-generation accuracy is not required for the observed policy improvement.

We also recognize an important distinction: a drop in next-state generation accuracy does not necessarily mean that all transition-related information has been lost from the model's internal representations. Our current new-fact F1 metric evaluates autoregressive generation of structured feedback. It does not directly measure what transition-related knowledge remains accessible to the policy after fine-tuning. We therefore do not claim to have directly established that the final policy retains and uses an accurate internal transition model.

...(Continued in the next post, post 1/2)

...(Continued from the previous post, post 2/2)

3. What evidence connects transition learning to improved policy behavior? (Q1 & Q2)

Beyond the final proof-success results, Section 4.5 provides an analysis of how WM-SFT changes policy learning during GRPO. We separate proof success into two components: whether the agent reaches a state where the target theorem becomes provable, and whether it successfully submits the target proposition. Between RL steps 150 and 175, the WM-SFT-initialized policy improves proof success by 10.8 percentage points, of which 10.4 points come from improved goal reachability. This suggests that the improvement is primarily associated with choosing constructions that make the target theorem provable, rather than simply becoming better at submitting the final answer. We believe this provides useful behavioral evidence that transition pretraining affects downstream action selection.

However, we agree that it does not fully explain the internal mechanism. A useful next step is to evaluate whether transition knowledge remains accessible through other measurements, such as selecting the correct next state from multiple candidates, rather than requiring the model to generate the complete feedback.


We appreciate the reviewer highlighting this distinction. We will further clarify that our central contribution is the downstream learning benefit of transition supervision, not the development of an accurate inference-time simulator.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.