- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes world-model supervised fine-tuning (WM-SFT) to teach a model to predict changes in the proof state after auxiliary geometric constructions. Applying WM-SFT before policy supervised fine-tuning (Policy-SFT) improves mean proof success from 65.9% to 72.9%. WM-SFT also provides an effective initialization for reinforcement learning: the sequence of WM-SFT, Policy-SFT, and RL achieves a proof success rate of up to 76.8%, compared with 71.2% for Policy-SFT followed by RL. However, the main policy-learning experiments are limited to a single backbone model, leaving generalization across models unexplored.
Critical strengths
-
The paper motivates improving policy learning by explicitly training the model to predict environment feedback following auxiliary geometric constructions.
-
The results show that WM-SFT improves subsequent supervised policy learning and provides an effective initialization for reinforcement learning.
Critical weaknesses
-
The main policy-learning experiments use a single backbone model. Could the authors evaluate additional models to demonstrate the robustness and generalizability of the method?
-
Could the proposed method extend to other settings, such as formal theorem proving in Lean 4? What adaptations would be needed to define the states, actions, and feedback?
-
Could the authors explain why Policy-SFT → GRPO achieves slightly higher proof success than WM-SFT () → GRPO (71.2% versus 70.4%)? Further analysis could clarify the respective contributions of action imitation and state-transition prediction to subsequent RL performance.
Questions for authors
Please refer to the weaknesses discussed above.
2 comments
Official Comment by Reviewer fg2g
Thank you for the authors' response. However, it would be better if the authors could provide additional experimental results to support their claims.
We thank the reviewer for the encouraging feedback and thoughtful suggestions. We are glad that the reviewer finds the motivation and the improvements from transition supervision meaningful.
TL;DR
Our main finding is that learning action-induced state changes provides a useful initialization for downstream policy learning, without requiring a learned simulator at inference time. We agree that broader model and environment evaluations are valuable next steps. We also clarify that Policy-SFT directly teaches action selection, which helps explain its slightly stronger performance than WM-SFT when each is used alone before RL.
1. Generalization across backbone models (W1)
We agree that evaluating additional backbone models would strengthen the generality of our findings.
Our main policy-learning experiments use Qwen3.5-27B. We also conducted experiments with a smaller 4B model to study transition supervision from different data sources (Table 2). These results show that transition prediction can also be learned by a smaller model, although they do not yet establish the same downstream policy-learning gains across model sizes.
We appreciate this suggestion and plan to further investigate different model sizes and architectures, including models outside the Qwen family.
2. Extension to Lean 4 (W2)
Thank you for this suggestion. We agree that Lean 4 would be an interesting setting for testing whether the proposed learning objective generalizes beyond geometry.
Our method only requires a structured state, an action, and verifiable feedback describing how the state changes. In Lean 4, the current proof goals and local hypotheses could define the state, a tactic could serve as the action, and the updated proof goals and hypotheses returned by Lean could provide transition supervision.
The model could first learn to predict these action-induced changes and then be fine-tuned for tactic selection or further optimized with RL. Importantly, the actual Lean environment would still execute and verify tactics during inference, without requiring the model to simulate their outcomes.
We see this as a natural extension of our approach and appreciate the suggestion.
3. Why does Policy-SFT -> GRPO slightly outperform WM-SFT -> GRPO? (W3)
We believe this difference is consistent with the different objectives of the two training stages.
Policy-SFT directly teaches the model which construction actions to select to solve a problem. WM-SFT instead teaches what geometric relations may result from a supplied construction, without directly teaching which action is useful for the current proof goal. Therefore, Policy-SFT provides more direct supervision for the downstream policy task.
We also note that the difference is small: 71.2% versus 70.4% on 250 evaluation problems, corresponding to only two problems. Since these RL results are from single runs, we do not interpret this difference as a meaningful performance gap.
More importantly, the two objectives appear complementary. WM-SFT -> Policy-SFT -> GRPO achieves 76.4–76.8%, outperforming either Policy-SFT -> GRPO (71.2%) or WM-SFT -> GRPO (70.4%). This supports our main finding that transition supervision can provide a useful initialization, while subsequent action imitation and RL teach the model how to use that initialization for problem solving.
We thank the reviewer again for these suggestions, which will help us clarify the scope and broader applicability of the work.