- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents MathRHS, a benchmark of 457 open-answer problems with conclusions (right-hand side) that separates conclusions from problem statements, and compares three things: standalone conclusion synthesis before formalization, joint synthesis of the conclusion and formal statement, and conclusion synthesis during proving. Experiments with GPT-4.1, DeepSeek-V3.2, and Goedel-Prover-V2 show that standalone synthesis improves conclusion accuracy and downstream performance over joint synthesis.
Critical strengths
Interesting task formulation is the critical strength. Masking the original answer and letting the LLM synthesize the RHS conclusion before proving is a relatively interesting idea. Existing benchmarks skip this by directly giving the answer. The paper presents a relatively interesting method for addressing that.
Critical weaknesses
- Outdated models in the experiments: The evaluated models are well behind the current state of the art. GPT-4.1 and DeepSeek-V3.2 are used for the entire analysis process, and Goedel-Prover-V2 is the only prover; all are outdated frameworks. The conclusion reached for those frameworks may not be valid for the latest research. Especially, the PutnamBench and IMO problems are considered resolved for the latest model, as the paper itself indicates
- Lack of analysis of joint synthesis underperformance: A central finding is that joint conclusion synthesis and authformalization performs much worse compared to direct answer generation. However, the paper does not investigate why that happens. And, combined with the first weakness, whether the problem still exists in the latest open-source model, say Qwen-3.8-27B.
- The paper’s idea of exploring open-ended problems through Lean has been explored before in [1], but not discussed in the paper.
Reference:
[1] Wang, R., Li, Y., Fung, Y. R., & Zhang, T. (2025, November). Let’s reason formally: Natural-formal hybrid reasoning enhances llm’s math capability. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (pp. 16794-16820).
Questions for authors
- How do the latest open-source and closed-source models, including both general reasoning agents and recent provers, perform on MathRHS under the three settings?
- Why does joint synthesis underperform direct answer generation? Can the authors provide an error analysis of failed joint conclusions? Can they also report results when the model is allowed to reason before producing the joint formal statement?
3 comments
Thanks a lot for the response. However, I am unable to agree with the authors' point regarding the benchmark's goal. In my view, the exact goal of a benchmark is to distinguish a good model from a bad one. Thus, I will keep my score.
Thank you for your response and for sharing your perspective. We completely agree that MathRHS can be used to compare model capabilities across each stage of the pipeline (conclusion synthesis, autoformalisation, and proving) and distinguish stronger models from weaker ones, as seen in the performance separation between GPT-4.1, DeepSeek-V3.2, and soon DeepSeek-V4-Flash.
At the same time, static model leaderboards on fixed benchmarks tend to saturate quickly as newer models are released and datasets become public. By removing the conclusion from the natural language problem statement, MathRHS is designed not only to measure model capability, but primarily to isolate where in the end-to-end theorem-proving pipeline conclusion synthesis should take place. Understanding how different pipeline placements affect conclusion accuracy and formalisation faithfulness provides structural guidance to our community for building better training data and more generalisable formal reasoning models. This contribution remains relevant beyond any single generation of models.
Thank you again for your time and thoughtful feedback on our work. We would be grateful if you would consider raising your score, or let us know what specific additions would help address any remaining concerns.
Thank you for your comments and feedback, please find an updated version of the paper with revision taking into account your feedback in red, as well as direct responses below.
We extended our experiments to include DeepSeek-V4-Flash and use Qwen3.6-27B for semantic evaluation. The advantage of standalone conclusion synthesis persists with DeepSeek-V4-Flash. Importantly, existing results on PutnamBench and IMO do not imply that conclusion synthesis is solved. These systems typically start from gold formal statements that already contain the answers. MathRHS removes the answers, requiring models to discover them before proving the resulting statements. To our knowledge, only one closed-source system, Midas Prover, reports solving all 346 answer-free Putnam problems, at a reported cost of $4,144. This suggests that answer synthesis remains challenging, particularly for open-source models. Our goal is not to compare theorem provers directly, but to investigate where conclusion synthesis should occur in the formal theorem-proving pipeline.
We find that the gap comes primarily from numerical and algebraic answers, while performance on binary answers is similar.
For binary questions, selecting an answer and encoding it in Lean is relatively straightforward. Numerical and algebraic questions instead require the model to discover a value or expression while simultaneously constructing a valid formal statement. We also observe a common failure mode in joint generation: the answer is omitted from the formal statement. In the GPT-4.1 few-shot setting, 51.3% of the generated statements end with “= conjecture” without specifying an answer. These statements can typecheck in Lean, but they leave conclusion synthesis to the prover rather than performing it during autoformalisation. Other outputs similarly assert only that an answer exists, for example “∃ n, …”, without identifying it.
Consequently, approximately half of the problems in the joint setting pass Lean’s typecheck, but only 6-10% yield a faithful formalisation. This helps explain why joint generation underperforms despite producing many syntactically valid Lean statements. The standalone advantage also persists with DeepSeek-V4-Flash, as discussed in Q1.
Thank you for pointing out this work. We cited and discussed Wang et al. (EMNLP 2025) in the updated manuscript. They rewrite QA problems as Lean existence theorems and give these to a prover as extra context. The answer is still taken from the prover's natural-language reasoning and graded against the reference. Lean is not used to verify answers (their Limitations and App. E). MathRHS asks a different question: where in a formal pipeline the answer should be found.