acceptodds
Under review as a conference paper at ICLR 2027

MathRHS: Conclusion Synthesis in the End-to-End Theorem-Proving Pipeline

Abstract

Many mathematical problems require synthesising a conclusion before a complete theorem statement can be formalised and proved. Evaluations that provide this conclusion in advance skip an important subtask: determining what must be proved. In this study, we investigate where conclusion synthesis should occur in the end-to-end theorem-proving pipeline. We introduce MathRHS, a benchmark of 457 mathematical problems with conclusions (right-hand side) that separates conclusions from problem statements, and compare three settings: standalone conclusion synthesis before autoformalisation, joint synthesis of the conclusion and formal statement, and conclusion synthesis during proving. By evaluating conclusion correctness, formalisation faithfulness, and verified proof success separately, we trace how synthesis performance translates into downstream results. Standalone synthesis substantially improves conclusion accuracy over joint synthesis: from 61.27% to 75.22% for GPT-4.1 and from 45.73% to 82.02% for DeepSeek-V3.2. These improvements also benefit subsequent stages: using DeepSeek-V3.2 for both standalone synthesis and autoformalisation increases faithful autoformalisation from 12.28% to 21.49% and verified proof success from 4.38% to 5.48%. These findings establish conclusion synthesis as an important pipeline design choice: separating it from autoformalisation improves conclusion accuracy and can strengthen downstream prover performance. Nevertheless, the strongest evaluated pipeline proves only 6.36% of problems, highlighting the challenge of integrating correct conclusions through faithful formalisation to verified proofs. We argue that more effort should be put into targeted training for conclusion synthesis to generate more successful end-to-end proofs.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Formal reasoning, Autoformalization, theorem-proving, conjecturing, Lean4, answer discovery

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oP58OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents MathRHS, a benchmark of 457 open-answer problems with conclusions (right-hand side) that separates conclusions from problem statements, and compares three things: standalone conclusion synthesis before formalization, joint synthesis of the conclusion and formal statement, and conclusion synthesis during proving. Experiments with GPT-4.1, DeepSeek-V3.2, and Goedel-Prover-V2 show that standalone synthesis improves conclusion accuracy and downstream performance over joint synthesis.

Critical strengths

Interesting task formulation is the critical strength. Masking the original answer and letting the LLM synthesize the RHS conclusion before proving is a relatively interesting idea. Existing benchmarks skip this by directly giving the answer. The paper presents a relatively interesting method for addressing that.

Critical weaknesses
  1. Outdated models in the experiments: The evaluated models are well behind the current state of the art. GPT-4.1 and DeepSeek-V3.2 are used for the entire analysis process, and Goedel-Prover-V2 is the only prover; all are outdated frameworks. The conclusion reached for those frameworks may not be valid for the latest research. Especially, the PutnamBench and IMO problems are considered resolved for the latest model, as the paper itself indicates
  2. Lack of analysis of joint synthesis underperformance: A central finding is that joint conclusion synthesis and authformalization performs much worse compared to direct answer generation. However, the paper does not investigate why that happens. And, combined with the first weakness, whether the problem still exists in the latest open-source model, say Qwen-3.8-27B.
  3. The paper’s idea of exploring open-ended problems through Lean has been explored before in [1], but not discussed in the paper.

Reference:

[1] Wang, R., Li, Y., Fung, Y. R., & Zhang, T. (2025, November). Let’s reason formally: Natural-formal hybrid reasoning enhances llm’s math capability. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (pp. 16794-16820).

Questions for authors
  1. How do the latest open-source and closed-source models, including both general reasoning agents and recent provers, perform on MathRHS under the three settings?
  2. Why does joint synthesis underperform direct answer generation? Can the authors provide an error analysis of failed joint conclusions? Can they also report results when the model is allowed to reason before producing the joint formal statement?
3 comments

Thank you for your comments and feedback, please find an updated version of the paper with revision taking into account your feedback in red, as well as direct responses below.

W1 & Q1. Outdated models and relevance to current theorem provers

We extended our experiments to include DeepSeek-V4-Flash and use Qwen3.6-27B for semantic evaluation. The advantage of standalone conclusion synthesis persists with DeepSeek-V4-Flash. Importantly, existing results on PutnamBench and IMO do not imply that conclusion synthesis is solved. These systems typically start from gold formal statements that already contain the answers. MathRHS removes the answers, requiring models to discover them before proving the resulting statements. To our knowledge, only one closed-source system, Midas Prover, reports solving all 346 answer-free Putnam problems, at a reported cost of $4,144. This suggests that answer synthesis remains challenging, particularly for open-source models. Our goal is not to compare theorem provers directly, but to investigate where conclusion synthesis should occur in the formal theorem-proving pipeline.

W2 & Q2. Why joint conclusion synthesis underperforms

We find that the gap comes primarily from numerical and algebraic answers, while performance on binary answers is similar.

SettingNumericalAlgebraicBinary
Standalone GPT-4.157.3%44.5%73.7%
During Autoformalisation GPT-4.122.5%30.5%76.3%
Standalone DeepSeek77.5%60.4%74.6%
During Autoformalisation DeepSeek25.3%27.4%70.2%

For binary questions, selecting an answer and encoding it in Lean is relatively straightforward. Numerical and algebraic questions instead require the model to discover a value or expression while simultaneously constructing a valid formal statement. We also observe a common failure mode in joint generation: the answer is omitted from the formal statement. In the GPT-4.1 few-shot setting, 51.3% of the generated statements end with “= conjecture” without specifying an answer. These statements can typecheck in Lean, but they leave conclusion synthesis to the prover rather than performing it during autoformalisation. Other outputs similarly assert only that an answer exists, for example “∃ n, …”, without identifying it.

Consequently, approximately half of the problems in the joint setting pass Lean’s typecheck, but only 6-10% yield a faithful formalisation. This helps explain why joint generation underperforms despite producing many syntactically valid Lean statements. The standalone advantage also persists with DeepSeek-V4-Flash, as discussed in Q1.

W3. Relation to prior work on open-ended problems in Lean

Thank you for pointing out this work. We cited and discussed Wang et al. (EMNLP 2025) in the updated manuscript. They rewrite QA problems as Lean existence theorems and give these to a prover as extra context. The answer is still taken from the prover's natural-language reasoning and graded against the reference. Lean is not used to verify answers (their Limitations and App. E). MathRHS asks a different question: where in a formal pipeline the answer should be found.

Thanks a lot for the response. However, I am unable to agree with the authors' point regarding the benchmark's goal. In my view, the exact goal of a benchmark is to distinguish a good model from a bad one. Thus, I will keep my score.

Thank you for your response and for sharing your perspective. We completely agree that MathRHS can be used to compare model capabilities across each stage of the pipeline (conclusion synthesis, autoformalisation, and proving) and distinguish stronger models from weaker ones, as seen in the performance separation between GPT-4.1, DeepSeek-V3.2, and soon DeepSeek-V4-Flash.

At the same time, static model leaderboards on fixed benchmarks tend to saturate quickly as newer models are released and datasets become public. By removing the conclusion from the natural language problem statement, MathRHS is designed not only to measure model capability, but primarily to isolate where in the end-to-end theorem-proving pipeline conclusion synthesis should take place. Understanding how different pipeline placements affect conclusion accuracy and formalisation faithfulness provides structural guidance to our community for building better training data and more generalisable formal reasoning models. This contribution remains relevant beyond any single generation of models.

Thank you again for your time and thoughtful feedback on our work. We would be grateful if you would consider raising your score, or let us know what specific additions would help address any remaining concerns.

Reviewer_zQXBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MathRHS contains 457 open-answer problems and compares answer synthesis before autoformalization, during autoformalization, and during proving from gold formal statements. GPT-4.1 and DeepSeek-V3.2 generate answers and formalizations; Goedel-Prover-V2 generates proofs. The paper reports higher answer accuracy for standalone synthesis and a best proof success rate of 6.36%.

Critical strengths
  1. Distinguishing answer correctness, formalization faithfulness, and proof validity is important. A kernel-checked proof does not establish that the generated statement matches the original problem.
  2. The open-answer benchmark and cross-model pairings provide a useful setting for studying where mathematical reasoning pipelines fail.
Critical weaknesses
  1. Section 4 confounds synthesis placement with oracle assistance and search budget. Standalone answers are filtered using reference answers, then each distinct accepted answer receives ten formalization attempts; the joint setting receives ten statements in total. The gold-statement condition bypasses autoformalization, and feedback-driven revision is not tested. These comparisons do not establish a general advantage for separating the stages.
  2. The semantic evaluation is too weak to support the end-to-end claims. Appendix A reports 67.5% human agreement for faithfulness grading; the 83% figure concerns the separate conclusion judge. The accepted subset needs direct expert validation. The motivating root example also contains false implications, and Table 2's counts do not reproduce Figure 2's percentages.
Questions for authors
  1. Do the gains persist without reference-based intermediate filtering and under matched total budgets? Please include a stronger reasoning model with an explicit reasoning budget.
  2. Can experts audit all reported successful final statements and provide separate error rates for the conclusion and faithfulness judges?
  3. Please reconcile Table 2's 344/457 and 374/457 with Figure 2: these counts give 75.27% and 81.84%, not 75.22% and 82.02%. In Section 2, x = 4 refutes the implication from x² - 4x = 0 to x = 0, and x = 0 refutes the implication to x = 4. Please correct the example.
3 comments

Thank you for your comments and feedback, please find an updated version of the paper with revision taking into account your feedback in red, as well as direct responses below.

W1 & Q1. Confounding effects of reference filtering and search budgets

We conducted an analysis on the search budget differences for our theorem proving pipelines and find that standalone conclusion synthesis produces more faithful formalisations that reach the final prover, with 201 statements across 62 distinct problems for GPT-4.1 and 309 statements across 98 problems for DeepSeek-V4-Flash. In comparison, joint synthesis produces 148 statements across 44 problems for GPT-4.1 and 119 statements across 28 problems for DeepSeek-V4-Flash. Thus, the difference in effective proof-search budget is smaller than the formalisation budgets might suggest.

Equivalent budget evaluation. We run our pipeline on a subset of the data for the standalone conclusion synthesis and the conclusion synthesis during autoformalisation setting with 100 samples using both GPT and DeepSeek. The results show faithful autoformalisation evaluated by the LLM-judge. We find that our findings hold when evaluating on a strictly equivalent budget. For the manuscript, we opted for evaluating a real-world pipeline. We clarified the evaluation setting in the paper and included the results below in the appendix.

SettingModelFaithful Formalisation
Standalone -> FormaliserGPT-4.1 -> GPT-4.19
Standalone -> FormaliserDeepSeek-V4-Flash -> DeepSeek-V4-Flash20
During autoformalisationGPT-4.11
During autoformalisationDeepSeek-V4-Flash4

Stronger model. We added DeepSeek-V4-Flash. The standalone advantage holds: 20 vs 4 problems with a correct and faithful statement.

We clarified the search budgets in the updated manuscript and will add the full pipeline results for the newest DeepSeek-V4 model in the camera-ready manuscript.

W2 & Q2. Reliability of semantic evaluation and reported results

Thank you for this remark. We conducted a manual analysis of a subset of the statements. We find that the conclusion judge rejects nearly all incorrect answers in the audited sample (98%), but consequently rejects instances that should be accepted (31%). Falsely accepted answers propagate to later stages, whereas rejected correct answers only reduce the number of candidates in later stages.

Q3. Numerical inconsistencies and incorrect root example

Thank you for raising these issues. The benchmark contains only 456 distinct problems (removed a duplicate). Figure 2 uses 456 as the denominator, while Table 2 uses 457. Table 2 also incorrectly reports 142 rather than 141 numerical answers for GPT-4.1. We will report every rate over 456 problems in the camera-ready version.

The motivating example with the root problem is meant to show that a Lean proof can succeed for an incomplete answer. Candidates x = 0 and x = 4 are correct roots, so Lean verifies conclusion x → x^2 - 4*x = 0, a one directional implication for each root to independently satisfy the equation. Only x = 0 ∨ x = 4 satisfies it, but the way the question is worded "If x^2 − 4*x = 0, what is x?", there's no explicit requirement for both answer in the text which generates ambiguity. But from the conclusion that is given, it is clear that both solution are required. Therefore, the point we make is that the formalisation can be less strict if it doesn't see the answer it's expecting upfront, therefore faithfulness is important.

Thank you for the additional experiments and clarifications. The subset comparison in Appendix C is a useful addition. Under the reported matched-budget setup, the standalone pipeline yields more formalizations judged faithful than the joint pipeline for both GPT-4.1 (9 vs. 1) and DeepSeek-V4-Flash (20 vs. 4). The result makes the case for standalone synthesis more convincing at the formalization stage.

However, I am still leaning toward keeping my original score of 2. Two points would most affect my assessment:

First, could you clarify what was matched in Table 7 (calls, samples, or tokens at each stage) and whether reference-answer filtering was used before formalization? The table reports LLM-judged faithfulness, so it leaves open whether the advantage carries through to verified proofs under an equal end-to-end budget without access to reference answers during generation or candidate selection.

Second, the manual audit is helpful for assessing the conclusion judge. It does not establish how often the separate faithfulness judge accepts a Lean statement that changes the meaning of the original problem. Since there are relatively few reported end-to-end successes, an expert audit of the final statements counted as successes would be especially informative. It would also test the revised manuscript's claim that the downstream rates are conservative lower bounds: false rejections by the conclusion judge alone do not establish that claim if some unfaithful statements are accepted and proved.

Thank you also for explaining the duplicate and the corrected count. Please use consistent counts and denominators throughout. For the quadratic example, I agree that proving 0 or 4 is a root illustrates a valid but incomplete formalization. Section 2 still displays the implication in the other direction. Please make the two statements explicit: x = 0 implies x^2 - 4x = 0, whereas the complete answer is captured by x^2 - 4x = 0 if and only if x = 0 or x = 4.

Thank you for your thoughtful follow-up and for recognising the value of the matched-budget comparison in Appendix C. We made changes in blue on the revised script. To clarify your remaining points:

1. Matched generation budget in Table 7 Table 7 matches the generation sample budget (k=10k = 10 generations per problem) across the two settings on the same 100-problem subset (where raw counts out of 100 directly equal percentages):

  • Standalone →\rightarrow Formaliser (10 conclusion generations + 1 formalisation call = 11 LLM calls total): We generate 10 standalone conclusion candidates; if a valid conclusion is identified, 1 deduplicated conclusion is passed to the formaliser for 1 formalisation call (compared to 10×1010 \times 10 in the full-scale pipeline).
  • During autoformalisation / joint (10 joint formalisations = 10 LLM calls total): We generate 10 joint formalisation candidates directly per problem.

Thus, the budget is equal in terms of candidate generation (1010 generations) and nearly identical in total LLM calls (1111 vs. 1010). Under this setup, standalone synthesis yields 9 vs. 1 faithful formalisations for GPT-4.1 and 20 vs. 4 for DeepSeek-V4-Flash. Due to the computational cost and time constraints of running Goedel-Prover-V2 at pass@32 within the rebuttal window, we have not yet run end-to-end proof generation on this subset; we will include the full downstream proof verification results in the camera-ready version.

2. Separate expert audit of faithfulness and conclusion judges (N=100N = 100) In our stratified manual audit of N=100N = 100 typechecked formalisations (50 DeepSeek, 50 GPT-4.1 but a different random set the the ones used in Table 7 as we conduct the manual annotation in parallel for the rebuttal; we will make this clear in the revised version), we evaluated both formalisation faithfulness and conclusion correctness independently against expert annotations:

Expert \ Model JudgeFaithfulNot FaithfulTotal
Faithful82634
Not Faithful56166
Total1387100
  • Formalisation faithfulness (69.0% agreement): The faithfulness judge rejects 92.4% (61/6661/66) of unfaithful statements while falsely rejecting 76.5% (26/3426/34) of faithful ones (accepting only 13/10013/100 vs. 34/10034/100 by the expert). Because false rejections (2626) outnumber false acceptances (55) by more than 5-to-1, the faithfulness filter acts as a conservative lower bound. Inspecting all 5 false positives shows that 3 weaken the goal to an existential (∃) without explicitly stating the requested value (e.g., asserting ∃! a rather than identifying a=2a = 2 in For what integer $a$ does $x^2-x+a$ divide $x^{13}+x+90$?), while 2 involve subtle domain/hypothesis mismatches.
Expert \ Model JudgeCorrectIncorrectTotal
Correct251338
Incorrect95362
Total3466100
  • Conclusion correctness (77.0% overall agreement treating null as incorrect; 84.1% on the 82 non-null verdicts): The judge rejects 84.1% (53/6353/63) of incorrect conclusions and accepts 34/10034/100 vs. 37/10037/100 by the expert. Of the 9 false positives, 7 occur on binary True/False problems (where negated theorem statements without an explicit ↔ False wrapper are misunderstood), 1 occur on numerical problems where the answer cast 1/2 as a natural number which would make it 0 instead of 0.5, and 1 on an algebraic problem where the generated set is not equivalent to the gold set.

3. Consistent counts, denominators, and Section 2 motivating example We have audited all tables and figures so that every rate uses the exact 456-problem denominator (and explicitly distinguishes full-set vs. 100-problem subset evaluations), but we will verify these again in the camera-ready version. Finally, apologies for not reversing the arrow in Section 2 earlier; we have now updated the manuscript so that the incomplete candidate is explicitly written as conclusion x → x ^ 2 - 4 * x = 0.

Reviewer_NwsDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates where conclusion synthesis should occur in the end-to-end theorem-proving pipeline. The reported experiments favor synthesizing conclusions before formalization and proof generation, although a substantial gap remains between obtaining a correct answer and constructing a rigorous formal proof. The low rate of faithful formalization further suggests that autoformalization and formal proof generation remain major bottlenecks in the evaluated pipeline.

Critical strengths
  1. The paper highlights a step often overlooked in conventional theorem-proving evaluations: discovering the answer itself. Proving a given conclusion and independently deriving that conclusion before proving it require distinct capabilities. By explicitly evaluating conclusion synthesis, the paper brings evaluation closer to the demands of mathematical problem solving.

  2. The paper distinguishes three often conflated criteria: conclusion correctness, formalization faithfulness, and proof validity. Evaluating these dimensions separately helps identify whether failures arise from incorrect answers, inaccurate translations of the original problem, or unsuccessful proof construction.

  3. The results reveal patterns that warrant further investigation. Standalone conclusion synthesis generally achieves higher answer accuracy than joint conclusion synthesis and formalization. However, improvements in answer accuracy do not consistently translate into more faithful formalizations or higher proof success rates, highlighting the need to study how progress at individual stages affects end-to-end performance.

Critical weaknesses
  1. LLM-based semantic grading may introduce evaluation errors. Have the authors considered assessing agreement across multiple LLM judges and manually validating a representative subset of their judgments?

  2. Could the authors clarify the dataset rewriting procedure and provide sufficient details to support its reproducibility?

Questions for authors

Please refer to the weaknesses discussed above.

2 comments

Thank you for your comments and feedback, please find an updated version of the paper with revision taking into account your feedback in red, as well as direct responses below.

W1. Reliability of LLM-based semantic evaluation

Thank you for the suggestion. Appendix A compares four conclusion judges against 100 human-labelled outputs. We additionally evaluated Qwen3.6-27B and re-judged the completed standalone and joint settings. Standalone synthesis retains higher conclusion accuracy:

GeneratorStandaloneDuring Autoformalisation
GPT-4.1 259/456 (56.8%) 177/456 (38.8%)
DS-V3.2322/456 (70.6%)170/456 (37.3%) 

W2. Dataset construction and reproducibility

We select instances whose conclusion can be separated from the problem while retaining its assumptions and domain. We then remove the conclusion and rewrite the request according to its type (Table 1):

Numerical: we replace a stated value or tight bound with a question asking for that value, e.g. “at least 1991 red points” becomes “What is the minimum number of red points?”. Algebraic: we ask for the expression instead of supplying it, e.g. “at most 2n − 1 subsets” becomes “What is the maximum number of subsets?” Binary: we replace “Prove P” with “Prove or disprove P”, retaining the proposition but withholding its true value.

Each instance stores the rewritten question, reference answer, formal statement and answer type. The formal answer is separated into “abbrev conjecture := …”. References are withheld from generation prompts outside oracle conditions.

Reviewer_NwsDOpenReview ↗

Official Comment by Reviewer NwsD

Thank you for your detailed response. I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.