acceptodds
Under review as a conference paper at ICLR 2027

Closing the Loop in Formal Theorem Proving: Coordinated and Revisable Proof Construction

Abstract

Agentic theorem provers increasingly combine mathematical reasoning, proof decomposition, and iterative interaction with Lean. When proof construction stalls, the underlying difficulty may arise in formal proving, the informal proof strategy, or the surrounding proof structure, and Lean's diagnostics alone do not identify which source is responsible. Effective recovery requires identifying the source of difficulty and controlling the scope of revision. We present OpenMathHarness (OMH), an agentic theorem-proving framework that coordinates decomposition, informal reasoning, formal proving, and revision within a unified proof construction process. OMH introduces a multi-level review mechanism that uses formal evidence to direct subsequent recovery effort to formal refinement, revision of the informal proof strategy under a fixed statement, or structural revision. Using DeepSeek-V4-Flash as its backbone, OMH solves 53/100 problems on FATE-X and 11/12 on Putnam 2025, at mean inference costs of $1.23 and $0.27 per solved problem respectively. Controlled experiments show that decomposition and informal guidance improve performance on effort-intensive problems, and strategy flexibility nearly halves token usage on commonly solved nodes. Analysis of review and routing decisions further shows how recovery unfolds across formal, strategy, and structural levels.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
formal theorem proving, automated theorem proving, large language models, agentic systems, Lean, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_G8sYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes OpenMathHarness, a persistent proof graph loop that asks a model to continue, revise local guidance, decompose a goal, or revise a parent decomposition while Lean checks the resulting proof artifacts. The end to end results are strong, but structured intermediate proof states, verifier feedback localization, layered recovery, and preservation of verified progress are already represented in prior public systems. The submission does not give a sufficiently specific novelty map or matched controls for its multi level routing, so the results do not establish a distinct contribution.

Critical strengths
  1. The paper clearly constrains what can be changed and anchors final acceptance to the original root theorem: local strategy changes cannot change the current statement, parent revisions cannot weaken the original problem, and final verification recompiles from the original context and checks axiom dependencies. This gives auditable boundaries between repairing code, changing a strategy, and changing a decomposition.

  2. The empirical and process evidence is unusually broad for this kind of systems claim: 212 main runs, a same backbone 10M control, decomposition and guidance ablations, node replay, a parent revision case study, and an external blind audit together show that the loop operates and can reuse verified progress when the structure changes. This supports operational validity and some local recovery, but does not by itself establish an independent gain over prior work or simpler routing.

Critical weaknesses
  1. The paper does not establish sufficient novelty. Its central pattern includes structured intermediate proof state, verifier feedback localization, local repair or strategy revision, decomposition/backtracking, and preservation of verified progress, and it is already represented in prior public verifier guided proof repair systems. The related work discussion does not map these mechanisms one by one or identify a clearly new algorithmic operation, so the broad framework claim remains unsubstantiated.
  2. The distinctive multi level router is not isolated by a matched control. The Without Decomposition and Without Informal Guidance ablations remove capabilities, while Fixed Strategy jointly disables multiple flexibilities; none compares the proposed router with a fixed or heuristic scheduler retaining the same actions, tools, model, and budget. Thus routing counts and successful traces cannot establish causal mechanism effectiveness.
  3. The component evidence has restricted coverage and stochastic support. The 21 problem ablation is selected by full system success and token use, node replay reports only shared successes, and the runs are single trials. These results cannot establish whole distribution stability or general efficiency.
  4. Reproducibility and attribution remain incomplete. Exact prompts/configuration for planner, decomposer, and reviewers, decision logs, model/tool calls, per problem budgets, root reconstruction artifacts, final proofs, and axiom audit outputs are not fully provided, making it difficult to determine which module produced each gain or to replay the claims.
Questions for authors
  1. Please provide a mechanism by mechanism novelty map against prior public verifier guided proof repair systems, covering structured intermediate states, verifier feedback localization, local revision, decomposition/expansion, upstream backtracking, and preservation of validated progress. Which operation in this paper is genuinely new, and which is an adaptation to Lean/Mathlib or a clearer exposition?
  2. Please compare the proposed multi level router with a prespecified fixed or heuristic router that retains the same actions, model, Lean tools, token budget, and randomization. Report the incremental effect of continuing, guidance revision, further decomposition, and parent revision separately.
  3. Please release prompts/configuration and decision logs for planner/decomposer and all reviewers, with acceptance/rejection counts, model calls, token usage, and reasons for the 43 parent revision decisions versus 16 applied revisions.
  4. Please strengthen the related work section by citing and directly comparing recent studies on large language model based formal theorem proving, verifier guided proof repair, proof state management, and hierarchical planning or multi level routing. Please specify which design elements of OpenMathHarness have appeared in these studies, clarify the technical differences and added contributions of each component, and discuss the closest related work in the main text. These comparisons are important for assessing the paper's novelty and positioning.
Reviewer_CKLrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a Lean theorem-proving system that reviews failed attempts and decides whether to repair the proof, change the mathematical strategy, or revise the decomposition. It preserves verified lemmas throughout this process.

Critical strengths
  1. Strong results on difficult algebra benchmarks. With the same model and a 10M-token ceiling, OMH solves 74 versus 28 problems on FATE-H and 23 versus 6 on FATE-X compared with AxProverBase.
  2. Final proofs undergo independent Lean verification and checks for unproved assumptions.
Critical weaknesses
  1. Limited demonstrated novelty. Decomposition, informal guidance, proof graphs, and failure-driven revision already appear in existing systems. OMH’s distinction is its explicit multi-level review process, but the benefit of this organization remains unclear.
  2. The experiments do not isolate that distinction. The controlled comparison is against a simpler agent, while the ablations remove established components such as decomposition and guidance. They do not show whether diagnostic routing improves over a simpler recovery policy with the same tools, actions, and budget. The results therefore establish a strong system, but not a clear advance over comparable structured harnesses.
Questions for authors

What capability does multi-level review provide beyond the revision mechanisms in other harnesses?

Reviewer_2XHxOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OpenMathHarness, which treats to proof failures by selecting Lean-code repair, strategic revision with fixed lemma statements, or lemma-decomposition revision. The local proving and node reviews determine the scope of rewriting the proof graph, while verified lemmas are reused in later trials. Evaluations on FATE-H, FATE-X, and Putnam 2025 report higher solve rates than AxProverBase under the same backbone and token cap. Ablations and lemma-level replays confirmed that proof decomposition and informal guidance surely contribute to the higher solve rates while strategy revision reduces token consumption.

Critical strengths
  1. Recovery from failure is explicitly designed and proposed. Local review selects continued formal proving, strategy revision, or escalation. Node review selects retry, further decomposition, or parent-decomposition revision. This supports revisions at different scopes while preserving verified progress.
  2. Experimental demonstrations on difficult algebra problems. With DeepSeek-V4-Flash, OMH solves 92/100 FATE-H and 53/100 FATE-X problems. On 21 selected problems, the full system solves 18, versus 8 without decomposition and 14 without informal guidance, supporting the usefulness of these components.
Critical weaknesses
  1. The differences from existing harnesses and their effects is not well demonstrated. LEAP already provides proof graphs and lemma reuse; Goedel-Architect also supports failure-driven graph revision. It seems that OMH mainly differs in two-level review and revision routing, but it lacks comparisons such as unified review or disabling parent-decomposition revision strategies. Whether the contribution goes beyond incremental design changes or not remains unclear.
  2. Table 1 mixes the proposed harness effects with the evaluation-condition differences as models, attempt counts, and additional guidance differ. OMH allows up to 100M tokens, versus 10M for the reimplemented AxProverBase. Table 6 shows improvements under the same model and 10M-token cap, but it lacks matched comparisons with Goedel-Architect or LEAP. We assume that this leaves superiority over closely related methods unverified. I also note that the comparison with M2F is missing. M2F reports 96/100 fully automatic FATE-H solves, exceeding OMH’s 92/100. This reminds us that these differences indeed come from models, budgets, and verification criteria. As M2F’s FATE-X score could not be confirmed, their relative performance on this benchmark cannot be separately assessed.
Questions for authors

See Weakness. Can you also clarify the comparison with M2F?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.