- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes OpenMathHarness, a persistent proof graph loop that asks a model to continue, revise local guidance, decompose a goal, or revise a parent decomposition while Lean checks the resulting proof artifacts. The end to end results are strong, but structured intermediate proof states, verifier feedback localization, layered recovery, and preservation of verified progress are already represented in prior public systems. The submission does not give a sufficiently specific novelty map or matched controls for its multi level routing, so the results do not establish a distinct contribution.
Critical strengths
-
The paper clearly constrains what can be changed and anchors final acceptance to the original root theorem: local strategy changes cannot change the current statement, parent revisions cannot weaken the original problem, and final verification recompiles from the original context and checks axiom dependencies. This gives auditable boundaries between repairing code, changing a strategy, and changing a decomposition.
-
The empirical and process evidence is unusually broad for this kind of systems claim: 212 main runs, a same backbone 10M control, decomposition and guidance ablations, node replay, a parent revision case study, and an external blind audit together show that the loop operates and can reuse verified progress when the structure changes. This supports operational validity and some local recovery, but does not by itself establish an independent gain over prior work or simpler routing.
Critical weaknesses
- The paper does not establish sufficient novelty. Its central pattern includes structured intermediate proof state, verifier feedback localization, local repair or strategy revision, decomposition/backtracking, and preservation of verified progress, and it is already represented in prior public verifier guided proof repair systems. The related work discussion does not map these mechanisms one by one or identify a clearly new algorithmic operation, so the broad framework claim remains unsubstantiated.
- The distinctive multi level router is not isolated by a matched control. The Without Decomposition and Without Informal Guidance ablations remove capabilities, while Fixed Strategy jointly disables multiple flexibilities; none compares the proposed router with a fixed or heuristic scheduler retaining the same actions, tools, model, and budget. Thus routing counts and successful traces cannot establish causal mechanism effectiveness.
- The component evidence has restricted coverage and stochastic support. The 21 problem ablation is selected by full system success and token use, node replay reports only shared successes, and the runs are single trials. These results cannot establish whole distribution stability or general efficiency.
- Reproducibility and attribution remain incomplete. Exact prompts/configuration for planner, decomposer, and reviewers, decision logs, model/tool calls, per problem budgets, root reconstruction artifacts, final proofs, and axiom audit outputs are not fully provided, making it difficult to determine which module produced each gain or to replay the claims.
Questions for authors
- Please provide a mechanism by mechanism novelty map against prior public verifier guided proof repair systems, covering structured intermediate states, verifier feedback localization, local revision, decomposition/expansion, upstream backtracking, and preservation of validated progress. Which operation in this paper is genuinely new, and which is an adaptation to Lean/Mathlib or a clearer exposition?
- Please compare the proposed multi level router with a prespecified fixed or heuristic router that retains the same actions, model, Lean tools, token budget, and randomization. Report the incremental effect of continuing, guidance revision, further decomposition, and parent revision separately.
- Please release prompts/configuration and decision logs for planner/decomposer and all reviewers, with acceptance/rejection counts, model calls, token usage, and reasons for the 43 parent revision decisions versus 16 applied revisions.
- Please strengthen the related work section by citing and directly comparing recent studies on large language model based formal theorem proving, verifier guided proof repair, proof state management, and hierarchical planning or multi level routing. Please specify which design elements of OpenMathHarness have appeared in these studies, clarify the technical differences and added contributions of each component, and discuss the closest related work in the main text. These comparisons are important for assessing the paper's novelty and positioning.