- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a reference-free iterative refinement framework for full-theorem autoformalization, aiming to translate natural-language theorem statements and proofs into machine-verified formalizations. The authors introduce a masked composite objective combining formal validity with several soft quality dimensions, and develop a monotonic acceptance policy for selecting candidates generated by different LLMs. Experiments on miniF2F and ProofNet demonstrate improvements in formal validity and overall formalization quality.
Critical strengths
- The paper aims at an important practical challenge: producing formally verified proofs while preserving the mathematical meaning of the original natural-language content.
- The proposed reference-free refinement framework and its integration of complementary LLM capabilities are potentially useful, with promising empirical results.
Critical weaknesses
I am concerned that the paper introduces considerable mathematical complexity without providing commensurate theoretical insight. Theoretical contributions should be evaluated by the new understanding they offer, rather than by the sophistication of their presentation. In this case, many of the derived properties appear to be generic consequences of the chosen optimization mechanism, rather than distinctive properties of autoformalization.
W1. The theoretical contribution of the proposed formalization is substantially overstated. The authors claim that formally defining full-theorem autoformalization is a major theoretical contribution. However, I find the proposed formalization largely amounts to defining a multi-dimensional scoring function, rather than providing a principled theoretical characterization of the problem. Specifically, the framework introduces several hard and soft quality dimensions and aggregates their scores into an optimization objective. Yet the authors provide little theoretical justification for the choice of these dimensions, their aggregation, or the relationships and potential trade-offs among them. Consequently, the formulation does not offer substantial new insight into the fundamental nature of full-theorem autoformalization. Moreover, multi-dimensional quality assessment is not new to autoformalization; prior work such as LeanScorer[1] has already explored decomposed semantic evaluation and score aggregation. I acknowledge that the authors provide an explicit mathematical formulation of their proposed objective. However, formalizing a particular evaluation function should not be conflated with establishing a novel theoretical understanding of the underlying problem. The authors should substantially moderate their theoretical novelty claims and clarify what insights their formulation provides beyond existing evaluation frameworks.
W2. The convergence analysis provides limited theoretical insight. The convergence analysis provides limited theoretical insight. Under the intended validity-preservation condition, the LCB differs from the estimated score only by a constant. Since the acceptance rule ensures that this bounded score is non-decreasing, convergence follows directly from the elementary monotone convergence theorem. This argument is generic to iterative refinement with bounded scoring functions and monotonic acceptance mechanisms, and applies equally to tasks such as automated program repair and machine translation. Therefore, the result does not, by itself, reveal any distinctive theoretical property of full-theorem autoformalization.
W3. The claimed monotonic improvement in true formalization quality is not established. The proposed optimization process relies on quality estimates provided by LLM judges, but there is a substantial gap between these estimated scores and the true quality of formalization results. Monotonic improvement in estimated quality does not necessarily imply monotonic improvement in true quality. Although the authors introduce uncertainty assumptions and lower confidence bounds (LCBs) to address this issue, their theoretical analysis does not bridge this gap. In particular, Theorem 2.4 only guarantees that either the true objective is non-decreasing or the LCB increases. An increasing LCB does not imply an increase in true quality, meaning that the proposed process may still accept candidates with lower actual quality. Therefore, the current theoretical results are insufficient to support the paper's central claim of certified monotonic refinement. I recommend conducting independent, paired human evaluations of consecutive accepted candidates to measure the proportions of genuine improvements, regressions, and unchanged results. This would provide more direct evidence that the proposed refinement mechanism improves actual autoformalization quality rather than merely optimizing the preferences of its LLM judges.
W4. The characterization of full-theorem autoformalization as inherently a multi-objective optimization problem is not justified. In Section 1, the authors state that full-theorem autoformalization is "inherently a multi-dimensional optimization problem." I disagree with this characterization. The fundamental task is to construct a formally verified theorem and proof that faithfully represent the original mathematical content. This task does not inherently require a multi-objective optimization formulation. For instance, it could equally be formulated as a constrained feasibility problem, where formal validity and semantic faithfulness are mandatory conditions, while properties such as clarity, conciseness, and structural quality are optional preferences. The distinction between hard and soft dimensions, the particular selection of FV, LP, MC, and FQ, and their aggregation into a weighted composite objective are modeling decisions introduced by the authors, rather than intrinsic properties of the underlying task. In particular, Formal Quality is a desirable property but not necessarily a defining requirement for successful autoformalization. Moreover, the paper ultimately optimizes a single scalarized objective whose weighting scheme is not established as canonical. I believe the authors should distinguish the mathematical task itself from their proposed optimization framework. A more accurate statement would be: "We formulate full-theorem autoformalization as a multi-dimensional optimization problem by introducing hard and soft quality dimensions." This would appropriately position the proposed formulation as a methodological choice, without making an unjustified claim about the fundamental nature of the problem.
References [1] Yu, Xuejun, et al. "Mathesis: Towards formal theorem proving from natural languages." International Conference on Learning Representations. Vol. 2026. 2026.
Questions for authors
Q1. How do the authors justify that the proposed objective (J_{\mathrm{OA}}) faithfully represents the true quality of full-theorem autoformalization? In particular, what justifies the additive aggregation and equal weighting of LP, MC, and FQ? Why should a severe failure in an essential dimension be compensable by high scores in the other dimensions? Furthermore, could the authors provide a theoretical or empirical analysis of the relationships and correlations among these soft dimensions, especially clarifying the substantive distinction between LP and MC?
3 comments
Rebuttal to Reviewer oxHd (continue)
Q1. On aggregation, weighting, and the distinction between LP and MC
We thank the reviewer for highlighting the importance of distinguishing the mathematical requirements of autoformalization from the particular objective used to guide optimization.
Aggregation and weighting. Our masked objective treats formal validity as a hard requirement while combining the soft dimensions into a graded quality signal. As clarified in Section 3.1, equal weights are adopted for simplicity and are not claimed to define a canonical measure of formalization quality. We acknowledge that additive aggregation allows compensation among soft dimensions. The composite objective is intended as a practical criterion for comparing and selecting candidate formalizations during refinement, rather than a complete definition of semantic correctness. For applications requiring stricter semantic preservation, the framework permits alternative weighting schemes or additional constraints. We therefore do not claim that the chosen scalarization uniquely represents true formalization quality.
Distinction between LP and MC. As defined in Section 2.1 and further specified through the evaluation criteria in Appendix I, LP and MC assess different aspects of semantic faithfulness:
-
Logical Preservation (LP) evaluates whether the formal theorem preserves the logical structure and content of the original natural-language statement.
-
Mathematical Consistency (MC) evaluates whether the formal theorem accurately represents the mathematical objects, relations, and operations appearing in the original statement.
These dimensions are conceptually distinguishable. A formalization may preserve the logical structure of a statement while misrepresenting a mathematical operation; conversely, it may represent the intended mathematical objects correctly while modifying logical assumptions or quantification. For example, consider the natural-language theorem . Formalizing it as preserves its logical structure but changes the intended mathematical operation, illustrating an MC error. Conversely, formalizing it as preserves the mathematical objects and operations but introduces an additional assumption, illustrating an LP error. Both statements are formally provable, yet neither faithfully represents the original theorem.
This distinction motivates evaluating LP and MC separately rather than treating semantic fidelity as a single undifferentiated score. Nevertheless, we do not claim that LP and MC are statistically independent, nor does the current experimental evaluation establish their empirical independence. Our contribution is to incorporate these complementary quality assessments into the refinement and candidate-selection procedure. We have clarified the interpretation and limitations of the objective accordingly.
Reviewer oxHd, thanks for your review. Could you take a look at the authors’ response and comment on which of your concerns have or have not been resolved?
Rebuttal to Reviewer oxHd
Dear Reviewer oxHd,
We thank the reviewer for the constructive feedback. We have substantially revised the manuscript to clarify the scope of these contributions and distinguish methodological design choices from fundamental properties of autoformalization. We address the main questions and concerns below:
W1 & W4. On theoretical novelty and the multi-dimensional formulation
We agree that full-theorem autoformalization does not uniquely require a multi-objective optimization formulation. In particular, the proposed scalarization and the choice of soft dimensions are methodological design decisions, rather than a canonical mathematical characterization of the task. Our motivation is that a complete formalization must satisfy multiple distinct quality requirements. Formal validity ensures that a theorem and proof are accepted by the theorem prover, but does not necessarily establish that the formal statement faithfully represents the original mathematical content. Conversely, semantic fidelity alone is insufficient without a machine-verified proof.
These requirements motivate our distinction between the hard dimension of formal validity and the soft dimensions of logical preservation, mathematical consistency, and formal quality. We also agree that decomposed semantic evaluation and score aggregation have been explored in prior work. We do not claim that multi-dimensional scoring is novel in itself. Rather, our contribution lies in integrating multi-dimensional quality assessment into a reference-free, iterative framework for joint statement–proof formalization.
Specifically, the framework combines:
The distinction from decomposed evaluation alone is that quality estimates are not used solely to assess final outputs; they also provide feedback and selection criteria within the iterative refinement process. We have revised the title, Introduction, and contribution statements to emphasize this methodological contribution rather than presenting the scalarized objective as a new theoretical characterization of the underlying task.
W2. On the convergence analysis
We agree that convergence of a bounded, eventually non-decreasing sequence follows from the classical monotone convergence theorem, and we do not claim that this mathematical principle is novel. The purpose of Theorem 2.5 is instead to characterize the behavior of our particular acceptance procedure under the stated uncertainty model. Specifically, it establishes eventual non-decreasing behavior and convergence of the LCB sequence, together with a conditional termination result when a candidate achieves the maximum estimated score.
These results describe the behavior of the proposed refinement process, rather than establishing a general convergence principle unique to autoformalization. Accordingly, we have moderated the theoretical novelty claims and repositioned the convergence analysis as a supporting characterization of the proposed method.
W3. On monotonic improvement in true formalization quality
We agree that monotonic improvement in estimated quality does not imply monotonic improvement in true semantic quality. As clarified in revised Section 2.3.2, the acceptance policy guarantees non-decreasing estimated composite scores. Theorem 2.4 establishes only the disjunctive criterion that either the true objective does not decrease or the LCB increases. The revised manuscript explicitly acknowledges that an increasing LCB does not necessarily imply higher true quality. We therefore no longer interpret the theorem as certifying monotonic improvement in true formalization quality.
The empirical evaluations provide complementary evidence regarding the behavior of the refinement process. Independent human evaluation shows increasing aggregate scores on both miniF2F and ProofNet (Table 4), while the cross-model results also reveal evaluator-dependent differences. These findings support improvement in aggregate evaluated quality, but they do not establish that every accepted transition improves true quality. We agree that paired independent human evaluations of consecutive accepted candidates would provide stronger validation, and we acknowledge this limitation.