acceptodds
Under review as a conference paper at ICLR 2027

Monotonic Reference-Free Refinement for Autoformalization

Abstract

While statement autoformalization has advanced rapidly, full-theorem autoformalization remains largely unexplored. Existing iterative refinement methods for statement autoformalization typically improve isolated aspects of formalization quality, such as syntactic correctness, but struggle to jointly optimize multiple dimensions of quality, a key requirement for full-theorem autoformalization. We introduce a reference-free iterative monotonic process for full-theorem autoformalization at inference time that leverages complementary feedback from theorem provers and LLM-based judges, without access to existing formalizations or human intervention. Our approach optimizes a masked composite objective over a hard dimension, formal validity as determined by a theorem prover, and a set of soft dimensions. We introduce an acceptance policy over candidate formalizations that provides a certified notion of monotonic progress, ensuring at each iteration that either the true objective does not decrease or its lower confidence bound increases, while establishing conditions for convergence and termination. We also propose a responsiveness map that characterizes how different LLMs, acting in different roles, preferentially improve these dimensions, enabling principled generator selection. Experiments show that our method simultaneously improves multiple dimensions of formalization quality, achieving overall scores of 90.27% on miniF2F and 52.45% on ProofNet.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
autoformalization, monotonic process, optimization at test-time

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oxHdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a reference-free iterative refinement framework for full-theorem autoformalization, aiming to translate natural-language theorem statements and proofs into machine-verified formalizations. The authors introduce a masked composite objective combining formal validity with several soft quality dimensions, and develop a monotonic acceptance policy for selecting candidates generated by different LLMs. Experiments on miniF2F and ProofNet demonstrate improvements in formal validity and overall formalization quality.

Critical strengths
  1. The paper aims at an important practical challenge: producing formally verified proofs while preserving the mathematical meaning of the original natural-language content.
  2. The proposed reference-free refinement framework and its integration of complementary LLM capabilities are potentially useful, with promising empirical results.
Critical weaknesses

I am concerned that the paper introduces considerable mathematical complexity without providing commensurate theoretical insight. Theoretical contributions should be evaluated by the new understanding they offer, rather than by the sophistication of their presentation. In this case, many of the derived properties appear to be generic consequences of the chosen optimization mechanism, rather than distinctive properties of autoformalization.

W1. The theoretical contribution of the proposed formalization is substantially overstated. The authors claim that formally defining full-theorem autoformalization is a major theoretical contribution. However, I find the proposed formalization largely amounts to defining a multi-dimensional scoring function, rather than providing a principled theoretical characterization of the problem. Specifically, the framework introduces several hard and soft quality dimensions and aggregates their scores into an optimization objective. Yet the authors provide little theoretical justification for the choice of these dimensions, their aggregation, or the relationships and potential trade-offs among them. Consequently, the formulation does not offer substantial new insight into the fundamental nature of full-theorem autoformalization. Moreover, multi-dimensional quality assessment is not new to autoformalization; prior work such as LeanScorer[1] has already explored decomposed semantic evaluation and score aggregation. I acknowledge that the authors provide an explicit mathematical formulation of their proposed objective. However, formalizing a particular evaluation function should not be conflated with establishing a novel theoretical understanding of the underlying problem. The authors should substantially moderate their theoretical novelty claims and clarify what insights their formulation provides beyond existing evaluation frameworks.

W2. The convergence analysis provides limited theoretical insight. The convergence analysis provides limited theoretical insight. Under the intended validity-preservation condition, the LCB differs from the estimated score only by a constant. Since the acceptance rule ensures that this bounded score is non-decreasing, convergence follows directly from the elementary monotone convergence theorem. This argument is generic to iterative refinement with bounded scoring functions and monotonic acceptance mechanisms, and applies equally to tasks such as automated program repair and machine translation. Therefore, the result does not, by itself, reveal any distinctive theoretical property of full-theorem autoformalization.

W3. The claimed monotonic improvement in true formalization quality is not established. The proposed optimization process relies on quality estimates provided by LLM judges, but there is a substantial gap between these estimated scores and the true quality of formalization results. Monotonic improvement in estimated quality does not necessarily imply monotonic improvement in true quality. Although the authors introduce uncertainty assumptions and lower confidence bounds (LCBs) to address this issue, their theoretical analysis does not bridge this gap. In particular, Theorem 2.4 only guarantees that either the true objective is non-decreasing or the LCB increases. An increasing LCB does not imply an increase in true quality, meaning that the proposed process may still accept candidates with lower actual quality. Therefore, the current theoretical results are insufficient to support the paper's central claim of certified monotonic refinement. I recommend conducting independent, paired human evaluations of consecutive accepted candidates to measure the proportions of genuine improvements, regressions, and unchanged results. This would provide more direct evidence that the proposed refinement mechanism improves actual autoformalization quality rather than merely optimizing the preferences of its LLM judges.

W4. The characterization of full-theorem autoformalization as inherently a multi-objective optimization problem is not justified. In Section 1, the authors state that full-theorem autoformalization is "inherently a multi-dimensional optimization problem." I disagree with this characterization. The fundamental task is to construct a formally verified theorem and proof that faithfully represent the original mathematical content. This task does not inherently require a multi-objective optimization formulation. For instance, it could equally be formulated as a constrained feasibility problem, where formal validity and semantic faithfulness are mandatory conditions, while properties such as clarity, conciseness, and structural quality are optional preferences. The distinction between hard and soft dimensions, the particular selection of FV, LP, MC, and FQ, and their aggregation into a weighted composite objective are modeling decisions introduced by the authors, rather than intrinsic properties of the underlying task. In particular, Formal Quality is a desirable property but not necessarily a defining requirement for successful autoformalization. Moreover, the paper ultimately optimizes a single scalarized objective whose weighting scheme is not established as canonical. I believe the authors should distinguish the mathematical task itself from their proposed optimization framework. A more accurate statement would be: "We formulate full-theorem autoformalization as a multi-dimensional optimization problem by introducing hard and soft quality dimensions." This would appropriately position the proposed formulation as a methodological choice, without making an unjustified claim about the fundamental nature of the problem.

References [1] Yu, Xuejun, et al. "Mathesis: Towards formal theorem proving from natural languages." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

Q1. How do the authors justify that the proposed objective (J_{\mathrm{OA}}) faithfully represents the true quality of full-theorem autoformalization? In particular, what justifies the additive aggregation and equal weighting of LP, MC, and FQ? Why should a severe failure in an essential dimension be compensable by high scores in the other dimensions? Furthermore, could the authors provide a theoretical or empirical analysis of the relationships and correlations among these soft dimensions, especially clarifying the substantive distinction between LP and MC?

3 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer oxHd

Dear Reviewer oxHd,

We thank the reviewer for the constructive feedback. We have substantially revised the manuscript to clarify the scope of these contributions and distinguish methodological design choices from fundamental properties of autoformalization. We address the main questions and concerns below:

W1 & W4. On theoretical novelty and the multi-dimensional formulation

We agree that full-theorem autoformalization does not uniquely require a multi-objective optimization formulation. In particular, the proposed scalarization and the choice of soft dimensions are methodological design decisions, rather than a canonical mathematical characterization of the task. Our motivation is that a complete formalization must satisfy multiple distinct quality requirements. Formal validity ensures that a theorem and proof are accepted by the theorem prover, but does not necessarily establish that the formal statement faithfully represents the original mathematical content. Conversely, semantic fidelity alone is insufficient without a machine-verified proof.

These requirements motivate our distinction between the hard dimension of formal validity and the soft dimensions of logical preservation, mathematical consistency, and formal quality. We also agree that decomposed semantic evaluation and score aggregation have been explored in prior work. We do not claim that multi-dimensional scoring is novel in itself. Rather, our contribution lies in integrating multi-dimensional quality assessment into a reference-free, iterative framework for joint statement–proof formalization.

Specifically, the framework combines:

  • A masked composite objective that treats formal validity as a hard requirement;
  • Three generator roles—One-Off Generators, FV-Repairers, and Recurrent Generators—with dimension-specific responsiveness;
  • An acceptance policy that uses theorem-prover verification and LLM-based judgments to guide candidate selection across refinement iterations.

The distinction from decomposed evaluation alone is that quality estimates are not used solely to assess final outputs; they also provide feedback and selection criteria within the iterative refinement process. We have revised the title, Introduction, and contribution statements to emphasize this methodological contribution rather than presenting the scalarized objective as a new theoretical characterization of the underlying task.

W2. On the convergence analysis

We agree that convergence of a bounded, eventually non-decreasing sequence follows from the classical monotone convergence theorem, and we do not claim that this mathematical principle is novel. The purpose of Theorem 2.5 is instead to characterize the behavior of our particular acceptance procedure under the stated uncertainty model. Specifically, it establishes eventual non-decreasing behavior and convergence of the LCB sequence, together with a conditional termination result when a candidate achieves the maximum estimated score.

These results describe the behavior of the proposed refinement process, rather than establishing a general convergence principle unique to autoformalization. Accordingly, we have moderated the theoretical novelty claims and repositioned the convergence analysis as a supporting characterization of the proposed method.

W3. On monotonic improvement in true formalization quality

We agree that monotonic improvement in estimated quality does not imply monotonic improvement in true semantic quality. As clarified in revised Section 2.3.2, the acceptance policy guarantees non-decreasing estimated composite scores. Theorem 2.4 establishes only the disjunctive criterion that either the true objective does not decrease or the LCB increases. The revised manuscript explicitly acknowledges that an increasing LCB does not necessarily imply higher true quality. We therefore no longer interpret the theorem as certifying monotonic improvement in true formalization quality.

The empirical evaluations provide complementary evidence regarding the behavior of the refinement process. Independent human evaluation shows increasing aggregate scores on both miniF2F and ProofNet (Table 4), while the cross-model results also reveal evaluator-dependent differences. These findings support improvement in aggregate evaluated quality, but they do not establish that every accepted transition improves true quality. We agree that paired independent human evaluations of consecutive accepted candidates would provide stronger validation, and we acknowledge this limitation.

AuthorsOpenReview ↗

Rebuttal to Reviewer oxHd (continue)

Q1. On aggregation, weighting, and the distinction between LP and MC

We thank the reviewer for highlighting the importance of distinguishing the mathematical requirements of autoformalization from the particular objective used to guide optimization.

Aggregation and weighting. Our masked objective treats formal validity as a hard requirement while combining the soft dimensions into a graded quality signal. As clarified in Section 3.1, equal weights are adopted for simplicity and are not claimed to define a canonical measure of formalization quality. We acknowledge that additive aggregation allows compensation among soft dimensions. The composite objective is intended as a practical criterion for comparing and selecting candidate formalizations during refinement, rather than a complete definition of semantic correctness. For applications requiring stricter semantic preservation, the framework permits alternative weighting schemes or additional constraints. We therefore do not claim that the chosen scalarization uniquely represents true formalization quality.

Distinction between LP and MC. As defined in Section 2.1 and further specified through the evaluation criteria in Appendix I, LP and MC assess different aspects of semantic faithfulness:

  • Logical Preservation (LP) evaluates whether the formal theorem preserves the logical structure and content of the original natural-language statement.

  • Mathematical Consistency (MC) evaluates whether the formal theorem accurately represents the mathematical objects, relations, and operations appearing in the original statement.

These dimensions are conceptually distinguishable. A formalization may preserve the logical structure of a statement while misrepresenting a mathematical operation; conversely, it may represent the intended mathematical objects correctly while modifying logical assumptions or quantification. For example, consider the natural-language theorem ∀x∈R, x2≥0\forall x\in\mathbb R,\ x^2\geq 0. Formalizing it as ∀x∈R, x4≥0\forall x\in\mathbb R,\ x^4\geq 0 preserves its logical structure but changes the intended mathematical operation, illustrating an MC error. Conversely, formalizing it as ∀x∈R, x≥0⇒x2≥0\forall x\in\mathbb R,\ x\geq 0\Rightarrow x^2\geq 0 preserves the mathematical objects and operations but introduces an additional assumption, illustrating an LP error. Both statements are formally provable, yet neither faithfully represents the original theorem.

This distinction motivates evaluating LP and MC separately rather than treating semantic fidelity as a single undifferentiated score. Nevertheless, we do not claim that LP and MC are statistically independent, nor does the current experimental evaluation establish their empirical independence. Our contribution is to incorporate these complementary quality assessments into the refinement and candidate-selection procedure. We have clarified the interpretation and limitations of the objective accordingly.

Reviewer oxHd, thanks for your review. Could you take a look at the authors’ response and comment on which of your concerns have or have not been resolved?

Reviewer_g8XoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a reference-free framework for jointly formalizing natural-language theorem statements and proofs. It combines role-specialized generators and repairers, Lean-based verification, and LLM judgments of semantic fidelity and formal quality. Experiments on miniF2F and ProofNet show improvements over sampling and iterative-refinement baselines.

Critical strengths
  1. The paper addresses an important limitation of verification-only evaluation.
    A Lean-verified proof does not establish that the formal statement faithfully represents the original problem. Combining formal validity with semantic and quality assessments is therefore well motivated and relevant to reliable autoformalization.

  2. The framework offers evidence of complementary model capabilities.
    Separating generation, validity repair, and semantic refinement provides a promising way to combine specialized and general-purpose models. Improvements under external and human evaluation suggest that the gains are not solely artifacts of the acceptance judge, although their magnitude remains evaluator-dependent.

Critical weaknesses
  1. The experiments do not sufficiently isolate the contributions of the proposed design.
    Table 7 reports 53.35 generation calls and 191.55 judge calls per instance for the ensemble, substantially exceeding Best-of-8. The gains therefore cannot be clearly attributed to the proposed mechanism rather than additional computation. Best-of-32 and Best-of-64 would provide starting points for approximately matching judge and generation budgets, respectively, with actual calls and preferably token costs used to calibrate the comparisons.

    The necessity of joint statement–proof optimization also lacks a direct ablation. A budget-matched two-stage baseline that fixes the generated statement, alongside an oracle experiment using correct statements, would disentangle statement and proof formalization bottlenecks. Statement revisions should also be checked for genuine semantic corrections rather than improved provability through stronger assumptions or weaker conclusions.

  2. Validation of judge reliability and practical applicability remains insufficient.
    Despite single-expert and cross-model evaluations, scores differ substantially across evaluators, and instance-level human–LLM agreement is not reported. It remains unclear whether independent experts also consider accepted refinements better, rather than merely higher-scoring under the LLM judges. This directly affects confidence in the claimed soft-quality improvements.

    Evaluation is also limited to miniF2F and ProofNet, without proof-length distributions or length-stratified results. Real-world mathematical problems may require very long proofs with interdependent lemmas, where full-proof refinement could introduce non-local errors and judges could miss semantic issues while incurring substantial costs. Representative long-proof experiments would better establish practical applicability.

Questions for authors
  1. Budget-controlled comparison: Does the ensemble retain its advantage over larger Best-of-k baselines under comparable generation and judge budgets? Please report actual computational costs alongside performance.

  2. Design ablation: How does joint optimization compare with fixing an automatically generated statement or providing a correct oracle statement, while retaining the same natural-language proof input? How often do statement revisions correct versus introduce semantic errors?

  3. Judge validation: What is the instance-level agreement between LLM and human judgments? In particular, how often do independent experts prefer an accepted refinement over its predecessor when both pass Lean verification?

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer g8Xo

Dear Reviewer g8Xo,

We thank the reviewer for the constructive feedback. We address the main questions and concerns below:

  1. On budget-controlled comparisons

Please refer to Point 2 of our general rebuttal.

  1. On the necessity of joint statement–proof optimization

We agree that a direct comparison with fixed-statement and oracle-statement pipelines would more precisely quantify the contribution of joint optimization. Our motivation, as clarified in Section 1, is that statement formalization and proof generation are interdependent. In a two-stage pipeline that fixes the formal statement before proof generation, semantic errors introduced during statement formalization cannot subsequently be corrected in light of the proof.

In contrast, our framework represents a full formalization as a statement–proof pair and permits recurrent generators to revise both components using theorem-prover and LLM feedback. This allows formal validity and semantic alignment to be considered throughout the refinement process, rather than treating the generated statement as immutable.

Moreover, Lean verification alone cannot establish semantic faithfulness: a formally verified proof may still establish a statement that differs from the intended natural-language theorem. This motivates incorporating LP and MC alongside formal validity when evaluating candidate formalizations.

We emphasize that our contribution is a framework enabling such joint refinement, rather than an empirical demonstration that joint optimization is necessarily superior to every fixed-statement pipeline. The current experiments do not directly quantify how frequently statement revisions correct or introduce semantic errors, and we acknowledge this limitation.

  1. On judge reliability and human evaluation

We agree that the reliability of LLM-based quality judgments is important for evaluating the proposed refinement framework. To mitigate evaluation bias, our experiments employ three complementary sources of assessment: Lean4 verification for formal validity, GPT-4.1-mini judges for candidate acceptance, and independent Qwen2.5-Coder-7B and human evaluations for soft dimensions (Figure 2; Table 4). The human evaluation shows increasing overall scores across iterations, from 49.46% to 66.60% on miniF2F and from 15.71% to 40.96% on ProofNet. Human-evaluated LP and MC also improve on both benchmarks. These results provide supporting evidence that the aggregate gains are not exclusively attributable to preferences of the acceptance judges.

Nevertheless, the evaluations also reveal differences between evaluators. In particular, Qwen-evaluated LP and MC decrease on ProofNet, while human-evaluated LP and MC increase. Furthermore, human-evaluated FQ on ProofNet decreases slightly. We therefore do not interpret the optimization-judge scores as ground-truth semantic quality. As discussed in Appendix J, the human evaluation relies on a single annotator, and inter-annotator agreement has not been established. The current results also do not provide instance-level agreement or paired independent expert preferences between consecutive accepted candidates. We regard the independent evaluations as supporting evidence for aggregate improvement, rather than verification that every accepted refinement improves true semantic quality.

  1. On applicability to longer proofs

We agree that evaluation on substantially longer proofs with interdependent lemmas is important for establishing practical applicability. Our experiments focus on miniF2F and ProofNet, which provide natural-language statements together with corresponding proofs and therefore support evaluation of full-theorem autoformalization. ProofNet offers more complex mathematical problems than miniF2F, providing an additional setting in which to evaluate the framework.

Nevertheless, these benchmarks do not establish robustness to very long proofs, where non-local dependencies, semantic errors, and computational costs may present additional challenges. We have therefore limited our empirical conclusions to the evaluated benchmarks and acknowledge that length-stratified evaluation and more complex formalization tasks are needed to assess practical scalability.

Reviewer g8Xo, thanks for your review. Could you take a look at the authors’ response and comment on which of your concerns have or have not been resolved?

Reviewer_ywYeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a reference-free iterative refinement framework for full-theorem autoformalization, jointly generating formal theorem statements and proofs. It formulates the task as multi-objective optimization, combining theorem-prover-verified formal validity with LLM-judged semantic and formal quality dimensions. The method uses role-specialized generators, a responsiveness map, and an acceptance policy with theoretical guarantees under judge uncertainty. Experiments on miniF2F and ProofNet show improvements over iterative self-refinement and Best-of-k baselines, achieving overall scores of 90.27% and 52.45%, respectively.

Critical strengths
  1. A meaningful formulation of full-theorem autoformalization. The paper moves beyond statement-level formalization by considering both formal statements and their proofs. Treating formal validity as a hard requirement while optimizing semantic fidelity and formal quality addresses an important limitation of existing refinement-based methods. This provides a useful perspective for designing more reliable autoformalization systems.

  2. Substantial empirical evaluation with complementary validation. The paper evaluates multiple LLMs and generator roles on miniF2F and ProofNet, including comparisons against Best-of-k and iterative self-refinement. The additional evaluation with external LLM judges and a human expert is particularly valuable, as it provides evidence that the observed improvements are not solely artifacts of the optimization judge.

Critical weaknesses
  1. The theoretical guarantee is weaker than the claimed monotonic improvement in formalization quality. Theorem 2.4 guarantees that either the true objective does not decrease or its lower confidence bound increases, but this does not imply monotonic improvement of the true objective. In fact, the proof in Appendix B explicitly permits transitions from formally valid to invalid candidates under this criterion. Moreover, the acceptance policy operates on the estimated objective rather than directly optimizing calibrated lower confidence bounds. The confidence guarantees also depend on assumptions about judge reliability that are not fully validated for adaptively selected candidates. Consequently, the theoretical results provide a weaker justification for the claimed reliability than the paper suggests.

  2. The experimental comparisons do not sufficiently isolate the benefits of the proposed optimization strategy. According to Table 7, the ensemble method uses an average of 53.35 generator calls per instance, compared with only eight for Best-of-k. Without a compute-matched baseline, it is unclear how much of the improvement comes from the proposed acceptance policy and role specialization rather than additional sampling. Furthermore, the independent Qwen evaluation on ProofNet shows decreases in LP and MC despite improvements in the overall score, suggesting that improvements across quality dimensions are less consistent than claimed. These issues limit the strength of the empirical conclusions.

Questions for authors
  1. Clarification on theoretical guarantees: Can the authors clarify what meaningful notion of progress is guaranteed when the true objective decreases but its lower confidence bound increases? How are the confidence guarantees justified under adaptive candidate selection and potentially correlated LLM judges?

  2. Clarification on empirical attribution: Can the authors provide a compute-matched comparison against Best-of-N or multi-model candidate sampling with the same judge-based selection policy? In particular, how much improvement remains when the number of generator and judge calls is controlled? This would help establish whether the gains come from the proposed optimization framework rather than additional test-time computation.

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer ywYe

Dear Reviewer ywYe,

We thank the reviewer for the constructive feedback. We address the main questions and concerns below:

  1. On the theoretical guarantee and judge uncertainty

We agree that Theorem 2.4 does not establish unconditional monotonic improvement in true formalization quality. We have revised Section 2.3.2 to explicitly distinguish the guarantees:

  • Estimated-objective monotonicity: The acceptance policy in Eqs. (8)–(9) guarantees that the estimated composite objective is non-decreasing across iterations.

  • Conditional theoretical progress: Theorem 2.4 establishes that either the true objective does not decrease or the lower confidence bound (LCB) increases. An increasing LCB does not necessarily imply improvement in true quality.

The acceptance rule also has a conditional formal-validity preservation property, as established in the proof of Theorem 2.5 (Appendix C). Once the current candidate is formally valid and its estimated composite score exceeds ϵ\epsilon, all subsequent accepted candidates must remain formally valid. This follows because any invalid candidate has an estimated score at most ϵ\epsilon and therefore cannot satisfy the strict acceptance condition.

Thus, although true semantic quality is not guaranteed to improve monotonically, the acceptance mechanism provides a well-defined estimated-score progression and preserves formal validity once this condition is satisfied.

Regarding judge uncertainty, Assumption 2.1 explicitly models asymptotic stability and one-sided estimation uncertainty, with calibration margins accounting for factors including inter-judge correlation and distributional drift. Theorem 2.2 additionally assumes independence across soft-dimension evaluations when deriving its composite LCB. We acknowledge that these assumptions do not establish simultaneous confidence coverage for adaptively selected candidates. Accordingly, we have moderated our theoretical claims and clarified that the LCB characterizes the process under the stated assumptions rather than certifying monotonic improvement in true quality.

  1. On compute-controlled comparisons

Please refer to Point 2 of our general rebuttal.

  1. On independent evaluation and dimensional consistency

We agree that improvements across individual quality dimensions are not necessarily monotonic. In particular, the independent Qwen evaluation shows decreasing LP and MC scores on ProofNet, despite an increase in aggregate score. However, the human evaluation in Table 4 provides complementary evidence. On miniF2F, human-evaluated overall scores increase from 49.46% to 66.60%, with improvements in LP, MC, and FQ. On ProofNet, human-evaluated overall scores increase from 15.71% to 40.96%, with improvements in LP and MC, although FQ decreases slightly.

These findings suggest that the aggregate improvements are not solely artifacts of the acceptance judges, while also demonstrating that the magnitude and dimensional consistency of the improvements depend on the evaluator. We have therefore revised our claims to emphasize aggregate quality improvement rather than coordinate-wise monotonicity, and acknowledge that the current evaluations do not establish improvement in true quality at every accepted transition.

Reviewer ywYe, thanks for your review. Could you take a look at the authors’ response and comment on which of your concerns have or have not been resolved?

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Rebuttal

We thank all reviewers for their constructive feedback. We have substantially revised the manuscript to clarify our contributions, strengthen the experimental comparisons, and address the limitations of the theoretical analysis.

  1. We have clarified the scope of our theoretical contributions. We distinguish monotonic improvement in the estimated objective from improvement in true formalization quality, and explicitly state the conditional nature of Theorems 2.4 and 2.5. We have also changed the title to "Multi-Dimensional Reference-Free Refinement for Autoformalization" to better reflect the methodological contribution.

  2. We have strengthened the experimental evaluation by adding DeepSeek-Prover Best-of-64 comparisons (Table 1) and reporting detailed generation and judge-call counts (Tables 7–8).

On miniF2F, our ensemble achieves a slightly higher overall score (90.27% vs. 89.06%) and judge-estimated perfect-formalization rate (49.18% vs. 47.54%), while using fewer generator calls (43.10 vs. 64.00) and judge calls (147.39 vs. 175.98). Both methods achieve 100% formal validity. These results provide evidence that structured refinement can be competitive with substantially larger independent-sampling budgets while requiring fewer model calls on this benchmark. On ProofNet, Best-of-64 achieves higher formal validity (89.25% vs. 77.96%) and overall score (56.37% vs. 52.45%). However, our ensemble achieves higher LP, MC, and FQ scores, together with a higher judge-estimated perfect-formalization rate (17.74% vs. 16.13%). This demonstrates that improvements in formal validity and aggregate score do not necessarily coincide with improvements across the soft dimensions.

We therefore do not claim uniform superiority over large-scale sampling. Instead, the results demonstrate benchmark-dependent trade-offs and suggest that structured refinement provides a complementary approach to independent sampling. We also acknowledge that model-call counts do not fully control for differences in model composition or token-level costs but the revised experiments still strengthen the comparison.

  1. We have clarified the empirical conclusions and limitations. In particular, we distinguish improvements in aggregate quality from improvements in individual dimensions, and clarify what can and cannot be concluded from independent LLM and human evaluations.

We address each reviewer's concerns below.

AuthorsOpenReview ↗

Update: Additional Best-of-64 Experiments and Revised Manuscript

We thank the reviewers and Area Chair for their time and consideration. Following our previous rebuttal, we have further updated the manuscript with additional Goedel-Prover-V2-8B Best-of-64 experiments, extending our compute-controlled comparisons beyond DeepSeek-Prover-V2-7B (Table 1; Tables 7–8 in Appendix K).

The new results provide additional evidence regarding the effectiveness and computational trade-offs of our approach:

  • On miniF2F, our ensemble achieves a higher overall score (90.27% vs. 84.25%) and judge-estimated perfect-formalization rate (49.18% vs. 30.74%) than Goedel-Prover Best-of-64, while using fewer generator calls (43.10 vs. 64.00) and judge calls (147.39 vs. 297.55).
  • On ProofNet, our ensemble also achieves a higher overall score (52.45% vs. 48.92%) and perfect-formalization rate (17.74% vs. 7.53%) than Goedel-Prover Best-of-64, with nearly identical judge-call counts (249.48 vs. 248.76).

These additional experiments strengthen our comparisons with large-scale independent sampling, while we acknowledge that model-call counts do not fully control for token-level costs or model composition.

We hope these new results and the revisions help address the reviewers' concerns, particularly regarding experimental attribution and computational budgets. As the discussion period approaches its deadline, we would greatly appreciate any further feedback from the reviewers, and would be happy to clarify any remaining questions.

Thank you again for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.