acceptodds
Under review as a conference paper at ICLR 2027

Sage: Formalization with Semantic Correction

Abstract

While neural theorem provers have achieved impressive milestones in formal mathematics, they largely operate on the assumption that faithful formal statements are already provided. Translating informal natural language into a formal language is a critical data bottleneck plagued by an "illusion of rigor": standard type-checkers accept statements that compile but drop hypotheses, introduce vacuous truths, or subtly alter mathematical bounds. To resolve this, we introduce Sage (Semantic Agent-Guided Formalization Engine), an agentic framework that replaces monolithic translation with a four-stage decomposed generation pipeline coupled with a dual-signal semantic correction loop. By pairing formal compiler diagnostics with multi-dimensional semantic feedback, our correction loop enforces mathematical fidelity alongside syntactic validity. By explicitly accounting for the gap between open-ended queries and declarative formal targets, our pipeline prevents models from achieving high formalization rates by guessing unverified answers (exhibiting a 70.9% answer leakage rate in monolithic baselines). Consequently, Sage suppresses leakage to 2.7% while achieving 73.3% pass@4 joint compilation and semantic fidelity on the Omni-MATH benchmark without proofs (compared to 42.0% for a fine-tuned Goedel-Formalizer-V2-32B baseline). Finally, on IMO-Unformalized, a novel frontier of 175 unformalized International Mathematical Olympiad problems, Sage demonstrates effective zero-shot generalization with 87.4% pass@4 verified fidelity compared to just 19.4% for the baseline, winning over 79% of blind pairwise evaluations.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Autoformalization, Large Language Models, Formal Mathematics, Semantic Verification, Multi-Agent Systems, Lean 4

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cfFwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes Sage, a multi-agent framework for formalizing natural-language mathematical problems in Lean, with an iterative Draft–Verify–Refine pipeline. It introduces semantic verification during formalization to detect meaning drift, uses existential encodings to handle answer-seeking problems, and incorporates specialized agents for semantic and syntactic correction. Experiments on mathematical benchmarks show promising formalization performance, though the novelty and reliability of semantic verification require further clarification.

Critical strengths

S1: Clear presentation and logical organization.

The paper is generally well written, with a clear presentation and an easy-to-follow logical structure. The overall framework is organized in an intuitive manner, making the proposed approach and its motivation relatively easy to understand.

S2: Clear problem motivation.

The paper aims at a meaningful challenge in mathematical formalization: ensuring that generated Lean statements preserve the semantics of the original natural-language problems. This provides a clear motivation for incorporating semantic verification into the formalization pipeline.

Critical weaknesses

W1: Unclear incremental contributions.

The claimed contributions largely overlap with existing work, but the paper does not clearly establish what is novel. In particular, the distinction between Assertion- and Question-type problems is not new. For example, Loc-Decomp [1] explicitly distinguishes proof-type and solve-type problems and adopts different formalization strategies. Its use of a free variable for solve-type problems is mathematically equivalent to the existential encoding proposed here. Thus, the paper appears to overstate its primary theoretical contribution. Similarly, the decomposed pipeline, semantic feedback, and joint semantic-syntactic correction resemble existing approaches[2,3]. The authors should provide a more precise comparison with prior work and clearly identify the substantive methodological advances beyond existing methods.

W2: Insufficient validation of semantic fidelity.

Semantic fidelity is central to this work, yet its evaluation relies primarily on LLM-as-a-judge. Although the paper reports agreement with human annotators and accuracy on existing benchmarks, the human audit is limited to 45 examples, and the reported results do not establish reliable equivalence checking across the full evaluation set. Stronger validation is needed, including a larger expert audit and ground-truth-based equivalence checks (e.g., BEq) where reference formalizations are available. Without such evidence, the reliability of the proposed semantic correction mechanism and the reported fidelity improvements remains uncertain.

W3: Limited evidence for the generality of the reported improvements.

The evaluation primarily focuses on mathematical competition problems and uses a specific family of language models. Moreover, the strongest results rely on pass@4, which may obscure the reliability of a single generated formalization in practical applications. Additional evaluation across diverse mathematical domains and model families, together with a clearer analysis of pass@1 performance and inference costs, would strengthen the claims regarding generality and practical utility.

References

[1]Shi, Jiangze, et al. LoC-Decomp: LLM Autoformalization via Logical Concept Decomposition and Iterative Feedback Correction. International Conference on Learning Representations (ICLR 2026). (2026)

[2]Zhang, Jiyao, et al. "KELPS: A Framework for Verified Multi-Language Autoformalization via Semantic-Syntactic Alignment." arXiv preprint arXiv:2507.08665 (2025).

[3]Liu, Chengwu, et al. "Fimo: A challenge formal dataset for automated theorem proving." arXiv preprint arXiv:2309.04295 (2023).

Questions for authors

Q1:

Could the authors clarify the key technical differences between Sage and prior approaches such as Loc-Decomp, particularly regarding problem-type distinctions, existential encoding, and semantic verification? More importantly, which specific empirical results demonstrate that these differences provide substantial improvements over existing methods? The answer would materially affect the assessment of the paper’s novelty and contribution.

11 comments
AuthorsOpenReview ↗

Response Part 1/3: Architectural & Theoretical Distinctions — W1 / Q1

We thank Reviewer cfFw for the detailed feedback. LoC-Decomp [1] (Shi et al., ICLR 2026), KELPS [2] (Zhang et al., ICML 2025), and FIMO [3] (Liu et al., 2023) are now discussed in §2, §3 and compared in Appendix B.6 (Table 8). Below, we address W1 & Q1 (Part 1), W2 & Concrete Gains (Part 2), and W3 (Part 3).


W1 - Architectural & Theoretical Distinctions Over [1, 2, 3] (W1, Q1)

W1.1 - Verification Decomposition (LoC-Decomp) vs. Generation Decomposition (Sage) (Appendix B.6)

The term "decomposition" refers to two different phases: LoC-Decomp [1] generates the entire Lean specification in a single LLM call (FormalTrans) using -->>declare_... markers and decomposes only post-hoc semantic checking (slicing markers for segmented NL back-translation), whereas Sage decomposes the generation process itself across four isolated stages (Distiller →\to Preprocessor →\to Formalizer →\to Formatter), resolving mathematical ontology and domain guards before any Lean code is written. Full architectural contrasts with [1, 2, 3] are summarized in the table below.

W1.2 - Why [1]'s Free-Variable def (α → Prop) Is Not Equivalent to Our Existential theorem (Prop)

While prior works note the distinction between proof-type and solve-type problems (miniF2F, FIMO [3], KELPS [2], LoC-Decomp [1]; cited in §3), [1]'s free-variable def is not equivalent to our existential encoding for two fundamental reasons:

  1. Free Variable / Universal Binder vs. Existential (∃): We encode answer-agnostic queries via existential quantification (∃s:α,P(s)\exists s : \alpha, P(s)) in a native Mathlib theorem, whereas [1] wraps problem parameters in a structure MySystem and leaves (solution : α) as an unquantified free parameter in def problem_statement (solution : α) : Prop := ∀ sys : MySystem, ... (and placing (solution : α) in a Lean 4 theorem binder means universal quantification ∀s\forall s, not ∃s\exists s).
  2. Bridging from an Open def (α → Prop) to an ATP-Ready theorem (Prop): While [1]'s def is an elegant specification template, it has type α → Prop rather than Prop (theorem root : problem_statement := by sorry fails type-checking) and is not directly usable for ATP provers, which synthesize tactics strictly inside := by sorry on an immutable theorem. Turning [1]'s def into a closed theorem directly hits our Feasibility Trilemma (§3, Table 1): (i) Answer injection (problem_statement c) guesses unverified answers (70.9%–72.6% leakage, 46.2%–48.5% false; Table 11); (ii) Placeholders (def sol := sorry or KELPS's \{x \mid \dots\} = sorry) leave two sorrys, breaking single-sorry ATP harnesses; and (iii) Naive ∃ (∃ s, problem_statement s) collapses into vacuity on sets/yes-no queries via the Triviality Trap (use {x | P x}; rfl) or Polarity Cheat (Classical.em), unless α\alpha is hardened via Type-Theoretic Locks (Subtype, constructive Finset in Type, PSum P (¬P), Appendix A.4).

W1.3 - Relational Mismatch in [1]'s Showcased Example

In [1]'s showcased solve-type example (Fig. F-13), the informal statement asks to express a quantity in terms of parameters (a,b,c,Ra, b, c, R inside sys : MySystem), yet their formalization declares (solution : ℝ) as a single scalar number outside ∀ sys : MySystem (def problem_statement (solution : ℝ) : Prop := ∀ sys : MySystem, ...). While we do not know how [1] handles every problem, the example showcased in their paper is not semantically matched to the informal statement, as it asks for a single constant number independent of a,b,c,Ra, b, c, R rather than a function of the system's parameters. Sage models functional dependencies directly.

W1.4 - Direct Lean 4 Evaluation (Sage-SM) vs. Segmented Back-Translation (Q1)

Segmented NL back-translation (BackTrans [1]) is lossy: truncating ℕ-division (1 : ℕ) / 2 = 0 back-translates benignly as "half of..." (hiding arithmetic collapse), while translating constraints in isolation loses cross-segment binder dependencies. Sage-SM evaluates the complete Lean 4 AST and types directly (Appendix B.2).

DimensionFIMO [3]KELPS [2]LoC-Decomp [1]Sage (Ours)
DecompositionNone2-stage KE DSL + ANTLR4Single-call gen; post-hoc slice4-stage generation isolation
Solve-TypeAnswer in prompt\{x \mid \dots\} = sorry (2×2\timessorry)Open def (s : α) : PropClosed Prop + Type-Theoretic Locks
Sem. CheckManual reviewOffline post-hoc filterSegmented NL back-translationDirect Lean 4 Sage-SM gate
AuthorsOpenReview ↗

Response Part 2/3: Multi-Layer Semantic Fidelity Validation & Empirical Gains Over [1–3] — W2 / Q1

W2 - Multi-Layer Validation of Semantic Fidelity

To validate semantic faithfulness beyond LLM judges, our revised manuscript combines four layers of verification: (1) machine-checked Lean 4 kernel equivalence, (2) blinded multi-expert human audits (225 total expert ratings), (3) expert-verified Olympiad geometry and combinatorics case studies (Appendix G.1), and (4) zero-shot calibration on 4,611 external human-labeled pairs (Appendix F).

W2.1 - Machine-Checked Lean 4 Kernel Equivalence on IMO-Formalized (§6.2; Appendix E.2, Table 19)

We evaluated bidirectional logical equivalence (BEq) across the chronological first 40 compiling, Goedel-SM-passing assertion problems in IMO-Formalized (IMO 1959–1988):

  • Verbatim-Header Protocol: The human Compfiles reference and Sage's pass@1 formalization are loaded verbatim into isolated namespaces (BeqGold and BeqGen) with source proofs replaced by := by sorry. We query DeepSeek 4.1 Flash with iterative Lean 4 compiler feedback to prove statements_equivalent : S_Gold ↔ S_Sage. Equivalence is certified by the Lean 4 kernel iff: (i) the bridge proves the exact biconditional between the unaltered propositions, (ii) no helper definitions are rewritten, and (iii) recursive Lean.collectAxioms inspection restricts dependencies strictly to {propext, Classical.choice, Quot.sound}—rejecting any sorryAx (preventing invocation of either unproved theorem).
Outcome Category (IMO-Formalized, IMO 1959–1988)CountShareVerification Basis
Certified Lean 4 Kernel Bridges (S_Gold ↔ S_Sage)3280.0%Sorry-free kernel proof under foundational axioms only
Compfiles Human Reference Defects / Incomplete410.0%Imo1972P1, Imo1972P5, Imo1969P2, Imo1987P1
Representation / Packaging Differences37.5%Same math, distinct Mathlib types (Imo1970P6, 1976P2, 1983P6)
Unilateral Sage Formalization Slip12.5%Imo1967P3 (truncating Nat subtraction m - 1)
  • Forensic Breakdown of Non-Bridged Pairs: Because proving Olympiad assertions AA and BB from scratch is infeasible, every closed S_Gold ↔ S_Sage bridge is genuinely non-vacuous. Inspecting the 8 non-bridged pairs shows only 1/40 (2.5%) is a unilateral Sage error (Imo1967P3), while 3/40 (7.5%) express the identical mathematical claim across distinct Mathlib representations (EuclideanSpace ℝ (Fin 2) vs. ℝ × ℝ in Imo1970P6; Multiset.Nodup roots vs. Polynomial.discr P ≠ 0 in Imo1976P2; metric vs. side-length triangles in Imo1983P6).
  • 4 Human Benchmark Defects Caught in Compfiles (10.0%; full per-problem details in Appendix E.2 & Appendix G.1): The remaining 4/40 non-bridges expose verified defects in the human Compfiles reference itself (36/40 = 90.0% of Sage statements faithful or strictly cleaner than gold): vacuous empty-subset solutions in Imo1972P1 (Appendix G.1), weakened boundedness in Imo1972P5, an unstated f(0)≠0f(0) \neq 0 hypothesis in Imo1969P2, and an open sorry inside a Fintype instance in Imo1987P1.

W2.2 - Blinded 3-Expert Study (225 Ratings), IMO Case Studies & 4,611 External Pairs

  • Blinded 3-Expert Comparison (Appendix E.1, Table 18): Adding to our submission's 45×3=13545\times 3 = 135 expert ratings (Appendix E), 3 independent Lean-proficient experts blindly compared Sage vs. Goedel-V2 on 30 random compiling IMO-Formalized assertions (30×3=9030\times 3 = 90 new ratings; 225 total): majority vote favored Sage on 12/30 (40.0%, 7 unanimous), Tie on 17/30 (56.7%), and Goedel-V2 on 0/30 (0.0%) (1 split)—a 12–0 sweep.
  • IMO Case Studies (Appendix G.1) & External Calibration (4,611 Pairs, Appendix F, Table 20): We provide expert-audited Lean 4 case studies of previously unformalized 2D/3D IMO geometry and combinatorics problems in Appendix G.1, while Sage-SM reaches 88.8% zero-shot accuracy (MCC = 0.74) on ProofNetVerif (N=3,752N=3{,}752) and 85.7% (MCC = 0.67) on ConsistencyCheck (N=859N=859).

Q1 - Concrete Empirical Gains Over [1, 2, 3]

Whereas [1]'s open α → Prop defs and [2]'s dual-sorry templates cannot be fed to standard single-sorry ATP provers without triggering the Trilemma, Sage emits prover-ready closed Prop theorems with 1.0%–2.7% leakage (48.7% Leak-Free Joint at pass@1, 69.3% at pass@4). Moreover, while prior frameworks excluded Olympiad Geometry and Combinatorics (FIMO [3]) or relied on closed 40-concept DSLs (KELPS [2]), Sage reaches 60.0% pass@1 (87.4% pass@4) on IMO-Unformalized (69.1% Geometry, 24.6% Combinatorics) — historical IMO problems so difficult they had never previously been formalized in Lean.

AuthorsOpenReview ↗

Response Part 3/3: `pass@1`, Token Economics, Open-Weight Scaling & Breadth — W3

W3. pass@1 Reporting, Token Economics, Open-Weight Scaling & Domain Breadth

W3.1 Why Upfront Formalization Compute & pass@4 Matter in Practice (Appendix D.2, Tables 15–16)

  • Statement Formalization vs. Proof Search (Why Upfront Compute & pass@4 Are Essential in Practice): In any real-world workflow—whether a mathematician is formalizing a theorem for a paper/thesis or an automated pipeline is preparing a target for downstream ATP—proof search is orders of magnitude more expensive than statement formalization (days of human effort or 103–10510^3\text{–}10^5 prover rollouts per problem). If the formal statement is unfaithful or leaks a false witness (which occurs in 46.2%–70.7% of compiling leaks in monolithic and SFT baselines; Appendix C.4.1, Table 11), 100% of downstream proving compute is wasted on a false or vacuous specification. Consequently, in practice the priority is the exact opposite of a single-shot (pass@1) constraint: it is far better to spend a modest upfront token budget (18.5k–63.8k tokens) across a few trajectories (k=4k=4) and let Sage's in-loop compiler + Sage-SM gate automatically select a verified statement (87.4% on IMO-Unformalized, 73.3% on Omni-MATH) before launching proof search.
  • pass@4 Is Purely Additive & Single-Attempt Sage (pass@1) Already Beats Goedel-V2 (pass@4): Reporting pass@4 removes zero information—every table (Tables 2, 4, 5) reports pass@1 and pass@4 side-by-side. Even under a strict single-attempt budget, Sage (pass@1) already outperforms 4-sample Goedel-Formalizer-V2 (pass@4) across every benchmark with 0 SFT GPU hours—e.g., on IMO-Unformalized (N=175N=175), Sage pass@1 reaches 60.0% Joint (87.4% at pass@4) vs. 19.4% for 4-sample Goedel-V2 pass@4 (12.6% at pass@1), with the same reversal on IMO-Formalized (61.4% vs. 50.7%) and Omni-MATH Leak-Free Joint (48.7% vs. 20.0%).
  • Hidden SFT Cost in Single-Shot Baselines: While Goedel-V2's inference uses 1.3k tokens (5.3k at pass@4), that hides the massive upfront cost of full 32B SFT on 50k distilled traces—which baked 56.2%–58.1% answer leakage directly into its weights, with 70.7% of its compiling leaks asserting mathematically FALSE theorems (87/123 false witnesses; Table 11).

W3.2 Why Competition Formalization Is Harder & Open-Weight Reproducibility

  • Why Competition Problems Are Structurally Harder Than Standard Textbook Theorems: Unlike standard textbook theorems, competition benchmarks (IMO-Unformalized: 69.1% Geometry, 24.6% Combinatorics; Omni-MATH: 98.7% open-ended questions) are substantially harder for autoformalization because they require: (i) converting open answer-finding/extremal/relational queries into closed propositions without leaking answers or triggering the Feasibility Trilemma, and (ii) synthesizing complex custom geometric configurations and combinatorial games from scratch that have no pre-packaged definitions in Mathlib.
  • Deliberate Commitment to Open-Weight Models (No Closed Commercial APIs): We intentionally built and evaluated Sage exclusively on open-weight models rather than proprietary black-box APIs to guarantee scientific reproducibility, weight-level stability, and zero API paywalls (Appendix C.2). Holding all prompts and repair budgets (T=5T=5, pass@1) fixed, Sage scales monotonically across three open-weight generator generations—Qwen3-32B (30.3%) →\to Qwen3.6-27B (77.3%) →\to Qwen3.8-27B (85.7%) (Table 3, Appendix D.1)—while leveraging three distinct open-weight model families across the study: Qwen (generation), Gemma4-31B (external Goedel-SM scoring and leakage auditing), and DeepSeek 4.1 Flash (Lean 4 kernel bridge synthesis).

We sincerely thank Reviewer cfFw again for the rigorous engagement and references, which substantially strengthened both the theoretical positioning and empirical validation of our manuscript. Upon publication, we will open-source the complete Sage framework, prompts, Lean 4 kernel equivalence suite, and datasets. We hope these formal comparisons, controlled ablations, blinded multi-expert audits, and machine-checked IMO kernel proofs fully address your concerns, and we warmly welcome any further questions during the discussion period.

Reviewer_t5ySOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Sage formalizes mathematical statements in Lean through four generation stages and a feedback loop using compiler and semantic checks. Experiments on Omni-MATH and two IMO sets distinguish translation with and without access to informal solutions.

Critical strengths

The answer-leakage analysis catches a failure that compilation alone misses: translating a question is different from inserting a guessed answer. The same-backbone ablations are helpful, as are the external alignment benchmarks and the three-annotator audit. Separating answer-aware and answer-agnostic settings is important here. It prevents a high formalization rate from hiding a change in the task itself. The paper makes this issue visible rather than relying only on whether Lean accepts a statement.

Critical weaknesses

My main concern is how much to trust the reported fidelity gains. Success still depends on automated semantic judgments. The expert audit finds 71.1% agreement on 45 deliberately selected drafts, rather than comparing final Sage and baseline outputs on a random sample. Appendix C.5 also reports systematic bias in the post-hoc judge. The external validation supports the semantic gate, but does not estimate the final error difference between systems. A judge can be useful overall and still favor one system’s characteristic outputs.

Equal round limits do not establish strict compute parity. Sage has four generation stages, semantic-rating calls, and compilation of proposed repairs. Without actual call and token counts, I cannot tell how much of the gain comes from additional inference.

Questions for authors

Could you provide a small blinded human comparison of randomly sampled final outputs? What are the actual model calls and generated tokens per method, and does Sage retain its advantage at a matched token budget? The human sample should include both systems and hide their identities, rather than selecting only known failure types.

2 comments

We thank Reviewer t5yS for the constructive review and for highlighting the value of separating answer-aware and answer-agnostic formalization. Below, we address both concerns via a blinded random-sample human study (§6.2; Appendix E.1, Table 18), judge rationale & Lean 4 kernel equivalence (§6.2; Appendix E.2, Table 19), and token accounting (§5, §7; Appendix D.2, Tables 15–16).


1. Blinded Random-Sample Human Study, Judge Rationale & Kernel BEq (W1, Q1a)

1a. Blinded 3-Expert Study on Randomly Sampled Final Outputs (Appendix E.1, Table 18; 30×3=9030 \times 3 = 90 ratings):
To compare randomly sampled final outputs across both systems with hidden identities rather than failure-selected drafts, 3 independent Lean-proficient annotators blindly evaluated Sage vs. Goedel-V2 on 30 random assertion-type IMO-Formalized problems where both compile (pass@1; headers/comments stripped, targets renamed theorem statement, randomized A/B order):

  • Outcome (κ=0.72\kappa = 0.72): Majority vote preferred Sage on 12/30 (40.0%, 7 unanimous), Tie on 17/30 (56.7%), and Goedel-V2 on 0/30 (0.0%) (1 split)—a 12–0 sweep on decisive cases, confirming genuine semantic error reduction on final outputs.

1b. Why We Used External Goedel-SM (and Why Its Bias Works Against Sage):
We avoided creating our own evaluation metric alongside our method, as co-designing the post-hoc judge with Sage would risk self-preference bias or tailoring a metric that favors us. We therefore adopted the external community standard Goedel-SM (running Goedel-Prover-V2's exact prompt on Gemma4-31B, 4/4 consensus), even though it is not optimal for open-ended questions. Crucially (Appendix C.5), Goedel-SM's few-shot prompt rewards explicit answer injection (f 3 = 0, characteristic of Goedel-V2 and Monolithic) and penalizes open existential encodings (76.8% of Sage rejections; Table 13)—working against Sage and making our Goedel-SM scores a conservative lower bound.

1c. Judge-Free Lean 4 Kernel Equivalence (Appendix E.2, Table 19):
On the first 40 compiling, Goedel-SM-passing assertions in IMO-Formalized (IMO 1959–1988), we froze Compfiles gold and Sage pass@1 verbatim in isolated namespaces with := by sorry and queried DeepSeek 4.1 Flash with compiler feedback to prove S_Gold ↔ S_Sage under Lean.collectAxioms ⊆ {propext, Classical.choice, Quot.sound} (zero sorryAx). The Lean 4 kernel certifies 32 / 40 (80.0%) bridges; 4 / 40 (10.0%) expose Compfiles reference defects (Appendix E.2, G.1), 3 / 40 (7.5%) reflect Mathlib representation differences, and 1 / 40 (2.5%) is a unilateral Sage slip (Imo1967P3)—so 36 / 40 (90.0%) of Sage's statements are faithful or strictly cleaner than gold.


2. Token Accounting & Advantage Under Lower Token Budgets (W2, Q1b)

You are completely right: writing "matched compute" in our submission was our mistake—we meant matched loop count (T=5T=5 on Omni-MATH, T=10T=10 on IMO), not equal token compute. We corrected this in §2, §5, and Appendix C.3, and ran a full generation token accounting experiment on Omni-MATH (N=300N=300) in Appendix D.2 (Tables 15–16).

Directly answering Q1b, our experiments in Appendix D.2 confirm that Sage's gains come from structural stage isolation rather than extra tokens, and hold even when baselines use more tokens:

  1. Zero-Repair Stage Isolation Beats Multi-Round & Multi-Sample Monolithic Using Fewer Tokens: Without any repair loop, Decomposed (T=1, 18.5k tok) uses half the tokens of 5-round Monolithic + Dual (37.3k tok) and fewer tokens than 4-sample Monolithic pass@4 (25.3k tok), yet beats both on Leak-Free Joint (29.7% vs. 18.0% and 16.7%). Why? Single-pass monolithic generation falls into the Discovery Trap (guessing answers on >70% of open questions, even with an Anti-Solver prompt in Appendix D.3, Table 17), and once a witness is hardcoded in round 1, extra repair rounds only polish syntax around a leaked (46%–49% false; Table 11) statement.
  2. At 32% Fewer Tokens, Decomposed + Syntax More Than Doubles Monolithic + Dual: Adding compiler repair (Decomposed + Syntax, 25.5k vs. 37.3k tok) more than doubles Leak-Free Joint (42.0% vs. 18.0%), and Sage (63.8k tok) reaches 48.7% pass@1 / 69.3% pass@4.
  3. Training-Free Sage pass@1 Outperforms 4-Sample SFT Goedel-V2 pass@4: While Goedel-V2 amortizes compute into 32B SFT on 50k traces, SFT baked answer guessing into its weights (70.7% false witnesses; Table 11). Thus, single-attempt Sage pass@1 beats 4-sample Goedel-V2 pass@4 on Omni-MATH Leak-Free Joint (48.7% vs. 20.0%) and IMO-Unformalized (60.0% vs. 19.4%).

We sincerely thank Reviewer t5yS again for the thoughtful comments, which directly strengthened our paper.

Dear Reviewer t5yS,

As a brief update on Section 1c of our response above: the rate-limited beq audit run on IMO-Formalized has now finished across 100% of all compiling, judge-passing IMO assertions (N=91/91N = 91 / 91, zero subsampling):

  • 70 / 91 (76.9%) close with sorry-free Lean 4 kernel equivalence proofs (S_gold ↔ S_Sage, excluding 1 vacuous True ↔ True proof caught by our screen),
  • 8 / 91 (8.8%) expose provable defects or sorrys in the human Compfiles ground truth itself vs. only 1 / 91 (1.1%) unilateral Sage slip (11 representation/packaging gaps)—yielding 85.7% (78 / 91) contest faithfulness.

Together with our blinded 3-expert random-sample comparison (12–0 decisive preference for Sage, 17 ties, κ=0.72\kappa = 0.72) and matched/unfavorable token-budget tables, we hope this fully resolves both of your questions. Please let us know if any further clarification would be helpful before the discussion period concludes!

Reviewer_VYKKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Sage combines a multi-stage natural-language-to-Lean pipeline with compiler feedback and an LLM-based semantic correction loop. The paper distinguishes answer-aware from answer-agnostic formalization and reports large gains in compilation-plus-fidelity scores, along with reduced answer leakage. The main comparison is against Goedel-Formalizer-V2 on Omni-MATH and two IMO sets.

Critical strengths
  • The distinction between translating a mathematical question and supplying its answer is important, and the leakage measurements are informative.
  • The same-backbone ablation and use of a separate post-hoc judge are helpful steps toward a controlled evaluation.
Critical weaknesses
  • What does routing confidence measure? Its correlations with reference PRM scores are modest. The score may also favor longer, non-terminal steps because it uses within-step variation and head-token statistics. Random-step revision is not a strong enough control to rule this out. Length- and position-matched baselines are missing.
  • Limited validation as a PRM. RoCo-PRM is mainly tested as a step selector in the authors' revision pipeline. Direct step-error detection or best-of-N evaluation would better support the process-supervision claim. Some reported values also need clarification, particularly Table 15's OLMoE “before” scores. Also, Vanilla is described as greedy decoding, yet it has nonzero variance across seeds in Table 1.
Questions for authors
  • The headline baseline comparison is not compute-matched: Sage uses several stages and correction rounds, while Goedel-Formalizer is evaluated largely in a single pass. The experiments also do not fully isolate the effects of decomposition and semantic feedback. For example, does the Monolithic prompt include the same "don't compute the answer" instruction as the Distiller? If not, the drop in leakage may just come from the prompt.
  • The main semantic-fidelity scores rely on LLM judgments, especially on IMO-Unformalized where gold formalizations are unavailable. An independent audit of a sample would make the large gains more convincing.
1 comment

We thank Reviewer VYKK for the supportive and constructive review
(Note: Critical Weaknesses appears to come from another review on RoCo-PRM / OLMoE; below we address all your Sage questions.)

1. Compute Budgets, Decomposition vs. Feedback & "Don't Compute" Prompt Control (Q1)

  • Matched Iteration Budgets & Token Accounting (§2, §5; Appendix D.2, Tables 15–16): Single-pass token parity across paradigms does not hold because Goedel-V2 amortizes compute into offline 32B full SFT (no repair), whereas Sage is training-free under matched iteration caps (T=5T=5). Full token accounting and draft-vs.-repair phase breakdowns are in Appendix D.2 (Tables 15–16).
  • Insight — Decomposition Matters More Than Raw Tokens (Appendix D.2): Adding iterative repair to Monolithic (Monolithic + Dual) cannot fix answer leakage: once the model guesses an answer in Round 1, extra rounds merely polish Lean syntax around the leaked statement. Thus, zero-repair Decomposed (T=1T=1) already beats 5-round Monolithic + Dual (T=5T=5) on Leak-Free Joint (29.7% vs. 18.0%) at fewer than half the tokens (18.5k vs. 37.3k), while single-attempt Sage (pass@1) outperforms 4-sample Goedel-V2 (pass@4) across every benchmark (Appendix D.2, Table 15).
  • Controlled "Don't Compute" Prompt Study (Appendix D.3, Table 17): Adding the Distiller's "do not compute/solve or inject witnesses" rules directly into Monolithic (T=1T=1) (Appendix D.3, Table 17) shows that when comprehension and Lean coding occur in one LLM call, the model still solves internally to form a concrete Lean expression—confirming stage isolation, not prompt phrasing, prevents answer guessing:
Pipeline (Omni-MATH Answer-Agnostic, pass@1)Leak-Free JointAnswer Leakage ↓
Monolithic (T=1T=1, standard prompt)11.3%72.6%
Monolithic + Anti-Solver (T=1T=1, "don't compute")14.3%63.0%
Decomposed (T=1T=1, stage isolation, no repair)29.7%1.0%
Sage (Decomposed + Dual, T=5T=5)48.7%2.7%

2. Independent Fidelity Validation & IMO-Unformalized Audit (Q2)

  • Same-Source IMO Validation & Case Studies (Appendix G.1): Because IMO-Unformalized lacks human references (precluding BEq) and is mostly Olympiad Geometry and Combinatorics—where manual formal review is very tedious—we note that: (i) it comes from the exact same IMO source and uses the exact same evaluation protocol (Compile + Gemma4-31B consensus) as IMO-Formalized, audited directly below; and (ii) Appendix G.1 provides two expert-verified Lean 4 case studies (imo_1966_p6 and imo_1968_p4).
  • Blinded 3-Expert Study (Appendix E.1, Table 18): Complementing our rubric audit (Appendix E) and external calibration (Appendix F, Table 20), 3 independent Lean experts blindly compared Sage vs. Goedel-V2 on 30 random compiling IMO-Formalized assertions: Sage 12/30, Tie (both faithful) 17/30, Goedel-V2 0/30 (1 split)—a 12–0 sweep (κ=0.72\kappa=0.72).
  • Lean 4 Kernel Equivalence (Appendix E.2, Table 19): On the first 40 compiling, Goedel-SM-passing IMO-Formalized assertions, the Lean 4 kernel certifies 32 / 40 (80.0%) bidirectional proofs (S_Gold ↔ S_Sage, zero sorryAx); of the remaining 8 (Appendix E.2), 4 stem from Compfiles reference defects, 3 from benign Mathlib choices, and only 1 (2.5%) from a Sage slip.

3. Ground-Truth Accuracy of Leaked Answers on Omni-MATH (Q3)

As detailed in §3, §6.3, and §7, an autoformalizer must act as a translator, not an unverified solver: in Answer-Agnostic mode, injecting a guessed answer violates the contract even when right, as (1) Lean 4 gives zero compiler signal on wrong guesses (x = 42 vs. x = 43 under sorry), (2) supplying the answer turns discovery into verification, and (3) right guesses reflect pretraining recall. With that established, we audited how often leaked answers in compiling Lean 4 statements match the ground-truth Omni-MATH solution (Appendix C.4.1, Table 11):

Method (Omni-MATH, pass@1)Leaked Compiling StatementsMatches SolutionFalse Witness (%)
Monolithic1588573 (46.2%)
Monolithic + Dual20210498 (48.5%)
Goedel-Formalizer-V21233687 (70.7%)
Sage (Ours)835 (62.5%)

Baselines frequently turn open questions into mathematically false theorems (46.2%–70.7% false witnesses), whereas Sage reduces leaked compiling statements from 123–202 down to just 8 / 300 (2.7%).

We hope these token-budget analyses, anti-solver prompt study, blinded expert and Lean 4 kernel audits, and witness-accuracy results resolve your concerns on compute parity, prompt engineering, semantic fidelity, and answer leakage.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions: New Empirical Validation, Anti-Solver Ablation, and Prior-Work Positioning

We thank the Area Chair and Reviewers (VYKK, t5yS, cfFw) for their thoughtful feedback on (1) validating semantic fidelity beyond LLM judges, (2) isolating stage decomposition from prompt engineering and token budgets, and (3) situating our theoretical and architectural contributions relative to prior autoformalization frameworks. Below is an overview of our new empirical studies and manuscript revisions (marked in dark blue in the PDF):


1. Validating Semantic Fidelity Beyond LLM Judges

  • Blinded 3-Expert Random-Sample Comparison (§6.2; Appendix E.1, Table 18): Complementing our rubric audit (Appendix E) and external calibration on 4,611 human-labeled pairs (Appendix F, Table 20), 3 independent Lean-proficient experts blindly compared final Sage vs. Goedel-Formalizer-V2 outputs on 30 randomly sampled compiling IMO-Formalized assertions (headers stripped, randomized A/B order):
    • Majority Vote: Sage Preferred: 12/30 (7 unanimous) | Tie (both faithful): 17/30 | Goedel-V2 Preferred: 0/30 (1 split) — a 12–0 clean sweep on decisive cases.
  • Lean 4 Kernel Equivalence Audit (§6.2; Appendix E.2, Table 19): On the first 40 compiling, Goedel-SM-passing IMO-Formalized assertions, the Lean 4 kernel certifies 32 / 40 (80.0%) exact bidirectional equivalence proofs (S_Gold ↔ S_Sage, zero sorryAx). Forensic inspection of the remaining 8 pairs (Appendix E.2) shows that 4 of the 8 pairs stem from defects in the human Compfiles reference itself (we listed them in the appendix), 3 reflect benign Mathlib representation choices, and only 1 (2.5%) is a unilateral Sage slip.
  • Expert-Verified IMO Case Studies (Appendix G.1): Full Lean 4 formalizations and audits of previously unformalized 2D and 3D Olympiad geometry problems (imo_1966_p6, imo_1968_p4), plus imo_1972_p1 where Sage catches an omitted non-emptiness precondition in Compfiles.

2. Isolating Stage Decomposition from Prompting & Compute Budgets

  • Prompting Alone Cannot Replace Stage Isolation (§6.3; Appendix D.3, Table 17): Adding explicit "do not compute/solve or inject witnesses" rules into a single-call Monolithic prompt across all benchmarks barely reduces answer leakage on Omni-MATH (72.6% →\to 63.0%) while hurting overall formalization quality. When problem comprehension and Lean coding share one context, the LLM still solves internally to form a concrete Lean expression—whereas zero-repair stage isolation (Decomposed, T=1T=1) collapses leakage to 1.0% (29.7% vs. 14.3% Leak-Free Joint).
  • Why Leaked Answers Are Dangerous (§6.3; Appendix C.4.1, Table 11): Auditing compiling Question-type leaks against ground-truth Omni-MATH solutions shows that 46.2%–70.7% of baseline leaks silently inject mathematically false theorems that compile cleanly under sorry. Sage reduces leaked compiling statements from 123–202 down to 8 / 300 (2.7%).
  • Decomposition Beats Raw Tokens & Offline SFT (§5, §7; Appendix D.2, Tables 15–16): Iterative repair on Monolithic (Monolithic + Dual, T=5T=5) merely polishes Lean syntax around already-leaked answers. Thus, zero-repair Decomposed (T=1T=1) beats 5-round Monolithic + Dual (29.7% vs. 18.0% Leak-Free Joint) at fewer than half the tokens (18.5k vs. 37.3k). Moreover, with zero SFT GPU hours, single-attempt Sage (pass@1) outperforms 4-sample Goedel-V2 (pass@4, 32B SFT) across every benchmark—reaching 60.0% vs. 19.4% on IMO-Unformalized (Table 15).

3. Theoretical Positioning & Open Science

  • Comparison with Prior Frameworks (§2, §3; Appendix B.6, Table 8): We expanded our comparison with LoC-Decomp (Shi et al., ICLR 2026), KELPS (Zhang et al., ICML 2025), and FIMO (Liu et al., 2023), clarifying three distinctions:
    1. Verification vs. Generation Decomposition: LoC-Decomp generates Lean monolithically and decomposes post-hoc verification; Sage decomposes generation itself across isolated stages.
    2. Closed theorem vs. Open def: LoC-Decomp's free-variable def (solution : α) : Prop is not equivalent to our closed existential theorem : Prop—it cannot be proved directly by ATP provers without conversion (Feasibility Trilemma, solved via Type-Theoretic Locks), and binding a scalar solution outside ∀ sys struggles on relational queries (Appendix B.6).
    3. Direct Lean Evaluation: Sage-SM audits compiled Lean 4 ASTs directly, avoiding lossy NL back-translation.
  • Open Science: Upon publication, we will release all Sage code, prompts, kernel-equivalence scripts, and datasets on GitHub, and upstream verified IMO-Unformalized statements to Compfiles.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.