- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes Sage, a multi-agent framework for formalizing natural-language mathematical problems in Lean, with an iterative Draft–Verify–Refine pipeline. It introduces semantic verification during formalization to detect meaning drift, uses existential encodings to handle answer-seeking problems, and incorporates specialized agents for semantic and syntactic correction. Experiments on mathematical benchmarks show promising formalization performance, though the novelty and reliability of semantic verification require further clarification.
Critical strengths
S1: Clear presentation and logical organization.
The paper is generally well written, with a clear presentation and an easy-to-follow logical structure. The overall framework is organized in an intuitive manner, making the proposed approach and its motivation relatively easy to understand.
S2: Clear problem motivation.
The paper aims at a meaningful challenge in mathematical formalization: ensuring that generated Lean statements preserve the semantics of the original natural-language problems. This provides a clear motivation for incorporating semantic verification into the formalization pipeline.
Critical weaknesses
W1: Unclear incremental contributions.
The claimed contributions largely overlap with existing work, but the paper does not clearly establish what is novel. In particular, the distinction between Assertion- and Question-type problems is not new. For example, Loc-Decomp [1] explicitly distinguishes proof-type and solve-type problems and adopts different formalization strategies. Its use of a free variable for solve-type problems is mathematically equivalent to the existential encoding proposed here. Thus, the paper appears to overstate its primary theoretical contribution. Similarly, the decomposed pipeline, semantic feedback, and joint semantic-syntactic correction resemble existing approaches[2,3]. The authors should provide a more precise comparison with prior work and clearly identify the substantive methodological advances beyond existing methods.
W2: Insufficient validation of semantic fidelity.
Semantic fidelity is central to this work, yet its evaluation relies primarily on LLM-as-a-judge. Although the paper reports agreement with human annotators and accuracy on existing benchmarks, the human audit is limited to 45 examples, and the reported results do not establish reliable equivalence checking across the full evaluation set. Stronger validation is needed, including a larger expert audit and ground-truth-based equivalence checks (e.g., BEq) where reference formalizations are available. Without such evidence, the reliability of the proposed semantic correction mechanism and the reported fidelity improvements remains uncertain.
W3: Limited evidence for the generality of the reported improvements.
The evaluation primarily focuses on mathematical competition problems and uses a specific family of language models. Moreover, the strongest results rely on pass@4, which may obscure the reliability of a single generated formalization in practical applications. Additional evaluation across diverse mathematical domains and model families, together with a clearer analysis of pass@1 performance and inference costs, would strengthen the claims regarding generality and practical utility.
References
[1]Shi, Jiangze, et al. LoC-Decomp: LLM Autoformalization via Logical Concept Decomposition and Iterative Feedback Correction. International Conference on Learning Representations (ICLR 2026). (2026)
[2]Zhang, Jiyao, et al. "KELPS: A Framework for Verified Multi-Language Autoformalization via Semantic-Syntactic Alignment." arXiv preprint arXiv:2507.08665 (2025).
[3]Liu, Chengwu, et al. "Fimo: A challenge formal dataset for automated theorem proving." arXiv preprint arXiv:2309.04295 (2023).
Questions for authors
Q1:
Could the authors clarify the key technical differences between Sage and prior approaches such as Loc-Decomp, particularly regarding problem-type distinctions, existential encoding, and semantic verification? More importantly, which specific empirical results demonstrate that these differences provide substantial improvements over existing methods? The answer would materially affect the assessment of the paper’s novelty and contribution.
11 comments
Response Part 2/3: Multi-Layer Semantic Fidelity Validation & Empirical Gains Over [1–3] — W2 / Q1
W2 - Multi-Layer Validation of Semantic Fidelity
To validate semantic faithfulness beyond LLM judges, our revised manuscript combines four layers of verification: (1) machine-checked Lean 4 kernel equivalence, (2) blinded multi-expert human audits (225 total expert ratings), (3) expert-verified Olympiad geometry and combinatorics case studies (Appendix G.1), and (4) zero-shot calibration on 4,611 external human-labeled pairs (Appendix F).
W2.1 - Machine-Checked Lean 4 Kernel Equivalence on IMO-Formalized (§6.2; Appendix E.2, Table 19)
We evaluated bidirectional logical equivalence (BEq) across the chronological first 40 compiling, Goedel-SM-passing assertion problems in IMO-Formalized (IMO 1959–1988):
- Verbatim-Header Protocol: The human
Compfilesreference and Sage'spass@1formalization are loaded verbatim into isolated namespaces (BeqGoldandBeqGen) with source proofs replaced by:= by sorry. We queryDeepSeek 4.1 Flashwith iterative Lean 4 compiler feedback to provestatements_equivalent : S_Gold ↔ S_Sage. Equivalence is certified by the Lean 4 kernel iff: (i) the bridge proves the exact biconditional between the unaltered propositions, (ii) no helper definitions are rewritten, and (iii) recursiveLean.collectAxiomsinspection restricts dependencies strictly to{propext, Classical.choice, Quot.sound}—rejecting anysorryAx(preventing invocation of either unproved theorem).
Outcome Category (IMO-Formalized, IMO 1959–1988) | Count | Share | Verification Basis |
|---|---|---|---|
Certified Lean 4 Kernel Bridges (S_Gold ↔ S_Sage) | 32 | 80.0% | Sorry-free kernel proof under foundational axioms only |
Compfiles Human Reference Defects / Incomplete | 4 | 10.0% | Imo1972P1, Imo1972P5, Imo1969P2, Imo1987P1 |
| Representation / Packaging Differences | 3 | 7.5% | Same math, distinct Mathlib types (Imo1970P6, 1976P2, 1983P6) |
| Unilateral Sage Formalization Slip | 1 | 2.5% | Imo1967P3 (truncating Nat subtraction m - 1) |
- Forensic Breakdown of Non-Bridged Pairs: Because proving Olympiad assertions and from scratch is infeasible, every closed
S_Gold ↔ S_Sagebridge is genuinely non-vacuous. Inspecting the 8 non-bridged pairs shows only 1/40 (2.5%) is a unilateral Sage error (Imo1967P3), while 3/40 (7.5%) express the identical mathematical claim across distinct Mathlib representations (EuclideanSpace ℝ (Fin 2)vs.ℝ × ℝinImo1970P6;Multiset.Noduproots vs.Polynomial.discr P ≠ 0inImo1976P2; metric vs. side-length triangles inImo1983P6). - 4 Human Benchmark Defects Caught in
Compfiles(10.0%; full per-problem details inAppendix E.2&Appendix G.1): The remaining 4/40 non-bridges expose verified defects in the humanCompfilesreference itself (36/40 = 90.0% of Sage statements faithful or strictly cleaner than gold): vacuous empty-subset solutions inImo1972P1(Appendix G.1), weakened boundedness inImo1972P5, an unstated hypothesis inImo1969P2, and an opensorryinside aFintypeinstance inImo1987P1.
W2.2 - Blinded 3-Expert Study (225 Ratings), IMO Case Studies & 4,611 External Pairs
- Blinded 3-Expert Comparison (
Appendix E.1, Table 18): Adding to our submission's expert ratings (Appendix E), 3 independent Lean-proficient experts blindly compared Sage vs. Goedel-V2 on 30 random compilingIMO-Formalizedassertions ( new ratings;225total): majority vote favored Sage on 12/30 (40.0%, 7 unanimous), Tie on 17/30 (56.7%), and Goedel-V2 on 0/30 (0.0%) (1split)—a 12–0 sweep. - IMO Case Studies (
Appendix G.1) & External Calibration (4,611Pairs,Appendix F, Table 20): We provide expert-audited Lean 4 case studies of previously unformalized 2D/3D IMO geometry and combinatorics problems inAppendix G.1, whileSage-SMreaches 88.8% zero-shot accuracy (MCC = 0.74) onProofNetVerif() and 85.7% (MCC = 0.67) onConsistencyCheck().
Q1 - Concrete Empirical Gains Over [1, 2, 3]
Whereas [1]'s open α → Prop defs and [2]'s dual-sorry templates cannot be fed to standard single-sorry ATP provers without triggering the Trilemma, Sage emits prover-ready closed Prop theorems with 1.0%–2.7% leakage (48.7% Leak-Free Joint at pass@1, 69.3% at pass@4). Moreover, while prior frameworks excluded Olympiad Geometry and Combinatorics (FIMO [3]) or relied on closed 40-concept DSLs (KELPS [2]), Sage reaches 60.0% pass@1 (87.4% pass@4) on IMO-Unformalized (69.1% Geometry, 24.6% Combinatorics) — historical IMO problems so difficult they had never previously been formalized in Lean.
Response Part 3/3: `pass@1`, Token Economics, Open-Weight Scaling & Breadth — W3
W3. pass@1 Reporting, Token Economics, Open-Weight Scaling & Domain Breadth
W3.1 Why Upfront Formalization Compute & pass@4 Matter in Practice (Appendix D.2, Tables 15–16)
- Statement Formalization vs. Proof Search (Why Upfront Compute &
pass@4Are Essential in Practice): In any real-world workflow—whether a mathematician is formalizing a theorem for a paper/thesis or an automated pipeline is preparing a target for downstream ATP—proof search is orders of magnitude more expensive than statement formalization (days of human effort or prover rollouts per problem). If the formal statement is unfaithful or leaks a false witness (which occurs in 46.2%–70.7% of compiling leaks in monolithic and SFT baselines;Appendix C.4.1, Table 11), 100% of downstream proving compute is wasted on a false or vacuous specification. Consequently, in practice the priority is the exact opposite of a single-shot (pass@1) constraint: it is far better to spend a modest upfront token budget (18.5k–63.8ktokens) across a few trajectories () and let Sage's in-loop compiler +Sage-SMgate automatically select a verified statement (87.4%onIMO-Unformalized,73.3%onOmni-MATH) before launching proof search. pass@4Is Purely Additive & Single-AttemptSage (pass@1)Already BeatsGoedel-V2 (pass@4): Reportingpass@4removes zero information—every table (Tables 2, 4, 5) reportspass@1andpass@4side-by-side. Even under a strict single-attempt budget,Sage (pass@1)already outperforms 4-sampleGoedel-Formalizer-V2 (pass@4)across every benchmark with 0 SFT GPU hours—e.g., onIMO-Unformalized(),Sage pass@1reaches 60.0% Joint (87.4%atpass@4) vs. 19.4% for 4-sampleGoedel-V2 pass@4(12.6%atpass@1), with the same reversal onIMO-Formalized(61.4% vs. 50.7%) andOmni-MATHLeak-Free Joint (48.7% vs. 20.0%).- Hidden SFT Cost in Single-Shot Baselines: While
Goedel-V2's inference uses1.3ktokens (5.3katpass@4), that hides the massive upfront cost of full 32B SFT on 50k distilled traces—which baked 56.2%–58.1% answer leakage directly into its weights, with 70.7% of its compiling leaks asserting mathematically FALSE theorems (87/123false witnesses;Table 11).
W3.2 Why Competition Formalization Is Harder & Open-Weight Reproducibility
- Why Competition Problems Are Structurally Harder Than Standard Textbook Theorems: Unlike standard textbook theorems, competition benchmarks (
IMO-Unformalized: 69.1% Geometry, 24.6% Combinatorics;Omni-MATH: 98.7% open-ended questions) are substantially harder for autoformalization because they require: (i) converting open answer-finding/extremal/relational queries into closed propositions without leaking answers or triggering the Feasibility Trilemma, and (ii) synthesizing complex custom geometric configurations and combinatorial games from scratch that have no pre-packaged definitions in Mathlib. - Deliberate Commitment to Open-Weight Models (No Closed Commercial APIs): We intentionally built and evaluated Sage exclusively on open-weight models rather than proprietary black-box APIs to guarantee scientific reproducibility, weight-level stability, and zero API paywalls (
Appendix C.2). Holding all prompts and repair budgets (,pass@1) fixed, Sage scales monotonically across three open-weight generator generations—Qwen3-32B(30.3%)Qwen3.6-27B(77.3%)Qwen3.8-27B(85.7%) (Table 3,Appendix D.1)—while leveraging three distinct open-weight model families across the study:Qwen(generation),Gemma4-31B(externalGoedel-SMscoring and leakage auditing), andDeepSeek 4.1 Flash(Lean 4 kernel bridge synthesis).
We sincerely thank Reviewer cfFw again for the rigorous engagement and references, which substantially strengthened both the theoretical positioning and empirical validation of our manuscript. Upon publication, we will open-source the complete Sage framework, prompts, Lean 4 kernel equivalence suite, and datasets. We hope these formal comparisons, controlled ablations, blinded multi-expert audits, and machine-checked IMO kernel proofs fully address your concerns, and we warmly welcome any further questions during the discussion period.
Response Part 1/3: Architectural & Theoretical Distinctions — W1 / Q1
We thank Reviewer cfFw for the detailed feedback. LoC-Decomp [1] (
Shi et al., ICLR 2026), KELPS [2] (Zhang et al., ICML 2025), and FIMO [3] (Liu et al., 2023) are now discussed in §2, §3 and compared in Appendix B.6 (Table 8). Below, we address W1 & Q1 (Part 1), W2 & Concrete Gains (Part 2), and W3 (Part 3).W1 - Architectural & Theoretical Distinctions Over [1, 2, 3] (W1, Q1)
W1.1 - Verification Decomposition (
LoC-Decomp) vs. Generation Decomposition (Sage) (Appendix B.6)The term "decomposition" refers to two different phases: LoC-Decomp [1] generates the entire Lean specification in a single LLM call (
FormalTrans) using-->>declare_...markers and decomposes only post-hoc semantic checking (slicing markers for segmented NL back-translation), whereas Sage decomposes the generation process itself across four isolated stages (Distiller→Preprocessor→Formalizer→Formatter), resolving mathematical ontology and domain guards before any Lean code is written. Full architectural contrasts with [1, 2, 3] are summarized in the table below.W1.2 - Why [1]'s Free-Variable
def(α → Prop) Is Not Equivalent to Our Existentialtheorem(Prop)While prior works note the distinction between proof-type and solve-type problems (
miniF2F,FIMO[3],KELPS[2],LoC-Decomp[1]; cited in §3), [1]'s free-variabledefis not equivalent to our existential encoding for two fundamental reasons:∃): We encode answer-agnostic queries via existential quantification (∃s:α,P(s)) in a native Mathlibtheorem, whereas [1] wraps problem parameters in astructure MySystemand leaves(solution : α)as an unquantified free parameter indef problem_statement (solution : α) : Prop := ∀ sys : MySystem, ...(and placing(solution : α)in a Lean 4theorembinder means universal quantification ∀s, not ∃s).def(α → Prop) to an ATP-Readytheorem(Prop): While [1]'sdefis an elegant specification template, it has typeα → Proprather thanProp(theorem root : problem_statement := by sorryfails type-checking) and is not directly usable for ATP provers, which synthesize tactics strictly inside:= by sorryon an immutabletheorem. Turning [1]'sdefinto a closedtheoremdirectly hits our Feasibility Trilemma (§3, Table 1): (i) Answer injection (problem_statement c) guesses unverified answers (70.9%–72.6% leakage, 46.2%–48.5% false;Table 11); (ii) Placeholders (def sol := sorryor KELPS's\{x \mid \dots\} = sorry) leave twosorrys, breaking single-sorryATP harnesses; and (iii) Naive∃(∃ s, problem_statement s) collapses into vacuity on sets/yes-no queries via the Triviality Trap (use {x | P x}; rfl) or Polarity Cheat (Classical.em), unless α is hardened via Type-Theoretic Locks (Subtype, constructiveFinsetinType,PSum P (¬P),Appendix A.4).W1.3 - Relational Mismatch in [1]'s Showcased Example
In [1]'s showcased solve-type example (Fig. F-13), the informal statement asks to express a quantity in terms of parameters (a,b,c,R inside
sys : MySystem), yet their formalization declares(solution : ℝ)as a single scalar number outside∀ sys : MySystem(def problem_statement (solution : ℝ) : Prop := ∀ sys : MySystem, ...). While we do not know how [1] handles every problem, the example showcased in their paper is not semantically matched to the informal statement, as it asks for a single constant number independent of a,b,c,R rather than a function of the system's parameters. Sage models functional dependencies directly.W1.4 - Direct Lean 4 Evaluation (
Sage-SM) vs. Segmented Back-Translation (Q1)Segmented NL back-translation (
BackTrans[1]) is lossy: truncatingℕ-division(1 : ℕ) / 2 = 0back-translates benignly as "half of..." (hiding arithmetic collapse), while translating constraints in isolation loses cross-segment binder dependencies.Sage-SMevaluates the complete Lean 4 AST and types directly (Appendix B.2).\{x \mid \dots\} = sorry(2×sorry)def (s : α) : PropProp+ Type-Theoretic LocksSage-SMgate