acceptodds
Under review as a conference paper at ICLR 2027

SemantEq: Faithful Bidirectional Equivalence of Formal Statements

Abstract

Statement autoformalization translates mathematics from natural language into a proof assistant such as Lean 4, and models now do it at scale. As provers get stronger, the burden of correctness shifts from the proof to the statement, that is, to whether the formal statement faithfully translates its source. Given a trusted reference formalization, the natural test is logical equivalence, and it proves too much. P ↔ Q holds trivially for any two statements P and Q that the library proves, because a prover can prove each separately and pass through P ↔ True ↔ Q. The stronger the prover, the more unrelated pairs pass. Earlier methods restrict the prover and check that each proof names the other statement. We introduce SemantEq, a more principled relation that instead requires each statement to turn into the other only through a chain of allowed steps, such as logical rearrangement and rewriting with the library's own equations. Our programs judge a proof by the proof term Lean checks, not by its tactic script. SemantEq-Eval, a search without a model, reaches 100% precision and 65.6% recall on the ProofNetVerif test split under our adjudicated labels, against 95.8% and 61.8% for BEq+, and an 8.9-fold increase in its CPU time adds at most three pairs. A language model finds more proofs, but under two of four prompts it proves both directions of about two in five of the non-equivalent pairs. SemantEq-ProofAudit audits each model-written proof under a policy derived from the relation and accepts none of the 104 non-equivalent pairs, and it raises recall on 407 equivalent pairs from 65.4% to 81.1% under the best prompt. Separating the generation of equivalence proofs from their acceptance lets a stronger prover raise recall without changing what an acceptance means. We release code and data at: https://anonymous.4open.science/r/SemantEq-2FBF/

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Autoformalization, Formalization Evaluation, Automated Theorem Proving, Lean4, Semantic Equivalence

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_Nv44OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on evaluating statement autoformalization: determining whether a generated formal statement is semantically equal to a reference formal statement. It introduces a principled equivalence relation, SemantEq: whether two formal statements can prove each other by a fixed set of steps. Based on this relation, it implements a Lean metaprogram, SemantEq-Eval for determining such relatin in Lean 4. It also proposes SemantEq-ProofAudit, an auditing policy for externally generated proofs motivated by the relation.

Critical strengths

This paper brings a novel, reasonable, and intelligent idea to the community: for reference-based autoformalization evaluation, restricting the finer-grained steps in elaborated proof terms instead of coarse-grained tactics. They implement this idea in Lean 4 metaprogramming and conduct experiments to validate its effectiveness.

Critical weaknesses
  1. Absence of implication from ProofAudit to Definition 2. This paper lacks a formal derivation or intuitive explanation that once a proof passes SemantEq-ProofAudit, there exists chains from P to Q and from Q to P.
  2. Insufficient literature review and comparison. LeanScorer[1] and ASSESS[2] are a closely related work but is not discussed.

[1] Mathesis: Towards Formal Theorem Proving from Natural Languages. ICLR 2026. https://openreview.net/forum?id=CJdX82odge [2] ASSESS: A Semantic and Structural Evaluation Framework for Statement Similarity. ICLR 2026. https://openreview.net/forum?id=avwNGWtiHF

Questions for authors
  1. Could the authors please explain the connection between the implementation of ProofAudit and the original definition?
  2. Could LeanScorer[1] and ASSESS[2] be included for comparison?

[1] Mathesis: Towards Formal Theorem Proving from Natural Languages. ICLR 2026. https://openreview.net/forum?id=CJdX82odge [2] ASSESS: A Semantic and Structural Evaluation Framework for Statement Similarity. ICLR 2026. https://openreview.net/forum?id=avwNGWtiHF

Reviewer_qbc3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SemantEq, a framework in Lean 4 designed to verify whether two formal mathematical statements are semantically equivalent. Standard logical equivalence checks fail for evaluating statement autoformalization because a strong prover can independently prove two unrelated true statements from the math library, ignoring whether one actually translates the other. To solve this, SemantEq defines equivalence strictly through four permitted operations: logical rearrangement, bridge rewriting with library equations, change of variables, and structural weakening. The framework provides SemantEq-Eval, a deterministic proof search tool, and SemantEq-ProofAudit, an auditor that verifies proofs generated by external language models against these strict rules, ensuring the antecedent is genuinely used to derive the target.

Critical strengths

Exposes Systemic Vulnerabilities in Prior Work: The paper empirically demonstrates that the previous state-of-the-art method (BEq+) is scientifically flawed, showing it falsely accepts 54 out of 100 adversarial non-equivalent pairs because AI provers can exploit its constraints.

Architectural Decoupling: By separating proof generation from proof acceptance, SemantEq-ProofAudit successfully harnesses the creative reasoning power of language models to find complex bridges (boosting recall to 81.1%) while relying on a rigid, deterministic Lean kernel auditor to maintain 100% precision.

Strict Proof-Term Auditing: The system moves beyond checking tactic scripts or compilation success. By auditing the fully elaborated proof term, it guarantees the antecedent is actively doing the mathematical work in every branch, effectively blocking models from bypassing the statement using independent library facts.

Critical weaknesses

Lack of Downstream Application (The "Pure Tool" Limitation): The paper heavily motivates SemantEq as a necessary defense against reward hacking during the reinforcement learning (RL) training of autoformalization models. However, it only evaluates the tool on static datasets. Without deploying it in an actual RL training loop, the paper lacks the definitive scientific insight of proving whether this tool actually survives a live agent actively trying to exploit it.

Semantic Drift via "Heavy Machinery" Bridges (This is just my own point of view, very subjective, should be viewed as a question, not a real weakness): The framework relies on the math library's existing theorems to bridge statements. In mathematics, applying highly sophisticated theorems (e.g., using projective geometry to trivialize a middle-school geometry problem) can mathematically link two statements while completely obliterating the original educational or semantic intent of the problem. Relying purely on structural library bridges risks accepting these extreme semantic shifts.

Reliance on Trusted Formal References: The system does not compare the candidate against the informal natural language source directly. It requires a trusted, human-written formal reference statement, limiting its utility on completely novel problem sets.

Questions for authors

While the static adversarial tests in Section 4 are compelling, have you tested SemantEq as a reward signal in a live reinforcement learning loop? How do we know an active RL agent won't find a novel way to reward-hack the proof-term auditor (i think this should be very unlikely, unless lean kernal has bugs)? Do you think that using SemantEq in the RL loop will make the model behave better than very rigid "equivalence" definition like exact match with just variable substitution? Will the model actually learn to "hallucinate" since the model can bridge two things that connected by very heavy theorems (but the model might not understand), also see the following question.

How does the framework handle cases where a model uses "heavy machinery" library theorems to bridge two statements? If a highly advanced theorem mathematically connects statement A and statement B, but completely bypasses the original semantic intent of the natural language problem, does SemantEq consider this a valid translation?

The deterministic search saturates quickly. Are there plans to expand the ruleset for permitted transformations, or do you view the LLM-auditor pairing as the permanent solution to the recall ceiling?

Reviewer_G9nwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper points out an issue in evaluating formal statements with Lean proofs: logical equivalence does not imply that two formal statements express the same mathematical claim. Thus, the authors propose SemantEq, an equivalence relation defined through chains of four allowed transformations: logical rearrangement, bridge rewriting, change of variables, and structural weakening. They use these rules to guide proof search and auditing with SemantEq-Eval, and to check whether supplied proofs establish equivalence through acceptable transformations with SemantEq-ProofAudit. The results show high precision in accepting equivalent statement pairs, while combining an LLM prover with SemantEq-ProofAudit allows wider coverage of equivalent pairs while maintaining high precision.

Critical strengths
  1. The paper addresses a meaningful problem: P <--> True <--> Q allows unrelated provable statements to pass logical equivalence checks. The proposed chain of allowed transformations is a reasonable approach to this problem.
  2. SemantEq-Eval performs proof search without LLMs and achieves fair coverage with a reported 100% precision. The evaluation under the adjudicated labels is solid, and the baseline is chosen well.
  3. The audit ablation is solid and insightful, showing how each audit contributes to rejecting non-equivalent pairs.
  4. Separating proof generation from auditing is useful. SemantEq-ProofAudit does not restrict the tactics used to generate proofs, and combining it with an LLM prover increases coverage while maintaining the reported high precision. The search saturation results meaningfully motivate this design.
Critical weaknesses
  1. The abstract and introduction do not clearly distinguish faithfulness to the informal source from equivalence between formal statements. The paper keeps motivating the former while actually evaluating the latter. The research question is meaningful, but the framing needs to be clearer.
  2. The theoretical justification is insufficient. Proposition 1 proves that SemantEq implies logical equivalence, but the authors themselves explain that logical equivalence does not establish that two statements express the same claim. It therefore does not show that SemantEq captures the intended notion of equivalence.
  3. The connection between the definition and the implementations is not clear enough. There is no explicit mapping from the operations/stages in SemantEq-Eval to the four allowed steps, nor a proof or guarantee that the acceptances of SemantEq-Eval abide by Definition 1. The same issue applies to SemantEq-ProofAudit.
  4. The published labels do not exactly match the evaluation target, since they assess faithfulness to the natural-language statement rather than equivalence with the formal reference. Their interpretation and necessity in this evaluation therefore needs more justification.
Questions for authors

I like this paper a lot. The problem is meaningful, and the proposed theory is reasonable. However, I cannot ignore the gap between the theory and the actual implementations, as well as the need to refine parts of the theoretical justification. Please address the first three weaknesses, particularly the framing of the research goal, the justification beyond logical equivalence, and the connection between Definition 1 and the implementations. Resolving these concerns would allow me to raise my score.

  1. Is the intended goal to evaluate faithfulness to the informal source or equivalence with the formal reference?
  2. Beyond Proposition 1’s guarantee of logical equivalence, what justifies that SemantEq captures when two statements express the same mathematical claim?
  3. How do the operations in SemantEq-Eval and the checks in SemantEq-ProofAudit map to the four allowed steps? What guarantees that an accepted pair satisfies Definition 1?
Reviewer_h5vWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SemantEq, which evaluates whether a formalized mathematical theorem in Lean is equivalent to the formalized reference. This is the follow-up work to BEq and BEq+. SemantEq is implemented in two ways, which are denoted as SemantEq-Eval and SemantEq-ProofAudit. SemantEq-Eval is implemented as a Lean metaprogram without relying on LLMs, while SemantEq-ProofAudit uses LLMs to prove the equivalence, which is subsequently verified with Lean.

Critical strengths
  • This paper addresses a crucial and challenging problem in the domain of AI for Math: evaluating whether a Lean statement faithfully captures the intended mathematical claim (Although the contribution of this paper is limited; see weaknesses)
  • This paper proposes two variants: an LLM-free evaluator and a more flexible one using LLMs.
Critical weaknesses
  • The contribution of this paper is limited. SemantEq requires existing formalized references, which is unavailable when formalizing new mathematical theorems. Thus, the practical applicability is largely limited to evaluating the performance of autoformalizers on existing benchmarks. While this limitation is shared with BEq and BEq+, those works provide additional contributions, such as the improvement of autoformalizers and development of the evaluation benchmarks. In contrast, this paper does not provide contributions beyond the equivalence evaluation method and only follow-ups to these works, which limits the overall contribution.
  • The authors use their modified version of ProofNetVerif, but this modification is not entirely fair. Since the revision is based on the result of SemantEq, it might be favorable for SemantEq, even if they are checked by a human.
  • The experimental evaluation is only conducted on ProofNetVerif. This makes it difficult to assess whether the reported gains are coming from the overfit on this specific dataset or coming from the truly generalizable improvements.
  • The paper introduces Proposition 2 to show the strictness compared to the logical equivalence in the library, but this can only hold with the assumption that there is no Bridge Rewriting. Consequently, this proposition does not guarantee the behavior of the whole SemantEq, which includes Bridge Rewriting as one of the major components.
  • The LLM experiments in Section 4 (SemantEq-ProofAudit) are conducted through Codex. The paper does not clarify whether the filesystem/network access and command execution were restricted during those experiments. If no such restrictions were enforced, Codex might have had access to results from other ablation runs, ground truth, or other information in the working directory. It could also have inspected the audit implementation and potentially exploited loopholes in the auditing process through preliminary Lean execution. This potential information leakage and exploitation of the evaluator undermines the reliability of the experimental results reported in Section 4.
  • The reported 100% precision of SematEq-Audit (Section 4) is misleading, since it does not use the whole test split. Instead, it uses a selected subset consisting of 407 originally equivalent pairs and some other relabeled non-equivalent pairs. The authors should evaluate SemantEq-ProofAudit on the full test split, including the originally non-equivalent pairs.
  • Ablation studies on the proving steps of SemantEq-Eval (the steps described in the second paragraph of Section 2.2 and displayed in the blue boxes in Figure 2) would improve understanding of the contributions of these steps. Reporting how many theorems passed each step and how many of them are rejected at the audits would help.

The paper also has several presentational issues:

  • The notations such as D1, W1, D8, DW2 in Figure 3 are not described in the main text. The definitions only exist in the Appendix, which makes Figure 3 hard to interpret.
  • The notations and descriptions in Section 2.1 are cluttered. These contents can and should be written more simply and intuitively. Currently, they are written overly formally.
  • Since ProofNetVerif was created in the BEq+ work, relating them would improve the description of related work.
Questions for authors

Please see and address the weaknesses

Reviewer_uiF8OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the equivalence checking problem between 2 formal mathematical statements, and proposes SemantEq as the definition of formal-formal equivalence through bidirectional chains of logical rearrangement, library-based rewriting, changes of variables, and structural weakening. Based on the definition, the paper implements 2 complementary tools: SemantEq-Eval, a deterministic search-based equivalence checker, and SemantEq-ProofAudit, which examines elaborated Lean proof terms supplied by an external prover.

Critical strengths

Novelty

  1. The detailed proof-term audit is a meaningful contribution beyond tactic-text checks. Figure 4 and Appendix G distinguish goal derivations, antecedent arguments, and rewrite justifications. They also address unused local bindings, dependencies in branches, locally proved lemmas, and inappropriate instantiations. This combination is more specific and technically substantive than simply requiring the antecedent’s name in a script.

Soundness

  1. The stricter-policy experiment makes an important tradeoff visible. Table 4 reduces the faithful-prompt count from 330 to 323 accepted equivalent pairs, while retaining zero observed pair-level false acceptances on the evaluated negatives. This shows that most of the reported coverage improvement persists after removing the specifically designated severe tolerated uses.

Usefulness

  1. Higher precision and recall than BEq+ under both labeling schemes. SemantEq-Eval outperforms BEq+ on both original and adjudicated labels (Table 1), accepting more valid reformulations with fewer false acceptances. The gains therefore do not depend solely on relabeling.

  2. Complementary coverage with audited model-generated proofs. Algorithmic and model-assisted checking provide complementary coverage, jointly accepting 341/407 equivalent pairs (83.8%). SemantEq-ProofAudit supports this combination through explicit proof-term checks on how model-generated proofs use assumptions and library facts (§4.2).

Critical weaknesses

Novelty

  1. The conceptual justification for admissible bridges remains incomplete. Proposition 2 establishes strictness only for the bridge-free fragment under its first-order interpretation. The full relation therefore depends critically on bridge rewriting. Although the paper specifies syntactic admission and search rules, these do not establish that admitted bridges preserve the intended mathematical claim—a limitation acknowledged in §5.

    Suggestions: Clarify the bridge-ranking and premise-discharge procedures, provide a stronger justification for bridge admissibility, and foreground this acknowledged limitation alongside the definition.

Soundness

  1. Validation beyond the main benchmark remains limited. Most evidence comes from ProofNetVerif, supplemented by a small curated recall suite (§3.3; Appendix H). Generalization to research-level mathematics and different formalization conventions remains insufficiently tested.

    Suggestions: Add a small, independently labeled benchmark of research-level examples containing both faithful and unfaithful formalizations.

Usefulness

  1. Accuracy improvements come with higher computational cost. Tables 5–6 report mean CPU times of 30.5 seconds per pair for SemantEq-Eval versus 6.3 seconds for BEq+, making the default evaluator approximately 4.8× more expensive on the evaluated equivalent pairs.

    Suggestions: Extend the efficiency analysis to mixed positive and negative workloads and investigate ways to reduce the cost of difficult or unsuccessful searches.

Questions for authors

Clarification Questions

  1. What exactly does a successful certificate guarantee? Is acceptance intended to establish membership in Definition 2, compliance with a separate implementation policy, or empirical evidence of human-judged faithfulness? How should tolerated uses and additional axioms change the guarantee being reported?

  2. Can the auditor be evaluated on the original negative test population? What are its results on the 832 originally-negative elaborating ProofNetVerif pairs, rather than only the relabeled-negative certification subset?

  3. What happens under proof refactoring? Appendix G says that arguments passed through locally named partial applications are not checked. Do aliases, eta-expansion, or alternate local-lemma packaging change acceptance for otherwise corresponding proofs?

Discussion Questions

  1. Where should faithful reformulation draw the boundary between representation and additional mathematical content? In particular, when should a constraint encoded in a subtype be allowed to appear explicitly in a conclusion?

  2. How should library growth affect acceptance? If a new equation or biconditional becomes available, should previously non-equivalent statements become equivalent under the policy, or should some transformation judgments remain stable across library versions?

  3. Could the auditor guide proof generation without encouraging policy gaming? What evidence would distinguish improved discovery of faithful proofs from increasingly sophisticated exploitation of the acceptance rules?

Comments on the paper

1 comment

Thank you for the response and the revisions to the manuscript.

  1. The revision clarifies the scope, but I still think the motivation should center on deciding whether two formal statements express the same claim, while faithfulness to the informal source is a precondition instead of a motivation for this paper.
  2. Proposition 2 and the comparison with human judgments make a stronger case for SemantEq.
  3. The mapping and worked example clarify how the implementations relate to the four allowed steps, though it remains unproven that every accepted pair satisfies the definition.
  4. Comparability with prior work justifies reporting the published labels, though those labels remain an imperfect proxy for equivalence with the formal reference.

While some gaps remain, the paper addresses a meaningful problem in the field, provides valuable insights, and presents promising evaluation results. I would therefore raise my score to 3 (weak acceptance).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.