acceptodds
Under review as a conference paper at ICLR 2027

Transformal: Towards Semantically Faithful Formal Repository Translation in the Wild

11 views

Abstract

Large verified developments are scattered across proof assistants with different logics, such as Coq, Isabelle, and Lean. Formal repository translation enables their reuse across systems, but ensuring that translations preserve meaning remains an open problem. Even checking this translation faithfulness is difficult, because it requires relating declarations across two logics. Most existing faithfulness metrics check a statement against a reference or an informal description within a single logic, so they do not directly apply to translation. To bridge this gap, we propose counterexample-based faithfulness (CEF), a reference-free metric that reduces faithfulness checking to counterexample finding. A counterexample consists of source and target inputs that semantically represent the same object but lead to incompatible observations. To make counterexample search more reliable and efficient, we build Transformal, an LLM agent supported by four stages of program-analysis tools. The stages extract dependencies and elaborated signatures from the compilers, derive candidate inputs from branch conditions, and replay each candidate in both proof assistants. To evaluate Transformal, we construct TF-Judge, a benchmark of 297 expert-labeled declaration pairs from 18 real-world Coq and Isabelle repositories. With Claude Opus 5, Transformal reaches 97.67% accuracy, compared with 78.79% for the same model judging directly. The counterexamples of Transformal also provide precise feedback for translation, because each one shows where a translation diverges. On our translation benchmark TF-Translation, this feedback raises the share of GPT-5.6 Sol's declarations that satisfy CEF from 80.25% to 92.48%, four times the gain of compiler feedback. Counterexamples thus make translation faithfulness both measurable and repairable, and Transformal finds them more reliably and efficiently than baselines. Our code and data are available at https://anonymous.4open.science/r/transformal.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
formal verification, large language models, AI agents, program synthesis, formal repository translation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FkRBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies formal repository translation faithfulness checking and propose counterexample-based faithfulness (CEF), a reference-free metric that reduces faithfulness checking to counterexample finding. The paper then build an LLM agent named TRANSFORMAL to make counterexample search more efficient. The proposed agent achieves 94.28% accuracy with GPT-5.6 Sol on the constructed TF-JUDGE benchmark. Besides, the counterexamples also provide feedback on translation and improve CEF of GPT-5.6 Sol’s declaration form 80.25% to 92.48% on TF-TRANSLATION benchmark. Overall, the paper proposes a novel counterexample-based approach that can both evaluate translation faithfulness and improve translation quality through iterative refinement.

Critical strengths
  1. This paper addresses the important challenge of semantic faithfulness in formal repository translation and propose a counterexample-based evaluation metric that provides verifiable feedback beyond conventional compilation check and LLM judges.

  2. The proposed method achieves substantial improvements over baselines in both faithfulness evaluation and repository translation while reducing token consumption.

Critical weaknesses
  1. The repository translation experiments use the proposed TRANSFORMAL both to generate feedback for iterative refinement and to evaluate the translation quality. This raise the concern that the improvements in CEF scores may not necessarily reflect genuine gains in semantic faithfulness. An independent human evaluation would help validate the reported improvements.

  2. Although the benchmarks are constructed from real-world repositories, TF-JUDGE only contains 297 declaration pairs, with over half originating from FloatSpec, while TF-TRANSLATION includes only 4 repositories. Furthermore, all translation tasks target Lean, leaving the method’s effectiveness on other translation directions unverified. Additional evaluations on more diverse repositories or existing benchmarks would help strengthen the empirical evidence for the generalizability of the proposed method.

Questions for authors
  1. Since CEF relies on finding counterexamples, failure to find a counterexample does not prove semantic equivalence. Do the authors have any estimate of the false-alignment rate?

  2. The proposed framework heavily relies on dependency analysis. Did the authors evaluate the accuracy of the dependency extraction component or analyze how errors in dependency analysis impact the final faithfulness assessment results?

8 comments
AuthorsOpenReview ↗

Response to Reviewer FkRB [1/3]

We thank the reviewer for recognizing the value of the problem and method, and for the constructive suggestions. We address each weakness and question below. (Wi denotes weakness i; Qi denotes question i.)

W1: Transformal both generates the refinement feedback and scores the translations. An independent human evaluation would help.

Transformal's effectiveness is evaluated against the expert labels of TF-judge: 297 declaration pairs from 18 source projects, with 94.28% accuracy for GPT-5.6 Sol. These judgments use repository context: a TF-judge pair depends transitively on a median of 5 source declarations and up to 91 (Table 1), similar to the setting of TF-translation.

To further assess translation refinement independently, two formal verification experts blindly annotated a 150-sample translation set.

  • The first 100 case set covers 10 source declarations randomly sampled from TF-translation, each with its original translation and the outputs of three feedback methods (compiler, LLM judge, Transformal) over three rounds (i.e., the results in Figure 3). Expert labels agree with CEF on 92 of the 100 samples (92%).
  • To broaden coverage within the rebuttal period, we also sampled 50 random translation cases from the results in Figure 3, excluding the previous 100. Expert labels agree with the CEF score on 100% of these cases, which is also consistent with the 94.28% judge accuracy on TF-judge.

CEF agrees with the expert labels on 28/30, 29/30, and 29/30 translations refined with compiler, LLM-judge, and Transformal feedback, respectively. CEF is thus about equally accurate on translations refined with its own feedback, so the evaluation is not biased toward Transformal feedback. We will describe the human annotation explicitly in Section 5.3.

Besides expert annotation, the evaluation protocol also limits overfitting: each round's translation artifacts are scored by a fresh Transformal search rather than by replaying the witnesses supplied as feedback. A patch that only fits a reported witness therefore does not automatically pass the subsequent evaluation. Appendix J documents such patches.

AuthorsOpenReview ↗

Response to Reviewer FkRB [2/3]

W2: TF-judge has 297 pairs, over half from FloatSpec; TF-TRANSLATION has 4 repositories; all translations target Lean.

Size and context. TF-Translation contains 1,185 source declarations and 20,372 source lines from 39 files in four repositories. TF-judge contains 297 declaration pairs from 18 source projects; a pair depends on a median of 5 source declarations and up to 91, which the experts inspect when labeling (Section 4 and Appendix F). For comparison, ITPEval [1] contains 6,848 theorems across 1,560 source files in four proof assistants; MiniF2F in Rocq [3] evaluates 488 theorems, with 478 successfully translated; and lf-lean [4] contains 1,276 distinct Logical Foundations statements.

FloatSpec concentration. Excluding all 156 FloatSpec [2] pairs, Transformal correctly judges 132/141 remaining pairs (93.6%), compared with 82/141 (58.2%) for Raw LLM and 89/141 (63.1%) for Agent, computed from Table 6. Accuracy in each individual domain ranges from 90.00% to 96.67% (Table 5).

Translation direction. To test a non-Lean target, we evaluated Lean-to-Isabelle translation on ITPEval [1] (74 problems); we also report Isabelle-to-Lean results on its 316-problem miniF2F subset.

For Lean to Isabelle, we evaluated all 74 ITPEval proof-translation problems with GPT-5.6 Sol over three refinement rounds under compiler, LLM judge, and Transformal feedback. The following table reports target proof validity under each feedback condition. All three conditions share the same initial candidates; before refinement, 3/74 (4.05%) passed the compiler checks, 52 failed, and 19 were unknown.

Refinement roundCompiler feedbackLLM judge feedbackTransformal feedback
Initial3/74 (4.05%)3/74 (4.05%)3/74 (4.05%)
Round 16/74 (8.11%)7/74 (9.46%)7/74 (9.46%)
Round 29/74 (12.16%)7/74 (9.46%)11/74 (14.86%)
Round 311/74 (14.86%)7/74 (9.46%)15/74 (20.27%)

For Isabelle to Lean, we used ITPEval's 316-problem miniF2F subset with BEq evaluation. We excluded 35 items whose specifications are self-contradictory. Transformal evaluated the 281 eligible items over three refinement rounds. The table shows the initial result and refinement rounds 1–3 under Transformal and BEq metrics.

StageTransformal alignment (self-scored)BEq score
Initial274/281 (97.51%)58/281 (20.64%)
Round 1275/281 (97.86%)58/281 (20.64%)
Round 2277/281 (98.57%)58/281 (20.64%)
Round 3277/281 (98.57%)58/281 (20.64%)

The miniF2F task is close to saturation for GPT-5.6 Sol, so its marginal gains could be within noise, and we did not compare the other feedback signals there. Official BEq scores remained unchanged across all three refinement rounds. Using the full 316-problem dataset as the denominator, the BEq score is 58/316 (18.35%) at every stage. This also illustrates the limitations of a reference-based metric. Of BEq's 223 rejections, Transformal judged 209 aligned. For at least 121 items rejected by BEq and judged aligned by Transformal, we proved in Lean that the translation is equivalent to the Lean reference statement, which BEq's automated proof search failed to establish.

BEq therefore under-accepts logically equivalent translations when its proof search fails; a reference-free judge like Transformal avoids this dependency.

AuthorsOpenReview ↗

Response to Reviewer FkRB [3/3]

Q1: Failure to find a counterexample does not prove equivalence. What is the false-alignment rate?

Of Transformal's 229 aligned verdicts, 6 are wrong (2.62%). To quantify the agent-decided component, Table 6 reports the following false-negative and false-positive rates, with aligned treated as the positive class:

MethodFalse-negative rate (FN/234)False-positive rate (FP/63)
Raw LLM13.25% (31/234)19.05% (12/63)
Agent12.82% (30/234)22.22% (14/63)
Transformal3.85% (9/234)9.52% (6/63)

For GPT-5.6 Sol, Raw LLM and Agent did not return valid definitive judgments on 25.59% (76/297) and 27.27% (81/297) of cases, even when enforced to; these outcomes are excluded from the FP and FN counts but count as errors in current overall accuracy. Transformal reduces false rejections from 31 to 9 and false acceptances from 12 to 6 relative to Raw LLM (14 to 6 relative to Agent), with only 2/297 undecided. The residual errors arise from agent decisions about the cross-system matching relation (Appendix I). We consider these rates acceptable for the current setting, while identifying more reliable correspondence judgments as a direction for future work.

Q2: How accurate is the dependency extraction, and how do its errors affect the final assessment?

S0 reads dependencies from compiler indexes; it does not infer them with an LLM. Adding S0 introduces 0 additional false-negative cases in the reported ablation. The indexes are Coq coqc/coqdep and .glob declaration/reference facts; Isabelle Export_Theory exports of types, constants, theorem propositions, proof dependencies, and locales; and Lean compiled-environment declarations and elaborated types from Lean.getEnv.

Name resolution is exact for the compiled snapshot. S3 replays both harnesses against the full compiled repositories, so an omitted dependency in the supplied slice cannot make an invalid harness compile or change its observation.

Failure to find a counterexample does not certify equivalence, and more reliable correspondence judgments remain future work. We will describe the independent annotation in Section 5.3, add the FloatSpec-excluded result to Table 2, and make the false-alignment and undecided rates explicit in the text. Table 2 will report both all-pair accuracy (undecided counted as errors) and definitive-only accuracy with coverage.

We hope these responses address the reviewer's concerns and welcome further questions.

  • [1] Wu, Jiayi, et al. "ITPEval: Benchmarking formal translation across interactive theorem provers." arXiv preprint arXiv:2607.19407.
  • [2] Beneficial AI Foundation. "FloatSpec: Formally Verified Float Implementation with Lean 4." GitHub repository, 2025.
  • [3] Viennot, Jules, et al. "MiniF2F in Rocq: Automatic Translation Between Proof Assistants -- A Case Study." arXiv preprint arXiv:2503.04763, 2025.
  • [4] Saraswat, Vishesh, et al. "lf-lean: The frontier of verified software engineering." Theorem blog, 2026.
Reviewer_PozzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

What the paper is about is translating Coq/Isabelle repositories to Lean 4, and then checking whether the translated declarations still preserve meaning. Their proposal is counterexample-based faithfulness (CEF). A pair gets marked unfaithful when matched inputs give incompatible observations (different values, provable on one side and refutable on the other, or asymmetric type class instances), and this gets checked natively in each assistant.

The search for those counterexamples is done by Transformal, a Codex harness, using dependency slicing, binder matching, symbolic branch extraction, and dual compile-replay. On TF-JUDGE (297 hard pairs), GPT-5.6 Sol with Transformal reaches 94% accuracy, against 60% for direct LLM judging and 23% for round-trip+BEq. Used as translation feedback, it takes CEF-satisfying declarations from 80% to 92% over three rounds, beating simple compiler feedback at 83%. In the ablations, prompting for counterexamples on its own already hits 85%, and the four pipeline stages add another 9 points while also lowering token usage.

Critical strengths
  1. A replayable evidence standard for unfaithful translations

The failure cases are concrete and convincing. A "not_aligned" verdict comes with two harnesses that compile in their own proof assistants, apply the judged declaration , and disagree on matched inputs.

  1. Scale of the data.

18 real Coq/Isabelle repositories, 5,097 translated declarations judged and four repository-translation tasks are larger than anything in the LLM cross-assistant translation literature I know of.

Critical weaknesses

1 No motivation for why this is a significant problem to solve.

The paper says verified developments (seL4, CompCert, Flocq, AFP) live in Coq and Isabelle while AI work targets Lean, and stops there. What it never says is who needs a translated repository, what they would do with it, or why a translation is the right tool at all, as opposed to a certified bridge or just working in the source assistant. The one concrete case is FloatSpec, an AI-generated port of Flocq, which is also where 156 of the 297 benchmark pairs come from. With no stated use case the reader cannot judge whether a 94% judge or a 92% translation is useful.

2 The technical contribution is modest.

At the end of the day CEF is differential testing between two proof assistants. You look for one input where the two sides give different outputs. It is a good idea but a simple one, and the paper spends a lot of space on it. The 94 vs 60 gap in Table 2 gets presented as Transformal versus judging directly, but their own ablation in Figure 4 says that 85 of it comes from the criterion alone (just telling the model to search for a counterexample) and 9 from the system, with the first stage changing a single prediction out of 192. That simple version should be in Table 2, so the reader can see what the machinery is actually buying. What is also missing is a proper test-time baseline that is not told about counterexamples at all. Same model, same tools, same token budget, told to check faithfulness however it wants within budget X, with more rounds or best-of-k as the knob. Tool Call is sort of that but with a one shot judging prompt and no reported budget, so it is not surprising it does bad. Cost is not the issue here, Transformal actually spend less tokens than the raw judge (61k vs 140k per case) because the raw judge get the whole dependency closure pasted in the prompt. Figure 3(b) is the only plot with tokens on the axis and it compares feedback signals not strategies.

3 Can we trust the reviewer?

Whether two inputs "match" and whether two observations "differ" is decided by the agent. All the proof assistants check is that each harness compiles. The paper itself says every residual error comes from the agent choosing a convenient matching relation. So "not_aligned" verdicts are replayable but not model-free, and "aligned" verdicts are just "no counterexample found plus one example that agreed". The benchmark labels come from an unstated number of experts with no agreement figure. On top of that, 192 of 297 pairs were selected because an LLM judge got them wrong (so the raw-LLM accuracy and the gap are products of the selection), and the translation experiment in Figure 3 is scored by Transformal itself, which is also the feedback signal being evaluated.

Questions for authors

What are the practical situations where someone would actually translate a whole Coq or Isabelle repository into Lean and rely on it? Who is the user, and what do they do with the result?

Could you add two rows to Table 2 for all four models? One for the criterion alone (instruction to search for a counterexample, no stages), and one for a generic agent with the same tools and token budget that is told to check faithfulness however it wants. Please report token usage for every row, including Tool Call.

What do the numbers look like on the 105 pairs that were not selected for being hard, or on a random sample of the 5,097 declarations?

Who decides that two inputs match and two observations differ, and how often did the expert labellers disagree with the agent on this? Related to that, how many experts labelled TF-JUDGE and what was their agreement?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Pozz [1/3]

We thank the reviewer for the detailed feedback and concrete experimental requests. We have run the requested experiments and report the results below. (Wi denotes weakness i; Qi denotes question i.)

W1/Q1: Who needs a translated repository, and what do they do with it?

Our primary contributions are TF-judge, a benchmark for cross-proof-system semantic alignment; CEF, a faithfulness metric; and Transformal, the judge that computes it. The translation experiment evaluates whether a more accurate faithfulness signal also provides more useful feedback for iterative refinement; it does not claim a new translation algorithm. We will make this scope explicit in the Introduction.

As for the reviewer’s concern about motivation, formal repository translation is widely needed and studied in both academia and the open-source community:

  • In open-source community, the target users of formal repository translation are projects that build verified computer science in one popular yet relatively new proof system (Lean, Dafny, etc.) and want to reuse what already exists in other proof systems (Coq, Isabelle, etc.). CSLib [1], the pioneer Lean library launched in 2026 to be "for computer science what Mathlib is for mathematics", states that the base of computer science knowledge in Lean is currently quite limited, and emphasized that translation from other formal proof systems will be one of their major tasks.

  • On the academia side, formal repository translation is also a widely-explored yet open domain in formal verification research. FormalWeaver [2], lf-lean [3], and ITPEval [4] are concrete instances of this effort. In our Transformal, we introduce TF-judge with 297 declaration pairs from 18 source projects (234 aligned and 63 not_aligned), and a reference-free faithfulness judge that achieves 94.28% accuracy with GPT-5.6 Sol, making it another pioneering effort.

In the meantime, formal codebases are much easier to translate & port compared with building from scratch. The Flocq codebase, which served as part of the TF-judge benchmark, took over 10 years to build from scratch and consists of 72,728 lines of code. Re-deriving such developments by hand is rarely affordable, so porting them, increasingly with AI agents, is the practical path.

The alternatives raised by the reviewer do not fully address this reuse problem:

  • A certified bridge must import an entire proof system into another, which remains challenging even with AI assistance. Existing implementations have limited scope and regularly fail on complex formal repositories. In our TF-judge experiment, the manually created Rocq–Lean Round-trip + BEq baseline [3, 5] could translate only 30.43% of the Rocq–Lean cases, and the resulting proof-check-based evaluation achieved 22.56% overall accuracy, illustrating these scope limits.

  • Remaining in the source assistant does not resolve the need to combine developments from different systems. A project that needs both a Coq library such as Flocq and an Isabelle theory from the AFP has no single source assistant in which to use both. Their results require translation before they can be combined in the target system.

AuthorsOpenReview ↗

Response to Reviewer Pozz [2/3]

W2/Q2: The technical contribution is modest; most of the gain comes from the criterion alone. Add a criterion-only row and a budget-matched generic agent to Table 2, with tokens.

Through our experiment results, we respectfully disagree that most of Transformal's gain comes from criterion alone. Both requested controls are included below for Agent (the Tool Call row of Table 2). Agent+CE adds counterexample-search instructions without the additional stages. Agent+Budget keeps the generic instructions and prompts the agent to continue checking until a completed turn reaches or exceeds the full Transformal token budget for that case. The token column records observed usage.

The budget control shows that the criterion alone does not outperform a generic agent given more tokens (81.14%); the stages are what turn it into a 94.28% judge. With GPT-5.6 Sol, Agent+Budget reaches 81.14% and Agent+CE 80.47% on all 297 pairs (84.15% on its 284 definitive predictions), both well below Transformal's 94.28% in Table 2. Counterexample instructions improve the agent via increasing accuracy while maintaining similar or less amount of token consumption, whereas full Transformal pipeline went further by providing essential information from compiler to reduce the effort of search and elaboration.

ModelMethodsCoverageAcc., all 297 (%)Acc., definitive (%)Macro-F1 (%)*MCC*Bal. acc. (%)*Observed tokens (K/case)
GPT-5.6 SolAgent+Budget297/29781.1481.1474.800.509778.17262.09
GPT-5.6 SolAgent+CE284/29780.4784.1576.370.527576.53131.55
Opus 5Agent+Budget297/29779.1279.1273.050.484377.47329.17
Opus 5Agent+CE261/29780.1391.1987.010.755792.18305.51
GLM-5.3Agent+Budget297/29775.7675.7669.680.427374.76243.95
GLM-5.3Agent+CE211/29759.2683.4176.230.525175.67221.44
V4 ProAgent+Budget297/29778.1178.1172.310.476777.412294.87
V4 ProAgent+CE183/29753.5486.8979.670.608784.402150.15

*Computed on definitive predictions.

We will include token usage and cost for every row in the appendix and discuss the comparison in a new paragraph of Section 5.5.

W3/Q4: Who decides that inputs match and observations differ, how often do experts disagree, and how many experts labeled TF-judge?

The proof assistants check that each harness compiles without trust escapes, such as axioms or sorry, and that it applies the judged declaration to produce the stated observation. The LLM agent decides the cross-system matching relations: source input ≡ target input and source output ≡ target output. Existing cross-proof-system semantics do not cover the full range of input and output structures, so the protocol has to retain LLM-based correspondence checks.

MethodFalse-negative rate: faithful pairs falsely rejectedFalse-positive rate: unfaithful pairs falsely accepted
Raw LLM13.25% (31/234)19.05% (12/63)
Agent12.82% (30/234)22.22% (14/63)
Transformal3.85% (9/234)9.52% (6/63)

The remaining trust gap concerns concrete values that a reviewer can inspect directly, rather than a universal claim over all inputs. All remaining Transformal errors arise from the matching relation ≡ (Section 5.4). Of Transformal's 66 not_aligned verdicts on TF-judge, 9 are wrong (13.6%), compared with 31 of 82 (37.8%) for Raw LLM and 30 of 79 (38.0%) for Agent (Table 6).

Expert disagreement with the agent. Across all 15 Transformal false acceptances and false rejections, the disagreement concerns the matching relation or the choice of public interface (Appendix I). Table 6 quantifies these errors: with aligned treated as the positive class, the false-negative rate is 3.85% (9/234) and the false-positive rate is 9.52% (6/63). We consider these rates acceptable for the current setting, while identifying more reliable correspondence judgments as a direction for future work.

Benchmark labeling & curation. For TF-judge 297 pairs, we had two experts annotate & curate the data under the pipeline discribed in Q3. Both reviewers have PhD level knowledge and expertise in programming languages, and have relative opensource/academia experience with Lean and other proof systems. For the rebuttal, the same two formal-methods experts re-labeled a 60-pair TF-judge sample. They agree on 59 of 60 pairs (98.33%, Cohen's κ = 0.97). The disagreement concerns the binding of a dependency parameter in the target. On this sample, Transformal's predictions from the main run agree with the experts on 54 of 60 pairs. The original benchmark labels were fixed before any method was evaluated (Section 4).

AuthorsOpenReview ↗

Response to Reviewer Pozz [3/3]

Q3: The hard suite was selected by LLM errors, so the gap may be a selection artifact. What about the 105 other pairs, or a random sample of the 5,097 declarations?

Our selection procedure is carefully curated, and is fair towards different evaluated methods. TF-judge contains 297 declaration pairs from 18 source projects; each pair includes its repository context, with a median of 5 dependencies and up to 91 (Section 4 and Appendix F). The hard subset has 192 pairs, supplemented by 105 Isabelle-to-Lean pairs. The 192-pair subset of the 297 cases was constructed as follows:

  • We used a combination of GPT-5.6 Sol and Claude Opus 5 agents to translate the repositories listed in Appendix F, obtaining 5,097 translation pairs.
  • A shallow rule-based filter excluded incomplete or extremely short items, leaving 4,513 translations. We then ran two independent semantic-judgment rounds using GPT-5.5 (xhigh) + Codex agent.
  • The rounds disagreed on 870 pairs. We selected 200 with the largest confidence gaps for human annotation. The experts jointly discarded 8, primarily because semantic alignment was difficult to define for those cases.

The initial 192 pairs are concentrated in Coq-to-Lean translation (184) and in the aligned label (162). We therefore added an Isabelle-to-Lean batch using the same pipeline. Among 462 translation pairs, two GPT-5.5 (xhigh) + Codex agent sweeps produced 160 cases with differing judgments. Expert annotation retained 105 Isabelle-to-Lean pairs, including 72 aligned, reducing the concentration in that label.

A separate random sample tests the prevalence outside that curated benchmark: two experts agreed on 90 aligned and 8 not_aligned pairs among 100 sampled translations, with 2 disagreements concerning target features that change behavior. On these 98 pairs, the accuracies of GPT-5.6 Sol are 91/98 for Raw LLM, 93/98 for Agent, and 97/98 for Transformal. There are still gaps between different methods, but the gap would be much smaller and harder to tell between randomness, without our curation process.

The benchmark is enriched for difficult judgments and does not estimate natural error prevalence. We will make the contribution scope explicit in the Introduction, label Tool Call as Agent, add the criterion and budget controls with their scoring denominators to Table 2, report token usage and cost in the appendix, and clarify curation in Section 4 and Appendix F. Table 2 will report both all-pair accuracy (undecided counted as errors) and definitive-only accuracy with coverage.

We hope these responses address the reviewer's concerns and welcome further questions.

Reviewer_AGvTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies semantic faithfulness when translating Coq and Isabelle repositories into Lean. It proposes counterexample-based faithfulness (CEF) and Transformal, an LLM agent that searches for discrepancies using dependency context, elaborated signatures, branch inspection, and witness replay in both proof assistants.

Critical strengths
  1. The paper focuses on a useful problem: proof translatation. Using counterexamples makes this problem easier and more tractable.
  2. Checking witnesses in both proof assistants helps filter out invalid evidence. The ablation supports this design: adding S3 raises accuracy from 85.4% to 93.2% on the hard suite. The failure analysis is also useful for understanding where the method still makes mistakes.
Critical weaknesses
  1. The methodological novelty of searching counterexamples is compromised. Prior work has used LLM-generated, machine-checked counterexamples for theorem disproving and proof repair[1][2].
  2. The translation experiment uses the same framework for feedback (in main method) and evaluation. Transformal guides repairs, and Transformal with GPT-5.6 Sol also computes the final CEF score. This poses concerns on fairness of the evaluation and risks of overfitting. An independent assessment of the repaired translations is needed to support the stronger conclusion.
  3. The proposed counterexamples are only partially validated. Each proof assistant checks its own harness, but neither certifies that the source and target inputs correspond or that the observed difference demonstrates a translation error. Appendix I explicitly describes cases where both observations are locally correct but the matching relation is wrong. Thus, the method remains an LLM-based semantic judge supported by mechanically checked evidence, rather than an end-to-end counterexample validator. The paper acknowledges this limitation, but claims such as “compiler-verified counterexamples” read confusing and misleading.

[1] Yang, Jun, Yuechun Sun, Yi Wu, Rodrigo Caridad, Yongwei Yuan, Jianan Yao, Shan Lu, and Kexin Pei. "ExVerus: Verus Proof Repair via Counterexample Reasoning." arXiv preprint arXiv:2603.25810 (2026).

[2] Li, Zenan, Zhaoyu Li, Kaiyu Yang, Xiaoxing Ma, and Zhendong Su. "Learning to disprove: Formal counterexample generation with large language models." arXiv preprint arXiv:2603.19514 (2026).

Questions for authors
  1. What specific technical mechanisms distinguish Transformal from existing LLM-based counterexample generation and validation methods? In particular, what new machinery handles cross-language correspondence beyond prompting the agent with compiler-derived context?
  2. Was the improvement on TF-TRANSLATION independently checked by experts or another evaluation procedure? If so, does the improvement remain when the evaluator is independent of the repair feedback?
  3. Which parts of input matching and observation comparison are mechanically checked, and which are decided by the agent?
3 comments
AuthorsOpenReview ↗

Response to Reviewer AGvT [1/2]

We thank the reviewer for the constructive feedback and for recognizing the value of two-sided witness checking and the failure analysis. We address each weakness and question below. (Wi denotes weakness i; Qi denotes question i.)

W1/Q1: Prior work already uses LLM-generated, machine-checked counterexamples. What distinguishes Transformal?

Our contribution is a reference-free criterion and judge for whether a translated theorem statement or specification preserves its source meaning across proof systems. We don't claim novelty for counterexample search on isolated theorems. In theorem proving, counterexamples have been generated to disprove isolated statements [1] and to guide proof repair under a fixed specification [2]. Formal theorem translation, however, must also relate declarations, inputs, and observations across different proof systems, which neither setting requires. Existing approaches rely either on logical-equivalence checks with limited scope [4, 5, 6, 7] or on unchecked LLM judgments [3], which are difficult to supervise and perform poorly when meaning depends on complex dependencies.

Transformal combines compiler-derived information with counterexample search to address this setting: S0 retrieves compiler-indexed dependencies, S1 exposes elaborated binder roles and inherited assumptions, S2 derives inputs from branch conditions, and S3 replays both harnesses.

  • For example, S1 exposes the swapped parameter order in Coq FLT_exp (-10) 3 5 and Lean FLT_exp 3 (-10) 5, preventing a positional match from becoming a false counterexample (Appendix I).

On the 297-pair TF-judge benchmark, GPT-5.6 Sol achieves 57.91% as Agent, 80.47% with counterexample instructions alone (Agent+CE), and 94.28% with Transformal (all 297 pairs, undecided outputs counted as errors).

Nevertheless, we agree that the paper should distinguish single-language counterexamples from cross-proof-system counterexamples more clearly. We will explain this comparison more clearly to Section 2.

W2/Q2: The same framework produces the feedback and the final score. Was the improvement checked independently?

Transformal's effectiveness is evaluated against the expert labels of TF-judge: 297 declaration pairs from 18 source projects, with 94.28% accuracy for GPT-5.6 Sol. These judgments use repository context: a TF-judge pair depends transitively on a median of 5 source declarations and up to 91 (Table 1), similar to the setting of TF-translation.

To further assess translation refinement independently, two formal verification experts blindly annotated a 150-sample translation set.

  • The first 100 case set covers 10 source declarations randomly sampled from TF-translation, each with its original translation and the outputs of three feedback methods (compiler, LLM judge, Transformal) over three rounds (i.e., the results in Figure 3). Expert labels agree with CEF on 92 of the 100 samples (92%).
  • To broaden coverage within the rebuttal period, we also sampled 50 random translation cases from the results in Figure 3, excluding the previous 100. Expert labels agree with the CEF score on 100% of these cases, which is also consistent with the 94.28% judge accuracy on TF-judge.

CEF agrees with the expert labels on 28/30, 29/30, and 29/30 translations refined with compiler, LLM-judge, and Transformal feedback, respectively. CEF is thus about equally accurate on translations refined with its own feedback, so the evaluation is not biased toward Transformal feedback. We will describe the human annotation explicitly in Section 5.3.

Besides expert annotation, the evaluation protocol also limits overfitting: each round's translation artifacts are scored by a fresh Transformal search rather than by replaying the witnesses supplied as feedback. A patch that only fits a reported witness therefore does not automatically pass the subsequent evaluation. Appendix J documents such patches.

AuthorsOpenReview ↗

Response to Reviewer AGvT [2/2]

W3/Q3: Counterexamples are only partially validated; which parts are mechanically checked and which are decided by the agent? "Compiler-verified counterexamples" is misleading.

We agree that the wording should distinguish local verification from cross-system semantic judgment. In Transformal, the proof assistants check that each harness compiles without trust escapes, such as axioms or sorry, and that it faithfully applies the judged declaration to produce the stated observation. The LLM agent decides the cross-system matching relations: source input ≡ target input and source output ≡ target output. Existing cross-proof-system semantics do not cover the full range of input and output structures, so the protocol has to retain LLM-based correspondence checks. We will replace "compiler-verified counterexample" with "two-sided replayed counterexample" throughout and describe Transformal as an LLM semantic judge whose verdicts carry natively checked evidence.

Nevertheless, we still see our methodology as a meaningful contribution. Existing faithfulness checks for translation are either unchecked LLM verdicts [3], comparisons to a fixed reference that in-the-wild translations do not have [4, 5], or human-written bridges [6, 7], which are limited in scope and rarely apply in practice (e.g., Round-trip + BEq can only deal with less than 1/3 of the cases, and its accuracy is therefore limited, as shown in Table 2). CEF narrows the unchecked claim from all inputs to a few concrete values, improving judge quality. To quantify the agent-decided component, Table 6 reports the following false-negative and false-positive rates, with aligned treated as the positive class:

MethodFalse-negative rate (FN/234)False-positive rate (FP/63)
Raw LLM13.25% (31/234)19.05% (12/63)
Agent12.82% (30/234)22.22% (14/63)
Transformal3.85% (9/234)9.52% (6/63)

For GPT-5.6 Sol, Raw LLM and Agent did not return valid definitive judgments on 25.59% (76/297) and 27.27% (81/297) of cases, even when enforced to; these outcomes are excluded from the FP and FN counts but count as errors in current overall accuracy. Transformal reduces false rejections from 31 to 9 and false acceptances from 12 to 6 relative to Raw LLM (14 to 6 relative to Agent), with only 2/297 undecided. The residual errors arise from agent decisions about the cross-system matching relation (Appendix I). We consider these rates acceptable for the current setting, while identifying more reliable correspondence judgments as a direction for future work.

We hope these clarifications address the reviewer's concerns. We are incorporating them into the paper and will make the revision available within the rebuttal period. We welcome further questions.

AuthorsOpenReview ↗

Official Comment by Authors

Thanks again for the constructive feedback. We have provided detailed responses to all your questions and have conducted the additional experiments you requested. We have also revised our manuscript to address all the mentioned concerns.

The changes are highlighted in blue. Specifically, we made the following changes:

  • W1/Q1: We added a paragraph to Section 2 (line 156-182) that distinguishes single-language counterexamples from cross-proof-system counterexamples and cites ExVerus. We also added the Agent+CE and Agent+Budget controls to Table 2 and discuss them in Section 5.5 (line 480-508).
  • W2/Q2: We added the expert validation of CEF on 150 translations to Section 5.3 (line 431-437) and Appendix H (line 1059-1067).
  • W3/Q3: We replaced "compiler-verified counterexample" with "two-sided replayed counterexample" throughout (Figure 1, Section 3.2, Appendix H). We state which parts the proof assistants check and which the agent decides in Section 3.2 (line 223-228), and we describe Transformal as an LLM semantic judge whose verdicts carry natively checked evidence in Section 1 (line 92-93). We added the false-negative, false-positive, and undecided rates to Section 5.2 (line 371-382), Table 7, and Appendix G (line 902-910).

We would greatly appreciate any additional feedback you might have on our responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.