- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies formal repository translation faithfulness checking and propose counterexample-based faithfulness (CEF), a reference-free metric that reduces faithfulness checking to counterexample finding. The paper then build an LLM agent named TRANSFORMAL to make counterexample search more efficient. The proposed agent achieves 94.28% accuracy with GPT-5.6 Sol on the constructed TF-JUDGE benchmark. Besides, the counterexamples also provide feedback on translation and improve CEF of GPT-5.6 Sol’s declaration form 80.25% to 92.48% on TF-TRANSLATION benchmark. Overall, the paper proposes a novel counterexample-based approach that can both evaluate translation faithfulness and improve translation quality through iterative refinement.
Critical strengths
-
This paper addresses the important challenge of semantic faithfulness in formal repository translation and propose a counterexample-based evaluation metric that provides verifiable feedback beyond conventional compilation check and LLM judges.
-
The proposed method achieves substantial improvements over baselines in both faithfulness evaluation and repository translation while reducing token consumption.
Critical weaknesses
-
The repository translation experiments use the proposed TRANSFORMAL both to generate feedback for iterative refinement and to evaluate the translation quality. This raise the concern that the improvements in CEF scores may not necessarily reflect genuine gains in semantic faithfulness. An independent human evaluation would help validate the reported improvements.
-
Although the benchmarks are constructed from real-world repositories, TF-JUDGE only contains 297 declaration pairs, with over half originating from FloatSpec, while TF-TRANSLATION includes only 4 repositories. Furthermore, all translation tasks target Lean, leaving the method’s effectiveness on other translation directions unverified. Additional evaluations on more diverse repositories or existing benchmarks would help strengthen the empirical evidence for the generalizability of the proposed method.
Questions for authors
-
Since CEF relies on finding counterexamples, failure to find a counterexample does not prove semantic equivalence. Do the authors have any estimate of the false-alignment rate?
-
The proposed framework heavily relies on dependency analysis. Did the authors evaluate the accuracy of the dependency extraction component or analyze how errors in dependency analysis impact the final faithfulness assessment results?
8 comments
Response to Reviewer FkRB [2/3]
W2: TF-judge has 297 pairs, over half from FloatSpec; TF-TRANSLATION has 4 repositories; all translations target Lean.
Size and context. TF-Translation contains 1,185 source declarations and 20,372 source lines from 39 files in four repositories. TF-judge contains 297 declaration pairs from 18 source projects; a pair depends on a median of 5 source declarations and up to 91, which the experts inspect when labeling (Section 4 and Appendix F). For comparison, ITPEval [1] contains 6,848 theorems across 1,560 source files in four proof assistants; MiniF2F in Rocq [3] evaluates 488 theorems, with 478 successfully translated; and lf-lean [4] contains 1,276 distinct Logical Foundations statements.
FloatSpec concentration. Excluding all 156 FloatSpec [2] pairs, Transformal correctly judges 132/141 remaining pairs (93.6%), compared with 82/141 (58.2%) for Raw LLM and 89/141 (63.1%) for Agent, computed from Table 6. Accuracy in each individual domain ranges from 90.00% to 96.67% (Table 5).
Translation direction. To test a non-Lean target, we evaluated Lean-to-Isabelle translation on ITPEval [1] (74 problems); we also report Isabelle-to-Lean results on its 316-problem miniF2F subset.
For Lean to Isabelle, we evaluated all 74 ITPEval proof-translation problems with GPT-5.6 Sol over three refinement rounds under compiler, LLM judge, and Transformal feedback. The following table reports target proof validity under each feedback condition. All three conditions share the same initial candidates; before refinement, 3/74 (4.05%) passed the compiler checks, 52 failed, and 19 were unknown.
| Refinement round | Compiler feedback | LLM judge feedback | Transformal feedback |
|---|---|---|---|
| Initial | 3/74 (4.05%) | 3/74 (4.05%) | 3/74 (4.05%) |
| Round 1 | 6/74 (8.11%) | 7/74 (9.46%) | 7/74 (9.46%) |
| Round 2 | 9/74 (12.16%) | 7/74 (9.46%) | 11/74 (14.86%) |
| Round 3 | 11/74 (14.86%) | 7/74 (9.46%) | 15/74 (20.27%) |
For Isabelle to Lean, we used ITPEval's 316-problem miniF2F subset with BEq evaluation. We excluded 35 items whose specifications are self-contradictory. Transformal evaluated the 281 eligible items over three refinement rounds. The table shows the initial result and refinement rounds 1–3 under Transformal and BEq metrics.
| Stage | Transformal alignment (self-scored) | BEq score |
|---|---|---|
| Initial | 274/281 (97.51%) | 58/281 (20.64%) |
| Round 1 | 275/281 (97.86%) | 58/281 (20.64%) |
| Round 2 | 277/281 (98.57%) | 58/281 (20.64%) |
| Round 3 | 277/281 (98.57%) | 58/281 (20.64%) |
The miniF2F task is close to saturation for GPT-5.6 Sol, so its marginal gains could be within noise, and we did not compare the other feedback signals there. Official BEq scores remained unchanged across all three refinement rounds. Using the full 316-problem dataset as the denominator, the BEq score is 58/316 (18.35%) at every stage. This also illustrates the limitations of a reference-based metric. Of BEq's 223 rejections, Transformal judged 209 aligned. For at least 121 items rejected by BEq and judged aligned by Transformal, we proved in Lean that the translation is equivalent to the Lean reference statement, which BEq's automated proof search failed to establish.
BEq therefore under-accepts logically equivalent translations when its proof search fails; a reference-free judge like Transformal avoids this dependency.
Response to Reviewer FkRB [3/3]
Q1: Failure to find a counterexample does not prove equivalence. What is the false-alignment rate?
Of Transformal's 229 aligned verdicts, 6 are wrong (2.62%). To quantify the agent-decided component, Table 6 reports the following false-negative and false-positive rates, with aligned treated as the positive class:
| Method | False-negative rate (FN/234) | False-positive rate (FP/63) |
|---|---|---|
| Raw LLM | 13.25% (31/234) | 19.05% (12/63) |
| Agent | 12.82% (30/234) | 22.22% (14/63) |
| Transformal | 3.85% (9/234) | 9.52% (6/63) |
For GPT-5.6 Sol, Raw LLM and Agent did not return valid definitive judgments on 25.59% (76/297) and 27.27% (81/297) of cases, even when enforced to; these outcomes are excluded from the FP and FN counts but count as errors in current overall accuracy. Transformal reduces false rejections from 31 to 9 and false acceptances from 12 to 6 relative to Raw LLM (14 to 6 relative to Agent), with only 2/297 undecided. The residual errors arise from agent decisions about the cross-system matching relation (Appendix I). We consider these rates acceptable for the current setting, while identifying more reliable correspondence judgments as a direction for future work.
Q2: How accurate is the dependency extraction, and how do its errors affect the final assessment?
S0 reads dependencies from compiler indexes; it does not infer them with an LLM. Adding S0 introduces 0 additional false-negative cases in the reported ablation. The indexes are Coq coqc/coqdep and .glob declaration/reference facts; Isabelle Export_Theory exports of types, constants, theorem propositions, proof dependencies, and locales; and Lean compiled-environment declarations and elaborated types from Lean.getEnv.
Name resolution is exact for the compiled snapshot. S3 replays both harnesses against the full compiled repositories, so an omitted dependency in the supplied slice cannot make an invalid harness compile or change its observation.
Failure to find a counterexample does not certify equivalence, and more reliable correspondence judgments remain future work. We will describe the independent annotation in Section 5.3, add the FloatSpec-excluded result to Table 2, and make the false-alignment and undecided rates explicit in the text. Table 2 will report both all-pair accuracy (undecided counted as errors) and definitive-only accuracy with coverage.
We hope these responses address the reviewer's concerns and welcome further questions.
- [1] Wu, Jiayi, et al. "ITPEval: Benchmarking formal translation across interactive theorem provers." arXiv preprint arXiv:2607.19407.
- [2] Beneficial AI Foundation. "FloatSpec: Formally Verified Float Implementation with Lean 4." GitHub repository, 2025.
- [3] Viennot, Jules, et al. "MiniF2F in Rocq: Automatic Translation Between Proof Assistants -- A Case Study." arXiv preprint arXiv:2503.04763, 2025.
- [4] Saraswat, Vishesh, et al. "lf-lean: The frontier of verified software engineering." Theorem blog, 2026.
Response to Reviewer FkRB [1/3]
We thank the reviewer for recognizing the value of the problem and method, and for the constructive suggestions. We address each weakness and question below. (Wi denotes weakness i; Qi denotes question i.)
W1: Transformal both generates the refinement feedback and scores the translations. An independent human evaluation would help.
Transformal's effectiveness is evaluated against the expert labels of TF-judge: 297 declaration pairs from 18 source projects, with 94.28% accuracy for GPT-5.6 Sol. These judgments use repository context: a TF-judge pair depends transitively on a median of 5 source declarations and up to 91 (Table 1), similar to the setting of TF-translation.
To further assess translation refinement independently, two formal verification experts blindly annotated a 150-sample translation set.
CEF agrees with the expert labels on 28/30, 29/30, and 29/30 translations refined with compiler, LLM-judge, and Transformal feedback, respectively. CEF is thus about equally accurate on translations refined with its own feedback, so the evaluation is not biased toward Transformal feedback. We will describe the human annotation explicitly in Section 5.3.
Besides expert annotation, the evaluation protocol also limits overfitting: each round's translation artifacts are scored by a fresh Transformal search rather than by replaying the witnesses supplied as feedback. A patch that only fits a reported witness therefore does not automatically pass the subsequent evaluation. Appendix J documents such patches.