- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces Lean2Isabelle, a framework for translating Lean theorems and proofs into Isabelle/HOL. It separates statement translation from proof reconstruction and uses supervised fine-tuning followed by verifier-guided RL with a Minimal Valid Progress (MVP) reward.
The authors construct two datasets, Herald-ISA (26K examples) and LeanWorkbook-ISA (9.7K examples), and evaluate on MiniF2F-derived problems as well. The system achieves 54.3% SemPass@1 with reference Isabelle statements and 39.1% end-to-end on LeanWorkbook-ISA. The main idea is to make cross-assistant proof translation verifiable while explicitly measuring semantic drift.
Critical strengths
I really like the motivation behind this work :) As more mathematics gets formalized in different proof assistants, finding ways to reuse proofs across systems seems increasingly important. I particularly appreciate that the authors distinguish between generating a valid Isabelle theorem and actually preserving the meaning of the original Lean theorem.
The dataset construction is also a valuable contribution. I liked the source-proof ablations, especially the observation that removing tactics or shuffling proof steps significantly reduces performance. This provides useful evidence that the model is learning something from the original Lean proof rather than simply proving the target Isabelle statement independently.
Critical weaknesses
1. Semantic preservation is still difficult to guarantee.
My biggest concern is how we actually know that the translated Isabelle theorem means the same thing as the Lean theorem. Isabelle checking alone cannot establish this, so the authors use an LLM-based semantic judge. I appreciate the manual audits, but Appendix B reports that 92.5% of sampled judge-rejected Reference-Statement candidates were actually considered valid by human reviewers. This makes me wonder how much the reported success rates depend on the judge's decisions.
2. The improvement from MVP seems relatively small.
I find MVP interesting, but closing proof contexts does not necessarily mean making mathematical progress :) In the equal-budget experiments, MVP improves SemPass@1 from 51.0% to 52.3% over verifier-only GRPO. I would have liked stronger evidence that this reward correlates with actual progress and cannot be exploited through unnecessary intermediate proof steps.
3. Stronger baselines and larger migration experiments would help.
The structural translation baseline is fairly limited. I would be interested in comparing against an LLM agent with Isabelle execution, library search, and iterative repair. The 10-theorem matrix pilot is encouraging, but it remains unclear how well the approach scales to larger libraries with shared definitions and dependencies.
Questions for authors
-
Semantic equivalence: How sensitive are the SemPass results to errors in the LLM judge? Could some translations be checked through stronger, possibly machine-verified correspondence conditions?
-
MVP reward: Does closing Isabelle proof contexts reliably predict eventual proof success? Could a model game this reward by proving easy but irrelevant intermediate lemmas?
-
RL improvements: The equal-budget MVP gain is only 1.3 percentage points. Is this consistent across multiple training seeds, and is it statistically significant?
-
Stronger baselines: How would the system compare against a general-purpose LLM agent with Isabelle/PISA access, lemma retrieval, and iterative feedback? I think this would be a particularly interesting comparison :)
-
Library migration: Could the authors evaluate a larger dependency-connected collection of Lean theorems? I am curious how the system handles shared definitions, typeclasses, and reused lemmas.
-
Source-proof information: Which aspects of the Lean proof contribute most to translation success: tactic names, intermediate statements, or dependencies on existing lemmas?