- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper tackles the problem of autoformalization and test-time selection of samples for both informal and formal math domains. The general idea relies on building a graph, defining persisting objects/statements, and operating on them. The work shows experimental advantage of the proposed NF-Bridge method.
Critical strengths
Overall, the paper presents a straightforward idea for scaling inference and bridging natural language and Lean.
Strengths
- Useful construction. The authors use both lean and NL to construct aligned DAGs, which are subsequently used for TTL/reasoning.
- Wide array of benchmarks. This work includes many baselines and different scenarios.
Critical weaknesses
Weaknesses
- Missing baselines. The authors pose a question in the introduction: How can formal verification be translated into effective signals for TTS?, where TTS is test-time selection. And then Table 2 reports how the proposed method is better than other ORM/PRM models. But there already was the work on this: Safe [1]. And what is even more jarring is that they cite this work without elaborating on it in Related works or main section. Including a citation for it just being shown in the References section is already questionable, but omitting the work that directly proposes an answer from your introduction section is purposefully misleading. [1] proposed simple fine-tuning using generated Lean steps and combining them with ORM/PRM models for best test-time selection.
Moreover, this work is eerily similar to Hermes [2], where the abstracts bring up similar problems, proof of and , which I believe was proposed there (correct me if I'm wrong, I am unsure on this point), and the general idea of interleaving and combining natural language with formal language verification. I understand that the settings are slightly different, but it seems like a lot of re-used parts from that agent combined with DAG (which is not novel too) and wrapping as a new method. Overall, reading this work was giving me a serious déjà vu and the presentation of this work leans me more towards negative assessment, especially given that authors do not explicitly compare NF-Bridge with Safe, a rather straightforward baseline. Given that Safe was cited but not acknowledged anywhere and many re-used parts from another work without citation, I have rather serious concerns regarding this work.
- Unreliable evaluations. Moreover, I would argue that the auto formalization experiments and results are not adequate enough. First, the work lacks any back translation mechanism that is a staple established by [3] and discussed in many auto formalization works. The omission of it is an oversight. Second, the work makes confident claims as ~50% boost in proof auto formalization compared to some strong models. And who evaluated this? Another LLM. For a work that tries to answer a notoriously difficult and intricate question of auto formalization, this work lacks any kind of human ablation. And manual inspection is the golden standard, we cannot rely on semantic faithfulness made by LLM when evaluating another set of LLMs.
Moreover, I have a separate problem with Table 1, where first for models were not built for proof autoformalziation. This comparison is misleading and unfair. And in general it is hard to believe these massive improvements on datasets like MATH-500/miniF2F that are pretty much solved. I would like to see some confidence intervals to verify how stable this method is. And pick suitable strong baseline models for this task, instead of opting for models that were designed for theorem proving and not proof autoformalziation.
- Novelty and writing. In short: the writing and presentation of this work are abysmal.
The following are their most jarring examples: the narrative jumps around too much; Table 3 is never discussed; Tables 4 and 6 do not reveal which benchmarks were used; unnecessary and confusing introduction with Holistic-DAG/Uni-DAG that are not standard in literature and so weakly connected to the overall message; misleading and messy abstract that seemingly describes an agent but the text talks about fine-tuning and GRPO; figures use comic sans for crying out loud; and many, many more mistakes and inconsistencies I found but there is not enough space to lay them out here.
The writing is so messy, I had to read it thrice, and even then I am confused on the message and what authors wanted to say. None of the moving parts make sense; the introduction is talking about DAG, and then jumps to method, no related works in the main section. Descriptions and analysis are lacking. Somehow, authors could not put any clear, complete idea onto 9 pages. I would highly recommend authors to scrap everything and write from scratch, and stick to 1-2 main results with everything else in the Appendix. As it stands right now, this work is borderline unreadable.
And I think because writing is so messy, the novelty is buried. I am already not convinced by the results and baselines choices, but the abstract and introduction themselves are quite poorly written. The contributions mention improved autoformalization and Lean-powered TTS, but the results are weak, lack human inspection, and some baselines do not even appear. Overall, the paper should be re-worked and I recommend authors spend more time on writing instead of releasing an incoherent manuscript that is so jarringly difficult to follow.
References
[1] Liu et al. 2025. Safe: Enhancing Mathematical Reasoning in Large Language Models via Retrospective Step-aware Formal Verification. ACL
[2] Ospanov et al. 2026. HERMES: Towards Efficient and Verifiable Mathematical Reasoning in LLMs. ICML
[3] Gao et al. 2025. Herald: A Natural Language Annotated Lean 4 Dataset. ICLR
Questions for authors
Questions
-
What is the novelty compared to other DAG methods, Safe, Hermes?
-
Why Safe was cited but not acknowledged? Why is it not a baseline?
-
What is the purpose of Figure 1 in the narrative of this work? How is it even connected to the proposed method?