- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Goal–Context Delta Alignment (GCDA), a reference-conditioned score for how well a natural-language proof follows a given verified Lean proof, i.e., whether its key steps, dependencies and route are realized in the formal proof. It extracts reasoning transitions and their dependencies from the natural proof with an LLM, represents the Lean proof as goal-state transitions labeled with the change in goals and local context induced by each tactic (the Goal–Context Delta), aligns each informal transition with a connected tactic region, and combines step support, coverage of required formal regions, whole-proof coherence, dependency direction and a cycle penalty into one score. Built on this, the paper introduces ProofRouteBench, 670 theorem groups from ReasBook textbook formalizations and miniF2F, where each verified proof pair comes with process-preserving variants (paraphrase, removal of nonrequired steps) and five controlled changes (bridge removal, invalid rewrite, wrong dependency, circular dependency, alternative strategy). The method reaches a MacroAUC of 0.9398 on ReasBook and 0.8788 on miniF2F, compared with 0.6848 and 0.6818 for the strongest baseline, a direct LLM judge.
Critical strengths
The paper identifies an important problem that statement-level semantic alignment does not cover: a Lean proof can be correct and still not follow the argument it is supposed to formalize, which matters in scenarios where the intermediate steps of a proof should stay visible in the formal code, such as auditing model-generated solutions and textbook formalization, as the authors mention. This is a real bottleneck when formalizing textbooks, since a proof that closes the goal with heavy automation is correct but tells the reader little about the argument of the book. Using the change in goals and local context as the meaning of each tactic is a sensible design, since it lets several tactics realize one natural step and does not depend on the tactic text. The gain on the text-level changes is also clear, as GCDA reaches 0.9691 and 0.9237 on invalid rewrites, and its WorstAUC (0.8541 and 0.8197) stays far above every baseline, none of which exceeds 0.5832. The statistics are generally careful, with theorem-group bootstrap intervals and an independent rerun (ordering agreement 0.9177).
Critical weaknesses
- I notice in Section 4.1 that the wrong-dependency and circular-dependency variants only edit the dependency graph extracted from the natural proof, while the proof text stays the same. Only the proposed metric sees this graph, so the baselines cannot tell these variants apart from the original proof. The main comparison table reports only an average over all five types, and once the two dependency types are left out, its lead drops to about half. It is still clearly ahead on the remaining types, so I think the conclusion holds, but the per-type comparison should be in the main text, and a fair comparison would need a judge that also sees the graph.
- The main use case of GCDA in the paper is auditing natural-language proofs against a verified formalization, but it is only tested on synthetic edits of correct proofs, which may not fully reflect the errors that model-generated solutions actually make. Expert-labeled datasets of flawed proofs already exist, and the expert-rated pairs in the paper are not scored with the method either. The scores are also only comparable within one theorem, so it is hard to set a threshold for auditing a single solution. In addition, ReasBook covers analysis, algebra, probability and related fields, but no elementary geometry or combinatorics, where it is less clear that a formal proof should follow the book route (I am not asking the authors to cover these fields). Could the authors please state the intended scope? Is the method meant for fields where following the key steps matters, or as a general metric? Designing a metric takes some taste/ to be scenario-specific and I do not expect a perfect one, but this deserves a discussion.
Questions for authors
Clarification
- Since the main scenario of GCDA is auditing model-generated solutions, have the authors tried it on real flawed proofs, such as the expert-labeled errors in ProofLoc paired with a verified Lean proof of the same theorem, or on the expert-rated pairs in Section 4.5? This question matters a lot for my evaluation, and I would be glad to raise my score if the authors can show that it works on such data.
- For textbook formalization, which the paper also mentions, I'm a bit confused about why the key intermediate steps are not simply fixed as sublemmas. A common practice is to decompose the book proof into sublemmas for its key steps, check that each sublemma statement is faithful, and check that the final proof depends on them. The setting is also reversed there, since the book proof is fixed and different Lean proofs are compared against it, whereas the benchmark fixes the Lean proof. What does GCDA add in this setting, and have the authors tried ranking several Lean proofs of the same theorem against one natural proof?
- The certificate keeps a score for each informal step and each required formal region, and the paper checks if the edited structure can be localized. In practice a user often cares about a few key steps of the book proof. Is it possible to specify these steps and check if each of them is realized in the formal proof, and how reliable is such a per-step judgment compared with the global score?
Future work suggestions
- Is there a plan to train a specialized model for this task? The way ProofRouteBench builds its variants looks like a way to synthesize training data in batch, although each example needs a natural proof paired with a verified Lean proof, and I am wondering how the authors would obtain such pairs at scale. The score could also be used as a reward for a prover that should follow a given book proof.
- Reporting the cost and latency of the multi-stage GPT-5.4 pipeline would help readers judge if it can be used on a large formalization project.
Minor in write-up
- • ProofBridge and Safe are closely related and could be discussed in the related work, and recent arXiv work on proof-route faithfulness such as Pistis, as well as the paper by DeDeo and Duede on the informal–formal correspondence problem, may help the framing.
- Most of the evidence behind the main claims, including the per-type comparison and the caveats on the dependency graph, is in the appendix, and the many named objects such as ProofUnit and Formal Delta Card make the main text hard to follow.
- I could not find which model the LLM baselines use, since Appendix G.3 only refers to the judge, and the later adaptive score overlay mentioned in the appendix is not defined.
3 comments
Thank you for the response. This problem is vital and the formulation is valuable, so I raise my score to 3. However, the experiments remain incomplete, and I would encourage you to report the comparison on the full benchmark in the revision.
Thank you for reconsidering our submission. The matched-evidence comparison on the full benchmark is still in progress. We’ll post the results for each edit category here as soon as the run is complete.
Per-category comparison with matched evidence
Wrong and circular dependencies edit the extracted graph while preserving the prose used at construction. In the rerun, Direct and the iterative judge receive the same graph and Lean evidence as GCDA. Given the rebuttal deadline, the comparison covers about one quarter of the original benchmark, selected using observed GCDA performance. For each edit type, the table averages AUC over proof groups. Both graph edits favor GCDA on this subset. The individual prose edits show mixed results, and the text-only average is close for GCDA and Direct.
Original proof pairs, thresholds, and scope
We paired the route ratings for the original Section 4.5 proof pairs with scores from the separate runs on the same source proofs. On the 19 pairs with a numeric route rating and valid scores from both methods, Spearman ρ was 0.8300 for GCDA and 0.7142 for Direct. Kendall τ-b was 0.6935 and 0.6256, respectively. The paired difference in Spearman ρ was 0.1158 (95% bootstrap CI [−0.1821, 0.4363]). This comparison measures route agreement on these proof pairs.
We also tested single-proof thresholds by selecting each threshold on training theorems and applying it unchanged to held-out theorems in five folds. On the constructed development labels, the mean balanced accuracy was 0.6342. Across 263 scored candidates, the low-score flag had precision 0.7544 and recall 0.7457. GCDA is intended for tasks that specify a route to preserve, such as formalizing a textbook proof or checking that an explanation follows the chosen Lean proof. A mathematically correct alternative may score lower when it bypasses the specified argument. The ReasBook evaluation covers analysis, algebra, probability, and related areas.
Textbook routes, key steps, and presentation
Faithful sublemma statements and their checked use in the final proof directly establish the intended correspondence. GCDA helps examine existing Lean proofs without a step map, including cases where a single textbook step spans several formal actions. With a fixed prose proof, each Lean implementation supplies its own trace for inspection.
Users can select existing informal step IDs and inspect the aligned Lean action regions and state changes. An audit of 45 supplied key steps from 24 theorems yielded 44 jointly evaluable judgments. The annotators agreed on 43 (97.73%, κ = 0.6563). This is agreement on the supplied local alignments. The revised Section 3 explains both alignment directions in plain language and follows one proof pair through the method.
Resources, model settings, and related work
In the gpt-5.6-terra rerun, candidate scoring used a median of four logical calls and about 49,500 recorded tokens per pair. Median summed request time was 340 seconds. In 12 successful reruns with prepared formal evidence, median elapsed time was 632 seconds. Evaluations of variants can reuse the trace and required regions. ProofRouteBench reuses each verified reference across its variants, yielding 4,888 candidates from 670 paired proof groups.
The archived Direct scalar and adapted ProofScore runs requested GPT-5.4. The matched rerun uses gpt-5.6-terra for GCDA and both judges. The fixed five-component formula reproduces all 339 audited historical Terra scores. A construction audit separately found score-conditioned selection of 149 miniF2F originals and 24 alternative-route variants.
ProofBridge generates and repairs Lean formalizations. Safe uses step-aware verification to check mathematical reasoning. Pistis specifies the route a generated formalization should follow. DeDeo and Duede discuss informal–formal correspondence at the level of statements and steps. GCDA evaluates step support, coverage, and dependencies in a supplied pair. ProofBridge · Safe · Pistis · DeDeo and Duede