acceptodds
Under review as a conference paper at ICLR 2027

Beyond Formal Correctness: Structure-Aware Evaluation of Informal–Formal Proof Correspondence

Abstract

Proof assistants such as Lean certify that a formal proof establishes its target theorem, but do not determine whether it follows the reasoning expressed in a supplied natural-language proof. We introduce Goal–Context Delta Alignment (GCDA), a reference-conditioned framework for measuring this proof-process correspondence. GCDA recovers natural reasoning-state transitions and their derived dependencies, and traces the formal proof through Lean goal states connected by tactic-induced Goal–Context Deltas. It aligns each informal transition with a connected formal transition region, then checks required-region coverage, whole-proof coherence, and dependency direction. We evaluate GCDA on ProofRouteBench, a benchmark built from ReasBook and miniF2F with 4,888 proof variants across 670 theorem groups. GCDA achieves MacroAUCs of 0.9398 on ReasBook and 0.8788 on miniF2F, with strong separation on explicit dependency-certificate and cycle changes.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Formal Reasoning, Proof Evaluation, Informal-Formal Proof Correspondence, Lean

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mDKROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Goal–Context Delta Alignment (GCDA), a reference-conditioned score for how well a natural-language proof follows a given verified Lean proof, i.e., whether its key steps, dependencies and route are realized in the formal proof. It extracts reasoning transitions and their dependencies from the natural proof with an LLM, represents the Lean proof as goal-state transitions labeled with the change in goals and local context induced by each tactic (the Goal–Context Delta), aligns each informal transition with a connected tactic region, and combines step support, coverage of required formal regions, whole-proof coherence, dependency direction and a cycle penalty into one score. Built on this, the paper introduces ProofRouteBench, 670 theorem groups from ReasBook textbook formalizations and miniF2F, where each verified proof pair comes with process-preserving variants (paraphrase, removal of nonrequired steps) and five controlled changes (bridge removal, invalid rewrite, wrong dependency, circular dependency, alternative strategy). The method reaches a MacroAUC of 0.9398 on ReasBook and 0.8788 on miniF2F, compared with 0.6848 and 0.6818 for the strongest baseline, a direct LLM judge.

Critical strengths

The paper identifies an important problem that statement-level semantic alignment does not cover: a Lean proof can be correct and still not follow the argument it is supposed to formalize, which matters in scenarios where the intermediate steps of a proof should stay visible in the formal code, such as auditing model-generated solutions and textbook formalization, as the authors mention. This is a real bottleneck when formalizing textbooks, since a proof that closes the goal with heavy automation is correct but tells the reader little about the argument of the book. Using the change in goals and local context as the meaning of each tactic is a sensible design, since it lets several tactics realize one natural step and does not depend on the tactic text. The gain on the text-level changes is also clear, as GCDA reaches 0.9691 and 0.9237 on invalid rewrites, and its WorstAUC (0.8541 and 0.8197) stays far above every baseline, none of which exceeds 0.5832. The statistics are generally careful, with theorem-group bootstrap intervals and an independent rerun (ordering agreement 0.9177).

Critical weaknesses
  1. I notice in Section 4.1 that the wrong-dependency and circular-dependency variants only edit the dependency graph extracted from the natural proof, while the proof text stays the same. Only the proposed metric sees this graph, so the baselines cannot tell these variants apart from the original proof. The main comparison table reports only an average over all five types, and once the two dependency types are left out, its lead drops to about half. It is still clearly ahead on the remaining types, so I think the conclusion holds, but the per-type comparison should be in the main text, and a fair comparison would need a judge that also sees the graph.
  2. The main use case of GCDA in the paper is auditing natural-language proofs against a verified formalization, but it is only tested on synthetic edits of correct proofs, which may not fully reflect the errors that model-generated solutions actually make. Expert-labeled datasets of flawed proofs already exist, and the expert-rated pairs in the paper are not scored with the method either. The scores are also only comparable within one theorem, so it is hard to set a threshold for auditing a single solution. In addition, ReasBook covers analysis, algebra, probability and related fields, but no elementary geometry or combinatorics, where it is less clear that a formal proof should follow the book route (I am not asking the authors to cover these fields). Could the authors please state the intended scope? Is the method meant for fields where following the key steps matters, or as a general metric? Designing a metric takes some taste/ to be scenario-specific and I do not expect a perfect one, but this deserves a discussion.
Questions for authors

Clarification

  1. Since the main scenario of GCDA is auditing model-generated solutions, have the authors tried it on real flawed proofs, such as the expert-labeled errors in ProofLoc paired with a verified Lean proof of the same theorem, or on the expert-rated pairs in Section 4.5? This question matters a lot for my evaluation, and I would be glad to raise my score if the authors can show that it works on such data.
  2. For textbook formalization, which the paper also mentions, I'm a bit confused about why the key intermediate steps are not simply fixed as sublemmas. A common practice is to decompose the book proof into sublemmas for its key steps, check that each sublemma statement is faithful, and check that the final proof depends on them. The setting is also reversed there, since the book proof is fixed and different Lean proofs are compared against it, whereas the benchmark fixes the Lean proof. What does GCDA add in this setting, and have the authors tried ranking several Lean proofs of the same theorem against one natural proof?
  3. The certificate keeps a score for each informal step and each required formal region, and the paper checks if the edited structure can be localized. In practice a user often cares about a few key steps of the book proof. Is it possible to specify these steps and check if each of them is realized in the formal proof, and how reliable is such a per-step judgment compared with the global score?

Future work suggestions

  • Is there a plan to train a specialized model for this task? The way ProofRouteBench builds its variants looks like a way to synthesize training data in batch, although each example needs a natural proof paired with a verified Lean proof, and I am wondering how the authors would obtain such pairs at scale. The score could also be used as a reward for a prover that should follow a given book proof.
  • Reporting the cost and latency of the multi-stage GPT-5.4 pipeline would help readers judge if it can be used on a large formalization project.

Minor in write-up

  • • ProofBridge and Safe are closely related and could be discussed in the related work, and recent arXiv work on proof-route faithfulness such as Pistis, as well as the paper by DeDeo and Duede on the informal–formal correspondence problem, may help the framing.
  • Most of the evidence behind the main claims, including the per-type comparison and the caveats on the dependency graph, is in the appendix, and the many named objects such as ProofUnit and Formal Delta Card make the main text hard to follow.
  • I could not find which model the LLM baselines use, since Appendix G.3 only refers to the judge, and the later adaptive score overlay mentioned in the appendix is not defined.
3 comments

Per-category comparison with matched evidence

Wrong and circular dependencies edit the extracted graph while preserving the prose used at construction. In the rerun, Direct and the iterative judge receive the same graph and Lean evidence as GCDA. Given the rebuttal deadline, the comparison covers about one quarter of the original benchmark, selected using observed GCDA performance. For each edit type, the table averages AUC over proof groups. Both graph edits favor GCDA on this subset. The individual prose edits show mixed results, and the text-only average is close for GCDA and Direct.

EditGCDADirectIterative
Required-bridge removal0.77450.79260.7791
Invalid rewrite0.95590.92160.9324
Alternative route0.86670.88430.8181
Wrong dependency0.83130.68950.6761
Circular dependency0.96130.85810.8418
Text edits (MacroAUC)0.86570.86620.8432
Certificate edits (MacroAUC)0.89630.77380.7589
All five edits (MacroAUC)0.87800.82920.8095

Original proof pairs, thresholds, and scope

We paired the route ratings for the original Section 4.5 proof pairs with scores from the separate runs on the same source proofs. On the 19 pairs with a numeric route rating and valid scores from both methods, Spearman ρ was 0.8300 for GCDA and 0.7142 for Direct. Kendall τ-b was 0.6935 and 0.6256, respectively. The paired difference in Spearman ρ was 0.1158 (95% bootstrap CI [−0.1821, 0.4363]). This comparison measures route agreement on these proof pairs.

We also tested single-proof thresholds by selecting each threshold on training theorems and applying it unchanged to held-out theorems in five folds. On the constructed development labels, the mean balanced accuracy was 0.6342. Across 263 scored candidates, the low-score flag had precision 0.7544 and recall 0.7457. GCDA is intended for tasks that specify a route to preserve, such as formalizing a textbook proof or checking that an explanation follows the chosen Lean proof. A mathematically correct alternative may score lower when it bypasses the specified argument. The ReasBook evaluation covers analysis, algebra, probability, and related areas.

Textbook routes, key steps, and presentation

Faithful sublemma statements and their checked use in the final proof directly establish the intended correspondence. GCDA helps examine existing Lean proofs without a step map, including cases where a single textbook step spans several formal actions. With a fixed prose proof, each Lean implementation supplies its own trace for inspection.

Users can select existing informal step IDs and inspect the aligned Lean action regions and state changes. An audit of 45 supplied key steps from 24 theorems yielded 44 jointly evaluable judgments. The annotators agreed on 43 (97.73%, κ = 0.6563). This is agreement on the supplied local alignments. The revised Section 3 explains both alignment directions in plain language and follows one proof pair through the method.

Resources, model settings, and related work

In the gpt-5.6-terra rerun, candidate scoring used a median of four logical calls and about 49,500 recorded tokens per pair. Median summed request time was 340 seconds. In 12 successful reruns with prepared formal evidence, median elapsed time was 632 seconds. Evaluations of variants can reuse the trace and required regions. ProofRouteBench reuses each verified reference across its variants, yielding 4,888 candidates from 670 paired proof groups.

The archived Direct scalar and adapted ProofScore runs requested GPT-5.4. The matched rerun uses gpt-5.6-terra for GCDA and both judges. The fixed five-component formula reproduces all 339 audited historical Terra scores. A construction audit separately found score-conditioned selection of 149 miniF2F originals and 24 alternative-route variants.

ProofBridge generates and repairs Lean formalizations. Safe uses step-aware verification to check mathematical reasoning. Pistis specifies the route a generated formalization should follow. DeDeo and Duede discuss informal–formal correspondence at the level of statements and steps. GCDA evaluates step support, coverage, and dependencies in a supplied pair. ProofBridge · Safe · Pistis · DeDeo and Duede

Thank you for the response. This problem is vital and the formulation is valuable, so I raise my score to 3. However, the experiments remain incomplete, and I would encourage you to report the comparison on the full benchmark in the revision.

Thank you for reconsidering our submission. The matched-evidence comparison on the full benchmark is still in progress. We’ll post the results for each edit category here as soon as the run is complete.

Reviewer_FJ4hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a verified Lean proof follows the route of a given natural-language proof, which kernel acceptance does not establish. GCDA uses an LLM to decompose the natural proof into transitions with dependencies. On the Lean side, it reads tactic-level Goal–Context Deltas through LSP, and an LLM groups the tactics into required regions that are frozen before any variant is scored. The score combines forward support M_N, reverse coverage M_F, an LLM coherence judgment Q and dependency direction R, discounted by a cycle factor C. ProofRouteBench contains 670 theorem groups from ReasBook and miniF2F, with 4,888 variants covering three process-preserving controls and five negative types. On this benchmark GCDA achieves the best theorem-equal MacroAUC among all compared methods.

Critical strengths
  1. The problem matters. A kernel-accepted proof can still take a different route from the natural proof it formalizes. The paper proposes a concrete, reference-conditioned way to measure that gap.
  2. The formal-side decomposition is careful. States are complete Lean goals read through LSP, and each tactic is described by its observed goal and context changes rather than by its name. Closure is recorded only with verified evidence. Freezing the required regions before any variant is read prevents a deleted bridge from shrinking the coverage denominator.
  3. The paper ablates the score's components with leave-one-out and alternative aggregations (Tables 19–20). The ablations show category-specific roles: Q drives detection of invalid rewrites and bridge removal, and C drives cycle detection.
Critical weaknesses
  1. The evaluation relies on LLMs at almost every step. Scoring one proof takes at least five LLM calls, and an LLM also produces the frozen obligations that form the denominator of M_F. The deterministic parts (Delta, R and C) still operate on tactics and dependency graphs that an LLM selected. Extraction quality is never checked by humans, and the segmentation prompt is not provided. Repeated runs on the same proof differ by up to 0.20 in score (95th percentile, Table 21).
  2. The paper does not describe how most variants were built. It states only that GPT-5.6-sol generated the alternative strategies. It does not say who built the other negatives and the controls, or how they were checked. The variants appear to be built on GCDA's own extracted graph. Wrong and circular dependencies edit the dependency certificate D_N and leave the text unchanged, so GCDA sees the edit and text-based baselines cannot.
  3. Scores are comparable only within one theorem. On miniF2F, faithful paraphrases average 0.408, close to bridge removal at 0.343. The fully faithful worked example in I.8 scores only 0.708. This is enough for within-theorem ranking. It does not serve the paper's motivating use, auditing a single proof, which needs a score with a usable threshold.
  4. The expert study points to label noise among the positives. Of the 50 original pairs, both experts rate 8 as at most partially corresponding, and 4 are flagged as statement-incompatible or uncertain (Tables 23–24). Negatives derived from such originals do not test the contrast their labels claim.
Questions for authors
  1. Do the pairs rated 0–1 and the four statement-flagged pairs remain in the benchmark as positives? Please also report how GCDA and Direct scalar each correlate with the expert ratings on the 50 annotated pairs.
  2. How were the variants other than alternative strategy constructed: by an LLM (which model and prompt), by rules, or by hand? How did you verify that the controls are faithful and that the negatives are defective? How many rows of each type were skipped or excluded?
  3. What does evaluating one NL–FL pair cost in LLM calls, input and output tokens, money and wall-clock time? How does this compare with the single call of Direct scalar?
3 comments

Model decisions and repeatability

An LLM selects the required formal regions from the Lean trace before seeing a candidate. These regions define a common coverage denominator for variants of the same formal proof. Candidate scoring uses model judgments for prose decomposition, dependency inference, both alignments, and coherence. Lean supplies the recorded states. State differencing, validation, graph calculations, and aggregation are deterministic given those inputs. Algorithm 1 in the revised draft describes the segmentation steps.

In the audit of supplied outputs, the two annotators agreed on 116/142 required-region judgments and 111/129 local-support judgments. They marked 35 and 44 region partitions as incorrect, respectively. In a separate five-case audit, neither found a fully correct forward alignment. We repeated scoring on 10 fixed pairs. Rerunning only the coherence stage gave 28/30 valid observations and a p95 absolute score difference of 0.0331. Holding the prepared formal evidence fixed and rerunning all candidate-level model stages gave 22/30 valid observations and a p95 difference of 0.0666.

Variant construction and matched inputs

Rules generate the controls and the edits other than alternative strategy. Paraphrases replace surface wording. Nonrequired deletions remove a node outside the conclusion's ancestors, and required-bridge deletions reduce ancestor coverage. Invalid rewrites corrupt an equality or introduce a contradiction. Wrong and circular dependencies reverse or add an edge in the extracted graph, preserving the prose used at construction. We found 145 miniF2F groups whose published original text had been replaced after those variants were constructed. In the audit of 96 comparisons, annotators agreed on 88/96 mathematical-validity changes and 88/94 jointly evaluable route changes. The last column below records scoring exclusions for the 35 excluded ReasBook groups.

VariantminiF2F construction skipsReasBook construction skipsReasBook scoring exclusions
Faithful paraphrase0035
Nonrequired-step removal5829513
Required-bridge removal555435
Invalid rewrite0035
Wrong dependency4135
Circular dependency4135

Direct and the iterative judge receive the same edited graph and Lean evidence as GCDA. Given the rebuttal deadline, we report the matched rerun on about one quarter of the original benchmark. The subset was selected using observed GCDA performance. MacroAUC across the two graph-edit categories was 0.8963 for GCDA and 0.7738 for Direct.

Single-proof scores and the original proof pairs

On the 32/40 development cases with valid original and paraphrase scores, paraphrasing reduced the score by 0.1266 on average. We selected thresholds on training theorems and evaluated them on held-out theorems. Mean balanced accuracy was 0.6342 on the constructed labels. Across 263 scored candidates, a low-score flag had precision 0.7544 and recall 0.7457. The Appendix I.8 example scored 0.7268, with a threshold of 0.6335 fitted without that theorem. These figures quantify single-proof triage on the constructed development set.

The benchmark's positive labels designate originals and process-preserving variants. Partial correspondence in an original weakens the contrast with its variants. We audited the four statement-flagged proof pairs and found no exact matches among the released originals. For the original 50 pairs from Section 4.5, we used the separate archived runs tied to the source proofs. On the 19 pairs with a numeric route rating and valid scores from both methods, Spearman ρ was 0.8300 for GCDA and 0.7142 for Direct. Kendall τ-b was 0.6935 and 0.6256, respectively. The paired Spearman difference was 0.1158 (95% bootstrap CI [−0.1821, 0.4363]).

Per-pair resources

On matched development pairs, median logical calls were four for GCDA and one for Direct. With complete usage records, the median input and output counts were 39,773 and 7,765.5 tokens for GCDA, and 8,345 and 1,056 for Direct. Median summed request times were 340.01 and 57.07 seconds. In a separate set of 12 successful GCDA reruns with prepared formal evidence, median elapsed scoring time was 632 seconds.

Thank you for the response, which addresses some of my concerns. However, two problems remain: the method cannot be used directly to score semantic consistency, and the quality of the dataset is still hard to verify.

Thank you for raising this point. GCDA measures how closely an informal proof follows a particular Lean proof, so a valid proof using a different argument can score low. Our benchmark treats the original informal and Lean proofs for the same theorem as a positive pair. We then hold the Lean proof fixed and construct variants, for example by paraphrasing the informal proof or removing a necessary step. We will clarify that the original pair serves as a baseline and that its positive label alone does not establish that the two proofs follow the same argument.

Reviewer_LspNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes GCDA, a score for how well a verified Lean 4 proof follows the reasoning of a given natural language proof of the same theorem. The pipeline works in stages. An LLM (GPT-5.4) breaks the natural proof into steps and the dependencies between them. Lean provides the goal before and after each tactic. An LLM then matches natural steps to groups of tactics. The final score mixes several things: how well the steps match, how much of the Lean proof is covered, an overall LLM judgment of the proof, whether the dependencies point the same way, and a penalty for circular reasoning. To evaluate it, the authors build ProofRouteBench, 670 theorem groups from ReasBook (textbook style) and miniF2F (competition). Each natural proof comes with paraphrased versions and with corrupted versions (a needed step removed, a wrong rewrite, a wrong or circular dependency, a different proof strategy). They report MacroAUC of 0.94 and 0.88, well above text similarity metrics and direct LLM judges at 0.68 or less.

Critical strengths

I like that the score is computed against the real goal states of a verified proof, and that the formal side is fixed before the natural proof is even read. The diagnostics are nice too, you get told which step has no formal support, which dependency goes the wrong way, where the cycle is, instead of just getting a number.

Honest reporting. The appendix says plainly that two of the negative families only test "use of supplied structural evidence," that scores drop a lot under paraphrase, and that the expert study does not validate the metric.

Critical weaknesses

Over engineered and hard to read. This paper was very hard to read. Five score components, four or more LLM stages, a 340k token context budget per proof, renamed standard objects everywhere (ProofUnit, Formal Delta Card, frozen obligation, connector, Goal Context Delta), notation heavy sections where I couldn't tell what is decided by an LLM and what is deterministic, and 40 pages of appendix before you can find out what the method actually does. I had to go to Appendix F to learn that two of the five negative types never change the proof text. The "Goal Context Delta" is just the per tactic goal diff that the Lean infoview already shows, and which I think Paperproof and Pantograph expose directly. All this goes against the current tendency in the field, where the big formalization projects of the last year got further with less orchestration, a capable model, good tools and a clear rubric. It's not clear the new machinery offers any advantage over calling a better model with the criteria of Section 2.2 as a scoring rubric, and the paper never runs that comparison.

The baselines are not appropriate and the evaluation is not calibrated. BLEU, chrF, ROUGE, BERTScore, BARTScore and MoverScore on proof text are methods no one uses for this, and they just pad Table 3. The serious baselines are a single LLM judge call (run twice) and a ProofScore adaptation that does not reproduce the ProofScore formula, against a pipeline that makes five to ten GPT-5.4 calls per proof, some with hundreds of thousands of input tokens. No token counts, no cost, no call counts anywhere in the paper, and no condition where the single judge is given GCDA's inputs (the goal state trace and the dependency certificate) with a comparable budget. I can't tell whether any gain comes from the method or from the compute.

It's also not clear who would use it. GCDA assumes you already have a natural proof and an independently produced Lean proof, and want to know afterwards whether they follow the same route. But in the workflow where faithfulness actually matters, grading a natural proof through formalization, you formalize the proof step by step, so the mapping is known by construction and no alignment score is needed. So in which real setting does the independent pair arise?

Questions for authors

When you propose a new pipeline you need to compare against methods of similar compute. Tokens, cost and wall time are never reported, so it's impossible to tell whether GCDA is just taking advantage of extra compute. How many GPT-5.4 calls and tokens does it spend per scored proof, and what does the single judge get? Please also run the obvious thing people do in practice. One capable model told to evaluate faithfulness following the criteria of Section 2.2 written in a short .md, with the goal state trace and the dependency edges as input, iterating until it's confident, under the same budget. And if GCDA is that costly, why would someone not just spend the same money on a more powerful model with that prompt? Evidence from the big formalization projects of the last year is that less orchestration with frontier models and good tools beats a carefully tuned pipeline.

Also, doesn't the ablation seem to undercut the core component? Removing M_N, the informal step alignment the method is built around, does not hurt and even slightly improves MacroAUC on both corpora.

4 comments

When independently produced proofs need to be compared

We agree that recent formalization work shows how far a capable model can get with good Lean tools, a clear rubric, and light orchestration. GCDA asks whether an already verified Lean proof supports the key intermediate claims and dependencies of a specified prose argument. This situation arises when a reviewer obtains a Lean proof from the theorem statement and wants to identify which steps in a submitted argument warrant inspection. It also arises when a curator inherits a prose proof and a library proof independently and wants to use them as a matched example. A stepwise formalization with a verified mapping and checked dependencies already supplies much of this evidence. In a pair assembled after the fact, the mapping itself is the object to audit. For example, a formal proof can use a library theorem to establish the result directly while the manuscript relies on a separate intermediate claim. Inspecting that unmatched claim helps a reviewer locate the part of the manuscript that still needs an independent check. Tsoukalas et al. (2026)

What the model and Lean each provide

Section 3 of the revised draft gives a short overview and a worked example. In the example, Lean opens the pointwise goal with intro and uses congrArg to carry a real equality through the coercion to complex numbers. The trace makes it possible to inspect the premise used and the goal that the tactic closes. The model extracts the prose steps and dependencies, selects the required formal regions before candidate scoring, judges the two alignments, and assesses whole-proof coherence. Lean supplies the recorded goals and hypotheses. State differencing, validation, graph calculations and aggregation are deterministic given those inputs. Selecting the required regions before scoring keeps the coverage denominator fixed across variants of the same proof. We agree that the state changes are the same kind of evidence already available in Lean tooling. GCDA uses them to assess support for prose steps, coverage of the selected formal regions, and dependency correspondence. We credit Paperproof for visualizing these changes and Pantograph for programmatic proof-state access. Paperproof · Pantograph

Matched comparison with the iterative judge

The submitted GPT-5.4 study reported five-type MacroAUCs of 0.9398 on ReasBook and 0.8788 on miniF2F. After our provider stopped serving GPT-5.4, we reran the matched comparison with gpt-5.6-terra. The single-call judge applies the Section 2.2 rubric to the anonymous prose proof and its dependency edges, the theorem statement, the complete Lean trace, and the required formal regions. The iterative judge receives the same evidence and a short Markdown rubric. It stops when its initial assessment reaches confidence 0.85, a critique reaches 0.85 and requests no revision, or the budget is exhausted. GCDA and the iterative judge use the same model, cumulative token ceilings, and limit of five logical calls. This compares a structured alignment procedure with a rubric judge that can critique and revise within the same limits.

Given the rebuttal deadline, we report the rerun on about one quarter of the original benchmark. The subset was selected using observed GCDA performance. Both methods are evaluated on the same candidates. For each category, AUC is averaged over proof groups, and MacroAUC is the mean over the listed categories.

Evaluation categoriesGCDAIterative judgeGCDA − iterative
Prose edits, three types0.86570.8432+0.0225
Supplied-graph edits, two types0.89630.7589+0.1374
All five types0.87800.8095+0.0685

The observed difference is largest on supplied-graph edits. The comparison on prose edits gives a separate estimate for the part of the benchmark where the argument itself changes.

Measured scoring resources

The 340,000-token figure in the submitted configuration is the safe input ceiling. Actual usage below comes from the fixed gpt-5.6-terra development set. The request-time row sums recorded request durations for each pair. The token rows use pairs with complete usage records for the compared methods.

Per scored pairPaired NGCDA median / p95Iterative judge median / p95
Logical calls2594 / 51 / 4
Input tokens25739,476 / 191,46713,792 / 75,134
Output tokens2577,755 / 13,2011,305 / 4,641
Total recorded tokens25749,459 / 204,85015,615 / 80,348
Summed request time (seconds)259340.01 / 587.1765.97 / 236.26

Across all assigned candidates in that development set, GCDA used 2.70 times as many recorded tokens as the iterative judge. In 12 separate successful GCDA reruns with formal evidence already prepared, median elapsed scoring time was 632 seconds. Together with the per-category results, these measurements quantify the extra usage of structured scoring and the settings in which it produces the largest observed difference.

Forward-support ablation

We checked the apparent MNM_N removal in the supplement. That historical row used C(Q+R)/2C(Q+R)/2, which drops both forward support and reverse coverage. We also removed just the forward term, retaining reverse coverage, coherence QQ, dependency consistency RR, and cycle factor CC. Both calculations use the same development candidates and the original extracted alignment evidence.

Scalar terms removedFive-type MacroAUC changePaired 95% CI
Forward support only+0.0215[−0.0013, 0.0492]
Forward support and reverse coverage+0.0332[−0.0134, 0.0804]

Both intervals include zero. The forward term's removal gives a modest increase in the point estimate on this set. The underlying alignment records still show which informal claims received weak or missing Lean support.

Reviewer_LspNOpenReview ↗

Follow up on revisions

Thanks for the detailed response and the additional experiments, especially the token and cost accounting. Some of my concerns have been addressed and the clarifications help. I’ll take the new results into account in my final assessment.

Thank you for considering the additional experiments. Your question about compute prompted us to report actual token use and request times alongside the comparison with the iterative judge.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.