- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes align-then-label, a formulation of semantic alignment evaluation in which the evaluator first links informal regions to formal regions and then labels each link, so that every whole-item verdict can be traced to the expressions responsible for it. It targets theory-scale autoformalization, where a formal statement depends on definitions, notation and conventions outside the target, so a link may connect the informal target, its source context or implicit assumptions to the formal target or its formal context, either endpoint may be empty, and each link is labeled faithful, unfaithful or undecidable. Built on align-then-label, the paper introduces FaithAlign (460 items and 4,982 expert-annotated links from LCS-Bench, ProofNet-Verified, MA-Align and LeanEuclid–Pistis) and FaithAlign-Gap (146 items in which only the two targets and the formal context are given), where each of the 85 undecidable items is paired with two full-context versions of opposite verdicts. Across 8 models, the best one (GPT-6 Astra) reaches 47.8% defect F1 and finds every defect in only 34.3% of defective items, although its balanced verdict accuracy is 93.1%.
Critical strengths
The paper identifies an important gap in how semantic alignment is evaluated, since item-level verdicts hide which parts of a formalization are wrong. The link types in Figure 3 (paired regions, NL-only and Lean-only, each faithful or unfaithful) are a clean and fine-grained way to separate harmless omissions such as 'Show that' from missing conditions, and the annotation could facilitate future research on finer evaluators. The sources (logic and linear algebra textbooks, Euclid's Elements, MA-Align) extend evaluation to textbook material with its surrounding context, which complements the competition-style sets that most evaluators are built on, and since the defects are found in released formalizations, FaithAlign can serve as a natural held-out test. The triplet design is also worth noticing: with full context GPT-6 Astra judges 96.9% and 100% of the two readings correctly, yet it flags only 28.2% of the undecidable items once the informal context and assumptions are removed.
Critical weaknesses
- It is hard to tell what the headline metric measures, since there is no human ceiling. The only reliability figure is the 81.9% link overlap between the reviewed and initial annotations (Appendix A.4), and the reviewer saw the initial version, so this is not independent agreement. Under reference grouping, Astra's joint F1 rises from 47.6 to 67.5 (Table 11), and the full-context prompt (Appendix D.1) asks models to group coherent contributions, whereas annotators narrow each defect region as much as possible (Section 4.1). That's a thin basis for reading 47.8% defect F1 as low. The benchmark idea is good, but the annotation quality needs to be documented in more detail, such as the full guideline, per-label agreement from blind double annotation on a subset, and examples where annotators disagreed.
- The headline result on undecidable items is hard to interpret. The undecidable-item rate (28.2% for Astra) is a recall over the 85 undecidable items with no usable precision at the item level, since an unfaithful finding overrides an undecidable one and only 12 faithful controls remain where a wrong undecidable verdict would show. The rate also credits undecidable labels placed on the wrong link: DeepSeek V3.2 reaches 25.9% at the item level but only 4.3% undecidable-link F1 (Table 8), compared with 28.2% and 17.4% for Astra. There is also no agreement figure for the undecidable label, and the informal context of the opposite-verdict counterparts is written by the authors, with 54 of these counterparts also counted as FaithAlign items (Appendix A.3). The link-level F1 or a paired success rate (undecidable on the right link and both full-context versions judged correctly) would be a better headline.
Questions for authors
Clarification
- As a benchmark the align-then-label target is well designed. Is there a plan to train a specialized model on this formulation, possibly covering standalone statements as well? I am curious how the authors would synthesize training data with link-level labels, such as injecting errors into faithful pairs whose links are known, or dropping the context to create undecidable cases, and if the scorer could serve as an RL reward given its sensitivity to annotation granularity (Table 11). This question matters a lot for my evaluation, and I am willing to raise my score if the authors can give a convincing answer.
- For one-defect items Astra locates the defect in 70.9% of cases (Figure 8), yet its balanced verdict accuracy is 93.1%. How many correct unfaithful verdicts rest on findings that match no annotated defect? Reporting defect precision and recall separately would make this clear.
- I'm a bit confused about how the formalizations in each source were produced. Were they written by humans or generated by models?
- RQ2 mentions that links involving context score 15.7 points below the overall joint F1, but this deficit is pooled over sources, while LCS-Bench has a median informal context of 13,850 characters and the other sources have under 250. Could the authors split it by source?
Future work suggestions
- A correction would provide autoformalization pipelines with a more useful signal than a verdict, since it shows how to fix the statement, but it is hard to reward directly. It would be interesting to see if correction becomes easier when the model first produces the links, since the defect is already located.
- Since only general-purpose models are evaluated, I would also be interested in how specialized semantic alignment evaluators such as LeanScorer or FormalRx behave on FaithAlign at the verdict level, though I understand they cannot produce links.
- Relating the work to criterion-level LLM judging of autoformalization (Zhang et al., Beyond Gold Standards, 2025) would help the positioning.
Minor in write-up
- Figure 2 is too hard to read. I tried a range of zoom levels and still could not make out the text, although it is the main figure that shows what an alignment looks like. A larger version with two or three items would help readers a lot.
- The claim of the first theory-scale faithfulness benchmark could be narrowed to context-linked, span-level annotation. Table 8's full-context accuracy on the original items equals Table 1's value for the faithful versions (96.9 ± 0.7), which needs a short clarification.
5 comments
Q1: “ Is there a plan to train a specialized model on this formulation, possibly covering standalone statements as well? ”
We thank the reviewer for the encouraging question. Training a specialized evaluator on the align-then-label formulation is a promising direction for future work. We see the following three dimensions to address:
Specialized evaluator
- Expert annotation might not scale to theory-scale, so a specialized evaluator would need to learn from weaker supervision and generalize beyond it [1]. In our align-then-label task, since each link is a small, locally checkable claim, it suits such supervision. FaithAlign can then serve as the human-annotated test of such generalization.
- In terms of standalone statements, they could be regarded as the special case with empty context, so the formulation covers it without needing additional adaptation.
Training data
- Synthesizing data could be achieved by deterministic edits such as constant, exponent, or type changes [2], or LLM error injection guided by an error taxonomy [3]. They both edit a known aligned pair at a known location, so they could be easily extended to link-level labels.
- During our FaithAlign, especially FaithAlign-Gap construction, we have explored a way of construction via constructing item families. That could potentially be scaled up with the aid of LLMs. The procedure is as follows
- We first identify items that are undecidable with only their targets. They are usually underspecified, hence we can produce a family of items from the same idea, e.g., sharing the informal target, each fixing a different interpretation by more specific assumptions or slight changes to statements in the context.
- Dropping the context and assumptions of a family member with no definite defect on its target-to-target links directly yields an undecidable item.
- Error injection would simply reduce to mixing and matching members' targets with each other's contexts and assumptions.
RL reward
- The design of the RL reward could consider these points:
- With the revised verdict accuracy as mentioned in the response to W2, a correct verdict earns credit only through a correct link.
- The precision term penalizes flooding the alignment with links.
- Sensitivity to granularity could be mitigated by prediction splitting and reference grouping, the exact effect is reported in Table 15. Hence the reward would need to combine various ways of aligning regions, depending on the granularity we desire.
- [1]: Burns et. al., Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision -- ICML 2024
- [2]: Lu et. al., FormalAlign: Automated Alignment Evaluation for Autoformalization -- ICLR 2025
- [3]: Wang et. al., FormalRx: Rectify and eXamine Semantic Failures in Autoformalization -- ICML 2026
Q2: “How many correct unfaithful verdicts rest on findings that match no annotated defect? Reporting defect precision and recall separately would make this clear.”
We thank the reviewer for the question. Following the reviewer's suggestion in W2, the revised verdict accuracy no longer counts these verdicts as correct. As shown below, GPT-6 Astra's unfaithful-item accuracy falls from 94.7 to 76.8 under the revised counting, and the drop is largest on one-defect items.
We also report defect precision and recall separately. Precision exceeds recall for every model, so missed defects lower defect F1 more than false alarms do.
(NEW) Unfaithful-item verdict accuracy of GPT-6 Astra under the original and the revised counting
| defective items | items | original (%) | revised (%) |
|---|---|---|---|
| one defect | 133 | 93.5 | 70.9 (−22.6) |
| two defects | 75 | 93.3 | 73.8 (−19.5) |
| three or more defects | 88 | 97.7 | 88.3 (−9.4) |
| all | 296 | 94.7 | 76.8 (−17.9) |
(NEW) Defect precision and recall
| model | precision (%) | recall (%) |
|---|---|---|
| GPT-5.6 Sol | 53.5 | 35.6 |
| Opus 5 | 53.1 | 22.8 |
| GPT-6 Astra | 59.8 | 39.8 |
| Opus 4.8 | 60.3 | 16.9 |
| Kimi K3 | 54.4 | 30.1 |
| GLM-5 | 19.7 | 10.3 |
| DeepSeek V3.2 | 26.0 | 14.6 |
| MiniMax M2.5 | 29.4 | 4.0 |
Q3: “I'm a bit confused about how the formalizations in each source were produced. Were they written by humans or generated by models?”
The formalizations are obtained from the source datasets. As described in Section 4.1, we use the informal–formal pairs provided by each source as targets, and for LCS-Bench and LeanEuclid, the formalizations were provided directly by their authors. Each formalization was therefore produced by its source's own construction process, as described in the cited source papers.
Q4: “RQ2 mentions that links involving context score 15.7 points below the overall joint F1, but this deficit is pooled over sources. Could the authors split it by source?”
We thank the reviewer for the suggestion. We split the comparison in RQ2 by source below. We have added the per-source breakdown to Appendix C.3 as Table 11.
Each role column gives the joint F1 of links with that role minus the overall joint F1 of the same source, in percentage points, and the context-links column counts the annotated context links. Following Section 3.2 Regions and links, a link is implicit if an endpoint lies in the assumptions , context if it has no endpoint in but one in the informal or formal context, and target otherwise.
Joint F1 of each link role relative to the overall joint F1 of each source, pooled over models and repetitions
| source | context links | overall joint F1 (%) | target (pp) | context (pp) | implicit (pp) |
|---|---|---|---|---|---|
| LCS-Bench | 690 | 41.2 | +9.9 | −20.1 | −15.1 |
| ProofNet-Verified | 42 | 50.4 | +2.7 | −11.6 | −8.1 |
| MA-Align | 174 | 29.0 | +1.0 | −8.1 | +3.6 |
| LeanEuclid–Pistis | 0 | 32.0 | −2.0 | – | +27.9 |
| All | 906 | 37.6 | +4.1 | −15.7 | −2.5 |
M1: “Figure 2 is too hard to read.”
We have replaced Figure 2 with a larger version that shows three representative items.
M2: “The claim of the first theory-scale faithfulness benchmark could be narrowed to context-linked, span-level annotation.”
We haved narrowed the claim to the first theory-scale faithfulness benchmark with context-linked, span-level annotation.
M3: “Table 8's full-context accuracy on the original items equals Table 1's value for the faithful versions (96.9 ± 0.7).”
Table 12 (Table 8 in the submission) reports accuracy on the 85 original items, consisting of 72 faithful and 13 unfaithful, while Full F in Table 1 reports accuracy on the 85 faithful versions, consisting of the 72 faithful originals and the 13 faithful counterparts we wrote for the unfaithful originals. Both sets share the 72 faithful originals, where all of Astra's full-context errors occur, and Astra judges the 13 remaining items in each set correctly, so the two values coincide.
Following the reviewer's suggestion in W2, the revised verdict accuracy additionally requires a correct unfaithful verdict to rest on a matched unfaithful link, which applies to the 13 unfaithful originals in Table 12 but not to Full F in Table 1, so the two values now differ slightly in the revised PDF.
S1: “It would be interesting to see if correction becomes easier when the model first produces the links, since the defect is already located.”
We thank the reviewer for the insightful suggestion. We agree that this is a very promising direction. Correction guided by the links is a natural next step, and it is part of our plan for future work.
S2: “Since only general-purpose models are evaluated, I would also be interested in how specialized semantic alignment evaluators behave on FaithAlign at the verdict level.”
We thank the reviewer for the suggestion. We run LeanScorer with its released implementation and threshold α = 0.6 on a random quarter of FaithAlign (115 items, proportional to each source and verdict), giving it only the informal and formal targets. Since LeanScorer produces no links, its verdicts are scored alone, while align-then-label uses the revised verdict accuracy from W2, which also requires a matched link. As shown below, with GPT-6 Astra, align-then-label is more accurate than LeanScorer despite the stricter counting, while with DeepSeek V3.2 it is less accurate.
(NEW) Balanced verdict accuracy on a random quarter of FaithAlign (115 items)
| backbone | LeanScorer (%) | align-then-label, 3 runs (%) |
|---|---|---|
| GPT-6 Astra | 71.5 | 87.2 ± 2.0 |
| DeepSeek V3.2 | 61.2 | 54.1 ± 2.3 |
S3: “Relating the work to criterion-level LLM judging of autoformalization (Zhang et al., Beyond Gold Standards, 2025) would help the positioning.”
We thank the reviewer for the suggestion. We have included the work in our revision.
We conducted a blind re-annotation study. Two human experts who were not among the original annotators independently re-annotated a random sample of 30 FaithAlign items, proportional to each source and verdict and 10 FaithAlign-Gap items. Each annotated 15 FaithAlign items and 5 FaithAlign-Gap items without overlaps. Each expert received exactly the inputs given to the models. We score the re-annotation against the reference with the same evaluation scorer used for the models, and report GPT-6 Astra on the same items. We present here the reannotation statistics and an example of annotator disagreement. They have been added in the revised PDF in more detail.
As shown in the table below, human agreement exceeds the strongest model by a wide margin on alignment structure, defect labels, and undecidable labels.
Human agreement and GPT-6 Astra on the same items
Example of annotator disagreement: ProofNet-Verified PN-123 (Munkres, Exercise 26.11)
(hAne : A.Nonempty), which the informal statement does not state.(hAne : A.Nonempty).(A : Set (Set X)) (hAne : A.Nonempty). Both identify the same defect and verdict, and differ only in whether the added hypothesis has an informal counterpart. All six uncredited re-annotated defects identify the same defect as the reference, three with a different empty endpoint and three at a coarser grain.We thank the reviewer for the suggestion. Following it, we revise item-level verdict accuracy so that a correct verdict must be supported by a correct link:
Headline numbers under the revised verdict accuracy
The revised PDF updates all affected numbers.