acceptodds
Under review as a conference paper at ICLR 2027

Aligning Faithfulness for Theory-Scale Autoformalization

15 views

Abstract

Autoformalization (AF) requires more than compilation: formal statements must preserve the meaning of their informal sources, a property known as faithfulness. An emerging frontier is the scalable evaluation of this property, yet most existing methods operate at the statement level and produce whole-item verdicts that do not reveal how individual expressions correspond or which contextual interpretations support the judgment. This omission is consequential because AF is inherently theory-scale: target statements derive their meaning from surrounding mathematical material and unstated conventions. To address these limitations, we reformulate faithfulness evaluation as a context-aware align-then-label problem: identifying phrase-level correspondences between informal and formal statements and assigning a faithfulness label to each under explicitly specified contextual interpretations and assumptions. We also introduce FaithAlign, the first theory-scale faithfulness benchmark comprising 460 items and 4,982 annotated phrase-level links, alongside FaithAlign-Gap, a companion set of 146 items with informal context and assumptions withheld. Each FaithAlign item includes the target statement pair, relevant context, formal dependencies, and, where needed, assumptions specifying the interpretation under evaluation. Across 8 models, GPT-6 Astra achieves the highest defect F1 at 47.8% and identifies every defect in only 4.5% of items containing two or more defects. On FaithAlign-Gap, Astra correctly identifies only 28.2% of undecidable items as such. These results highlight the difficulty of recovering context-dependent semantic correspondences, establishing FaithAlign as a testbed for fine-grained, context-aware faithfulness evaluation.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Autoformalization, faithfulness evaluation, benchmark, formal verification, large language models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_L2SUOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes align-then-label, a formulation of semantic alignment evaluation in which the evaluator first links informal regions to formal regions and then labels each link, so that every whole-item verdict can be traced to the expressions responsible for it. It targets theory-scale autoformalization, where a formal statement depends on definitions, notation and conventions outside the target, so a link may connect the informal target, its source context or implicit assumptions to the formal target or its formal context, either endpoint may be empty, and each link is labeled faithful, unfaithful or undecidable. Built on align-then-label, the paper introduces FaithAlign (460 items and 4,982 expert-annotated links from LCS-Bench, ProofNet-Verified, MA-Align and LeanEuclid–Pistis) and FaithAlign-Gap (146 items in which only the two targets and the formal context are given), where each of the 85 undecidable items is paired with two full-context versions of opposite verdicts. Across 8 models, the best one (GPT-6 Astra) reaches 47.8% defect F1 and finds every defect in only 34.3% of defective items, although its balanced verdict accuracy is 93.1%.

Critical strengths

The paper identifies an important gap in how semantic alignment is evaluated, since item-level verdicts hide which parts of a formalization are wrong. The link types in Figure 3 (paired regions, NL-only and Lean-only, each faithful or unfaithful) are a clean and fine-grained way to separate harmless omissions such as 'Show that' from missing conditions, and the annotation could facilitate future research on finer evaluators. The sources (logic and linear algebra textbooks, Euclid's Elements, MA-Align) extend evaluation to textbook material with its surrounding context, which complements the competition-style sets that most evaluators are built on, and since the defects are found in released formalizations, FaithAlign can serve as a natural held-out test. The triplet design is also worth noticing: with full context GPT-6 Astra judges 96.9% and 100% of the two readings correctly, yet it flags only 28.2% of the undecidable items once the informal context and assumptions are removed.

Critical weaknesses
  1. It is hard to tell what the headline metric measures, since there is no human ceiling. The only reliability figure is the 81.9% link overlap between the reviewed and initial annotations (Appendix A.4), and the reviewer saw the initial version, so this is not independent agreement. Under reference grouping, Astra's joint F1 rises from 47.6 to 67.5 (Table 11), and the full-context prompt (Appendix D.1) asks models to group coherent contributions, whereas annotators narrow each defect region as much as possible (Section 4.1). That's a thin basis for reading 47.8% defect F1 as low. The benchmark idea is good, but the annotation quality needs to be documented in more detail, such as the full guideline, per-label agreement from blind double annotation on a subset, and examples where annotators disagreed.
  2. The headline result on undecidable items is hard to interpret. The undecidable-item rate (28.2% for Astra) is a recall over the 85 undecidable items with no usable precision at the item level, since an unfaithful finding overrides an undecidable one and only 12 faithful controls remain where a wrong undecidable verdict would show. The rate also credits undecidable labels placed on the wrong link: DeepSeek V3.2 reaches 25.9% at the item level but only 4.3% undecidable-link F1 (Table 8), compared with 28.2% and 17.4% for Astra. There is also no agreement figure for the undecidable label, and the informal context of the opposite-verdict counterparts is written by the authors, with 54 of these counterparts also counted as FaithAlign items (Appendix A.3). The link-level F1 or a paired success rate (undecidable on the right link and both full-context versions judged correctly) would be a better headline.
Questions for authors

Clarification

  1. As a benchmark the align-then-label target is well designed. Is there a plan to train a specialized model on this formulation, possibly covering standalone statements as well? I am curious how the authors would synthesize training data with link-level labels, such as injecting errors into faithful pairs whose links are known, or dropping the context to create undecidable cases, and if the scorer could serve as an RL reward given its sensitivity to annotation granularity (Table 11). This question matters a lot for my evaluation, and I am willing to raise my score if the authors can give a convincing answer.
  2. For one-defect items Astra locates the defect in 70.9% of cases (Figure 8), yet its balanced verdict accuracy is 93.1%. How many correct unfaithful verdicts rest on findings that match no annotated defect? Reporting defect precision and recall separately would make this clear.
  3. I'm a bit confused about how the formalizations in each source were produced. Were they written by humans or generated by models?
  4. RQ2 mentions that links involving context score 15.7 points below the overall joint F1, but this deficit is pooled over sources, while LCS-Bench has a median informal context of 13,850 characters and the other sources have under 250. Could the authors split it by source?

Future work suggestions

  • A correction would provide autoformalization pipelines with a more useful signal than a verdict, since it shows how to fix the statement, but it is hard to reward directly. It would be interesting to see if correction becomes easier when the model first produces the links, since the defect is already located.
  • Since only general-purpose models are evaluated, I would also be interested in how specialized semantic alignment evaluators such as LeanScorer or FormalRx behave on FaithAlign at the verdict level, though I understand they cannot produce links.
  • Relating the work to criterion-level LLM judging of autoformalization (Zhang et al., Beyond Gold Standards, 2025) would help the positioning.

Minor in write-up

  • Figure 2 is too hard to read. I tried a range of zoom levels and still could not make out the text, although it is the main figure that shows what an alignment looks like. A larger version with two or three items would help readers a lot.
  • The claim of the first theory-scale faithfulness benchmark could be narrowed to context-linked, span-level annotation. Table 8's full-context accuracy on the original items equals Table 1's value for the faithful versions (96.9 ± 0.7), which needs a short clarification.
5 comments

W1: “It is hard to tell what the headline metric measures, since there is no human ceiling.”

We conducted a blind re-annotation study. Two human experts who were not among the original annotators independently re-annotated a random sample of 30 FaithAlign items, proportional to each source and verdict and 10 FaithAlign-Gap items. Each annotated 15 FaithAlign items and 5 FaithAlign-Gap items without overlaps. Each expert received exactly the inputs given to the models. We score the re-annotation against the reference with the same evaluation scorer used for the models, and report GPT-6 Astra on the same items. We present here the reannotation statistics and an example of annotator disagreement. They have been added in the revised PDF in more detail.

As shown in the table below, human agreement exceeds the strongest model by a wide margin on alignment structure, defect labels, and undecidable labels.

Human agreement and GPT-6 Astra on the same items

item setlabelhumanGPT-6 Astra
FaithAlign (30 items)joint (alignment structure)74.350.3 ± 0.5
FaithAlign (30 items)defect64.252.7 ± 1.5
FaithAlign-Gap (10 items)joint75.943.5 ± 0.7
FaithAlign-Gap (10 items)defect91.640.7 ± 1.5
FaithAlign-Gap (10 items)undecidable72.319.0 ± 0.5

Example of annotator disagreement: ProofNet-Verified PN-123 (Munkres, Exercise 26.11)

  • Informal target: “Let XX be a compact Hausdorff space. Let A\mathcal{A} be a collection of closed connected subsets of XX that is simply ordered by proper inclusion. Then Y=⋂A∈AAY=\bigcap_{A \in \mathcal{A}} A is connected.”
  • Formal target: it adds the hypothesis (hAne : A.Nonempty), which the informal statement does not state.
  • Reference: marks the defect as an unmatched formal addition, with an empty informal endpoint linked to (hAne : A.Nonempty).
  • Re-annotation: marks the same defect but links it to “Let A\mathcal{A} be a collection of closed connected subsets of XX that is simply ordered by proper inclusion.”, with the formal endpoint (A : Set (Set X)) (hAne : A.Nonempty). Both identify the same defect and verdict, and differ only in whether the added hypothesis has an informal counterpart. All six uncredited re-annotated defects identify the same defect as the reference, three with a different empty endpoint and three at a coarser grain.

W2: “The headline result on undecidable items is hard to interpret. The rate also credits undecidable labels placed on the wrong link. The link-level F1 or a paired success rate would be a better headline.”

We thank the reviewer for the suggestion. Following it, we revise item-level verdict accuracy so that a correct verdict must be supported by a correct link:

  • an unfaithful or undecidable verdict counts as correct only when at least one predicted link with that label matches an annotated link with the same label;
  • a faithful verdict only when every predicted link is faithful. The revised rule applies to every verdict accuracy in the paper, including the undecidable-item rate. Under it, the undecidable-item rate of DeepSeek V3.2 falls well below that of GPT-6 Astra. We also report the suggested paired success rate alongside the undecidable-item rate. The headline numbers change as shown below.

Headline numbers under the revised verdict accuracy

metricoriginal (%)revised (%)
GPT-6 Astra, balanced verdict accuracy93.184.1
GPT-6 Astra, undecidable-item rate28.224.7
DeepSeek V3.2, undecidable-item rate25.911.4
GPT-6 Astra, both full-context versions correct96.983.9
GPT-6 Astra, paired success rate (NEW)-23.9

The revised PDF updates all affected numbers.

Q1: “ Is there a plan to train a specialized model on this formulation, possibly covering standalone statements as well? ”

We thank the reviewer for the encouraging question. Training a specialized evaluator on the align-then-label formulation is a promising direction for future work. We see the following three dimensions to address:

Specialized evaluator

  • Expert annotation might not scale to theory-scale, so a specialized evaluator would need to learn from weaker supervision and generalize beyond it [1]. In our align-then-label task, since each link is a small, locally checkable claim, it suits such supervision. FaithAlign can then serve as the human-annotated test of such generalization.
  • In terms of standalone statements, they could be regarded as the special case with empty context, so the formulation covers it without needing additional adaptation.

Training data

  • Synthesizing data could be achieved by deterministic edits such as constant, exponent, or type changes [2], or LLM error injection guided by an error taxonomy [3]. They both edit a known aligned pair at a known location, so they could be easily extended to link-level labels.
  • During our FaithAlign, especially FaithAlign-Gap construction, we have explored a way of construction via constructing item families. That could potentially be scaled up with the aid of LLMs. The procedure is as follows
    • We first identify items that are undecidable with only their targets. They are usually underspecified, hence we can produce a family of items from the same idea, e.g., sharing the informal target, each fixing a different interpretation by more specific assumptions or slight changes to statements in the context.
    • Dropping the context and assumptions of a family member with no definite defect on its target-to-target links directly yields an undecidable item.
  • Error injection would simply reduce to mixing and matching members' targets with each other's contexts and assumptions.

RL reward

  • The design of the RL reward could consider these points:
    • With the revised verdict accuracy as mentioned in the response to W2, a correct verdict earns credit only through a correct link.
    • The precision term penalizes flooding the alignment with links.
  • Sensitivity to granularity could be mitigated by prediction splitting and reference grouping, the exact effect is reported in Table 15. Hence the reward would need to combine various ways of aligning regions, depending on the granularity we desire.

  • [1]: Burns et. al., Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision -- ICML 2024
  • [2]: Lu et. al., FormalAlign: Automated Alignment Evaluation for Autoformalization -- ICLR 2025
  • [3]: Wang et. al., FormalRx: Rectify and eXamine Semantic Failures in Autoformalization -- ICML 2026

Q2: “How many correct unfaithful verdicts rest on findings that match no annotated defect? Reporting defect precision and recall separately would make this clear.”

We thank the reviewer for the question. Following the reviewer's suggestion in W2, the revised verdict accuracy no longer counts these verdicts as correct. As shown below, GPT-6 Astra's unfaithful-item accuracy falls from 94.7 to 76.8 under the revised counting, and the drop is largest on one-defect items.

We also report defect precision and recall separately. Precision exceeds recall for every model, so missed defects lower defect F1 more than false alarms do.

(NEW) Unfaithful-item verdict accuracy of GPT-6 Astra under the original and the revised counting

defective itemsitemsoriginal (%)revised (%)
one defect13393.570.9 (−22.6)
two defects7593.373.8 (−19.5)
three or more defects8897.788.3 (−9.4)
all29694.776.8 (−17.9)

(NEW) Defect precision and recall

modelprecision (%)recall (%)
GPT-5.6 Sol53.535.6
Opus 553.122.8
GPT-6 Astra59.839.8
Opus 4.860.316.9
Kimi K354.430.1
GLM-519.710.3
DeepSeek V3.226.014.6
MiniMax M2.529.44.0

Q3: “I'm a bit confused about how the formalizations in each source were produced. Were they written by humans or generated by models?”

The formalizations are obtained from the source datasets. As described in Section 4.1, we use the informal–formal pairs provided by each source as targets, and for LCS-Bench and LeanEuclid, the formalizations were provided directly by their authors. Each formalization was therefore produced by its source's own construction process, as described in the cited source papers.

Q4: “RQ2 mentions that links involving context score 15.7 points below the overall joint F1, but this deficit is pooled over sources. Could the authors split it by source?”

We thank the reviewer for the suggestion. We split the comparison in RQ2 by source below. We have added the per-source breakdown to Appendix C.3 as Table 11.

Each role column gives the joint F1 of links with that role minus the overall joint F1 of the same source, in percentage points, and the context-links column counts the annotated context links. Following Section 3.2 Regions and links, a link is implicit if an endpoint lies in the assumptions II, context if it has no endpoint in II but one in the informal or formal context, and target otherwise.

Joint F1 of each link role relative to the overall joint F1 of each source, pooled over models and repetitions

sourcecontext linksoverall joint F1 (%)target (pp)context (pp)implicit (pp)
LCS-Bench69041.2+9.9−20.1−15.1
ProofNet-Verified4250.4+2.7−11.6−8.1
MA-Align17429.0+1.0−8.1+3.6
LeanEuclid–Pistis032.0−2.0–+27.9
All90637.6+4.1−15.7−2.5

M1: “Figure 2 is too hard to read.”

We have replaced Figure 2 with a larger version that shows three representative items.

M2: “The claim of the first theory-scale faithfulness benchmark could be narrowed to context-linked, span-level annotation.”

We haved narrowed the claim to the first theory-scale faithfulness benchmark with context-linked, span-level annotation.

M3: “Table 8's full-context accuracy on the original items equals Table 1's value for the faithful versions (96.9 ± 0.7).”

Table 12 (Table 8 in the submission) reports accuracy on the 85 original items, consisting of 72 faithful and 13 unfaithful, while Full F in Table 1 reports accuracy on the 85 faithful versions, consisting of the 72 faithful originals and the 13 faithful counterparts we wrote for the unfaithful originals. Both sets share the 72 faithful originals, where all of Astra's full-context errors occur, and Astra judges the 13 remaining items in each set correctly, so the two values coincide.

Following the reviewer's suggestion in W2, the revised verdict accuracy additionally requires a correct unfaithful verdict to rest on a matched unfaithful link, which applies to the 13 unfaithful originals in Table 12 but not to Full F in Table 1, so the two values now differ slightly in the revised PDF.

S1: “It would be interesting to see if correction becomes easier when the model first produces the links, since the defect is already located.”

We thank the reviewer for the insightful suggestion. We agree that this is a very promising direction. Correction guided by the links is a natural next step, and it is part of our plan for future work.

S2: “Since only general-purpose models are evaluated, I would also be interested in how specialized semantic alignment evaluators behave on FaithAlign at the verdict level.”

We thank the reviewer for the suggestion. We run LeanScorer with its released implementation and threshold α = 0.6 on a random quarter of FaithAlign (115 items, proportional to each source and verdict), giving it only the informal and formal targets. Since LeanScorer produces no links, its verdicts are scored alone, while align-then-label uses the revised verdict accuracy from W2, which also requires a matched link. As shown below, with GPT-6 Astra, align-then-label is more accurate than LeanScorer despite the stricter counting, while with DeepSeek V3.2 it is less accurate.

(NEW) Balanced verdict accuracy on a random quarter of FaithAlign (115 items)

backboneLeanScorer (%)align-then-label, 3 runs (%)
GPT-6 Astra71.587.2 ± 2.0
DeepSeek V3.261.254.1 ± 2.3

S3: “Relating the work to criterion-level LLM judging of autoformalization (Zhang et al., Beyond Gold Standards, 2025) would help the positioning.”

We thank the reviewer for the suggestion. We have included the work in our revision.

Reviewer_9mf2OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces FaithAlign, a benchmark for checking whether Lean formalizations preserve the meaning of mathematical text. Models match parts of the text and context to Lean code, then label each match. FaithAlign contains 460 items and 4,982 annotated links; FaithAlign-Gap tests 146 items with context withheld. Across eight models, the strongest reaches 47.8% defect F1 and recognizes only 28.2% of undecidable cases.

Critical strengths
  • Matching individual parts exposes missing assumptions and context mismatches that a single faithful or unfaithful verdict can hide. Including an undecidable label also tests whether models recognize insufficient information.

  • Expert review, repeated evaluations, and alternative scoring provide useful diagnostics. Comparing the same statements under different contexts helps distinguish judging an interpretation from recognizing that context is missing. Defect detection remains difficult even when broader matches receive credit.

Critical weaknesses
  • Independent annotation agreement is not established. The reported 81.9% overlap compares annotations before and after review, rather than separately produced annotations.

  • The missing-context result needs a prompt check. The prompt treats an “unsupported restriction” as unfaithful, while some undecidable cases contain restrictions that could be justified by withheld context. Although the prompt also defines “undecidable,” this boundary may be unclear.

Questions for authors
  • Can you report blind independent agreement on a representative subset, separately for alignment structure, defect labels, and undecidable labels?

  • How do results change when the prompt explicitly distinguishes definite defects from conditions that could be justified by missing context?

2 comments

Q1 (W1): “Can you report blind independent agreement on a representative subset, separately for alignment structure, defect labels, and undecidable labels?”

We conducted a blind re-annotation study. Two human experts independently re-annotated a random sample of 30 FaithAlign items, proportional to each source and verdict and 10 FaithAlign-Gap items. Each annotated 15 FaithAlign items and 5 FaithAlign-Gap items without overlaps. Each expert received exactly the inputs given to the models. We score the re-annotation against the reference with the same evaluation scorer used for the models, and report GPT-6 Astra on the same items.

As shown below, human agreement exceeds the strongest model by a wide margin on alignment structure, defect labels, and undecidable labels.

Human agreement and GPT-6 Astra on the same items

item setlabelhumanGPT-6 Astra
FaithAlign (30 items)joint (alignment structure)74.350.3 ± 0.5
FaithAlign (30 items)defect64.252.7 ± 1.5
FaithAlign-Gap (10 items)joint75.943.5 ± 0.7
FaithAlign-Gap (10 items)defect91.640.7 ± 1.5
FaithAlign-Gap (10 items)undecidable72.319.0 ± 0.5

We have added the study, its protocol, and the sampled items to the revision.

Q2 (W2): “How do results change when the prompt explicitly distinguishes definite defects from conditions that could be justified by missing context?”

We thank the reviewer for the careful reading. Following the suggestion, we rerun FaithAlign-Gap with a revised context-omitted prompt that replaces “an unsupported restriction” in the definition of unfaithful with “a restriction not specified by the target that no reasonable assumption could justify.” As shown below, the revised prompt changes the undecidable-item rate substantially only for GPT-6 Astra, and undecidable-link F1 remains low for all models.

We also note that many of GPT-6 Astra’s errors on undecidable items are faithful verdicts as shown in Table 1, where it commits to the permissive reading, which the “unsupported restriction” wording cannot cause.

GPT-5.6 Sol, Opus 5, GPT-6 Astra, and Opus 4.8 on FaithAlign-Gap under the original and the revised prompt

modelpromptundecidable-item rate (%)undecidable-link F1 (%)unfaithful-control accuracy (%)
GPT-5.6 Soloriginal12.511.571.4
GPT-5.6 Solrevised16.5 (+4.0)10.5 (−1.0)75.5 (+4.1)
Opus 5original1.21.754.4
Opus 5revised2.4 (+1.2)1.5 (−0.2)57.1 (+2.7)
GPT-6 Astraoriginal24.717.474.8
GPT-6 Astrarevised41.2 (+16.5)24.2 (+6.8)73.5 (−1.3)
Opus 4.8original0.40.448.3
Opus 4.8revised0.0 (−0.4)0.0 (−0.4)49.0 (+0.7)

Additionally, following Reviewer L2SU's suggestion, we revise item-level verdict accuracy so that a correct verdict must be supported by at least one correctly matched link. The numbers above use this revised counting.

Dear Reviewer 9mf2,

Thank you again for your thoughtful review. As the discussion period is nearing its end, we wanted to follow up on our response. We added blind independent agreement by label type, and a rerun with a revised prompt under which undecidable-link F1 remains low. We hope these address your concerns.

If any concern remains, we would be grateful for the opportunity to clarify it before the discussion period closes.

Thank you for your time and effort in reviewing our paper.

Reviewer_Ao8oOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces FaithAlign, a context-aware benchmark that aligns informal mathematical expressions with Lean formalizations and labels their faithfulness. FaithAlign-Gap additionally tests whether models recognize cases that cannot be judged without missing context. The results expose substantial gaps in defect localization and uncertainty recognition. For example, the best model gets most verdicts right but reaches only 47.8% defect F1.

Critical strengths
  • Phrase-level links and explicit contextual assumptions make faithfulness judgments more informative than a single statement-level label.
  • The FaithAlign-Gap triplets provide convincing evidence that models often miss underspecification even when they can judge the explicit interpretations correctly.
Critical weaknesses
  • Annotation reliability is unclear. The paper does not report independent inter-annotator agreement or a human performance ceiling. This matters for a task with many possible phrase boundaries and alignment choices.
  • Sensitivity to matching conventions. Models often produce coarser links than the reference annotations. Strict one-to-one matching could penalize reasonable alternative segmentations or contextual anchors. The reported defect F1 should therefore be accompanied by more relaxed matching analyses. This is a possible evaluation confound, not proof that the model errors are merely artifacts.
Questions for authors
  1. Can you report independent human agreement using the same scoring protocol?
  2. How much do defect F1 and all-defects accuracy change with relaxed span or endpoint matching?
  3. Could you manually inspect unmatched predicted defects to distinguish genuine misses from valid alternative alignments?
2 comments

Q1 (W1): “Can you report independent human agreement using the same scoring protocol?”

We conducted a blind re-annotation study. Two human experts independently re-annotated a random sample of 30 FaithAlign items, proportional to each source and verdict and 10 FaithAlign-Gap items. Each annotated 15 FaithAlign items and 5 FaithAlign-Gap items without overlaps. Each expert received exactly the inputs given to the models. We score the re-annotation against the reference with the same evaluation scorer used for the models, and report GPT-6 Astra on the same items.

As shown below, human agreement exceeds the strongest model by a wide margin on alignment structure, defect labels, and undecidable labels.

Human agreement and GPT-6 Astra on the same items

item setlabelhumanGPT-6 Astra
FaithAlign (30 items)joint (alignment structure)74.350.3 ± 0.5
FaithAlign (30 items)defect64.252.7 ± 1.5
FaithAlign-Gap (10 items)joint75.943.5 ± 0.7
FaithAlign-Gap (10 items)defect91.640.7 ± 1.5
FaithAlign-Gap (10 items)undecidable72.319.0 ± 0.5

We have added the study, its protocol, and the sampled items to the revision.

Q2 (W2): “How much do defect F1 and all-defects accuracy change with relaxed span or endpoint matching?”

We thank the reviewer for the valid concern and suggestion. Our primary scorer credits partial span overlap at each endpoint, and the all-defects rate requires only a positive overlap with each annotated defect. In appendix we also provid relaxed matching results in Table 15 (Table 11 in the submission), consisting of prediction splitting and reference grouping. Prediction splitting allows several predictions to describe one annotated link, and reference grouping allows one coarse prediction to describe several annotated links. Under reference grouping, defect identification remains low for all models.

We additionally provide new scores by further relaxing the remaining matching constraints. First, we report the all-defects rate under reference grouping for all models. Second, we relax endpoint matching, so that a predicted defect is also credited when it selects an endpoint that the annotation leaves empty, as long as it overlaps every endpoint that the annotation fills.

Defect F1 and all-defects rate under standard scoring, reference grouping, and relaxed endpoint matching

modeldefect F1 (%)defect F1, grouping (%)defect F1, relaxed endpoint (%) (NEW)all-defects (%)all-defects, grouping (%) (NEW)all-defects, relaxed endpoint (%) (NEW)
GPT-5.6 Sol42.749.3 (+6.6)44.6 (+1.9)30.934.8 (+3.9)33.0 (+2.1)
Opus 531.935.7 (+3.8)32.9 (+1.0)22.224.3 (+2.1)22.6 (+0.4)
GPT-6 Astra47.855.8 (+8.0)48.7 (+0.9)34.339.2 (+4.9)35.4 (+1.1)
Opus 4.826.429.7 (+3.3)27.0 (+0.6)19.420.8 (+1.4)20.2 (+0.8)
Kimi K338.745.1 (+6.4)39.3 (+0.6)28.233.2 (+5.0)29.4 (+1.2)
GLM-513.517.0 (+3.5)13.9 (+0.4)11.413.0 (+1.6)12.0 (+0.6)
DeepSeek V3.218.722.6 (+3.9)18.5 (−0.2)13.315.2 (+1.9)13.5 (+0.2)
MiniMax M2.57.110.0 (+2.9)7.0 (−0.1)7.38.4 (+1.1)7.7 (+0.4)

Q3: “Could you manually inspect unmatched predicted defects to distinguish genuine misses from valid alternative alignments?”

We thank the reviewer for the suggestion, we have included the comparison in detail in the revision. We inspect some unmatched predicted defects, and here we present the items in Appendix C.4.1 and identify both cases:

  • Valid alternative alignment (MA-072). The annotation links “n×nn\times n matrix,” to ℝ as unfaithful. DeepSeek V3.2 instead flags the coarser link from “Let NN be a strictly upper triangular n×nn\times n matrix” to (n : ℕ) (N : Matrix (Fin n) (Fin n) ℝ), which contains the defect and is credited under reference grouping in Table 15.
  • Genuine miss (isosceles triangles). On this faithful item, GLM-5 flags “For isosceles triangles, the angles at the base are equal to one another, …” as missing, although the formalization states it as (∠ a:b:c = ∠ a:c:b) ∧ (∠ c:b:d = ∠ b:c:e).

Dear Reviewer Ao8o,

Thank you again for your careful review. As the discussion period is nearing its end, we wanted to follow up on our response. We added a blind re-annotation study by two independent experts, along with relaxed matching results and a manual inspection of unmatched predicted defects. We hope these address your concerns about annotation reliability and matching sensitivity.

If any concern remains, we would be grateful for the opportunity to clarify it before the discussion period closes.

Thank you for your time and effort in reviewing our paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their thoughtful and constructive feedback. We are encouraged that the reviewers found that phrase-level links and explicit context make faithfulness judgments more informative than a single statement-level verdict (Ao8o, 9mf2, L2SU), regarded the align-then-label target as well designed and the link types as a clean, fine-grained way to separate harmless omissions from missing conditions (L2SU), and found the FaithAlign-Gap triplets convincing evidence that models miss underspecification even when they judge the explicit interpretations correctly (Ao8o, 9mf2, L2SU). Reviewer L2SU also noted that the textbook sources extend evaluation beyond competition-style sets and that FaithAlign can serve as a natural held-out test.


Response to the Key Concerns

  • Annotation reliability and human ceiling (Ao8o, 9mf2, L2SU). We conducted a blind re-annotation study. Two experts who were not among the original annotators, and had no access to the prior annotations, re-annotated a random sample of FaithAlign and FaithAlign-Gap items, and we scored their annotations with the same scorer. Their agreement with the reference exceeds the strongest model on alignment structure, defect labels, and undecidable labels. Every uncredited re-annotated defect identifies the same defect as the reference and differs only in its anchoring.
  • Sensitivity to matching conventions (Ao8o, L2SU). We report relaxed endpoint matching and the all-defects rate under reference grouping for all models, and defect identification remains low under both. We also provide examples of manual inspection of unmatched predicted defects.
  • Prompt check for missing context (9mf2). We reran FaithAlign-Gap on four models with a revised prompt that separates definite defects from restrictions a reasonable assumption could justify. The undecidable-item rate rises only for GPT-6 Astra, and undecidable-link F1 remains low for all models.
  • Verdicts without located defects (L2SU). The revised verdict accuracy no longer counts correct unfaithful/undecidable verdicts that match no annotated defect. We also report defect precision and recall separately, where precision exceeds recall for every model.
  • Context deficit by source and specialized evaluators (L2SU). We added the per-source breakdown. Context links score below the overall joint F1 in every source. We also compared with LeanScorer on a random quarter of FaithAlign. With GPT-6 Astra, align-then-label verdicts are more accurate than LeanScorer.
  • Plan for training a specialized evaluator (L2SU). We outlined how link-level training data can be synthesized from the item families used to construct FaithAlign-Gap, together with deterministic and LLM-based error injection, and how the scorer could serve as an RL reward.

All revisions are marked in blue in the revised PDF.


We would like to thank all reviewers for their valuable time and effort in reviewing our manuscript; their suggestions were instrumental in helping us improve our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.