acceptodds
Under review as a conference paper at ICLR 2027

Beyond Field F1: Concept-Binding Accuracy and the Hindsight Benchmark for Document AI

Abstract

Document AI systems often extract text correctly but assign it to the wrong semantic field, for example, reading “$4,523.67” correctly from a paystub but labeling it net pay instead of gross pay. Value-only matching, which we term Value-Only Recall (Recval), credits a ground-truth value if it appears anywhere in the output, irrespective of its concept key, and therefore misses this failure. We introduce Concept-Binding Accuracy (CBA), a metric family that decomposes joint key-value error into extraction and binding components, and Hindsight, a benchmark of nine datasets across five document domains with hierarchical concept ontologies. For Claude Haiku 4.5 and GPT-4o-mini, Recval overstates joint key-value accuracy by up to 33.3 percentage points: on commercial contracts, GPT-4o-mini reaches 100% Recval but only 66.7% CBA, with 250 misbindings across 50 documents. Even fixed-layout IRS W-2 forms yield 194 (Haiku) and 115 (GPT-4o-mini) misbindings when 44 same-type fields compete, whereas low-ambiguity domains show nearzero gaps. We identify two drivers, concept density and semantic overlap, and confirm both as independent causal effects in a controlled 5 × 4 factorial study on W-2: overlap alone moves binding accuracy by 25 percentage points, and density alone by up to 16 percentage points. An open-weights model (Qwen2-VL-7B) shows the same pattern, with markedly worse binding on the highest-ambiguity domain (CUAD) and comparable binding on low-ambiguity ones. We release all ontologies, scoring code, and experiment notebooks.

Primary area:
datasets and benchmarks
Keywords:
Document AI, information extraction, concept-binding accuracy, semantic misbinding, document understanding, field-level F1, hierarchical concept ontologies, structured document parsing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VnK7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This draft claims that the metrics used for value-only studies only considers if the targeted value is extracted from a document, but they omit if the value is associated with the correct key. Thus, the previously used Value-only recall rate (Rec_val) did not penalize the wrong binding(the value appears in the ground-truth but matched to a wrong key.)

To solve this problem, this draft proposes 1: a new metric, CONCEPT-BINDING ACCURACY, which penalizes this key-value mismatching error. 2: a new benchmark for this key-value extraction which consists of previously published datasets.

The proposed benchmark and metric are claimed to better evaluate information extraction methods.

Critical strengths
  1. The overall writing of this draft is clear, the proposed idea is easy to follow.
  2. Around 250 synthetic data samples are added for schema-driven information extraction.
Critical weaknesses

There exist several key problems in this current draft.

  1. This paper mainly compares their proposed “new” metric against value-only recall rate, this could be a scope mismatch in computer science. Those datasets are proposed or used for value-only studies on purpose, in which only the coverage of the retrieved information or value is important. When taking key-value into account, there exists a long history in schema IE studies, which will also penalize the wrong “binding” scenario. For instance, it is less reasonable to criticize the metric of accuracy used for image classification that omits pixel-level information, then rename a metric to compare against the accuracy for image segmentation.

  2. Line 68, “None isolates the case in which the correct value is attached to the wrong concept.” This claim seems to be wrong to me, if most of those datasets are proposed for value-only studies, then this study should compare their proposed method against schema driven IE tasks. Further, most of the listed datasets could be used for schema-driven extraction, e.g., the mismatched key-value pairs from SROIE, CORD v2 will also be penalized.

  3. Another main issue in this draft is that the proposed CBA metric seems to be a renamed existing metric in statistics. If my understanding is correct, Section 3 is explaining the proposed CONCEPT-BINDING ACCURACY. Based on the context, Eq(1) is the recall rate, how many correct values are retrieved out of the ground-truth, so called Value-Only Recall. Then Eq(2) introduces the Conditional Concept-Binding Accuracy, among those retrieved correct values, how many of them are truly correct, the keys are also matched. This can be confirmed by Figure 1 right, 3 values are matched(retrieved), only one of them the key is also correct, so CBA=33.3%. In basic statistics, we normally consider this CBA as Precision. Then Eq(3) is essentially Recall * Precision, which is the numerator of F1. That is the proposed new metric is essentially precision we commonly used in statistics.

  4. The newly defined delta Eq(6), is a customized combination of those basic statistics terms, it is an overlap of existing statistics measures.

  5. The strict or soft matching is a pre-defined scoring rule, which is essentially the same as exact matching, partial matching and fuzzy matching.

  6. Section 3.2, this study admits schema-aware F1 can already detect the key mismatch, but the Fig 1. Left still uses standard F1 to emphasize the difference, this seems to be not right.

  7. The claim contribution of a new benchmark, which is essentially recomputing basic statistics over previously published datasets. The only newly added data is about 200 newly created paystubs and tax forms. This can hardly be considered a solid contribution.

Questions for authors

Please refer to the above section.

3 comments
AuthorsOpenReview ↗

Response to reviewer VnK7

We thank the reviewer for the careful evaluation, constructive criticism and insightful questions.

CW-01: Scope

Reviewer Concern

This paper mainly compares their proposed “new” metric against value-only recall rate, this could be a scope mismatch in computer science. Those datasets are proposed or used for value-only studies on purpose, in which only the coverage of the retrieved information or value is important. When taking key-value into account, there exists a long history in schema IE studies, which will also penalize the wrong “binding” scenario. For instance, it is less reasonable to criticize the metric of accuracy used for image classification that omits pixel-level information, then rename a metric to compare against the accuracy for image segmentation.

Response

Value-only recall is legitimate for value-only tasks. We will frame CBA as a schema-extraction diagnostic alongside joint F1 and compare identical tasks/outputs.

CW-02: Schema evaluation

Reviewer Concern

Line 68, “None isolates the case in which the correct value is attached to the wrong concept.” This claim seems to be wrong to me, if most of those datasets are proposed for value-only studies, then this study should compare their proposed method against schema driven IE tasks. Further, most of the listed datasets could be used for schema-driven extraction, e.g., the mismatched key-value pairs from SROIE, CORD v2 will also be penalized.

Response

The claim is too broad. SROIE Task 3 requires content and category agreement (Huang et al., Sec. III-C); CORD has structured labels. We will replace the claim and compare native schema-aware evaluators.

CW-03: Precision

Reviewer Concern

Another main issue in this draft is that the proposed CBA metric seems to be a renamed existing metric in statistics. If my understanding is correct, Section 3 is explaining the proposed CONCEPT-BINDING ACCURACY. Based on the context, Eq(1) is the recall rate, how many correct values are retrieved out of the ground-truth, so called Value-Only Recall. Then Eq(2) introduces the Conditional Concept-Binding Accuracy, among those retrieved correct values, how many of them are truly correct, the keys are also matched. This can be confirmed by Figure 1 right, 3 values are matched(retrieved), only one of them the key is also correct, so CBA=33.3%. In basic statistics, we normally consider this CBA as Precision. Then Eq(3) is essentially Recall * Precision, which is the numerator of F1. That is the proposed new metric is essentially precision we commonly used in statistics.

Response

Let G=gold items, M=predictions, R=one-to-one value matches, J=joint matches. CBA=J/R; joint precision=J/M. They coincide when M=R (Figure 1), not generally: G=3,M=4,R=3,J=1 gives CBA=1/3, precision=1/4. Rec_val*CBA=J/G is joint recall; F1=2J/(G+M). We will clarify this conditional-precision interpretation, without claiming a new statistical primitive.

CW-04: Delta

Reviewer Concern

The newly defined delta Eq(6), is a customized combination of those basic statistics terms, it is an overlap of existing statistics measures.

Response

Delta=(R-J)/G is a derived misbinding rate, not an independent statistical invention. We will present it as an error-category diagnostic and test controlled errors alongside joint F1.

CW-05: Scoring rules

Reviewer Concern

The strict or soft matching is a pre-defined scoring rule, which is essentially the same as exact matching, partial matching and fuzzy matching.

Response

Strict uses exact concept equality after value matching; soft gives predefined family-based partial credit, not fuzzy value matching. We will acknowledge these established ideas; soft credit is ontology-relative, not calibrated severity.

CW-06: Figure 1

Reviewer Concern

Section 3.2, this study admits schema-aware F1 can already detect the key mismatch, but the Fig 1. Left still uses standard F1 to emphasize the difference, this seems to be not right.

Response

We will correct Figure 1 and the F1 framing. For G=M=3 and J=1, schema-aware precision, recall and F1 are 1/3; 100% is value-only recall.

CW-07: Benchmark

Reviewer Concern

The claim contribution of a new benchmark, which is essentially recomputing basic statistics over previously published datasets. The only newly added data is about 200 newly created paystubs and tax forms. This can hardly be considered a solid contribution.

Response

Most documents are reused. The manuscript reports 200 generated paystubs plus 50 W-2 forms, with 50 of each evaluated; count alone does not establish novelty. We will audit inherited versus added annotations, mappings, protocols and software, check reproducibility and diagnostic value against native metrics, and narrow claims accordingly.

Questions for Authors: "Please refer to the above section." Addressed in CW-01–CW-07 above.

Reviewer_VnK7OpenReview ↗

The core idea is still lame

I appreciate the authors respond each question separately and accordingly. Unfortunately, I don't think they resolved my concerns. I will point one thing here about the main concept,

"Let G=gold items, M=predictions, R=one-to-one value matches, J=joint matches. CBA=J/R;" If we formulate the "retrieved" pool as those one-to-one value matches(R), then we wonder how many of the "retrieved" are true postives, joint matches J, then this is essentially PRECISION(numerator) in statistics. It is simply how we formulate the problem, I could be wrong, but I could not see why this is a new statistics to publish.

We thank the reviewer again. The initial rebuttal was already posted; this note reports the evidence now completed and exactly where it lives in the revised manuscript. Color legend: changes addressing VnK7 are purple, and 2+ reviewers orange.

CW-03: "CBA is just a renamed existing metric (precision)"

We ran the exact counterexample through our scorer: G=3,M=4,R=3,J=1 gives CBA=1/3 but joint precision=1/4, genuinely different. Removing the spurious 4th prediction collapses M=R=3, and all four quantities become 1/3, exactly our Figure 1 example. CBA is conditional precision restricted to the value-matched subset, an established construction, not a new primitive. Locator: §3.2 (p.4, orange — shared with mF6u's QA-02, since the same CUAD/LEDGAR numbers answer both).

CW-04: Delta "is a customized combination of existing statistics"

A matched-pair construction, holding G, M, J identical by design:

ConditionGMRJRec_valCBA_condΔ
Value-error (values replaced)383818180.4741.0000.000
Key-misbinding (values transposed)383838181.0000.4740.526

Joint Precision and F1_joint are identical (0.474) in both rows by construction (they depend only on G, M, J); Rec_val/CBA_cond/Δ, whose denominator R we deliberately varied, differ completely and correctly. Locator: Table 1, §3.2 (p.4, orange).

CW-01/CW-02: scope and schema evaluation

Verified against primary sources, not inferred from dataset names: SROIE's official Task 3 protocol states "marked as correct if both submitted content and category... matches the groundtruth" (exact quote, fetched live); the widely-used Donut reference evaluator for CORD matches (schema-path, value) tuples. Both already penalize wrong-key bindings. We replaced the overbroad "None isolates the case..." claim accordingly and corrected a citation (a scene-text-recognition paper that does not define any field-value-pair evaluation protocol) that had been attached to the wrong claim. Locator: §2 Related Work (p.2, purple).

CW-06: Figure 1

The left panel was relabeled (prior response) to "Value-Only Recall (Rec_val) Scoring." This response additionally adds, to the right panel: "CBA = Joint F1 = 33.3%", so the figure itself now shows that true schema-aware F1 already equals CBA in this example. Locator: Figure 1 (p.3).

CW-07: benchmark contribution

200 newly generated paystubs (50 evaluated) are genuine and verified-reproducible: we regenerated 5 from the released seed=42 generator code and all 5 matched the saved ground truth exactly. The 50 evaluated W-2 forms are not generated by us, they are drawn from a pre-existing public dataset; the manuscript now states this explicitly and cites the source. Locator: Appendix A, "W-2 Tax Forms" paragraph (p.22, purple). We also found (and report, not hide) that our own paystub manifest.json disagrees with the per-document ground-truth files on rendering metadata for 92% of documents, though field values agree exactly on all checked documents.

CW-05: scoring rules

Confirmed: strict/soft scoring is established conditional-accuracy-plus-partial-credit, not fuzzy value matching or an invention; no manuscript text change was needed beyond what §3.1/Appendix B already state.

What remains unresolved

Native SROIE/CORD evaluator adapter engineering (comparing our saved outputs against the official/reference scorers end-to-end) was not completed; a frozen design is provided rather than a fabricated result.

Reviewer_S9YGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Concept-Binding Accuracy (CBA), an evaluation framework for structured document extraction that separates two sources of error: failure to recover the correct value and failure to assign a recovered value to the correct semantic field. It further presents Hindsight, a benchmark spanning nine datasets across five document domains, augmented with hierarchical concept ontologies. Using several vision-language models, the paper shows that value-only recall can substantially overestimate system reliability when correctly extracted values are assigned to incorrect fields. It also analyzes concept density and semantic overlap as factors contributing to such misbindings.

Critical strengths
  1. The paper targets a practically important failure in structured document extraction: values are read correctly but assigned to the wrong field, which passes type and format checks in automated pipelines. CBA decomposes joint accuracy into value recovery and conditional concept binding, so a system's failures can be attributed to reading or to binding without an LLM judge.

  2. The contribution goes beyond a metric. Hindsight covers nine datasets across five document domains with hierarchical ontologies, and the controlled 5×4 factorial study on W-2 isolates concept overlap and concept density as separate causes of misbinding. The ontologies, scoring code, and experiment notebooks are released.

Critical weaknesses
  1. The motivation relies on comparison with value-only recall, a weak baseline that already misses misbinding by construction. Schema-aware key-value F1 already penalizes misbinding, and ACCjointACC_{joint} is exactly its recall term, so CBA introduces no new notion of correctness. The paper does not show that the decomposition leads to different rankings, conclusions, or fixes than schema-aware F1 and per-field analysis.

  2. Unquantified sensitivity to design choices. CBA values depend on several choices whose effects are not systematically quantified: the domain-specific value normalization, the author-defined ontology families and 0.5 partial-credit weight in CBA-soft (which the authors note were not externally validated or pre-registered), and the output prompt format, which differs across domains in the main comparison yet is shown in Table 4 to shift CBAcondCBA_{cond} by up to about 14 points. It is therefore unclear how much of the cross-domain differences in Table 2 reflects domain difficulty rather than these choices.

Questions for authors
  • Q1. [Added value over F1] Is there any dataset or model pair where CBA changes a model ranking or deployment decision compared with F1jointF1_{joint} or per-field confusion analysis? The ranking differences in Sec. 4.3 are shown against RecvalRec_{val}, a weak baseline, whereas ACCjointACC_{joint}, the recall term of schema-aware F1, already separates the two models on CUAD (74.5% vs. 66.7%).

  • Q2. [Ontology sensitivity] How do CBA-soft and the soft–strict gap change under alternative or independently designed ontology families and partial-credit weights other than 0.5? Both the families and the 0.5 weight are author-defined, yet the paper gives no evidence of how stable the results are to these choices.

  • Q3. [Validity of rule-based judgments] Have the authors validated the flagged misbindings against human annotation on a sample, e.g., to separate genuine binding errors from normalization failures or label ambiguity? Misbinding counts depend on rule-based normalization and exact concept matching, and the authors themselves acknowledge label disagreement in CUAD. An LLM judge could also serve as a cheaper proxy for this validation, or as an alternative to hand-built ontology families in CBA-soft.

  • [Minor] Please spell out "Concept-Binding Accuracy (CBA)" at its first use in the main text. "CBA" appears in the Introduction (line 40) without expansion, although it is expanded in the abstract.

3 comments
AuthorsOpenReview ↗

Response to reviewer S9YG

We thank the reviewer for the careful evaluation, constructive criticism and insightful questions.

CW-01: Added value

Reviewer Concern

The motivation relies on comparison with value-only recall, a weak baseline that already misses misbinding by construction. Schema-aware key-value F1 already penalizes misbinding, and is exactly its recall term, so CBA introduces no new notion of correctness. The paper does not show that the decomposition leads to different rankings, conclusions, or fixes than schema-aware F1 and per-field analysis.

Response

We agree that schema-aware F1 penalizes misbinding. ACCjointACC_{joint}, rather than CBAcondCBA_{cond} in general, is its recall term (Sec. 3.2). Our contribution is a recovery-conditioned diagnostic and benchmark, not a new correctness criterion. We will correct the F1 framing and compare against joint F1 and recovery-aware per-field diagnostics; a sufficiently detailed confusion analysis can recover the same information.

CW-02: Design sensitivity

Reviewer Concern

Unquantified sensitivity to design choices. CBA values depend on several choices whose effects are not systematically quantified: the domain-specific value normalization, the author-defined ontology families and 0.5 partial-credit weight in CBA-soft (which the authors note were not externally validated or pre-registered), and the output prompt format, which differs across domains in the main comparison yet is shown in Table 4 to shift by up to about 14 points. It is therefore unclear how much of the cross-domain differences in Table 2 reflects domain difficulty rather than these choices.

Response

Table 2 reports strict CBA: changing family groupings or the soft weight alone cannot change it when concept keys, matching and outputs are fixed. Normalization and prompts can affect strict scores. Table 4 already shows prompt sensitivity, not invariance. We will quantify normalization, family/weight and matched-prompt sensitivity, including changes in domain/model ordering, and qualify cross-domain claims.

QA-01: Rankings and decisions

Reviewer Concern

[Added value over F1] Is there any dataset or model pair where CBA changes a model ranking or deployment decision compared with or per-field confusion analysis? The ranking differences in Sec. 4.3 are shown against , a weak baseline, whereas , the recall term of schema-aware F1, already separates the two models on CUAD (74.5% vs. 66.7%).

Response

CUAD does not establish added ranking information: with full recovery, CBAcond=ACCjointCBA_{cond}=ACC_{joint}. W-2 does show a trade-off: Qwen has higher CBA than Haiku (85.5% vs 75.3%) but lower value recall (23.9% vs 41.3%; Sec. 4.3). This is not a verified F1 reversal or deployment benefit. We will report precision, joint F1, paired ranking uncertainty and retrospective selection analyses with prespecified coverage/precision constraints, alongside per-field diagnostics.

QA-02: Ontology and weights

Reviewer Concern

[Ontology sensitivity] How do CBA-soft and the soft–strict gap change under alternative or independently designed ontology families and partial-credit weights other than 0.5? Both the families and the 0.5 weight are author-defined, yet the paper gives no evidence of how stable the results are to these choices.

Response

We have not established stability under alternative families or weights. For fixed matches, CBAsoft(w)=CBAstrict+wW/RCBA_{soft(w)}=CBA_{strict}+wW/R, where RR is recovered items and WW is within-family misbindings. We will sweep w=0,0.25,0.5,0.75,1 and compare the original, independently designed and controlled alternative families, reporting score gaps and rank changes. Soft credit remains ontology-relative, not validated error severity.

QA-03: Human validation

Reviewer Concern

[Validity of rule-based judgments] Have the authors validated the flagged misbindings against human annotation on a sample, e.g., to separate genuine binding errors from normalization failures or label ambiguity? Misbinding counts depend on rule-based normalization and exact concept matching, and the authors themselves acknowledge label disagreement in CUAD. An LLM judge could also serve as a cheaper proxy for this validation, or as an alternative to hand-built ontology families in CBA-soft.

Response

The manuscript does not report the requested human-validation study. We will audit a stratified sample of flagged errors and unflagged controls with independent human judgments, distinguishing genuine misbinding, normalization artifacts, ambiguous/multiple valid labels and reference errors. We will report agreement, weighted flag-validity estimates and uncertainty. If feasible, an LLM judge and LLM-proposed families will be assessed against held-out human judgments, not treated as ground truth.

Minor clarification. We will expand "Concept-Binding Accuracy (CBA)" at its first occurrence in the Introduction.

We thank the reviewer again. The initial rebuttal was already posted; this note reports the evidence now completed and exactly where it lives in the revised manuscript. Color legend: changes addressing S9YG are teal, and 2+ reviewers orange.

CW-01/QA-01: "Is there any dataset/model pair where CBA changes a model-selection decision versus F1_joint?"

We applied a prespecified protocol (coverage floors {.25,.50,.75} x precision floors {.75,.90,.95}) across all nine domains, comparing "pick the eligible model with the highest F1_joint" against "pick by highest CBA_cond."

CellsCount
Total (domain x coverage x precision)81
No eligible model73
Two eligible models (disagreement possible)5
Disagreements between F1 rule and CBA rule0

We did not find an instance where CBA_cond changes a selection decision F1_joint would not have already made. In this implementation, CBA_cond is computed from the same per-field ledger a confusion-matrix analysis would use, so this rule and a confusion-matrix-derived rule are the same computation, not two independently-agreeing ones. Locator: new sentence, §4.3 "Model Comparison and Open-Weights Generalization" (p.7, teal); full protocol, Appendix C.6 "Does CBA Change Model Selection Relative to Joint F1?" (p.18, teal).

CW-02/QA-02: family/weight sensitivity

Confirmed by direct code citation (no family lookup anywhere in the matching cost function) and 600 random-family-permutation re-scores, all exactly invariant: strict scores cannot change under family/weight choice. A genuine instability was found and is now reported: LEDGAR's soft ranking crosses at w≈0.697, so w=0.75/1.0 (two of our own five stated sweep points) favor GPT-4o-mini while w≤0.5 favors Haiku; W-2 and CUAD's soft rankings are stable across the full [0,1] range. Locator: Appendix L.2 "Soft-Strict Gap by Domain" (p.27, teal).

QA-03: human validation

We built the complete stratified sampling frame offline (18 domain/model strata, up to 20 flagged/10 joint/10 missed per stratum, seed 42, census where smaller; 486 of 720 possible items identified with inclusion weights). The labeling itself remains blocked, no qualified human annotators were available, and we do not substitute an agent for them.

Minor

MIN-01 (spell out "Concept-Binding Accuracy (CBA)" at first use) applied directly: §1 Introduction now reads "Concept-Binding Accuracy (CBA) is designed to diagnose this conflation..." at its first unexpanded main-text occurrence.

What remains unresolved

The independent-human-ontology-designer arm and matched-prompt re-inference both require resources, qualified annotators or paid API budget. Frozen, ready-to-run designs are in our supplementary materials rather than fabricated results.

I appreciate the authors’ careful response and additional analysis. The new results partially address my sensitivity concern. Nevertheless, my main concerns remain. CBA_cond produced no model-selection disagreement with joint F1, and the authors acknowledge that equivalent information can be obtained from per-field confusion analysis. It therefore remains unclear what practical value CBA adds beyond existing evaluation methods. Overall, the revision improves the framing and analysis, but these issues remain central to my assessment. I therefore maintain my recommendation.​‌

Reviewer_mF6uOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that value-only evaluation of document extraction (Rec_val, which credits a ground-truth value if it shows up anywhere in the output) hides "concept misbinding": the right value placed under the wrong field. It proposes Concept-Binding Accuracy (CBA_cond = fraction of recovered values that also carry the right key, with a strict and an ontology-based soft variant), ACC_joint = Rec_val x CBA_cond, and Delta = Rec_val - ACC_joint, computed with a value-first Hungarian matching. It also proposes Hindsight, 9 datasets in 5 domains (paystubs, SROIE, CORD, synthetic W-2, LEDGAR, CUAD, two VRDU subsets, GST invoices) with hand-built concept-family ontologies. Validation: Claude Haiku 4.5 and GPT-4o-mini on ~50 docs per domain (plus Qwen2-VL-7B 4-bit on 5 domains). The largest gaps are on CUAD (Delta 25.5-33.3 pts) and LEDGAR (21.9-24.9), moderate on W-2 (6.1-10.2), near zero on the low-ambiguity domains. A prompt-format ablation (free JSON vs fixed keys) on 3 domains shows a domain-dependent effect, and a 5x4 density x overlap factorial study on W-2 shows both factors lower CBA_cond.

Critical strengths
  1. The controlled factorial study on W-2 (Sec 4.5, App C.8) is the most convincing part. Holding documents and models fixed and varying only which N concepts are active isolates semantic overlap from density, which the cross-domain table can't do. The finding that 4 near-synonym fields already drop Haiku to 45% CBA_cond (vs 93% for 4 unrelated fields) is a concrete, useful result for people designing extraction schemas.

  2. The diagnostic split into "did the model read the value" vs "did it put it in the right slot" is practically useful, and the paper is careful and transparent about it: it clearly says CBA complements rather than replaces schema-aware F1 (Sec 3.2), reports bootstrap CIs, includes negative/mixed results (prompt ablation), and releases raw outputs and scoring code at very low cost. This makes the benchmark easy to adopt and check.

Critical weaknesses
  1. The main framing rests on a weak baseline and the headline numbers are partly by construction. The paper itself admits (Sec 1, 3.2, App B.1) that standard schema-aware key-value F1 already penalizes misbinding; the gap is only shown against Rec_val, a value-only baseline that the paper names itself and does not show is widely used (no citations of benchmarks or papers that report it). Yet the title ("Beyond Field F1"), Figure 1 (labelled "Standard Field F1 Scoring ... F1 = 100%") and the TL;DR ("up to 33.5% F1 overstatement") say field F1 hides the error, which contradicts Sec 3.2. Also, the biggest gaps come from CUAD and LEDGAR, where the model is given the clauses and asked to label each one, so Rec_val = 100% by design (App C.7 says the model is "forced to classify every clause it is given"). There, CBA_cond is just clause-classification accuracy and Delta = 1 - accuracy; calling this a 33-point "overstatement" of extraction is not really informative. Precision / joint F1 are defined in App M but never reported, so the reader can't see how CBA compares with the metric the authors say practitioners should use.

  2. Small and narrow evaluation for a benchmark paper. ~50 documents per domain, only small/cheap models (Haiku 4.5, GPT-4o-mini, 4-bit Qwen2-VL-7B), no frontier models and no dedicated document models (e.g. LayoutLM-style or Donut-style extractors). The two "most interesting" structured domains (W-2, paystubs) are synthetic, and GST ground truth was made with Claude Sonnet 4.6 (with Rec_val ~30%, it's hard to tell model failure from label noise). Results also shifted between two runs on different API routing (App D.2, W-2 moved most), which weakens per-model claims like "the ranking reverses on W-2". The "independent causal effect" language for density is also stronger than the data: the density effect is not monotone (CBA_cond goes up from N=32 to N=44) and the two models diverge sharply in some cells, but results are averaged over models.

Questions for authors

Questions that could change my assessment:

  1. Can you point to existing benchmarks or published papers that report value-only recall as their main metric? If it is rarely used, what does CBA show that per-field accuracy or a per-field confusion matrix under schema-aware F1 does not?
  2. Please report Precision, DupRate and F1_joint (App M) for Table 2. How large is the gap between Rec_val and F1_joint, and does the model ranking change?
  3. For CUAD/LEDGAR, is Rec_val = 100% guaranteed by the prompt (every given clause gets a label)? If so, how is this different from reporting clause-classification accuracy? How do you handle CUAD clauses that have more than one gold label?
  4. How does the matching treat the common case where several gold fields share the same value (e.g. W-2 boxes 1/3/5 are often identical; CGST = SGST on GST invoices)? If the model emits that value only once, is the item counted as a misbinding or a miss, and how often does this happen?
  5. For the factorial study, can you show the per-model grid and explain the non-monotone density effect beyond the denominator argument (e.g. by reporting misbindings per active concept)?

Clarifications (minor): the two anonymous repo links differ (Reproducibility statement vs App F); TL;DR says 33.5% but the paper says 33.3; Table 13 "Identity 31" vs Fig 3 caption "identif 30"; the sentence in Sec 5.1 "density and overlap affect the accuracy of binding for each move across identical documents and models" is garbled; App D.2 "evaluation began;."

2 comments
AuthorsOpenReview ↗

Response to reveiwer mF6u

We thank the reviewer for the careful evaluation, constructive criticism and insightful questions.

CW-01: Framing

Reviewer Concern

The main framing rests on a weak baseline and the headline numbers are partly by construction. The paper itself admits (Sec 1, 3.2, App B.1) that standard schema-aware key-value F1 already penalizes misbinding; the gap is only shown against Rec_val, a value-only baseline that the paper names itself and does not show is widely used (no citations of benchmarks or papers that report it). Yet the title ("Beyond Field F1"), Figure 1 (labelled "Standard Field F1 Scoring ... F1 = 100%") and the TL;DR ("up to 33.5% F1 overstatement") say field F1 hides the error, which contradicts Sec 3.2. Also, the biggest gaps come from CUAD and LEDGAR, where the model is given the clauses and asked to label each one, so Rec_val = 100% by design (App C.7 says the model is "forced to classify every clause it is given"). There, CBA_cond is just clause-classification accuracy and Delta = 1 - accuracy; calling this a 33-point "overstatement" of extraction is not really informative. Precision / joint F1 are defined in App M but never reported, so the reader can't see how CBA compares with the metric the authors say practitioners should use.

Response

F1 penalizes misbinding. We will correct the framing, separate classification/extraction, and report joint F1.

CW-02: Scope

Reviewer Concern

Small and narrow evaluation for a benchmark paper. ~50 documents per domain, only small/cheap models (Haiku 4.5, GPT-4o-mini, 4-bit Qwen2-VL-7B), no frontier models and no dedicated document models (e.g. LayoutLM-style or Donut-style extractors). The two "most interesting" structured domains (W-2, paystubs) are synthetic, and GST ground truth was made with Claude Sonnet 4.6 (with Rec_val ~30%, it's hard to tell model failure from label noise). Results also shifted between two runs on different API routing (App D.2, W-2 moved most), which weakens per-model claims like "the ranking reverses on W-2". The "independent causal effect" language for density is also stronger than the data: the density effect is not monotone (CBA_cond goes up from N=32 to N=44) and the two models diverge sharply in some cells, but results are averaged over models.

Response

We will qualify causal/generalization claims, audit GST, expand samples/models where feasible, and repeat matched-route runs. Rankings are run-specific.

QA-01: Precedent

Reviewer Concern

Can you point to existing benchmarks or published papers that report value-only recall as their main metric? If it is rarely used, what does CBA show that per-field accuracy or a per-field confusion matrix under schema-aware F1 does not?

Response

We cannot substantiate widespread use and will remove that claim. CBA summarizes recovery-conditioned assignment; recovery-aware confusion analysis contains the same information.

QA-02: F1

Reviewer Concern

Please report Precision, DupRate and F1_joint (App M) for Table 2. How large is the gap between Rec_val and F1_joint, and does the model ranking change?

Response

We will rescore Table 2 for these metrics, Rec_val-F1_joint and rankings. For N gold items, M predictions, J joint matches: Precision=J/M; F1_joint=2J/(N+M). M is unavailable.

QA-03: Legal task

Reviewer Concern

For CUAD/LEDGAR, is Rec_val = 100% guaranteed by the prompt (every given clause gets a label)? If so, how is this different from reporting clause-classification accuracy? How do you handle CUAD clauses that have more than one gold label?

Response

Full recovery is intended, not guaranteed by prompting. We will audit parser enforcement and CUAD multilabel handling, undocumented. With one label per recovered clause, CBA is classification accuracy.

QA-04: Duplicates

Reviewer Concern

How does the matching treat the common case where several gold fields share the same value (e.g. W-2 boxes 1/3/5 are often identical; CGST = SGST on GST invoices)? If the model emits that value only once, is the item counted as a misbinding or a miss, and how often does this happen?

Response

For three equal-valued fields, one emission gives one joint match (correct key) or misbinding (non-gold key), plus two misses. We will reconcile equations/Algorithm 1 and measure collision frequency.

QA-05: Factorial

Reviewer Concern

For the factorial study, can you show the per-model grid and explain the non-monotone density effect beyond the denominator argument (e.g. by reporting misbindings per active concept)?

Response

We will report per-model grids, misbindings per active gold item and shared-field N=32/44 changes. At N=44 all conditions use the same fields: order/composition need controls; dilution is not a causal explanation.

Minor. We will reconcile links, fix 33.5/33.3 and both sentences, and clarify Identity 31 = 30 within-category + 1 cross-category in the heatmap.

We thank the reviewer again. The initial rebuttal was already posted; this note reports the evidence now completed and exactly where it lives in the revised manuscript. Color legend: our revision marks changes addressing mF6u in blue, and changes addressing 2+ reviewers in orange.

QA-02 (prior ID: QA-02): "report Precision, DupRate, F1_joint for Table 2" (now Table 3, after this revision's new Table 1 and the pre-existing ontology table shifted the numbering)

Our scorer already computed these for all 18 rows; they were never printed. Now added.

DatasetModelPrecision (%)F1_joint (%)
CUADHaiku74.574.5
CUADGPT-4o-mini66.766.7
LEDGARHaiku78.178.1
W-2Haiku46.137.1
SROIEHaiku18.628.2

On CUAD/LEDGAR, Precision = ACC_joint = F1_joint = CBA_cond exactly (recovery is total, M=R=G); there is no separate "F1 hides it" story there. Locator: new sentence, §3.2 (p.4, orange); full 18-row table with 95% CIs, Appendix M, Table 17 (p.28, orange).

CW-01/CW-06: Figure 1 and the F1 framing

Figure 1's left panel was mislabeled "Standard Field F1 Scoring" while scoring value-only matches. Relabeled to "Value-Only Recall (Rec_val) Scoring." Locator: Figure 1, §3.1 (p.3).

QA-05/CW-02: "independent causal effects" and dilution

Decomposing the N=32→N=44 rise shows it is driven mostly by newly-added fields being bound almost perfectly (97–99%), while the originally-active 32 fields' own accuracy stays flat (e.g. near-synonym, Haiku: 33.0% at both N=32 and when re-isolated at N=44). We removed "independent causal effects" language throughout (abstract and two appendix sections) and replaced it with this mechanism. Locator: abstract (p.1, orange); Appendix C.9 "Factorial Study: Full Design and Error-Rate Grid" (p.19–20, orange); Appendix D.1 "When Does Misbinding Matter? (Extended)" (p.21, orange).

QA-04: duplicates/collisions

Confirmed exactly: three identical-valued gold fields with one correct-key emission give R=1, J=1, L=2 (not R=3), on 23/23 edge-case tests against the real scorer. W-2 has a value collision in 100% of its 50 documents; 18.3% of all gold items sit in a collision group. No scoring correction needed.

Minor items

MIN-03: Identity category = 31 total = 30 within-category + 1 cross-category (into a state/local field), re-derived exactly. Locator: Appendix C.2 "Concept Density: The W-2 Effect" (p.17, blue). MIN-04: garbled §5-equivalent sentence rewritten. Locator: §5 Discussion (p.9, blue). MIN-05: "evaluation began;." punctuation fixed. Locator: Appendix D.2 "Limitations (Extended)" (p.21, blue).

What remains unresolved

Broader sample/model coverage, same-route repeatability, controlled factorial intervention, human validation) all require paid API budget or human annotators. Frozen, ready-to-run designs are provided in our supplementary materials rather than fabricated results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.