- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This draft claims that the metrics used for value-only studies only considers if the targeted value is extracted from a document, but they omit if the value is associated with the correct key. Thus, the previously used Value-only recall rate (Rec_val) did not penalize the wrong binding(the value appears in the ground-truth but matched to a wrong key.)
To solve this problem, this draft proposes 1: a new metric, CONCEPT-BINDING ACCURACY, which penalizes this key-value mismatching error. 2: a new benchmark for this key-value extraction which consists of previously published datasets.
The proposed benchmark and metric are claimed to better evaluate information extraction methods.
Critical strengths
- The overall writing of this draft is clear, the proposed idea is easy to follow.
- Around 250 synthetic data samples are added for schema-driven information extraction.
Critical weaknesses
There exist several key problems in this current draft.
-
This paper mainly compares their proposed “new” metric against value-only recall rate, this could be a scope mismatch in computer science. Those datasets are proposed or used for value-only studies on purpose, in which only the coverage of the retrieved information or value is important. When taking key-value into account, there exists a long history in schema IE studies, which will also penalize the wrong “binding” scenario. For instance, it is less reasonable to criticize the metric of accuracy used for image classification that omits pixel-level information, then rename a metric to compare against the accuracy for image segmentation.
-
Line 68, “None isolates the case in which the correct value is attached to the wrong concept.” This claim seems to be wrong to me, if most of those datasets are proposed for value-only studies, then this study should compare their proposed method against schema driven IE tasks. Further, most of the listed datasets could be used for schema-driven extraction, e.g., the mismatched key-value pairs from SROIE, CORD v2 will also be penalized.
-
Another main issue in this draft is that the proposed CBA metric seems to be a renamed existing metric in statistics. If my understanding is correct, Section 3 is explaining the proposed CONCEPT-BINDING ACCURACY. Based on the context, Eq(1) is the recall rate, how many correct values are retrieved out of the ground-truth, so called Value-Only Recall. Then Eq(2) introduces the Conditional Concept-Binding Accuracy, among those retrieved correct values, how many of them are truly correct, the keys are also matched. This can be confirmed by Figure 1 right, 3 values are matched(retrieved), only one of them the key is also correct, so CBA=33.3%. In basic statistics, we normally consider this CBA as Precision. Then Eq(3) is essentially Recall * Precision, which is the numerator of F1. That is the proposed new metric is essentially precision we commonly used in statistics.
-
The newly defined delta Eq(6), is a customized combination of those basic statistics terms, it is an overlap of existing statistics measures.
-
The strict or soft matching is a pre-defined scoring rule, which is essentially the same as exact matching, partial matching and fuzzy matching.
-
Section 3.2, this study admits schema-aware F1 can already detect the key mismatch, but the Fig 1. Left still uses standard F1 to emphasize the difference, this seems to be not right.
-
The claim contribution of a new benchmark, which is essentially recomputing basic statistics over previously published datasets. The only newly added data is about 200 newly created paystubs and tax forms. This can hardly be considered a solid contribution.
Questions for authors
Please refer to the above section.
3 comments
The core idea is still lame
I appreciate the authors respond each question separately and accordingly. Unfortunately, I don't think they resolved my concerns. I will point one thing here about the main concept,
"Let G=gold items, M=predictions, R=one-to-one value matches, J=joint matches. CBA=J/R;" If we formulate the "retrieved" pool as those one-to-one value matches(R), then we wonder how many of the "retrieved" are true postives, joint matches J, then this is essentially PRECISION(numerator) in statistics. It is simply how we formulate the problem, I could be wrong, but I could not see why this is a new statistics to publish.
We thank the reviewer again. The initial rebuttal was already posted; this note reports the evidence now completed and exactly where it lives in the revised manuscript. Color legend: changes addressing VnK7 are purple, and 2+ reviewers orange.
CW-03: "CBA is just a renamed existing metric (precision)"
We ran the exact counterexample through our scorer: G=3,M=4,R=3,J=1 gives CBA=1/3 but joint precision=1/4, genuinely different. Removing the spurious 4th prediction collapses M=R=3, and all four quantities become 1/3, exactly our Figure 1 example. CBA is conditional precision restricted to the value-matched subset, an established construction, not a new primitive. Locator: §3.2 (p.4, orange — shared with mF6u's QA-02, since the same CUAD/LEDGAR numbers answer both).
CW-04: Delta "is a customized combination of existing statistics"
A matched-pair construction, holding G, M, J identical by design:
| Condition | G | M | R | J | Rec_val | CBA_cond | Δ |
|---|---|---|---|---|---|---|---|
| Value-error (values replaced) | 38 | 38 | 18 | 18 | 0.474 | 1.000 | 0.000 |
| Key-misbinding (values transposed) | 38 | 38 | 38 | 18 | 1.000 | 0.474 | 0.526 |
Joint Precision and F1_joint are identical (0.474) in both rows by construction (they depend only on G, M, J); Rec_val/CBA_cond/Δ, whose denominator R we deliberately varied, differ completely and correctly. Locator: Table 1, §3.2 (p.4, orange).
CW-01/CW-02: scope and schema evaluation
Verified against primary sources, not inferred from dataset names: SROIE's official Task 3 protocol states "marked as correct if both submitted content and category... matches the groundtruth" (exact quote, fetched live); the widely-used Donut reference evaluator for CORD matches (schema-path, value) tuples. Both already penalize wrong-key bindings. We replaced the overbroad "None isolates the case..." claim accordingly and corrected a citation (a scene-text-recognition paper that does not define any field-value-pair evaluation protocol) that had been attached to the wrong claim. Locator: §2 Related Work (p.2, purple).
CW-06: Figure 1
The left panel was relabeled (prior response) to "Value-Only Recall (Rec_val) Scoring." This response additionally adds, to the right panel: "CBA = Joint F1 = 33.3%", so the figure itself now shows that true schema-aware F1 already equals CBA in this example. Locator: Figure 1 (p.3).
CW-07: benchmark contribution
200 newly generated paystubs (50 evaluated) are genuine and verified-reproducible: we regenerated 5 from the released seed=42 generator code and all 5 matched the saved ground truth exactly. The 50 evaluated W-2 forms are not generated by us, they are drawn from a pre-existing public dataset; the manuscript now states this explicitly and cites the source. Locator: Appendix A, "W-2 Tax Forms" paragraph (p.22, purple). We also found (and report, not hide) that our own paystub manifest.json disagrees with the per-document ground-truth files on rendering metadata for 92% of documents, though field values agree exactly on all checked documents.
CW-05: scoring rules
Confirmed: strict/soft scoring is established conditional-accuracy-plus-partial-credit, not fuzzy value matching or an invention; no manuscript text change was needed beyond what §3.1/Appendix B already state.
What remains unresolved
Native SROIE/CORD evaluator adapter engineering (comparing our saved outputs against the official/reference scorers end-to-end) was not completed; a frozen design is provided rather than a fabricated result.
Response to reviewer VnK7
We thank the reviewer for the careful evaluation, constructive criticism and insightful questions.
CW-01: Scope
Reviewer Concern
Response
Value-only recall is legitimate for value-only tasks. We will frame CBA as a schema-extraction diagnostic alongside joint F1 and compare identical tasks/outputs.
CW-02: Schema evaluation
Reviewer Concern
Response
The claim is too broad. SROIE Task 3 requires content and category agreement (Huang et al., Sec. III-C); CORD has structured labels. We will replace the claim and compare native schema-aware evaluators.
CW-03: Precision
Reviewer Concern
Response
Let G=gold items, M=predictions, R=one-to-one value matches, J=joint matches. CBA=J/R; joint precision=J/M. They coincide when M=R (Figure 1), not generally: G=3,M=4,R=3,J=1 gives CBA=1/3, precision=1/4. Rec_val*CBA=J/G is joint recall; F1=2J/(G+M). We will clarify this conditional-precision interpretation, without claiming a new statistical primitive.
CW-04: Delta
Reviewer Concern
Response
Delta=(R-J)/G is a derived misbinding rate, not an independent statistical invention. We will present it as an error-category diagnostic and test controlled errors alongside joint F1.
CW-05: Scoring rules
Reviewer Concern
Response
Strict uses exact concept equality after value matching; soft gives predefined family-based partial credit, not fuzzy value matching. We will acknowledge these established ideas; soft credit is ontology-relative, not calibrated severity.
CW-06: Figure 1
Reviewer Concern
Response
We will correct Figure 1 and the F1 framing. For G=M=3 and J=1, schema-aware precision, recall and F1 are 1/3; 100% is value-only recall.
CW-07: Benchmark
Reviewer Concern
Response
Most documents are reused. The manuscript reports 200 generated paystubs plus 50 W-2 forms, with 50 of each evaluated; count alone does not establish novelty. We will audit inherited versus added annotations, mappings, protocols and software, check reproducibility and diagnostic value against native metrics, and narrow claims accordingly.
Questions for Authors: "Please refer to the above section." Addressed in CW-01–CW-07 above.