- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper extends document structure recovery beyond heading hierarchies to include non-heading and virtual parents, cross-page element identity, and document-wide reference links. It introduces DocVine, comprising 1,217 documents and 12,736 pages, and VinePipe, a staged pipeline driven by the fine-tuned VineVLM-2B model. On DocVine, the system achieves 66.6 Soft-TEDS, compared with 52.0 for the strongest prompted baseline. External evaluation reports improvements in heading hierarchy and truncated-text recovery, although the interpretation of the aggregate score requires clarification.
Critical strengths
- A practically useful formulation. Non-heading parents, virtual groups, cross-page continuity, and reference links capture aspects of real documents that heading-only representations cannot adequately express.
- A substantial joint annotation resource. DocVine annotates these relationships over the same document elements. The appendix provides detailed annotation guidelines, prompts, output schemas, and metric definitions.
- Promising structural results and thoughtful comparisons. The reported DocVine gains are substantial. Capability-aligned projections and the use of oracle elements for both methods in the DOSA comparison help account for differences in baseline scope. The external heading-hierarchy results provide additional evidence of transfer.
Critical weaknesses
- The benefits to other OCR and document-parsing tasks remain unclear. The experiments demonstrate structural recovery and cross-page reconstruction, but do not isolate whether adding hierarchy and reference-relation supervision improves ordinary text, table, or formula recognition. The truncated-text and truncated-table metrics combine recognition with merging, while reading-order performance is weaker than several baselines. A controlled comparison with and without the additional relation supervision would clarify whether the broader formulation also benefits existing parsing capabilities.
- The external “Overall” score appears to use a different aggregation from the benchmark. MPDocBench-Parse defines Overall as the average of eight task scores. However, Table 3’s values can be reproduced, within rounding, by averaging only the four displayed metrics: the displayed values yield approximately 78.82 for VineVLM and 76.15 for PaddleOCR. A four-task structural aggregate can be informative, but it should be explicitly identified as such. The reported table does not establish an overall lead under the benchmark’s original eight-task protocol.
- Annotation reliability needs empirical support. Although Appendix C provides detailed guidelines, independent annotation agreement, adjudication procedures, and measured quality-audit results are not reported. These are particularly important for semantic parents, virtual groups, and in-tree versus out-of-tree membership. Corpus sources, document/template deduplication, and overlap checks with the external benchmark also need clarification.
- Some reproducibility details are missing or inconsistent. Section 5.2 describes DOSA as run “as released,” whereas the reproducibility statement says it was reimplemented because no public code was available. Appendix I does not provide the promised implementation deviations or sufficiently precise projection rules. The reproducibility statement also mentions bootstrap intervals for the main comparisons, but numerical intervals are not included in the supplied paper.
Questions for authors
- Can the authors compare the same backbone, OCR front-end, and comparable training budget with and without hierarchy/reference-relation supervision? Reporting ordinary text recognition, table recognition, formula recognition, reading order, and cross-page merging would clarify which capabilities benefit and whether any regress.
- What exact aggregation produces Table 3’s Overall score? Can the authors report the original eight-task results, or explicitly label the four-task subset aggregate and narrow the corresponding claim? Please also specify the benchmark and baseline-result versions.
- What independent agreement or audit evidence supports the structural annotations, particularly virtual groups and semantic parents? How were corpus sources selected, near-duplicate documents or templates handled, and overlap with MPDocBench-Parse checked?
- Can the authors provide the exact DOSA implementation and training settings, capability-projection rules, and the bootstrap confidence intervals mentioned in the reproducibility statement?
3 comments
Respond to annotation reliability and corpus overlap
- Independent agreement and audit evidence.
Agreement is measured on the 13-document exam, which all five annotators annotated, giving 65 submissions; the audit runs over the full released set, and the 78 documents it rejected were corrected and re-accepted.
| Stage | Item | Value |
|---|---|---|
| Calibration | Gold exam | 13 documents, all five annotators (A1–A5), 65 submissions |
| Gold source | The administrator who wrote the guideline; no annotator took part | |
| Alignment | Per page, greedy one-to-one at IoU ≥ 0.7, category verified; whole cross-page forest compared, virtual nodes included | |
| Feedback | Itemized diff for every mismatch | |
| Agreement | Virtual grouping | 76 virtual parents, 190 members; F1 84.8 |
| Hierarchy parents | 861 edges; F1 76.5 | |
| Membership | 1,068 blocks (910 in, 158 out); accuracy 87.4 | |
| Merging | 80 pairs; F1 88.9 | |
| Reference linking | 25 links; F1 91.2 | |
| Audit | Flagged, by rule | semantic parent 478; virtual typing 0; reference target 26; title-to-table 443; aside order 0; date/seal/signature order 97; prelabel coverage 466; date category 0 |
| Review | Released | 1,217 |
| Rejected | 78 (6.41%), 167 reasons (2.14 each) | |
| Re-accepted | 78 (100%) |
Why the scores are independent. Gold was written by the administrator who authored the guideline, so each score measures an annotator against a structure that annotator did not write. Every submission is aligned per page at IoU ≥ 0.7 with category verified, the whole cross-page forest is compared as one structure, and each score is a macro-average over the 65 submissions on the 0–100 scale.
The two structures you name. Virtual groups: 76 virtual parents hosting 190 members, Virtual F1 84.8. Semantic parents: 861 parent–child edges, Parent-edge F1 76.5; the parenting rule flags 478 documents whose media box hangs from a title or plain text, while the virtual-typing rule flags none.
How the audit works. Eight deterministic rules run over every released document, not a sample, each stating a condition a correct annotation cannot violate. A flagged document is read in full by the administrator, who either passes it or returns an itemized record naming stage, page and issue; the annotator revises and does not decide the outcome. The rules are over-inclusive by design, so most flagged documents are cleared and the rejections are residue.
- Corpus sources.
The corpus is drawn from FinePDFs (HuggingFaceFW/finepdfs), assembled by Hugging Face from Common Crawl snapshots and released under the Open Data Commons Attribution Licence 1.0. Selection used the corpus's own labels rather than a hand-picked list: every record carries a language–script pair, and Simplified and Traditional Chinese are distinct labels in the source, not a distinction drawn during annotation. Each document is assigned the language of its dominant script, so the labels below are mutually exclusive; the five English and three Japanese documents are those whose primary language covers more than 90% of the text, and mixed documents are counted under their dominant language. No result is reported per language. A document was kept when it belongs to one of the 18 types of Appendix B.2, and the PDF was kept rather than extracted text, because annotation is over page images.
| Language | Docs | Ratio |
|---|---|---|
| Simplified Chinese | 756 | 62.1% |
| Traditional Chinese | 453 | 37.2% |
| English | 5 | 0.4% |
| Japanese | 3 | 0.2% |
| Total | 1,217 | 100% |
- Near-duplicate documents and templates.
The check runs on text extracted after header and footer blocks are removed, since documents from the same publisher share running heads and page furniture and the template would otherwise drive the similarity. Exact duplicates are caught by SHA-256 over the extracted text; near-duplicates by word-level 3-gram shingles, MinHash and LSH banding, with pairs at Jaccard ≥ 0.85 linked and one representative kept per connected component. Screening precedes annotation and splitting, so a removed document enters neither set. No cluster formed at this threshold and the two exact duplicates are the only removals, so 1,219 candidates yield 1,217 released documents.
- Overlap with MPDocBench-Parse.
Checked at two levels. Document level: the text of every benchmark document was hashed with SHA-256 and compared with the DocVine hashes, and no document matches. Page level: MinHash signatures over the text of every page on both sides, and no DocVine page among the 12,736 reaches Jaccard ≥ 0.9 with any benchmark page, that is, 0%.
Respond to the DOSA implementation and bootstrap intervals
All three items are now given in the appendix; the two that are lists of settings are reproduced here so they can be read without a cross-reference.
| Component | Setting |
|---|---|
| Visual backbone | ResNet-50 (ImageNet-1k), FPN over P2–P6; C2–C5 features, 7×7 RoI align |
| Text encoder | multilingual SentenceTransformer, 512-d, 128 tokens |
| Scalar features | category, normalized box geometry |
| Frozen | backbone with FPN, text encoder |
| Trainable | RoI projection, fusion layers, prediction heads |
| Fusion | 6 Transformer layers, width 1024, 8 heads, FFN 2048 |
| Heads | parent and predecessor pair scoring; focal loss |
| Chunk / context | 256 objects / 8 historical objects |
| Input | 800×800 |
| Optimizer | AdamW, 5×10⁻⁵, weight decay 10⁻⁴ |
| Batch / schedule | 2 / 100 epochs, tenfold decay after epoch 90 |
DOSA has no public implementation, so this is a reimplementation of the paper, released with the code; the numbers in Table 2 are what it obtains under our protocol, not a reproduction of the published ones, which were obtained on different data.
| Structure in DocVine | Projection |
|---|---|
| virtual node | removed, its children promoted to its parent |
| multi-member node | split into siblings |
| element merged across pages | split at the page boundary |
| element detection, OCR | outside this block: both systems receive the ground-truth elements |
| sibling order | derived reading order (the block does not measure reading-order recovery) |
The projection is applied identically to the gold annotation and to the prediction.
| Metric | Strongest baseline | VinePipe + VineVLM-2B |
|---|---|---|
| Soft-TEDS | 52.0±2.8 (VinePipe + Claude-Fable-5) | 66.6±2.3 |
| Hierarchy Edge F1 | 37.9±2.6 (Direct Kimi-K3) | 56.9±2.9 |
| Link F1 | 55.5±4.8 (VinePipe + Qwen3.8-27B) | 65.2±4.4 |
| Merge F1 | 66.0±6.6 (Direct Kimi-K3) | 77.3±5.6 |
Intervals are over the 91 test documents, B = 1,000 replicates, 95% percentile (2.5th–97.5th); the resampling itself is defined in Appendix H.6, where all 13 rows and the interval of every cell are given in Table 23.
Respond to the supervision ablation and overall aggregation
Supervision ablation
We thank the reviewer for requesting this comparison. Both arms share the backbone (Qwen3.5-2B), OCR front end (PaddleOCR-VL), corpus and schedule; the ablated arm only drops the hierarchy and reference-relation targets from the multi-task mixture. Both are evaluated on MPDocBench-Parse under the same protocol, and the supervised arm is the model in Table 3, so the columns below are that row's cells. Reading-order ED and truncated-text ED are error rates (lower is better).
Removing the supervision degrades all three metrics. Truncated-text ED halves (0.200 → 0.100), reading-order ED falls 24% (0.189 → 0.143), and truncated-table TEDS drops 1.07 points (84.29 → 83.22). The supervision thus does not trade structural correctness for recognition quality: the structure it adds carries the merging and ordering.
This also addresses the concern that truncated metrics conflate recognition with merging. The recognition half is fixed: text, cells and formulas come from the same frozen front end in both arms, and no character, cell or formula target enters training. Any score difference therefore comes from the merging decision alone. We do not report text, table or formula recognition as separate tasks, since those outputs are not predicted by our model but by the OCR front end; scoring them would measure that front end twice. Our formulation starts above the recognition layer, hence the structure-based evaluation.
On reading order being weaker than some baselines: the ablation shows the gap is not caused by the supervision, which moves the metric in the right direction. Our hierarchy construction yields a better hierarchical order. Documents contain many floats; a system ordering blocks by page position places a float at that position, whereas our method keeps floats in the external set and attaches them through reference links, so their page positions never enter the predicted sequence.
This ablation is now given in Appendix E.
Overall aggregation
The aggregation is the benchmark's own: the unweighted mean of the task scores. MPDocBench-Parse averages eight tasks; Table 3 averages the four we report. Each score is mapped to a 0–100 scale, 100 × (1 − NED) over merged text and over reading order, and 100 × TEDS over merged tables and over the heading hierarchy, and the Overall column is their mean. Every value in that column is computed by us from the four task columns under this rule, so the column is consistent across rows, and we do not use the benchmark's published eight-task aggregate. The definition is now given in Appendix E.1.
We report these four tasks rather than all eight because they are the ones whose unit of evaluation is not bounded by a page: a text run interrupted by a break, a table continued on the next page, the reading order of the whole document, and the heading hierarchy above the page level. They are exactly the tasks our formulation covers. The remaining tasks score recognition within a page, which our pipeline takes from its OCR front end rather than produces, so scoring them would measure that front end and not the structure we contribute.
The text now states which four tasks the column averages, so the claim attaches to that aggregate and not to an unspecified overall score.