acceptodds
Under review as a conference paper at ICLR 2027

Document Structure Beyond Heading Hierarchy: Dataset, Method, and Benchmark

Abstract

Documents organize meaning through their structure. Explicit structure helps retrieval and question answering, yet document parsing optimizes word-level accuracy and page-level fidelity while structure recovery targets the heading hierarchy alone. That formulation under-determines the structure of real documents: a parent need not be a title, an element may fall outside the ordered forest and rejoin it through a reference link to a target many pages away, and a unit need not be what one page yields, since page breaks split fragments and running headers take no part in it. We reformulate the problem as the joint prediction of a structure over a document: semantic elements that persist across page boundaries and carry a flag for taking part, a forest whose internal nodes need not be titles and whose groups may be named by virtual nodes, and directed reference edges from an anchor to a target anywhere in the document. We instantiate this with DocVine, a human-annotated dataset of 1,217 documents and 12,736 pages in which all three are annotated over the same elements. We design VinePipe, a staged pipeline, and train VineVLM-2B, a 2B vision language model fine-tuned to drive every stage of the pipeline. VinePipe with VineVLM-2B beats task-specific hierarchy parsers and frontier vision language models on DocVine, and leads on MPDocBench-Parse under its own protocol.

Primary area:
datasets and benchmarks
Keywords:
Document Structure Understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rBxxOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends document structure recovery beyond heading hierarchies to include non-heading and virtual parents, cross-page element identity, and document-wide reference links. It introduces DocVine, comprising 1,217 documents and 12,736 pages, and VinePipe, a staged pipeline driven by the fine-tuned VineVLM-2B model. On DocVine, the system achieves 66.6 Soft-TEDS, compared with 52.0 for the strongest prompted baseline. External evaluation reports improvements in heading hierarchy and truncated-text recovery, although the interpretation of the aggregate score requires clarification.

Critical strengths
  • A practically useful formulation. Non-heading parents, virtual groups, cross-page continuity, and reference links capture aspects of real documents that heading-only representations cannot adequately express.
  • A substantial joint annotation resource. DocVine annotates these relationships over the same document elements. The appendix provides detailed annotation guidelines, prompts, output schemas, and metric definitions.
  • Promising structural results and thoughtful comparisons. The reported DocVine gains are substantial. Capability-aligned projections and the use of oracle elements for both methods in the DOSA comparison help account for differences in baseline scope. The external heading-hierarchy results provide additional evidence of transfer.
Critical weaknesses
  1. The benefits to other OCR and document-parsing tasks remain unclear. The experiments demonstrate structural recovery and cross-page reconstruction, but do not isolate whether adding hierarchy and reference-relation supervision improves ordinary text, table, or formula recognition. The truncated-text and truncated-table metrics combine recognition with merging, while reading-order performance is weaker than several baselines. A controlled comparison with and without the additional relation supervision would clarify whether the broader formulation also benefits existing parsing capabilities.
  2. The external “Overall” score appears to use a different aggregation from the benchmark. MPDocBench-Parse defines Overall as the average of eight task scores. However, Table 3’s values can be reproduced, within rounding, by averaging only the four displayed metrics: the displayed values yield approximately 78.82 for VineVLM and 76.15 for PaddleOCR. A four-task structural aggregate can be informative, but it should be explicitly identified as such. The reported table does not establish an overall lead under the benchmark’s original eight-task protocol.
  3. Annotation reliability needs empirical support. Although Appendix C provides detailed guidelines, independent annotation agreement, adjudication procedures, and measured quality-audit results are not reported. These are particularly important for semantic parents, virtual groups, and in-tree versus out-of-tree membership. Corpus sources, document/template deduplication, and overlap checks with the external benchmark also need clarification.
  4. Some reproducibility details are missing or inconsistent. Section 5.2 describes DOSA as run “as released,” whereas the reproducibility statement says it was reimplemented because no public code was available. Appendix I does not provide the promised implementation deviations or sufficiently precise projection rules. The reproducibility statement also mentions bootstrap intervals for the main comparisons, but numerical intervals are not included in the supplied paper.
Questions for authors
  1. Can the authors compare the same backbone, OCR front-end, and comparable training budget with and without hierarchy/reference-relation supervision? Reporting ordinary text recognition, table recognition, formula recognition, reading order, and cross-page merging would clarify which capabilities benefit and whether any regress.
  2. What exact aggregation produces Table 3’s Overall score? Can the authors report the original eight-task results, or explicitly label the four-task subset aggregate and narrow the corresponding claim? Please also specify the benchmark and baseline-result versions.
  3. What independent agreement or audit evidence supports the structural annotations, particularly virtual groups and semantic parents? How were corpus sources selected, near-duplicate documents or templates handled, and overlap with MPDocBench-Parse checked?
  4. Can the authors provide the exact DOSA implementation and training settings, capability-projection rules, and the bootstrap confidence intervals mentioned in the reproducibility statement?
3 comments
AuthorsOpenReview ↗

Respond to the supervision ablation and overall aggregation

Supervision ablation

We thank the reviewer for requesting this comparison. Both arms share the backbone (Qwen3.5-2B), OCR front end (PaddleOCR-VL), corpus and schedule; the ablated arm only drops the hierarchy and reference-relation targets from the multi-task mixture. Both are evaluated on MPDocBench-Parse under the same protocol, and the supervised arm is the model in Table 3, so the columns below are that row's cells. Reading-order ED and truncated-text ED are error rates (lower is better).

SupervisionReading-order EDTruncated-text EDTruncated-table TEDS
Hierarchy + reference relations0.1430.10084.29
Without0.1890.20083.22

Removing the supervision degrades all three metrics. Truncated-text ED halves (0.200 → 0.100), reading-order ED falls 24% (0.189 → 0.143), and truncated-table TEDS drops 1.07 points (84.29 → 83.22). The supervision thus does not trade structural correctness for recognition quality: the structure it adds carries the merging and ordering.

This also addresses the concern that truncated metrics conflate recognition with merging. The recognition half is fixed: text, cells and formulas come from the same frozen front end in both arms, and no character, cell or formula target enters training. Any score difference therefore comes from the merging decision alone. We do not report text, table or formula recognition as separate tasks, since those outputs are not predicted by our model but by the OCR front end; scoring them would measure that front end twice. Our formulation starts above the recognition layer, hence the structure-based evaluation.

On reading order being weaker than some baselines: the ablation shows the gap is not caused by the supervision, which moves the metric in the right direction. Our hierarchy construction yields a better hierarchical order. Documents contain many floats; a system ordering blocks by page position places a float at that position, whereas our method keeps floats in the external set and attaches them through reference links, so their page positions never enter the predicted sequence.

This ablation is now given in Appendix E.

Overall aggregation

The aggregation is the benchmark's own: the unweighted mean of the task scores. MPDocBench-Parse averages eight tasks; Table 3 averages the four we report. Each score is mapped to a 0–100 scale, 100 × (1 − NED) over merged text and over reading order, and 100 × TEDS over merged tables and over the heading hierarchy, and the Overall column is their mean. Every value in that column is computed by us from the four task columns under this rule, so the column is consistent across rows, and we do not use the benchmark's published eight-task aggregate. The definition is now given in Appendix E.1.

We report these four tasks rather than all eight because they are the ones whose unit of evaluation is not bounded by a page: a text run interrupted by a break, a table continued on the next page, the reading order of the whole document, and the heading hierarchy above the page level. They are exactly the tasks our formulation covers. The remaining tasks score recognition within a page, which our pipeline takes from its OCR front end rather than produces, so scoring them would measure that front end and not the structure we contribute.

The text now states which four tasks the column averages, so the claim attaches to that aggregate and not to an unspecified overall score.

AuthorsOpenReview ↗

Respond to annotation reliability and corpus overlap

  1. Independent agreement and audit evidence.

Agreement is measured on the 13-document exam, which all five annotators annotated, giving 65 submissions; the audit runs over the full released set, and the 78 documents it rejected were corrected and re-accepted.

StageItemValue
CalibrationGold exam13 documents, all five annotators (A1–A5), 65 submissions
Gold sourceThe administrator who wrote the guideline; no annotator took part
AlignmentPer page, greedy one-to-one at IoU ≥ 0.7, category verified; whole cross-page forest compared, virtual nodes included
FeedbackItemized diff for every mismatch
AgreementVirtual grouping76 virtual parents, 190 members; F1 84.8
Hierarchy parents861 edges; F1 76.5
Membership1,068 blocks (910 in, 158 out); accuracy 87.4
Merging80 pairs; F1 88.9
Reference linking25 links; F1 91.2
AuditFlagged, by rulesemantic parent 478; virtual typing 0; reference target 26; title-to-table 443; aside order 0; date/seal/signature order 97; prelabel coverage 466; date category 0
ReviewReleased1,217
Rejected78 (6.41%), 167 reasons (2.14 each)
Re-accepted78 (100%)

Why the scores are independent. Gold was written by the administrator who authored the guideline, so each score measures an annotator against a structure that annotator did not write. Every submission is aligned per page at IoU ≥ 0.7 with category verified, the whole cross-page forest is compared as one structure, and each score is a macro-average over the 65 submissions on the 0–100 scale.

The two structures you name. Virtual groups: 76 virtual parents hosting 190 members, Virtual F1 84.8. Semantic parents: 861 parent–child edges, Parent-edge F1 76.5; the parenting rule flags 478 documents whose media box hangs from a title or plain text, while the virtual-typing rule flags none.

How the audit works. Eight deterministic rules run over every released document, not a sample, each stating a condition a correct annotation cannot violate. A flagged document is read in full by the administrator, who either passes it or returns an itemized record naming stage, page and issue; the annotator revises and does not decide the outcome. The rules are over-inclusive by design, so most flagged documents are cleared and the rejections are residue.

  1. Corpus sources.

The corpus is drawn from FinePDFs (HuggingFaceFW/finepdfs), assembled by Hugging Face from Common Crawl snapshots and released under the Open Data Commons Attribution Licence 1.0. Selection used the corpus's own labels rather than a hand-picked list: every record carries a language–script pair, and Simplified and Traditional Chinese are distinct labels in the source, not a distinction drawn during annotation. Each document is assigned the language of its dominant script, so the labels below are mutually exclusive; the five English and three Japanese documents are those whose primary language covers more than 90% of the text, and mixed documents are counted under their dominant language. No result is reported per language. A document was kept when it belongs to one of the 18 types of Appendix B.2, and the PDF was kept rather than extracted text, because annotation is over page images.

LanguageDocsRatio
Simplified Chinese75662.1%
Traditional Chinese45337.2%
English50.4%
Japanese30.2%
Total1,217100%
  1. Near-duplicate documents and templates.

The check runs on text extracted after header and footer blocks are removed, since documents from the same publisher share running heads and page furniture and the template would otherwise drive the similarity. Exact duplicates are caught by SHA-256 over the extracted text; near-duplicates by word-level 3-gram shingles, MinHash and LSH banding, with pairs at Jaccard ≥ 0.85 linked and one representative kept per connected component. Screening precedes annotation and splitting, so a removed document enters neither set. No cluster formed at this threshold and the two exact duplicates are the only removals, so 1,219 candidates yield 1,217 released documents.

  1. Overlap with MPDocBench-Parse.

Checked at two levels. Document level: the text of every benchmark document was hashed with SHA-256 and compared with the DocVine hashes, and no document matches. Page level: MinHash signatures over the text of every page on both sides, and no DocVine page among the 12,736 reaches Jaccard ≥ 0.9 with any benchmark page, that is, 0%.

AuthorsOpenReview ↗

Respond to the DOSA implementation and bootstrap intervals

All three items are now given in the appendix; the two that are lists of settings are reproduced here so they can be read without a cross-reference.

ComponentSetting
Visual backboneResNet-50 (ImageNet-1k), FPN over P2–P6; C2–C5 features, 7×7 RoI align
Text encodermultilingual SentenceTransformer, 512-d, 128 tokens
Scalar featurescategory, normalized box geometry
Frozenbackbone with FPN, text encoder
TrainableRoI projection, fusion layers, prediction heads
Fusion6 Transformer layers, width 1024, 8 heads, FFN 2048
Headsparent and predecessor pair scoring; focal loss
Chunk / context256 objects / 8 historical objects
Input800×800
OptimizerAdamW, 5×10⁻⁵, weight decay 10⁻⁴
Batch / schedule2 / 100 epochs, tenfold decay after epoch 90

DOSA has no public implementation, so this is a reimplementation of the paper, released with the code; the numbers in Table 2 are what it obtains under our protocol, not a reproduction of the published ones, which were obtained on different data.

Structure in DocVineProjection
virtual noderemoved, its children promoted to its parent
multi-member nodesplit into siblings
element merged across pagessplit at the page boundary
element detection, OCRoutside this block: both systems receive the ground-truth elements
sibling orderderived reading order (the block does not measure reading-order recovery)

The projection is applied identically to the gold annotation and to the prediction.

MetricStrongest baselineVinePipe + VineVLM-2B
Soft-TEDS52.0±2.8 (VinePipe + Claude-Fable-5)66.6±2.3
Hierarchy Edge F137.9±2.6 (Direct Kimi-K3)56.9±2.9
Link F155.5±4.8 (VinePipe + Qwen3.8-27B)65.2±4.4
Merge F166.0±6.6 (Direct Kimi-K3)77.3±5.6

Intervals are over the 91 test documents, B = 1,000 replicates, 95% percentile (2.5th–97.5th); the resampling itself is defined in Appendix H.6, where all 13 rows and the interval of every cell are given in Table 23.

Reviewer_LgtnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the limitations of existing document parsing methods, which primarily focus on page-level content extraction or reconstructing heading hierarchies, while overlooking more complex structural relationships in real-world documents. In particular, existing approaches struggle to represent hierarchical relationships involving non-title elements, semantic elements spanning multiple pages, and cross-references between distant document components. To address these limitations, the authors reformulate document structure understanding as the joint prediction of semantic elements, an ordered forest representing hierarchical relationships, and directed reference links. They introduce DocVine, a human-annotated dataset comprising 1,217 documents and 12,736 pages, and propose VinePipe, a three-stage pipeline for element recognition, cross-page forest construction, and reference linking. A fine-tuned 2B vision-language model, VineVLM-2B, is trained to perform all three stages. Experimental results show that the proposed approach outperforms task-specific document parsers and several frontier vision-language models across all four evaluation metrics on DocVine, while also achieving the highest overall score on the external MPDocBench-Parse benchmark.

Critical strengths
  1. The paper identifies meaningful limitations in conventional heading-based representations and extends document structure modeling to include non-heading parent elements, virtual nodes, cross-page element continuity, and reference links. The unified formulation provides a useful conceptual framework for studying document structure beyond page-level parsing and conventional section tree reconstruction.
  2. The paper constructs a valuable dataset for future work. It includes human annotations covering semantic elements, hierarchical relations, and reference links within a unified representation. In particular, the inclusion of virtual nodes, cross-page element merging, and long-range references distinguishes the dataset from existing benchmarks that primarily focus on page-level parsing or heading hierarchy reconstruction.
  3. The proposed VinePipe framework, combined with VineVLM-2B, achieves the best performance across all four evaluation metrics on DocVine, outperforming both task-specific parsers and substantially larger general-purpose vision-language models. The results are particularly encouraging given the relatively small model size. Furthermore, the evaluation on MPDocBench-Parse provides additional evidence that the proposed approach can generalize beyond the newly introduced benchmark and improve performance on conventional document parsing tasks.
Critical weaknesses
  1. It remains unclear to what extent virtual nodes, non-heading hierarchical relations, and reference links actually improve practical downstream tasks compared with simpler heading-based representations. Given that the proposed formulation substantially increases annotation complexity and introduces additional prediction stages, downstream experiments would help establish whether these extensions provide sufficient practical benefits to justify their additional complexity.
  2. The proposed formulation represents the main document structure as an ordered forest, where each element can have at most one parent. However, semantic relationships in real documents may not always conform to such a strict hierarchical organization. For example, a shared explanatory paragraph or a summary table may simultaneously support multiple sections or topics, making it difficult to assign a unique hierarchical parent without losing part of its semantic context.
  3. The main experimental results demonstrate strong performance for VinePipe combined with the fine-tuned VineVLM-2B, but it remains unclear whether the improvements primarily originate from the proposed pipeline or from task-specific supervision on DocVine. In particular, Table 1 shows that applying VinePipe to general-purpose vision-language models without specialized training does not consistently improve performance and, in some cases, substantially degrades it.
Questions for authors

Can the authors provide downstream experiments demonstrating the incremental benefits of the proposed structural representation over conventional heading hierarchies, particularly for document retrieval or long-document question answering? How does the proposed single-parent forest handle document elements that simultaneously belong to or support multiple semantic contexts? Can the authors disentangle the contributions of pipeline design and task-specific fine-tuning through controlled ablation experiments?

3 comments
AuthorsOpenReview ↗

Respond to downstream benefits

We thank the reviewer for this suggestion and agree that parsing quality should ultimately be measured by downstream use. We evaluated our representations on two multi-document question-answering sets.

Datasets. Both are multi-document variants of published benchmarks: a question is generated from the joint context of several documents rather than a single PDF, every generated sample is manually verified, and questions that are illogical, duplicated, or answerable from one document are discarded. The answer must therefore be assembled from evidence held in different documents, which is precisely where hierarchy quality shows.

  • SyllabusQA-Mdoc: 196 questions over 63 course-syllabus documents, asking about program requirements, prerequisites and policies stated in different sections or documents.
  • QASPER-Mdoc: 143 questions over 256 documents derived from NLP papers, each question spanning 2 to 5 papers, asking about methods, datasets and reported numbers whose evidence is scattered across sections and tables.

Setup. Every row shares the same reader (Qwen3.6-Plus) and judge (Qwen3.6-Plus); answers are scored by exact true/false matching, and a shared judge carries any bias equally, so the ranking is unchanged. All rows start from the same PaddleOCR-VL-1.5 parsing, and the only variable is the component that builds the cross-page hierarchy and serves as the retrieval index:

  1. PaddleOCR-VL-1.5: the parser's own pipeline, whose built-in stage builds the heading hierarchy and merges cross-page text and tables.
  2. MinerU-PoPo: a structure-repair system that post-processes page-level parser output to rebuild the title hierarchy, used as released with no task-specific training on our data.
  3. Qwen3.5-35B-A3B and Qwen3.5-27B: general-purpose LLMs constructing the hierarchy in exactly the same pipeline mode as VineVLM-2B (same parsing input, same hierarchy slot, same output interface) but without task-specific training.
  4. VineVLM-2B (ours): a Qwen3.5-2B fine-tuned for hierarchy construction, in the same slot of the same pipeline.
ParsingHierarchy builderSyllabusQA-MdocQASPER-Mdoc
PaddleOCR-VL-1.5PaddleOCR-v1.572.476.2
MinerU-PoPo–80.42
Qwen3.5-35B-A3B82.6587.4
Qwen3.5-27B91.894.4
VineVLM-2B93.3795.1

Accuracy improves monotonically with hierarchy quality: from the parser's built-in stage (72.4 / 76.2), to a structure-repair system (80.42 on QASPER), to general-purpose LLMs in the same hierarchy slot (82.65–94.4), and finally to VineVLM-2B (93.37 / 95.1). Our 2B model, trained for this task, surpasses both Qwen3.5 generalists, including the 27B model on the identical pipeline, under the same reader and judge, so the gain is attributable to the trained representation rather than to model scale. We will add the datasets, setup and table to the revision.

AuthorsOpenReview ↗

Respond to shared elements in a single-parent forest and the attribution of the gain

Shared elements in a single-parent forest

Semantic support in real documents can be many-to-many: a paragraph or a summary table may serve several sections at once, and a structure in which each element has at most one parent cannot represent that relation. We do not claim otherwise. The forest of Section 3 is the reading order and the parent–child containment of the document, single-parent by construction, so that the hierarchy remains a tree that can be compared edge by edge; it is not a graph of semantic support. At present a shared element is annotated once, at the position where the document prints it, and the other sections that draw on it are not recorded. When it is printed beside the hierarchy it stays out of the forest, recorded beside the structure rather than inside it.

We regard this as a boundary of the present schema rather than a property of the task, and the schema does not exclude the case. E_r carries no relation type: a link records that an anchor refers to a target, not what kind of reference it is. The target may be a leaf, a non-leaf element, or a virtual node, since a reference often names a group whose heading the document never prints. Nothing in the definition requires a target to be referenced once, since only the pairs are constrained to be distinct, and several links may already share one anchor. What is absent is a label that says the target is shared. Supplying it requires only a type on the existing link set, a supports relation recording that an element is shared by the sections it serves, and no change to V or to E_h: shared content would enter the representation through the link layer rather than as a second parent. The formal change is small, but annotating the relation across the corpus and scoring it in the metrics is not, so we state it as future work in the Conclusion.

Attribution of the gain

The new controlled comparison shows that supervision supplies most of the gain, and that the pipeline adds to it only when every stage is supervised.

SettingSupervisionSoft-TEDSHierarchy Edge F1Link F1Merge F1
Directnone5.35.616.50.0
end-to-end29.926.834.930.0
VinePipenone1.92.811.01.3
stage-wise (VineVLM-2B)66.656.965.277.3

Setup. One backbone (Qwen3.5-2B), both settings, off the shelf and after fine-tuning on DocVine; page images, PaddleOCR-VL output and training corpus held fixed. Four configurations, reported in the appendix section "Attribution of the Gain" and summarised in Section 5.

Without training, the pipeline lowers three of the four metrics: soft-TEDS 5.3 → 1.9, Hierarchy Edge F1 5.6 → 2.8, Link F1 16.5 → 11.0, with Merge F1 flat at 0.0 → 1.3. Table 1 shows the same at larger scale, where Merge F1 falls for every prompted model (Kimi-K3 66.0 → 20.1, Claude-Fable-5 55.7 → 0.1). A general-purpose model does not honour a stage's output contract: boxes drift, categories change, and single-page errors compound at the merge stage.

Supervision supplies most of the gain. Direct plus fine-tuning: soft-TEDS 5.3 → 29.9, Hierarchy Edge F1 5.6 → 26.8, Link F1 16.5 → 34.9, Merge F1 0 → 30.0.

The pipeline then adds to it, reaching 66.6 soft-TEDS, 56.9 Hierarchy Edge F1, 65.2 Link F1 and 77.3 Merge F1, that is +36.7, +30.1, +30.3 and +47.3 over the fine-tuned Direct model in the same order; the largest gain is in Merge F1, the metric that rewards carrying an element across a page boundary. Both supervised rows are fine-tuned for their respective settings (end-to-end for Direct, stage-wise for VinePipe), so this is the combined effect of stage-wise training and inference, not the pipeline alone.

This is the split behind the sentence already in Section 5: "the advantage comes from the pipeline and the supervision together, and the pipeline's contribution differs by setting."

Thank you for your response. Most of my concerns have been adequately addressed, and I have therefore increased my rating accordingly.

Reviewer_vWLEOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates document structure understanding as the joint prediction of semantic elements, an ordered forest, and directed reference links. The formulation also distinguishes elements that participate in the hierarchy, elements connected through references, and irrelevant content. The authors introduce DocVine, a human-annotated dataset containing 1,217 documents and 12,736 pages, and VinePipe, a staged pipeline for element correction, forest construction, cross-page merging, and reference resolution. A fine-tuned 2B vision-language model, VineVLM-2B, drives these stages.

Critical strengths
  1. DocVine jointly annotates cross-page document elements, hierarchical structure, and reference relations over elements, including 5,652 nodes and 2,800 textual instances. These annotations capture structural information that heading-only representations cannot express. Together with detailed annotation guidelines, the dataset benefits the study of interactions between element recovery, grouping, and reference resolution.

  2. VineVLM-2B leads on all four DocVine metrics, achieving 66.6 Soft-TEDS versus 52.0 for the strongest competing entry. Generalist comparisons share an OCR frontend, while the DOSA comparison uses identical oracle elements and dataset-specific training.

Critical weaknesses
  1. The conceptual novelty is insufficiently distinguished from prior work. GraphDoc already represents hierarchy, reading order, references, and non-heading parents, and DocGenome additionally annotates cross-page elements, subordinate relations, and explicit references. DocParser already models parent–child relationships involving non-heading entities, while DSG includes ordered and unordered groups. These works make the broad heading-only sentences and unqualified “first” over-claim insufficiently supported. arxiv.org. This paper should clarify its specific extensions in virtual grouping, annotation labels, and reference-target standard. Since task reformulation is a central contribution, this imprecise positioning weakens the novelty argument.

  2. Despite detailed guidelines and quality-inspection procedures, the paper reports no quantitative examples or independent evaluation results for virtual groups, membership labels, merges, and references. Besides, the main text treats footnotes as out-of-tree elements, whereas Appendix C.3 places them in the tree. These issues leave uncertainty about annotation consistency and the resulting evaluation target.

Questions for authors
  1. Given the median document length of four pages, the number of test instances involving long-range references, virtual targets, and long documents is necessary to assess how strongly these capabilities are validated.

  2. The main text restricts merge to an incoming node without children, whereas Equation 9 includes children among the concatenated attributes. These definitions can produce different behavior for a heading split across pages whose continuation already governs a subtree. The implemented operation needs clarification.

5 comments
AuthorsOpenReview ↗

Respond to insufficient novelty

We thank the reviewer for this careful critique and agree that three statements are too broad: the unqualified "first" in the contribution list, the survey sentences describing prior work as heading-only, and the description of DocGenome's relations as within-page. We will revise all three. Table R1 (new in Appendix C.5) states the deltas side by side.

Table R1 (new in Appendix C.5): dataset and annotation scope against prior datasets.

DimensionGraphDocDocParserDSGDocGenomeDocVine (ours)
Parents need not be headingyesyesyesyesyes
Corpus scopesingle-page images (80,000)scientific papers (arXivdocs)single-page imagesscientific papers only, LaTeX-derivedheterogeneous, human-annotated
Cross-page element identity------units sharing one LaTeX source (derived, scientific papers only)merges rebuild one element across the page seam: 1,975 text / 825 table
Cross-page reference resolution------typed, but the paper does not state whether they resolve across pagesyes: 2,343 of 6,769 links cross a page
Virtual parents for untitled groups----ordered/unordered groups as detected categories--5,652 corpus-wide
Membership of a unit------not represented; one of 13 category labels onlydecided per occurrence; 12,633 media blocks (11,609 in-tree, 1,024 out)
Reference targetspage-internal----printed scientific entitiesany element, virtual and out-of-tree included, across pages

(1) Cross-page references. GraphDoc, DocParser and DSG neither represent cross-page element identity nor resolve a reference whose anchor and target lie on different pages: GraphDoc predicts relations only among layout instances within a single-page image; DocParser and DSG have no explicit reference type (only parent_of and followed_by), and DSG is single-page. DocVine makes cross-page references first-class, and Link F1 and Merge F1 are defined over exactly this structure. No other dataset evaluates links and merges jointly.

(2) DocGenome. DocGenome is LaTeX-derived and covers scientific papers only. Its six relations map onto ours: Identical ↔ cross-page element identity, Subordinate ↔ parent–child edges, Title/Non-title adjacent ↔ sibling order, and the two Cite types ↔ reference links including caption–float associations. DocVine adds three extensions absent from all four systems: (i) virtual parents for heading-less groups, where DSG's ordered/unordered groups are detected entity categories and no system creates a parent where the page prints no text; (ii) instance-level membership, decided per occurrence rather than per category; (iii) reference targets drawn uniformly from in-tree, out-of-tree and virtual elements across pages, not only printed scientific entities.

(3) Revisions. We will (i) scope the contribution to "to our knowledge, the first human-annotated benchmark that jointly annotates elements, an ordered forest with virtual parents, and reference links, including cross-page links and virtual or out-of-tree targets, over 18 document types"; (ii) credit GraphDoc, DocParser and DSG for non-heading parents and grouping in the survey; (iii) expand the GraphDoc discussion to its full scope and state the delta explicitly (page-scoped, no cross-page element identity, no cross-page reference resolution, no virtual grouping); (iv) correct the DocGenome description and add the condensed mapping to related work.

AuthorsOpenReview ↗

Respond to insufficient novelty

We thank the reviewer for this careful critique and agree that three statements are too broad: the unqualified "first" in the contribution list, the survey sentences describing prior work as heading-only, and the description of DocGenome's relations as within-page. We will revise all three. Table R1 (new in Appendix C.5) states the deltas side by side.

Table R1 (new in Appendix C.5): dataset and annotation scope against prior datasets.

DimensionGraphDocDocParserDSGDocGenomeDocVine (ours)
Parents need not be headingyesyesyesyesyes
Corpus scopesingle-page images (80,000)scientific papers (arXivdocs)single-page imagesscientific papers only, LaTeX-derivedheterogeneous, human-annotated
Cross-page element identity------units sharing one LaTeX source (derived, scientific papers only)merges rebuild one element across the page seam: 1,975 text / 825 table
Cross-page reference resolution------typed, but the paper does not state whether they resolve across pagesyes: 2,343 of 6,769 links cross a page
Virtual parents for untitled groups----ordered/unordered groups as detected categories--5,652 corpus-wide
Membership of a unit------not represented; one of 13 category labels onlydecided per occurrence; 12,633 media blocks (11,609 in-tree, 1,024 out)
Reference targetspage-internal----printed scientific entitiesany element, virtual and out-of-tree included, across pages

(1) Cross-page references. GraphDoc, DocParser and DSG neither represent cross-page element identity nor resolve a reference whose anchor and target lie on different pages: GraphDoc predicts relations only among layout instances within a single-page image; DocParser and DSG have no explicit reference type (only parent_of and followed_by), and DSG is single-page. DocVine makes cross-page references first-class, and Link F1 and Merge F1 are defined over exactly this structure. No other dataset evaluates links and merges jointly.

(2) DocGenome. DocGenome is LaTeX-derived and covers scientific papers only. Its six relations map onto ours: Identical ↔ cross-page element identity, Subordinate ↔ parent–child edges, Title/Non-title adjacent ↔ sibling order, and the two Cite types ↔ reference links including caption–float associations. DocVine adds three extensions absent from all four systems: (i) virtual parents for heading-less groups, where DSG's ordered/unordered groups are detected entity categories and no system creates a parent where the page prints no text; (ii) instance-level membership, decided per occurrence rather than per category; (iii) reference targets drawn uniformly from in-tree, out-of-tree and virtual elements across pages, not only printed scientific entities.

(3) Revisions. We will (i) scope the contribution to "to our knowledge, the first human-annotated benchmark that jointly annotates elements, an ordered forest with virtual parents, and reference links, including cross-page links and virtual or out-of-tree targets, over 18 document types"; (ii) credit GraphDoc, DocParser and DSG for non-heading parents and grouping in the survey; (iii) expand the GraphDoc discussion to its full scope and state the delta explicitly (page-scoped, no cross-page element identity, no cross-page reference resolution, no virtual grouping); (iv) correct the DocGenome description and add the condensed mapping to related work.

AuthorsOpenReview ↗

Respond to annotation evidence and footnote placement

(1) Quantitative evidence. The submission described the annotation workflow but reported no quantitative evidence of its enforcement or of annotation quality. The revision provides both in Appendix C.4: agreement on each structural relation (C.4.1), the eight deterministic audit rules with flag counts, and the outcome of production review.

Agreement. The gold annotation of the 13 exam documents was made by the administrator who authored the guideline; none of the five annotators took part, so each score measures an annotator against a structure that annotator did not write. All five annotate all 13 documents, giving 65 submissions, each aligned to gold per page (greedy one-to-one at IoU ≥ 0.7, category verified) and scored on the remapped trees as set-level F1, macro-averaged over the 65 submissions on the 0–100 scale.

Relation familyInstances in the 13 gold documentsMetricAgreement
Virtual grouping76 virtual parents, 190 real membersVirtual F184.8
Hierarchy parents861 parent–child edgesParent-edge F176.5
Membership1,068 blocks (910 in-tree, 158 out-of-tree)Accuracy87.4
Merging80 merged pairsMerge F188.9
Reference linking25 linksLink F191.2

Parent-edge F1 is the lowest of the five because one error near the root rewrites the parent of every block beneath it, and each rewritten edge counts against the score.

Audit. Audit is separate from scoring and the administrator holds final authority. Eight deterministic rules run over the released set, each stating a condition a correct annotation cannot violate; a flagged document is read in full by the administrator, and a failing one is returned with an itemized record of stage, page and issue, revised, resubmitted, and re-admitted only once the correction is confirmed. The two structures the schema leans on most are checked directly: every image or table box must be mounted under a semantic parent (478 documents flagged), and every virtual node must carry a type (none flagged); the remaining rules flag reference targets (26), title-to-table links (443), reading order (97) and prelabel recovery (466). The rules are over-inclusive by design, so most candidates are cleared on review, and the 78 rejections among 1,217 released documents (6.41%; 167 reasons, 2.14 each; 100% re-accepted) are the residue.

Counts and examples (Appendix B.1). Hierarchy parents 124,680; virtual parents 5,652 (12.97%), in 818 of 1,217 documents, hosting 26,319 real descendants; media blocks with a membership decision 12,633 (11,609 in-tree, 1,024 out); merged elements 2,800 (1,975 text, 825 table); reference links 6,769 (4,426 same-page, 2,343 cross-page, 2,123 to a virtual parent). Representative cases: Doc 002, one title governing five chapters and their article-level descendants across pages 0–6; Doc 003, a virtual list-item parent governing 14 descendants across pages 0–3; Doc 012, a table spanning pages 0–3 whose multi-bbox fragments merge into one element; Doc 010, a cross-page anchor resolving to a target seven pages away. Identifiers are anonymized, and the full annotation of every case will be verifiable in the dataset upon release.

(2) Footnote treatment. The correct convention, implemented in both the annotation pipeline and the metric code, is the one in the main text: footnotes are out-of-tree. They are annotated at page level with category footnote, but tree construction excludes them from the forest: no footnote node appears in the serialized trees, and no released reference link targets a footnote. The sentences in Appendix C.3.2 and C.3.4 describing footnotes as isolated in-tree nodes, and the inclusion of footnotes among reference targets in C.3.6, are errors in the guidelines and will be corrected, together with the corresponding example in Section 3. Since the evaluation target already excludes footnotes, all reported scores are unaffected.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.