acceptodds
Under review as a conference paper at ICLR 2027

Thinking with Anchors: Grounded and Efficient Document Reasoning

Abstract

Existing document understanding benchmarks have largely focused on locating page elements, yet real-world document intelligence requires models to reason jointly about region semantics, spatial relations, and visual structure. We present ADOPD 2026, a reasoning-oriented extension of ADOPD that turns page decomposition into spatially grounded document understanding. ADOPD 2026 enriches page anchors inherited from ADOPD 2024 dataset with human-cleaned captions, semantic tags, and generated chain-of-thought (CoT) traces grounded to document regions. Instead of treating boxes, masks, and tags as independent supervision signals, we cast text blocks, visual entities, semantic labels, bounding boxes, and polygon masks as a shared vocabulary of . This representation supports three connected capabilities. First, region-level semantic tagging asks models to identify document element types from both page context and local appearance, revealing long-tail semantic failures that standard layout benchmarks often hide. Second, unified vision-language grounding generates text regions and visual entities together with coordinates or polygonal outlines, transforming detection and segmentation outputs into structured anchors that can be reused by downstream reasoning systems. Third, current state-of-the-art models still struggle with dense counting tasks evaluated on DocCount, a benchmark derived from ADOPD 2026, highlighting the need for the Thinking-with-Anchors pipeline in document semantic understanding. By connecting page decomposition to verifiable visual-anchor reasoning, ADOPD 2026 provides a task framework that moves document understanding beyond localization toward anchor-grounded document intelligence.

Primary area:
datasets and benchmarks
Keywords:
document understanding, visual grounding, multimodal large language models, grounded document reasoning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_833WOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper extends an existing document dataset (ADOPD 2024) with human-cleaned captions, semantic region labels, and CoT reasoning traces grounded in generic (polygonal) visual regions. The resulting ADOPD 2026 dataset is used to study three document tasks (1) text block grounding (as bounding boxes) and visual entity grounding (as segmentation masks). (2) document region classification (3) visual entity counting. The authors evaluate existing detection and segmentation models. They also introduce an agentic refinement procedure that uses a VLM to tackle fragmented text boxes. For tagging, they compare zero-shot VLMs against a fine-tuned LocateAnything model. They also introduce DocCount, a benchmark of over 400 documents for counting of specific visual categories.

Critical strengths
  1. The extension of ADOPD 2024 dataset with addtional annotations (region labels, cleaned captions, and grounded reasoning traces) is a useful contribution.

  2. The improvements for text box region refinement obtained through VLM-guided grouping tackles a practical limitation of existing detectors.

Critical weaknesses
  1. The proposed extension to ADOPD adds useful annotations, but I am not convinced that it constitutes a sufficiently substantial contribution. Also, recent works such as DocLayout-VL (https://doclayoutvl.github.io/, ECCV 2026) and DoCoG (https://docog-eccv.github.io/, ECCV 2026) already address more challenging and comprehensive formulations of semantic document grounding and grounded reasoning. While these works don't explicitly target counting, they substantially overlap with the central motivation of this paper. To be fair, the proceedings for these works were available recently. I don't consider the absence of experimental comparisons to these concurrent works a weakness. But their existence does diminish the significance of the proposed contribution.

  2. The main counting experiments evaluate VLMs for final answer without using the proposed reasoning traces. Therefore, these experiments are not enough to say that the anchor-based reasoning formulation is beneficial.

  3. The DocCount benchmark is restricted to only four visual categories and the number of samples is also small.

  4. The proposed "agentic self-refinement" appears to be a fixed algorithm of sequential operations involving detector outputs, VLM-based grouping, and geometric checks. It is not clear what makes this procedure agentic. There does not appear to be any autonomous planning, iterative refinement, or adaptive decision-making. Merely involving a VLM in a processing pipeline does not make the approach agentic.

  5. The paper claims a unified framework for grounded document reasoning. But the tasks (localization, semantic tagging) and counting experiments are largely independent.

  6. The paper discusses challenges with CJK documents. But the analysis does not adequately establish whether these challenges arise from linguistic characteristics, visual complexity, dataset imbalance, or annotation conventions.

  7. Most experiments involve evaluating or fine-tuning existing detectors, segmentation models, and VLMs. The reported results are useful as diagnostics, but it is not clear what new methodological insight or generalizable learning principle emerges. In this respect, the work seems closer to a document-analysis benchmark and empirical study than a substantial contribution to machine learning. It also appears more naturally suited to venues such as CVPR, ICDAR, or ACL/EMNLP than ICLR.

Questions for authors

While additional experiments or annotation corrections may address individual issues, it's unlikely to substantially change my overall assessment. I'd be ok if the authors decide to focus on other reviews.

3 comments

Thank you for the careful review. We understand your overall assessment, and we still answer the main points, because several of them concern how the paper should be read. ADOPD 2026 is submitted under the datasets and benchmarks area. It does not propose a new model or method. Its claim is that the data is useful: it offers a combination of annotations that existing document datasets do not, and it exposes failure modes of current detectors and VLMs that existing benchmarks do not measure. We answer them in that light.

W1 / W7: contribution, concurrent work and venue.

  • What is new in the data. Of the 15 document-layout datasets we compare against in Appendix A.1, none combines boxes, arbitrary polygons, region-linked OCR blocks, semantic region classes and grounded spatial tasks; ADOPD 2026 provides all five on 120K pages. DocCount is the only benchmark in our counting comparison (Appendix A.1) that is built on document pages, defines its classes semantically, and grounds every counted instance with a polygon.
  • What the data reveals. Each of the following is a measurement that the data makes possible, not a new method:
    • Natural-image detectors and segmenters transfer poorly to document entities: zero-shot AP is near zero, and fine-tuning the same architectures on ADOPD lifts AP by 50–70 points (Table 1).
    • Document-pretrained layout detectors also miss many ADOPD entity types, because layout taxonomies have no class for them: zero-shot recall is 17–18% for dividers, 19–29% for dialog boxes and 47–64% for icons. Fine-tuning on ADOPD adds 35–44 AP (Appendix A.13).
    • Detectors over-fragment text blocks: each ground-truth block is covered by 3.3 predicted boxes on average, and 66% of blocks receive two or more (§4.1.2). Block-level text annotation is what makes this measurable.
    • Zero-shot VLMs rarely recognize role-type regions: at most 3.3% accuracy on Background Image and 6.7% on Decorative / Pattern, against 63–90% on Brand Logo and 70–87% on Table (Table 2).
    • VLMs struggle with dense, definition-following counting: the best evaluated model reaches 72.85% on DocCount (Table 3), and the new ablation shows the errors are mainly enumeration and grounding errors, not arithmetic (W2 below).
  • Concurrent work. Thank you for pointing us to DocLayout-VL and DoCoG (ECCV 2026). We agree they are relevant and will discuss them in the related work. We do not claim novelty for semantic document grounding as a task formulation; our contribution is the data resource and benchmark above, and the failure modes they measure.

W2: the counting experiments do not use the traces. We agree that the original Table 3 did not test this. We added a controlled ablation (General response §1, Appendix A.12): one backbone (Qwen3.6-35B-A3B), the same 442 questions and the same generation budget, varying only the region evidence given in the prompt.

  • Direct answer 50.7% → free reasoning 68.8% → ADOPD regions given as unlabeled anchors 78.7% → verified target regions 97.3%.
  • The gain from verified target regions over all regions (+18.6 points) is statistically significant on the same 442 questions (paired tests).
  • So most counting errors are enumeration and localization errors, and the information that removes them is the region information that ADOPD annotates.

W3: DocCount scope.

  • Four classes. Brand Logo, Photograph, Table and Chart / Graph were chosen because their definitions are clear and visually separable (§4.3.1), so that every reference count can be checked.
  • Size. 442 pages is comparable to several established counting benchmarks in Appendix A.1, such as UCF-QNRF (334), CARPK (459), ShanghaiTech (498) and CountBench (540). Each DocCount page is also dense: 2,555 target regions in total, 4–45 per page (median 5), all polygon-grounded.

W4: "agentic". In the camera-ready version, we will rename it "VLM-guided proposal grouping" and present it only as a diagnostic. The part we consider a contribution is the dataset-level finding it starts from: block-level text annotations show that current detectors over-fragment text blocks (3.3 predicted boxes per ground-truth block; 66% of blocks split).

W6: CJK. This question shows another use of the data. ADOPD's multilingual coverage and per-page metadata (language, complexity tier and text/visual profile; Appendix A.2) let us test the four explanations you list. We stratified the fine-tuned YOLOv12-X Doc2Box AP by language × the five complexity tiers × page profile, and compared each language with English at the same tier mix (Appendix A.16).

  • Visual complexity: CJK pages are denser and more often in the hardest tier, but re-weighting English to a CJK tier mix moves its AP by at most 0.6. After matching, a gap remains: Japanese −5.9 AP, Chinese −2.9, Korean +2.0 on validation, and −4.0, −3.4 and +2.2 on test.
  • Dataset imbalance: unlikely. Korean has the fewest pages (1,381 validation pages, against 3,283 for Chinese and 3,306 for Japanese) yet shows no gap.
  • Where the gap is: visual-leaning pages (Japanese 62.3 vs English 70.4) and sparse pages (46.2 vs 64.6).
  • Linguistic characteristics vs annotation conventions: the pattern is consistent with how CJK text is grouped into blocks on visually rich pages, that is, with text-block grouping conventions, rather than with page complexity. The data cannot fully separate linguistic properties from grouping conventions, and we state this as a limitation.

We hope these answers make clear that the paper's claims are about the utility of the data.

Reviewer_833WOpenReview ↗

Methodological issues not resolved

Thanks for the response. The point about highlighting DoCoG and DocLayoutVL is not about the architectures. The works show that the proposed benchmark is not comprehensive and challenging enough - the bar for a benchmark is higher.

When pointed out that the usage of 'agentic' lacks justification, the authors simply plan to re-brand it as VLM-guided proposal grouping. This is not a good sign and shows lack of understanding about the topic.

Even if contribution is to D&B track, benchmarking can still throw light on unresolved problems and frontiers. But the current benchmark is ill-equipped to do so.

Given all these major issues, I retain my assessment.

Reviewer_7FgxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper extends ADOPD 2024 with revised captions, region-level semantic labels, and reasoning traces linked to visual regions. It evaluates document localization and semantic tagging, and introduces DocCount, a 442-document benchmark for dense counting. The experiments compare specialized vision models and VLMs, examine class imbalance and decoding choices, and show that VLM-based grouping with geometric checks can improve fragmented text-box predictions.

Critical strengths

The dataset extension represents substantial annotation effort and is a meaningful, practical improvement to ADOPD 2024. Adding semantic roles to existing regions makes the resource more useful for studying document understanding beyond localization, and the detailed labeling guidelines help explain the intended distinctions between categories.

The paper is also carefully organized, with broad experimental coverage and a substantial appendix. In particular, the text-box grouping experiment addresses a concrete failure mode and separates the gains from semantic grouping and geometric checks. The rare-class sampling and decoding analyses add useful information beyond the main benchmark scores.

Critical weaknesses

My main reservation is the limited novelty. The work is well executed, but its main advance is enriching an existing dataset and evaluating established models on the resulting tasks. The additional analyses are useful, yet I am not convinced that they establish a sufficiently substantial new contribution for ICLR.

I also find the “Thinking with Anchors” framing somewhat stronger than the evidence. The paper generates Anchor-CoT from verified regions and labels, but does not show whether these traces improve training or inference. DocCount evaluates the final count without assessing the model’s use of anchors. The text-box grouping experiment is a useful inference procedure, but it does not validate the generated CoT data. The contribution would be clearer with a more modest framing or direct evidence of the value of these traces.

As a dataset paper, this is a coherent and well-developed piece of work, with substantial annotation effort and useful experiments. I see it as a potentially better fit for a data-centric venue such as KDD.

Questions for authors
  1. How many annotators independently labeled or checked each sample, and how were disagreements resolved? The guidelines are detailed, but I could not find agreement statistics for either semantic labels or instance counts.

  2. In the awards example on pp. 44–45, the reference count is 9 while all four models return 12. What excludes the three lower-right award/certificate regions under the Brand Logo definition in A.9? Is this an annotation omission or an exclusion rule that needs to be made explicit?

  3. Is there evidence that Anchor-CoT improves training or inference, beyond providing accompanying descriptions of the annotated regions? If it is intended mainly as a resource for future work, clarifying that scope would help.

2 comments

Thank you for the review and the detailed questions.

Q2: awards example (pp. 44–45). A.9 counts certification marks, quality seals and endorsement badges as Brand Logos. We read these as compact visual identity marks, such as the medal seals in the left column. We do not count document-style award items: the two lower-right award certificates (Les Olivalies, Expo Azeite), which carry the winner’s name, award text and signatures, and the rectangular Olive Japan award label below them. Logos printed inside such items are not counted separately either. Under this rule the page has 9 Brand Logos.

Q3 / W2: evidence for Anchor-CoT. We now test anchor-based inference directly (also stated in General response §1), which supports the value of the verified anchors that Anchor-CoT is built on.

ConditionAccuracy
Free reasoning68.8%
ADOPD regions as unlabeled anchors78.7%
Verified target anchors97.3%

W1: novelty / venue. ADOPD 2026 is submitted under the datasets and benchmarks area, and its predecessor appeared at ICLR 2024. The contribution is the new supervision layers and what they expose:

  • Document-specialized detectors fail on entity types outside layout taxonomies.
  • Fine-tuning on ADOPD adds 35–44 AP to these models over their best zero-shot class mapping.
  • VLM counting failures are largely enumeration failures that verified anchors remove.

We hope this makes the case that the released data enables measurements not possible before.

Thank you for the response and the added experiments evaluating anchor-based inference, as well as the clarifications regarding the annotation rules and venue positioning. I will maintain my current rating.

Reviewer_s7WjOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ADOPD 2026, an extension of the original ADOPD 2024 dataset aimed at moving document understanding beyond page-element localization toward semantically grounded reasoning. The main idea is to treat text blocks, visual entities, semantic labels, bounding boxes, and polygon masks as visual anchors that can serve as intermediate evidence for document-level reasoning. Building on approximately 120k document images from ADOPD 2024, the authors introduce fine-grained region-level semantic annotations, human-cleaned captions, and generated chain-of-thought (CoT) traces grounded in the annotated regions.

The paper evaluates three related tasks. The first focuses on visual-anchor localization through text-box detection and entity segmentation, comparing specialized detectors and segmentation models with a generative vision-language model. The authors also propose an agentic refinement procedure that groups fragmented text detections into coherent regions. The second task evaluates semantic tagging, examining how well models recognize the functional roles of document elements. Finally, the authors introduce DocCount, a counting benchmark derived from ADOPD 2026, containing 442 documents with verified target-region annotations and grounded reasoning traces. The benchmark evaluates modern VLMs on dense document counting, where the best reported model achieves 72.85% exact-count accuracy. Overall, the work provides a richer document-understanding resource and a collection of experiments highlighting the limitations of current models in fine-grained localization, semantic interpretation, and visual counting.

Critical strengths

I think one of the strongest aspects of this work is the effort to extend document understanding beyond conventional localization tasks. Rather than treating bounding boxes, polygon masks, and semantic labels as separate annotation types, the authors organize them around a common visual-anchor representation. This is a useful direction because many downstream document tasks require more than identifying where an element appears; they also require understanding what that element represents and how it relates to the rest of the page. The scale of the dataset is particularly valuable, with approximately 2.5 million tagged elements across visual regions and OCR text blocks. The annotation taxonomy and detailed disambiguation guidelines in Appendix A.5 are also useful, particularly given how subjective some document-specific semantic categories can be. I see this as a meaningful extension of the original ADOPD dataset that could support further research on grounded document understanding.

I also appreciate the breadth of the experimental analysis. The paper evaluates localization, segmentation, semantic tagging, and dense counting rather than focusing on a single downstream task. The localization results reveal a substantial gap between off-the-shelf models and their fine-tuned counterparts, while the semantic-tagging experiments expose difficulties with less frequent document categories. I found the agentic box-refinement experiment in Section 4.1.2 especially interesting. The authors identify over-fragmentation as a major detection error and show that grouping detector proposals with a VLM, followed by geometric validation, improves mean F1 from 33.6 to 47.7 without additional detector training. The DocCount benchmark is another useful contribution. Despite the progress of recent VLMs, the best model achieves only 72.85% exact-count accuracy, suggesting that reliable enumeration of semantically defined regions in dense documents remains challenging. These results provide several concrete directions for future work, and I think the resulting dataset and evaluation framework could be useful to the document AI community.

Critical weaknesses

My main concern is that the paper's central motivation is grounded document reasoning, but the experiments do not fully establish the benefit of the proposed Thinking-with-Anchors representation for reasoning itself. The authors introduce generated Anchor-CoT traces in which reasoning steps refer to specific document regions through polygon coordinates and visual-anchor identifiers. However, the main DocCount evaluation in Section 4.3 provides the models with only the original document image and the counting question, without the anchor annotations or generated reasoning traces. Consequently, the reported counting accuracies primarily measure how well existing VLMs perform on the new benchmark, rather than whether reasoning through explicit visual anchors actually improves their performance. I would have liked to see a controlled experiment comparing ordinary visual reasoning against an anchor-guided approach, ideally using predicted anchors at inference time. For example, does providing detected regions or encouraging explicit anchor-based reasoning improve counting accuracy, reduce missed instances, or make the generated explanations more reliable? This would help connect the proposed representation more directly to the paper's central claims.

A related concern is how the grounded reasoning traces are constructed and evaluated. Section 4.3.1 explains that the CoT generator receives the verified target polygons, class definitions, and instance assignments, while the final count and coordinates are determined from those annotations. This seems like a reasonable way to produce geometrically consistent supervision, but it does not necessarily establish that the generated explanations reflect a model's actual reasoning process. The traces may be grounded in the correct regions by construction, without demonstrating that they are useful for solving the task or that the accompanying natural-language explanations are consistently faithful. Some quantitative analysis of trace quality, grounding correctness, or downstream effectiveness would strengthen this contribution.

I also think the scope of DocCount should be discussed more carefully. Although the broader ADOPD 2026 dataset contains a much richer semantic taxonomy, DocCount evaluates only four categories: Brand Logo, Photograph, Table, and Chart/Graph. The final benchmark contains 442 documents, selected using a process that retains pages whose most frequent target class has more than three instances. This is a reasonable starting point for constructing a clean counting benchmark, but it represents a relatively restricted subset of the document-reasoning problems discussed in the introduction. In particular, the current evaluation does not directly test more complex relationships between anchors, such as matching visual elements to captions, interpreting reading order, or answering questions that require combining evidence across different regions. I would therefore be cautious about interpreting the counting results as evidence for general document-level reasoning. The benchmark is useful, but the claims should reflect the specific capabilities it currently measures.

Another point that deserves clarification is the extent of the new contribution relative to ADOPD 2024. The original dataset already contained approximately 120k document images with entity polygons, text boxes, semantic annotations, and human-refined captions. ADOPD 2026 adds much more detailed region-level tagging and grounded reasoning annotations, which are meaningful extensions, but the paper could distinguish the inherited and newly collected supervision more explicitly. I would also have liked more quantitative evidence about annotation quality. Appendix A.5 provides extensive labeling guidelines, but I could not find a systematic inter-annotator agreement analysis or an independent quality audit across the full taxonomy. This matters because several classes, such as decorative patterns, illustrations, background images, and icons, have potentially ambiguous boundaries. Moreover, the main semantic-tagging experiment in Table 2 uses only 360 class-balanced regions across a reduced 12-class taxonomy. While this provides useful diagnostic results, it is a relatively small evaluation for assessing the quality and difficulty of a dataset containing millions of annotated elements. A larger independently reviewed sample, or agreement statistics for ambiguous categories, would make the dataset contribution more convincing.

I have some reservations about the evaluation protocol as well. Appendix A.2 describes a model-guided procedure for selecting the final validation and test partitions, using model performance and challenge scores to choose among candidate splits. I understand the motivation for obtaining balanced and similarly challenging evaluation sets, and the authors provide considerable detail about the procedure. However, the main localization and semantic-tagging results appear to be reported primarily on the validation split, which is also used for checkpoint selection and model configuration. I would have preferred to see the principal results on the separate test split, particularly given that the paper has already constructed one. Reporting these results would provide stronger evidence that the findings generalize beyond the evaluation data used during model development.

Also, the efficiency claims could be better connected to the overall proposed workflow. Appendix Table 11 shows that hybrid decoding achieves a 2.2× speedup for text-box detection while retaining approximately 99% of autoregressive AP, which is encouraging. However, the same approach retains only 26% of autoregressive mask AP for polygon segmentation, suggesting that the efficiency-accuracy trade-off depends strongly on the task. Similarly, the agentic refinement procedure produces a substantial improvement in box-grouping accuracy, but it relies on additional inference from a 30B VLM, and the paper does not provide an end-to-end latency or computational-cost comparison for that workflow. Given the emphasis on efficient document reasoning, I think it would be valuable to quantify the additional inference overhead and compare the resulting accuracy-cost trade-off with simpler grouping alternatives.

But still I believe the dataset extension and empirical analysis remain valuable. My reservations are primarily about the strength of the claims concerning grounded reasoning and efficiency, rather than the usefulness of the underlying resource.

Questions for authors
  1. Could the authors provide more direct evidence that reasoning through visual anchors improves document understanding? The DocCount evaluation currently measures zero-shot counting using only the original image and question, while the Anchor-CoT traces are generated separately using verified region annotations. Have the authors considered comparing a standard VLM reasoning setup against one that explicitly uses predicted visual anchors or anchor-grounded intermediate steps? That would help establish whether the proposed representation improves reasoning accuracy or grounding consistency, rather than primarily serving as an additional annotation format.

  2. How was the quality of the new semantic annotations and Anchor-CoT traces evaluated? For the generated Anchor-CoT traces, was there any evaluation of whether the natural-language explanations correctly describe the referenced regions beyond having consistent coordinates and counts?

  3. Could the authors clarify the evaluation protocol and provide results on the held-out test split? I would also appreciate clarification on how the model-guided split selection avoids introducing evaluation bias, given that model-performance statistics are used when choosing the final partitions.

  4. What is the computational overhead of the agentic refinement and anchor-grounded generation procedures? A comparison with a simpler grouping heuristic would also help determine whether the added reasoning component provides enough improvement to justify its cost.

1 comment

Thank you for the thorough review. We address the main points below.

W1 / Q1: does reasoning through anchors help? We ran the controlled comparison with one backbone (Qwen3.6-35B-A3B), the same images and the same token budget (General response §1):

ConditionAccuracy
Direct answer50.7%
Standard reasoning68.8%
Reasoning over all annotated ADOPD regions as unlabeled numbered anchors78.7%
Oracle target anchors97.3%
  • Grounding consistency: in a selection variant of the all-regions condition, the model's selected anchors match the verified targets with precision 0.94 and recall 0.95.
  • Exact-set match: 76.5% of pages.
  • Token cost: anchors reduce reasoning tokens when they are correct (median 1,534 vs 2,127).

We agree this shows the value of verified anchors at inference, not of training on Anchor-CoT. We will scope the claim accordingly.

W2 / Q2: trace quality. The coordinates and counts in every Anchor-CoT trace are ground truth, not generated. The generator (GPT-5.5) writes only the natural-language reasoning, and it refers to each region by a placeholder ([[1]], [[2]], …) rather than by coordinates. We then assemble the trace from the DocCount ground truth (Sec. 4.3.1). So no coordinate or count in a trace comes from the generator: they are exactly the benchmark's verified annotations.

W3: DocCount scope. Agreed. DocCount measures dense counting of four semantically defined region classes, chosen for their separable definitions (§4.3.1). It does not test caption matching, reading order or multi-region evidence. We will restate the claims as "dense, semantically grounded enumeration" and list these relation types as future layers.

  • Class denominators: 163 / 180 / 75 / 24.
  • Count distribution: 4–45 per page (median 5). Filtering yield: 1,224 candidate pages → 442 (36.1%). Details in Appendix A.15.

W5 / Q3: test split and split selection.

  • (a) Test-split results: frozen checkpoints and frozen thresholds; validation numbers from the same scorer (AP50/AP75/mF1 in the new Appendix A.14):
ModelDoc2Box valDoc2Box testDoc2Mask valDoc2Mask test
YOLOv12-X68.568.5——
RF-DETR-N / RF-DETR-Seg-N50.450.356.255.8
RF-DETR-L / RF-DETR-Seg-2XL64.064.271.270.9
DocLayout-YOLO, fine-tuned (new)66.766.859.659.3
RT-DETR-L, fine-tuned (new)67.367.062.361.7
RT-DocLayout, fine-tuned (new)67.267.261.961.5
  • (b) Why selection does not bias these results: candidate partitions were scored by how similar validation and test are to each other and to the full held-out pool (taxonomy, language, complexity, and the val–test gap of an ensemble of zero-shot calibration models with mean AP ≈ 2.4; Table 7). Apart from a small mean-AP term that rules out an ultra-hard split, the objective rewards matched splits, not low or high scores of any model. (a) is the direct check: every row differs by at most 0.6 AP between validation and test, and the ranking within each task is the same except for one swap between two models 0.1–0.2 AP apart (RT-DETR-L and RT-DocLayout on Doc2Box).
Reviewer_YSBwOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ADOPD 2026, an extension of the approximately 120,000-page ADOPD corpus with revised captions, region-level semantic labels, and polygon-grounded counting narratives. It evaluates document localization, semantic tagging, and document counting through the 442-example DocCount benchmark. The experiments identify difficulties in contextual region classification and exact enumeration, with the strongest evaluated counting model reaching 72.85% accuracy. An additional agentic grouping procedure improves text-block decomposition by merging detector proposals, increasing mean F1 from 33.6 to 47.7 under the same proposal-pool evaluation.

Critical strengths
  1. The corpus reports approximately 2.5 million tagged elements, including 1.22 million visual polygons and 1.25 million text blocks. Adding contextual semantic roles to these regions provides supervision for interpreting document elements, beyond recovering their geometry. The annotation vocabulary covers document-specific distinctions such as photographs, background images, decorative patterns, logos, and textual functions.

  2. The evaluation holds the target region fixed and compares marked-page and page-plus-crop inputs on a class-balanced set. The results expose substantial failures on background and decorative categories that an aggregate localization score would not reveal. The decoding ablation evaluation distinguishes some classification failures from output-generation failures: length-normalized candidate scoring recovers Visual Motif recall from 0/30 to 7/30 without retraining.

Critical weaknesses
  1. This paper evaluates YOLOv12, RF-DETR, SAM3/SAM3.1, and LocateAnything, but does not include SOTA models or document-specialized models such as RT-DETR, RT-DocLayout, and DocLayout-YOLO. RT-DocLayout is particularly relevant because it jointly predicts document boxes and masks, while DocLayout-YOLO is already cited in the paper. Their inclusion would better establish how current document-specialized systems perform under the proposed benchmark evaluation. Moreover, near-zero performance from generic pretrained detectors shows poor transfer for those checkpoints, but the difficulty document-specialized pretrained models face remains untested. Therefore, the current results make it hard to provide a comprehensive evaluation baseline for characterizing the capabilities of existing document-localization systems.

  2. The benefit of anchor-based reasoning is not directly tested. Anchor-CoT generation receives verified target polygons, whereas DocCount evaluation supplies only the original image and counting question and scores the final integer. No controlled ablation study experiment establishes whether predicted-anchor reasoning improves counting or produces more faithful evidence than ordinary CoT. Besides, DocCount contains 442 pages from four classes and retains the most frequent target class with more than three instances, while tagging uses 30 regions per class. Class denominators, count distributions, and filtering yields are needed to interpret representativeness.

Questions for authors
  1. Could the authors compare direct counting, ordinary CoT, and reasoning with predicted anchors using the same backbone and a fixed inference budget, and report exact-count accuracy alongside grounding precision/recall and count–anchor consistency?

  2. Could they also include an oracle-anchor condition to help distinguish perception limitations from reasoning limitations?

  3. Could the authors include RT-DETR, RT-DocLayout, and DocLayout-YOLO as baselines for the applicable localization tasks?

  4. How would the native layout classes be mapped to the respective task targets, OCR text blocks for Doc2Box and foreground visual entities for Doc2Mask, without indiscriminately collapsing all categories into a single class?

3 comments

Thank you for the detailed review and the concrete baseline and ablation requests. We ran both.

W1 / Q3: document-specialized baselines. We added DocLayout-YOLO (DocStructBench weights), RT-DETR-L (COCO weights) and RT-DocLayout (official pretrained weights; boxes + masks). Setup and tables are in the new Appendix A.13.

  • Zero-shot: each model's native classes are mapped to the task targets (Q4 below).
  • Fine-tuning: on the same 80k train split: 10 epochs, multi-scale {384, 640, 768, 1024}, effective batch 128, AdamW + cosine, final epoch (no checkpoint selection). DocLayout-YOLO uses the YOLOv12 recipe; RT-DETR uses the RF-DETR optimizer settings; RT-DocLayout uses its official recipe with these shared settings.
  • Doc2Mask scoring rule: every model is scored by mask IoU against the human mask. A box-only model's box is scored as a filled rectangle; 88.6% of the GT masks have IoU ≥ 0.75 with their own box, so this is a fair output for most entities.

Doc2Box (text blocks, 19,440 validation pages):

ModelZS AP (class map)FT APFT AP50FT AP75FT APS/M/LFT mF1
RT-DETR-L (COCO)0.067.381.373.029.0/53.6/73.476.4
DocLayout-YOLO32.1 (text classes; all: 28.2)66.780.573.025.1/50.9/73.975.0
RT-DocLayout30.7 (text classes; all: 22.6)67.281.873.727.6/52.6/73.976.0
YOLOv12-X0.168.582.574.828.4/54.4/75.077.0

Doc2Mask (foreground entity masks, 19,072 validation pages):

ModelZS AP (class map)FT APFT AP50FT AP75FT APS/M/LFT mF1
RT-DETR-L (box)2.562.379.666.729.6/53.0/66.775.5
DocLayout-YOLO (box)15.6 (figure+table; all: 4.1)59.676.764.624.8/50.2/64.273.3
RT-DocLayout (mask)20.1 (image/chart/seal/table; all: 9.4)61.975.868.222.8/49.5/67.972.6
YOLOv12-Seg-X2.264.480.576.730.0/59.5/76.877.1

Findings:

  1. Zero-shot: document pre-training helps (≈0 → 31–32 AP on Doc2Box).
  2. Fine-tuned: the three models land within 1–2 AP of YOLOv12-X on Doc2Box. On Doc2Mask they stay 2.1–4.8 AP below the YOLOv12-Seg-X masks.
  3. Fine-tuning gains are the largest single effect: +34.6 (DocLayout-YOLO, Doc2Box) and +41.8 (RT-DocLayout, Doc2Mask). ADOPD supervision, not document pre-training alone, closes most of the gap.

On the test split (19,407 Doc2Box / 19,087 Doc2Mask pages), every fine-tuned row is within 0.6 AP of its validation value: Doc2Box 66.8 / 67.0 / 67.2, all below YOLOv12-X (68.5); Doc2Mask 59.3 / 61.7 / 61.5 (DocLayout-YOLO / RT-DETR-L / RT-DocLayout). Full test-split table: Appendix A.14, and we'll merge it to Table 1 in the main text in the camera-ready version.

Q4: class mapping. We map each model's native classes to the task target at scoring time and also report collapse-all as a reference.

  • Doc2Box (text blocks):
    • DocLayout-YOLO: title, plain text, abandon, figure/table caption, table footnote, isolated formula, formula caption.
    • RT-DocLayout: all 17 text-like classes (text, paragraph/doc/figure titles, header, footer, footnotes, abstract, reference, formula, …).
  • Doc2Mask (foreground visual entities):
    • DocLayout-YOLO: figure, table.
    • RT-DocLayout: image, chart, seal, table.
  • RT-DETR (COCO): has no document classes, so only collapse-all applies.
  • Effect: the mapping raises zero-shot Doc2Mask AP over collapse-all by about 3.8× for DocLayout-YOLO (4.1 → 15.6) and 2.1× for RT-DocLayout (9.4 → 20.1).

Per-label recall at IoU 0.5 (Appendix A.13) shows why mapping alone cannot close the gap: layout taxonomies have no class for many ADOPD entity types.

Entity type (n)Zero-shot recall, RT-DocLayoutZero-shot recall, DocLayout-YOLO
Line/Divider (2,062)18%17%
Dialog Box (2,098)29%19%
Text-block entity (43,504)46%35%
Color Block (3,121)44%45%
Icon (14,232)64%47%
Brand Logo (19,387)68%48%
Photograph / Table / Chart85–87%71–86%

We read this as a dataset-utility result: ADOPD's entity notion is broader than existing layout taxonomies.

W2 / Q1 / Q2: anchor ablation and oracle anchors. See General response §1 for the full table.

  • Q1 (same backbone and budget):
    • direct answer 50.7%;
    • free CoT 68.8%;
    • all ADOPD regions as unlabeled anchors 78.7%, with grounding precision/recall 0.94/0.95 for the selected anchors.
  • Q2 (oracle): target-only anchors reach 93.9% (thinking off) and 97.3% (on), with fewer reasoning tokens than free CoT (1,534 vs 2,127 median).
  • Perception vs. reasoning: given correct anchors, the model counts almost perfectly; the dominant failure is perceptual enumeration.

W2 (second part): DocCount representativeness. Details in the new Appendix A.15.

  • Class denominators: Brand Logo 163, Photograph 180, Table 75, Chart/Graph 24 pages (442 total).
  • Count distribution: 4–45 regions per page, median 5, mean 5.8 (Brand Logo has the long tail: mean 6.7, max 45; Chart/Graph max 8). 204 pages have exactly 4.
  • Filtering yield: 1,224 validation pages (7,837 polygons) met "top-1 target class has more than 3 instances" and entered verification; 442 survived label verification and the manual missing-instance review (36.1%; Brand Logo 23.5%, Photograph 45.8%, Table 78.9%, Chart 57.1%).

Thank you for the additional baselines, class-mapping analysis, and counting ablations, which substantially address my earlier concerns. The anchor-assisted counting gains are encouraging, although it remains unclear whether they persist with model-predicted anchors rather than the provided ADOPD regions. I maintain my original assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful reading. All four found the enriched dataset useful (YSBw, s7Wj, 7Fgx, 833W). The main concerns were: (1) whether anchor-based reasoning is actually useful and supported by evidence; (2) novelty and venue fit.

1. Do anchors help? (YSBw W2/Q1/Q2, s7Wj W1/Q1, 7Fgx W2/Q3, 833W W2)

One backbone (Qwen3.6-35B-A3B), the 442 DocCount questions, and the same generation budget in every condition. Every prompt is the released DocCount prompt; we vary only the region evidence, added after the question as a numbered list of box coordinates without class names. The revised appendix gives the exact prompt and input of every condition. Exact-count accuracy, thinking off / on:

Conditionthinking offthinking onmedian output tokens (on)
A. Direct answer (no reasoning allowed)50.7——
A. Free reasoning (DocCount prompt)—68.82,127
B. All annotated regions given as unlabeled numbered boxes (targets + distractors); the model counts55.978.73,897
C. Target regions given as numbered boxes in the prompt (oracle)93.997.31,534
  • Free reasoning: +18 points over a direct answer.
  • Unlabeled regions: adding ADOPD's unlabeled region anchors gives a further +9.9. The model must still decide class membership: in a selection variant with the same input (Appendix A.12), the regions it selects have precision/recall 0.94/0.95 (F1 0.95; exact-set match 76.5%).
  • Oracle target anchors: accuracy reaches 97.3%, while the model uses 28% fewer reasoning tokens than free reasoning.
  • Significance: on the same 442 questions (exact McNemar; paired-bootstrap 95% CI), all-region → target anchors is +18.6 pp [14.9, 22.2], p < 1e-20.

So most of the model's DocCount error is enumeration/localization, not arithmetic. This is the information the ADOPD 2026 anchors carry.

We present Anchor-CoT as a supervision resource whose value for training is left to future work (Appendix A.12).

2. Novelty and venue (7Fgx W1, 833W W1/W7)

ADOPD 2026 is submitted under the datasets and benchmarks area. Its predecessor, ADOPD (Gu et al., 2024), appeared at ICLR 2024. The new layers are:

  • semantic roles on human-drawn anchors;
  • cleaned captions;
  • verified dense-counting questions with anchor-grounded traces.

The new experiments show what these layers reveal that geometry-only benchmarks do not:

  • Off-the-shelf document-layout models miss many document entities that their taxonomies lack: zero-shot recall is 17–29% for dividers and dialog boxes, and 44–64% for color blocks and icons.
  • VLM counting errors are largely enumeration errors that verified anchors remove.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.