acceptodds
Under review as a conference paper at ICLR 2027

How Far Is Document Parsing from Solved? PureDocBench: A Source-Traceable Benchmark across Clean, Degraded, and Real-World Settings

19 views

Abstract

Document parsing models are improving rapidly, but existing benchmarks are still insufficient for a comprehensive evaluation of their capabilities. We introduce PureDocBench, a benchmark with 10 domains, 66 subcategories, and 1,475 pages. Each page has matched Clean, Digital, and Real versions, covering clean renders, simulated degradation, and real capture or sharing conditions. The resulting 4,425 images have source-derived annotations for text, formulas, tables, and reading order, checked through automated validation and human review. Keeping page content and annotations fixed allows direct comparisons across image conditions. We evaluate 58 models, including pipeline-based parsers, end-to-end parsers, and general vision-language models. The best three-track average score is 79.69 out of 100, and the mean across models is 63.84. Scores drop by 3.34 points on Digital and 11.61 points on Real on average, relative to Clean. TeleOCR leads on Clean, GLM-5.3-Flash on Digital, and Gemini 3.6 Flash on Real. We release the benchmark, source documents, generation tools, and evaluation code.

Primary area:
datasets and benchmarks
Keywords:
Document parsing, OCR, vision-language models

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_GWMYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PureDocBench, a benchmark for evaluating document parsing under clean, digitally degraded, and real-world capture conditions. It contains 1,475 source pages across 10 domains and 66 subcategories, with 4,425 images across the three settings. The authors use HTML/CSS-generated documents to obtain source-linked annotations and evaluate 58 models using established document-parsing metrics for text, formulas, tables, and reading order.

The main goal is to study how document-parsing performance changes as documents move from clean inputs to degraded or real-world images. The benchmark provides a structured way to compare models across these conditions. However, the paper's contribution is primarily a new dataset and evaluation setup rather than a new parsing method or evaluation metric. Its novelty therefore depends on how much additional insight and evaluation capability it provides beyond existing benchmarks.

Critical strengths

1. Source-linked annotations and controlled evaluation. Generating documents from HTML/CSS gives the authors access to structured source information and makes it possible to evaluate models under different visual conditions while keeping the underlying content consistent. This is useful for studying how degradation affects parsing performance and can make annotation easier to verify, but is not something very novel and has been already used to generate certain other benchmarks (also cited in the paper).

2. Evaluation across multiple document types and conditions. The benchmark covers 10 domains, 66 subcategories, and 58 models. Evaluating clean, digitally degraded, and real-world images in one framework can help researchers identify where models are robust and where they fail. The benchmark could be useful to the community if its added value over existing datasets is demonstrated more clearly.

Critical weaknesses
  1. The novelty over existing benchmarks is not sufficiently established. Several recent benchmarks already evaluate document parsing on photographed or degraded documents, compare digital and real-world inputs, or provide broad coverage of document types. In particular, Real5-OmniDocBench studies matched document content under physical capture conditions, while DocPTBench and MDPBench also consider photographed documents and real-world variation. OmniDocBench already provides a broad evaluation framework for text, formulas, tables, and reading order. As a result, the paper needs to explain more clearly what new research questions PureDocBench enables that these existing benchmarks cannot address. The combination of existing ideas in a new dataset is useful, but it is not by itself a strong enough novelty argument.

  2. The connection between synthetic and real-world documents needs stronger validation. The source-linked annotations are a useful feature, but generating documents with HTML/CSS does not necessarily capture the layout, content, and visual complexity of naturally occurring documents. This matters because the paper aims to draw conclusions about real-world document parsing. The authors should provide stronger evidence that the synthetic documents and degradation scenarios reflect real-world cases, and clarify which conclusions are supported by the real-image evaluation rather than the synthetic setup alone.

Questions for authors
  1. What can PureDocBench measure that existing benchmarks cannot? Please compare the benchmark directly with the most relevant prior work, especially Real5-OmniDocBench, DocPTBench, MDPBench, and OmniDocBench. Beyond the number of domains or evaluated models, what specific capability or research question is uniquely enabled by PureDocBench?

  2. How representative are the HTML/CSS-generated documents of real-world documents? Do the authors have evidence that the document structures and degradation scenarios reflect naturally occurring documents? How well do the findings from the synthetic tracks agree with results on the real-world track?

  3. What new findings emerge from this benchmark? Which conclusions about model behavior could not have been drawn from existing benchmarks? A direct comparison or analysis showing that PureDocBench reveals previously unobserved failure patterns would strengthen the paper's contribution.

5 comments
AuthorsOpenReview ↗

Response to Reviewer GWMY (Part 1/4): Weakness 1, Question 1, and Question 3

Dear Reviewer GWMY, we sincerely appreciate your thorough review and the effort behind it. Your comments on how PureDocBench compares with closely related benchmarks and on the connection between generated and real-world documents have helped us state our contribution much more clearly. We respond to each of them below, quoting your comments first.

Weakness 1. The novelty over existing benchmarks is not sufficiently established. Several recent benchmarks already evaluate document parsing on photographed or degraded documents, compare digital and real-world inputs, or provide broad coverage of document types. In particular, Real5-OmniDocBench studies matched document content under physical capture conditions, while DocPTBench and MDPBench also consider photographed documents and real-world variation. OmniDocBench already provides a broad evaluation framework for text, formulas, tables, and reading order. As a result, the paper needs to explain more clearly what new research questions PureDocBench enables that these existing benchmarks cannot address. The combination of existing ideas in a new dataset is useful, but it is not by itself a strong enough novelty argument.

Question 1. What can PureDocBench measure that existing benchmarks cannot? Please compare the benchmark directly with the most relevant prior work, especially Real5-OmniDocBench, DocPTBench, MDPBench, and OmniDocBench. Beyond the number of domains or evaluated models, what specific capability or research question is uniquely enabled by PureDocBench?

Question 3. What new findings emerge from this benchmark? Which conclusions about model behavior could not have been drawn from existing benchmarks? A direct comparison or analysis showing that PureDocBench reveals previously unobserved failure patterns would strengthen the paper's contribution.

Thank you for these thoughtful questions and for pointing us to the most closely related benchmarks. OmniDocBench (Ouyang et al., CVPR 2025) established the evaluation framework that PureDocBench adopts, and the recent Real5-OmniDocBench (Zhou et al., ECCV 2026), DocPTBench (Du et al., NeurIPS 2026), and MDPBench (Li et al., NeurIPS 2026), which are concurrent with our work, have each advanced the evaluation of parsers on photographed and physically recaptured documents. We see PureDocBench as complementary to these benchmarks. Below, we first compare PureDocBench directly with these four benchmarks and then describe the two questions it allows us to study, together with the findings they lead to.

Direct comparison. OmniDocBench v1.5 offers 1,355 collected pages in nine document types with detailed annotations. Real5-OmniDocBench recaptures all of these pages under five physical conditions (scanning, warping, screen photography, illumination, and skew) and reuses their annotations, which gives a large-scale view of how capture affects identical content. DocPTBench pairs 981 OmniDocBench pages with simulated photographs, photographs 100 of them under four physical conditions, and adds document translation. MDPBench extends photographed-document parsing to 17 languages, with three photographs of each of its 850 pages. The table below places PureDocBench alongside them.

BenchmarkPagesPage sourceDocument typesDigitally degraded versionReal acquisitionReference annotations
OmniDocBench v1.51,355Collected PDFs9 types: academic papers, books, textbooks, exam papers, magazines, newspapers, notes, research reports, slidesNoNoModel pre-annotation with human correction
Real5-OmniDocBench1,355OmniDocBench v1.5Same as OmniDocBenchNoAll pages, 5 conditionsInherited from OmniDocBench
DocPTBench981OmniDocBenchSame as OmniDocBenchAll pages (simulated photographs)100 pages, 4 conditionsInherited from OmniDocBench, with added translation references
MDPBench850Collected documents in 17 languagesAcademic papers, business reports, educational materials, notes, archives, newspapers, textbooks, comicsNoAll pages, 3 photographs eachModel-assisted with human review
PureDocBench1,475Generated HTML/CSS, sources retained10 domains and 66 subcategories, including invoices, bank statements, medical records, contracts, government documents, waybills, and certificatesAll pagesAll pages (photographs of prints, photocopies, and screens; social-media screenshots)New, human-reviewed, shared by the three tracks

Our response to Weakness 1, Question 1, and Question 3 continues in Part 2/4.

AuthorsOpenReview ↗

Response to Reviewer GWMY (Part 2/4): Weakness 1, Question 1, and Question 3 (continued)

This comment continues our response to Weakness 1, Question 1, and Question 3 from Part 1/4.

How PureDocBench differs. PureDocBench differs from these benchmarks in three ways. First, its 1,475 pages are newly generated and do not overlap with the pages of existing benchmarks; their HTML/CSS sources are retained, and their references are newly annotated, human-reviewed, and shared by all three tracks. Second, 49 of its 66 subcategories, covering 1,077 pages, are document types outside the categories of OmniDocBench and MDPBench, such as invoices, bank statements, tax and insurance documents, medical records and bills, contracts, court and government documents, waybills and bills of lading, and certificates. In this count, related types such as theses, technical reports, syllabi, and business or audit reports are treated as covered. Third, every page has both a digitally degraded image and a real acquisition. Among the four benchmarks, only DocPTBench contains both simulated and physically photographed images, and the photographs cover 100 of its 981 pages. The second and third differences let us study two questions directly.

How well digital degradation reflects real acquisition. Digital degradation is commonly used to test and train parsers for robustness, since it is inexpensive and can be applied to any page. Because both versions of every page are scored against the same reference, the two can be compared on identical content across the whole benchmark. With the ten degradation profiles of our Digital track, which include aging, photocopying, compression, uneven lighting, and geometric distortion, the 58 models lose 3.34 Overall points on average, compared with 11.61 points under real acquisition, so the Digital track reproduces about 29% of the real loss. The two tracks also order the models differently. The Spearman correlation between the models' digital and real losses is 0.64, and TeleOCR, which ranks second on Digital, ranks eighth on Real, while Gemini 3.6 Flash moves from third to first. The difference is largest for general VLMs, which lose 1.84 points under digital degradation and 9.30 points under real acquisition, so a digital-only evaluation would suggest that they are almost unaffected. In line with DocPTBench, specialist parsers lose more than general VLMs under real acquisition (14.23 points for pipeline and 12.99 points for end-to-end specialists), and in line with MDPBench, which finds Gemini 3 Pro relatively robust on photographed documents, the two Gemini models lose the least among the 58 models. These results show that robustness measured with digital degradation alone underestimates the loss under real acquisition and gives a different picture of which models are most robust.

How accurately business and administrative documents are parsed. Business and administrative documents are common in commercial document processing but are difficult to collect and release because they contain personal and financial information, and generating the pages makes it possible to include them. To examine what they reveal, we compared the academic, education, and publishing domains, whose documents are closest to those in existing benchmarks, with the other seven domains, using the 53 models with per-domain results in Appendix G. Both groups are generated and annotated in the same way, so the differences can be attributed to the document types. On the Clean track, all 53 models have lower text accuracy on the seven domains, by 11.0 points on average, and 51 and 52 models have lower table and reading-order scores, by 10.7 and 13.2 points on average. The gaps persist on the Digital and Real tracks and are largest on certificates and logistics documents, whose mean text accuracy on Clean is 56.4, compared with 85.3 on academic pages. Model rankings are largely consistent between the two groups (Spearman ρ=0.93\rho = 0.93 for text). Evaluation on the document types found in existing benchmarks therefore ranks parsers similarly, while overestimating their accuracy on business and administrative documents by about ten points.

We acknowledge that the submission did not state these two questions explicitly, and your comments helped us articulate them. In the revision, we will add the comparison table to Section 2.2, introduce the two questions in the introduction, report both analyses in Section 3 with per-model and per-domain results in the appendix, and note where our findings agree with DocPTBench and MDPBench.

AuthorsOpenReview ↗

Response to Reviewer GWMY (Part 3/4): Weakness 2 and Question 2

Weakness 2. The connection between synthetic and real-world documents needs stronger validation. The source-linked annotations are a useful feature, but generating documents with HTML/CSS does not necessarily capture the layout, content, and visual complexity of naturally occurring documents. This matters because the paper aims to draw conclusions about real-world document parsing. The authors should provide stronger evidence that the synthetic documents and degradation scenarios reflect real-world cases, and clarify which conclusions are supported by the real-image evaluation rather than the synthetic setup alone.

Question 2. How representative are the HTML/CSS-generated documents of real-world documents? Do the authors have evidence that the document structures and degradation scenarios reflect naturally occurring documents? How well do the findings from the synthetic tracks agree with results on the real-world track?

Thank you for raising this important point. We agree that pages generated from HTML/CSS cannot capture every aspect of naturally occurring documents. We chose generation mainly because it makes a few things possible that are hard to achieve with collected documents, and we see it as a complement to benchmarks built from them. For example, business documents such as invoices and medical records can be released together with their sources, the annotations can be checked against those sources, and the three images of each page can share one reference. Below, we present the evidence on the generated documents and the acquisition conditions, and then clarify which conclusions the Real track supports and how closely the synthetic and Real tracks agree.

Documents. In an additional human study, three raters who were not involved in building the benchmark scored 50 Clean pages, stratified by document type, layout complexity, and orientation and shown in random order with source information hidden. On a five-point scale from 1 (clearly synthetic) to 5 (fully natural), their ratings of typography, spacing, alignment, element placement, and overall coherence averaged 4.13 (95% CI 3.99 to 4.24; Krippendorff's α=0.63\alpha = 0.63), and 40 of the 50 pages averaged at least 4. The same raters scored the plausibility of the text for its document type at 4.28. We also compared our reference text with that of the 1,284 naturally sourced pages with body text in OmniDocBench v1.5 (Ouyang et al., CVPR 2025). Punctuation and symbols make up a similar share of the text (9.9% versus 9.7%), while our pages contain about twice as much text, about three times the digit density, and more mixed Chinese and English text (48% versus 15% of pages), in line with the amounts, dates, and codes that fill invoices, statements, and medical records.

Acquisition. The Real images come from actual capture and sharing: 1,275 phone photographs of printed pages, photocopies, and screens, and 200 screenshots passed through social-media compression. They cover flat, creased, bent, and photocopied paper, normal, low, flash, shadowed, and outdoor lighting, and top-down, oblique, and handheld views. The ten profiles of the Digital track, such as aging, photocopying, compression, uneven lighting, and geometric distortion, are simulated, and on the same pages they reproduce about 29% of the Real loss (3.34 versus 11.61 points). We therefore base conclusions about real-world robustness on the Real track.

Our response to Weakness 2 and Question 2 continues in Part 4/4.

Reviewer_DvkFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PureDocBench, a source-traceable benchmark designed to evaluate document parsing systems under clean, digitally degraded, and real-world acquisition conditions. The benchmark comprises 1,475 source pages spanning 10 domains and 66 subcategories, resulting in 4,425 matched images across three evaluation tracks. By generating documents from HTML/CSS sources and deriving annotations through automated validation and human review, the authors aim to improve annotation reliability and benchmark scalability. The paper evaluates 58 models from three architectural families, revealing substantial performance degradation under challenging acquisition conditions and notable differences in parsing capabilities.

Critical strengths
  1. The paper introduces a source-traceable annotation pipeline. By generating documents from HTML/CSS sources and linking annotations to the original document content, the authors provide a practical approach to constructing and maintaining document parsing benchmarks. The Clean, Digital, and Real tracks enable direct comparisons across different acquisition conditions, helping distinguish parsing accuracy from robustness under degradation.
  2. The evaluation of 58 models is extensive, covering pipeline-based specialists, end-to-end specialists, and general-purpose VLMs. The results reveal interesting differences across model families.
  3. The paper presents numerous case studies illustrating practical failure modes, including content omissions, numerical substitutions, and structural errors. These examples provide useful insights into the remaining challenges in document parsing.
Critical weaknesses
  1. Prior studies have synthesized document images from structured markup languages such as HTML/CSS and LaTeX, deriving supervision directly from source files. Examples include DocBank, CoSyn, Dolphin, and Image2Struct. Thus, PureDocBench appears to apply established document synthesis techniques to benchmark construction rather than introduce a fundamentally new generation or annotation method.
  2. The documents are generated from LLM-produced HTML/CSS rather than collected from real-world sources. Although the Real track introduces physical acquisition conditions, the underlying content and layouts remain synthetic, potentially limiting the benchmark's representativeness of real-world document distributions.
  3. The paper uses LLMs to generate HTML/CSS documents and then extract ground-truth annotations from the same sources. Since these sources already contain explicit textual and structural information, the need for LLM-based extraction instead of deterministic parsing is unclear, particularly for reading order. The paper also lacks details on how reading order is determined, whether rendered layouts are considered, and how many annotation errors are identified and corrected during human review. These omissions make it difficult to assess annotation reliability.
  4. My understanding is that a key motivation for the source-traceable annotation pipeline is to support automated benchmark construction at scale. However, source traceability does not guarantee valid image-level ground truth. For example, text present in source files may be illegible in rendered images due to small fonts or poor rendering quality. The paper should explain how such issues are systematically detected and filtered, and how annotation quality is maintained at scale. Otherwise, undetected annotation noise may undermine the reliability of benchmark evaluations.
Questions for authors
  1. How can the authors demonstrate that the LLM-assisted HTML/CSS documents sufficiently resemble real-world documents in terms of content, layout, and visual structure? What measures are taken to ensure and validate that the generated documents follow distributions representative of real-world documents?
  2. Why are multiple rounds of human cross-review still necessary after generating documents directly from source code? How much annotation cost or human effort does the proposed pipeline actually save compared with traditional annotation workflows?
  3. Among the annotations for text, tables, formulas, and reading order, which are extracted deterministically from HTML/CSS, and which require LLM inference? Why cannot these annotations be obtained entirely through source-code parsing and browser layout information? What is the accuracy of the initial LLM-assisted annotations? How many errors were identified and corrected during the multiple rounds of human review, and which annotation types accounted for most of these errors?
  4. How does the proposed source-traceable pipeline ensure that annotation quality remains reliable as the benchmark scales to substantially larger datasets?
8 comments
AuthorsOpenReview ↗

Response to Reviewer DvkF (Part 1/5): Weakness 1

Dear Reviewer DvkF, we are sincerely grateful for the time and care you devoted to our paper. Your comments on the relation to prior synthesis work, the realism of the generated documents, the annotation pipeline, and annotation quality at scale have helped us improve the paper substantially. We respond to each of them below, quoting your comments first.

Weakness 1. Prior studies have synthesized document images from structured markup languages such as HTML/CSS and LaTeX, deriving supervision directly from source files. Examples include DocBank, CoSyn, Dolphin, and Image2Struct. Thus, PureDocBench appears to apply established document synthesis techniques to benchmark construction rather than introduce a fundamentally new generation or annotation method.

Thank you for pointing us to these important works. We fully agree that synthesizing document images from markup and deriving supervision from source files are well-established techniques, and PureDocBench builds directly on this line of research. DocBank (Li et al., COLING 2020) showed that LaTeX sources can provide token-level layout labels at scale. CoSyn (Yang et al., ACL 2025) demonstrated that LLM-generated code can be rendered into diverse text-rich images, and our generation step follows the same idea. Dolphin (Feng et al., ACL 2025) rendered Wikipedia HTML into millions of training pages, and Image2Struct (Roberts et al., NeurIPS 2024) evaluated models by asking them to write code that reproduces a given image.

What PureDocBench adds. Building on these ideas, PureDocBench focuses on an evaluation question that we hope complements the works you mention: how a parser's recovery of text, tables, formulas, and reading order changes when the same page is captured under different conditions. It takes markup-based synthesis one step beyond the works above: each of the 1,475 generated pages is also degraded digitally and acquired in the real world, as phone photographs of prints, photocopies, and screens or as screenshots shared through social media, and all three versions are scored against the same source-linked, human-reviewed reference.

Two results from this design. First, the 58 evaluated models lose 3.34 points on Digital and 11.61 points on Real on average relative to Clean, so the digital degradations reproduce about 29% of the loss under real acquisition. The two losses also rank the models differently (Spearman ρ=0.64\rho = 0.64), and the leading model differs across the three tracks. Second, generating the source lets us include business and administrative documents that are difficult to collect and release, such as invoices, medical records, and waybills: 49 of our 66 subcategories (1,077 pages) lie outside the categories of OmniDocBench (Ouyang et al., CVPR 2025) and MDPBench (Li et al., NeurIPS 2026). On the seven domains other than academic, education, and publishing, all 53 models with per-domain results have lower text accuracy on Clean than on these three, by 11.0 points on average.

We acknowledge that the submission did not discuss this line of work sufficiently. In the revision, we will expand the discussion of markup-based document synthesis in Section 2.2 to cover DocBank, CoSyn, Dolphin, Image2Struct, Nougat (Blecher et al., ICLR 2024), and READoc (Li et al., Findings of ACL 2025), add Image2Struct and Dolphin to Table 1, and revise the introduction and contribution list to credit these techniques explicitly and describe our contribution more precisely.

AuthorsOpenReview ↗

Response to Reviewer DvkF (Part 2/5): Weakness 2 and Question 1

Weakness 2. The documents are generated from LLM-produced HTML/CSS rather than collected from real-world sources. Although the Real track introduces physical acquisition conditions, the underlying content and layouts remain synthetic, potentially limiting the benchmark's representativeness of real-world document distributions.

Question 1. How can the authors demonstrate that the LLM-assisted HTML/CSS documents sufficiently resemble real-world documents in terms of content, layout, and visual structure? What measures are taken to ensure and validate that the generated documents follow distributions representative of real-world documents?

Thank you for raising this important point. We agree that the content and layouts of PureDocBench are generated, and that a generated collection cannot fully reproduce the distribution of real-world documents. Below we describe the measures we take during construction, the evidence on layout, visual structure, and content, and how we handle distribution. We will also state this limitation clearly in the revision.

Measures during construction. We chose the 10 domains and 66 subcategories based on three kinds of evidence: market reports on intelligent document processing, which identify financial services as the largest end-use sector (Fortune Business Insights, 2026; Grand View Research, 2025); the document types for which major cloud services provide dedicated APIs, such as invoices, receipts, and lending documents (e.g., Amazon Textract); and the business documents that OmniDocBench (Ouyang et al., CVPR 2025), whose categories centre on academic and publishing sources, rarely covers. Finance and medical documents have the most subcategories (10 each). For each subcategory, the generation prompt specifies the document type, language, layout, and element composition, and human screening removed pages with collapsed layouts or repetitive content.

Layout and visual structure. To check visual realism, we conducted an additional human study. Three raters who were not involved in building the benchmark scored 50 Clean pages, stratified by document type, layout complexity, and orientation, and shown in random order with source information hidden. They rated typography, spacing, alignment, element placement, and overall coherence on a five-point scale from 1 (clearly synthetic) to 5 (fully natural). The mean score was 4.13 (95% CI 3.99 to 4.24), 40 of the 50 pages averaged at least 4, only one page averaged 2 or below, and Krippendorff's α\alpha was 0.63.

Content. In the same human study, the raters also scored whether the text is plausible for its document type, judging fields, terminology, figures, and internal consistency, with a mean score of 4.28 out of 5. We further compared our reference text with that of the 1,284 naturally sourced pages with body text in OmniDocBench v1.5. Punctuation and symbols make up a similar share of the text (9.9% versus 9.7%), while our pages contain about twice as much text, about three times the digit density, and more mixed Chinese and English text (48% versus 15% of pages). As a result, each page asks a parser to recover more figures, identifiers, and mixed-script text, such as amounts, dates, and codes on invoices and medical records. We will add these statistics to the appendix.

Acquisition. On the image side, the 1,475 Real images come from actual capture and sharing: 1,275 phone photographs of printed pages, photocopies, and screens, and 200 screenshots passed through social-media compression. As you point out, the synthetic component therefore lies in the content and layout.

Distribution. Each subcategory has 20 to 32 pages so that every document type is covered. As a result, the proportions of document types follow this coverage design and differ from real-world frequencies. We recognize that the evidence above concerns visual and content plausibility and does not establish that the collection matches real-world document distributions. We therefore see PureDocBench as a complement to benchmarks built from naturally sourced documents, such as OmniDocBench, Real5-OmniDocBench (Zhou et al., ECCV 2026), and MDPBench (Li et al., NeurIPS 2026). In the revision, we will add the realism study and the content comparison to the appendix. In the main text of the final version, we will discuss these limitations, state that handwriting and historical scans are not yet covered, and scope our conclusions to the covered domains and acquisition chains.

AuthorsOpenReview ↗

Response to Reviewer DvkF (Part 3/5): Weakness 3, Question 2, and Question 3

Weakness 3. The paper uses LLMs to generate HTML/CSS documents and then extract ground-truth annotations from the same sources. Since these sources already contain explicit textual and structural information, the need for LLM-based extraction instead of deterministic parsing is unclear, particularly for reading order. The paper also lacks details on how reading order is determined, whether rendered layouts are considered, and how many annotation errors are identified and corrected during human review. These omissions make it difficult to assess annotation reliability.

Question 2. Why are multiple rounds of human cross-review still necessary after generating documents directly from source code? How much annotation cost or human effort does the proposed pipeline actually save compared with traditional annotation workflows?

Question 3. Among the annotations for text, tables, formulas, and reading order, which are extracted deterministically from HTML/CSS, and which require LLM inference? Why cannot these annotations be obtained entirely through source-code parsing and browser layout information? What is the accuracy of the initial LLM-assisted annotations? How many errors were identified and corrected during the multiple rounds of human review, and which annotation types accounted for most of these errors?

Thank you for these careful questions. We agree that the generated HTML already contains explicit text and structure, and that deterministic parsing is a natural way to use it. We also agree that the submission did not describe the annotation steps in enough detail. We clarify them below and will add a full description to the appendix.

What is deterministic and what relies on the LLM. For each page, an LLM reads the HTML/CSS source and proposes candidates for all four annotation types: text blocks with their categories, tables in HTML, displayed formulas in LaTeX, and a reading order. Deterministic checks then compare these candidates with the source and the annotation schema, covering text matching, captions, LaTeX formulas, file paths, and element identifiers (Figure 2). Finally, annotators review every page against its rendered image and correct the remaining errors in content, categories, block boundaries, and reading order. The text of each block is thus checked deterministically against the source, while block boundaries, categories, table structure, formula LaTeX, and reading order come from the LLM proposals and are confirmed by annotators.

Why the source alone is not enough. We used an LLM for the first step because several decisions require interpreting the markup. Of the 782 displayed formulas, 734 come from pages whose HTML contains no TeX markup. On these pages, formulas are written with HTML subscripts and superscripts or plain Unicode characters, so their LaTeX references have to be transcribed. The evaluation schema also separates elements that the HTML often groups together, such as a displayed equation and the explanation of its variables placed in one container. Because the HTML was written by LLMs across 66 subcategories, the same kind of element appears in many different markup patterns, which is hard to cover with fixed rules. At the same time, we agree that deterministic rules are more reliable for some decisions. For example, elements whose tag or class marks them as page footers can be flagged by rule for review, and we found pages where real footers had been annotated as body text. In the revision, we will add DOM-based rule checks for these categories and re-examine all pages.

Reading order. The LLM proposes an order from the HTML/CSS, and annotators confirm or correct it on the rendered page, so the final order follows the rendered layout. Browser layout gives the position of each element, but turning positions into an order on grid and multi-panel layouts still requires judgment. The HTML order agrees with the final order on most pages. To compare orders, we used the 53,114 title and text blocks whose text appears exactly once in the source, leaving out blocks that cannot be located unambiguously, mostly short headings and labels. On 1,418 of the 1,458 pages with at least two such blocks (97.3%), these blocks appear in the same order in the source and in the reference. On the remaining 40 pages, at least one pair of blocks is ordered differently. In one conference poster, for example, the HTML lists Case Study before Model Architecture, while the CSS grid places Model Architecture in the second row and Case Study in the third, and the reference follows the rendered page.

Our response to Weakness 3, Question 2, and Question 3 continues in Part 4/5.

Reviewer_5nfkOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a document parsing benchmark with 1,475 source pages, each with matched clean, digitally degraded, and recaptured images. It evaluates 58 models on text, formulas, tables, and reading order.

Critical strengths
  • Comparing the same page across conditions is useful. It makes the robustness results easier to interpret than comparisons on unrelated image sets.
  • Keeping the HTML/CSS sources should help with checking and fixing annotations. Appendix C also addresses the unreadable images: removing those 83 images barely changes the rankings within the tested cohort.
Critical weaknesses
  • The inference setup needs more detail. I couldn't find model-specific image resizing settings, prompts, output limits, or how failed and truncated outputs were handled in the manuscript. These choices could affect the robustness comparison, especially on long pages. Figure 12 includes a truncated output. I couldn't access the anonymous code link to check the settings there.
  • The annotation pipeline includes automated checks and human correction, but the remaining error rate is unclear. Retaining the source helps with checking annotations, but does not by itself establish that reading order and table structure match the rendered page.
Questions for authors
  • Could you provide the per-model inference settings and explain how truncation and failed requests were handled? Do the architecture-level robustness conclusions hold after accounting for input-resolution differences?
  • Was an independent sample of the final annotations checked after correction? If so, what were the sample size and remaining error rates for each annotation type?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 5nfk (Part 1/3): Weakness 1 and Question 1

Dear Reviewer 5nfk, we sincerely appreciate the time and attention you gave to our paper. Your comments on the inference setup, input resolution, and the remaining annotation error rate have helped us improve the paper substantially. We respond to each of them below, quoting your comments first.

Weakness 1. The inference setup needs more detail. I couldn't find model-specific image resizing settings, prompts, output limits, or how failed and truncated outputs were handled in the manuscript. These choices could affect the robustness comparison, especially on long pages. Figure 12 includes a truncated output. I couldn't access the anonymous code link to check the settings there.

Question 1. Could you provide the per-model inference settings and explain how truncation and failed requests were handled? Do the architecture-level robustness conclusions hold after accounting for input-resolution differences?

Thank you for reading the setup so closely. The anonymous repository linked in the abstract is at https://anonymous.4open.science/r/puredocbenchv2reviewercode. We repeat the address here for convenience and have confirmed that it opens without signing in. Following your requests, we give below the per-model settings, the handling of truncated and failed outputs, and an analysis of input resolution.

Per-model settings. The repository now holds the settings of all 58 models. inference/settings/ gives, for each model, the checkpoint name or API model string together with its recorded image processing, prompt, output budget, decoding and thinking settings, and postprocessing, and model_settings_58.md in that folder gives an overview in one table. inference/original_configs/ holds the retained original configurations, scripts, and prompts, and predictions/ holds the raw predictions of all 58 models.

Prompts. Specialist parsers use the prompt of their official release or the task prompts built into their pipeline. Of the 26 general VLMs, 22 receive the same transcription prompt, which asks for Markdown that keeps the document structure, with formulas in LaTeX and tables in HTML. The settings files list the prompt of every model or name the SDK that supplies it.

Output limits and decoding. Decoding is greedy or uses temperature 0 wherever the interface allows it, and thinking is disabled or set to a low level. The output budget is 8,192 tokens for most models, including all API models with saved request settings. Other budgets follow the model's own pipeline, for example 4,096 tokens for layout and 5,000 per region for MonkeyOCRv2-B-Parsing, 16,384 for OvisOCR2, and 32,768 for Unlimited-OCR and HunyuanOCR-1.5.

Truncated and failed outputs. A prediction that ends early is scored as saved against the full-page reference, so the content it leaves out counts as errors. Figure 12 shows such a prediction: the Digital output of Gemini-3.1-Pro stops inside the proof at "Injective", while the Clean output of the same model for this page continues into the next section. An empty prediction is scored as empty, with a Text Edit and Reading Edit of 1.0 and a TEDS of 0. Among the 252,518 archived predictions, 249 (0.10%) are empty. API scripts retry transient errors, for example up to four attempts with waits of 5, 10, and 20 seconds for Kimi K2.6. For the 38 models evaluated first, a page that still has no output is scored as empty. For the 20 models added last, 201 of their 88,500 page-track inputs (0.2%) have no output and were left out of the averages of seven models: 69 for Claude Opus 5, 43 for GPT-5.6 Sol, 33 for Qwen3.8-27B, 29 for Gemini 3.6 Flash, 25 timeouts for MonkeyOCRv2-B-Parsing, and 1 each for MiniMax-M3 and Qwen3.6-27B. Scoring these inputs as empty lowers the three-track average by 1.14 points for Claude Opus 5 and by 0.56 points or less for the other six. In the revision, we will apply this single rule to all 58 models and report the updated scores.

Our response to Weakness 1 and Question 1 continues in Part 2/3.

AuthorsOpenReview ↗

Response to Reviewer 5nfk (Part 2/3): Weakness 1 and Question 1 (continued)

This comment continues our response to Weakness 1 and Question 1 from Part 1/3.

Input resolution. Every model receives the released image of each page as it is, and our scripts apply no resizing of their own. Resizing, cropping, and tiling are carried out by the image processor or pipeline of each model at the settings of its release, and the Clean, Digital, and Real images of a page go through the same processing. The Clean-to-Real loss of each model is therefore measured under one fixed input configuration. The settings file lists this configuration for every model. For example, DeepSeek-OCR and Unlimited-OCR use a base size of 1,024 with 640-pixel crops; Dolphin-v2 accepts up to 2.56 million pixels, OvisOCR2 up to 8.3 million, and WeVisDoc up to 16.8 million; MiniCPM-V-4.6 splits a page into at most 36 slices; most pipeline parsers detect the layout and recognize each region as a separate crop; and API models receive the original image bytes at the high-detail setting, with an image re-encoded or downscaled only when it exceeds a provider's size limit.

Using these settings, we examined in four ways whether input resolution accounts for the difference between the three groups.

First, models that share one input configuration still differ clearly in their loss. The four locally served Qwen3.5 models run with identical code and settings, and their Clean-to-Real losses are 12.84, 10.32, 9.98, and 8.42 points from 0.8B to 9B. The three Qwen3-VL models, again with identical code and settings, lose 12.28, 12.43, and 9.71 points from 2B to 8B. In both families the largest model has the smallest loss.

Second, the comparison between groups is unchanged among models with the same kind of input handling. Sixteen of the 19 end-to-end parsers pass the original page directly to the model's own image processor, as the general VLMs do, and they lose 12.72 points on average, compared with 9.30 for the general VLMs. The other three use a fixed base size with crops and lose 14.43 points.

Third, reading the page region by region does not come with a smaller loss. Pipeline parsers recognize each region of a page as a separate crop, and their average loss is the largest of the three groups (14.23 points).

Fourth, the trend remains after the pages most sensitive to resolution are removed. We identified the 104 pages whose width would fall below 256 pixels if the page were fitted to 1,024 pixels on its longest side, and removed them from both the Clean and the Real track for the 32 models whose page-level results match the submitted scores exactly. The average losses of pipeline, end-to-end, and general VLM models change from 15.33, 12.57, and 9.27 points to 14.77, 11.46, and 8.73 points. The Spearman correlation between the Real rankings before and after is 0.997, and the ten best models stay the same.

Together, these results indicate that the order of the three groups holds when input resolution is taken into account. In the revision, we will state the conclusion at the level of the complete system, meaning each model together with its input processing, and we will add the per-model settings table, the prompts, the rules for truncated and failed outputs, and these four analyses to the appendix.

AuthorsOpenReview ↗

Response to Reviewer 5nfk (Part 3/3): Weakness 2 and Question 2

Weakness 2. The annotation pipeline includes automated checks and human correction, but the remaining error rate is unclear. Retaining the source helps with checking annotations, but does not by itself establish that reading order and table structure match the rendered page.

Question 2. Was an independent sample of the final annotations checked after correction? If so, what were the sample size and remaining error rates for each annotation type?

We agree with both points. Following your question, we report below the audit of the released annotations that was carried out after the correction rounds, with its sample size and the error rate of each annotation type, and then the evidence that concerns reading order and table structure specifically.

Sample and protocol. An annotator who was not involved in building the benchmark audited the released annotations of 119 pages against the rendered images. The sample combines a 100-page list fixed in advance with 19 pages from an earlier audit round. It contains 8 to 18 pages from each of the ten domains, roughly in proportion to domain size, and covers 56 of the 66 subcategories. On these pages, the annotator checked 7,240 annotations against the rendered image and flagged every annotation with a problem, such as wrong, missing, or extra text, a wrong category, an inaccurate box, or a wrong position in the reading order.

Error rates by annotation type. Of the 7,240 annotations, 70 (0.97%) were flagged.

Annotation typeCheckedFlaggedRate
Text (including headers and captions)6,891691.00%
Tables27110.37%
Displayed formulas7800.00%
Total7,240700.97%

By type of problem, 47 flags concern the transcribed text, 11 the category, 8 the box, 3 the reading order, and 1 the segmentation. The three reading-order flags lie on 2 pages: two text blocks on an insurance page, and the one flagged table, which sat at the wrong position in the order of a business letter. One of the category flags is a region annotated as running text that is a table on the rendered page. We re-examined every flag and corrected 69 annotations. Rescoring five models (HunyuanOCR, PaddleOCR-VL-1.6, DeepSeek-OCR-2, Dolphin-v2, and Qwen3-VL-8B) against the corrected annotations, with their predictions fixed, raises each model's three-track average by less than 0.02 points over the full benchmark. Beyond this audit, we screened the HTML sources of all 1,475 pages for header and footer text annotated with a scored category and confirmed 140 such entries on 138 pages. They are separate from the 70 flags above, and we will correct them in the released annotations as well. With the predictions fixed, correcting them raises the three-track average of HunyuanOCR by 0.085 points.

Reading order on the rendered page. The reading order is first proposed from the HTML/CSS and then confirmed or corrected by annotators on the rendered page, so the final order follows the rendered layout. To see how often the two differ, we compared the order of the 53,114 heading and body-text blocks whose text occurs exactly once in the source. On 1,418 of the 1,458 pages with at least two such blocks (97.3%), these blocks appear in the same order in the source and in the reference. On the other 40 pages the reference departs from the source order. On a conference poster, for example, the HTML lists Case Study before Model Architecture, the CSS grid places Model Architecture in the second row and Case Study in the third, and the reference follows the rendered page.

Table structure. Each table is annotated in HTML and confirmed by annotators against the rendered page. For the first construction batches, our annotation records keep the annotations as they stood before the cross-review rounds, which shows how often review had to change a table. On the 376 pages of these batches whose HTML did not change in between, 857 of the 909 tables (94.3%) needed no change, and the structure of 16 tables was corrected. Among the 271 tables in the audit above, the only flag concerns the position of a table in the reading order.

In the revision, we will add the audit protocol and results, the reading-order comparison, and the review statistics for tables to the appendix.

Dear Reviewer 5nfk,

Thank you again for your helpful comments. If time permits, we would appreciate your thoughts on our response. We would be happy to follow up on anything that remains unclear.

Reviewer_bcUGOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PureDocBench, a source-traceable benchmark for document parsing. It contains 1,475 generated HTML/CSS pages spanning 10 domains and 66 subcategories, with matched Clean, digitally degraded, and real-recaptured versions, yielding 4,425 images. Source-linked annotations cover text, formulas, tables, and reading order, supported by automated checks and human review. The authors evaluate 58 specialist parsers and general-purpose VLMs using the OmniDocBench evaluation framework. The results show substantial degradation under real acquisition conditions, changes in model rankings across tracks, and competitive performance from compact specialist models. The paper also provides qualitative failure analyses and a readability sensitivity study.

Critical strengths
  1. A useful controlled evaluation design. Matching the same source page across Clean, Digital, and Real tracks controls content variation when measuring robustness to acquisition conditions. This makes the cross-track comparisons more interpretable than comparisons between unrelated document collections and provides a useful complement to clean-document evaluation.

  2. Broad empirical coverage with inspectable evidence. The evaluation covers 58 models from multiple architectural families. The supplied case artifacts connect input images, source HTML, annotations, and archived predictions, making selected failure claims directly inspectable. The readability audit and filtering analysis are also valuable: they acknowledge acquisition-related information loss and examine whether it changes the comparative conclusions.

Critical weaknesses
  1. The central data-quality and representativeness claims need stronger quantitative validation. Retaining source HTML improves traceability, but does not by itself establish annotation accuracy: extraction, element categorization, and reading-order assignment still require validation. Although automated checks and human review are described, the provided materials do not report a sufficiently detailed independent audit of residual annotation errors. Moreover, the Real track recaptures generated pages rather than independently sourced real documents. Evidence connecting this synthetic-source distribution to authentic document parsing would strengthen the benchmark’s claimed practical relevance. These issues matter because reliable supervision and meaningful coverage are central to the benchmark’s contribution.

  2. The provided artifacts do not enable verification of the benchmark-wide results. The inspected supplement contains eight selected source pages, thirteen images, thirteen archived predictions, and four representative inference examples, rather than a complete evaluation package. Its documentation explicitly notes that some historical API settings, postprocessing configurations, and dependency revisions could not be recovered. Consequently, the available materials do not allow verification of the full scoring pipeline or the consistency of inference protocols across the 58 models. This is a limitation of the evidence available for review, not a claim that the authors have no additional artifacts.

Questions for authors
  1. Annotation validation and scope. Beyond schema/source checks and human cross-review, was an independent quantitative annotation audit conducted? Please report the sampling protocol and residual error rates by component, if available. What existing evidence supports generalization of the benchmark’s findings to authentically sourced documents?

  2. Full evaluation artifacts. Are the complete evaluation scripts, configurations, dataset manifests, and benchmark-wide predictions accessible to reviewers elsewhere? Please clarify the exact model/checkpoint identities, preprocessing, prompts, output limits, postprocessing, and failure-handling rules needed to reproduce the main tables.

  3. Prompt–target alignment. The supplied HunyuanOCR prompt instructs the model to ignore headers and footers, while the bottom-of-page text in case C4 is annotated as text_block. Are these entries included in the evaluated text targets? If so, how is this instruction–target mismatch handled? This is a clarification request rather than an established scoring error.

6 comments
AuthorsOpenReview ↗

Response to Reviewer bcUG (Part 1/5): Weakness 1 and Question 1

Dear Reviewer bcUG, we are sincerely grateful for your careful and constructive assessment of our paper. Your comments on annotation quality, generalization to authentically sourced documents, reproducibility, and prompt-target alignment have helped us improve the paper substantially. We respond to each of them below, quoting your comments first.

Weakness 1. The central data-quality and representativeness claims need stronger quantitative validation. Retaining source HTML improves traceability, but does not by itself establish annotation accuracy: extraction, element categorization, and reading-order assignment still require validation. Although automated checks and human review are described, the provided materials do not report a sufficiently detailed independent audit of residual annotation errors. Moreover, the Real track recaptures generated pages rather than independently sourced real documents. Evidence connecting this synthetic-source distribution to authentic document parsing would strengthen the benchmark's claimed practical relevance. These issues matter because reliable supervision and meaningful coverage are central to the benchmark's contribution.

Question 1. Annotation validation and scope. Beyond schema/source checks and human cross-review, was an independent quantitative annotation audit conducted? Please report the sampling protocol and residual error rates by component, if available. What existing evidence supports generalization of the benchmark's findings to authentically sourced documents?

We agree with both points. Because every page keeps its source, an error found in an audit can be traced and corrected once for all three tracks, and document types such as invoices and medical records can be released in full together with their sources. Following your requests, we report below an audit of the released annotations, with the sampling protocol and the error rates by component that you asked for, and then the evidence on authentically sourced documents.

Audit protocol. An annotator who was not involved in building the benchmark audited the released annotations of 119 pages against the rendered images. The sample combines a 100-page list fixed in advance with 19 pages from an earlier audit round. It contains 8 to 18 pages from each of the ten domains, roughly in proportion to domain size, and covers 56 of the 66 subcategories. On these pages, the annotator checked 7,240 annotations against the rendered image and flagged every annotation with a problem, such as wrong, missing, or extra text, a wrong category, an inaccurate box, or a wrong position in the reading order.

Error rates by component. Of the 7,240 annotations, 70 (0.97%) were flagged.

ComponentCheckedFlaggedRate
Text (including headers and captions)6,891691.00%
Tables27110.37%
Displayed formulas7800.00%
Total7,240700.97%

By type of problem, 47 flags concern the transcribed text: 19 headings whose capitalization differed from the rendered page, 20 annotations with missing or incomplete text, and 8 annotations that should not be scored, such as duplicated or illegible text. A further 11 concern the category, 8 the box, 3 the reading order (on 2 pages), and 1 the segmentation. The one flagged table was flagged for its position in the reading order.

Effect on scores. We re-examined every flag and corrected 69 annotations. We then rescored five models against the corrected annotations with their predictions fixed. The flagged annotations lie on 23 pages. The table covers 22 of them, and the corrections on the remaining page only adjust boxes, which do not affect the scores. All values are Avg3, the mean of the Overall scores on the Clean, Digital, and Real tracks.

ModelBefore (22 pages)After (22 pages)Change (22 pages)Change (full benchmark)
HunyuanOCR78.6979.85+1.16+0.017
PaddleOCR-VL-1.681.4782.66+1.19+0.018
DeepSeek-OCR-268.9869.96+0.98+0.015
Dolphin-v272.2073.13+0.93+0.014
Qwen3-VL-8B83.5484.54+0.99+0.015

Changes are computed before rounding. The five models gain 0.93 to 1.19 points on these pages, and over the full benchmark each model's Avg3 rises by less than 0.02 points.

Our response to Weakness 1 and Question 1 continues in Part 2/5.

AuthorsOpenReview ↗

Response to Reviewer bcUG (Part 2/5): Weakness 1 and Question 1 (continued)

This comment continues our response to Weakness 1 and Question 1 from Part 1/5.

Evidence on authentically sourced documents. Two findings on our Real track agree with benchmarks built from collected documents. DocPTBench (Du et al., NeurIPS 2026), which builds on photographed OmniDocBench pages, reports that specialist parsers lose more accuracy than general multimodal models on photographed inputs, and on our Real track specialists lose 12.99 to 14.23 points while general VLMs lose 9.30. MDPBench (Li et al., NeurIPS 2026) finds Gemini 3 Pro relatively robust on photographed documents, and the two Gemini models lose the least among our 58 models.

Realism of the pages. For the pages themselves, an additional human study asked three raters who were not involved in building the benchmark to score 50 Clean pages, stratified by document type, layout complexity, and orientation, with source information hidden. On a five-point scale from 1 (clearly synthetic) to 5 (fully natural), visual realism averaged 4.13 (95% CI 3.99 to 4.24; Krippendorff's α=0.63\alpha = 0.63), and the plausibility of the text for its document type averaged 4.28. We also compared our reference text with the 1,284 naturally sourced pages with body text in OmniDocBench v1.5 (Ouyang et al., CVPR 2025). Punctuation takes a similar share (9.9% versus 9.7%), while our pages hold about twice as much text and about three times the digit density, in line with the amounts, dates, and codes on invoices and medical records. On the acquisition side, the 1,475 Real images come from actual capture and sharing: 1,275 phone photographs of printed pages, photocopies, and screens, and 200 screenshots passed through social-media compression.

Scope. This evidence supports the visual and textual plausibility of the pages and the agreement of our robustness findings with collected-document benchmarks. The proportions of document types follow our coverage design, with 20 to 32 pages per subcategory, and differ from real-world frequencies. We therefore see PureDocBench as a complement to benchmarks built from collected documents, such as OmniDocBench, Real5-OmniDocBench (Zhou et al., ECCV 2026), and MDPBench, and we limit our conclusions to the covered domains and acquisition chains.

In the revision, we will add the audit protocol and results, the realism study, and the content comparison to the appendix, and state this scope in the main text.

AuthorsOpenReview ↗

Response to Reviewer bcUG (Part 3/5): Weakness 2 and Question 2

Weakness 2. The provided artifacts do not enable verification of the benchmark-wide results. The inspected supplement contains eight selected source pages, thirteen images, thirteen archived predictions, and four representative inference examples, rather than a complete evaluation package. Its documentation explicitly notes that some historical API settings, postprocessing configurations, and dependency revisions could not be recovered. Consequently, the available materials do not allow verification of the full scoring pipeline or the consistency of inference protocols across the 58 models. This is a limitation of the evidence available for review, not a claim that the authors have no additional artifacts.

Question 2. Full evaluation artifacts. Are the complete evaluation scripts, configurations, dataset manifests, and benchmark-wide predictions accessible to reviewers elsewhere? Please clarify the exact model/checkpoint identities, preprocessing, prompts, output limits, postprocessing, and failure-handling rules needed to reproduce the main tables.

Thank you for examining the supplement so carefully. Beyond the supplement, further evaluation materials are available to you in the anonymous repository linked in the abstract (https://anonymous.4open.science/r/puredocbenchv2reviewercode), and we have extended them following your requests. Below, we first describe what the repository contains and then give the items you listed: model identities, preprocessing, prompts, output limits, postprocessing, and failure handling.

What you can access. The repository contains the scoring code, which is a pinned OmniDocBench (Ouyang et al., CVPR 2025) snapshot with our patch and a command-line entry point; the manifest of all 1,475 pages, with the capture condition of each Real image; the main table of 58 models with the 174 model-by-track component results and the per-domain results of 53 models; the generation and degradation code; and four inference adapters. We have now added three sets of files to this repository: the settings of all 58 models together with the retained original configurations, scripts, and prompts (inference/settings/ and inference/original_configs/); the raw predictions of all 58 models, one archive per model (predictions/, 252,518 files covering 172 of the 174 model-by-track runs); and per-page scores (results/page_scores_20_models/). We will release the full images, sources, and annotations as well.

Model identities. Each entry of the settings files gives the checkpoint name or the API model string, for example zenosai/MonkeyOCRv2-B-Parsing, tencent/WeVisDoc-4B, gemini-3.6-flash, and claude-opus-5. Two limits remain and are marked in the files. The exact weight revision was recorded for the most recent runs and was not pinned for the earlier locally served checkpoints, and for YouTu-Parsing and Gemini-3.1-Pro the original run configuration was not saved, so their entries are reconstructed from the archived adapters.

Preprocessing. Every model receives the full page image and applies the image processor or pipeline of its own release, so resizing, cropping, and tiling follow each model's design. Pipeline parsers run their layout detection and region recognition stages, and DeepSeek-OCR, for example, uses a base size of 1,024 with cropping enabled. API models receive the original image bytes. When an image exceeds a provider's size limit, it is re-encoded and, if still too large, downscaled, and the submitted size is logged.

Our response to Weakness 2 and Question 2 continues in Part 4/5.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers and the Area Chair

We thank all four reviewers for the careful and constructive feedback, and the area chair for handling our submission. The reviews centered on four points. We have uploaded a revised paper that addresses each of them; the main text now uses the tenth page, and Appendices H to K are new.

1. Novelty over prior work (DvkF, GWMY). The revision relates PureDocBench to markup-based document synthesis and compares it directly with OmniDocBench, Real5-OmniDocBench, DocPTBench, and MDPBench (Sections 1 and 2.2, Table 21). It adds two findings: digital degradation reproduces about 29% of the real-acquisition loss, and all 53 models with per-domain results read business and administrative documents less accurately (Section 3.2, Appendix J).

2. Realism of the generated pages (DvkF, GWMY, bcUG). Three raters who were not involved in building the benchmark scored the visual realism of 50 pages at 4.13 out of 5, all Real images are actual captures, and two findings on the Real track agree with DocPTBench and MDPBench, which are built from collected documents. The limitations now state the scope of the generated pages (Sections 2.1 and 3.2, Appendix I).

3. Annotation quality (DvkF, bcUG, 5nfk). An annotator who was not involved in building the benchmark audited 119 pages and flagged 70 of 7,240 annotations (0.97%), and rescoring five models against the corrected annotations changes Avg3 by less than 0.02 points. Appendix H reports the annotation steps, the corrections made during review, the audit by annotation type, and the safeguards for future releases.

4. Inference settings and reproducibility (bcUG, 5nfk). Appendix K lists the prompts, preprocessing, output limits, and failure handling of all 58 models and examines input resolution. It also applies a single failure rule to all 58 models, which changes the scores of seven (Table 22) and leaves the leader of every track unchanged. The anonymous repository (https://anonymous.4open.science/r/puredocbenchv2reviewercode) holds the settings and raw predictions of all 58 models.

We would be glad to answer any further questions during the discussion.

AuthorsOpenReview ↗

Summary of the Rebuttal and Revision

Dear Reviewers, AC, SAC, and PCs,

Thank you for your time and effort on our submission. As the discussion closes, we summarize the reviews, our responses, and the revised paper.

1. Ratings at a glance

ReviewerRatingConfidenceMain concernsOur response
bcUG3 (Weak acceptance)4Annotation audit; evaluation artifacts; prompt-target alignment5 comments; no follow-up yet
5nfk3 (Weak acceptance)4Inference settings; residual annotation errors3 comments; no follow-up yet
DvkF2 (Weak rejection)3Novelty over markup-based synthesis; realism; annotation pipeline; quality at scaleReplied Nov 18: partly addressed, rating maintained
GWMY2 (Weak rejection)4Novelty over existing benchmarks; generated versus real-world documents4 comments; no follow-up yet
Average2.5
  • All four reviewers rated Presentation 3 and described their soundness concerns as "minor or fixable".
  • We answered every weakness and question on November 11 and uploaded a revised paper (new Appendices H to K).

2. Strengths noted by the reviewers

  • Matched Clean, Digital, and Real versions of each page, which control for content (all four reviewers).
  • Source-linked annotations that are easier to verify and correct (DvkF, GWMY, 5nfk).
  • Evaluation of 58 models from three architectural families (bcUG, DvkF, GWMY).

3. Concerns of Reviewers DvkF and GWMY (rating 2)

We made every effort to address every concern raised by the two reviewers who rated 2. Each point below rests on new evidence now in the revised paper. Reviewer DvkF replied on Nov 18 that our clarifications, the independent audit, and the additional experiments "have addressed some of my concerns", and maintains the rating over novelty and representativeness (first two items below); we posted a further reply.

Novelty (DvkF: W1; GWMY: W1, Q1, Q3)

  • The revision credits markup-based synthesis (DocBank, CoSyn, Dolphin, Image2Struct) and compares PureDocBench directly with OmniDocBench, Real5-OmniDocBench, DocPTBench, and MDPBench (Sections 1 and 2.2, Table 21).
  • New finding 1: on identical pages, digital degradation reproduces about 29% of the loss under real acquisition (3.34 versus 11.61 points), and the two losses rank the models differently (Spearman 0.64).
  • New finding 2: all 53 models with per-domain results have lower Clean text accuracy on business and administrative documents, by 11.0 points on average. 49 of the 66 subcategories lie outside the categories of OmniDocBench and MDPBench (Section 3.2, Appendix J).

Realism of the generated pages (DvkF: W2, Q1; GWMY: W2, Q2)

  • Three raters who were not involved in building the benchmark scored 50 pages at 4.13/5 for visual realism and 4.28/5 for content plausibility.
  • All 1,475 Real images are actual captures: 1,275 phone photographs and 200 shared screenshots.
  • Two Real-track findings agree with DocPTBench and MDPBench, which are built from collected documents. Appendix J reports the tracks supporting each finding, and Appendix I states the scope of the generated pages.

Annotation pipeline and quality at scale (DvkF: W3, W4, Q2, Q3, Q4)

  • Appendix H specifies what the LLM proposes, what the rules check, and what annotators verify on the rendered page. On 376 pages with pre-review records, 91.5% of text blocks, 94.3% of tables, and 93.3% of formulas needed no change.
  • An annotator who was not involved in building the benchmark audited 119 pages and flagged 70 of 7,240 annotations (0.97%). Rescoring five models against the corrected annotations changes Avg3 by less than 0.02 points.

4. Concerns of Reviewers bcUG and 5nfk (rating 3)

Annotation audit and prompt-target alignment (bcUG: W1, Q1, Q3; 5nfk: W2, Q2)

The audit above is reported by annotation type: text 1.00%, tables 0.37%, displayed formulas 0.00% (Appendix H). Following Q3 of Reviewer bcUG, we screened all 1,475 pages and will recategorize 140 header and footer entries (HunyuanOCR three-track average +0.085 points).

Inference settings and reproducibility (bcUG: W2, Q2; 5nfk: W1, Q1)

Appendix K lists the prompts, preprocessing, output limits, and failure handling of all 58 models and applies one failure rule to all of them (Table 22; every track leader unchanged). Its four analyses of input resolution indicate that the order of the three architecture groups holds. The anonymous repository holds the settings and 252,518 raw predictions of all 58 models.

As three reviewers have not yet responded, we would be grateful if the responses and the revision could be considered in the upcoming discussion.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.