- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces PureDocBench, a benchmark for evaluating document parsing under clean, digitally degraded, and real-world capture conditions. It contains 1,475 source pages across 10 domains and 66 subcategories, with 4,425 images across the three settings. The authors use HTML/CSS-generated documents to obtain source-linked annotations and evaluate 58 models using established document-parsing metrics for text, formulas, tables, and reading order.
The main goal is to study how document-parsing performance changes as documents move from clean inputs to degraded or real-world images. The benchmark provides a structured way to compare models across these conditions. However, the paper's contribution is primarily a new dataset and evaluation setup rather than a new parsing method or evaluation metric. Its novelty therefore depends on how much additional insight and evaluation capability it provides beyond existing benchmarks.
Critical strengths
1. Source-linked annotations and controlled evaluation. Generating documents from HTML/CSS gives the authors access to structured source information and makes it possible to evaluate models under different visual conditions while keeping the underlying content consistent. This is useful for studying how degradation affects parsing performance and can make annotation easier to verify, but is not something very novel and has been already used to generate certain other benchmarks (also cited in the paper).
2. Evaluation across multiple document types and conditions. The benchmark covers 10 domains, 66 subcategories, and 58 models. Evaluating clean, digitally degraded, and real-world images in one framework can help researchers identify where models are robust and where they fail. The benchmark could be useful to the community if its added value over existing datasets is demonstrated more clearly.
Critical weaknesses
-
The novelty over existing benchmarks is not sufficiently established. Several recent benchmarks already evaluate document parsing on photographed or degraded documents, compare digital and real-world inputs, or provide broad coverage of document types. In particular, Real5-OmniDocBench studies matched document content under physical capture conditions, while DocPTBench and MDPBench also consider photographed documents and real-world variation. OmniDocBench already provides a broad evaluation framework for text, formulas, tables, and reading order. As a result, the paper needs to explain more clearly what new research questions PureDocBench enables that these existing benchmarks cannot address. The combination of existing ideas in a new dataset is useful, but it is not by itself a strong enough novelty argument.
-
The connection between synthetic and real-world documents needs stronger validation. The source-linked annotations are a useful feature, but generating documents with HTML/CSS does not necessarily capture the layout, content, and visual complexity of naturally occurring documents. This matters because the paper aims to draw conclusions about real-world document parsing. The authors should provide stronger evidence that the synthetic documents and degradation scenarios reflect real-world cases, and clarify which conclusions are supported by the real-image evaluation rather than the synthetic setup alone.
Questions for authors
-
What can PureDocBench measure that existing benchmarks cannot? Please compare the benchmark directly with the most relevant prior work, especially Real5-OmniDocBench, DocPTBench, MDPBench, and OmniDocBench. Beyond the number of domains or evaluated models, what specific capability or research question is uniquely enabled by PureDocBench?
-
How representative are the HTML/CSS-generated documents of real-world documents? Do the authors have evidence that the document structures and degradation scenarios reflect naturally occurring documents? How well do the findings from the synthetic tracks agree with results on the real-world track?
-
What new findings emerge from this benchmark? Which conclusions about model behavior could not have been drawn from existing benchmarks? A direct comparison or analysis showing that PureDocBench reveals previously unobserved failure patterns would strengthen the paper's contribution.
5 comments
Response to Reviewer GWMY (Part 2/4): Weakness 1, Question 1, and Question 3 (continued)
This comment continues our response to Weakness 1, Question 1, and Question 3 from Part 1/4.
How PureDocBench differs. PureDocBench differs from these benchmarks in three ways. First, its 1,475 pages are newly generated and do not overlap with the pages of existing benchmarks; their HTML/CSS sources are retained, and their references are newly annotated, human-reviewed, and shared by all three tracks. Second, 49 of its 66 subcategories, covering 1,077 pages, are document types outside the categories of OmniDocBench and MDPBench, such as invoices, bank statements, tax and insurance documents, medical records and bills, contracts, court and government documents, waybills and bills of lading, and certificates. In this count, related types such as theses, technical reports, syllabi, and business or audit reports are treated as covered. Third, every page has both a digitally degraded image and a real acquisition. Among the four benchmarks, only DocPTBench contains both simulated and physically photographed images, and the photographs cover 100 of its 981 pages. The second and third differences let us study two questions directly.
How well digital degradation reflects real acquisition. Digital degradation is commonly used to test and train parsers for robustness, since it is inexpensive and can be applied to any page. Because both versions of every page are scored against the same reference, the two can be compared on identical content across the whole benchmark. With the ten degradation profiles of our Digital track, which include aging, photocopying, compression, uneven lighting, and geometric distortion, the 58 models lose 3.34 Overall points on average, compared with 11.61 points under real acquisition, so the Digital track reproduces about 29% of the real loss. The two tracks also order the models differently. The Spearman correlation between the models' digital and real losses is 0.64, and TeleOCR, which ranks second on Digital, ranks eighth on Real, while Gemini 3.6 Flash moves from third to first. The difference is largest for general VLMs, which lose 1.84 points under digital degradation and 9.30 points under real acquisition, so a digital-only evaluation would suggest that they are almost unaffected. In line with DocPTBench, specialist parsers lose more than general VLMs under real acquisition (14.23 points for pipeline and 12.99 points for end-to-end specialists), and in line with MDPBench, which finds Gemini 3 Pro relatively robust on photographed documents, the two Gemini models lose the least among the 58 models. These results show that robustness measured with digital degradation alone underestimates the loss under real acquisition and gives a different picture of which models are most robust.
How accurately business and administrative documents are parsed. Business and administrative documents are common in commercial document processing but are difficult to collect and release because they contain personal and financial information, and generating the pages makes it possible to include them. To examine what they reveal, we compared the academic, education, and publishing domains, whose documents are closest to those in existing benchmarks, with the other seven domains, using the 53 models with per-domain results in Appendix G. Both groups are generated and annotated in the same way, so the differences can be attributed to the document types. On the Clean track, all 53 models have lower text accuracy on the seven domains, by 11.0 points on average, and 51 and 52 models have lower table and reading-order scores, by 10.7 and 13.2 points on average. The gaps persist on the Digital and Real tracks and are largest on certificates and logistics documents, whose mean text accuracy on Clean is 56.4, compared with 85.3 on academic pages. Model rankings are largely consistent between the two groups (Spearman for text). Evaluation on the document types found in existing benchmarks therefore ranks parsers similarly, while overestimating their accuracy on business and administrative documents by about ten points.
We acknowledge that the submission did not state these two questions explicitly, and your comments helped us articulate them. In the revision, we will add the comparison table to Section 2.2, introduce the two questions in the introduction, report both analyses in Section 3 with per-model and per-domain results in the appendix, and note where our findings agree with DocPTBench and MDPBench.
Response to Reviewer GWMY (Part 3/4): Weakness 2 and Question 2
Weakness 2. The connection between synthetic and real-world documents needs stronger validation. The source-linked annotations are a useful feature, but generating documents with HTML/CSS does not necessarily capture the layout, content, and visual complexity of naturally occurring documents. This matters because the paper aims to draw conclusions about real-world document parsing. The authors should provide stronger evidence that the synthetic documents and degradation scenarios reflect real-world cases, and clarify which conclusions are supported by the real-image evaluation rather than the synthetic setup alone.
Question 2. How representative are the HTML/CSS-generated documents of real-world documents? Do the authors have evidence that the document structures and degradation scenarios reflect naturally occurring documents? How well do the findings from the synthetic tracks agree with results on the real-world track?
Thank you for raising this important point. We agree that pages generated from HTML/CSS cannot capture every aspect of naturally occurring documents. We chose generation mainly because it makes a few things possible that are hard to achieve with collected documents, and we see it as a complement to benchmarks built from them. For example, business documents such as invoices and medical records can be released together with their sources, the annotations can be checked against those sources, and the three images of each page can share one reference. Below, we present the evidence on the generated documents and the acquisition conditions, and then clarify which conclusions the Real track supports and how closely the synthetic and Real tracks agree.
Documents. In an additional human study, three raters who were not involved in building the benchmark scored 50 Clean pages, stratified by document type, layout complexity, and orientation and shown in random order with source information hidden. On a five-point scale from 1 (clearly synthetic) to 5 (fully natural), their ratings of typography, spacing, alignment, element placement, and overall coherence averaged 4.13 (95% CI 3.99 to 4.24; Krippendorff's ), and 40 of the 50 pages averaged at least 4. The same raters scored the plausibility of the text for its document type at 4.28. We also compared our reference text with that of the 1,284 naturally sourced pages with body text in OmniDocBench v1.5 (Ouyang et al., CVPR 2025). Punctuation and symbols make up a similar share of the text (9.9% versus 9.7%), while our pages contain about twice as much text, about three times the digit density, and more mixed Chinese and English text (48% versus 15% of pages), in line with the amounts, dates, and codes that fill invoices, statements, and medical records.
Acquisition. The Real images come from actual capture and sharing: 1,275 phone photographs of printed pages, photocopies, and screens, and 200 screenshots passed through social-media compression. They cover flat, creased, bent, and photocopied paper, normal, low, flash, shadowed, and outdoor lighting, and top-down, oblique, and handheld views. The ten profiles of the Digital track, such as aging, photocopying, compression, uneven lighting, and geometric distortion, are simulated, and on the same pages they reproduce about 29% of the Real loss (3.34 versus 11.61 points). We therefore base conclusions about real-world robustness on the Real track.
Our response to Weakness 2 and Question 2 continues in Part 4/4.
Response to Reviewer GWMY (Part 1/4): Weakness 1, Question 1, and Question 3
Dear Reviewer GWMY, we sincerely appreciate your thorough review and the effort behind it. Your comments on how PureDocBench compares with closely related benchmarks and on the connection between generated and real-world documents have helped us state our contribution much more clearly. We respond to each of them below, quoting your comments first.
Thank you for these thoughtful questions and for pointing us to the most closely related benchmarks. OmniDocBench (Ouyang et al., CVPR 2025) established the evaluation framework that PureDocBench adopts, and the recent Real5-OmniDocBench (Zhou et al., ECCV 2026), DocPTBench (Du et al., NeurIPS 2026), and MDPBench (Li et al., NeurIPS 2026), which are concurrent with our work, have each advanced the evaluation of parsers on photographed and physically recaptured documents. We see PureDocBench as complementary to these benchmarks. Below, we first compare PureDocBench directly with these four benchmarks and then describe the two questions it allows us to study, together with the findings they lead to.
Direct comparison. OmniDocBench v1.5 offers 1,355 collected pages in nine document types with detailed annotations. Real5-OmniDocBench recaptures all of these pages under five physical conditions (scanning, warping, screen photography, illumination, and skew) and reuses their annotations, which gives a large-scale view of how capture affects identical content. DocPTBench pairs 981 OmniDocBench pages with simulated photographs, photographs 100 of them under four physical conditions, and adds document translation. MDPBench extends photographed-document parsing to 17 languages, with three photographs of each of its 850 pages. The table below places PureDocBench alongside them.
Our response to Weakness 1, Question 1, and Question 3 continues in Part 2/4.