acceptodds
Under review as a conference paper at ICLR 2027

SimDocBench: Benchmarking the Robustness of Document Parsing via Paper Realistic Simulation

Abstract

Modern optical character recognition (OCR) and document parsing systems convert document images into machine-readable text and structured content, and now achieve strong performance on clean, scanned documents. However, their robustness to real-world physical capture remains poorly understood, as photographed documents often exhibit diverse geometric, photometric, and environmental variations that are not systematically evaluated by existing benchmarks. To address this gap, we introduce SimDocBench with a Source-Neutral-Perturbed protocol that separates rendering effects from acquisition-induced degradation. Using physically based thin-sheet simulation, path-traced rendering, and camera-effect modeling, SimDocBench constructs 158,496 single-factor renders covering 31 acquisition factors across 5 dimensions, together with Regular and Hard multi-factor settings. Rather than measuring parsing accuracy alone, SimDocBench evaluates three complementary properties of document parsers: accuracy under physical acquisition changes, consistency across content-preserving observations of the same document, and recoverability through controlled input restoration. Extensive experiments reveal that physical robustness is not simply an extension of clean-document accuracy: models with similar clean performance can exhibit substantially different degradation, particularly under challenging acquisition conditions. Moreover, accuracy degradation and prediction consistency capture distinct robustness behaviors, while controlled restoration further reveals heterogeneous failure mechanisms across acquisition factors and parsers. Together, these results show that physical robustness is a distinct and multi-dimensional capability largely invisible to clean-document benchmarks.

Primary area:
datasets and benchmarks
Keywords:
Document Parsing, OCR, Robustness Benchmark, Benchmark Evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_KXAwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SimDocBench evaluates document parsing under controlled physical acquisition changes. Using thin-sheet simulation, path-traced rendering, and camera effects, it generates 158,496 single-factor images covering 31 factors, together with Regular and Hard multi-factor settings, from 1,651 annotated source pages. A Source–Neutral–Perturbed protocol distinguishes rendering-induced changes from acquisition-induced degradation. The benchmark evaluates parsing accuracy, prediction consistency, and recoverability through controlled input restoration.

Critical strengths
  1. The Source–Neutral–Perturbed protocol separates rendering effects from acquisition-induced degradation, while 31 controlled factors enable matched comparisons on the same document content. This provides more informative diagnosis than a single clean-versus-degraded evaluation.

  2. Eight parsers exhibit substantially different degradation despite similar clean accuracy, with large differences supported by paired source-page confidence intervals. The additional consistency and restoration analyses extend the benchmark evaluation spectrum.

Critical weaknesses
  1. Agreement with real acquisition is strong for Skew but weak or uncertain for Warping and Illumination. The factor-coverage analysis also does not establish whether physical rendering provides better real-world diagnostic value than simpler, severity-matched transformations.

  2. MCD lacks precise component-matching and missing-output rules, and the accuracy–consistency comparison does not clearly use identical pages and perturbation severities. Unexplained differences between Regular and Hard neutral scores further limit the reproducibility and interpretation of the diagnostic results.

Questions for authors

Please respond to my concern about weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer KXAw (Part 1/2): Question 1

We sincerely thank you for the careful evaluation of SimDocBench. We will address the concerns below and hope to clarify them.

Q1: Agreement with real acquisition is strong for Skew but weak or uncertain for Warping and Illumination. The factor-coverage analysis also does not establish whether physical rendering provides better real-world diagnostic value than simpler, severity-matched transformations.

A1: Thank you for this valuable suggestion. An important advantage of 3D physical simulation over conventional 2D image transformations is its ability to model physical constraints and interactions among different acquisition factors at the generation-mechanism level.

Traditional 2D augmentation typically applies image-space operations, such as rotation, deformation, blurring, or brightness adjustment, directly to the final image. In contrast, our approach first models a document as a three-dimensional paper surface with deformable geometry and material properties, and then renders it under different camera viewpoints, illumination conditions, and material configurations. Consequently, paper geometry deformations, local text distortions, perspective projection, and illumination variations induced by changes in surface orientation can arise naturally within a unified physical scene.

Physical rendering allows us to independently control paper shape, material, illumination, and camera parameters. For example, when the paper curvature changes, text projection, surface shading, and shadows change jointly with the scene, while the annotations are propagated consistently with the geometric transformations. This enables researchers to repeatedly construct specific acquisition conditions and investigate the circumstances under which models fail.

To examine whether these physical variations provide greater real-world diagnostic value than simple 2D augmentation, we conducted the following additional controlled comparison:

  1. Matched source pages: We used the same 240 test source pages. Each page has a corresponding real-capture photograph from Real5-OmniDocBench. We additionally constructed content-matched 2D and 3D simulated versions to investigate Skew, Warping, and Illumination, respectively.
  2. Severity-matched 2D baselines: We made the 2D baselines as comparable as possible to the 3D perturbations in terms of degradation severity. Specifically, the 2D baselines directly apply rotation/perspective transformations, grid-based deformation, or illumination changes to the corresponding neutral-rendered images, whereas the 3D variants are generated by modifying scene parameters. The 2D parameters are selected based on image characteristics on separate calibration pages, without using model performance scores. Three simulation presets are retained for each page and condition.
  3. Evaluation on eight parsers: We directly evaluated all eight parsers and calculated their performance drops under 2D, 3D, and real-world conditions. Simulated performance drops are calculated relative to the corresponding neutral images, while real-world performance drops are calculated relative to the scanned images. We then compare the rankings of the eight models according to their performance drops.

In other words, we investigate whether models that experience larger performance drops on real photographs also exhibit larger drops on simulated images. In the following table, a higher τ indicates stronger agreement between the rankings induced by simulation and those observed under real-world acquisition.

Conditionτ (2D vs. Real)τ (3D vs. Real)3D − 2D (95% CI)
Skew0.5000.7860.286 [−0.357, 0.357]
Warping0.2140.2140.000 [−0.429, 0.429]
Illumination−0.429−0.2140.214 [−0.357, 0.500]

The 3D simulation produces rankings that are more consistent with real-world acquisition under Skew. However, the confidence interval for its improvement over 2D still includes zero.

We manually inspected paired examples with identical content from six source-page strata and found that some Real5-OmniDocBench Illumination photographs exhibit additional variations in page scale, position, and skew, beyond illumination changes. The Illumination subset in Real5-OmniDocBench is therefore confounded by substantial skew variations.

In real-world acquisition, obtaining such controlled comparisons typically requires fixing the camera position and pose (e.g., using a tripod), paper position, and imaging parameters, while independently adjusting illumination. In contrast, our simulations can directly fix these conditions, allowing researchers to distinguish a model's sensitivity to illumination itself from its sensitivity to factors such as geometry and viewing angle.

AuthorsOpenReview ↗

Response to Reviewer KXAw (Part 2/2): Question 2

Q2: MCD lacks precise component-matching and missing-output rules, and the accuracy–consistency comparison does not clearly use identical pages and perturbation severities. Unexplained differences between Regular and Hard neutral scores further limit the reproducibility and interpretation of the diagnostic results.

A2: Thank you for pointing out these issues. We have checked our implementation and conducted additional comparisons using exactly the same evaluation inputs.

First, we clarify the MCD computation rules.

MCD compares two outputs generated by the same parser for the same page under Neutral and Perturbed conditions. It measures how much the outputs change, rather than comparing them against ground-truth annotations.

ComponentComparison
TextText content and block types, such as headings and paragraphs
FormulaNormalized LaTeX tokens and symbols
TableNumber of rows, number of columns, and cell contents
Reading OrderEach block is represented by its type and its first four words/symbols. We compare these representations and their adjacent relationships.

Content similarity considers the overlap of both individual words/symbols and adjacent word/symbol pairs.

For multiple formulas or tables, we identify the alignment that minimizes the total difference while preserving their original order, allowing insertions and deletions. The cost of a missing item is 1, while the cost of a matched pair is 1 − similarity. The total difference is then normalized by the larger number of items in the two outputs. Finally, we compute the equally weighted average of the distances across applicable components. The complete formulas and fixed component weights will be provided in the appendix.

If neither output contains formulas or tables, the corresponding component is excluded from the calculation. If a component is missing from only one output, its distance is assigned a value of 1. For text and reading-order features, the distance is 0 when both outputs are empty and 1 when only one output is empty. Consequently, two completely empty outputs can also yield MCD = 0, indicating that consistency should be interpreted together with accuracy. Missing prediction files caused by execution errors are not treated as empty outputs for scoring purposes.

Second, we use exactly the same evaluation inputs for both metrics.

Our original comparison did indeed use different evaluation subsets. We have now fixed exactly the same source pages, perturbed images, and Neutral reference images, and recomputed the accuracy drops and MCD. Each of the eight parsers is evaluated on 3,703 input pairs, covering 995 source pages and 22 factors.

Aggregation MethodSpearman ρ (95% CI)Pairwise Ranking Reversals
Equal weight per input pair0.119 [−0.018, 0.228]300/616
Equal weight per source page0.119 [−0.018, 0.230]296/616

The 616 comparisons consist of all pairwise comparisons among the eight models across 22 factors. The 300 ranking reversals indicate cases where accuracy degradation and MCD lead to opposite conclusions about which of two models is more robust. Confidence intervals are estimated using 2,000 bootstrap resamples with shared source-page sampling.

Therefore, even after matching the evaluation pages and perturbation images, the two metrics still produce different robustness rankings.

Third, why do the Neutral scores differ between Regular and Hard?

Upon checking, we found that the two settings use the same pages and reference inputs for their Neutral scores, but the Hard scores for several models were taken from an earlier scoring version, resulting in inconsistent scoring versions in the table.

Regarding the N and NH columns, the values in the N column are the correct scores for the Neutral pages. For example, FireRed's Neutral score is corrected from 88.48 to 89.60, while its Hard score remains unchanged at 68.72. Accordingly, the performance drop is corrected from 19.76 to 20.88.

We thank the reviewer for identifying this issue and will correct Table 1 and the corresponding results throughout the final revised manuscript.

Thank you for the detailed response and the additional experiments.

The MCD specification, the matched-support recomputation, and the N/N_H discrepancy are adequately addressed. It would also be good to add them to the main paper

But still, some concerns I raised have not been resolved. In the comparison, all three 3D−2D confidence intervals include zero, and Warping shows no difference. The per-parser table also shows simulated drops far below real ones. The additional real-world evaluation value of physical simulation is not proven. Besides, only the four end-to-end parsers have N ≠ NHN_H. Was H rescored with the current version?

Overall, I would keep my current rating.

Reviewer_WM1VOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SimDocBench is a benchmark for systematically evaluating full document parsing under controlled physical acquisition changes. It combines physically based thin-sheet simulation, path-traced rendering, and camera-effect modeling to construct 158,496 single-factor renders covering 31 acquisition factors across five dimensions, together with Regular and Hard multi-factor settings. A Source–Neutral–Perturbed protocol separates rendering effects from acquisition-induced degradation. The paper evaluates eight document parsers from three complementary perspectives: accuracy under physical acquisition changes, prediction consistency, and recoverability through controlled input restoration. The experiments reveal model-dependent degradation and examine how simulated results transfer to real acquisition.

Critical strengths
  1. By varying one acquisition factor while holding other conditions fixed, the benchmark helps identify specific causes of parsing degradation.
  2. The benchmark covers 31 factors across five categories and includes multi-factor combinations, enabling evaluation under a broad range of acquisition changes.
  3. The extensive evaluation of eight parsing systems provides useful reference results for understanding and improving document parsing robustness.
Critical weaknesses
  1. Synthetic images still differ from real photographs, and the current validation does not establish that the benchmark broadly reflects parser robustness under real acquisition conditions.
  2. The ranking of influential factors may depend on the selected perturbation strengths and sampling choices. Its stability under alternative settings needs further validation.
  3. Prior work has already established the importance of physical acquisition robustness. The paper should more clearly demonstrate what its differences from existing benchmarks enable researchers to discover.
Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer WM1V (Part 1/3): Question 1

We sincerely thank you for recognizing the value of our controlled evaluation design and fine-grained factor analysis. We further clarify the three concerns below.

Q1: Synthetic images still differ from real photographs, and the current validation does not establish that the benchmark broadly reflects parser robustness under real acquisition conditions.

A1: Thank you for raising this important question. The goal of SimDocBench is not to completely replace real-capture data, but rather to explore a low-cost, controllable, and scalable approach to generating physical acquisition data. Compared with relying on physical acquisition using smartphones, cameras, and real-world environments, physical simulation enables large-scale generation of paired test data with explicitly defined acquisition conditions while preserving the correspondence between document content and annotations.

More importantly, the value of our 3D approach lies in modeling the relationships among paper geometry, material properties, illumination, and cameras through a unified physical scene generation mechanism. For example, paper wrinkles not only alter the spatial shape of text but also simultaneously affect surface illumination and shadow formation. These effects can arise naturally within the same physical scene.

Regarding validation on real-world data, the Kendall's τ between Pose and Skew reaches 0.79, while that between Geometry and Warping reaches 0.36. These results indicate that our current simulation can already reflect differences in model robustness under real-world acquisition conditions for certain factors.

The long-term value of this work lies in opening up a new research direction: moving from reliance on real-world data collection toward proactively generating and controlling acquisition conditions through physical mechanisms. In the future, more complex imaging factors, such as ink reflectance, paper surface optical properties, and moiré patterns, can be further incorporated to progressively cover a wider range of real-world acquisition phenomena.

AuthorsOpenReview ↗

Response to Reviewer WM1V (Part 2/3): Question 2

Q2: The ranking of influential factors may depend on the selected perturbation strengths and sampling choices. Its stability under alternative settings needs further validation.

A2: Thank you for raising this concern. Our perturbation parameters are not selected arbitrarily; instead, they are designed according to the physical properties of individual factors. For example, multiple intensity levels are configured for factors such as Roll, Pitch, and Motion Blur, while different candidate configurations are adopted for factors such as Paper Material and Watermark. The specific parameters and sampling rules are provided in Appendix B.1.

Furthermore, we employ paired source-page bootstrap resampling to estimate the uncertainty of performance differences, avoiding the treatment of multiple variants generated from the same document as independent samples.

We would also like to emphasize that the factor rankings reported in this paper reflect model sensitivity under the predefined parameter configurations and sampling protocol, rather than universal rankings that hold across arbitrary perturbation intensities. We will further clarify this point in the revised manuscript and explicitly state that the parameters of different factors do not constitute a unified physical severity scale.

AuthorsOpenReview ↗

Response to Reviewer WM1V (Part 3/3): Question 3

Q3: Prior work has already established the importance of physical acquisition robustness. The paper should more clearly demonstrate what its differences from existing benchmarks enable researchers to discover.

A3: Thank you for raising this key question. The primary difference between SimDocBench and existing benchmarks such as Real5-OmniDocBench and PureDocBench is that we not only evaluate model performance under degraded conditions, but also analyze the effects of individual physical factors through controllable single-factor interventions.

By separating performance changes introduced by rendering itself from additional degradation caused by acquisition factors, SimDocBench provides three important insights:

  1. Models exhibit different factor sensitivities. The average PertDrop across the eight models for Wrinkles reaches 4.89 points, while the performance drops for PaddleOCR-VL-1.5 and Qianfan-OCR under the same factor are 0.50 and 14.97, respectively.
  2. Clean accuracy is insufficient to predict physical robustness. The Spearman correlation between Neutral Accuracy and Acquisition Gap is only −0.357.
  3. Accuracy and prediction consistency capture different failure behaviors. The correlation between PertDrop and MCD is only 0.199, and 316 out of 616 pairwise model comparisons across factors exhibit ranking reversals.

These findings suggest that SimDocBench provides not only finer-grained evaluation data, but also factor-level attribution and model failure diagnosis capabilities that are difficult to achieve with existing holistic benchmarks. We will incorporate these findings and the corresponding discussion into the revised manuscript to further clarify the unique contributions of SimDocBench over existing benchmarks. We hope that this controllable physical simulation framework can serve as a foundation for subsequent research on robustness analysis, targeted data augmentation, and model improvement.

The authors' response has addressed my concerns, I will maintain the positive rating.

Reviewer_hvLAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SimDocBench, a benchmark designed to evaluate the robustness of document parsing models in real-world physical acquisition scenarios. Addressing the limitation that existing benchmarks struggle to isolate the impact of specific physical acquisition factors on document parsing performance, the authors—leveraging the 1,651-page document corpus from OmniDocBench—constructed a dataset containing approximately 158,000 images. This dataset incorporates 31 distinct physical degradation factors, simulated through a combination of physical paper modeling, photorealistic rendering, and camera effect modeling. Furthermore, the authors established both "Regular" and "Hard" multi-factor test scenarios. For evaluation, they propose a "Source–Neutral–Perturbed" protocol, which systematically analyzes mainstream document parsing models across three dimensions: parsing accuracy, prediction consistency, and recoverability. Experiments reveal that a model’s accuracy on clean documents is not a sufficient indicator of its physical robustness; different models exhibit varying degrees of sensitivity to factors such as paper creases, material variations, and geometric distortions. Additionally, the authors validated the effectiveness of several simulation results using real-world captured data. In summary, the primary contribution of this work is the provision of a fine-grained, controllable benchmark for assessing the influence of various physical degradation factors on document parsing models.

Critical strengths
  1. This paper focuses on the robustness of document parsing under real-world physical acquisition conditions, providing a means to evaluate the reliability of existing OCR and document parsing models in complex scenarios.
  2. By employing physical paper simulation, photorealistic rendering, and camera-effect modeling, the authors constructed a test set encompassing 31 types of physical degradation factors, enabling fine-grained, single-factor analysis while keeping the document content constant.
  3. Beyond parsing accuracy, the study incorporates analyses of prediction consistency and recoverability. It compares eight mainstream document parsing models, revealing how their sensitivity to various physical degradation factors differs.
Critical weaknesses
  1. Existing benchmarks, such as Real5-OmniDocBench[1] and PureDocBench[2], have already explored the robustness of document parsing under various physical acquisition or degradation conditions, and they support paired evaluations. While this paper expands the scope of degradation factors to a more granular set of 31 types, it has not yet sufficiently demonstrated what new insights—unattainable through existing benchmarks—are gained from this finer-grained categorization.
  2. The paper employs a relatively complex 3D physical simulation pipeline; however, it lacks a direct comparison with traditional 2D image augmentation methods. Consequently, it remains unclear whether this simulation process can uncover model failure modes that conventional image perturbations fail to reveal.
  3. Real-to-Sim experiments indicate that the changes in model performance under simulated degradations align strongly with those observed under real-world acquisition conditions only for a subset of factors. For instance, the Kendall’s τ coefficients for "lighting" and "screen capture" are merely 0.07 and 0.00, respectively—a finding that, to some extent, undermines the representativeness of the simulation results for real-world scenarios.

References: [1] Real5-OmniDocBench: A full-scale physical reconstruction benchmark for robust document parsing in the wild. ECCV 2026. [2] How Far Is Document Parsing from Solved? PureDocBench: A Source-TraceableBenchmark across Clean, Degraded, and Real-World Settings. ArXiv 2026.

Questions for authors

I may consider raising my score if the authors provide a deeper analysis, particularly regarding the second point:

  1. The geometric distortions, skew, and various forms of image degradation addressed in the paper could potentially be mitigated by existing document rectification or image restoration methods. Although the authors conduct an "Oracle Restoration" analysis, they lack experiments involving the integration of actual document rectification models with the parsing system. Consequently, it remains unclear whether these degradation factors continue to pose a significant challenge following standard preprocessing.
  2. While the paper primarily demonstrates performance degradation caused by various physical factors, it lacks a systematic analysis of the interactions between these factors. Furthermore, it does not verify whether the fine-grained diagnostic results can effectively guide model training or data augmentation. Therefore, the practical utility of this benchmark for future research on enhancing model robustness remains to be validated.
3 comments
AuthorsOpenReview ↗

Response to Reviewer hvLA (Part 1/2): Question 1

We sincerely thank you for the constructive suggestions regarding document rectification, interactions among physical factors, and robustness enhancement. We will answer the questions below and hope to clarify the concerns.

Q1: The geometric distortions, skew, and various forms of image degradation addressed in the paper could potentially be mitigated by existing document rectification or image restoration methods. Although the authors conduct an "Oracle Restoration" analysis, they lack experiments involving the integration of actual document rectification models with the parsing system. Consequently, it remains unclear whether these degradation factors continue to pose a significant challenge following standard preprocessing.

A1: Thank you for raising this important question. We agree that existing document rectification and image restoration methods can mitigate certain physical degradations to some extent, such as perspective transformations, page warping, and geometric distortions. However, this does not necessarily imply that degradation caused by real-world physical acquisition can already be adequately addressed by existing preprocessing methods.

In fact, several recent studies have continued to investigate document parsing robustness under real-world physical acquisition conditions. For example, Real5-OmniDocBench evaluates document parsing models under real-world acquisition scenarios through physical reconstruction, while PureDocBench systematically evaluates model performance under different conditions, including Clean, Degraded, and Real-World settings. The emergence of these benchmarks indicates that document parsing under real-world physical acquisition conditions remains a practically relevant and insufficiently addressed problem. In particular, performance degradation caused by real-world acquisition cannot simply be attributed to geometric distortions, but instead involves multiple complex physical factors.

More importantly, existing document rectification methods are typically designed to address specific types of degradation. For example, geometric rectification methods primarily handle spatial distortions such as perspective variations and page warping. However, their effectiveness may be considerably limited when dealing with factors such as paper material variations, folds, wrinkles, shadows, non-uniform illumination, blur, and screen photography. Furthermore, the simultaneous presence of multiple physical factors may introduce even more complex visual degradations. Therefore, even when existing standard preprocessing methods are applied, we cannot assume in advance that all physical degradations can be effectively eliminated. Their actual effectiveness still requires systematic experimental validation.

We would like to emphasize that the goal of SimDocBench is not to replace existing document rectification methods, but rather to systematically characterize the effects of different physical acquisition factors on document parsing models and identify the vulnerabilities that existing systems still exhibit under real-world physical acquisition conditions.

AuthorsOpenReview ↗

Response to Reviewer hvLA (Part 2/2): Question 2

Q2: While the paper primarily demonstrates performance degradation caused by various physical factors, it lacks a systematic analysis of the interactions between these factors. Furthermore, it does not verify whether the fine-grained diagnostic results can effectively guide model training or data augmentation. Therefore, the practical utility of this benchmark for future research on enhancing model robustness remains to be validated.

A2: Thank you for raising this important question. To investigate whether synthesizing physical degradation factors can effectively guide model training or data augmentation, we conducted an additional experiment.

As shown in Figure 3, wrinkles are among the most influential factors affecting OCR model performance. Therefore, to examine whether targeted training can improve robustness to paper deformation, we fine-tuned FireRed-OCR using only wrinkle images generated by the SimDocBench pipeline. We split 1,355 source pages into training, validation, and test sets of 948, 135, and 272 pages, respectively, grouped by document and near-duplicate clusters to prevent data leakage. Among these, 824 eligible training pages provided 2,472 SimDocBench images. We trained for four epochs with LoRA rank 128 and a learning rate of 2e-5, selecting checkpoints exclusively based on SimDocBench validation performance.

The following table compares FireRed-OCR's performance before and after fine-tuning on SimDocBench wrinkle images, original OmniDocBench images, and the Real5-OmniDocBench evaluation dataset.

DatasetOriginalFine-tunedChange
SimDocBench Wrinkles90.9992.56+1.57
OmniDocBench Original94.0393.49−0.54
Real5-OmniDocBench Warping90.3991.07+0.67

The results demonstrate that SimDocBench's fine-grained diagnostic findings can effectively guide targeted robustness improvements. By fine-tuning FireRed-OCR exclusively on SimDocBench-generated wrinkle images, targeting a degradation factor identified as particularly influential in our analysis, we achieved a 1.57-point improvement on the SimDocBench wrinkle test set and a 0.67-point improvement on the real-world Real5-OmniDocBench Warping subset.

These findings suggest that targeted training interventions for specific physical degradation factors can not only improve performance under corresponding simulated conditions but also transfer to real-world acquisition scenarios involving similar physical deformations. Overall, these preliminary results demonstrate that SimDocBench can serve not only as a diagnostic benchmark for identifying model sensitivities to different physical factors, but also as a practical guide for targeted training data generation and robustness enhancement.

I appreciate the author’s additional experiments; the current response has slightly increased my confidence in the paper. However, as this guidance does not appear to have been broadly validated across a variety of scenarios, the paper remains limited to an analytical level and lacks systematic guidance or analysis for practical applications. Therefore, I will maintain my current rating.

Reviewer_rz3yOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SimDocBench, a benchmark for evaluating document parsing under controlled physical acquisition changes. Starting from 1,651 OmniDocBench pages, the authors combine thin-sheet simulation, physically based rendering, and camera-effect modeling to construct 158,496 single-factor images covering 31 factors, alongside Regular and Hard multi-factor settings. A Source–Neutral–Perturbed protocol distinguishes rendering losses from additional acquisition losses. Eight parsers are evaluated using task-balanced accuracy, prediction consistency, and oracle-assisted restoration, with additional comparison against content-matched real captures. The experiments reveal substantial differences in degradation between parsers with similar clean-document accuracy.

Critical strengths
  1. Useful controlled evaluation design. The Source–Neutral–Perturbed protocol explicitly accounts for the accuracy changes introduced by rendering, while factor-matched neutral controls support more informative analysis of individual acquisition factors (Sections 4.1–4.2). Combining these controls with propagated annotations and visibility screening makes the benchmark useful for diagnosing failures that aggregate clean-document scores cannot adequately characterize.

  2. Substantial empirical evidence of model-dependent vulnerability. The eight-parser evaluation demonstrates meaningful robustness differences. For example, MinerU2.5-Pro and PaddleOCR-VL-1.6 have similar source scores but achieve Hard scores of 54.19 and 78.61, respectively (Table 1). Paired source-page bootstrap intervals, restoration diagnostics, and the content-matched real-capture comparison strengthen the empirical contribution and help establish the benchmark’s practical relevance.

Critical weaknesses
  1. Real-capture agreement is uneven across acquisition families. Table 3 shows strong rank agreement for Skew (Kendall τ = 0.79), but weaker evidence for Warping (τ = 0.36, 95% CI [0.00, 0.71]) and Illumination (τ = 0.07, 95% CI [−0.43, 0.36]). The authors acknowledge these limitations, but the evidence currently supports transfer most clearly for pose changes. This limits how broadly the simulated robustness rankings can be interpreted as reflecting performance under real acquisition conditions.

  2. The consistency analysis requires a more explicit specification and comparison protocol. Section 4.3 and Appendix C.3 do not fully specify MCD component similarities, prediction alignment, or the treatment of missing components. Moreover, MR2 uses selected mild/medium variants on a subset of pages, whereas the main PertDrop evaluation uses broader support. It is unclear whether the reported accuracy–consistency correlation was computed on identical page–variant pairs. These details matter because metric implementation and sampling differences could influence the conclusion that accuracy degradation and prediction stability capture distinct behaviors.

Questions for authors
  1. Clarification of real-capture validation: For Warping and Illumination, could you provide the underlying per-parser real and simulated score drops and clarify whether the weak agreement is driven by particular factor presets or differences in acquisition severity? This would help establish the scope of the benchmark’s real-world validity.

  2. Clarification of consistency evaluation: Could you specify the MCD component similarities, alignment procedure, and missing-component rules? Were the reported correlation of 0.199 and the ranking reversals computed with PertDrop and MCD evaluated on identical pages and perturbation variants? If not, does the conclusion remain after matching this support?

3 comments
AuthorsOpenReview ↗

Response to Reviewer rz3y (Part 1/2): Question 1

We sincerely thank you for the careful evaluation of SimDocBench and for recognizing the value of our controlled evaluation design and empirical investigation. We will address the questions point by point below and hope to clarify the concerns.

Q1: Clarification of real-capture validation: For Warping and Illumination, could you provide the underlying per-parser real and simulated score drops and clarify whether the weak agreement is driven by particular factor presets or differences in acquisition severity? This would help establish the scope of the benchmark’s real-world validity.

A1: Thank you for your valuable question. We have supplemented the performance drops of all eight parsers under real and simulated conditions on the same set of 1,355 source pages. We further investigated the differences between real acquisition conditions and single-factor simulations.

The performance drop under real conditions is calculated as Scan Image Score − Real-capture Image Score, while the simulated performance drop is calculated as Corresponding Neutral-rendered Image Score − Perturbed Image Score.

ParserWarping (Real)Warping (Simulated)Illumination (Real)Illumination (Simulated)
GLM-OCR1.280.530.32−0.10
MinerU3.410.750.51−0.10
OvisOCR23.530.551.430.06
Paddle 1.52.280.301.050.21
Paddle 1.62.030.331.220.00
Hunyuan3.24−0.012.28−3.59
FireRed3.210.560.60−0.01
Qianfan8.882.293.52−0.10

Real acquisition categories and single-factor simulations are not perfectly matched comparisons. We manually inspected paired examples with identical content across six source strata and found that some Real5-OmniDocBench Illumination photographs exhibit additional variations in page scale, position, and skew, beyond illumination changes. Therefore, the performance drop relative to scanned images may reflect the combined effects of illumination and other acquisition variations, rather than illumination alone.

Controlled single-factor simulation is precisely the complementary capability provided by SimDocBench. In our single-factor rendering experiments, page content, paper geometry, material, and camera configurations are held fixed, while only the designated factor is varied.

In real-world acquisition, obtaining such controlled comparisons typically requires fixing the camera position and pose (e.g., using a tripod), paper position, and imaging parameters, while independently adjusting illumination. Real5-OmniDocBench does not provide such controlled conditions; consequently, its performance drops may be confounded by substantial skew variations. In contrast, our simulations can directly fix these conditions, allowing researchers to distinguish a model's sensitivity to illumination itself from its sensitivity to factors such as geometry and viewing angle.

Therefore, real-world data is used to evaluate performance under combined acquisition conditions, whereas controlled simulations are used to identify specific failure conditions. The two serve complementary purposes.

We appreciate the reviewer's constructive feedback. We will include the per-parser performance drops in the revised manuscript and further emphasize the role of controlled single-factor evaluation in failure localization.

AuthorsOpenReview ↗

Response to Reviewer rz3y (Part 2/2): Question 2

Q2: Clarification of consistency evaluation: Could you specify the MCD component similarities, alignment procedure, and missing-component rules? Were the reported correlation of 0.199 and the ranking reversals computed with PertDrop and MCD evaluated on identical pages and perturbation variants? If not, does the conclusion remain after matching this support?

A2: We thank the reviewer for pointing out this issue. Indeed, our original comparison did not use exactly the same pages and perturbed images for both metrics. We have therefore recomputed the results using a matched evaluation set. Below, we describe the computation procedure and the updated results.

First, how is MCD computed?

MCD compares the outputs of the same model on the same page under two different conditions, measuring how much the text, formulas, tables, and reading order change. It does not compare predictions against ground-truth annotations. Therefore, MCD measures output stability rather than recognition correctness.

The four types of components are compared as follows:

ComponentComparison
TextText content and block types, such as headings and paragraphs
FormulaMathematical commands and symbols after removing LaTeX delimiters and other formatting markers
TableNumber of rows, number of columns, and cell contents
Reading OrderEach block is represented by its type and its first four words/symbols. We then compare the ordering of these representations and their adjacent relationships.

We use Dice similarity to measure the overlap between words, symbols, or block markers, considering both individual elements and combinations of adjacent elements. Dice similarity is used for component-level comparisons, while MCD aggregates the differences across components.

For multiple formulas or tables, we identify corresponding items while preserving their original output order, allowing items to be missing or extra on either side. An unmatched item is assigned a difference of 1, while a matched pair is assigned a difference of 1 − similarity. We select the alignment that minimizes the total difference and then normalize it by the larger number of items in the two outputs to obtain the component-level difference.

Finally, we take the equally weighted average of the differences across applicable components to obtain MCD. A lower MCD indicates that the two outputs are more similar.

Second, how are missing components handled?

For formulas and tables, if both outputs contain no such components, the corresponding component is excluded from the calculation. If only one output contains such components, the difference is assigned a value of 1.

For text and reading-order features, if both outputs are empty, the difference is 0; if only one output is empty, the difference is 1. Consequently, when both outputs are entirely empty, the resulting MCD is also 0.

Third, do the conclusions still hold when exactly the same images are used?

We recomputed the comparison using the common MR2 evaluation images shared by all eight models. Each model contributes 3,703 image pairs, covering 22 factors and 995 distinct source pages. Both metrics use exactly the same source pages, perturbed variants, neutral reference images, and weighting schemes.

Aggregation MethodCorrelation Coefficient (ρ)Number of Pairwise Model Comparisons with Opposite Rankings
Equal weight per image pair0.119300/616 (48.7%)
Equal weight per source page0.119296/616 (48.1%)

The 616 comparisons consist of 28 pairwise model comparisons among the eight models for each of the 22 factors. Confidence intervals were estimated using 2,000 source-page-level bootstrap resamples.

These results indicate that even when exactly the same images are used, models with smaller accuracy drops do not necessarily produce more stable outputs. Thus, this observation still holds under a strictly matched evaluation setting.

We thank the reviewer for identifying this issue. In the final revised manuscript, we will provide the detailed MCD computation rules and update the correlation analysis using exactly the same evaluation images for both metrics.

Thank you for the detailed responses and additional experiments. The rebuttal has clarified some of my concerns, and I appreciate the authors' efforts to address the feedback. However, some of the limitations discussed in my original review remain. Overall, my assessment remains unchanged, and I will keep my original rating.

Reviewer_QRByOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SimDocBench evaluates document parsing under simulated acquisition changes. It builds on 1,651 annotated source pages and uses paper simulation, rendering, and image postprocessing to generate 31 types of perturbations, together with Regular and Hard multi-factor settings. The Source–Neutral–Perturbed protocol separates changes introduced by rendering from those caused by acquisition perturbations. Eight parsers are evaluated in terms of accuracy, prediction consistency, and recovery after oracle input correction.

Critical strengths

The paired evaluation design is useful. Rendering alone changes model scores, so comparing perturbed images directly with the original pages would mix rendering effects with acquisition effects. Introducing neutral controls makes the analysis more informative. Resampling source pages rather than treating all rendered variants as independent observations is also appropriate.

The benchmark exposes differences that are not obvious from clean-page scores. For example, MinerU2.5-Pro and PaddleOCR-VL-1.6 have similar neutral accuracy but perform very differently under the Hard setting.

Critical weaknesses

The benchmark is the main contribution of this paper, but I have serious concerns about how easily it could be gamed. As I understand it, SimDocBench is an augmented version of existing document benchmarks, generated using reproducible tools for 2D transformations, 3D deformation, and rendering. What prevents a model from quickly improving its benchmark scores simply by applying the same tools to its training data? Such gains could reflect adaptation to the augmentation pipeline rather than better robustness to real-world document capture. The authors need to address this concern directly. Without evidence that the benchmark can distinguish these two outcomes, its value as a robustness evaluation is substantially weakened.

The added value of physical simulation over simpler augmentation is not established. Physically plausible paper geometry and rendering do not by themselves establish realistic acquisition conditions. Applying a source image as a texture may miss differences between paper and ink reflectance, while postprocessed motion blur may not capture the spatially varying effects of camera or document motion. The paper acknowledges some of these limitations, but the current evidence supports a controlled synthetic diagnostic more clearly than a general benchmark of real-world physical robustness.

Questions for authors
  1. Could training with the same augmentation tools easily boost SimDocBench scores without improving real-world robustness? Please test this and compare the gains on SimDocBench with those on real-capture benchmarks.

  2. Since the augmentation pipeline combines 2D and 3D transformations, could the authors clarify whether it covers corner cases such as light reflecting off ink and text deformation caused by paper wrinkles?

  3. The evaluation of specialized OCR models is not sufficiently detailed. Could the authors compare how much end-to-end and two-stage parsers are affected, and separate layout detection failures from recognition errors?

  4. The dataset combines OmniDocBench v1.5 with 296 additional pages. Where do the annotations for these additional pages come from, and how was their quality verified to be comparable to OmniDocBench v1.5?

5 comments
AuthorsOpenReview ↗

Response to Reviewer QRBy (Part 1/4): Question 1

We sincerely thank you for recognizing the value of our neutral control mechanism and benchmark analysis. We will answer the questions below, and we hope this clears up your concerns.

Q1: Could training with the same augmentation tools easily boost SimDocBench scores without improving real-world robustness? Please test this and compare the gains on SimDocBench with those on real-capture benchmarks.

A1: Thank you for highlighting this potential risk. To directly investigate this concern, we conducted an additional small-scale fine-tuning experiment on FireRed-OCR using SimDocBench-generated wrinkle images.

We split the 1,355 source pages shared by Real5-OmniDocBench into training, validation, and test sets of 948, 135, and 272 pages, respectively. Images derived from the same source page in OmniDocBench, SimDocBench, and Real5-OmniDocBench were always assigned to the same split. We fine-tuned FireRed-OCR for four epochs using only 2,472 SimDocBench wrinkle images generated from 824 eligible training source pages, with LoRA rank 128 and a learning rate of 2e-5. Checkpoints were selected exclusively based on SimDocBench validation performance.

We compared the original and fine-tuned models on the SimDocBench wrinkle test set, the original OmniDocBench images, and the Real5-OmniDocBench Warping photographs. Following the OmniDocBench paper, the evaluation score was computed as the equally weighted average of three task metrics: text, formula CDM, and table TEDS.

DatasetOriginalFine-tunedChange
SimDocBench Wrinkle90.9992.56+1.57
OmniDocBench Original94.0393.49−0.54
Real5-OmniDocBench Warping90.3991.07+0.67

The results show that, even when trained exclusively on SimDocBench synthetic data and with checkpoints selected solely using synthetic validation data, FireRed-OCR achieves a +0.67 improvement on real-world Warping photographs, rather than improving only on the synthetic test set (+1.57). This suggests that training on SimDocBench does not merely lead to overfitting to a specific synthetic pipeline; some of the learned robustness can also transfer to real-world acquisition conditions.

Meanwhile, the smaller improvement on real-world data compared with synthetic data indicates that a synthetic-to-real gap remains. Nevertheless, these results provide direct evidence that SimDocBench can not only diagnose robustness weaknesses in document parsers but also potentially guide improvements in real-world robustness.

AuthorsOpenReview ↗

Response to Reviewer QRBy (Part 2/4): Question 2

Q2: Since the augmentation pipeline combines 2D and 3D transformations, could the authors clarify whether it covers corner cases such as light reflecting off ink and text deformation caused by paper wrinkles?

A2: Thank you for the question regarding the physical simulation details. We would like to further clarify the scope of our current pipeline in modeling paper geometry and material optical properties.

First, regarding paper geometry, we employ ARCSim-based thin-shell simulation to model physical deformations of paper, including folding, curling, twisting, and wrinkling. Since the original page content is mapped as a texture onto the deformed paper surface, the text undergoes corresponding spatial deformations as the paper geometry changes. Consequently, text bending, local geometric distortions, and spatial variations in reading regions caused by paper wrinkles are already covered by our current simulation framework. In addition, our framework models the effects of paper smoothness and reflectance on surface illumination and reflections. These effects are incorporated into the 'Paper material' factor.

Second, our pipeline includes a stain simulation factor, as illustrated by the 'Stain' example in Figure 1(a), as well as simulations of paper-surface reflections. These components allow our current synthetic pipeline to approximate certain visual effects associated with ink glare.

We envision SimDocBench's 3D simulation pipeline not merely as a fixed collection of image augmentation operations, but as an extensible physical scene simulation framework. In the future, more sophisticated paper materials, ink optical properties, illumination conditions, and camera imaging factors could be incorporated to progressively improve its coverage of real-world acquisition processes.

An important advantage of 3D physical simulation over conventional 2D image transformations is its ability to model physical constraints and interactions among different acquisition factors at the generation-mechanism level. Traditional 2D augmentation typically applies operations such as rotation, deformation, blurring, or brightness adjustment directly to the final image. Although these operations can approximate certain acquisition effects, they generally require individual image-space transformations to be defined separately.

In contrast, our approach first models a document as a three-dimensional paper surface with deformable geometry and material properties, and then renders it under different camera viewpoints, illumination conditions, and material configurations. Consequently, paper geometry deformations, local text distortions, perspective projection, and illumination variations induced by changes in surface orientation can arise naturally within a unified physical scene.

AuthorsOpenReview ↗

Response to Reviewer QRBy (Part 3/4): Question 3

Q3: The evaluation of specialized OCR models is not sufficiently detailed. Could the authors compare how much end-to-end and two-stage parsers are affected, and separate layout detection failures from recognition errors?

A3: Thank you for this valuable suggestion. We have conducted two additional experiments: one comparing the performance degradation of different parsing paradigms, and another investigating whether the degradation originates from Layout Detection or Text Recognition.

First, we compared the degradation of all eight OCR models. The two-stage group includes GLM-OCR, MinerU, and PaddleOCR-VL-1.5/1.6, which first detect document layouts and then recognize individual regions. The end-to-end group includes Hunyuan, OvisOCR2, FireRed, and Qianfan, which directly parse the entire page. All models were evaluated on the same 1,651 source pages: Regular contains five images per page (8,255 images in total), while Hard contains one image per page (1,651 images in total).

Each entry in the following table reports the mean performance drop for the two-stage / end-to-end groups, compared with their respective matched unperturbed reference images. Text, formula, and table performance are measured using edit-based accuracy, CDM, and TEDS, respectively. The overall score is the average of these three metrics.

DatasetOverall DropText DropFormula DropTable Drop
Regular2.31 / 2.902.45 / 2.721.89 / 2.032.57 / 3.94
Hard23.17 / 23.5726.16 / 19.0619.27 / 20.8224.08 / 30.84

Under Regular conditions, the two-stage group exhibits smaller overall and table performance drops. Under Hard conditions, however, the two-stage group experiences greater degradation in text recognition, whereas the end-to-end group experiences greater degradation in table recognition.

Second, to distinguish between different failure stages, we evaluated GLM-OCR on the same images using two pipelines:

  1. Standard pipeline: The model independently detects the locations of text, formula, and table regions before recognizing their contents.
  2. Ground-truth layout pipeline: We provide the model with the ground-truth annotated regions, region types, and reading order, and then let the same recognizer process the corresponding content.

Importantly, we do not provide the ground-truth text content or perform any image restoration. The original distortions, including skew, warping, and illumination variations, are preserved.

Both pipelines were evaluated on Neutral, all Regular images, and all Hard images, with identical recognition and decoding configurations.

The following table reports the results. The first three metrics measure performance directly, where higher values are better. The last two report performance drops, where lower values are better.

MetricNeutralRegularHard
Precision of detected regions (%)79.0978.7381.61
Recall of detected regions (%)84.9882.8857.36
Recognition score with ground-truth text regions (%)94.5191.6266.71
Overall drop (standard pipeline)—1.9221.09
Overall drop (ground-truth layout)—1.8012.55

Under Hard conditions, Recall decreases from 84.98% to 57.36%, indicating that a substantially larger proportion of regions fail to be correctly detected. Even when ground-truth text regions are provided, the recognition score still decreases from 94.51% to 66.71%, demonstrating that the recognition stage itself also suffers considerable degradation.

Providing the ground-truth layout reduces the overall performance drop under Hard conditions from 21.09 to 12.55, but does not eliminate the degradation. Under Regular conditions, the additional recovery is only 0.11 points, suggesting that the improvement is limited and not conclusive.

These results indicate that severe acquisition perturbations affect both Layout Detection and Text Recognition.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.