- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SimDocBench evaluates document parsing under controlled physical acquisition changes. Using thin-sheet simulation, path-traced rendering, and camera effects, it generates 158,496 single-factor images covering 31 factors, together with Regular and Hard multi-factor settings, from 1,651 annotated source pages. A Source–Neutral–Perturbed protocol distinguishes rendering-induced changes from acquisition-induced degradation. The benchmark evaluates parsing accuracy, prediction consistency, and recoverability through controlled input restoration.
Critical strengths
-
The Source–Neutral–Perturbed protocol separates rendering effects from acquisition-induced degradation, while 31 controlled factors enable matched comparisons on the same document content. This provides more informative diagnosis than a single clean-versus-degraded evaluation.
-
Eight parsers exhibit substantially different degradation despite similar clean accuracy, with large differences supported by paired source-page confidence intervals. The additional consistency and restoration analyses extend the benchmark evaluation spectrum.
Critical weaknesses
-
Agreement with real acquisition is strong for Skew but weak or uncertain for Warping and Illumination. The factor-coverage analysis also does not establish whether physical rendering provides better real-world diagnostic value than simpler, severity-matched transformations.
-
MCD lacks precise component-matching and missing-output rules, and the accuracy–consistency comparison does not clearly use identical pages and perturbation severities. Unexplained differences between Regular and Hard neutral scores further limit the reproducibility and interpretation of the diagnostic results.
Questions for authors
Please respond to my concern about weaknesses
3 comments
Response to Reviewer KXAw (Part 2/2): Question 2
Q2: MCD lacks precise component-matching and missing-output rules, and the accuracy–consistency comparison does not clearly use identical pages and perturbation severities. Unexplained differences between Regular and Hard neutral scores further limit the reproducibility and interpretation of the diagnostic results.
A2: Thank you for pointing out these issues. We have checked our implementation and conducted additional comparisons using exactly the same evaluation inputs.
First, we clarify the MCD computation rules.
MCD compares two outputs generated by the same parser for the same page under Neutral and Perturbed conditions. It measures how much the outputs change, rather than comparing them against ground-truth annotations.
| Component | Comparison |
|---|---|
| Text | Text content and block types, such as headings and paragraphs |
| Formula | Normalized LaTeX tokens and symbols |
| Table | Number of rows, number of columns, and cell contents |
| Reading Order | Each block is represented by its type and its first four words/symbols. We compare these representations and their adjacent relationships. |
Content similarity considers the overlap of both individual words/symbols and adjacent word/symbol pairs.
For multiple formulas or tables, we identify the alignment that minimizes the total difference while preserving their original order, allowing insertions and deletions. The cost of a missing item is 1, while the cost of a matched pair is 1 − similarity. The total difference is then normalized by the larger number of items in the two outputs. Finally, we compute the equally weighted average of the distances across applicable components. The complete formulas and fixed component weights will be provided in the appendix.
If neither output contains formulas or tables, the corresponding component is excluded from the calculation. If a component is missing from only one output, its distance is assigned a value of 1. For text and reading-order features, the distance is 0 when both outputs are empty and 1 when only one output is empty. Consequently, two completely empty outputs can also yield MCD = 0, indicating that consistency should be interpreted together with accuracy. Missing prediction files caused by execution errors are not treated as empty outputs for scoring purposes.
Second, we use exactly the same evaluation inputs for both metrics.
Our original comparison did indeed use different evaluation subsets. We have now fixed exactly the same source pages, perturbed images, and Neutral reference images, and recomputed the accuracy drops and MCD. Each of the eight parsers is evaluated on 3,703 input pairs, covering 995 source pages and 22 factors.
| Aggregation Method | Spearman ρ (95% CI) | Pairwise Ranking Reversals |
|---|---|---|
| Equal weight per input pair | 0.119 [−0.018, 0.228] | 300/616 |
| Equal weight per source page | 0.119 [−0.018, 0.230] | 296/616 |
The 616 comparisons consist of all pairwise comparisons among the eight models across 22 factors. The 300 ranking reversals indicate cases where accuracy degradation and MCD lead to opposite conclusions about which of two models is more robust. Confidence intervals are estimated using 2,000 bootstrap resamples with shared source-page sampling.
Therefore, even after matching the evaluation pages and perturbation images, the two metrics still produce different robustness rankings.
Third, why do the Neutral scores differ between Regular and Hard?
Upon checking, we found that the two settings use the same pages and reference inputs for their Neutral scores, but the Hard scores for several models were taken from an earlier scoring version, resulting in inconsistent scoring versions in the table.
Regarding the N and NH columns, the values in the N column are the correct scores for the Neutral pages. For example, FireRed's Neutral score is corrected from 88.48 to 89.60, while its Hard score remains unchanged at 68.72. Accordingly, the performance drop is corrected from 19.76 to 20.88.
We thank the reviewer for identifying this issue and will correct Table 1 and the corresponding results throughout the final revised manuscript.
Thank you for the detailed response and the additional experiments.
The MCD specification, the matched-support recomputation, and the N/N_H discrepancy are adequately addressed. It would also be good to add them to the main paper
But still, some concerns I raised have not been resolved. In the comparison, all three 3D−2D confidence intervals include zero, and Warping shows no difference. The per-parser table also shows simulated drops far below real ones. The additional real-world evaluation value of physical simulation is not proven. Besides, only the four end-to-end parsers have N ≠ . Was H rescored with the current version?
Overall, I would keep my current rating.
Response to Reviewer KXAw (Part 1/2): Question 1
We sincerely thank you for the careful evaluation of SimDocBench. We will address the concerns below and hope to clarify them.
Q1: Agreement with real acquisition is strong for Skew but weak or uncertain for Warping and Illumination. The factor-coverage analysis also does not establish whether physical rendering provides better real-world diagnostic value than simpler, severity-matched transformations.
A1: Thank you for this valuable suggestion. An important advantage of 3D physical simulation over conventional 2D image transformations is its ability to model physical constraints and interactions among different acquisition factors at the generation-mechanism level.
Traditional 2D augmentation typically applies image-space operations, such as rotation, deformation, blurring, or brightness adjustment, directly to the final image. In contrast, our approach first models a document as a three-dimensional paper surface with deformable geometry and material properties, and then renders it under different camera viewpoints, illumination conditions, and material configurations. Consequently, paper geometry deformations, local text distortions, perspective projection, and illumination variations induced by changes in surface orientation can arise naturally within a unified physical scene.
Physical rendering allows us to independently control paper shape, material, illumination, and camera parameters. For example, when the paper curvature changes, text projection, surface shading, and shadows change jointly with the scene, while the annotations are propagated consistently with the geometric transformations. This enables researchers to repeatedly construct specific acquisition conditions and investigate the circumstances under which models fail.
To examine whether these physical variations provide greater real-world diagnostic value than simple 2D augmentation, we conducted the following additional controlled comparison:
In other words, we investigate whether models that experience larger performance drops on real photographs also exhibit larger drops on simulated images. In the following table, a higher τ indicates stronger agreement between the rankings induced by simulation and those observed under real-world acquisition.
The 3D simulation produces rankings that are more consistent with real-world acquisition under Skew. However, the confidence interval for its improvement over 2D still includes zero.
We manually inspected paired examples with identical content from six source-page strata and found that some Real5-OmniDocBench Illumination photographs exhibit additional variations in page scale, position, and skew, beyond illumination changes. The Illumination subset in Real5-OmniDocBench is therefore confounded by substantial skew variations.
In real-world acquisition, obtaining such controlled comparisons typically requires fixing the camera position and pose (e.g., using a tripod), paper position, and imaging parameters, while independently adjusting illumination. In contrast, our simulations can directly fix these conditions, allowing researchers to distinguish a model's sensitivity to illumination itself from its sensitivity to factors such as geometry and viewing angle.