- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a synthetic document generation framework that preserves exact geometric correspondences during document rendering and degradation. By tracking geometric transformations, the proposed pipeline generates accurate grounding annotations without relying on OCR-based pseudo-labeling. The authors construct a large-scale synthetic dataset and introduce a two-stage training strategy to improve document grounding and extraction. Experiments investigate grounding accuracy, label noise, catastrophic forgetting, and cross-domain generalization.
Critical strengths
-
The paper addresses a practically relevant problem in document understanding, particularly the generation of reliable grounding annotations without expensive manual labeling.
-
The synthetic data generation pipeline is carefully engineered, and the analysis of different label noise patterns provides useful empirical observations.
-
The experiments cover several model scales, training configurations, and evaluation benchmarks, providing a relatively comprehensive empirical investigation.
Critical weaknesses
-
Limited methodological novelty. The main idea of transporting annotations through known geometric transformations is closely related to established synthetic data generation and geometric supervision techniques. More importantly, the homography-only baseline achieves comparable or slightly better grounding performance than the proposed full correspondence field (0.840 vs. 0.838 mIoU), raising questions about the necessity of the central technical contribution.
-
The improvements appear strongly dependent on output-format adaptation. The 500-sample experiment already achieves 0.767 mIoU, suggesting that much of the improvement comes from teaching the model to produce valid coordinates rather than learning substantially stronger visual grounding representations. More controlled comparisons are needed to isolate these effects.
-
The practical benefits for document understanding remain insufficiently demonstrated. Although the method improves exact-match accuracy on real documents, the semantic correctness analysis suggests that much of this gain results from output formatting rather than improved understanding. Furthermore, the pure task-QA baseline achieves higher exact-match accuracy than the final proposed model (0.738 vs. 0.697), indicating a trade-off between grounding and extraction performance.
-
The generalization evidence is not sufficiently convincing. The main real-document evaluation relies on relatively small, non-public subsets, limiting independent verification. Meanwhile, improvements on several public benchmarks involve replaying their training data, making it difficult to attribute the gains to the proposed synthetic supervision. Additional evaluation on independently held-out real-world benchmarks would strengthen the conclusions.
-
The experimental comparisons require further investigation. The comparison between exact labels and OCR-generated pseudo-labels is affected by differences in retained training samples, which may confound the effects of label accuracy and data quantity. In addition, the relatively low accuracy at strict localization thresholds ([email protected] = 0.255) indicates that precise evidence localization remains challenging.
Questions for authors
-
Given that homography-only supervision performs comparably to the full correspondence field, what is the practical advantage of explicitly maintaining dense non-rigid geometric correspondences?
-
Can the authors provide controlled experiments that disentangle improvements from coordinate-format learning, annotation quality, and training-data quantity?
2 comments
I appreciate the author’s time and response, which have addressed my concerns regarding label quality, data volume, and the experiments conducted on real-world datasets. However, I still believe this paper lacks innovation, and the actual improvement in document understanding capabilities remains limited. Furthermore, the quality of the paper does not meet the standards required for ICLR. Therefore, I am maintaining my score.
Response to Reviewer Q4dA
We thank the reviewer for identifying the confounds in the submitted experiments. The new controls changed several of our conclusions.
1. The dense field is unnecessary at our production dose
The homography-only result is a genuine negative result for the central representation. At the production crumple amplitude, homography-only supervision reaches 0.840 mIoU and the full field reaches 0.838. We no longer claim that the dense component always improves training.
We re-rendered matched page sets at larger amplitudes to measure when it does help.
Each pair uses the same source pages and training-item count within its row. Only the crumple correction applied to the training boxes differs. The production and largest doses use three seeds; the middle dose uses two. Different rows use different corpora and test slices, so only within-row arm differences are comparable. At the middle dose, the full field improves [email protected] by 0.055, but 81 test items cannot resolve the stricter-threshold differences. At the largest dose, three seeds show consistent gains at all three thresholds.
The practical advantage is therefore conditional. The dividing region is on the order of a 10 px median shift, not a universal requirement for dense geometry. Persisting the field also supplies the instrument needed to measure this boundary for a new rendering setting.
2. Format learning, annotation quality, and sample count are separated
We added two controls for the requested disentanglement.
The format-versus-precision control compares training-set sizes at several IoU thresholds. With 500 exact labels, the model reaches 0.928 [email protected] but only 0.187 [email protected]. With 20K labels, these values become 0.968 and 0.313. Most of the loose-threshold improvement is indeed obtained early. The larger corpus mainly buys stricter geometric precision.
The annotation-quality control uses exactly the 24,287 image–question pairs retained by the OCR pipeline in both arms.
Dropping 19% of the exact-label samples has no measurable effect. The 0.089 mIoU difference between the matched arms is therefore attributable to label quality rather than sample retention.
3. Independent real-domain evaluation
We added public, third-party human-annotated XFUND-zh and FUNSD tasks, with 150 items per task and no evaluation-image overlap with training.
Ref2box uses three training seeds; the other rows are seed-42 probes. The ± values are seed standard deviations, not sampling confidence intervals.
No FUNSD evaluation image occurs in training, and this evaluation uses no public replay from either target dataset. The final recipe does contain unrelated English DocVQA and InfographicVQA QA replay, but no English box annotations. We therefore claim cross-language grounding transfer, not an English-free training set.
By contrast, synthetic-only training does not improve the public QA benchmarks. We now state this directly. Final-recipe gains on DocVQA, InfographicVQA, and DocVQA-degraded use their training data as replay and are labeled replay repair rather than synthetic transfer.
4. Strict precision and the QA trade-off remain limitations
We agree that 0.255 [email protected] is not sufficient for high-precision automated redaction. The real-domain results are lower still: 0.180 on XFUND-zh and 0.016 on FUNSD. These numbers now appear in the main limitations.
We also agree that the pure-QA arm exposes a trade-off. It reaches 0.738 exact-EM, compared with 0.697 for the final model, but emits no coordinates on any of the 278 ref2box queries. The unified model gives up 4.1 exact-EM points to provide grounding. We no longer describe it as an unqualified reading improvement.
Finally, prompt alignment accounts for 87% of the original +14.8 exact-EM difference. The prompt-aligned base reaches 0.678, leaving a 1.9-point difference to the final model. The revised abstract, results, and conclusion withdraw the unqualified headline.