acceptodds
Under review as a conference paper at ICLR 2027

Beyond Visual Bounds: Generation-Aware Layout Analysis for Editable Visual Documents

Abstract

Vision-language models (VLMs) have advanced text recognition and document parsing. However, when a parsed text element is edited, bounding boxes that capture only its visible extent do not specify the space available for replacement text or how that text should be aligned within that space. We introduce Generation-aware Layout Analysis (GLA), a task that recovers text, allocated regions, semantic categories, and alignment to provide spatial constraints for subsequent element-level visual editing. This representation could support editable slide and document reconstruction, translation with layout preservation, and automated design editing. For training and evaluation on this task, we construct a 37K-image training dataset spanning nine domains and introduce GLA-Bench, a 1,000-image benchmark with four tasks. To predict dense coordinate and attribute sequences accurately and efficiently, we introduce joint supervised fine-tuning (SFT), reinforcement learning (RL) with layout-specific rewards, and accelerated decoding. To recover local feedback obscured by image-level aggregation, we introduce EAA-GRPO, which augments group-relative policy optimization (GRPO) with Element-level Advantage Assignment (EAA). EAA compares predictions for the same element across rollouts to provide field-specific supervision. To accelerate decoding, we adapt a block-diffusion drafter to the repetitive structure of parsing outputs. On GLA-Bench, our 4B parser reaches 46.59 [email protected] in full parsing, versus 6.26 for the best evaluated zero-shot baseline. Layout-only RL improves full parsing on both backbones. Task-adapted drafting achieves 2.18 autoregressive throughput with the SFT target fixed.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Layout Analysis, Vision-Language Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RWTMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Generation-aware Layout Analysis (GLA), a task that recovers text, per-element allocated regions (the space available for replacement content rather than visible text bounds), semantic categories, and alignment from raster document images, as spatial constraints for element-level editing and translation. The authors build a 37K-image HTML-rendered training set across nine domains and curate GLA-Bench, a 1,000-image human-verified benchmark with four diagnostic tasks. They train 4B VLM parsers with multi-task SFT, propose EAA-GRPO (assigning element-level advantages by comparing the same element across rollouts), and accelerate decoding with MTP and task-adapted DFlash. The 4B parser reaches 46.59 [email protected] on full parsing versus 6.26 for the best zero-shot baseline; layout-only RL transfers to full parsing on both backbones; task-adapted drafting reaches 2.18× autoregressive throughput.

Critical strengths
  1. Well-motivated and clearly formulated task (Sec. 1, Fig. 1). The visible-bounds vs. allocated-region distinction is concrete and practically relevant; the representation is independently measurable, decoupled from full document derendering (Sec. 2).
  2. Solid dataset engineering: a hierarchical, cue-based annotation policy (Sec. 3.1), human correction with cross-checking and adjudication for the benchmark (App. B.4), and a complete category/alignment vocabulary (App. B.3).
  3. EAA is well-motivated: image-level GRPO aggregation can obscure element-level differences in dense outputs, and the across-rollout comparison is cleanly differentiated from MCR-GRPO's within-response normalization (Sec. 2, Sec. 4.2, Fig. 5).
  4. Broad evaluation: four complementary tasks with strict joint metrics (Sec. 3.3), plus a blinded 20-reviewer human arena with bootstrap CIs whose rankings agree with benchmark rankings (Table 8).
  5. Good engineering transparency: hardware, hyperparameters, and measurement boundaries are documented (App. C.1), including the differing timing protocols for MTP and DFlash.
Critical weaknesses
  1. The benchmark's annotation quality is not yet established. Sec. 3.2 states "We plan an independent agreement assessment" — i.e., no IAA exists at submission time. Since allocated regions are admittedly ambiguous and resolved by a conservative heuristic policy, IAA is essential to interpret what any [email protected] score means. Relatedly, App. C.3 defines Fleiss' κ, P_o, U, Spearman's ρ and Kendall's τ but never reports their values. Both are fixable and I ask the authors to provide them in the rebuttal.
  2. All external baselines are zero-shot general-purpose VLMs (Table 2). The headline "46.59 vs 6.26" mostly reflects task-specific adaptation (the 4B base models score 0.00 Joint with CER far above 100%, indicating format failure). Missing: (a) an equally sized VLM fine-tuned on the same data, to separate data vs. method contributions; (b) a pipeline baseline (OCR + layout detector + heuristics). dots.ocr is cited but never evaluated.
  3. No end-to-end validation of the motivating applications. The paper's premise is that allocated regions help editing/translation, but no experiment feeds GLA outputs into an actual editing or translation pipeline to compare against visible-bounds constraints. Current evidence shows the representation is learnable and measurable, not that it helps generation.
  4. Evidence for EAA-GRPO is thinner than its narrative weight. The benchmark ablation covers a single backbone (Table 4), the gain over GRPO is +1.05 [email protected] without any variance across seeds, Fig. 3 uses only 64 validation images, and the GRPO row for Qwen3-VL-4B is missing. Key design choices (ε_box=0.05, the 0.95 trusted threshold, η=0.5) are not ablated.
  5. Table 3's interpretation is confounded: after SFT on allocated-region targets, the same model is compared against two different reference types, so the crossover mixes training-target alignment with task difficulty. It demonstrates that explicit supervision works, but only indirectly that spatial allocation is intrinsically learnable.
  6. The 867/934 element-level subset sampling is unspecified; no per-domain breakdown; MTP (Table 5) uses [email protected] and a different timing protocol than DFlash (Table 6); related work on layout-aware RL for document parsing (e.g., Infinity Parser, arXiv:2506.03197) and speculative decoding for document parsing VLMs (e.g., HSD, arXiv:2602.12957) is not discussed — please verify and comment.
Questions for authors
  1. Can you provide preliminary IAA for allocated regions (e.g., IoU statistics on a doubly-annotated subset) and category/alignment agreement, plus the human-agreement ceiling on [email protected]?
  2. Please report the Fleiss' κ\kappa, PoP_o, U, Spearman's ρ and Kendall's τ\tau defined in App. C.3.
  3. How were the 867 bbox2text and 934 text2bbox instances sampled from the 19,858 elements?
  4. Can you add a same-data, same-scale SFT baseline on a different backbone (or a pipeline baseline) to isolate the contribution of the training data vs. the framework?
  5. Can you report GRPO (without EAA) on Qwen3-VL-4B, and mean±std over at least 2–3 seeds for Table 4?
  6. Why is RL restricted to img2layout? Does adding img2full RL (with a text reward) help or hurt?
  7. Can you provide a small end-to-end comparison (allocated region vs. visible bounds as constraints in a text-replacement or translation pipeline)?
  8. Does the 2.18× DFlash speedup hold under the MTP timing protocol (including initialization), and under higher concurrency?
7 comments
AuthorsOpenReview ↗

Response to Reviewer RWTM (Part 1/7)

We thank you for your positive assessment of our task, dataset, EAA design, and evaluation. Please kindly find our response to your comments below. Additionally, all modifications to the manuscript have been highlighted in blue for easy reference. Please feel free to let us know if you have any additional concerns or questions.

W1 & Q1 — Annotation agreement and the human Joint reference

We randomly sample 100 images from the nine domains of GLA-Bench, containing 1,621 elements, for independent double annotation. We report boundary IoU, category agreement, and alignment agreement, grouped by explicit containers versus regions inferred from structure, whitespace, or neighboring elements. We also compute inter-annotator layout [email protected] under the same evaluation protocol as a human-agreement reference.

Region typeImages / ElementsMean / Median IoU (%)IoU ≥ .75 (%)Category agreement (%) / κAlignment agreement (%) / κHuman layout [email protected] (%)
All regions100 / 1,62185.93 / 88.2687.5094.39 / 0.93496.32 / 0.89879.73
Explicit containers or separators78 / 64594.08 / 94.9097.7897.21 / 0.96598.35 / 0.94892.79
Inferred from structure, whitespace, or neighboring elements91 / 97680.55 / 82.9580.7092.52 / 0.91294.98 / 0.87073.30

The overall human layout [email protected] is 79.73%, with 92.79% for explicitly constrained regions and 73.30% for inferred regions. Boundary agreement is higher when explicit constraints are available, while category and alignment agreement remain high in both groups.

We have added the protocol and agreement results to Appendix B.4, “Human Annotation and Quality Control” (Table 8), and summarized the main findings in Section 3.2.


W1 & Q2 — Human-study agreement and ranking correlations

Thank you for pointing out the missing statistics in Appendix C.3. Using the three initial judgments for each comparison, before discussion or adjudication, we compute Fleiss’ κ\kappa, pairwise observed agreement PoP_o, and unanimity rate UU for both arenas. We also compare the human preference rankings with benchmark rankings based on [email protected] on the shared set of 100 images.

ArenaComparisons / VotesFleiss’ κP_oUSpearman’s ρ_sKendall’s τ_b
Training-stage150 / 4500.70282.89%76.00%1.0001.000
Zero-shot250 / 7500.64579.60%71.60%1.0001.000

The human and benchmark rankings agree completely in both arenas, with Spearman’s ρs\rho_s and Kendall’s τb\tau_b both equal to 1.000. We have added these statistics in Table 11, alongside the human rankings in Table 10, in Appendix C.3.

AuthorsOpenReview ↗

Response to Reviewer RWTM (Part 2/7)

W2 & Q4 — Separating data and method contributions (1/2)

(1) The reason why the base models have zero Joint scores with normal CER.

As explained in Appendix C.4, CER measures transcription given a target box, whereas full-image [email protected] requires the same element to satisfy IoU ≥ .75, correct category and alignment, and exact normalized text matching simultaneously. The zero Joint scores arise from incorrect category predictions rather than the format failure: the unadapted models have 0.00 category F1, preventing elements from contributing to Joint. In Fig. 15, for example, the prediction is valid JSON, has IoU 0.898, and predicts the correct alignment, but labels a Text element as affiliation. Thus, incorrect field values can yield zero Joint even when the output is parseable.

(2) Contribution of task-specific data.

The base-versus-SFT comparisons on the two 4B backbones show the gains from supervised task adaptation, with the same four-task metrics as the main table:

Modelbbox2text CER ↓bbox2text EM ↑text2bbox mIoU ↑text2bbox mAcc ↑img2layout mF1 ↑img2layout [email protected] ↑img2full mF1 ↑img2full Cat.F1 ↑img2full Align. ↑img2full [email protected] ↑
Qwen3-VL-4B68.7161.9447.1524.2411.250.0011.160.0057.690.00
Qwen3-VL-4B + SFT10.0087.7773.1359.7651.8243.7050.9479.9791.2743.96
Qwen3.5-4B127.6448.1033.2114.057.840.000.170.0048.860.00
Qwen3.5-4B + SFT13.7186.3966.0547.4740.4133.8838.9578.6493.0232.93

SFT increases img2full mF1 by 39.78 points on Qwen3-VL-4B and 38.78 points on Qwen3.5-4B, and raises [email protected] from 0.00 to 43.96 and 32.93, respectively. On text2bbox, allocated-region mIoU increases by 25.98 and 32.84 points. On bbox2text, EM increases by 25.83 and 38.29 points. These comparisons establish the contribution of task-specific supervision on both backbones.

(3) Contribution of EAA.

Comparing EAA-GRPO with standard GRPO under the same reward function, SFT initialization, and training data isolates the contribution of element-level advantage assignment. We report the complete four-task comparison as means ± standard deviations across three seeds:

BackboneMethod after SFTbbox2text CER ↓bbox2text EM ↑text2bbox mIoU ↑text2bbox mAcc ↑img2layout mF1 ↑img2layout [email protected] ↑img2full mF1 ↑img2full Cat.F1 ↑img2full Align. ↑img2full [email protected] ↑
Qwen3.5-4BGRPO12.74 ± 1.6385.97 ± 1.2666.98 ± 0.2948.84 ± 0.3743.10 ± 0.2837.08 ± 0.5141.84 ± 0.2480.79 ± 0.4193.39 ± 0.0636.03 ± 0.44
Qwen3.5-4BEAA-GRPO11.95 ± 1.3586.74 ± 1.1767.25 ± 0.2249.39 ± 0.3644.69 ± 0.2138.16 ± 0.3743.20 ± 0.1782.43 ± 0.3893.52 ± 0.0837.21 ± 0.39
Qwen3-VL-4BGRPO9.93 ± 0.9588.67 ± 0.8273.51 ± 0.3360.31 ± 0.3952.86 ± 0.5145.12 ± 0.7151.65 ± 0.4680.34 ± 0.3491.40 ± 0.1545.20 ± 0.46
Qwen3-VL-4BEAA-GRPO9.89 ± 1.0788.92 ± 0.9973.75 ± 0.3060.59 ± 0.4355.53 ± 0.3247.07 ± 0.6854.13 ± 0.2380.66 ± 0.3292.05 ± 0.1346.88 ± 0.52

On Qwen3.5-4B, EAA improves img2layout mF1 and [email protected] by 1.59 and 1.08 points, and img2full mF1 and [email protected] by 1.36 and 1.18 points. On Qwen3-VL-4B, the corresponding gains are 2.67 and 1.95 points for img2layout and 2.48 and 1.68 points for img2full. The regional task means also improve on both backbones. These controlled comparisons distinguish EAA's additional gains from those due to supervised task adaptation. We have added the complete comparison to Appendix C.10 (Table 16).

AuthorsOpenReview ↗

Response to Reviewer RWTM (Part 3/7)

W2 & Q4 — Separating data and method contributions (2/2)

(4) Same-data document-parsing baseline.

Thank you for the suggestion. We include SFT controls for two related models: dots.ocr, which targets document layout and content parsing, and LocateAnything, which provides regional localization and partial parsing capabilities. We fine-tune both on the same GLA training data, four-task mixture, and output schema as our Qwen-based parsers, and evaluate them before and after SFT using the same benchmark protocol. The metric order follows Table 2 of the paper.

Modelbbox2text CER ↓bbox2text EM ↑text2bbox mIoU ↑text2bbox mAcc ↑img2layout mF1 ↑img2layout [email protected] ↑img2full mF1 ↑img2full Cat.F1 ↑img2full Align. ↑img2full [email protected] ↑
dots.ocr34.5560.210.000.0012.840.0013.2014.660.000.00
dots.ocr + SFT38.6259.8640.4420.4914.239.4314.1924.4892.8210.17
LocateAnything138.8044.5254.9830.1210.350.009.710.100.000.00
LocateAnything + SFT27.4166.7866.8048.0426.7314.8925.3351.2490.2710.31

SFT increases img2full mF1 from 13.20% to 14.19% for dots.ocr and from 9.71% to 25.33% for LocateAnything, gains of 0.99 and 15.62 percentage points, respectively. Their [email protected] scores increase from 0.00% to 10.17% and 10.31%. On text2bbox, mIoU increases from 0.00% to 40.44% for dots.ocr and from 54.98% to 66.80% for LocateAnything.

Native dots.ocr targets full-page document parsing, whereas text2bbox asks it to return the region corresponding to a specified text query. Under the current query protocol, its responses do not provide valid boxes accepted by the evaluation interface, giving 0.00 for both mIoU and mAcc. Its native output also lacks the alignment field required by GLA, so Align. and [email protected] are 0.00. SFT adapts dots.ocr to the GLA query and output schema, yielding text2bbox mIoU and mAcc of 40.44% and 20.49% and img2full alignment accuracy of 92.82%. The gains from initially zero scores reflect improved instruction following and output-schema compliance, enabling the model to produce valid GLA-structured predictions, rather than improvements in localization alone. These results demonstrate the value of task-interface adaptation. We have added the settings, four-task results, and explanation of the zero scores to Appendix C.7, “SFT Adaptation of Related Models” (Table 13).

[1] LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding

[2]dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model

Reviewer_A4hJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces generation-aware layout analysis, which recovers text, allocated regions, semantic categories, and alignment from document images. The distinction between visible text bounds and the space available for replacement text is the main motivation. The authors construct 37,418 automatically annotated HTML-rendered training images and a 1,000-image, human-corrected benchmark covering four tasks. They train a parser with multi-task SFT and EAA-GRPO, which compares the same element across sampled outputs and adds local advantages to its coordinate tokens. Experiments show improvements in parsing on two 4B backbones. A separate decoding study achieves 2.18× autoregressive throughput by adapting a DFlash drafter while fixing the SFT target.

Critical strengths
  1. The paper addresses a useful distinction between visible text bounds and available editing space, and provides a benchmark for studying it.

  2. EAA-GRPO provides a simple, well-motivated way to assign local feedback by comparing the same element across rollouts.

  3. RL improves parsing on both backbones, and the comparison with GRPO under the same reward supports the additional benefit of EAA.

Critical weaknesses
  1. The editing motivation is convincing, but the experiments mainly measure parsing quality. The human study also compares parsing outputs rather than edited documents. A small controlled editing experiment would help show whether better allocated-region predictions lead to better edits.

  2. Allocated-region boundaries can have several reasonable interpretations, especially when there are no clear containers or separators. The annotation rules and manual checks are helpful, but Section 3.2 still lists the independent agreement study as planned. Without these results, it is hard to judge how consistently different annotators interpret the target.

  3. The training data cover nine source domains, but the images are HTML-rendered reconstructions. The current evaluation does not clearly establish performance on scanned or photographed pages, naturally degraded documents, or layouts that differ substantially from these reconstructions. The coverage of rotated and vertical text is also unclear. These cases matter for the practical use of the method.

Questions for authors
  1. Could you report agreement between annotators before adjudication for region boundaries, category, and alignment? Even a small independently annotated subset would be useful. I would be particularly interested in whether agreement differs between regions with explicit containers and those inferred from whitespace or neighboring elements.

  2. Please specify the detection and format rewards in Eq. 4, the matching-cost weights and rules for leaving elements unmatched, and the criterion for an ambiguous match.

  3. Do you have results on scanned or photographed pages, naturally degraded images, or layouts that differ substantially from the HTML reconstructions used for training? It would also help to clarify how rotated or vertical text is handled and whether it is covered by evaluation.

5 comments
AuthorsOpenReview ↗

Response to Reviewer A4hJ (Part 1/3)

We thank you for recognizing the value of allocated regions and EAA-GRPO. Please kindly find our response to your comments below. Additionally, all modifications to the manuscript have been highlighted in blue for easy reference. Please feel free to let us know if you have any additional concerns or questions.

W1 — Controlled editing evaluation

Thank you for the suggestion. We conduct a controlled text-replacement experiment on 100 randomly sampled document images from GLA-Bench. We use a fixed editing and rendering workflow to replace the text. Specifically, we modify the structural metadata of the same target text block with the same replacement text, using either conventional visible-text bounds or manually annotated allocated regions as the layout constraint.

We evaluate readability, layout preservation, typesetting failure rate, and mean font-size retention. Readability covers text clarity, completeness, line breaks, and spacing. Layout preservation covers alignment, column structure, whitespace, visual hierarchy, and consistency with neighboring elements. Typesetting failure rate measures the proportion of replacements that cannot fit within the given region even at the preset minimum font size. Mean font-size retention measures the ratio of the final rendered font size to the original font size, averaged across cases.

Ten experts participate in a blinded pairwise evaluation of readability and layout preservation, choosing A, B, or a tie for each dimension and briefly recording their reasons. The left–right order is randomized, and method names are hidden. We compute the two typesetting metrics as

FailureRate=NfailNedit×100%,\mathrm{FailureRate}=\frac{N_{\mathrm{fail}}}{N_{\mathrm{edit}}}\times100\%,

FontRetention=1∣S∣∑i∈Ssieditsioriginal×100%.\mathrm{FontRetention}=\frac{1}{|\mathcal S|}\sum_{i\in\mathcal S}\frac{s_i^{\mathrm{edit}}}{s_i^{\mathrm{original}}}\times100\%.

Here, NfailN_{\mathrm{fail}} is the number of typesetting failures and Nedit=100N_{\mathrm{edit}}=100 is the number of editing cases in each constraint group. The shared set S\mathcal S contains the 89 paired cases in which both constraints yield successful typesetting. The variables siedits_i^{\mathrm{edit}} and sioriginals_i^{\mathrm{original}} denote the final rendered and original font sizes.

Evaluation dimensionVisible-text bounds winTieAllocated regions win
Readability12%24%64%
Layout preservation10%29%61%
MetricVisible-text boundsAllocated regions
Typesetting failure rate ↓10/100 (10%)3/100 (3%)
Mean font-size retention ↑81.7%92.4%

Under the same editing workflow, allocated regions win 64% of readability comparisons and 61% of layout-preservation comparisons, while reducing the typesetting failure rate from 10% to 3%. On the 89 successful pairs, mean font-size retention is 10.7 percentage points higher. These results show that accurate allocated regions allow more replacement text to fit while better preserving the original font size and page layout. This experiment uses manually annotated regions to directly test the editing value of the representation.

Figure 17 shows a cover title, a handwritten note, and a table header. Allocated regions accommodate longer replacements while preserving title prominence, note readability, and header consistency. Fitting the same text into visible-text bounds requires substantially smaller fonts. Together, these results demonstrate the practical editing value of accurate allocated regions.

We have added the experimental protocol, metric definitions, quantitative results, and visual comparisons to Appendix C.6, “Text Replacement with Allocated Regions” (Table 12 and Figure 17).

AuthorsOpenReview ↗

Response to Reviewer A4hJ (Part 2/3)

W2 & Q1 — Agreement on ambiguous allocated regions

Thank you for the suggestion. We randomly sample 100 images spanning the nine domains of GLA-Bench, containing 1,621 elements, for independent double annotation and evaluate inter-annotator agreement on allocated-region boundaries, categories, and alignment. We report the results separately for regions constrained by explicit containers or separators and those inferred from structure, whitespace, or neighboring elements.

Region typeImages / ElementsMean / Median IoU (%)IoU ≥ .75 (%)Category agreement (%) / κAlignment agreement (%) / κ
All regions100 / 1,62185.93 / 88.2687.5094.39 / 0.93496.32 / 0.898
Explicit containers or separators78 / 64594.08 / 94.9097.7897.21 / 0.96598.35 / 0.948
Inferred from structure, whitespace, or neighboring elements91 / 97680.55 / 82.9580.7092.52 / 0.91294.98 / 0.870

Allocated-region annotations show high human agreement. Boundary agreement is higher for explicitly constrained regions, while regions inferred from structure, whitespace, or neighboring elements exhibit more boundary variation. Category and alignment agreement remain high in both groups.

We have added the protocol and agreement results to Appendix B.4, “Human Annotation and Quality Control” (Table 8), and summarized the main findings in Section 3.2.


W3 & Q3 — Generalization to scanned and photographed pages

To evaluate generalization to real scanned and photographed pages, we conduct supplementary inference experiments with Qwen3-VL-4B + SFT on the Scanning and Skew subsets of Real5-OmniDocBench (Zhou et al., 2026). Direct GLA annotation of these physical images requires manual specification of allocated regions, categories, and alignment. Given the labor involved and the rebuttal timing budget, we randomly sample 100 images from each subset and establish GLA reference annotations with manual consistency checks. We use the same img2full evaluation protocol as in the main paper.

ConditionmF1 ↑Cat.F1 ↑Align. ↑[email protected] ↑
Scanning (N = 100)64.4187.5684.0258.28
Skew (N = 100)61.2975.3379.8856.72

We compute the following inference statistics on the full subsets, separately from the 100-image annotation subsets:

ConditionTruncation rate (%) ↓Format compliance (%) ↑
Scanning10.888.4
Skew14.884.4

The higher scores relative to GLA-Bench may reflect differences in content and layout composition. GLA-Bench spans nine domains, including interfaces, slides, infographics, and documents, whereas this supplementary evaluation focuses on scanned and photographed pages. Scanning images typically retain regular page geometry and text orientation, making these characteristics closer to those of HTML-rendered training images. Skew introduces perspective distortion and tilted text, which can increase the difficulty of region localization and attribute prediction. The Skew scores are correspondingly lower than the Scanning scores. The results support the generalization of the parser trained on HTML-rendered images to these real acquisition conditions.

Figure 19 illustrates predictions on an English scanned page, a skewed photographed page, and a Chinese scanned page, showing recovered text, allocated regions, semantic categories, and alignment. The tilted text in the photographed example also motivates future exploration of polygonal regions and rotation or perspective transformations for non-axis-aligned layouts.

We have added the protocol, quantitative results, and qualitative examples to Appendix C.9, “Generalization to Scanned and Photographed Pages” (Table 15 and Figure 19).

AuthorsOpenReview ↗

Response to Reviewer A4hJ (Part 3/3)

Q2 — Reward definitions and matching rules

Thank you for pointing out these reproducibility details. We have clarified the detection and format rewards in Section 4.2 and added the detailed reward definitions, matching rules, and parameters to Appendix A.2.

Detection and format rewards. In Eq. (4), the weights of FgdetF_g^{\mathrm{det}} and FgfmtF_g^{\mathrm{fmt}} are 0.1 and 0.05, respectively. The detection term is geometric F1 at IoU ≥ .50:

Fgdet=2TPgN+Kg,F_g^{\mathrm{det}} = \frac{2\mathrm{TP}_g}{N + K_g},

where NN is the number of reference elements, KgK_g is the number of predicted records, and TPg\mathrm{TP}_g counts pairs from the one-to-one matching below that meet the IoU threshold, without requiring correct category or alignment.

FgfmtF_g^{\mathrm{fmt}} is a binary reward: it is 1 when a complete, untruncated JSON list passes checks for required fields, coordinate ranges and ordering, and the category and alignment vocabularies. It is 0 otherwise. The output schema is given in Appendices B.5 and C.2. When format validation fails, recoverable valid boxes still contribute to the other rewards, and parsed invalid records remain in the detection denominator.

Matching cost and unmatched elements. Section 4.2 introduces the matching of predicted and reference elements for each rollout. Appendix A.2 gives the following matching cost and the detailed rules:

Cji=14∥b^j−bi∥1+1−GIoU⁡(b^j,bi).C_{ji} = \frac{1}{4}\left\|\hat{\boldsymbol b}_j - \boldsymbol b_i\right\|_1 + 1 - \operatorname{GIoU}(\hat{\boldsymbol b}_j,\boldsymbol b_i).

Coordinates are divided by 1,000 before computing the cost. The coefficients of the mean L1 term and the GIoU term are both 1. A predicted element and a reference element can be paired only if their center distance is ≤ .25 and their matching cost is < 1.5. We choose the one-to-one matching with the lowest total cost, allowing predicted or reference elements to remain unmatched. Unmatched reference elements receive zero element reward under Eq. (3) and remain in the denominator NN in Eq. (4).

Ambiguous assignments. We check whether the predictions can be reliably paired with the reference elements in each rollout. Let J∗J^* be the minimum total matching cost. For each reference element ii, we temporarily disallow its current pairing and recompute the best overall matching. If the element is currently unmatched, we disallow its corresponding unmatched option. Let Ji−J_i^{-} be the total cost of this alternative matching. If

Ji−−J∗≤0.05,J_i^{-} - J^* \le 0.05,

the alternative matching has nearly the same cost as the original. We mark every reference element whose pairing changes, together with the affected predictions, as ambiguous. Equal-cost alternatives also satisfy this rule. For these elements in that rollout, we exclude their rewards from the local comparison across rollouts and skip the additional coordinate-level update in Eqs. (6)–(7). Their rewards remain in the page-level score in Eq. (4), as specified in Appendix A.2. The 0.05 here is the threshold on the increase in total matching cost. The other 0.05 in Eq. (6) and Appendix C.1 is the minimum standard deviation used when normalizing local advantages.

[3] Real5-OmniDocBench: A Full-Scale Physical Reconstruction Benchmark for Robust Document Parsing in the Wild

Reviewer_Dde5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Generation-Aware Layout Analysis (GLA) for identifying text content and its layout information from visual documents. Unlike conventional text detection boxes, the core annotation of GLA is the "assigned area" where text elements can be replaced, and it simultaneously predicts semantic categories and horizontal alignment methods. The authors have constructed GLA-Bench, which includes 1,000 manually verified images, as well as a training set automatically annotated from 37,418 images; the benchmark supports four types of tasks: text recognition, text localization, layout parsing, and complete parsing. In addition, the author conducted multi-task supervised fine-tuning on two 4B visual language models and employed EAA-GRPO for the layout task, providing a more fine-grained reward through element matching across sampled outputs. The paper also investigated MTP and task-adapted DFlash speculative decoding. The author reported that SFT significantly improved the performance of complete parsing, EAA-GRPO further enhanced the metrics based on SFT, and task-adapted DFlash could increase inference throughput.

Critical strengths

1.An extension of document parsing. Predicting the space available for each text element, along with its content, semantic role, and alignment, is a sensible step toward editable visual documents.

2.A substantial benchmark and training set. GLA-Bench contains 1,000 manually verified images from nine domains, while the larger training set supports experiments across both element-level and full-image tasks.

3.Consistent gains from task-specific training. SFT improves results on both 4B backbones, and EAA-GRPO provides further gains. The paper also measures the throughput of its task-adapted decoding setup.

Critical weaknesses

1.The experiments do not directly test the editing motivation. Higher parsing scores do not yet show that allocated regions produce better text replacement, reflow, or visual preservation than conventional text bounds.

2.The methodological contributions are not equally well established. EAA-GRPO is the clearest new method, but its gain over standard GRPO is modest and lacks multi-seed evidence. MTP and DFlash are existing techniques adapted to this task.

3.The comparison does not adequately test the post-training claim. Large gains over zero-shot VLMs show that supervised task data helps, but the paper does not fine-tune a relevant document-parsing or layout-analysis model on the same data and output schema.

4.The data pipeline and annotation reliability need clarification. The paper should state unambiguously whether GLA-Bench evaluates original source images or HTML reconstructions, explain why the generated HTML/DOM is not used more directly for supervision, and report annotator agreement on allocated regions.

5.The backbone results lack error analysis. Qwen3.5-4B trails Qwen3-VL-4B substantially on localization and joint accuracy despite slightly better category and alignment scores. The paper does not establish whether this comes from visual grounding, element discovery, output formatting, or training settings.

Questions for authors

See the weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Dde5 (Part 1/3)

We thank you for your constructive feedback and recognition of our task and benchmark. Please kindly find our response to your comments below. Additionally, all modifications to the manuscript have been highlighted in blue for easy reference. Please feel free to let us know if you have any additional concerns or questions.

W1 — Editing benefits of allocated regions

Thank you for the suggestion. We conduct a controlled text-replacement experiment on 100 randomly sampled document images from GLA-Bench. We use a fixed editing and rendering workflow to replace the text. Specifically, we modify the structural metadata of the same target text block with the same replacement text, using either conventional visible-text bounds or manually annotated allocated regions as the layout constraint.

We evaluate readability, layout preservation, typesetting failure rate, and mean font-size retention. Readability covers text clarity, completeness, line breaks, and spacing. Layout preservation covers alignment, column structure, whitespace, visual hierarchy, and consistency with neighboring elements. Typesetting failure rate measures the proportion of replacements that cannot fit within the given region even at the preset minimum font size. Mean font-size retention measures the ratio of the final rendered font size to the original font size, averaged across cases.

Ten experts participate in a blinded pairwise evaluation of readability and layout preservation, choosing A, B, or a tie for each dimension and briefly recording their reasons. The left–right order is randomized, and method names are hidden. We compute the two typesetting metrics as

FailureRate=NfailNedit×100%,\mathrm{FailureRate}=\frac{N_{\mathrm{fail}}}{N_{\mathrm{edit}}}\times100\%,

FontRetention=1∣S∣∑i∈Ssieditsioriginal×100%.\mathrm{FontRetention}=\frac{1}{|\mathcal S|}\sum_{i\in\mathcal S}\frac{s_i^{\mathrm{edit}}}{s_i^{\mathrm{original}}}\times100\%.

Here, NfailN_{\mathrm{fail}} is the number of typesetting failures and Nedit=100N_{\mathrm{edit}}=100 is the number of editing cases in each constraint group. The shared set S\mathcal S contains the 89 paired cases in which both constraints yield successful typesetting. The variables siedits_i^{\mathrm{edit}} and sioriginals_i^{\mathrm{original}} denote the final rendered and original font sizes.

Evaluation dimensionVisible-text bounds winTieAllocated regions win
Readability12%24%64%
Layout preservation10%29%61%
MetricVisible-text boundsAllocated regions
Typesetting failure rate ↓10/100 (10%)3/100 (3%)
Mean font-size retention ↑81.7%92.4%

Under the same editing workflow, allocated regions win 64% of readability comparisons and 61% of layout-preservation comparisons, while reducing the typesetting failure rate from 10% to 3%. On the 89 successful pairs, mean font-size retention is 10.7 percentage points higher. These results show that accurate allocated regions allow more replacement text to fit while better preserving the original font size and page layout. This experiment uses manually annotated regions to directly test the editing value of the representation.

Figure 17 shows a cover title, a handwritten note, and a table header. Allocated regions accommodate longer replacements while preserving title prominence, note readability, and header consistency. Fitting the same text into visible-text bounds requires substantially smaller fonts. Together, these results demonstrate the practical editing value of accurate allocated regions.

We have added the experimental protocol, metric definitions, quantitative results, and visual comparisons to Appendix C.6, “Text Replacement with Allocated Regions” (Table 12 and Figure 17).


W2 — EAA-GRPO gains and the role of decoding techniques

Thank you for the suggestion. We report means ± standard deviations across three seeds for Qwen3.5-4B on img2full.

MethodmF1 ↑Cat.F1 ↑Align. ↑[email protected] ↑
GRPO w/o Det./Fmt.41.62 ± 0.2180.43 ± 0.4693.25 ± 0.0635.42 ± 0.51
GRPO41.84 ± 0.2480.79 ± 0.4193.39 ± 0.0636.03 ± 0.44
EAA-GRPO43.20 ± 0.1782.43 ± 0.3893.52 ± 0.0837.21 ± 0.39

EAA-GRPO achieves higher mean scores than GRPO on all four metrics, improving Cat.F1 and [email protected] by 1.64 and 1.18 percentage points, respectively. The mean gains exceed the corresponding reported standard deviations, supporting consistent improvements across seeds. We have updated Table 4 and added the complete four-task comparison in Appendix C.10 (Table 16).

GLA produces structured sequences containing thousands of tokens, making MTP and DFlash useful for improving decoding efficiency. We adapt these existing techniques to the task and verify their practical acceleration benefits. We position this as an efficiency study. The main contributions are the GLA task and benchmark, together with EAA-GRPO.

AuthorsOpenReview ↗

Response to Reviewer Dde5 (Part 2/3)

W3 — Same-data document-parsing baseline

Thank you for the suggestion. We include SFT controls for two related models: dots.ocr, which targets document layout and content parsing, and LocateAnything, which provides regional localization and partial parsing capabilities. We fine-tune both on the same GLA training data, four-task mixture, and output schema as our Qwen-based parsers, and evaluate them before and after SFT using the same benchmark protocol. The metric order follows Table 2 of the paper.

Modelbbox2text CER ↓bbox2text EM ↑text2bbox mIoU ↑text2bbox mAcc ↑img2layout mF1 ↑img2layout [email protected] ↑img2full mF1 ↑img2full Cat.F1 ↑img2full Align. ↑img2full [email protected] ↑
dots.ocr34.5560.210.000.0012.840.0013.2014.660.000.00
dots.ocr + SFT38.6259.8640.4420.4914.239.4314.1924.4892.8210.17
LocateAnything138.8044.5254.9830.1210.350.009.710.100.000.00
LocateAnything + SFT27.4166.7866.8048.0426.7314.8925.3351.2490.2710.31

SFT increases img2full mF1 from 13.20% to 14.19% for dots.ocr and from 9.71% to 25.33% for LocateAnything, gains of 0.99 and 15.62 percentage points, respectively. Their [email protected] scores increase from 0.00% to 10.17% and 10.31%. On text2bbox, mIoU increases from 0.00% to 40.44% for dots.ocr and from 54.98% to 66.80% for LocateAnything.

Native dots.ocr targets full-page document parsing, whereas text2bbox asks it to return the region corresponding to a specified text query. Under the current query protocol, its responses do not provide valid boxes accepted by the evaluation interface, giving 0.00 for both mIoU and mAcc. Its native output also lacks the alignment field required by GLA, so Align. and [email protected] are 0.00. SFT adapts dots.ocr to the GLA query and output schema, yielding text2bbox mIoU and mAcc of 40.44% and 20.49% and img2full alignment accuracy of 92.82%. The gains from initially zero scores reflect improved instruction following and output-schema compliance, enabling the model to produce valid GLA-structured predictions, rather than improvements in localization alone. These results demonstrate the value of task-interface adaptation. We have added the settings, four-task results, and explanation of the zero scores to Appendix C.7, “SFT Adaptation of Related Models” (Table 13).

[1] LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding

[2]dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model

AuthorsOpenReview ↗

Response to Reviewer Dde5 (Part 3/3)

W4 — Image provenance, DOM supervision, and annotation reliability

(1) Image provenance and DOM supervision.

As described under “HTML rendering and image provenance” in Appendix B.1, the images of the training data and GLA-Bench are both rendered from HTML reconstructions.

As explained in Section 3.2, we use text, containers, and spatial organization from HTML/DOM to initialize candidate elements and regions, then refine them into unified GLA supervision targets. DOM nodes do not correspond one-to-one to GLA semantic elements and their allocated regions: a single semantic element may span multiple nodes, while a parent container may contain several independent elements. DOM geometry provides candidate boundaries, and a VLM further determines element granularity and the corresponding region hierarchy under the shared GLA guidelines.

We therefore use HTML/DOM to construct initial structured supervision and apply task-specific mapping and VLM correction to obtain the final GLA annotations. The unified representation covers pages, slides, charts, infographics, and interfaces without depending on a particular webpage’s DOM implementation. We have clarified the relationship between image inputs, candidate annotations, and final supervision targets in Appendix B.1 and added the annotation-agreement results in Section 3.2 and Appendix B.4.

(2) Annotation reliability.

We randomly sample 100 images spanning the nine domains of GLA-Bench, containing 1,621 elements, for independent double annotation and evaluate inter-annotator agreement on allocated-region boundaries, categories, and alignment. We report the results separately for regions constrained by explicit containers or separators and those inferred from structure, whitespace, or neighboring elements.

Region typeImages / ElementsMean / Median IoU (%)IoU ≥ .75 (%)Category agreement (%) / κAlignment agreement (%) / κ
All regions100 / 1,62185.93 / 88.2687.5094.39 / 0.93496.32 / 0.898
Explicit containers or separators78 / 64594.08 / 94.9097.7897.21 / 0.96598.35 / 0.948
Inferred from structure, whitespace, or neighboring elements91 / 97680.55 / 82.9580.7092.52 / 0.91294.98 / 0.870

Allocated-region annotations show high human agreement. Boundary agreement is higher for explicitly constrained regions, while regions inferred from structure, whitespace, or neighboring elements exhibit more boundary variation. Category and alignment agreement remain high in both groups. The two region types can occur in the same image, so their image counts overlap.

We have added the protocol and agreement results to Appendix B.4, “Human Annotation and Quality Control” (Table 8), and summarized the main findings in Section 3.2.


W5 — Localization errors across backbones

Thank you for pointing this out. We perform a paired error analysis of the two SFT checkpoints on GLA-Bench. The diagnostic statistics appear below and in Appendix C.8, “Backbone Error Analysis” (Table 14). Figure 18 provides the visual comparison.

Diagnostic metricQwen3-VL-4BQwen3.5-4B
Output parsing failure rate, img2layout / img2full0.4% / 1.0%0.4% / 0.9%
Mean IoU for the same text elements (%) ↑73.3267.23
Same-text elements with IoU < .75 ↓36.99%50.06%
GT elements with almost no predicted-box coverage ↓4.73%5.66%
Category error rate on commonly matched elements ↓14.15%12.16%
Alignment error rate on commonly matched elements ↓8.07%7.41%

The main difference lies in box localization for already identified elements. Among 12,877 elements with identical, unique normalized text in the reference and both predictions, 2,437 reach IoU ≥ .75 only with Qwen3-VL, compared with 754 only with Qwen3.5. After excluding parsing failures, the [email protected] gaps remain 8.70 and 9.04 percentage points for img2layout and img2full, respectively.

Qwen3.5 makes fewer category and alignment errors on commonly matched elements, explaining why slightly higher attribute scores can coexist with lower Joint scores. The localization gap persists when outputs are parseable and target texts have been identified. Because Joint also requires accurate localization, the small attribute gains do not offset this gap. Figure 18 illustrates this distinction: Qwen3-VL-4B localizes “Stay healthy” more accurately (IoU 0.933 vs. 0.500), while Qwen3.5-4B correctly classifies “About” as ui_button when both models localize it with IoU above .75. We have added these diagnostics and examples to Appendix C.8.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We sincerely thank the AC and all reviewers for their constructive feedback. We appreciate their recognition of the distinction between visible text bounds and allocated editing regions (all reviewers), our dataset and benchmark (Reviewer Dde5, RWTM), and the motivation for element-level advantage assignment (Reviewer A4hJ, RWTM).

All manuscript changes are highlighted in blue. We summarize our responses to the reviewers' comments as follows:

  • Direct validation of the editing value of allocated regions (all reviewers, Appendix C.6). Beyond parsing evaluation, we additionally conducted a controlled text-replacement experiment on 100 GLA-Bench images, keeping the target blocks, replacement texts, and editing workflow fixed while varying the region constraint. In a blinded evaluation by 10 experts, allocated regions win 64% of readability comparisons and 61% of layout-preservation comparisons, versus 12% and 10% for visible-text bounds. Typesetting failures decrease from 10% to 3%. Mean font-size retention increases from 81.7% to 92.4% on the 89 pairs where both constraints allow successful typesetting. This experiment uses manually annotated regions to directly demonstrate the representation's editing value.

  • Annotation reliability and region ambiguity (all reviewers, Section 3.2 and Appendix B.4). Alongside our annotation guidelines and quality checks, we independently double-annotated 100 images across nine domains, covering 1,621 elements. Mean boundary IoU is 85.93%, category and alignment agreement are 94.39% and 96.32%, and human layout [email protected] is 79.73%. We further distinguish explicitly constrained and inferred regions, with mean IoUs of 94.08% and 80.55%, to characterize agreement under different spatial cues. Appendix C.3 also reports human-study agreement and correlations with benchmark rankings.

  • Cross-backbone validation and design analysis of EAA-GRPO (Reviewer Dde5, and RWTM, Table 4 and Appendices C.10-C.12). We additionally provide matched GRPO controls on both backbones and three-seed results across all four tasks. With rewards, SFT initialization, and training data held fixed, EAA improves full-parsing [email protected] from 36.03 ± 0.44 to 37.21 ± 0.39 on Qwen3.5-4B and from 45.20 ± 0.46 to 46.88 ± 0.52 on Qwen3-VL-4B. Design ablations, reward-variance analysis, and a text-reward comparison further examine EAA and our training choices.

  • Same-data adaptation of related models (Reviewer Dde5, and RWTM, Appendix C.7). We additionally fine-tuned dots.ocr and LocateAnything using the same GLA data, four-task mixture, and output schema. Their full-parsing [email protected] scores reach 10.17% and 10.31%. These controls complement our base-to-SFT and GRPO-to-EAA comparisons in distinguishing task adaptation from EAA.

  • Extended evaluation on real scanned and photographed images (Reviewer A4hJ, Appendix C.9). Beyond GLA-Bench's nine domains, we additionally established GLA annotations with manual consistency checks for 100 Scanning and 100 Skew images from Real5-OmniDocBench. Without further training, Qwen3-VL-4B + SFT achieves full-parsing [email protected] of 58.28% and 56.72%, supporting transfer to real acquisition conditions.

We also highlight the broader motivation for GLA:

  1. GLA recovers the space needed for editing, including how much room replacement text can occupy and how it should align. This information matters when translation or content updates change text length. Our controlled experiment shows that accurate allocated regions help accommodate replacements while preserving readability, font size, and layout.

  2. By explicitly predicting allocated regions and alignment alongside text and semantic categories, GLA provides a measurable intermediate representation that can be evaluated independently of a complete document reconstruction system. GLA-Bench and the agreement study support measuring progress while documenting uncertainty in inferred regions. This representation provides a foundation for editable document reconstruction, layout-preserving translation, and automated design editing.

  3. For dense structured outputs, aggregating many element predictions into one image-level score can obscure local differences. EAA-GRPO compares the same element across rollouts to provide element-level supervision for coordinate tokens. Our matched multi-seed controls further support its contribution beyond supervised task adaptation and motivate studying local credit assignment for structured outputs.

We would appreciate the reviewers' feedback on our detailed responses and additional analyses. Please let us know if any concerns remain during the discussion period. We welcome further questions and would be happy to continue the discussion. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.