acceptodds
Under review as a conference paper at ICLR 2027

ReflowDiff: Can Vision-Language Models Reliably Compare Documents Across Layouts?

Abstract

Vision-language models (VLMs) increasingly read, write, and revise documents. Yet it remains unclear whether they can report every textual revision when a document's layout also changes. Concretely, one changed digit or sign alters what a contract or financial report says, and catching it requires fine-grained character recognition, matching corresponding text across layouts, and exhaustive reporting. We introduce ReflowDiff, a benchmark of 4,488 comparison tasks built from 374 dense rich-text pages spanning six generated domains, real web pages, and financial reports. Each page is rendered in two layouts that preserve its text and structural context, and the same set of one to five character edits is applied to both. Two no-edit controls, Identical and Reflow only, test whether models report edits when the text is unchanged. We leverage exact match as the primary metric, which requires every reference edit, with exact before and after text at the correct occurrence, and no additional edits. Across 30 configurations of 26 VLMs, the best reaches 76.4% exact match when edits are combined with reflow. Reflow lowers exact match on the same edits in all 30 configurations, including a 3.8-percentage-point drop for the strongest, and elicits false reports even from configurations that make none on identical pairs. Even when models recover most individual edits, their reports remain incomplete and vary across layouts. These failures highlight a gap between reading document content and reliably verifying what has changed.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, document understanding, fine-grained textual change detection, layout robustness, multimodal benchmarks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gxFxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReflowDiff, a benchmark of 4,488 tasks evaluating whether vision-language models can exhaustively identify character-level edits across document layouts. Using controlled comparisons with identical edits under different layouts, the authors evaluate 30 configurations of 26 VLMs. Results show that layout changes consistently reduce exact-match accuracy, induce false change reports, and lead to incomplete or inconsistent predictions even in strong models. The findings highlight limitations in reliable document comparison beyond conventional document understanding tasks.

Critical strengths

The paper addresses a practically relevant and underexplored challenge in reliable document comparison.

The controlled paired comparisons effectively isolate the impact of reflow while preserving identical textual edits.

Extensive experiments across 26 VLMs reveal meaningful failure patterns beyond aggregate accuracy.

The analysis reveals that small aggregate performance drops can conceal substantial prediction instability across layouts, and that high per-edit recall does not necessarily translate into complete and accurate reports.

Critical weaknesses

Reflow simultaneously changes spatial arrangement, line wrapping, and character size, making it difficult to isolate the source of performance degradation.

The paper identifies several failure patterns but does not sufficiently disentangle perception, correspondence, and reporting errors.

Programmatically generated character-level edits may not fully reflect realistic document revisions.

While the paper identifies several failure patterns, the analysis remains largely empirical and outcome-driven, without sufficiently investigating the underlying mechanisms responsible for these failures. As a result, although the benchmark effectively exposes limitations of current VLMs, the findings provide limited actionable insights into how future models could be improved.

Questions for authors

Q1. Can the authors better isolate the effect of layout changes from changes in character size and visual resolution?

Q2. Can the authors provide an oracle-based analysis to distinguish perception failures from correspondence and reporting failures?

Q3. How sensitive are the conclusions to the evaluation protocol and response variability?

Q4. What are the potential underlying mechanisms responsible for these failures? Are they primarily caused by limitations in the vision encoder, insufficient vision-language interaction, or a lack of task-specific training? What do the authors consider to be the main contributing factors?

Reviewer_QXphOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ReflowDiff, a benchmark for evaluating the ability of vision-language models to identify fine-grained textual edits across document layouts. The authors construct comparison tasks from generated documents, real web pages, and financial reports by preserving textual content and structural context during layout reflow and applying identical character-level edits across different layouts. The benchmark requires models to report all textual changes and their locations, using exact match and false-report rates to assess performance. Experiments across proprietary and open-weight VLMs show that layout changes consistently degrade complete edit reporting, can induce false reports on unchanged text, and reveal a gap between individual edit recall and complete reporting accuracy.

Critical strengths

Well-controlled benchmark design. The paper preserves textual content and structural context during layout reflow, applies identical character-level edits across layouts, and includes no-edit controls. This design enables a more reliable assessment of how layout changes affect document comparison performance.
Comprehensive evaluation and valuable empirical findings. The paper evaluates a broad range of VLMs and analyzes cross-layout consistency, false reports, and complete reporting of multiple edits. These analyses reveal an important gap between individual edit recognition and reliable document-level comparison, providing useful insights for future research.

Critical weaknesses

Fine-grained visual difference detection, controlled perturbations, and no-change controls have been explored in prior research. The paper mainly combines these evaluation ideas in a cross-layout document comparison setting. While the benchmark is well designed, its conceptual novelty beyond existing evaluation paradigms remains limited.
The benchmark primarily relies on generated documents and synthetically injected single-character edits, including on real-source pages. Natural document revisions often involve more complex textual and structural changes, leaving the generalizability of the findings to practical document comparison scenarios insufficiently established.
Layout reflow simultaneously changes character size, line wrapping, and spatial arrangement. Although the paper analyzes character size effects, it does not sufficiently distinguish visual recognition difficulties from cross-layout correspondence failures, leaving the underlying causes of performance degradation unclear.

Questions for authors

Could the authors clarify what fundamentally new evaluation capabilities or insights the benchmark provides beyond combining existing fine-grained change detection and controlled perturbation settings?
Is there evidence that similar cross-layout performance degradation and false-report patterns also occur in naturally revised documents rather than synthetically injected character edits?
Could the authors further distinguish the effects of reduced character readability and cross-layout correspondence difficulties on performance degradation to better explain the main experimental findings?

Reviewer_aRQ4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReflowDiff, a benchmark for measuring whether vlms can identify textual revisions across layouts. The documents span various sources (e.g., education, engineering) and edits span multiple families (e.g., punctuation, numeric). Comprehensive evaluation across 30 configurations suggests most vlms cannot do so reliably.

Critical strengths
  1. The paper is well-written and the method is clear.
  2. The evaluations are comprehensive, spanning 30 configurations.
  3. Most models perform poorly on the dataset, so the benchmark will be hard to saturate. Moreover, the complexity can be arbitrarily increased by scaling the number of edits.
  4. The authors offer practical guidance for improving performance, such as increasing the reasoning effort or the image quality.
Critical weaknesses
  1. Figure 1 is confusing. I think the white space around the arrows cuts off the text making it difficult to track the edits. For instance, E3 highlights FD-13, FD-3, and D-31 respectively.
  2. It would be helpful to have a clearer description of the edit families. For instance, the glyphs and spelling families overlap quite a bit.
  3. Could the font-size differ after reflow? If so, smaller text could explain the gap between the old and new layout.
  4. Ocr would be a meaningful baseline to add.
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.