- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ReflowDiff, a benchmark of 4,488 tasks evaluating whether vision-language models can exhaustively identify character-level edits across document layouts. Using controlled comparisons with identical edits under different layouts, the authors evaluate 30 configurations of 26 VLMs. Results show that layout changes consistently reduce exact-match accuracy, induce false change reports, and lead to incomplete or inconsistent predictions even in strong models. The findings highlight limitations in reliable document comparison beyond conventional document understanding tasks.
Critical strengths
The paper addresses a practically relevant and underexplored challenge in reliable document comparison.
The controlled paired comparisons effectively isolate the impact of reflow while preserving identical textual edits.
Extensive experiments across 26 VLMs reveal meaningful failure patterns beyond aggregate accuracy.
The analysis reveals that small aggregate performance drops can conceal substantial prediction instability across layouts, and that high per-edit recall does not necessarily translate into complete and accurate reports.
Critical weaknesses
Reflow simultaneously changes spatial arrangement, line wrapping, and character size, making it difficult to isolate the source of performance degradation.
The paper identifies several failure patterns but does not sufficiently disentangle perception, correspondence, and reporting errors.
Programmatically generated character-level edits may not fully reflect realistic document revisions.
While the paper identifies several failure patterns, the analysis remains largely empirical and outcome-driven, without sufficiently investigating the underlying mechanisms responsible for these failures. As a result, although the benchmark effectively exposes limitations of current VLMs, the findings provide limited actionable insights into how future models could be improved.
Questions for authors
Q1. Can the authors better isolate the effect of layout changes from changes in character size and visual resolution?
Q2. Can the authors provide an oracle-based analysis to distinguish perception failures from correspondence and reporting failures?
Q3. How sensitive are the conclusions to the evaluation protocol and response variability?
Q4. What are the potential underlying mechanisms responsible for these failures? Are they primarily caused by limitations in the vision encoder, insufficient vision-language interaction, or a lack of task-specific training? What do the authors consider to be the main contributing factors?