acceptodds
Withdrawn from ICLR 2027

DiffBench: How Well Do MLLMs Understand Visual Differences?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-to-code tasks aim to reconstruct visual inputs into editable and executable artifacts with high visual fidelity. During reconstruction, visual difference detection is a key tool for locating reconstruction errors and guiding subsequent refinement. It makes evaluation of visual difference detection increasingly important. Existing difference detection benchmarks largely focus on synthetic artifacts or object-centric natural images, leaving complex documents in real-world underexplored. To address this gap, we introduce DiffBench, a high-quality benchmark of visual differences detection, featuring non-synthetic discrepancies in diverse real-world document types such as slides, magazines and reports. Besides, we propose an automated evaluation framework for assessing visual difference predictions across four dimensions including layout, relation, content integrity and visual properties. Our evaluation reveals that GPT-6 Astra achieves the strongest overall performance and is better than GPT-5.6 Sol, while Kimi K3 leads among open-weight models including its previous version Kimi K2.5. Across models, content integrity discrepancies like missing text and images are generally easier to detect, whereas precise alignment, visual properties, and multi-region layout changes remain substantially more challenging. Performance further drops on highly similar cases with subtle differences, where models increasingly miss true discrepancies. Overall, DiffBench remains challenging for current MLLMs while effectively capturing progress across successive model generations.

Primary area:
datasets and benchmarks
Keywords:
Visual Difference Detection, Multimodal Large Language Models, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_G5QaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces DiffBench, a benchmark for evaluation of MLLMs on detecting visual differences in documents. The authors curate 578 document images and prompt 5 MLLM models to reconstruct the documents, resulting in pairs of original and reconstructed document pairs simulating natural reconstruction differences. Using these pairs, the authors evaluate multiple MLLMs on detecting the visual differences, across 4 evaluation dimensions, and show that some MLLMs struggle on this task.

Critical strengths
  1. Clear motivation: tackling more realistic and fine-grained document differences for vision-to-code related tasks has clear and simple motivation, and could be useful for capabilities assessment.
  2. The paper is generally clear and simple to follow.
  3. The evaluation dimensions introduced could be useful for more fine-grained and interpretable understanding of visual difference detection capabilities..
Critical weaknesses
  1. The benchmark seems to have a major circularity problem: the authors create the ground truth using the same models that they later evaluate with (both the selection of GT samples, and the finding the differences between source and target and majority voting). Specifically, GPT-5.6 Sol, Gemini-3.8-Flash and Grok-4.6 do element discovery, difference discovery, and verification and all three are also evaluated in Table 1. This introduces a few critical potential issues:

    (a) The difficulty of the benchmark is bounded by what the evaluated models can perceive. If the models used for detecting the differences in the benchmark pairs cannot perceive a difference which does exist, it will not be part of the benchmark, limiting its reliability and its usefulness and biasing and narrowing the scope.

    (b) Furthermore, the ground truth descriptions are written with the annotator models phrasing and styling, which skews the results towards favoring these models. This gives the annotator models an advantage during the semantic matching. Note that the fact that some non-annotator models performed well on the benchmark (e.g., GPT-6 Astra) does not mean that there is no bias, only a controlled ablation can potentially measure it (see below).

    These limitations severely undermine the reliability of the benchmark. I suggest that the authors conduct an experiment, where they exclude a model from participation in the annotation and then compare its performance in the benchmark when it’s excluded and when it’s not. Another option is to perform human annotations which are tasked with finding all the differences, and then measuring the portion of differences the annotator models missed.

  2. The evaluation protocol is unclear and underspecified. A major part of the benchmark is the evaluation protocol. However, which models are used as the judges for evaluation, the prompts used, etc. are not specified. Are the evaluated models also being used as the judge?

  3. No human evaluation baseline: It’s hard to interpret the scores without comparing it to how humans perform at this task. Human performance on this task can establish a reference point to judge performance with, but currently a performance of e.g., 64% has little interpretation.

  4. While less critical, it would be nice if the authors could actually link the ability to spot differences to actual performance improvement on the paper’s main motivation task of vision-to-code. As of now, the link is only speculated but not tested.

Questions for authors

The font in some of the figures (e.g., Figure 3, Figure 4) is very small, can the authors increase the font?

Reviewer_pzetOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DiffBench evaluates the ability of multimodal models to describe visual differences between "actual document pages" and "model-reconstructed pages." It comprises 578 image pairs, accompanied by 12,267 annotations detailing these differences. The annotation process incorporates multiple stages to ensure robustness, and the paper assesses several state-of-the-art models to demonstrate the effectiveness of the benchmark.

Critical strengths
  1. The evaluation scenarios possess practical value, providing a benchmark for studying discrepancies in text-heavy contexts and helping to uncover error patterns related to typography and layout.
  2. The evaluation dimensions are decoupled—assessing layout, relations, content integrity, and visual properties separately—which yields a diverse and dense set of evaluation signals, offering far more information than a single, unified metric.
Critical weaknesses
  1. The evaluation setup and results need clarification. It is unclear how DeepSeek-V4-Pro processed images, and the model settings and image preprocessing are not fully described. In Figure 3, GPT-6 Astra has an overall Strict Accuracy of 24.70%, but the subgroup scores are 17.95%, 17.50%, and 14.12%. If these groups make up the full evaluation set, their weighted average cannot exceed 17.95%. These issues make the model comparison difficult to assess.

  2. The scoring method does not clearly explain how false positives are handled. Equations (1) and (2) measure coverage of gold differences, but are incorrect predictions that match no gold difference penalized? Without this penalty, a model may get a high score while reporting changes that do not exist, making the ranking less reliable.

Questions for authors
  1. How were images processed when evaluating DeepSeek-V4-Pro? Please clarify the model version and whether an additional vision component was used.

  2. In Figure 3, why is the overall Global Composition accuracy higher than every subgroup’s accuracy? Please explain how these results were calculated.

  3. Do Equations (1) and (2) penalize incorrect predictions that do not match any gold difference? It is unclear how false positives affect the score.

  4. Did human annotators check for differences missed by the model-generated candidates? The judge model, prompts, and inference settings also need clarification.

Reviewer_mPedOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes DiffBench, a benchmark for fine-grained visual difference detection between a reference document image and an MLLM-reconstructed candidate. It is motivated by verification in vision-to-code. The benchmark contains 578 pairs rendered from PDFs across five domains. Golden differences (12,267) are built by an agentic, region-based MLLM pipeline with self-verification and multi-model majority voting. Models are evaluated with an MLLM-as-a-judge protocol across four dimensions (Layout, Relation, Content Integrity, Visual Properties). Results show that models handle content discrepancies better than alignment, visual-property, and multi-region layout differences, and that they degrade on subtle hard cases.

Critical strengths
  • Difference detection is a key verification step in agentic vision-to-code.
  • Discrepancies arise naturally from real reconstruction errors rather than synthetic perturbations.
  • The hierarchical, region-based annotation addresses cascaded differences.
Critical weaknesses
  1. Mismatch between annotation and evaluation settings. Ground truth is produced by an agentic pipeline: region division, dual comparison views, self-verification, a multi-model ensemble, and majority voting. The evaluated models appear to be tested without such scaffolding. The same models used as annotators (GPT-5.6-Sol, Gemini-3.8-Flash, Grok-4.6) score only 19–52%, even though 97.2% of pipeline candidates were retained. A construction agent would therefore likely score near the ceiling. The measured gap may largely reflect the evaluation setup rather than intrinsic model capability, which weakens the main conclusion.
  2. Heavy reliance on MLLMs and potential circularity. Candidate generation, annotation, verification, dimension tagging, and scoring are all performed by MLLMs, and the annotator models overlap with the evaluated models. Human validation is under-specified. The paper does not state who audited the data, how many annotators were involved, or the inter-annotator agreement. It does not validate the completeness (recall) of the golden set or the accuracy of the text-only dimension tagging.
  3. Limited scale and diversity. There are only 578 pairs, and the subsets are too small (46 hard cases; Layout accounts for only 5.5% of differences). All references come from PDF pages, with about half from editorial publications. Variation in document type and size is also limited, and web and mobile UIs are not covered. The error distribution is tied to the five reconstruction models. No variance or confidence intervals are reported.
  4. Metric design. The scores are recall-only, so false positives are not penalized. Criticality is annotated but not used in scoring. "Overall" is undefined; it appears to be a micro-average dominated by Relation and Visual Properties. Because tagging is multi-label, the dimension scores are not independent, and a mis-tagged prediction can fail to match.
  5. Ambiguous taxonomy. Sub-dimensions are defined only by keyword lists in the appendix, and the main text does not restate them. Boundaries overlap: Global Composition includes alignment, which overlaps with Relation/Alignment, and Layout Skeleton vs. Global Composition is unclear. As a result, the sub-dimension findings in Sec. 5.2 may reflect tagging noise.
  6. Descriptive rather than explanatory analysis. The paper does not analyze why Content Integrity is easier than Visual Properties or Alignment. Plausible factors include OCR/text-salience bias, input resolution, cross-image spatial correspondence, and description granularity. Ablations are needed, such as resolution, crop/region inputs, and providing the annotation skill to evaluated models.
  7. Weak supporting evidence. Within-family improvements are expected and do not by themselves demonstrate benchmark validity. Nor is it shown that better detection improves vision-to-code reconstruction.
  8. Presentation. Writing needs proofreading (e.g., sentences beginning with "And", "Specifically, The").
Questions for authors

please refer to the Weaknesses Section and address all of them

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.