- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces DiffBench, a benchmark for evaluation of MLLMs on detecting visual differences in documents. The authors curate 578 document images and prompt 5 MLLM models to reconstruct the documents, resulting in pairs of original and reconstructed document pairs simulating natural reconstruction differences. Using these pairs, the authors evaluate multiple MLLMs on detecting the visual differences, across 4 evaluation dimensions, and show that some MLLMs struggle on this task.
Critical strengths
- Clear motivation: tackling more realistic and fine-grained document differences for vision-to-code related tasks has clear and simple motivation, and could be useful for capabilities assessment.
- The paper is generally clear and simple to follow.
- The evaluation dimensions introduced could be useful for more fine-grained and interpretable understanding of visual difference detection capabilities..
Critical weaknesses
-
The benchmark seems to have a major circularity problem: the authors create the ground truth using the same models that they later evaluate with (both the selection of GT samples, and the finding the differences between source and target and majority voting). Specifically, GPT-5.6 Sol, Gemini-3.8-Flash and Grok-4.6 do element discovery, difference discovery, and verification and all three are also evaluated in Table 1. This introduces a few critical potential issues:
(a) The difficulty of the benchmark is bounded by what the evaluated models can perceive. If the models used for detecting the differences in the benchmark pairs cannot perceive a difference which does exist, it will not be part of the benchmark, limiting its reliability and its usefulness and biasing and narrowing the scope.
(b) Furthermore, the ground truth descriptions are written with the annotator models phrasing and styling, which skews the results towards favoring these models. This gives the annotator models an advantage during the semantic matching. Note that the fact that some non-annotator models performed well on the benchmark (e.g., GPT-6 Astra) does not mean that there is no bias, only a controlled ablation can potentially measure it (see below).
These limitations severely undermine the reliability of the benchmark. I suggest that the authors conduct an experiment, where they exclude a model from participation in the annotation and then compare its performance in the benchmark when it’s excluded and when it’s not. Another option is to perform human annotations which are tasked with finding all the differences, and then measuring the portion of differences the annotator models missed.
-
The evaluation protocol is unclear and underspecified. A major part of the benchmark is the evaluation protocol. However, which models are used as the judges for evaluation, the prompts used, etc. are not specified. Are the evaluated models also being used as the judge?
-
No human evaluation baseline: It’s hard to interpret the scores without comparing it to how humans perform at this task. Human performance on this task can establish a reference point to judge performance with, but currently a performance of e.g., 64% has little interpretation.
-
While less critical, it would be nice if the authors could actually link the ability to spot differences to actual performance improvement on the paper’s main motivation task of vision-to-code. As of now, the link is only speculated but not tested.
Questions for authors
The font in some of the figures (e.g., Figure 3, Figure 4) is very small, can the authors increase the font?