- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether multi-image VLMs treat different renderings of the same photograph as independent evidence. Through controlled experiments and product-listing tasks, the authors analyze this overcounting behavior and propose PAEN, a training-free preprocessing method that detects and removes provenance-based duplicates before inference.
Critical strengths
The authors evaluate multiple datasets, model architectures, model scales, and API-served models. They compare exact duplicates, re-compressed images, crops, screenshots, new viewpoints, and different instances. The experiments also control for image position and include neutral images, genuine new viewpoints, and shuffled-mask controls, providing a relatively comprehensive evaluation of how repeated images affect VLM inference.
Critical weaknesses
-
The paper shows that multi-image VLMs may aggregate evidence according to the number of images rather than the number of independent observations, and it proposes PAEN as a remedy. However, it does not yet provide a unified account connecting independent observations, visual novelty, newly acquired evidence, and the reinforcement of repeated visual tokens. As a result, it is difficult to tell whether the paper is proposing a theory of evidence aggregation or reporting an empirical failure mode together with an engineering fix. The paper should define its central theoretical claim more explicitly and explain what it adds beyond existing work on repeated information and multi-image VLMs.
-
The main finding and the proposed mechanism also lack sufficient theoretical and causal support. The paper gives an operational distinction between renderings, new viewpoints, and independent instances, but it does not explain formally why these categories should correspond to different amounts of new evidence, or why log-odds gain is an appropriate proxy for that evidence. In Table 3, the +cross-image condition changes both text-side attention and image-to-image attention, so the contribution of cross-image attention cannot be isolated. The nearly constant attention mass from the answer position to the images also does not imply that the information contributed by those images is unchanged. The explanation for the different behavior of LLaVA-OneVision is likewise largely speculative. A factorial attention ablation and a more direct causal analysis would be needed to support the current mechanistic claims.
-
The assumption that different versions of the same observation are always redundant is also too strong. A crop may contain a task-relevant region that is absent from the original image, a recolored version may provide useful evidence for a color question, and a nearby viewpoint may reveal additional geometry or previously occluded content. The appendix already shows that PAEN sometimes merges partial crops, recolored variants, and nearby views, and that removing a crop can discard information unique to that crop. The paper should therefore state more clearly when removal is safe. At minimum, the method should be limited to high-confidence, provenance-preserving transformations that do not change task-relevant content, while partial crops, recolored variants, nearby views, and uncertain matches should be retained or handled with a softer rule.
4.The real-world validation and reproducibility are also limited. Although the paper uses real product images, the main conflicting images and re-uploaded versions are generated through fixed, manually designed transformation pipelines. The natural-duplicate analysis in the appendix is useful, but it is a diagnostic replacement experiment rather than a full end-to-end evaluation on unmodified product listings. The current results therefore establish that PAEN works in simulated re-upload settings, but do not yet show that it generalizes to naturally occurring image distributions or other downstream tasks.
Questions for authors
See the “Critical Weaknesses” mentioned above.