acceptodds
Withdrawn from ICLR 2027

Same Photo, More Evidence? Multi-Image VLMs Overcount Repeated Images

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision–language models increasingly answer questions about a set of images rather than one: the photographs on a product listing, the images attached to an incident report, the results of a web search. In such sets the same photograph routinely re-enters re-compressed, cropped or screenshotted. Existing multi-image benchmarks vary how many images a model sees and what they depict, but never whether an added image is a copy of one already present. The standard remedy, de-duplication by embedding similarity, does not settle this either: a re-upload of a photograph and a genuinely new photograph of the same object look alike to it, so any threshold either keeps the copies or throws away real evidence. We show that current multi-image VLMs count images rather than new observations. On a controlled spectrum that fixes image and token count, renderings of one photograph raise a model’s belief almost as much as genuinely new viewpoints; on real product listings, three re-uploads of one mis-attributed photo change up to a third of decisions, on open 3B–72B models and on API-served ones alike. Copies of a correct photograph inflate confidence as well, and stating in the prompt that copies count once does not help. Attention analysis places the effect not in how much the answer attends to the repeated images, which is flat across conditions, but in reinforcement among those images’ own tokens in the middle layers. The remedy is to remove the copies before the model sees the set. Provenance-Aware Evidence Aggregation (PAEN) scores each pair of images for provenance, asking whether they are renderings of one observation, by geometric verification: SIFT correspondences under a RANSAC homography, backed by a perceptual hash. This separates re-uploads from nearby viewpoints where semantic similarity cannot, even with a descriptor trained for copy detection. PAEN is training-free, returns four open architectures to within two points of their base accuracy with near-zero decision change while leaving genuinely new photographs untouched, and, applied before the images are sent, does the same for three models served only through an API (DeepSeek, GPT-5.6, Kimi). The signal multi-image aggregation needs is provenance, not similarity.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multi-image vision-language models, evidence aggregation, near-duplicate images, image provenance

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VahXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether multi-image VLMs treat different renderings of the same photograph as independent evidence. Through controlled experiments and product-listing tasks, the authors analyze this overcounting behavior and propose PAEN, a training-free preprocessing method that detects and removes provenance-based duplicates before inference.

Critical strengths

The authors evaluate multiple datasets, model architectures, model scales, and API-served models. They compare exact duplicates, re-compressed images, crops, screenshots, new viewpoints, and different instances. The experiments also control for image position and include neutral images, genuine new viewpoints, and shuffled-mask controls, providing a relatively comprehensive evaluation of how repeated images affect VLM inference.

Critical weaknesses
  1. The paper shows that multi-image VLMs may aggregate evidence according to the number of images rather than the number of independent observations, and it proposes PAEN as a remedy. However, it does not yet provide a unified account connecting independent observations, visual novelty, newly acquired evidence, and the reinforcement of repeated visual tokens. As a result, it is difficult to tell whether the paper is proposing a theory of evidence aggregation or reporting an empirical failure mode together with an engineering fix. The paper should define its central theoretical claim more explicitly and explain what it adds beyond existing work on repeated information and multi-image VLMs.

  2. The main finding and the proposed mechanism also lack sufficient theoretical and causal support. The paper gives an operational distinction between renderings, new viewpoints, and independent instances, but it does not explain formally why these categories should correspond to different amounts of new evidence, or why log-odds gain is an appropriate proxy for that evidence. In Table 3, the +cross-image condition changes both text-side attention and image-to-image attention, so the contribution of cross-image attention cannot be isolated. The nearly constant attention mass from the answer position to the images also does not imply that the information contributed by those images is unchanged. The explanation for the different behavior of LLaVA-OneVision is likewise largely speculative. A factorial attention ablation and a more direct causal analysis would be needed to support the current mechanistic claims.

  3. The assumption that different versions of the same observation are always redundant is also too strong. A crop may contain a task-relevant region that is absent from the original image, a recolored version may provide useful evidence for a color question, and a nearby viewpoint may reveal additional geometry or previously occluded content. The appendix already shows that PAEN sometimes merges partial crops, recolored variants, and nearby views, and that removing a crop can discard information unique to that crop. The paper should therefore state more clearly when removal is safe. At minimum, the method should be limited to high-confidence, provenance-preserving transformations that do not change task-relevant content, while partial crops, recolored variants, nearby views, and uncertain matches should be retained or handled with a softer rule.

4.The real-world validation and reproducibility are also limited. Although the paper uses real product images, the main conflicting images and re-uploaded versions are generated through fixed, manually designed transformation pipelines. The natural-duplicate analysis in the appendix is useful, but it is a diagnostic replacement experiment rather than a full end-to-end evaluation on unmodified product listings. The current results therefore establish that PAEN works in simulated re-upload settings, but do not yet show that it generalizes to naturally occurring image distributions or other downstream tasks.

Questions for authors

See the “Critical Weaknesses” mentioned above.

Reviewer_yMV1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how repetition affects multi-image VLMs, specifically whether VLMs treat multiple copies of the same image as new observations. The paper evaluates this across multiple open and closed-weight models, and finds that repeated photos can affect the decisions made by models and inflate their confidence. The authors analyze the attention of open-weights models and find that the overconfidence stems largely from intra-image attention. The authors then propose a training-free method to mitigate this, which scores pairs of images to determine which are duplicates or re-uploads.

Critical strengths
  • The problem studied could come up in real settings. The authors study it across open and closed-weights models.
  • The attention analysis adds some bits of information.
Critical weaknesses
  • Weak baselines and models. The intro motivates the problem by stating that ``embedding similarity is the wrong instrument'' (L080-081). This is a strawman argument: there could be many other cheap ways to detect similarity (indeed, the authors recruit a handful of these in PAEN). The authors do explore a prompting approach but it's only tested on open-weights models, not frontier models. It would also be trivial to add a 2-stage approach where the first prompt asks the model if any of the images are duplicates and the second prompt takes this into account. Most of the open-weights results are on an outdated generation of models.

  • It's not clear to me what the research contribution here is. The authors motivate the problem in settings where images are uploaded/compressed such that they end up in the context multiple times; however, there's no clear research question here besides ``how do current models handle this setting'', and then a pipeline approach to fixing it.

  • The writing is sloppy and very LLM-generated. Despite clear signs of heavy LLM writing use, the writing is also somehow very disorganized, e.g. PAEN is touted as a contribution in Fig1 and abstract but barely described in the introduction at all. This raises questions about how much the authors actually read the submission themselves.

Questions for authors

Video models can handle a large amount of repetition (most frames are very similar to adjacent frames). How do you square this with your results? Is this a product of the task or model, or both?

Reviewer_JnyqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether multi-image VLMs recognize that edited copies of the same photograph do not provide new observations. Its experiments show that repeated photos can increase confidence and change answers almost as much as genuinely new photos. The authors investigate how attention between images contributes to this effect and propose PAEN, which detects and removes copies before sending the images to the model. PAEN uses geometric image matching (SIFT/RANSAC) and image hashes, without training the VLM. Across open-weight models and models accessed through APIs, it largely restores the accuracy lost when misleading photos are repeated, while generally retaining the benefit of new photos.

Critical strengths
  • Novel problem and findings. The paper asks whether multi-image VLMs count edited copies of one photograph as additional observations. This distinction highlights an underexplored failure mode for applications that collect images from multiple sources.
  • Well-controlled experiments across models and answer formats. The study compares exact and edited copies, crops, new viewpoints, and different objects. It varies image order, uses gray or noise images as controls, and repeats both correct and misleading photos. The findings hold across several model families and sizes, product-type and color questions, and binary, four-choice, and open-ended answers.
Critical weaknesses
  1. Limited evidence of practical impact. The downstream evaluations use deliberately introduced conflicting images, including the experiment with naturally occurring duplicates (Appendix A.4). It remains unclear how frequently repetition causes errors in unmodified real-world image collections and how much PAEN would improve performance in these settings.
  2. Unclear scope of applicability. The paper demonstrates PAEN's effectiveness on product recognition tasks, but does not establish when provenance-based deduplication should be applied. Whether repeated photographs should be removed can depend on the downstream query: different renderings of the same photograph may provide complementary information useful to a VLM. Since PAEN operates independently of the query, it remains unclear how to determine when the method is beneficial versus potentially harmful in practical applications.
  3. Incomplete ablation of the proposed mechanism. Table 3 evaluates cross-image attention masking only in combination with text-side attention scaling. Although the results suggest that cross-image attention contributes substantially to evidence amplification, it remains unclear whether masking these connections alone is sufficient to eliminate the effect. A standalone cross-image masking ablation would better substantiate the proposed mechanism.
Questions for authors
  1. How frequently does duplicate-induced evidence amplification cause errors in unmodified real-world image collections? Can the authors evaluate whether PAEN improves performance without deliberately introducing conflicting images?
  2. Does cross-image attention masking alone, without text-side attention scaling, eliminate the amplification observed in Table 3?
  3. How can one determine whether PAEN should be applied to a given query? In particular, how does PAEN perform on tasks where different renderings of the same photograph (e.g., crops highlighting different details) may provide complementary information useful for answering the question?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.