- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies source misbinding in multi-image VQA: cases where a question explicitly identifies a target image, the model can answer correctly from that image alone, but adding other images causes the answer to become consistent with a distractor image. The paper argues that this is not primarily a failure to represent source identity; instead, source information remains decodable internally but fails to sufficiently constrain evidence use during generation.
Critical strengths
The problem is formulated clearly, which is backed by some mechanistic analysis. The experiments shows improvements from the method.
Critical weaknesses
- Lack of baseline methods and recent backbones: the experiment is based on Llava 1.5, which is outdated. Only DPO is compared, other recent multi-image papers are neglected (there are quite some other multi-image papers). The central mechanistic claims are derived almost entirely from LLaVA-1.5-7B.
- Method and evaluation design is unsound: every benchmark question explicitly names the target source, e.g. “Answer using Image 1 only.” The target image index is therefore already available symbolically in the prompt. Yet Source-First Inference introduces a learned 35K-parameter reader whose purpose is to infer which image the question specifies. This creates an artificial intermediate problem that does not need to exist in the benchmark as formulated. Benchmark scope is narrow relative to the broad framing.
Questions for authors
See weakness.