- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies why irrelevant images hurt VLM performance. The authors distinguish between two failure modes: failing to identify the relevant image, and what they call "in-context corruption", where distractors still hurt performance even when the relevant image is correctly identified. They evaluate 3 VLMs on five perception benchmarks augmented with distractor images and show that correct target selection alone does not remove most of the degradation. They then perform attention analysis and targeted attention knockouts on Qwen2.5-VL and identify middle layer distractor-to-text connections as an important interference pathway. Finally, they propose answer consistency regularization, which encourages similar answer distributions with and without distractors. This improves robustness over the base model and also provides additional gains over distractor augmented cross entropyh fine-tuning.
Critical strengths
1. The paper identifies an interesting failure mode beyond incorrect image selection. The experiments show that even explicitly telling the model which image is relevant recovers only a small part of the performance loss, while removing the distractors restores much more performance. I think this is a useful distinction and an important result for understanding multi image VLM robustness.
2. The mechanistic analysis is relatively strong. I liked that the authors use causal attention knockouts rather than relying only on attention visualization. Blocking distractor-to-text attention in the middle layers recovers 47% of broken examples while damaging only about 5% of previously correct examples, which gives fairly strong evidence that this pathway contributes to the observed interference.
Critical weaknesses
1. The gain from the proposed consistency objective is much smaller than the headline improvement over the base model. At 5 distractors, the base model achieves 51.4 circular accuracy and crossentropy+KL reaches 67.4, giving the reported 16 point improvement. However, the matched crosserntropy-only control already reaches 63.6, so the incremental gain attributable to the KL objective is only about 3.8 points. Similarly, the pooled distracted-condition gain over CE-only is about 3 points. I think the paper should more clearly separate the benefit of distractor-augmented finetuning from the benefit of the proposed consistency loss.
2. The mechanistic claims are only demonstrated on one model. The behavioral analysis includes three VLMs, but the attention analysis, causal knockouts, and mitigation experiments are all performed on Qwen2.5-VL. Since the models show quite different behavior under distraction and image selection, I am not sure how general the identified distractor-to-text pathway is.
Questions for authors
Mitigation attribution: How should the improvement from the KL consistency objective be interpreted relative to the crossentropy-only distractor-augmented baseline? Since most of the improvement over the base model already appears with crossentropy-only training, I think it would help to clarify what part of the robustness gain the authors attribute specifically to the proposed objective.
Generalization of the mechanism: Do the authors have evidence that the middle-layer distractor-to-text pathway also appears in InternVL3 or LLaVA-OneVision? Even a smaller knockout experiment on a second model would materially strengthen the claim that this is a general VLM failure mechanism rather than something specific to Qwen2.5-VL.
Training robustness: Were the crossentropy-only and crossentroorpy+KL finetuning experiments repeated across multiple random seeds? Since the incremental gain of the KL objective over crossentrorpy-only is relatively modest, knowing the variation across training runs would affect how strongly I interpret this result. The paper states that the arms share the same data order and LoRA configuration, but I did not find statistics (multi seed) results.
5 comments
Generalization of the mechanism. We repeated the knockout analysis on InternVL3-8B across all five benchmarks (Tables 22-23). Blocking distractor-to-text attention in layers 9-17 recovers 43% of broken examples while damaging 6% of preserved ones, close to the 47% and 5% observed for Qwen2.5-VL. Recovery is positive on every benchmark (26-62%). The middle-layer concentration also replicates: the same mask recovers only 14% in layers 0-8 and 6% in layers 18–27. Inter-image masking again recovers far less (10%). The pathway and its layer profile therefore hold in a second architecture rather than being specific to Qwen2.5-VL.
Prompted by other reviewers, we also added three controls on Qwen2.5-VL (table below; Table 3 and Tables 20-21). Restricting the mask to question tokens (Q) retains nearly all of the recovery (46% vs. 47%), indicating that the question tokens carry most of the effect. Preserving system-prefix access in the inter-image comparator (M) lowers its damage from 57% to 8%, but its recovery stays at 12%, so the advantage of distractor-to-text masking does not depend on the original comparator's removal of the system prefix. Finally, a random image-to-text mask that removes the same number of connections (Y) as distractor-to-text masking (D) but draws them at random from all images recovers 12-14% at one and two distractors, compared with 47-49% for distractor-specific masking. (Tables 20 and 21)
Attention knockouts in layers 9-17 at five distractors, both architectures. Recovery is the fraction of broken examples repaired; damage is the fraction of preserved examples made incorrect. Q was run on Qwen2.5-VL only.
| Cond. | Intervention | Qwen2.5-VL Recovery | Qwen2.5-VL Damage | InternVL3-8B Recovery | InternVL3-8B Damage |
|---|---|---|---|---|---|
| P | everything | 0.56 | 0.05 | 0.47 | 0.09 |
| MM | all preceding IMG | 0.10 | 0.57 | 0.18 | 0.49 |
| M | prior IMG IMG | 0.12 | 0.08 | 0.10 | 0.10 |
| D | all text | 0.47 | 0.05 | 0.43 | 0.06 |
| Q | question (stem options) | 0.46 | 0.05 | -- | -- |
| R | readout | 0.03 | 0.02 | 0.02 | 0.01 |
We hope these additions resolve the reviewer's questions, and we are happy to provide further detail on any of them.
Follow-up on our response before the discussion deadline
Dear Reviewer Tpya, thank you again for your thoughtful review. As the discussion period ends tomorrow (November 18), we wanted to briefly check in on our response above. In short:
- Mitigation attribution: over three matched training seeds, CE fine-tuning accounts for 12.1 of the 14.6-point gain at five distractors, and the KL objective for the remaining 2.5, positive in every seed and on every benchmark. We now present CE fine-tuning as the main mitigation and KL as an exploratory extension, and we removed the 16-point headline claim.
- Generalization: the knockout result replicates on InternVL3-8B (43% recovery, 6% damage), with the same middle-layer concentration. We also added a random control that blocks the same number of image-to-text connections, chosen at random across all images: it recovers only 12–14% of broken examples, compared with 47–49% when blocking distractor connections specifically.
We hope these changes address your questions, and we are happy to clarify anything further before the deadline.
We thank the reviewer for highlighting the distinction between selection failure and in-context corruption, and for the positive assessment of the knockout analysis. We have reframed the mitigation’s contribution, repeated training across three matched seeds, and extended the knockout experiments to InternVL3-8B. All references below refer to the revised manuscript we have uploaded.
Mitigation attribution and Training robustness. We agree with the reviewer and have reframed the mitigation accordingly. The revised abstract, introduction, and Section 6 present distractor-augmented supervised fine-tuning with standard cross-entropy (CE) as the main mitigation, and answer consistency (KL) as an exploratory extension with smaller, benchmark-dependent additional gains. The 16-point headline no longer appears. Table 4 now reports three matched training seeds (summarized below; per-seed values in Table 26). With three seeds, CE + KL improves five-distractor circular accuracy over the base model by 14.6 points: 12.1 points (about 83%) are already obtained with CE alone, and 2.5 points (about 17%) are attributable to the consistency objective.
CE vs. CE + KL across training seeds. Qwen2.5-VL-7B with LoRA arms, pooled circular accuracy macro-averaged over the five benchmarks. Trained arms report mean ± s.d. over three training seeds; the base model is deterministic and has no seed variation. Paired gain from KL is the per-seed difference (CE + KL − CE), averaged over seeds, with the standard deviation of those paired differences.
The CE gain over the base model (12.1 points at k=5) far exceeds its seed variation (s.d. 0.87). The additional KL gain (+2.47±1.09 at k=5) is positive in all three seeds and on all five benchmarks, grows with the number of distractors (from +0.42 at k=0 to +2.47 at k=5; Table 24), and lowers seed variance on four of five benchmarks. The gain is largest on CLEVR-size and What'sUp (+6.57 and +2.76 points at k=5). With three seeds, the interval on the pooled gain remains wide, which is why we present answer consistency as an exploratory extension. The weight λ=1 is selected in all five partitions of a 20/80 split-stability analysis (held-out gain +1.73 vs. +1.72 on the full set).
For the final version, we will also report the prior-method baselines (DTS, CE + inter-image mask) over multiple seeds. We are also extending the fine-tuning experiments to another model; we will report the results during the discussion period if they are completed in time, or in the final version otherwise.