acceptodds
Under review as a conference paper at ICLR 2027

Back to the Question: Tracing Visual Distraction Through Text Tokens

Abstract

The visual world is cluttered: robust vision-language reasoning requires ignoring irrelevant information. Yet vision-language models (VLMs) can suffer substantial accuracy losses when answering questions about a target image with distractors present. Existing mitigations focus on selecting relevant images or limiting information exchange between images. Through a systematic analysis of three open 7–8B VLMs on five perception benchmarks adapted to include distractor images, we show that distractors reduce accuracy even when models correctly identify the relevant image, suggesting that selection alone is not enough. To understand the mechanism behind this interference, we combine attention analysis with targeted knockouts in Qwen2.5-VL and identify middle-layer distractor-to-text connections as a pathway through which irrelevant visual information affects reasoning. We introduce answer consistency regularization, which encourages consistent answer distributions with and without distractors. Without directly constraining attention, this objective reduces middle-layer attention from question tokens to distractor images, consistent with the pathway our analysis implicates. With five distractors, our approach improves circular accuracy by an average of 16 percentage points over the Qwen2.5-VL baseline across the five benchmarks while preserving single-image performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, visual distraction, multi-image reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TpyaOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why irrelevant images hurt VLM performance. The authors distinguish between two failure modes: failing to identify the relevant image, and what they call "in-context corruption", where distractors still hurt performance even when the relevant image is correctly identified. They evaluate 3 VLMs on five perception benchmarks augmented with distractor images and show that correct target selection alone does not remove most of the degradation. They then perform attention analysis and targeted attention knockouts on Qwen2.5-VL and identify middle layer distractor-to-text connections as an important interference pathway. Finally, they propose answer consistency regularization, which encourages similar answer distributions with and without distractors. This improves robustness over the base model and also provides additional gains over distractor augmented cross entropyh fine-tuning.

Critical strengths

1. The paper identifies an interesting failure mode beyond incorrect image selection. The experiments show that even explicitly telling the model which image is relevant recovers only a small part of the performance loss, while removing the distractors restores much more performance. I think this is a useful distinction and an important result for understanding multi image VLM robustness.

2. The mechanistic analysis is relatively strong. I liked that the authors use causal attention knockouts rather than relying only on attention visualization. Blocking distractor-to-text attention in the middle layers recovers 47% of broken examples while damaging only about 5% of previously correct examples, which gives fairly strong evidence that this pathway contributes to the observed interference.

Critical weaknesses

1. The gain from the proposed consistency objective is much smaller than the headline improvement over the base model. At 5 distractors, the base model achieves 51.4 circular accuracy and crossentropy+KL reaches 67.4, giving the reported 16 point improvement. However, the matched crosserntropy-only control already reaches 63.6, so the incremental gain attributable to the KL objective is only about 3.8 points. Similarly, the pooled distracted-condition gain over CE-only is about 3 points. I think the paper should more clearly separate the benefit of distractor-augmented finetuning from the benefit of the proposed consistency loss.

2. The mechanistic claims are only demonstrated on one model. The behavioral analysis includes three VLMs, but the attention analysis, causal knockouts, and mitigation experiments are all performed on Qwen2.5-VL. Since the models show quite different behavior under distraction and image selection, I am not sure how general the identified distractor-to-text pathway is.

Questions for authors

Mitigation attribution: How should the improvement from the KL consistency objective be interpreted relative to the crossentropy-only distractor-augmented baseline? Since most of the improvement over the base model already appears with crossentropy-only training, I think it would help to clarify what part of the robustness gain the authors attribute specifically to the proposed objective.

Generalization of the mechanism: Do the authors have evidence that the middle-layer distractor-to-text pathway also appears in InternVL3 or LLaVA-OneVision? Even a smaller knockout experiment on a second model would materially strengthen the claim that this is a general VLM failure mechanism rather than something specific to Qwen2.5-VL.

Training robustness: Were the crossentropy-only and crossentroorpy+KL finetuning experiments repeated across multiple random seeds? Since the incremental gain of the KL objective over crossentrorpy-only is relatively modest, knowing the variation across training runs would affect how strongly I interpret this result. The paper states that the arms share the same data order and LoRA configuration, but I did not find statistics (multi seed) results.

5 comments

We thank the reviewer for highlighting the distinction between selection failure and in-context corruption, and for the positive assessment of the knockout analysis. We have reframed the mitigation’s contribution, repeated training across three matched seeds, and extended the knockout experiments to InternVL3-8B. All references below refer to the revised manuscript we have uploaded.

Mitigation attribution and Training robustness. We agree with the reviewer and have reframed the mitigation accordingly. The revised abstract, introduction, and Section 6 present distractor-augmented supervised fine-tuning with standard cross-entropy (CE) as the main mitigation, and answer consistency (KL) as an exploratory extension with smaller, benchmark-dependent additional gains. The 16-point headline no longer appears. Table 4 now reports three matched training seeds (summarized below; per-seed values in Table 26). With three seeds, CE + KL improves five-distractor circular accuracy over the base model by 14.6 points: 12.1 points (about 83%) are already obtained with CE alone, and 2.5 points (about 17%) are attributable to the consistency objective.

CE vs. CE + KL across training seeds. Qwen2.5-VL-7B with LoRA arms, pooled circular accuracy macro-averaged over the five benchmarks. Trained arms report mean ±\pm s.d. over three training seeds; the base model is deterministic and has no seed variation. Paired gain from KL is the per-seed difference (CE + KL −- CE), averaged over seeds, with the standard deviation of those paired differences.

MethodMean over k>0k{>}0k=5k{=}5
Qwen2.5-VL-7B (base)59.3351.37
CE only67.09±1.2067.09 \pm 1.2063.49±0.8763.49 \pm 0.87
CE + KL, λ=1\lambda{=}168.81±0.91\mathbf{68.81 \pm 0.91}65.96±1.15\mathbf{65.96 \pm 1.15}
Paired gain from KL+1.72±1.17+1.72 \pm 1.17+2.47±1.09+2.47 \pm 1.09

The CE gain over the base model (12.1 points at k=5k{=}5) far exceeds its seed variation (s.d. 0.87). The additional KL gain (+2.47±1.09+2.47 \pm 1.09 at k=5k{=}5) is positive in all three seeds and on all five benchmarks, grows with the number of distractors (from +0.42+0.42 at k=0k{=}0 to +2.47+2.47 at k=5k=5; Table 24), and lowers seed variance on four of five benchmarks. The gain is largest on CLEVR-size and What'sUp (+6.57+6.57 and +2.76+2.76 points at k=5k{=}5). With three seeds, the interval on the pooled gain remains wide, which is why we present answer consistency as an exploratory extension. The weight λ=1\lambda{=}1 is selected in all five partitions of a 20/80 split-stability analysis (held-out gain +1.73+1.73 vs. +1.72+1.72 on the full set).

For the final version, we will also report the prior-method baselines (DTS, CE + inter-image mask) over multiple seeds. We are also extending the fine-tuning experiments to another model; we will report the results during the discussion period if they are completed in time, or in the final version otherwise.

Generalization of the mechanism. We repeated the knockout analysis on InternVL3-8B across all five benchmarks (Tables 22-23). Blocking distractor-to-text attention in layers 9-17 recovers 43% of broken examples while damaging 6% of preserved ones, close to the 47% and 5% observed for Qwen2.5-VL. Recovery is positive on every benchmark (26-62%). The middle-layer concentration also replicates: the same mask recovers only 14% in layers 0-8 and 6% in layers 18–27. Inter-image masking again recovers far less (10%). The pathway and its layer profile therefore hold in a second architecture rather than being specific to Qwen2.5-VL.

Prompted by other reviewers, we also added three controls on Qwen2.5-VL (table below; Table 3 and Tables 20-21). Restricting the mask to question tokens (Q) retains nearly all of the recovery (46% vs. 47%), indicating that the question tokens carry most of the effect. Preserving system-prefix access in the inter-image comparator (M) lowers its damage from 57% to 8%, but its recovery stays at 12%, so the advantage of distractor-to-text masking does not depend on the original comparator's removal of the system prefix. Finally, a random image-to-text mask that removes the same number of connections (Y) as distractor-to-text masking (D) but draws them at random from all images recovers 12-14% at one and two distractors, compared with 47-49% for distractor-specific masking. (Tables 20 and 21)

Attention knockouts in layers 9-17 at five distractors, both architectures. Recovery is the fraction of broken examples repaired; damage is the fraction of preserved examples made incorrect. Q was run on Qwen2.5-VL only.

Cond.InterventionQwen2.5-VL RecoveryQwen2.5-VL DamageInternVL3-8B RecoveryInternVL3-8B Damage
PIMGD\text{IMG}_D →\rightarrow everything0.560.050.470.09
MMall preceding →\rightarrow IMG0.100.570.180.49
Mprior IMG →\rightarrow IMG0.120.080.100.10
DIMGD\text{IMG}_D →\rightarrow all text0.470.050.430.06
QIMGD\text{IMG}_D →\rightarrow question (stem ∪\cup options)0.460.05----
RIMGD\text{IMG}_D →\rightarrow readout0.030.020.020.01

We hope these additions resolve the reviewer's questions, and we are happy to provide further detail on any of them.

AuthorsOpenReview ↗

Follow-up on our response before the discussion deadline

Dear Reviewer Tpya, thank you again for your thoughtful review. As the discussion period ends tomorrow (November 18), we wanted to briefly check in on our response above. In short:

  • Mitigation attribution: over three matched training seeds, CE fine-tuning accounts for 12.1 of the 14.6-point gain at five distractors, and the KL objective for the remaining 2.5, positive in every seed and on every benchmark. We now present CE fine-tuning as the main mitigation and KL as an exploratory extension, and we removed the 16-point headline claim.
  • Generalization: the knockout result replicates on InternVL3-8B (43% recovery, 6% damage), with the same middle-layer concentration. We also added a random control that blocks the same number of image-to-text connections, chosen at random across all images: it recovers only 12–14% of broken examples, compared with 47–49% when blocking distractor connections specifically.

We hope these changes address your questions, and we are happy to clarify anything further before the deadline.

Reviewer_sEKYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why irrelevant images impair visual question answering even when the target image is correctly identified. Attention knockouts in Qwen2.5-VL implicate middle-layer distractor-to-text connections. The authors then introduce answer-consistency regularization, aligning predictions with and without distractors. Across five benchmarks, the method improves five-distractor circular accuracy from 51.4% to 67.4%, compared with 63.6% for matched CE-only training.

Critical strengths
  1. Meaningful behavioral controls. Target naming, two-stage selection, physical distractor removal, and black-image controls help distinguish selection errors, content interference, and sequence-length effects.

  2. The paper measures both recovery and damage under attention knockouts. CE-only training also provides an appropriate comparison for assessing the added consistency objective.

Critical weaknesses
  1. Generalization and incremental benefit remain limited. Behavioral evaluation covers three models, but the mechanism and mitigation are tested only on Qwen2.5-VL. It is unclear whether the findings extend to newer multimodal architectures. The KL objective adds 3.8 points over matched CE training at five distractors, rather than the headline 16-point gain over the original model. Training-seed uncertainty is not reported, and clean What’sUp accuracy decreases from 92.2% to 88.4%.

  2. The experimental interpretation needs tighter controls and clearer settings. The main knockout blocks all post-image text, not only question tokens, while the cross-image comparator also removes system-prefix access. This limits precise pathway attribution. Distractor validity partly depends on VLM judgments without a reported human-audited error rate. MIMIC filtering descriptions differ across Tables 5 and 7 and the surrounding text; the stated 1,280-token image budget also needs reconciliation with the reported 28K sequence estimate.

Questions for authors
  1. Does the distraction effect persist in a recent, similarly sized model, and does the proposed mitigation still help?

  2. Which model performs distractor screening, and what is its human-audited false-acceptance rate? Please reconcile the MIMIC filtering descriptions.

  3. Please explain how the per-image token budget produces the reported sequence lengths, distinguishing pre-merge and decoder tokens if applicable.

7 comments

We thank the reviewer for the detailed review and for recognizing the behavioral controls and the recovery/damage analysis. We replicated the knockouts on InternVL3-8B, evaluated a newer model, revised the fine-tuning contribution, added multiple training seeds, added pathway controls, and documented distractor screening with a human audit. All references below refer to the revised manuscript we have uploaded.

Generalization to other architectures. We repeated the knockout analysis on InternVL3-8B across all five benchmarks (Tables 22-23). Blocking distractor-to-text attention in layers 9–17 recovers 43% of broken examples while damaging 6% of preserved ones, close to the 47% and 5% observed for Qwen2.5-VL. The middle-layer concentration also replicates: the same mask recovers only 14% in layers 0-8 and 6% in layers 18–27.

To test whether the phenomenon persists in a more recent model, we evaluated Qwen3.5-9B on the same distractor-augmented benchmarks (Appendix A.4.1, Tables 13-14). From k=0k{=}0 to k=5k{=}5, mean-rotation accuracy falls by 4.0 points on average (2.3-6.6 per benchmark), and circular accuracy falls by 4.4-7.9 points on four of the five benchmarks; on BLINK, circular accuracy instead rises by 4.7 points. The degradation, although smaller than for Qwen2.5-VL thus persists for Qwen3.5-9B. We are extending the fine-tuning experiments to another model to test whether the mitigation transfers, and will report the results during the discussion period if they complete in time, or in the final version otherwise.

Incremental benefit of the consistency objective. We agree, and have reframed the mitigation throughout the revised manuscript. The revised abstract, introduction, and Section 6 present distractor-augmented supervised fine-tuning with cross-entropy (CE) as the main mitigation, and answer consistency (KL) as an extension with additional, smaller, benchmark-dependent gains. The 16-point headline no longer appears. Table 4 now reports three matched training seeds (summarized below; per-seed values in Table 26).

CE vs. CE + KL across training seeds. Qwen2.5-VL-7B with LoRA arms, pooled circular accuracy macro-averaged over the five benchmarks. Trained arms report mean ±\pm s.d. over three training seeds; the base model is deterministic and has no seed variation. Paired gain from KL is the per-seed difference (CE + KL −- CE), averaged over seeds, with the standard deviation of those paired differences.

MethodMean over k>0k{>}0k=5k{=}5
Qwen2.5-VL-7B (base)59.3351.37
CE only67.09±1.2067.09 \pm 1.2063.49±0.8763.49 \pm 0.87
CE + KL, λ=1\lambda{=}168.81±0.91\mathbf{68.81 \pm 0.91}65.96±1.15\mathbf{65.96 \pm 1.15}
Paired gain from KL+1.72±1.17+1.72 \pm 1.17+2.47±1.09+2.47 \pm 1.09

The CE gain over the base model (12.1 points at k=5k{=}5) far exceeds its seed variation (s.d. 0.87). The additional KL gain (+2.47±1.09+2.47 \pm 1.09 at k=5k{=}5) is positive in all three seeds and on all five benchmarks, grows with the number of distractors (from +0.42+0.42 at k=0k{=}0 to +2.47+2.47 at k=5k=5; Table 24), and lowers seed variance on four of five benchmarks. The gain is largest on CLEVR-size and What'sUp (+6.57+6.57 and +2.76+2.76 points at k=5k{=}5). With three seeds, the interval on the pooled gain remains wide, so we present distractor-augmented CE fine-tuning as our main mitigation and answer consistency as an exploratory extension with smaller, benchmark-dependent additional gains. The weight λ=1\lambda{=}1 is selected in all five partitions of a 20/80 split-stability analysis (held-out gain +1.73+1.73 vs. +1.72+1.72 on the full set).

The main text now acknowledges the What'sUp trade-off: fine-tuning lowers clean accuracy there by 3-4 points (92.2% →\rightarrow 89.1% CE, 87.9% CE + KL) but raises five-distractor accuracy by 17-20 points (52.7% →\rightarrow 69.8% and 72.6%). Clean accuracy improves on the other four benchmarks and on average (70.7% →\rightarrow 74.0% and 74.4%).

Pathway attribution. We agree that more controls help sharpen the pathway attribution. We therefore added three controls on Qwen2.5-VL (table below; Table 3 and Tables 20–21):

  • Question only (Q). Restricting the mask to question tokens (stem and options) retains nearly all of the recovery: 46% versus 47% for all post-image text (D), with 5% damage for both. The question tokens therefore carry most of the effect.
  • Inter-image masking with the system prefix readable (M). Preserving system-prefix access lowers the comparator's damage from 57% to 8%, but its recovery stays at 12%. The recovery advantage of distractor-to-text masking therefore does not depend on the original comparator's removal of the system prefix.
  • Budget-matched random mask (Y). Removing the same number of image-to-text connections at random across all images recovers 14% and 12% of broken examples at one and two distractors, compared with 49% and 47% for D, while damaging more preserved examples (10% vs. 2% at two distractors). (Tables 20 and 21)

Attention knockouts in layers 9-17 at five distractors, both architectures. Recovery is the fraction of broken examples repaired; damage is the fraction of preserved examples made incorrect. Q was run on Qwen2.5-VL only.

Cond.InterventionQwen2.5-VL RecoveryQwen2.5-VL DamageInternVL3-8B RecoveryInternVL3-8B Damage
PIMGD\text{IMG}_D →\rightarrow everything0.560.050.470.09
MMall preceding →\rightarrow IMG0.100.570.180.49
Mprior IMG →\rightarrow IMG0.120.080.100.10
DIMGD\text{IMG}_D →\rightarrow all text0.470.050.430.06
QIMGD\text{IMG}_D →\rightarrow question (stem ∪\cup options)0.460.05----
RIMGD\text{IMG}_D →\rightarrow readout0.030.020.020.01

Distractor screening, human audit, and MIMIC. Screening uses Qwen2.5-VL-7B-Instruct, which is asked, for each candidate in isolation, whether the image contains any queried object or could on its own support an answer (prompt in Figure 8). A set is rejected if any candidate fails, and it is redrawn up to 15 times. One annotator audited 200 rows at k=5k{=}5 from the three VLM-gated benchmarks. The overall false-acceptance rate is 3.5% (7/200; 95% Wilson interval $[1.7, 7.0] $; Table 8). All seven errors are on CV-Bench (7/100), with none on BLINK or What'sUp. CLEVR uses exact scene-graph vetting and needs no audit. MIMIC uses its native per-item distractors, with no external pool and no additional VLM filtering; Tables 5-7 and the text now describe it identically.

Visual-token budget and sequence length. We set max_pixels to 1,003,5201{,}003{,}520, which caps the total pixel area of each resized image. With Qwen2.5-VL's 14×1414{\times}14 patches and 2×22{\times}2 spatial merging, this permits at most 1,003,520/282=1,2801{,}003{,}520 / 28^2 = 1{,}280 decoder visual tokens per image (5,1205{,}120 pre-merge patches). With one target and five distractors, the decoder therefore receives at most 6×1,280=7,6806 \times 1{,}280 = 7{,}680 visual tokens, plus text and formatting tokens. The "28K" figure in the original Figure 11(b) came from a plotting-script error: we counted pre-merge patches rather than post-merge decoder tokens, inflating the visual-token count by a factor of four. We have removed it and clarified in Appendix A.2.1 that 1,280 is a per-image cap rather than a fixed count.

We hope these additions address the reviewer's concerns, and we are happy to provide further detail on any of them.

AuthorsOpenReview ↗

Follow-up on our response before the discussion deadline

Dear Reviewer sEKY, thank you again for your detailed review. As the discussion period ends tomorrow (November 18), we wanted to briefly check in on our response above. In short:

  • Generalization: the knockout result replicates on InternVL3-8B (43% recovery, 6% damage), with the same middle-layer concentration. Distraction also still degrades accuracy in the newer Qwen3.5-9B, though by less than in Qwen2.5-VL.
  • Pathway controls: blocking distractor attention to the question tokens alone retains nearly all of the recovery (46% vs. 47% for all post-image text). When the cross-image comparator is allowed to read the system prefix, its damage drops but its recovery stays at 12%, so the advantage of distractor-to-text masking does not come from removing system-prefix access.
  • Distractor screening: a human audit of 200 items gives a 3.5% false-acceptance rate. We also made the MIMIC descriptions consistent and corrected the sequence-length figure, which came from a plotting error that counted pre-merge patches.
  • Mitigation: over three training seeds, we now present distractor-augmented CE fine-tuning as the main mitigation and the KL objective as an exploratory extension (2.5 additional points at five distractors), and we removed the 16-point headline claim.

We hope these changes address your concerns, and we are happy to clarify anything further before the deadline.

Reviewer_AFHAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies why VLMs lose accuracy when irrelevant “distractor” images are added alongside the image needed to answer a question. The authors add 1-5 distractors to five benchmarks (CLEVR, CV-Bench, What’sUp, BLINK, MIMIC) and test Qwen2.5-VL, InternVL3, and LLaVA-OneVision. They make three claims:

a) Selection isn’t enough. Even when the prompt names the correct image, most of the accuracy loss remains. A black-image control shows that about 75% of the loss comes from distractor content, not longer context.

b) Distractors interfere through text tokens. In Qwen2.5-VL, blocking attention from distractor images to text tokens in middle layers (9-17) fixes 47% of broken examples while harming only 5% of correct ones. A cross-image mask from prior work fixes 10% and harms 56%.

c) A simple fix. Fine-tuning with a KL loss that pushes the model’s answer distribution with distractors toward its answer without distractors improves pooled circular accuracy by 10.4 points on average and 16 points at five distractors, compared to the base model.

Critical strengths
  1. Clean decomposition of the failure. Comparing regular, target-named, two-step, and hard-isolation prompting, plus the black-image control, separates selection errors, sequence length, and content interference. Showing that removing distractors fully restores accuracy while naming the target does not is a clear and useful finding.

  2. Well-designed knockout analysis. Reporting both recovery on broken items and damage on correct items, across layer bands and benchmarks, makes the claim about middle-layer distractor-to-text attention concrete and convincing for Qwen2.5-VL.

Critical weaknesses
  1. The headline training gain mostly comes from data, not the proposed loss. Training with plain cross-entropy on distractor-augmented data already gives +7.3 points. The KL term adds about 3 more points pooled (69.5 vs. 66.4), and at five distractors 67.4 vs. 63.6. The gain is inconsistent across benchmarks: on CV-Bench, KL with λ=1.0 is worse than CE-only. Results come from a single run, without confidence intervals, and λ appears to be picked by test-benchmark performance. The attention shift the paper links to its mechanism also appears in the CE-only model. The “+16 points” headline compares to the untrained base model, which overstates what the KL loss itself contributes, and the connection between the knockout findings and the method is loose.

  2. The “naming the target doesn’t help” result is confounded by the prompt change. The Ground-Truth prompt is worded differently from the Regular prompt, and it seems to hurt on its own. At k=1, Ground-Truth circular accuracy is 52.6 vs. 64.7 for Regular, and on CV-Bench it drops from about 68 to 42. This looks like a prompt-format effect, not distraction. Without a k=0 baseline using the same prompt, the claim that target naming recovers only 12% of the loss is unreliable. The hard-isolation results support the broader point, but the main number needs fixing.

Questions for authors
  1. What is Ground-Truth and Two-Step accuracy with the same prompt format at k=0? Why does Ground-Truth prompting drop CV-Bench circular accuracy so sharply even with one distractor?

  2. How was λ chosen? Can you report CE-only vs. KL over multiple seeds with confidence intervals?

  3. Do the knockout and fine-tuning results hold for InternVL3 or LLaVA-OneVision?

  4. Is blocking distractor-to-text attention different from simply softly removing distractors? A control that blocks the same number of non-distractor tokens, or target-to-text attention, would help.

  5. Why does the KL model also improve k=0 accuracy by 4 points? Does this mean part of the gain is general fine-tuning rather than distractor robustness?

4 comments

We thank the reviewer for recognizing the behavioral decomposition and knockout analysis, and for identifying the attribution and prompt-format issues. We have revised the mitigation claims and added multi-seed training results, same-format clean baselines, cross-model knockouts, and a budget-matched control. All section and table references below refer to the revised manuscript.

Reframing of the mitigation, multiple seeds, λ\lambda selection We thank the reviewer for highlighting the importance of separating the effects of fine-tuning and consistency regularization. The revised manuscript now presents distractor-augmented CE fine-tuning as our main mitigation and answer consistency as an exploratory extension with smaller, benchmark-dependent additional gains. The abstract, introduction, Section 6 and conclusion have been revised accordingly, and the 16-point headline no longer appears. Table 4 now reports three matched training seeds (summarized below).

CE vs. CE + KL across training seeds. Qwen2.5-VL-7B with LoRA arms, pooled circular accuracy macro-averaged over the five benchmarks. Trained arms report mean ±\pm s.d. over three training seeds; the base model is deterministic and has no seed variation. Paired gain from KL is the per-seed difference (CE + KL −- CE), averaged over seeds, with the standard deviation of those paired differences.

MethodMean over k>0k{>}0k=5k{=}5
Qwen2.5-VL-7B (base)59.3351.37
CE only67.09±1.2067.09 \pm 1.2063.49±0.8763.49 \pm 0.87
CE + KL, λ=1\lambda{=}168.81±0.91\mathbf{68.81 \pm 0.91}65.96±1.15\mathbf{65.96 \pm 1.15}
Paired gain from KL+1.72±1.17+1.72 \pm 1.17+2.47±1.09+2.47 \pm 1.09

CE fine-tuning alone improves five-distractor accuracy by 12.1 points over the base model, far beyond its seed variation (s.d. 0.87). The additional KL gain (+2.47±1.09+2.47 \pm 1.09 at k=5k{=}5) is positive in all three seeds (Table 26) and on all five benchmarks, and it grows with the number of distractors, from +0.42+0.42 at k=0k{=}0 to +2.47+2.47 at k=5k{=}5 (Table 24). The gain is largest on CLEVR-size and What'sUp (+6.57+6.57 and +2.76+2.76 points at k=5k{=}5). Conditional on the trained models, the gain is stable across evaluation items (item-level bootstrap 95% CI [+1.98,+2.97][+1.98, +2.97] at k=5k{=}5; Table 25), but with three training seeds the seed-level interval remains wide, which is why we present answer consistency as an exploratory extension.

All models use the epoch-3 checkpoint, without early stopping or checkpoint selection. We evaluate λ∈{0.1,0.5,1,2}\lambda \in \{0.1, 0.5, 1, 2\}. In a retrospective stability analysis using five stratified 20/80 partitions, selection on the 20% subset chooses λ=1\lambda = 1 in all five partitions; its mean improvement on the remaining 80% is 1.73 points, close to the full-set improvement of 1.72. We explicitly distinguish this analysis from independent test-set evaluation in the manuscript.

We also agree that the attention shift is not specific to KL. The revised Section 6 attributes it to fine-tuning in general and does not claim that the attention changes cause the accuracy gains. More broadly, we present the knockouts as identifying the pathway through which distractors interfere, and fine-tuning as a mitigation whose attention changes are consistent with that pathway, rather than as a method derived from it. For the final version, we will also report the prior-method baselines (DTS, CE + inter-image mask) over multiple seeds.

Prompt-format confounding. We agree that the original cross-format comparison conflated prompt-format effects with distraction. We have added same-format k=0k{=}0 baselines for Ground-Truth and Two-Step in Figures 4, 12, and 13, and replaced the cross-format recovery claim in the introduction and Section 4.

Substantial degradation persists relative to each format's own clean baseline. With five distractors, Qwen2.5-VL loses 6.56.5 mean-rotation accuracy points (9.49.4 circular) under Ground-Truth and 7.47.4 (10.610.6 circular) under Two-Step, averaged across five benchmarks. Even conditioning on correct phase-1 selection leaves a 6.06.0-point drop (8.68.6 circular), comparing clean and distracted performance on the same examples. Degradation likewise persists for InternVL3 and LLaVA-OneVision.

The controls also show a prompt-format penalty in Qwen2.5-VL on CV-Bench, MIMIC, and BLINK. On CV-Bench, this penalty is already present at k=0k{=}0 and is concentrated in circular accuracy: switching from Regular to Ground-Truth lowers clean circular accuracy from 72.0% to 45.8%, but costs only 6.7 vanilla and 9.1 mean-rotation points (78% to 69%). The distraction effect comes in addition to it: Ground-Truth still loses 8.1 circular points from k=0k{=}0 to k=5k{=}5 against its own clean baseline (mean-rotation: 69% to 63%). Thus, prompt format explains part of the original deficit, while a residual distraction penalty remains.

We have revised the introduction and Section 4 accordingly: our conclusion is that target guidance does not eliminate interference, rather than that it recovers a fixed small fraction of the Regular prompt's loss.

Generalization to another architecture We extended the knockout analysis to InternVL3-8B across all five benchmarks (Tables 22–23). Middle-layer distractor-to-text masking recovers 43% of broken examples while damaging approximately 6% of preserved examples. The same intervention recovers only 14% and 6% in early and late layers, respectively. Thus, the pathway and its middle-layer concentration replicate in a second architecture.

We are currently extending the fine-tuning experiments to another model and will report the results during the discussion period if they complete in time, or in the final version otherwise.

Attention Knockout (Random Control). We added a random control that removes the same number of image-to-text connections per text query as distractor-to-text masking, sampling across both target and distractor images. We use one and two distractors because, at higher counts, the random mask increasingly overlaps with distractor-specific masking, making the comparison less informative.

Across What’sUp and CV-Bench, distractor-specific masking (D) recovers 49% of broken examples versus 14% for the random control (Y) with one distractor, and 47% versus 12% with two distractors. At two distractors, it also damages fewer preserved examples: 2% versus 10% (Tables 20–21).

This control matches the number of removed connections, rather than their attention mass. The results support the importance of selectively blocking distractor connections beyond simply removing an equal number of visual connections.

Clean-image improvement. Yes, some improvement reflects better clean-image accuracy (Table 24). One hypothesis is that distractor-augmented training also improves selection of question-relevant information within a single image, which may itself contain irrelevant objects or background content. We have added this possible explanation to Section 6, while noting that our experiments do not directly test it.

We hope these revisions address the reviewer's concerns, and we would be glad to provide further detail on any of them.

AuthorsOpenReview ↗

Follow-up on our response before the discussion deadline

Dear Reviewer AFHA, thank you again for your careful review. As the discussion period ends tomorrow (November 18), we wanted to briefly check in on our response above. In short:

  • We reframed distractor-augmented CE fine-tuning as the main mitigation and removed the 16-point headline. Over three training seeds, the KL term adds 2.5 points at five distractors, positive in every seed and on every benchmark, and we now present it as an exploratory extension.
  • We added same-format clean baselines for the Ground-Truth and Two-Step prompts. Prompt format explains part of the original deficit, but a distraction penalty remains against each format's own clean baseline, and we revised the claim accordingly.
  • The knockout result replicates on InternVL3-8B (43% recovery, 6% damage). We also added a random control that blocks the same number of image-to-text connections, chosen at random across all images: it recovers only 12–14% of broken examples, compared with 47–49% when blocking distractor connections specifically.

We hope these changes address your concerns, and we are happy to clarify anything further before the deadline.

AuthorsOpenReview ↗

Fine-tuning results on InternVL3 and LLaVA-OneVision

Following up on your question about whether the fine-tuning results hold for InternVL3 or LLaVA-OneVision, we completed these experiments and added them as Appendix A.9 to the revised manuscript. We repeated the Section 6 protocol on InternVL3-8B and LLaVA-OneVision-7B with three training seeds per arm (two for LLaVA-OneVision CE + KL). All hyperparameters (learning rate, LoRA configuration, epochs, batch size, and λ=1.0\lambda=1.0) were taken unchanged from Qwen2.5-VL, with only image preprocessing following each model's processor. Due to time constraints, we currently evaluate on three of the five held-out benchmarks (CLEVR-size, CV-Bench, BLINK). What'sUp and MIMIC will be added in the final version.

Circular accuracy at k=5k=5 (percentage points, mean over seeds):

ModelBenchmarkCE − base(CE + KL) − CE
Qwen2.5-VLCLEVR-size+33.5+6.6
CV-Bench+5.3+0.8
BLINK+5.2+0.9
InternVL3CLEVR-size+18.5+1.0
CV-Bench+1.1+1.7
BLINK−2.6+0.4
LLaVA-OneVisionCLEVR-size+3.70.0
CV-Bench+3.1−1.4
BLINK+5.20.0

Distractor-augmented CE fine-tuning transfers. At five distractors, it improves circular accuracy over the untrained base in 8 of 9 model–benchmark combinations (mean +8.1 points) and narrows the clean-to-distracted gap in 8 of 9. The exception is InternVL3 on BLINK, where fine-tuning lowers accuracy by 2.6–2.9 points at both k=0k=0 and k=5k=5, while leaving the clean-to-distracted drop nearly unchanged (7.3 vs. 7.0 points). Since no hyperparameter was re-tuned per model, these gains do not depend on model-specific choices, supporting the robustness of the mitigation.

The additional KL gain does not transfer consistently. On InternVL3, it is positive but small and comparable to seed variation. On LLaVA-OneVision, it is absent. This is consistent with the reframing we made in response to the reviews: distractor-augmented CE fine-tuning is the robust mitigation, and answer consistency is an exploratory extension. The revised text states this explicitly.

Limitations. The LLaVA-OneVision CE + KL arm has two seeds rather than three, so its standard deviations are indicative only. The third seed will be included in the final version.

Thank you again for your careful review. We hope these results, together with our earlier responses, address your concerns and can be considered in your final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.