- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 3(+1)
- Soundness
- 2
Read review
Paper summary
The paper studies modality distraction, where a model correctly answers a question using one type of input, such as an image or text, but gives an incorrect answer when irrelevant information from another modality is added. To study this problem, the authors introduce three datasets: MoGround-Base, a generated and filtered dataset covering four visual domains; MoGround-Human, a collection of human-written examples designed to challenge models; and MoGround-Retrieved, built from existing visual question answering and reading comprehension benchmarks. The authors find that across seven VLMs, higher single-modality accuracy is associated with lower distraction. Their fine-tuning method helps models avoid being distracted by irrelevant visual information on unseen datasets, without significantly affecting their performance on standard benchmarks.
Critical strengths
- The paper introduces a useful benchmark for measuring modality distraction.
- The three evaluation pools are well designed, and the human-generated examples are a valuable contribution.
- The proposed method consistently reduces vision-side distraction on held-out examples.
- The method is practical and adds no extra inference passes.
Critical weaknesses
-
The comparison between vision and text is difficult to interpret: The authors compare accuracy on image-based questions with accuracy on a different set of text-based questions. However, they do not establish that the two sets have similar difficulty. If the text questions are easier, models could answer them more accurately and resist distraction more easily. This would not necessarily show that the models understand text better than images. Can the authors explain how they account for question difficulty when drawing this conclusion?
-
The distinction from Cai et al. (2025): The paper says prior work does not ensure that the answer comes from only one modality. However, Cai et al. study tasks such as image classification and text-only QA, where the relevant information is meant to come from one modality. Their task design differs from MoGround’s explicit item-level verification, but the paper should acknowledge this overlap. Cai et al. also study irrelevant inputs in both directions, irrelevant text on image tasks and irrelevant images on text tasks, so describing their work mainly as contradictory text is incomplete. They report results across models and propose a fine-tuning remedy. MoGround’s analysis of model-dependent distraction asymmetry offers a more specific distinction, which the authors should explain precisely rather than describing prior work as treating interference as “a universal property of VLMs.” The verified benchmark and conditional flip-rate metric are useful contributions, but the authors should clarify what their analysis and mitigation add beyond Cai et al. Can the authors clarify how their approach differs from Cai et al.'s and include a direct comparison with their mitigation method?
-
The presentation of Tables 1 and 2 makes the findings difficult to follow: Table 2 introduces a “Gap” column, the difference between text-only and vision-only accuracy, which is absent from Table 1. This makes it harder to compare the tables and understand how the gap relates to distraction in each direction. The different metrics and table formats also make the main findings less clear. Could the authors use consistent columns and definitions across both tables and explain the main takeaway from each more directly?
Minor suggestion: model-size ablation within the same family. The comparison between Qwen2.5-VL-3B and 7B is useful, but the broader claim that model size does not predict distraction relies on different model families within a narrow size range. If the authors could evaluate additional sizes within one family, this would help separate the effects of model size and grounding accuracy, and show whether the mitigation remains effective for larger models.
Questions for authors
Check the weaknesses.
5 comments
Response to Reviewer pone (2/2)
W2. Relation to Cai et al. [1], and a direct comparison. The reviewer notes that our Related Work describes Cai et al. [1] incompletely, and asks what our paper adds. The revised Related Work now describes their work accurately. MoGround adds three things [1] does not have:
- Realistic distractors. [1] adds unrelated content in both directions (text on image tasks, images on text tasks), or misleading text that contradicts the image, which is a conflict rather than a distraction. MoGround adds a realistic on-topic caption of the same scene that does not contain the answer (except MoGround-Retrieved). This realistic caption distracts more than any other kind of caption we tested (Table 13).
- Verified items. [1]'s tasks are designed so that the answer comes from one modality, but items are not checked one by one. Every MoGround-Base item is checked by two LLMs, and a human audit confirms these checks, so the answer is in exactly one modality, and the added content does not give the answer. A changed answer therefore must be a distraction, not an item that became unanswerable.
- A conditional distraction rate. [1] reports accuracy. We report the conditional distraction rate: the share of correct answers a model loses. This makes models of different accuracy comparable, and it reveals that which modality distracts more depends on the model, and its text−image accuracy gap predicts it (W1). [1] instead attributes robustness to model scale, but size does not predict distraction (r=+0.09), while grounding strength does (r=-0.90).
Direct comparison. We trained [1]'s objective in our pipeline so that only the objective differs: Qwen2.5-VL-3B, same training data, same LoRA rank and merge strength, two seeds, their published settings (App. E.7, Table 19).
| Qwen2.5-VL-3B | Relative change in v-distraction, MoGround-Base test / Human / Retrieved | MoGround-Retrieved accuracy | General capability |
|---|---|---|---|
| Robustness vector (4 seeds) | -53 / -58 / -20% | +0.1 | -0.2 |
| Cai et al.'s objective [1] (2 seeds) | -50 / -36 / -21% | -1.2 | +0.0 |
Their objective nearly matches ours on MoGround-Base test, the source of the training data, but not beyond it:
- MoGround-Human: they reduce distraction less (-36% vs -58%).
- MoGround-Retrieved: it reduces vision distraction as much as ours (-21% vs -20%), but text items are distracted 2.2 points more often than with ours, and accuracy falls 1.2 points below the base model (ours: +0.1).
Takeaway: verified items and a conditional rate are what let MoGround show that the direction of distraction is model-dependent and predictable. On equal data, our method also outperforms [1]'s overall and has a smaller side effect.
[1] Cai et al. (2025). Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models
W3. Tables 1&2 are not easy to follow. The reviewer notes that Table 2 has a Gap column (text-only minus image-only accuracy) that Table 1 lacks, that the two tables use different formats, and asks for consistent columns and a direct takeaway for each. We agree and have updated the PDF to use the same column names in both. Table 1 has no Gap column because its text side is at ceiling (99-100%) by construction. Each table now clarifies with one takeaway sentence in the text:
- Table 1 (MoGround-Base): every model is distracted on vision items, more when its image-only accuracy is lower.
- Table 2 (MoGround-Retrieved): for each model, the more distracted modality is the one where its accuracy is lower.
Takeaway: both tables now use the same columns and state their main point directly.
Minor. Model size within one family. The reviewer suggests testing several sizes within one model family, to separate model size from grounding accuracy and to check that the mitigation works on larger models. We ran two sizes of Qwen2.5-VL and of InternVL3, plus Qwen3-VL-32B (App. E.10, Table 22):
| Family | Size | V-distraction before training, Base test | Reduction, Base test | Reduction, Retrieved |
|---|---|---|---|---|
| Qwen2.5-VL | 3B / 7B | 5.6 / 5.7% | -53 / -65% | -20 / -16% |
| InternVL3 | 8B / 14B | 2.2 / 3.5% | -33 / -30% | -38 / -31% |
| Qwen3-VL | 32B | 3.7% | -40% | -36% |
Doubling size within a family does not lower distraction on its own (5.6 to 5.7%, 2.2 to 3.5%), as across models (App. E.1). On the two largest models, 14B and 32B, general capability changes by at most 0.4 points, and both still fall on the grounding relation fitted to the original seven models.
Takeaway: bigger size alone does not automatically reduce distraction, and the robustness vector reduces it at every size we tested, up to 32B.
We hope our responses and new results address the reviewer concerns, and we would be glad to clarify anything that remains unclear.
Gentle Reminer
Thank you once more for your thoughtful review and the time you invested in it. We hope our responses address your concerns about question difficulty, comparison with Cai et al., and clarity on Tables 1 and 2. Since presentation was one of your concerns, each answer opens with a summary of your concern and closes with a takeaway. We also integrated all feedback into the revised version of the paper. We would be glad to hear your thoughts, and we are happy to clarify anything that remains unclear.
Response to Reviewer pone (1/2)
We are grateful the reviewer finds the benchmark useful, our evaluation framework well designed, and our proposed method practical. The general response lists every change to the paper.
W1. Comparing vision and text is difficult to interpret. The reviewer notes that our vision and text items are different questions, so if the text questions were easier, models would look more distracted on vision without necessarily understanding text better than images. We do not claim that models understand one modality better than the other. Our claim is per model: on MoGround-Retrieved, each model is more distracted on whichever set it answers less accurately, its vision items or its text items. Measuring distraction on both modalities on the same items is impossible by construction. Each item has its answer in one and only one modality, so a vision item can only test distraction by added text, and a text item can only test distraction by an added image.
If text items were universally easier, every model would be more distracted on vision. Three analyses show this is not the case. Let's define the correlation r: across the seven models, how well a model's accuracy gap between text and image predicts which modality distracts it more. An r near +1 means the modality a model answers less accurately is reliably its more distracted one. Difficulty could differ at two levels: one modality's items (e.g., text items) could be easier for every model, or individual items could differ in how hard they are (e.g., item X is harder than Y). We check both.
Modality-level difficulty:
Sample-level difficulty (new in the rebuttal, App. E.2):
Takeaway: the distraction asymmetry is not caused by one modality's questions being universally easier or harder. Even on the same questions, each model is more distracted in whichever modality it answers less accurately.