acceptodds
Under review as a conference paper at ICLR 2027

MoGround: Measuring and Mitigating Modality Distraction in Vision-Language Models

Abstract

We release MoGround, a vision-language dataset spanning four visual domains in which the answer to every question is guaranteed to be available from exactly one modality. This guarantee enables us to measure modality distraction, the failure in which a model answers a question correctly from one modality alone and then flips to a wrong answer once irrelevant content from the other modality is added. Existing probes rarely establish single-modality answerability this way, making it hard to isolate distraction in the first place. Across seven open-source VLMs, we find that modality distraction is not universal but model-dependent. The weaker-grounded modality is the more distracted one (r = +0.86), and distraction scales inversely with grounding strength (r = -0.90). The single-modality guarantee also enables a mitigation method that needs to distinguish between relevant and irrelevant context. Trained on one split of MoGround alone, a weight-space robustness vector reduces distraction on all seven models by 29% to 51%, at a cost of only 0.1 average points of accuracy on standard multimodal tasks.

Primary area:
datasets and benchmarks
Keywords:
VLM, multimodality

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_poneOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The paper studies modality distraction, where a model correctly answers a question using one type of input, such as an image or text, but gives an incorrect answer when irrelevant information from another modality is added. To study this problem, the authors introduce three datasets: MoGround-Base, a generated and filtered dataset covering four visual domains; MoGround-Human, a collection of human-written examples designed to challenge models; and MoGround-Retrieved, built from existing visual question answering and reading comprehension benchmarks. The authors find that across seven VLMs, higher single-modality accuracy is associated with lower distraction. Their fine-tuning method helps models avoid being distracted by irrelevant visual information on unseen datasets, without significantly affecting their performance on standard benchmarks.

Critical strengths
  • The paper introduces a useful benchmark for measuring modality distraction.
  • The three evaluation pools are well designed, and the human-generated examples are a valuable contribution.
  • The proposed method consistently reduces vision-side distraction on held-out examples.
  • The method is practical and adds no extra inference passes.
Critical weaknesses
  • The comparison between vision and text is difficult to interpret: The authors compare accuracy on image-based questions with accuracy on a different set of text-based questions. However, they do not establish that the two sets have similar difficulty. If the text questions are easier, models could answer them more accurately and resist distraction more easily. This would not necessarily show that the models understand text better than images. Can the authors explain how they account for question difficulty when drawing this conclusion?

  • The distinction from Cai et al. (2025): The paper says prior work does not ensure that the answer comes from only one modality. However, Cai et al. study tasks such as image classification and text-only QA, where the relevant information is meant to come from one modality. Their task design differs from MoGround’s explicit item-level verification, but the paper should acknowledge this overlap. Cai et al. also study irrelevant inputs in both directions, irrelevant text on image tasks and irrelevant images on text tasks, so describing their work mainly as contradictory text is incomplete. They report results across models and propose a fine-tuning remedy. MoGround’s analysis of model-dependent distraction asymmetry offers a more specific distinction, which the authors should explain precisely rather than describing prior work as treating interference as “a universal property of VLMs.” The verified benchmark and conditional flip-rate metric are useful contributions, but the authors should clarify what their analysis and mitigation add beyond Cai et al. Can the authors clarify how their approach differs from Cai et al.'s and include a direct comparison with their mitigation method?

  • The presentation of Tables 1 and 2 makes the findings difficult to follow: Table 2 introduces a “Gap” column, the difference between text-only and vision-only accuracy, which is absent from Table 1. This makes it harder to compare the tables and understand how the gap relates to distraction in each direction. The different metrics and table formats also make the main findings less clear. Could the authors use consistent columns and definitions across both tables and explain the main takeaway from each more directly?

Minor suggestion: model-size ablation within the same family. The comparison between Qwen2.5-VL-3B and 7B is useful, but the broader claim that model size does not predict distraction relies on different model families within a narrow size range. If the authors could evaluate additional sizes within one family, this would help separate the effects of model size and grounding accuracy, and show whether the mitigation remains effective for larger models.

Questions for authors

Check the weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer pone (1/2)

We are grateful the reviewer finds the benchmark useful, our evaluation framework well designed, and our proposed method practical. The general response lists every change to the paper.


W1. Comparing vision and text is difficult to interpret. The reviewer notes that our vision and text items are different questions, so if the text questions were easier, models would look more distracted on vision without necessarily understanding text better than images. We do not claim that models understand one modality better than the other. Our claim is per model: on MoGround-Retrieved, each model is more distracted on whichever set it answers less accurately, its vision items or its text items. Measuring distraction on both modalities on the same items is impossible by construction. Each item has its answer in one and only one modality, so a vision item can only test distraction by added text, and a text item can only test distraction by an added image.

If text items were universally easier, every model would be more distracted on vision. Three analyses show this is not the case. Let's define the correlation r: across the seven models, how well a model's accuracy gap between text and image predicts which modality distracts it more. An r near +1 means the modality a model answers less accurately is reliably its more distracted one. Difficulty could differ at two levels: one modality's items (e.g., text items) could be easier for every model, or individual items could differ in how hard they are (e.g., item X is harder than Y). We check both.

Modality-level difficulty:

  • Same items for every model. On MoGround-Retrieved (Table 2), all seven models answer the same 4757 items, yet the more-distracted modality differs: vision for 2 models, text for 4, and 1 tie. If text items were universally easier, every model would be more distracted on vision.

Sample-level difficulty (new in the rebuttal, App. E.2):

  1. Difficulty-matched subsets. We built 20 random subsets in which vision and text items are equally hard, hardness scored by how many of the other six models solve an item. On them, r = +0.88.
  2. A new split shared by all models. We built a split of the 1617 vision and 1318 text items that all seven models answer correctly, so every model has the same conditioning set. On it, r = +0.82.

Takeaway: the distraction asymmetry is not caused by one modality's questions being universally easier or harder. Even on the same questions, each model is more distracted in whichever modality it answers less accurately.

AuthorsOpenReview ↗

Response to Reviewer pone (2/2)

W2. Relation to Cai et al. [1], and a direct comparison. The reviewer notes that our Related Work describes Cai et al. [1] incompletely, and asks what our paper adds. The revised Related Work now describes their work accurately. MoGround adds three things [1] does not have:

  • Realistic distractors. [1] adds unrelated content in both directions (text on image tasks, images on text tasks), or misleading text that contradicts the image, which is a conflict rather than a distraction. MoGround adds a realistic on-topic caption of the same scene that does not contain the answer (except MoGround-Retrieved). This realistic caption distracts more than any other kind of caption we tested (Table 13).
  • Verified items. [1]'s tasks are designed so that the answer comes from one modality, but items are not checked one by one. Every MoGround-Base item is checked by two LLMs, and a human audit confirms these checks, so the answer is in exactly one modality, and the added content does not give the answer. A changed answer therefore must be a distraction, not an item that became unanswerable.
  • A conditional distraction rate. [1] reports accuracy. We report the conditional distraction rate: the share of correct answers a model loses. This makes models of different accuracy comparable, and it reveals that which modality distracts more depends on the model, and its text−image accuracy gap predicts it (W1). [1] instead attributes robustness to model scale, but size does not predict distraction (r=+0.09), while grounding strength does (r=-0.90).

Direct comparison. We trained [1]'s objective in our pipeline so that only the objective differs: Qwen2.5-VL-3B, same training data, same LoRA rank and merge strength, two seeds, their published settings (App. E.7, Table 19).

Qwen2.5-VL-3BRelative change in v-distraction, MoGround-Base test / Human / RetrievedMoGround-Retrieved accuracyGeneral capability
Robustness vector (4 seeds)-53 / -58 / -20%+0.1-0.2
Cai et al.'s objective [1] (2 seeds)-50 / -36 / -21%-1.2+0.0

Their objective nearly matches ours on MoGround-Base test, the source of the training data, but not beyond it:

  • MoGround-Human: they reduce distraction less (-36% vs -58%).
  • MoGround-Retrieved: it reduces vision distraction as much as ours (-21% vs -20%), but text items are distracted 2.2 points more often than with ours, and accuracy falls 1.2 points below the base model (ours: +0.1).

Takeaway: verified items and a conditional rate are what let MoGround show that the direction of distraction is model-dependent and predictable. On equal data, our method also outperforms [1]'s overall and has a smaller side effect.

[1] Cai et al. (2025). Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models


W3. Tables 1&2 are not easy to follow. The reviewer notes that Table 2 has a Gap column (text-only minus image-only accuracy) that Table 1 lacks, that the two tables use different formats, and asks for consistent columns and a direct takeaway for each. We agree and have updated the PDF to use the same column names in both. Table 1 has no Gap column because its text side is at ceiling (99-100%) by construction. Each table now clarifies with one takeaway sentence in the text:

  • Table 1 (MoGround-Base): every model is distracted on vision items, more when its image-only accuracy is lower.
  • Table 2 (MoGround-Retrieved): for each model, the more distracted modality is the one where its accuracy is lower.

Takeaway: both tables now use the same columns and state their main point directly.


Minor. Model size within one family. The reviewer suggests testing several sizes within one model family, to separate model size from grounding accuracy and to check that the mitigation works on larger models. We ran two sizes of Qwen2.5-VL and of InternVL3, plus Qwen3-VL-32B (App. E.10, Table 22):

FamilySizeV-distraction before training, Base testReduction, Base testReduction, Retrieved
Qwen2.5-VL3B / 7B5.6 / 5.7%-53 / -65%-20 / -16%
InternVL38B / 14B2.2 / 3.5%-33 / -30%-38 / -31%
Qwen3-VL32B3.7%-40%-36%

Doubling size within a family does not lower distraction on its own (5.6 to 5.7%, 2.2 to 3.5%), as across models (App. E.1). On the two largest models, 14B and 32B, general capability changes by at most 0.4 points, and both still fall on the grounding relation fitted to the original seven models.

Takeaway: bigger size alone does not automatically reduce distraction, and the robustness vector reduces it at every size we tested, up to 32B.


We hope our responses and new results address the reviewer concerns, and we would be glad to clarify anything that remains unclear.

AuthorsOpenReview ↗

Gentle Reminer

Thank you once more for your thoughtful review and the time you invested in it. We hope our responses address your concerns about question difficulty, comparison with Cai et al., and clarity on Tables 1 and 2. Since presentation was one of your concerns, each answer opens with a summary of your concern and closes with a takeaway. We also integrated all feedback into the revised version of the paper. We would be glad to hear your thoughts, and we are happy to clarify anything that remains unclear.

Reviewer_nVWbOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary

This paper investigates modality distraction in vision-language models (VLMs), where irrelevant information from one modality causes a model to change a correct answer into an incorrect one. The authors introduce MoGround, a benchmark that verifies single-modality answerability, and demonstrate a strong negative correlation between grounding strength and distraction (r = −0.90). They further propose a LoRA-based robustness vector that reduces distraction by 29–51% across three evaluation datasets while largely preserving general multimodal performance.

Critical strengths

S1. Clear problem definition and benchmark construction

The paper clearly defines modality distraction and introduces MoGround to isolate this phenomenon through single-modality answerability verification, supported by cross-oracle validation and human annotation.

S2. Comprehensive empirical analysis

The authors investigate modality distraction across multiple VLMs and visual domains, and further examine the grounding–distraction relationship through within-model resolution perturbations and additional robustness analyses.

S3. Practical mitigation approach

The proposed robustness vector consistently reduces modality distraction across three evaluation datasets while largely preserving general multimodal capabilities and introducing no additional inference-time forward passes after weight merging.

Critical weaknesses

W1. Potential reliance on dataset-specific modality cues.

Appendix E.4 reports that a bag-of-words classifier can predict the grounding modality from question text alone with 95.7% accuracy (AUROC 0.989), indicating strong dataset-specific linguistic cues. Although the authors acknowledge this limitation, it raises concerns about whether the proposed robustness vector learns genuine resistance to irrelevant cross-modal information or partially exploits superficial cues associated with the grounding modality. Additional experiments controlling for these cues would strengthen the validity and generalizability of the proposed approach.

W2. Insufficient evidence for the distinct contribution of the proposed robustness vector.

The proposed approach builds on standard LoRA fine-tuning, while its claimed advantages in inference efficiency and deployability are also achievable through conventional LoRA weight merging. Without a direct comparison against standard LoRA fine-tuning using the same training data, it remains unclear whether the observed improvements stem from the proposed training strategy or from fine-tuning itself.

Questions for authors

Q1. Dataset-specific modality cues.

Could the authors evaluate whether the robustness vector remains effective when linguistic cues associated with the grounding modality are controlled? In particular, do similar improvements hold on subsets where the grounding modality cannot be easily predicted from question text alone?

Q2. Comparison with standard LoRA fine-tuning.

Could the authors compare the proposed robustness vector against standard LoRA fine-tuning using the same training data? Which components of the proposed training procedure contribute most to reducing modality distraction while preserving general capabilities?

Q3. Comparability of conditional distraction rates.

Since distraction rates are computed over different subsets of correctly answered items for each model, could the authors evaluate distraction on a shared subset of items that all models answer correctly? Would the relative differences in distraction rates remain consistent under this controlled evaluation?

Q4. Grounding–distraction relationship.

While the within-model resolution experiments provide additional evidence for the grounding–distraction relationship, reducing image resolution also changes visual input quality. Could the authors clarify to what extent the observed relationship reflects grounding strength itself rather than other effects of image degradation? Additionally, how consistent is this relationship across individual models and domains? Minor Question Q5. The citation formatting appears to differ from the standard ICLR style, particularly in the citation link colors. Could the authors verify that the manuscript follows the official ICLR 2027 formatting guidelines?

5 comments
AuthorsOpenReview ↗

Response to Reviewer nVWb

We thank the reviewer for the positive assessment and the precise questions.


W1 / Q1. Dataset-specific linguistic cues. The reviewer notes that on MoGround-Base a bag-of-words classifier predicts the grounding modality from the question alone in App. E.6, and asks whether the robustness vector learns real robustness to distraction or exploits these cues. To address the concern, we show that the robustness vector's gain does not depend on the cues (App. D.8, Table 11). We split vision items into subsets by the classifier's confidence that they are vision items. Each cell is the relative change in v-distraction over all seven models:

DatapoolLeast likely vision items (weakest cues)MiddleMost likely vision items (strongest cues)
MoGround-Human-48%-48%-55%
MoGround-Retrieved, held-out-31%-31%-22%

If the robustness vector exploited the cues, its gain would shrink on the items with the weakest cues. It does not: on MoGround-Human, weak-cue items improve almost as much as strong-cue ones, and on MoGround-Retrieved, they even improve more.

Takeaway: the vector learns robustness to distraction, not the word patterns. Its gain holds where the cues are weakest too.


W2 / Q2. Ablation of the method. The reviewer notes that the efficiency of the robustness vector also holds for plain LoRA, and asks whether the gains come from our training recipe or from LoRA itself. We trained, with 2 seeds, on the same split at w=0.5, plain LoRA (cross-entropy SFT) and our method without adversarial captions (App. E.9, Table 20):

Relative change in v-distraction, MoGround-Base test / Human / RetrievedQwen2.5-VL-3BLLaVA-NeXT-8B
Full method-53 / -58 / -20%-47 / -47 / -39%
Plain LoRA SFT-32 / -35 / -15%-39 / -41 / -32%
Without adversarial captions-39 / -33 / -10%-27 / -19 / -13%

The ablations show where the gains come from:

  • Our verified data itself helps. Plain LoRA SFT on MoGround-Base train already reduces distraction, showing that the verified data itself is already useful to mitigate distraction.
  • Adversarial captions improve further. Removing them shrinks the reduction on every pool for both models, and the full recipe wins on every model and datapool.
  • The full method also protects the text side. On Qwen2.5-VL-3B / MoGround-Retrieved, plain LoRA has 1.3 points more text distraction and 0.6 points lower accuracy than the full method.

Takeaway: the adversarial captions, which are specific to our method, contribute to improving over plain LoRA SFT.


Q3. A shared set of items. The reviewer notes that each model's distraction rate is computed on the items that model answers correctly, so models are scored on different items, and asks whether the findings hold on items all models answer correctly. They do (App. E.2).

  • MoGround-Base: on the vision items all seven models answer from the image, the model ranking matches Table 1, and distraction in a modality is still lower when a model is stronger at that modality.
  • MoGround-Retrieved: on vision and text items all seven models solve, the more distracted modality for each model mostly matches Table 2. Across the seven models, the distraction asymmetry on these items still follows each model's accuracy gap at r = +0.82.

Takeaway: on the common items all seven models solve, the main findings hardly change.


Q4. Resolution and the grounding–distraction relation. The reviewer asks whether lower resolution raises distraction through weaker grounding or through other effects of image damage, and how consistent the relation is (App. E.3):

  • Lost accuracy is not counted as distraction. Distraction is counted only on items the model still answers correctly from the damaged image, so the rise comes from correct answers lost once the distracting caption is added.
  • Image damage has little effect of its own. A downscaled model is about as distracted as a full-resolution model with the same grounding strength. Its points fall on the relation fitted to full-resolution models on MoGround-Retrieved.
  • The relation is consistent. Within each model, the resolution sweep gives r=-0.90 to -0.99, so the lower the model's accuracy on downscaled images, the more it gets distracted.

Takeaway: lower resolution raises distraction by weakening grounding, not by lowering accuracy, and the relation holds within each model.


Q5. Citation formatting. We thank the reviewer for noting that our citation style differs from the ICLR template. The revised PDF now uses the default hyperref settings.


We hope the additional analyses and clarifications resolve the reviewer concerns.

AuthorsOpenReview ↗

Follow-up on W1 / Q1 (Modality Cues)

W1 / Q1. Training on data with few word cues. Following up on our earlier reply to W1/Q1: there, we showed that the robustness vector's gains do not depend on word cues in the test items. Here we address the remaining part of the question with training data that has few cues: we train the robustness vector (Qwen2.5-VL-3B, same configuration) only on items whose modality is hard to predict from the text.

Filtering. We represent each item's question, options, and caption as a multi-hot bag of words concatenated with a sentence embedding (all-mpnet-base-v2), and score all 8172 items of MoGround-Base and MoGround-Retrieved with a logistic regression under 5-fold cross-validation, so no item is scored by a classifier trained on it. We keep the items it cannot classify confidently, with a probability in [0.3, 0.7] (strong filter) or [0.2, 0.8] (weak filter), and add MoGround-Human, giving 297 and 400 items. Note that, unlike in the paper, this experiment trains on MoGround-Human too.

The filter removes the cues this classifier relies on.* This classifier separates vision from text at AUROC 0.998 on all items. Retrained from scratch on the kept items, it falls to chance: 0.46 [0.38, 0.54] (strong filter) and 0.50 [0.44, 0.57] (weak filter). With MoGround-Human added, it reaches 0.61 [0.55, 0.67] and 0.60 [0.55, 0.65]. Brackets are 95% bootstrap intervals: we resample the items with replacement 1000 times and recompute the AUROC each time.

Training and evaluation. We again use 5-fold cross-validation, now with the same mix of pools, data sources, and modalities in each fold, and items that share an image kept in the same fold. The robustness vector trains on four folds and is tested on the fifth, so every item is evaluated by a vector that never saw it. As a baseline, each round trains on items with cues instead: every training item with few cues is replaced by a random item the filter rejected, matched on pool, source, and modality. MoGround-Human items stay the same. Both vectors are tested on the same fold. As in the paper, training requires adversarial captions. Most vision items already had one. For those that had never been given one, we generated one with the same pipeline.

Relative change in v-distraction [95% CI]Strong filter (297 items)Weak filter (400 items)
Trained on items with few cues−47% [−71, −23]−57% [−77, −37]
Trained on the same number of items with cues−36% [−60, −14]−48% [−69, −27]
Difference (pp)−11 [−27, 0]−9 [−30, +12]

The confidence intervals come from a paired bootstrap. V-distraction is measured on vision items, so we resample those with replacement 4000 times, using the same draw for both vectors and the base model, and recompute both relative changes and their difference.

Results. Trained on items with few cues, the robustness vector still reduces distraction significantly, and at least as much as training on items with cues (the difference is not significant). It also comes close to the original vector trained on the full 2,050-item train split. For a fair comparison, we drop items the original vector was trained on and items sharing an image with them, leaving 188 and 222 items, most from MoGround-Human. There, the original vector changes v-distraction by −52% and −53%, and the vectors trained on items with few cues by −43% and −53%, with no significant difference.

Takeaway: the robustness vector does not need word cues to learn. Trained on items with few cues, it reduces distraction nearly as much as the original.

AuthorsOpenReview ↗

Gentle Reminder

We are grateful for the time you dedicated to reviewing our work. We hope our responses answer your questions. Our follow-up on W1/Q1 trained the robustness vector on filtered data with fewer word patterns. Each answer opens with a summary of your concern and ends with a takeaway. We are happy to discuss anything further.

Reviewer_aGxHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors focus on the issue of modality distraction in vision-language models (VLMs). To measure this, they introduce MoGround, a dataset built across four visual domains. The dataset uses LLM-based verification and human audits to check that each question is answerable from only one modality. Experiments on seven VLMs show that distraction varies across models and is strongly related to their grounding strength. The authors also analyze answer confidence and internal model states to better understand this behavior. To reduce distraction, they propose a LoRA-based training with modified captions and regularization. Experiments show that the method reduces vision distraction across the tested models.

Critical strengths

The paper addresses a meaningful problem. The issue of modality distraction has been studied recently. Understanding when it occurs and how to reduce it remains valuable.

The proposed dataset is a useful contribution. MoGround provides a controlled way to measure modality distraction. The dataset covers several visual domains and includes LLM-based verification and human audits to support its quality.

The paper provides extensive experimental analysis. The authors evaluate several VLMs and examine distraction from different perspectives. The additional controls and analyses help support the findings.

The findings themselves are helpful, and the mitigation results are positive. In particular, the authors show that the direction of distraction varies across models and is related to grounding strength. They also explore ways to reduce distraction and report improvements with little average loss in general performance.

Critical weaknesses

The novelty over existing work is somewhat limited. Prior work has already studied modality distraction and proposed ways to reduce it [Cai et al., 2025;Yang et al., 2026]. The mitigation method in this work mainly combines existing techniques, including LoRA fine-tuning, adversarial data augmentation, and regularization.

The dataset is partly artificial and may contain systematic biases. MoGround is built from existing datasets using LLM-generated questions and captions. It appears that text-grounded questions are often too easy, with nearly 100% accuracy across models. The authors introduce harder evaluation sets to address this issue, but the human-written subset contains only 125 items.

The experimental comparisons are not fully controlled, and the model coverage is limited. Vision and text distraction are measured using different questions and datasets. Differences in task difficulty may therefore affect the observed asymmetry between modalities. The main experiments also use relatively old open-weight models.

Some of the main findings are expected and offer limited new insights. The authors find that weaker grounding is associated with greater distraction. Also, smaller answer margins make predictions more likely to change. Both findings are intuitive. The experiments provide useful evidence for these relationships, but their significance as new insights is somewhat limited.

Questions for authors

How does the proposed benchmark differ from the existing benchmarks specifically designed to study modality distraction or interference in VLMs?

How do the authors control for differences in the difficulty and dataset sources when comparing vision with text distraction for the direction?

5 comments
AuthorsOpenReview ↗

Response to Reviewer aGxH (1/2)

We thank the reviewer for finding the problem meaningful, the dataset useful, and the analysis extensive. We address their comments below.


W1 / Q1. Novelty over Cai et al. [1] and Yang et al. [2]. The reviewer finds the novelty limited because prior work already studies modality distraction and proposes remedies, and our mitigation combines existing ingredients (LoRA fine-tuning, adversarial data augmentation, regularization). The revised Related Work now describes both works accurately. MoGround adds three things neither work has:

  • Realistic distractors. [1] adds unrelated content in both directions (text on image tasks, images on text tasks), or misleading text that contradicts the image, which is a conflict rather than a distraction. [2] adds unrelated images or text. MoGround adds a realistic on-topic caption of the same scene that does not contain the answer (except MoGround-Retrieved). This realistic caption distracts more than other kinds of captions we tested (Table 13).
  • Verified items. Every MoGround-Base item is checked by two LLMs, and a human audit confirms these checks, so the answer is in exactly one modality, and the added content does not give the answer. A changed answer therefore must be a distraction, not an item that became unanswerable. No prior work verifies this per item.
  • A conditional distraction rate. [1] reports accuracy and [2] answer agreement. We report the conditional distraction rate: the share of correct answers a model loses due to distraction. This design reveals that which modality distracts more depends on the model, and it can be predicted. [1] instead attributes distraction robustness to model scale, while we show that grounding strength, not scale, predicts distraction.

On the mitigation, we do not claim the ingredients are new. The contribution is that verified single-modality data makes a simple recipe work where existing baselines fall short (Table 4). New ablations (App. E.9) show that both ingredients help: plain LoRA SFT on our data already reduces distraction, and adversarial captions reduce it further. Our method also beats the fine-tuning objective of [1] on MoGround-Human (-58% vs -36% distraction), and their method comes at a bigger performance drop on MoGround-Retrieved (details in our response to reviewer pone W2 and App. E.7).

Takeaway: verified items and a conditional rate are what let MoGround show that the direction of distraction is model-dependent and predictable. The verified data is also what makes a simple recipe beat existing methods.

[1] Cai et al. (2025). Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models

[2] Yang et al. (2026). Defying Distractions in Multimodal Tasks: A Novel Benchmark for Large Vision-Language Models


W2. Artificial data, easy text items, small human set. The reviewer notes that MoGround-Base is LLM-written and may carry systematic biases, that its text questions are too easy, and that MoGround-Human is small. Two results show that LLM writing does not affect our conclusions:

  • If LLM writing affected our findings, they would change on data not written by an LLM. They do not. On MoGround-Retrieved, whose questions are not LLM-written, the weaker modality is still the more distracted one, distraction still falls as grounding rises, and the robustness vector still reduces distraction on all models. On MoGround-Human, used only for evaluation, our method still reduces the distraction rate on average by 47%.
  • LLM wording does not make items easy to resist distraction. Being easy to answer is not the same as being hard to distract. The distraction rate counts only answers the model already gets right without distraction, so high accuracy does not automatically mean low distraction. If LLM wording made items easy to resist distraction, models would rarely be distracted on MoGround-Base. Yet they lose 3–14% of their correct vision answers to distraction there (Table 1). Moreover, distraction depends on how confident the model is, not just on whether it answers correctly: the gap between its top two answers predicts which correct answers flip (Sec. 4.2).

We agree MoGround-Human is small. It is a first step toward a high-quality, human-written resource for modality distraction. Still, 125 items suffice to measure clear effects: v-distraction averages 27.4% there vs 7.6% on MoGround-Base, and the robustness vector's 47% reduction is statistically significant (Table 3). For difficulty at scale, MoGround-Retrieved adds 4,757 audited items with text-only accuracy of 66–96% (Table 2).

Takeaway: the findings do not depend on LLM-written data. They hold on the two pools no LLM wrote, and the small human-written set serves for confirmation.

AuthorsOpenReview ↗

Response to Reviewer aGxH (2/2)

W3a / Q2. Different questions for vision and text. The reviewer asks how we control for difficulty, since vision and text distraction are measured on different questions, so a difference in difficulty could produce the asymmetry rather than the models themselves.

Measuring distraction on both modalities on the same items is impossible by construction. Each item has its answer in only one modality, so a vision item can only test distraction by added text, and a text item can only test distraction by an added image. Our claim is per model and per dataset: for each model, on a given set of vision and text items, it is more distracted on whichever of the two it answers less accurately.

If the text items were universally easier, every model would be more distracted on vision. Three analyses show this is not the case. Let's define the correlation r: across the seven models, how well a model's accuracy gap between text and image predicts which modality distracts it more. An r near +1 means the modality a model answers less accurately is reliably the more distracted one. Difficulty could differ at two levels: one modality's items (e.g., text items) could be easier for every model, or individual items could differ in how hard they are (e.g., item X is harder than Y). We check both.

Modality-level difficulty:

  • Same items for every model. On MoGround-Retrieved (Table 2), all seven models answer the same 4757 items, yet the more-distracted modality differs: vision for 2 models, text for 4, and 1 tie. If text items were universally easier, every model would be more distracted on vision.

Sample-level difficulty (new in the rebuttal, App. E.2):

  1. Difficulty-matched subsets. We built 20 random subsets in which vision and text items are equally hard, hardness scored by how many of the other six models solve an item. On them, r = +0.88.
  2. A new split shared by all models. We built a split of the 1617 vision and 1318 text items that all seven models answer correctly, so every model has the same conditioning set. On it, r = +0.82.

Takeaway: the distraction asymmetry is not caused by one modality's questions being universally easier or harder. Even on the same questions, each model is more distracted in whichever modality it answers less accurately.


W3b. The models are relatively old. The reviewer notes that our open-weight models are relatively old. We ran the full study on three newer or larger models. All claims made on the original models still hold, and the robustness vector still reduces distraction (App. E.10, Table 21):

ModelRelative change in v-distraction, MoGround-Base test / Human / RetrievedGeneral capability
Qwen3-VL-32B−40 / −65 / −36%+0.4
InternVL3-14B−30 / −35 / −31%−0.4
Gemma-4-E4B−44 / −43 / −18%−0.3

Two of them, InternVL3-14B and Qwen3-VL-32B, are also larger than the original models (up to 8B), showing that our method also works on larger models.

All three also fall on the grounding relation fitted to the original seven models. The paper also showed that two frontier models in Sec. 4.1 (Gemini 3.5 Flash, Claude Sonnet 5) are less distracted than open models but not unaffected.

Takeaway: the findings and the method apply to newer and larger models.


W4. The findings are intuitive. The reviewer finds two findings expected: weaker grounding goes with more distraction, and answers with small margins are more likely to change. These are easy to state in retrospect, but the first was not settled:

  • The direction was disputed. One line of work on modality bias reports text-over-vision as the pattern [3][4], another reports both directions [1]. We show the direction is a property of the model and predict it from its grounding gap.
  • Lower accuracy does not mechanically mean more distraction. The rate is conditional on a correct answer under no distraction, so a weak model may solve fewer items, but distraction is still scored only on the items it solves. Neither model size (r=+0.09) nor accuracy in the distracting modality (r=-0.22) predicts distraction, while grounding strength does (r=-0.90).

We agree it is expected that small-margin answers are more likely to be flipped. So, we measured how much it matters: given an image and a question on it, even without adding any caption, the confidence of the model (i.e., margin) predicts which correct answers flip (Sec. 4.2).

[3] Lin et al. (2024) Revisiting the role of language priors in vision-language models

[4] Lee et al. (2025) Vlind-bench: Measuring language priors in large vision-language models

Takeaway: the grounding finding was not settled by previous work, and we measure how strongly the margin predicts distraction.


We hope these clarifications and new analyses address the reviewer concerns, and we are happy to discuss any remaining points.

AuthorsOpenReview ↗

Gentle Reminder

Thanks again for the time spent reviewing our paper. We hope our responses address your concerns about novelty, question difficulty, and model coverage. To make them quick to read, each answer opens with a summary of your concern and closes with a takeaway. We would be glad to hear your thoughts, and we are happy to clarify anything further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for their careful and constructive feedback. We are encouraged that they found the problem meaningful, our per-item verification of modality grounding valuable (aGxH, nVWb), the three evaluation pools well designed and the human-written set a useful contribution (pone), and the analysis extensive (aGxH, nVWb).


In summary, the new experiments show that question difficulty does not explain which modality is more distracted, that the findings hold on the two datapools not written by an LLM, that the robustness vector does not need word cues to learn, that the method’s gain comes from the verified data and the adversarial captions rather than from LoRA SFT alone, and that every claim replicates on three newer or larger models. We also describe related works Cai et al. [1] and Yang et al. [2] more accurately and compare directly with the fine-tuning method of [1].


What is new in this rebuttal, and where it is in the revised PDF

We have uploaded a revised PDF. Every change is shown in red. Below we list the edits:

  • We added three controls showing that question difficulty does not explain which modality is more distracted: the same items for every model, difficulty-matched subsets, and a split of the items all seven models answer correctly (aGxH W3a/Q2, pone W1). App. E.2, Table 12. Sec. 4.1 now states that the asymmetry claim is per model and per item set: each model is more distracted on whichever of its vision or text items it answers less accurately, which does not mean that it understands one modality better in general.
  • We replicated the main findings of the paper on the same controlled dataset that contains only items all models answer correctly under no distraction, so their conditioning set is the same for all. The model ranking by v-distraction on MoGround-Base and the more distracted modality on MoGround-Retrieved hardly change (nVWb Q3). App. E.2.
  • We added a comparison with [1] and [2] (aGxH W1/Q1, pone W2). Sec. 2, App. E.7, Table 19.
  • We ablated the mitigation: we compare against plain LoRA SFT on the same data, and our recipe without the adversarial captions. Both the verified data and the adversarial captions contribute to the gain (aGxH W1, nVWb W2/Q2). App. E.9, Table 20.
  • We tested modality distraction and the robustness vector on three newer or larger models (Qwen3-VL-32B, InternVL3-14B, Gemma-4-E4B), and on two sizes within each of two model families (aGxH W3b, pone minor). App. E.10, Tables 21-22.
  • We tested whether the word patterns of MoGround-Base transfer to the other pools or affect the mitigation's gain. They do not (nVWb W1/Q1). App. D.8, Table 11.
  • We tested whether lower image resolution raises distraction through weaker grounding or through other effects of image damage, and how consistent the grounding–distraction relation is within each model and each data source (nVWb Q4). App. E.3. Sec. 4.1 now states that on MoGround-Base, downscaling adds a little distraction beyond its effect on grounding.
  • Tables 1 and 2 now have matching columns and one takeaway each (pone W3).
  • The Limitations section now discusses the small size of MoGround-Human and the word patterns of MoGround-Base (aGxH W2, nVWb W1).
  • We restored the template's default citation formatting (nVWb Q5).
  • We trained the robustness vector only on items with few word cues, whose modality is hard to predict from their text. It still reduces v-distraction about as much as when trained on the same number of items with cues, and comes close to the original vector (nVWb W1/Q1). App. D.8, Table 12.

[1] Cai et al. (2025). Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models

[2] Yang et al. (2026). Defying Distractions in Multimodal Tasks: A Novel Benchmark for Large Vision-Language Models

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.