- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether a user can choose which evidence source a VLM follows when image and text disagree. It proposes a conflict-preserving counterfactual protocol: FLIP requires correct answers both before and after changing the requested source, while STAY requires correct answers before and after changing only the competing source. These are joint correctness measures, not merely answer-change or answer-agreement rates.
To learn the control, the authors add modality-specific vocabulary entries and optimize only their input embeddings in a frozen model. The counterfactual twin-margin objective couples the likelihood of the same answer across original and edited contexts, encouraging the old answer to become less likely and the replacement to become more likely after a requested-source edit. Main comparisons use SFT+CTM across learned methods.
Experiments evaluate three VLMs on four controlled benchmarks, held-out values, objective and placement controls, and a separate news-frame application with newly trained checkpoints.
Critical strengths
1. Counterfactual evaluation of modality control.
The evaluation distinguishes answering correctly from following the requested evidence source. FLIP tests whether predictions change correctly when the requested source changes, while STAY tests whether predictions remain correct when only the competing source changes. Requiring correctness on both members of each pair provides a more informative measure of source dependence than accuracy on isolated examples. This extends the analysis of cross-modal interference studied in prior work. [1]
2. Parameter-efficient control of modality preference.
The method adapts modality preference by optimizing a small number of input embeddings while keeping the backbone frozen. The improvements over English instructions across the evaluated models support the effectiveness of this approach within the tested settings. Its application of neologism learning to user-directed evidence selection provides a compact interface for controlling which modality determines the answer. [2]
References
[1] Tianze Hua, Tian Yun, and Ellie Pavlick. How Do Vision-Language Models Process Conflicting Information Across Modalities? 2025. https://arxiv.org/abs/2507.01790. Project page: https://vlm-conflicting-information-processing.github.io/
[2] John Hewitt, Oyvind Tafjord, Robert Geirhos, and Been Kim. Neologism Learning for Controllability and Self-Verbalization. 2025. https://arxiv.org/abs/2510.08506
Critical weaknesses
1. Insufficient justification for retaining conflicting evidence.
The formulation requires both modalities to remain available after the user selects the trusted source, but the necessity of this constraint is not established for the evaluated tasks. In the image–caption settings, the unrequested evidence can be removed before inference. Prior work establishes cross-modal interference [1]; the remaining motivation question is when retaining both sources is necessary and learned control offers an advantage over input selection.
2. Missing input-filtering baseline.
The reported comparisons lack a training-free baseline that retains the requested evidence and removes the competing source, while preserving the question, instructions, and answer candidates. A target-modality-only comparison also has precedent in related work. [1] Evaluating this baseline with the same backbone and splits is the most direct test of the method's practical value: comparable performance would weaken the case for learned control, whereas a clear improvement would substantiate its benefit. Accuracy and requested-source FLIP are particularly relevant, since filtering removes interference by construction.
3. Limited evaluation on complex tasks.
The main benchmarks use controlled, relatively synthetic conflicts. The news-frame experiment broadens the evaluation, but uses separately trained checkpoints and includes human-annotated crops. Evaluation on more complex benchmarks, such as A-OKVQA or MMMU, with manually introduced and verified errors in accompanying textual evidence would better test whether the method remains effective beyond simple conflicts. The resulting examples should retain a well-defined answer for each requested source. [3, 4]
References
[1] Tianze Hua, Tian Yun, and Ellie Pavlick. How Do Vision-Language Models Process Conflicting Information Across Modalities? 2025. https://arxiv.org/abs/2507.01790. Project page: https://vlm-conflicting-information-processing.github.io/
[3] Dustin Schwenk et al. A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge. 2022. https://arxiv.org/abs/2206.01718
[4] Xiang Yue et al. MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. CVPR 2024. https://arxiv.org/abs/2311.16502
Questions for authors
-
How does learned control compare with requested-modality input filtering under the same evaluation protocol?
-
Which applications require retaining both conflicting sources after the user selects the trusted modality?
-
Does the method remain effective on more complex benchmark examples with manually introduced evidence errors?
Following-up Response
Dear reviewer ggAV,
We have provided detailed responses and conducted all three additional experiments you requested, addressing (1) the comparison with an input-filtering baseline, (2) the motivation for retaining both modalities through compositional requests, and (3) generalization to more complex natural-image questions using A-OKVQA-CF.
We would greatly appreciate it if you could let us know whether these new results resolve your concerns.
Thank you for your time and consideration.
Best regards,
The authors