acceptodds
Under review as a conference paper at ICLR 2027

Grounding on Request: Learning Modality Control in Vision-Language Models

Abstract

When an image and its caption disagree, can a user reliably control which one a vision-language model (VLM) follows? Accuracy alone cannot answer this question, since a model might give the correct answer from parametric knowledge without following the requested evidence. We therefore evaluate modality grounding control through conflict-preserving counterfactual interventions. Specifically, Flip measures whether an initially correct answer follows an edit to the requested modality, while Stay measures whether it remains stable when the competing modality is edited. To learn this control, we build on neologism learning by introducing new words for visual and textual grounding, training only their embeddings while keeping the pretrained model fixed. We also introduce Counterfactual Twin-Margin (CTM) supervision, which uses paired interventions to teach how answer probabilities should change when the requested evidence changes. Across three VLMs on three existing benchmarks and our multi-attribute Ground-CF benchmark, we find that learned modality words improve control over natural-language instructions. Surprisingly to us, a single learned embedding per modality provides strong lightweight visual control and remains stable on held-out attribute values. LoRA remains strongest overall, but its visual advantage narrows on the held-out evaluations. Moreover, counterfactual supervision improves control in several settings where standard supervision is weaker. Finally, we test our method on real-world news video, and find that visual control improves when the relevant evidence is made easier to access. Overall, our results show that reliable modality control requires testing not only whether an answer is correct, but whether it continues to follow edits to the modality of choice.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, multimodal grounding, modality control, neologism learning, counterfactual evaluation, parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ggAVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether a user can choose which evidence source a VLM follows when image and text disagree. It proposes a conflict-preserving counterfactual protocol: FLIP requires correct answers both before and after changing the requested source, while STAY requires correct answers before and after changing only the competing source. These are joint correctness measures, not merely answer-change or answer-agreement rates.

To learn the control, the authors add modality-specific vocabulary entries and optimize only their input embeddings in a frozen model. The counterfactual twin-margin objective couples the likelihood of the same answer across original and edited contexts, encouraging the old answer to become less likely and the replacement to become more likely after a requested-source edit. Main comparisons use SFT+CTM across learned methods.

Experiments evaluate three VLMs on four controlled benchmarks, held-out values, objective and placement controls, and a separate news-frame application with newly trained checkpoints.

Critical strengths

1. Counterfactual evaluation of modality control.

The evaluation distinguishes answering correctly from following the requested evidence source. FLIP tests whether predictions change correctly when the requested source changes, while STAY tests whether predictions remain correct when only the competing source changes. Requiring correctness on both members of each pair provides a more informative measure of source dependence than accuracy on isolated examples. This extends the analysis of cross-modal interference studied in prior work. [1]

2. Parameter-efficient control of modality preference.

The method adapts modality preference by optimizing a small number of input embeddings while keeping the backbone frozen. The improvements over English instructions across the evaluated models support the effectiveness of this approach within the tested settings. Its application of neologism learning to user-directed evidence selection provides a compact interface for controlling which modality determines the answer. [2]

References

[1] Tianze Hua, Tian Yun, and Ellie Pavlick. How Do Vision-Language Models Process Conflicting Information Across Modalities? 2025. https://arxiv.org/abs/2507.01790. Project page: https://vlm-conflicting-information-processing.github.io/

[2] John Hewitt, Oyvind Tafjord, Robert Geirhos, and Been Kim. Neologism Learning for Controllability and Self-Verbalization. 2025. https://arxiv.org/abs/2510.08506

Critical weaknesses

1. Insufficient justification for retaining conflicting evidence.

The formulation requires both modalities to remain available after the user selects the trusted source, but the necessity of this constraint is not established for the evaluated tasks. In the image–caption settings, the unrequested evidence can be removed before inference. Prior work establishes cross-modal interference [1]; the remaining motivation question is when retaining both sources is necessary and learned control offers an advantage over input selection.

2. Missing input-filtering baseline.

The reported comparisons lack a training-free baseline that retains the requested evidence and removes the competing source, while preserving the question, instructions, and answer candidates. A target-modality-only comparison also has precedent in related work. [1] Evaluating this baseline with the same backbone and splits is the most direct test of the method's practical value: comparable performance would weaken the case for learned control, whereas a clear improvement would substantiate its benefit. Accuracy and requested-source FLIP are particularly relevant, since filtering removes interference by construction.

3. Limited evaluation on complex tasks.

The main benchmarks use controlled, relatively synthetic conflicts. The news-frame experiment broadens the evaluation, but uses separately trained checkpoints and includes human-annotated crops. Evaluation on more complex benchmarks, such as A-OKVQA or MMMU, with manually introduced and verified errors in accompanying textual evidence would better test whether the method remains effective beyond simple conflicts. The resulting examples should retain a well-defined answer for each requested source. [3, 4]

References

[1] Tianze Hua, Tian Yun, and Ellie Pavlick. How Do Vision-Language Models Process Conflicting Information Across Modalities? 2025. https://arxiv.org/abs/2507.01790. Project page: https://vlm-conflicting-information-processing.github.io/

[3] Dustin Schwenk et al. A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge. 2022. https://arxiv.org/abs/2206.01718

[4] Xiang Yue et al. MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. CVPR 2024. https://arxiv.org/abs/2311.16502

Questions for authors
  1. How does learned control compare with requested-modality input filtering under the same evaluation protocol?

  2. Which applications require retaining both conflicting sources after the user selects the trusted modality?

  3. Does the method remain effective on more complex benchmark examples with manually introduced evidence errors?

1 comment
AuthorsOpenReview ↗

Following-up Response

Dear reviewer ggAV,

We have provided detailed responses and conducted all three additional experiments you requested, addressing (1) the comparison with an input-filtering baseline, (2) the motivation for retaining both modalities through compositional requests, and (3) generalization to more complex natural-image questions using A-OKVQA-CF.

We would greatly appreciate it if you could let us know whether these new results resolve your concerns.

Thank you for your time and consideration.

Best regards,

The authors

Reviewer_vxTvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Flip and Stay counterfactual metrics for whether a VLM follows a request to use visual or textual evidence. It learns modality tokens such as <look> and <read> via neologism learning and introduces a Counterfactual Twin-Margin (CTM) objective. Across VCF, ColorSwap, COCO-CF, and the new Ground-CF benchmarks, prompt tuning, neologism learning, and LoRA substantially improve visual Flip/Stay behavior over the base models, with some out-of-distribution retention. A MultiVENT-G news-frame study shows that full-frame visual requests remain difficult and that crop-based evidence helps, while the authors acknowledge intervention confounds.

Critical strengths
  1. This work propose a useful counterfactual evaluation design that reveals behavior missed by single-input accuracy. FLIP and STAY separately test whether predictions track requested evidence and resist changes to competing evidence. Preserving image–text conflict after intervention avoids attributing success merely to the removal of disagreement.
  2. Base on the observation, a parameter-efficient control is introduced with relevant experimental ablations. A single learned embedding per modality substantially improves control over natural-language instructions in the controlled evaluations. Paired-SFT, placement, and explicit-STAY ablations help examine the respective effects of counterfactual exposure, coupled supervision, and interface design.
Critical weaknesses
  1. The distinct advantage of neologisms over prompt tuning is not sufficiently established. Both methods learn input embeddings while freezing the backbone. The main comparison changes both placement and the surrounding natural-language carrier. The placement ablation is useful, but the paper should identify what computational difference remains when initialization, carrier, and position are fully matched. This is particularly important for MultiVENT-G, where the methods already use a shared end-of-prompt carrier.
  2. The training and held-out protocols are insufficiently specified to assess the generalization claims.The appendix provides evaluation counts and optimization settings, but does not clearly specify training-set sizes, whether controls are trained jointly across benchmarks, or how source images and their counterfactual variants are grouped during splitting.
Questions for authors
  1. What implementation difference remains between neologisms and prompt tuning when position, carrier, initialization, and training examples are identical? Please report multiple training seeds and investigate whether the Gemma-3-4B/COCO-CF collapse reproduces consistently.
  2. Please specify training-set sizes, data mixtures, checkpoint sharing, and splitting rules for source images and counterfactual families. Can held-out attribute values occur in training text or counterfactual variants?
  3. Can you add paired SFT for Qwen3.5-4B, which exhibits the largest CTM improvement but lacks this control? Also, what explains cases such as Qwen3.5-2B, where held-out visual FLIP decreases from 0.645 with SFT to 0.619 with CTM?
1 comment
AuthorsOpenReview ↗

Following-up Response

Dear reviewer vxTv,

We have provided detailed responses and additional experiments addressing your concerns about (1) the distinction between neologisms and prompt tuning, including zero-shot composition and multi-seed results, (2) training and held-out protocols, and (3) the contribution of CTM compared with paired SFT across all benchmarks.

We would greatly appreciate hearing whether these additional results and clarifications resolve your concerns.

Thank you for your time and consideration.

Best regards,

The authors

Reviewer_SiibOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how users can control which modality a VLM relies on when image and text evidence conflict. It introduces FLIP, measuring whether the model changes its answer when the requested modality is altered, and STAY, measuring whether it preserves its answer when the competing modality is altered. Averaged across three models and four controlled benchmarks, visual requests with the learned tokens achieve ID FLIP/STAY of 0.664/0.733 and held-out scores of 0.657/0.717. Natural-language instructions achieve 0.367/0.455. LoRA is stronger overall, but visual FLIP decreases from 0.800 in ID evaluation to 0.706 on held-out data. On MultiVENT-G, full-frame visual FLIP is low; cropping to a human-annotated evidence region improves it. The paper notes that this crop evaluation does not test whether the model can locate the evidence by itself.

Critical strengths
  1. The evaluation is more directly aligned with modality following than accuracy alone. FLIP and STAY separately test whether a model changes or preserves its answer under the relevant counterfactual intervention.
  2. The training objective is linked to counterfactual evaluation. CTM explicitly compares candidate-answer scores before and after interventions, and paired-SFT controls help distinguish the objective from merely observing counterfactual examples.
  3. The comparisons cover multiple models, benchmarks, and baselines. The paper includes ID and held-out evaluations, natural-language instructions, prompt tuning, and LoRA, and reports the parameter counts of the learned-token method.
Critical weaknesses
  1. The meaning of the learned-token interface is not fully interpretable. The model can behaviorally distinguish <look> from <read>, but has difficulty accurately describing which modality each token refers to. This limits claims about an interpretable control interface.
  2. The held-out generalization setting is narrow. It mainly tests unseen attribute values or concepts within related tasks and does not establish broad out-of-distribution generalization. On Ground-CF, held-out visual FLIP also falls from 0.725 ID to 0.608.
  3. The real-world evaluation is preliminary. MultiVENT-G contains 173 visual-swap pairs from 22 videos, and methods are trained separately in that setting. Full-frame FLIP is low, while human-box evidence crops do not test whether models can locate relevant regions independently.
Questions for authors
  1. Please expand transfer tests across datasets, tasks, and model sizes to assess whether the learned-token interface supports compositional generalization.
  2. Please evaluate evidence localization without human-provided boxes, separating perception, evidence retrieval, and modality control.
  3. Please analyze how CTM can jointly optimize FLIP and STAY; the Stay-term ablation suggests a potential trade-off between the two.
1 comment
AuthorsOpenReview ↗

Following-up Response

Dear reviewer Siib,

We have provided detailed responses and additional experiments addressing your concerns about (1) held-out generalization and cross-dataset transfer, (2) compositional modality control, (3) evidence localization, and (4) the relationship between Flip and Stay under CTM supervision.

We would greatly appreciate it if you could let us know whether these answers resolve your concerns.

Thank you for your time and consideration.

Best regards,

The authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

Summary of the revised paper

Dear reviewers and Area Chair,

Thank you all for your time and for the constructive feedback! Your reviews have made the paper considerably stronger, and we have now uploaded the revised version, with all changes marked in blue. Hereby a short summary of what we changed and where to find it.

Contributions. Following your feedback, the Introduction now states our contributions explicitly: (1) Flip and Stay, which measure whether a model follows the requested modality under conflict-preserving edits, something accuracy alone cannot show, (2) CTM supervision, which teaches how answer preferences should change when the requested modality changes, and (3) neologism learning as a lightweight, user-selectable control for frozen VLMs, to our knowledge its first application to visual and textual grounding.

Reviewer vxTv

  • Neologisms versus prompt tuning. The new Section 5.3 ("Why Neologisms?") states that the two methods are computationally equivalent when fully matched, and shows what the difference in use gives us. Two independently trained neologisms can be composed zero-shot, answering both questions correctly on .45/.35/.17 versus .12/.18/.15 for prompt tuning (chance .11; Figure 5, Appendix B.1).
  • Seeds and the Gemma-3-4B/COCO-CF collapse. The collapse reproduces over three seeds, where prompt tuning's best seed is below the neologism's worst (Figure 6, Table 11).
  • Protocols. Training sizes, split units, and held-out rules are now given in Section 4.1 and Table 5.
  • Paired SFT. We trained paired SFT for all models and benchmarks with three seeds. CTM achieves higher visual Flip in 18 of 24 comparisons and higher Stay in 17 of 24, with average visual Flip increasing from .614 to .655 (ID) and from .614 to .652 (OOD). We also report where CTM does not help (Section 5.2, Table 18, Appendix D.4).
  • Ethics. We added an Ethics Statement.

Reviewer ggAV

  • Filtering baseline. Added to Table 2 and Appendix A.1 (Table 6). Filtering is a strong reference indeed, yet neologisms outperform it while still seeing both conflicting sources (visual Flip .664 versus .609).
  • Why retain both sources. Sections 5.2 and 5.3 now motivate this. Separation is not always straightforward, and different questions within one prompt may require different modalities, which our composition experiment tests directly.
  • More complex tasks. The new A-OKVQA-CF evaluation on 100 hand-verified items gives .75/.82/.73 image accuracy for neologisms, versus .46/.45/.24 for the instruction and .68/.79/.63 for filtering (Section 6, Appendix F, Table 22).

Reviewer Siib

  • Interpretability. We now separate behavioral control from self-verbalization and moved the probes to Appendix G, since our claims concern the former.
  • Held-out setting. The Ground-CF decline is concentrated in counting, where the frozen model given the image alone also falls (Section 5.2, Appendix D.3, Table 16, Figure 8).
  • Transfer and composition. Next to the composition experiment, we added cross-dataset transfer (Figure 7, Appendix D.2, Table 15). Neologisms transfer between related tasks, while transfer between unrelated ones remains weaker, which we state in the Limitations.
  • Localization. Section 6.1 and the Limitations now clarify that the crop shows that evidence access limits visual control, and does not test whether the model can find the evidence itself.
  • Flip versus Stay. An explicit Stay term reduces Flip in seven of eight comparisons, and as such we retain CTM without it (Section 5.2, Appendix D.5, Table 19).

What remains open. Evidence localization without human boxes, larger models, and open-ended answers were not feasible within this period, and we now state them clearly as limitations.

We believe that the revised paper addresses the concerns raised in all three reviews, and we would still be very happy to hear whether this is the case for you as well. Thank you again for helping us improve this work!

Best regards,

The authors

AuthorsOpenReview ↗

Official Comment by Authors

Dear reviewer Siib,

Thank you for your reply, and for taking the time to read our response! We would like to clarify two points about the interventions, since they are important for interpreting the results.

The caption is edited as well. Every benchmark includes interventions that change the caption while keeping the image fixed (Table 4). These measure caption Flip (.917 for neologisms, Table 2) and Stay under an image request, where the answer should remain with the image despite the caption changing. The question and candidate answers remain fixed within each counterfactual pair, but the correct answer changes when the requested evidence changes. Keeping the question fixed therefore allows us to test whether the answer follows the edited evidence.

Detecting that an image was modified is not enough for Flip. Each context is scored on its own, and Flip requires the correct answer on both the original and the edited image, chosen from the same fixed candidates. Since the edited value differs from both the original value and the value in the caption, knowing that an image was modified does not tell the model which answer is correct. Moreover, the sources still conflict after the edit, so falling back to the caption remains possible and fails the test. Looking at the edited images across VCF, ColorSwap, and COCO-CF and all three models, instructions lead to the caption's value in 44% of cases, compared with 20% for neologisms. Thus, the improvement concerns which evidence the answer follows, beyond detecting a modification. On held-out evaluations, visual Flip remains .657, compared with .664 ID (Table 2).

The improvement also occurs with unedited photographs. In A-OKVQA-CF, the images are original photographs and only the accompanying rationale is made misleading. Neologisms improve image accuracy from .46 / .45 / .24 with instructions to .75 / .82 / .73 (Table 22). This improvement cannot be explained by detecting an image modification.

We agree that broader generalization remains an important question. The revised paper adds cross-dataset transfer (Table 15), A-OKVQA-CF (Table 22), and real news-frame evaluation (Section 6.1), while stating the limits of each. We hope these clarifications help distinguish that remaining question from the modification-detection concern.

Thank you again for your time!

Best regards,

The authors

Thank you for your effort to furher clear my concerns. I will keep my positive score.

AuthorsOpenReview ↗

Official Comment by Authors

Dear reviewer ggAV,

Thank you for your reply, and for the careful reading of the new experiments! We're glad to hear that the filtering baseline “provides an important practical comparison” and that the additional experiments strengthen the paper. We would like to clarify how these results address your concern about effectiveness beyond simple conflicts.

Size and complexity of A-OKVQA-CF. Following your suggestion, we evaluated natural-image questions from A-OKVQA with manually introduced misleading textual evidence and verified all 100 evaluation items. Neologisms improve image accuracy from .46 / .45 / .24 with instructions to .75 / .82 / .73 across the three models: gains of 29 / 37 / 49 percentage points (p < 1e-5 for each). They also meet or exceed filtering on all three models, despite retaining the misleading rationale (Table 22). We would also like to distinguish the answer format from the complexity of the task. Evaluating the final answer against fixed candidates does not remove the visual reasoning or world knowledge needed to select it. Our controlled benchmarks deliberately isolate modality following, while A-OKVQA-CF tests natural-image questions with misleading rationales and MultiVENT-G tests visual counterfactual behavior in complex news frames. These evaluations serve different purposes, and together show that the benefit is not restricted to simple attribute conflicts. Broader reasoning evaluations would further characterize the scope of the method, but the contribution already demonstrated is substantial: a single learned embedding can improve which evidence a frozen VLM follows despite persistent conflicting information.

Visual Flip and real-world inputs. We already evaluate visual Flip on complex real-world inputs through MultiVENT-G, using frames from real news videos where relevant evidence may occupy only a small part of a cluttered scene (Section 6.1, Table 3). A-OKVQA-CF adds a complementary test of natural-image question answering under misleading rationales. Since it lacks image counterfactual twins, we report image Accuracy and Stay there. Thus, our real-world evidence includes both visual counterfactual evaluation on news frames and question answering on unedited photographs. Full-frame MultiVENT-G performance remains challenging, and we explicitly distinguish the human-annotated crop evaluation from autonomous localization.

Prompt tuning. Prompt tuning performs similarly on A-OKVQA-CF indeed, which supports the effectiveness of lightweight learned modality control in this setting. Our contribution does not require neologisms to outperform prompt tuning on every task. Their use within individual requests also allows us to test independently learned controls together: without composition training, neologisms answer both questions correctly in .45 / .35 / .17 of prompts, versus .12 / .18 / .15 for the evaluated fixed-prefix prompt-tuning baseline (Section 5.3, Figure 5). This directly addresses the practical setting in which both sources must remain available because different questions request different modalities.

CTM. We assess CTM's contribution beyond counterfactual exposure on the four controlled benchmarks. Across three models, SFT+CTM achieves higher visual Flip than paired SFT in 18 of 24 ID/OOD comparisons, averaged over three seeds. Mean visual Flip increases from .614 to .655 ID and from .614 to .652 on held-out evaluations (Section 5.2, Table 18), with exceptions reported explicitly. A-OKVQA-CF supports the applicability of learned controls to natural-image questions; the paired-SFT comparison supports CTM's additional benefit. We do not separately establish CTM's contribution on A-OKVQA-CF.

The central result is that a single learned embedding per modality can substantially change which evidence a frozen VLM follows, even when the competing source continues to support a conflicting answer. Across the controlled benchmarks, these embeddings recover 69% of LoRA's visual Flip improvement over instructions in ID evaluation and 84% on held-out evaluations, using 5,300–11,600× fewer trainable parameters. Together with the filtering comparison, zero-shot composition, and natural-image experiment, this provides evidence for both the effectiveness and practical value of learned modality control. While generalization across a wider range of reasoning tasks remains open, we believe these findings constitute a substantive contribution. We would be grateful if you could reconsider your rating in light of these results.

Thank you again for your time and constructive suggestions!

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.