acceptodds
Under review as a conference paper at ICLR 2027

P-GVAR: Selective Visual Re-Grounding under Image-Text Grounding Drift

Abstract

Vision–language models (VLMs) can follow misleading textual context even when the image supports a different answer. We study a simple recovery mechanism: querying the same frozen VLM again after removing potentially conflicting context. This fresh visual query repairs many context-induced errors, but unconditional context removal is not generally safe because reliable context can also improve prediction. We therefore formulate recovery as a selective visual re-grounding problem with two distinct components: constructing a fresh visual candidate and deciding whether it should replace the original response. We introduce Policy-Guided Visual Answer Refresh (P-GVAR), a training-free response-level policy that uses controlled cross-condition responses to make this retain-or-refresh decision without task labels or access to model internals. Across three open VLMs and four stress benchmarks, selective re-grounding preserves most of the gain of unconditional visual reset, while controlled context experiments show that arbitration becomes important when context can be either helpful or misleading. These findings motivate evaluating candidate quality and routing quality separately when studying recovery from image–text grounding drift.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, multimodal reasoning, visual grounding, hallucination mitigation, robust inference, training-free methods, response-level arbitration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_6sdmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

VLMs can be distracted by the textual context that conflicts with the image and generate wrong response, even though the model can generate the correct answer when it is not given the context. Therefore, the authors propose P-GVAR, a training-free method, which selectively uses the response obtained with the input context removed. P-GVAR decodes several responses for different input conditions, and considers the context as not useful when the response with context and the response without context are different, and the response given the context with the image and without the image are the same.

Critical strengths

None

Critical weaknesses

1. The authors’ suggested contributions are not supported by their experiments, or some of them are well-known general knowledge that prior works have dealt with.

Observation in Section 3: Compared to the VLM’s performance in the context-free setting, higher performance with matching context and lower performance with corrupted context is how the evaluated benchmark (Words-or-Vision[1]) was already designed, and it is a generally known fact [2, 3].

Also P-GVAR selectively withholds only the unhelpful context, but the usefulness of this candidate selection is rather refuted by their own experiments. In Table 1, the proposed P-GVAR performs worse than not showing the context at all (B-only). Even in Table 4, where only helpful context is given (matching), P-GVAR performs similarly to B-only, and this contradicts the authors’ claim.

Also, comparisons with directly related works are lacking. Among the compared methods, none other than RVD deals with the situation where mismatched context is given. Prior training-free methods which deal with modality conflict check if the image and the context are matched by prompting [2], or more directly and almost similar to P-GVAR, solve the task using image input, then solve the task using the text input, and then inspect whether any inconsistencies between the outputs exist [3].

[1] Deng et al. Words or Vision: Do Vision-Language Models Have Blind Faith in Text?. CVPR 2025

[2] Zhang et al. Robust Multimodal Large Language Models Against Modality Conflict. ICML 2025

[3] Pezeshkpour et al. Mixed Signals: Decoding VLMs’ Reasoning and Underlying Bias in Vision-Language Conflict. EMNLP Findings 2025

2. The paper makes a precise understanding of the method and reproducibility impossible.

There is no concrete description or example of the prompts used. There is no explanation of the visual-applicability check used in the method. There is no explanation of how response validity is measured. There is no description of the main benchmarks. Also the authors use unintuitive and inconsistent terminology (visual reset: just the ordinary VQA setting without external context, re-grounding: this also seems to be just the ordinary VQA setting). The table captions do not follow the placement rule. And there are typos and unclear sentences, symbols used without definition or explanation, some contents described without referencing the related figure, and a lack of explanation for some results (What are Refresh, Recovered, Harmful, and Invalid in Table2? What is Rule in Table 1 and Table 7?)

Questions for authors

See the weakness.

Reviewer_nx5xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to address the image-text grounding shift problem in vision-language models, where the given text context can conflict with the image and lead to incorrect answers. To this end, the authors propose a selective re-grounding formulation with two steps: candidate construction for alternative answer generation and candidate selection for final answer prediction. Moreover, two training-free routing rules are proposed for candidate selection, namely Policy-Guided Visual Answer Refresh (P-GVAR) and P-GVAR-S. The controlled stress test and evaluation on MathVista show that selectively using supplemental context is helpful for improving accuracy.

Critical strengths

The paper shows that selective re-grounding can help discard misleading context while retaining helpful context, providing insights into the image-text grounding shift problem in VLMs.

Critical weaknesses

The reported results do not fully support the central contribution. The design goal of P-GVAR is to keep context when it is helpful and refresh the answer when context is misleading. However, Table 4 in Section 6.4 shows that P-GVAR does not necessarily outperform setting B (i.e., context is always discarded) in terms of averaged accuracy. This suggests that P-GVAR fails to reliably keep useful context, raising concerns regarding the validity of the proposed method.

Questions for authors
  • The formulation in Section 4 contains many undefined symbols and operations, making it difficult to understand the detailed operations. For example, PA/PB/PC/PDP_A/P_B/P_C/P_D, IDI_D, ss, and G(s)G(s) are not well-defined. In addition, there are inconsistent notations between the equations and Figure 1. Equations 6 and 7 use A≢BA \not\equiv B, but Figure 1 uses A≠BA \neq B.
  • Underspecified implementation details. While Appendix A provides the procedure of P-GVAR, it is still unclear how valid, some_target, equivalent, and visual_admissible are defined. This raises concerns regarding reproducibility.
  • Inference cost is discussed in Appendix E, but no inference time is provided. Since the proposed P-GVAR requires four controlled response branches to produce the final prediction, detailed computational cost should be given.
Reviewer_i7mVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies image–text grounding drift, where VLMs follow misleading supplementary text instead of the image. It observes that re-querying the same frozen VLM with the context removed fixes many such errors, but that removing context can also discard useful information. Recovery is therefore split into candidate construction (producing BB) and candidate selection (deciding whether BB should replace the context-conditioned answer. P-GVAR is a training-free, black-box rule: when AA and BB are valid and disagree, it queries the model text-only (C=Mθ(q,h)C=M_\theta(q,h)) and switches to BB if C≡AC\equiv A and a visual-applicability check G(s)G(s) passes; otherwise it keeps AA. P-GVAR-S adds an independently prompted visual query DD and switches when the ordinal score is positive. The paper have some experiments. However

I am concerned that this submission may have been produced largely by an automated research pipeline, beyond the AI use it discloses. Some evidence is as follows:

  • Unedited process notes in the body: “The complete matched-budget table is moved to Appendix D” (§6.5); “To keep the two trends directly comparable without consuming a full page, we place them side by side” (§6.4); “This distinction supports the first central claim of the paper” (§6.1), although no claims are ever enumerated; “highest executed value” (Table 1); “official-repository executions” (§5); “stored decision traces” (Table 11).
  • Run names that look like they come from an experiment log and are never defined: the baseline “Rule-GVAR”, “the original refresh trigger” F0F_0, G(s)G(s), TDT_D, and labels such as P0/P1, C-margin, Vote-B-4 and “Verify-AB cond.”
  • Benchmarks used without attribution (RH-Bench; VisualSwap from Shi et al., 2026), and a series of small, unspecified control experiments (Tables 5–6) that read like pre-emptive answers to anticipated reviewer objections.
  • The figures in the paper are extremely sloppy, and the "Related Work" section is overly simplistic; in contrast, the "Limitations and Future Work" section is quite extensive, organized into multiple bulleted paragraphs with the opening of each paragraph formatted in bold. This is unusual in normal paper.

I will keep my 1 score and suggest to desk reject this paper.

Critical strengths

None

Critical weaknesses

1.The selector does not solve the selection problem the paper poses.

  • It never reliably beats the better of the two fixed policies: always keeping the original answer, or always re-querying without context (B-only). Neither routing signal measures whether the context is reliable. The text-only check only shows that the original answer can be reproduced from the context; when the context is correct, it passes just as well, so a correct answer is replaced whenever the reset answer disagrees. The extra visual query in P-GVAR-S is another context-free query of the same model, so its agreement with the reset answer says little beyond prompt stability. In effect, the policy is “trust the image on conflict.”
  • The results confirm this. With helpful context on Words-or-Vision, the original answer beats the reset answer by 9–15 points, but P-GVAR-S recovers at most 2 of them, and none for InternVL3, where both variants trail B-only under every context condition. On the main suite, P-GVAR trails B-only in 9 of 12 cells and on average (49.1 vs. 50.2), and it is still behind in the compute-matched control despite using about 2.5 times as many calls. Its few wins over both fixed policies are within about 2 points, untested for significance, and matched on RHBench by simple validity handling.
  1. The evaluation cannot be audited or reproduced as reported, and its main gains are confounded.
  • Unattributed and undescribed data: RHBench appears to be RH-Bench (Liu et al., NeurIPS 2025), which is never cited. VisualSwap is the benchmark introduced by Shi et al. (2026), which the paper cites only in the introduction and related work, never as the dataset’s source. Beyond the Words-or-Vision context conditions, the paper never specifies what hh is for each benchmark (including MathVista), nor how the 500-example subsets were drawn.
  • Undefined method components: the baseline “Rule-GVAR” (Tables 1 and 7), G(s)G(s), TDT_D/same-target, validity VXV_X, answer equivalence e(⋅,⋅)e(\cdot,\cdot), and all prompts are never defined. The text also contains process notes that were apparently never edited out: “The complete matched-budget table is moved to Appendix D”; “without consuming a full page, we place them side by side”; and “the first central claim of the paper”, although no claims are ever enumerated. A reader cannot tell what was actually run, or whether hand-written rules encode benchmark-specific knowledge.
Questions for authors
  • Refer to the concerns regarding auto research in the summary.
  • Refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.