acceptodds
Withdrawn from ICLR 2027

Text-Conditioned Visual Steering for Hallucination Mitigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) are becoming a general interface for visual question answering, captioning and image-grounded reasoning. Their fluency makes them useful in practice, while also making unsupported objects and relations difficult to detect. We evaluate our method on POPE, CHAIR and complementary multimodal benchmarks. Recent inference-time methods use contrastive decoding, attention intervention, visual-token reinjection or latent steering. A representative class of visual-steering methods contrasts image-conditioned and text-only activations, extracting a reusable direction for generation. Yet this direction typically compresses the whole visual sequence into one global shift, leaving the text-dependent organization of visual evidence implicit. As a result, a correction that is useful for one question can be poorly aligned with another question about the same image, while methods that preserve token-level structure often add decoding-time computation. To address this gap, we introduce (Text-Conditioned Visual Steering), which combines global image–text contrast with text-conditioned visual-token routing. TVS uses the attention weights from the current text query to estimate which visual tokens should receive more or less emphasis, reweights them around a uniform reference, and pools their hidden states with signed zero-sum coefficients. The resulting correction is added to the inherited global direction, so the method remains training-free, preserves the steering interface and caches one direction for ordinary decoding. Across POPE, CHAIR and general multimodal evaluations, TVS improves the visual-steering baseline on Qwen and on LLaVA MSCOCO, with smaller but positive average gains on LLaVA A-OKVQA and GQA and variation across individual splits. Component ablations verify that the global and token-level terms are complementary, while efficiency experiments show that the cached implementation retains reference decoding accuracy with lower overhead. Code is available at https://anonymous.4open.science/status/TVS-E8F0.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Hallucination Mitigation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SZcKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Text-Conditioned Visual Steering (TVS), a training-free method for mitigating hallucinations in vision-language models. Building on the visual steering vector from VISTA, TVS augments the global image-versus-text-only direction with a prompt-conditioned correction derived from attention over visual tokens. The proposed correction uses zero-sum coefficients to emphasize relevant visual content. The resulting direction is cached after two prefills and applied during generation with one model forward per generated token. The paper evaluates LLaVA-1.5-7B and Qwen2.5-VL-7B on POPE, alongside caption hallucination and general capability evaluations. It also presents direction-component ablations and an efficiency analysis. However, discrepancies between the stated method, released implementation, and reported experimental results currently limit my confidence in the conclusions.

Critical strengths

A simple and potentially useful extension of visual steering. TVS introduces prompt-conditioned visual-token weighting into the existing VSV framework while retaining a cached steering direction. This is a plausible way to improve the specificity of visual guidance without introducing an additional decoding branch. The relationship to the steering component of VISTA is explicitly acknowledged.

Useful controls for interpreting hallucination reductions. The direction-component experiments include spatial-only, shuffled, and norm-matched uncentered controls. The captioning experiments also report length and object recall, helping assess whether lower hallucination rates arise from reduced output coverage. These controls address meaningful alternative explanations for the observed improvements.

Critical weaknesses

The released implementation does not match the central zero-sum construction. Equation (5) defines gi=αsi+(1−α)uig_i=\alpha s_i+(1-\alpha)u_i, ensuring ∑igi=1\sum_i g_i=1 and consequently ∑i(gi−ai(ℓ))=0\sum_i(g_i-a_i^{(\ell)})=0. However, qwen_compute_vsv_att in the released Qwen evaluation code implements alpha * saliency + max(0.0, 1.0 - alpha) * uniform. At the reported default α=2\alpha=2, this gives gi=2sig_i=2s_i, so the correction coefficients sum to 11, rather than 00. This changes the method and removes the stated shared-offset cancellation property. The batched evaluation script calls this function, making the discrepancy directly relevant to the experimental pipeline. Additionally, essential steering routines are imported from a local pade package that is absent from the released repository. It is therefore unclear whether the reported experiments validate the method described in the paper.

The released experimental records do not support several reported LLaVA results. Table 7 and the repository’s three-dataset results summary give substantially different mean accuracies:

DatasetMethodTable 7Released records
A-OKVQAVSV82.2182.54
A-OKVQATVS87.9182.24
GQAVSV83.3382.67
GQATVS83.7881.34

The launcher identifies vista as the global-direction baseline and psv as the attention-conditioned variant, with the latter using the stated λ=0.17\lambda=0.17, α=2\alpha=2, final-layer attention, and object-token query. Thus, the released records indicate decreases on both datasets, whereas Table 7 reports improvements. Different experiment versions could explain these discrepancies, but no sufficient mapping is provided. This materially affects the claimed generality of the method’s benefits.

Limited methodological novelty beyond VISTA’s visual steering component. TVS inherits the image-versus-text-only direction extraction, cached steering framework, and decoding-time intervention from VSV. Its main addition is an attention-weighted correction to the existing direction. This is a reasonable extension, but the current evidence does not establish a sufficiently substantial advance: in the matched component study, TVS improves mean accuracy over VSV by only 0.300.30 percentage points, while centering contributes just 0.060.06 points over the norm-matched uncentered variant. Although the paper appropriately acknowledges the limited evidence for centering, the incremental construction and modest gains in this controlled comparison weaken the case for a significant methodological contribution.

Questions for authors

Clarification of the evaluated algorithm. Which implementation generated the reported Qwen results: the signed weighting in Equation (5), or the clipped expression in the released code? Please identify the corresponding code version and provide the missing steering dependency. If the clipped expression was used, does the improvement persist with the stated zero-sum formulation? A representative matched comparison would be sufficient.

Clarification of experimental provenance. Please reconcile the Table 7 results with the released LLaVA three-dataset records, particularly the reversed TVS–VSV comparisons on A-OKVQA and GQA. Which configurations and artifacts reproduce the table, and do any aggregate results or conclusions require correction?

Limited additional analysis. In the matched component study, TVS improves mean accuracy over VSV by approximately 0.300.30 percentage points and over the norm-matched uncentered variant by only 0.060.06 points. Could you provide a paired uncertainty estimate using the existing predictions, preferably accounting for questions sharing an image? This would help determine how much evidence supports the practical benefit of the added correction.

Reviewer_7RG3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a text-conditioned visual steering method to mitigate visually ungrounded hallucinations by encouraging vision-language models (VLMs) to focus more on image regions relevant to the given question.

The method first constructs a global visual steering vector by computing the difference between the last-token hidden states obtained with and without the image. It then uses the model's attention distribution over image patches as a text-dependent routing signal, rather than treating attention as a ground-truth measure of semantic relevance. By comparing this attention distribution with a uniform reference distribution, the method constructs an additional steering vector that increases the influence of query-relevant visual representations. This encourages the model to rely more on visual evidence when generating responses.

Critical strengths

Simple and intuitive methodology: The proposed method is relatively simple and intuitive. By leveraging the model's existing attention distributions, it constructs a query-dependent visual steering signal without requiring an additional relevance prediction model.

Comprehensive experimental evaluation: The paper provides comprehensive experimental results supporting the effectiveness of the proposed approach in reducing visually ungrounded hallucinations.

Critical weaknesses

Missing relevant recent work: The paper overlooks two relevant studies published in 2026: (a) Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation and (b) Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models. Both explore adaptive activation steering for hallucination mitigation in VLMs. The authors should discuss how the proposed method differs from these approaches, particularly in its steering mechanism, and clarify its methodological novelty and potential advantages. Comparisons with these methods would also help establish the contribution of the current work.

Questions for authors

NA

Reviewer_KLoVOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Text-Conditioned Visual Steering (TVS), a training-free method for mitigating hallucinations in MLLMs. It augments a global visual steering vector with an attention-derived, query-conditioned correction over visual tokens. The combined direction is cached before generation. Experiments on LLaVA-1.5 and Qwen2.5-VL report improvements over the VSV baseline.

Critical strengths

The method is lightweight and easy to integrate into the existing steering framework. It adds token-level information to the steering direction while retaining one model forward per decoding step, making the refinement potentially practical for frozen models.

Critical weaknesses
  1. The motivation for query-conditioned spatial correction is not directly validated. Shuffling visual tokens tests spatial alignment, but does not isolate the role of question-specific attention. A useful control would keep the original question unchanged while using attention from a different question about the same image.

  2. Broad comparisons with competing methods are largely restricted to POPE. CHAIR and the general capability benchmarks mainly compare Vanilla, VSV, and TVS, leaving the method's competitiveness in open-ended generation unclear. The absence of full VISTA, rather than just its VSV component with self-logit augmentation disabled, is also a notable omission.

  3. The engagement with recent work feels incomplete: all cited papers are from 2025 or earlier. For an ICLR 2027 submission, I would expect a more up-to-date discussion and comparison with recent hallucination-mitigation methods. The current coverage makes the positioning of the contribution less convincing.

Questions for authors

Qwen2.5-VL's POPE performance is unexpectedly low, even falling below LLaVA-1.5. I am concerned that a suboptimal baseline configuration could affect the interpretation of the reported gains.

Related works have reported Qwen2.5-VL's performance: [1] Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats. NIPS25 [2] Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidance. CVPR26

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.