acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Steering of Layer-wise Uncertainty to Mitigate Hallucination in LVLMs

Abstract

Large vision-language models (LVLMs) suffer from diverse hallucinations, such as errors in object presence, spatial reasoning, counting, and text recognition. Existing inference-time mitigation methods largely focus on object hallucination and often apply coarse or uniform interventions, overlooking variations in hallucination risk across model depth. In this paper, we propose a dynamic steering method based on uncertainty that adaptively controls intervention strength at each decoder layer. Our analysis reveals that hallucination patterns vary substantially across both decoder layers and models, and no single attention-derived signal consistently captures hallucination risk. To overcome this variability, we integrate multiple complementary attention-derived signals to estimate layer-wise hallucination uncertainty as a dynamic control signal for steering. It applies stronger interventions to high-risk layers while preserving reliable representations, enabling adaptive steering across both decoder layers and generation steps without additional training. Experiments on multiple LVLMs across four hallucination benchmarks show that our method consistently improves performance across various hallucination tasks over existing inference-time baselines, achieving an average gain of 5%p and demonstrating that attention-derived layer-wise uncertainty provides an effective control signal for dynamically steering LVLMs against hallucinations beyond object-centric settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Hallucination

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yM4ROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DUST, a training-free method for mitigating hallucinations in large vision-language models. DUST extracts layer-specific visual steering directions through PCA on image-conditioned versus no-image activations. It combines attention entropy, image attention, question attention, and concentration using fuzzy patterns and a Choquet integral, then uses the resulting score to dynamically steer MLP outputs while preserving their norms. Experiments report improvements across multiple LVLMs and evaluations covering object existence, spatial relations, text, color, and counting.

Critical strengths
  1. The paper addresses hallucinations beyond object existence and evaluates several visual capabilities. This broader evaluation is valuable because methods that improve object-centric benchmarks may degrade spatial reasoning or text recognition. The reported results suggest that DUST offers a more balanced intervention across these tasks.

  2. The method provides a practical way to adapt steering across both layers and generation steps without updating backbone parameters. Ablations show that uncertainty-aware steering improves over fixed PCA steering, particularly on VizWiz, while experiments across four backbones support its potential applicability.

Critical weaknesses
  1. The interpretation of the final gate as hallucination uncertainty is insufficiently validated. The probing analysis reports AUROC from separately fitted logistic classifiers, which does not directly establish the predictive validity or directionality of the handcrafted DUST score. Moreover, the symmetric Choquet operator reduces to an ordered weighted average, so its advantage does not itself demonstrate meaningful pattern-specific interactions. Direct evaluation of the combined score and comparisons with strength-matched constant or shuffled gates are needed to establish the benefit of risk-aware intervention.

  2. Experimental inconsistencies and missing protocol details limit confidence in the comparisons. For example, Table 5 reports LLaVA VCD Random precision/recall of 91.42/72.80, which imply approximately 82.98% accuracy under the stated balanced evaluation, rather than 87.73%. ICD Random has a similar discrepancy. The paper also leaves unclear whether the 50 calibration instances are excluded from evaluation and how baseline hyperparameters and decoding settings are selected. These issues matter for assessing both reproducibility and the reported improvements.

  3. The evidence for hallucination mitigation in open-ended generation is limited. On CHAIR (Table 6), DUST reduces sentence-level hallucination from 20.00 to 18.40, but slightly increases instance-level hallucination from 6.18 to 6.23 and reduces object recall from 62.97 to 58.04. These results suggest a trade-off between hallucination suppression and information coverage, leaving unclear whether the gains on constrained QA tasks translate into more reliable, informative free-form responses.

Questions for authors
  1. How accurately does the final DUST score distinguish hallucinated from correct outputs across models and layers? Does it outperform its individual cues, and do its gains remain against constant-strength or shuffled gates with comparable intervention magnitude?

  2. Can the authors reconcile the accuracy, precision, and recall values in Table 5? Please clarify whether they use different subsets or aggregation procedures, and provide corrected results or confusion counts under a consistent protocol.

  3. Are calibration instances excluded from evaluation, and are directions and normalization statistics estimated separately for each benchmark? Please clarify the selection of λ, baseline hyperparameters, and decoding settings, including whether the reported gains are stable across calibration samples.

Reviewer_BiSNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an adaptive activation steering method for hallucination mitigation. Specifically, for each layer and each time step, an uncertainty value R is calculated to weight the steering vector. R is calculated by integrating different forecasting signals including attention head entropy, image and text token attention mass, and maximum attention. Experiment establishes advantage over existing training-free hallucination mitigation methods, particularly on non-object hallucination benchmarks.

Critical strengths
  • Layer-wise and step-wise steering provides more fine-grained decisions on how much to steer or preserve the original activations.
  • Probing experiments for hallucination-related metrics are informative, though they could be further investigated.
  • Experiment establishes advantage over existing training-free hallucination mitigation methods, particularly on non-object hallucination benchmarks.
Critical weaknesses
  • While a large number of metrics are considered in the probing experiments, the choice of the final four metrics does not seem to be very motivated. Figure 3 is somewhat confusing and does not motivate including attention mass as a metric. Could there be more quantitative evidence to decide which ones to include for the calculation of uncertainty?
  • While the paper demonstrates an unusual advantage on non-object-related hallucinations, it does not establish why particularly layer-wise activation steering helps with these tasks. Some intuition could be more helpful.
  • Although adding uncertainty improve upon fixed PCA steering, adding multiple metrics for R seems to offer limited improvement.
  • The evaluated backbones are quite dated from a 2026 perspective.
  • A related paper on step-wise activation steering [1] is worth discussing in the paper.

[1] Zhang et al. Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation, ICML 26

Questions for authors
  • What does the uncertainty distribution R look like, and is it possible for it to directly forecast hallucination?
  • On the algorithm: Are the steering weights R calculated from an non-steered first pass, or are the next-layer uncertainty updated according to current layer's steering?
Reviewer_vFNfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DUST, a training-free approach to mitigating hallucinations in large vision-language models through uncertainty-aware layer-wise steering. It combines multiple attention-derived signals using a λ-Choquet integral to estimate layer-wise uncertainty and dynamically adjust visual steering strength. Experiments across multiple LVLMs and benchmarks demonstrate improvements in both object-centric and non-object visual tasks.

Critical strengths
  1. Broad evaluation across hallucination categories. Unlike methods primarily targeting object hallucinations, DUST addresses object, text, color, counting, and spatial reasoning errors. The results in Table 1 and Figure 6 demonstrate improvements across multiple visual tasks, supporting the practical relevance of the approach.

  2. Effective adaptive steering framework. The combination of layer-wise uncertainty estimation and dynamic visual steering is interesting. The ablation results in Figure 7(a) show that the full method outperforms fixed PCA steering and several simpler uncertainty-based variants.

Critical weaknesses
  1. Insufficient validation of the uncertainty estimation. The paper assumes that attention-derived uncertainty reflects hallucination risk, but does not directly evaluate the hallucination detection performance of the final aggregated uncertainty score. The AUROC analyses in Section 3 primarily examine individual features rather than the proposed uncertainty gate. This weakens the justification for uncertainty-aware steering.

  2. Limited justification for the Choquet aggregation. The symmetric λ-Choquet integral used in the paper is equivalent to an ordered weighted aggregation, as shown in Appendix B. Therefore, its ability to model interactions among specific attention patterns is unclear. Additional comparisons with simpler aggregation methods would strengthen the methodological contribution.

Questions for authors
  1. Uncertainty estimation: Can the authors directly evaluate whether the aggregated uncertainty score correlates with actual hallucination risk across different layers, models, and hallucination categories?

  2. Choquet aggregation: Given that the symmetric λ-Choquet integral is equivalent to an ordered weighted aggregation, what advantages does it provide over simpler fusion strategies?

  3. Dynamic steering: Can the authors compare DUST with constant, random, or shuffled uncertainty gates to better isolate the contribution of uncertainty-aware adaptation?

  4. Experimental consistency: Table 1(a) reports an improvement of +5.97 for InstructBLIP, whereas the reported scores (80.63 to 86.80) indicate +6.17. Additionally, the stated improvement range in Section 5.2 does not match Table 1(a), and the VizWiz scores differ between Tables 1(b) and 4. Could the authors clarify these inconsistencies and verify the reported results?

  5. Hallucination evaluation: Since VizWiz and What's Up primarily evaluate visual understanding and reasoning, how do the authors distinguish genuine hallucination reduction from improvements in general task accuracy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.