acceptodds
Under review as a conference paper at ICLR 2027

SAKED: Mitigating Hallucination in Large Vision-Language Models via Stability-Aware Knowledge Enhanced Decoding

Abstract

Hallucinations in Large Vision-Language Models (LVLMs) pose significant security and reliability risks in real-world applications. Inspired by the observation that humans are more error-prone when uncertain or hesitant, we investigate how instability in a model’s internal knowledge contributes to LVLM hallucinations. We conduct extensive empirical analyses from three perspectives, namely attention heads, model layers, and decoding tokens, and identify three key hallucination patterns: (i) visual activation drift across attention heads, (ii) pronounced knowledge fluctuations across layers, and (iii) visual focus distraction between neighboring output tokens. Building on these findings, we propose Stability-Aware Knowledge-Enhanced Decoding (SAKED), which introduces a layer-wise Knowledge Stability Score (KSS) to quantify knowledge stability throughout the model. By contrasting the most stability-aware and stability-agnostic layers, SAKED suppresses decoding noise and dynamically leverages the most reliable internal knowledge for faithful token generation. Moreover, SAKED is training-free and can be seamlessly integrated into different architectures. Extensive experiments demonstrate that SAKED achieves state-of-the-art performance for hallucination mitigation on various models, tasks, and benchmarks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Vision Language Model, Hallucination, Decoding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_u1eJOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SAKED, a training-free decoding approach for mitigating hallucinations in large vision-language models. It constructs a Knowledge Stability Score (KSS) using cross-head visual-attention agreement (CHSS), cross-layer prediction consistency (CLSS), and cross-token visual-attention consistency (CTSS). At each decoding step, SAKED selects the most and least stable candidate layers and contrasts their logits to revise the final prediction. The method achieves encouraging average results on CHAIR and POPE, with additional evaluations on MME, AMBER, and several larger or more recent models. However, I have substantial reservations about the evidence supporting the central stability-based hypothesis and the methodological contribution beyond existing intermediate-layer decoding approaches.

Critical strengths
  • The problem is important, and the proposed intervention is training-free and requires no external supervision.
  • The average results on CHAIR and POPE are competitive across five LVLMs, and the paper also provides component ablations, efficiency measurements, and supplementary evaluations.
Critical weaknesses

Major Weaknesses

1. The central stability–hallucination hypothesis is not sufficiently established.

The paper's main premise is that agreement across attention heads, continuity across network layers, and consistency across adjacent tokens indicate more reliable visual knowledge. However, none of these properties necessarily implies that a prediction is visually grounded: multiple heads or layers can consistently support the same incorrect object, while legitimate changes in the described object may cause attention shifts between adjacent tokens. Figures 2–5 provide illustrative examples, but they do not establish these relationships systematically. Figure 6 offers useful quantitative evidence, yet it only reports layer-wise averages for Qwen3-VL-8B and Qwen3-VL-30B-A3B, both from the same model family, rather than the five principal evaluation backbones. The separation between true and hallucinated tokens also varies across the proposed scores and layers. Without sample counts, uncertainty estimates, or a direct evaluation of hallucination discrimination (e.g., AUROC/AUPRC against simple confidence-based signals), it is difficult to determine whether the proposed measures reliably identify hallucination-prone predictions. Importantly, the supplementary theory assumes a monotonic relationship between KSS and deviation from ideal grounded logits; it does not independently demonstrate this crucial premise. This leaves the main scientific motivation insufficiently supported.

2. The incremental methodological contribution beyond existing layer-aware decoding methods is unclear.

The actual decoding mechanism consists of handcrafted consistency scores, selection of two intermediate layers, and a contrastive correction to final-layer logits. The general strategy of exploiting intermediate-layer information to mitigate hallucinations has already been explored by DeCo, as well as related approaches such as DAMO (ICLR 2025) and MoLE (AAAI 2025). SAKED introduces a particular stability-based criterion, but the experiments do not convincingly isolate why this criterion, or the use of both highest- and lowest-KSS layers, is necessary. Removing individual score components is not an adequate substitute for comparing against alternative layer-selection rules under the same logit-correction formula and inference budget—for example, fixed-depth selection, random selection, confidence/entropy-based selection, or selection using only one score. Although the full method achieves the best average result in the reported ablation, some individual model–metric combinations favor reduced variants. Thus, the experiments establish that the overall recipe can work, but do not sufficiently demonstrate that its central design offers a distinct and necessary advance over simpler layer-aware decoding strategies.

3. The empirical results do not directly validate the claimed hallucination-correction mechanism.

The improvements on CHAIR and POPE are meaningful and should be acknowledged. Nevertheless, the claimed contribution is more specific than improving aggregate benchmark scores: SAKED is intended to locate more reliable internal knowledge and suppress unstable predictions to produce better visually grounded tokens. The paper does not analyze which tokens are changed by its intervention, how often those changes correct a hallucination, or how often they replace an originally correct prediction. Such evidence is especially important because changes in caption length or the number of mentioned objects can affect CHAIR scores; language-quality metrics alone do not fully resolve this issue. Moreover, the broader evaluations on POPE, MME, and AMBER do not provide corresponding analyses of the three stability components, despite the different generation regimes involved. Consequently, the existing results demonstrate useful empirical performance but do not establish that the proposed stability-aware selection is the mechanism responsible for improved visual grounding.

Minor Comments

  • The results are not uniformly improved across model–metric pairs. For instance, on CHAIR, InstructBLIP's Ci changes from 13.3 (greedy) to 14.0 (SAKED); on POPE, InternVL3's F1 changes from 0.9074 to 0.9027. These exceptions should be discussed rather than interpreted as failures of the entire method.
  • The larger/recent-model experiments in Appendix Table 6 compare SAKED only with vanilla decoding. Comparisons with competitive decoding methods at those scales would better support the generalization claim.
  • The limited role of CTSS on short answers, acknowledged in the supplement, merits task-specific ablations on POPE or MME.
  • Section 4.6 reports average Ci/Cs of 10.0/35.7, whereas Table 4 reports 10.2/36.1 for the complete method. This discrepancy should be reconciled.
Questions for authors
  1. How predictive are KSS and the three individual scores of hallucination at the token level on the five primary LVLMs, relative to confidence- or entropy-based alternatives? What are the sample sizes and variances underlying Figure 6?
  2. With the decoding correction and computational budget fixed, how does KSS-based layer selection compare with simpler layer-selection policies? How much benefit specifically comes from the least-stable-layer contrast?
  3. Among the tokens changed by SAKED, how many correspond to corrected hallucinations versus newly introduced errors? How do caption length and object coverage change relative to the baselines?
Reviewer_gxUaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, they first conduct empiricalanalyses to examine the relationship between knowledge stability and hallucination from three perspectives, including attention heads, model layers, and decoding steps. Based on the observations, they introduce a hallucination mitigation method termed stability-aware knowledge enhanced decoding to quantify knowledge stability for each layer. Experiments on five LVLMs show that the proposed method can achieve good hallucination mitigation performance.

Critical strengths
  1. They conduct empirical analyses to examine the relationship between knowledge stability and hallucination. Then, the SAKED method is introduced based on these observations and analyses.
  2. Extensive experiments are conducted on several models.
Critical weaknesses
  1. In Sections 3.2-3.4, this paper tried to show how instability in internal model knowledge contributes to hallucinations in LVLMs. However, they only show the observations on several examples. It is questionable whether these phenomena can be widely observed?
  2. According to the results summarized in Table 2, the performance of SAKED is very close to Deco and SAVER on four models. The performance gain is mainly obtained on Qwen2.5VL. This needs further discussion.
  3. In Table 10, why the performance of MiniGPT-4 is not reported?
  4. Missing related work: Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats (NeurIPS 2025). The method is highly similar and should therefore be compared with SAKED.
  5. In reference, the publication information should be updated. Many references have already been published on conferences/ journals, no longer in open review phase.
Questions for authors

See weakness.

Reviewer_gg5POpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the relationship between internal knowledge stability and hallucination in LVLMs. The authors define three signals: cross-head attention consistency, cross-layer prediction consistency, and cross-token visual consistency. SAKED combines these signals into a Knowledge Stability Score. At each decoding step, it selects the most and least stable layers and uses their logits to correct the final prediction. Experiments on five LVLMs show strong average performance on CHAIR, POPE, MME, and AMBER.

Critical strengths
  1. The three stability signals are well motivated. The paper first examines instability across attention heads, model layers, and adjacent tokens. It then uses these observations to define CHSS, CLSS, and CTSS. SAKED is training-free and does not require an additional forward pass.
  2. The experiments cover a wide range of models and tasks.SAKED is evaluated on five LVLMs and four benchmarks, achieving the best average results on CHAIR and POPE. The paper also includes component and hyperparameter ablations, experiments on larger models, language-quality results, and efficiency measurements. Together, these results show that the improvements are not limited to a single model or benchmark.
Critical weaknesses
  1. The method is not described in enough detail to be reproduced. The paper does not explain how the concentration term in Equation 2 is calculated or how many attention heads are retained. It also tunes the decoding strength and candidate layers separately for each model, but does not report the final settings.
Questions for authors

Can the authors provide the complete implementation settings for each model and clarify how the hyperparameters were selected?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.