- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SAKED, a training-free decoding approach for mitigating hallucinations in large vision-language models. It constructs a Knowledge Stability Score (KSS) using cross-head visual-attention agreement (CHSS), cross-layer prediction consistency (CLSS), and cross-token visual-attention consistency (CTSS). At each decoding step, SAKED selects the most and least stable candidate layers and contrasts their logits to revise the final prediction. The method achieves encouraging average results on CHAIR and POPE, with additional evaluations on MME, AMBER, and several larger or more recent models. However, I have substantial reservations about the evidence supporting the central stability-based hypothesis and the methodological contribution beyond existing intermediate-layer decoding approaches.
Critical strengths
- The problem is important, and the proposed intervention is training-free and requires no external supervision.
- The average results on CHAIR and POPE are competitive across five LVLMs, and the paper also provides component ablations, efficiency measurements, and supplementary evaluations.
Critical weaknesses
Major Weaknesses
1. The central stability–hallucination hypothesis is not sufficiently established.
The paper's main premise is that agreement across attention heads, continuity across network layers, and consistency across adjacent tokens indicate more reliable visual knowledge. However, none of these properties necessarily implies that a prediction is visually grounded: multiple heads or layers can consistently support the same incorrect object, while legitimate changes in the described object may cause attention shifts between adjacent tokens. Figures 2–5 provide illustrative examples, but they do not establish these relationships systematically. Figure 6 offers useful quantitative evidence, yet it only reports layer-wise averages for Qwen3-VL-8B and Qwen3-VL-30B-A3B, both from the same model family, rather than the five principal evaluation backbones. The separation between true and hallucinated tokens also varies across the proposed scores and layers. Without sample counts, uncertainty estimates, or a direct evaluation of hallucination discrimination (e.g., AUROC/AUPRC against simple confidence-based signals), it is difficult to determine whether the proposed measures reliably identify hallucination-prone predictions. Importantly, the supplementary theory assumes a monotonic relationship between KSS and deviation from ideal grounded logits; it does not independently demonstrate this crucial premise. This leaves the main scientific motivation insufficiently supported.
2. The incremental methodological contribution beyond existing layer-aware decoding methods is unclear.
The actual decoding mechanism consists of handcrafted consistency scores, selection of two intermediate layers, and a contrastive correction to final-layer logits. The general strategy of exploiting intermediate-layer information to mitigate hallucinations has already been explored by DeCo, as well as related approaches such as DAMO (ICLR 2025) and MoLE (AAAI 2025). SAKED introduces a particular stability-based criterion, but the experiments do not convincingly isolate why this criterion, or the use of both highest- and lowest-KSS layers, is necessary. Removing individual score components is not an adequate substitute for comparing against alternative layer-selection rules under the same logit-correction formula and inference budget—for example, fixed-depth selection, random selection, confidence/entropy-based selection, or selection using only one score. Although the full method achieves the best average result in the reported ablation, some individual model–metric combinations favor reduced variants. Thus, the experiments establish that the overall recipe can work, but do not sufficiently demonstrate that its central design offers a distinct and necessary advance over simpler layer-aware decoding strategies.
3. The empirical results do not directly validate the claimed hallucination-correction mechanism.
The improvements on CHAIR and POPE are meaningful and should be acknowledged. Nevertheless, the claimed contribution is more specific than improving aggregate benchmark scores: SAKED is intended to locate more reliable internal knowledge and suppress unstable predictions to produce better visually grounded tokens. The paper does not analyze which tokens are changed by its intervention, how often those changes correct a hallucination, or how often they replace an originally correct prediction. Such evidence is especially important because changes in caption length or the number of mentioned objects can affect CHAIR scores; language-quality metrics alone do not fully resolve this issue. Moreover, the broader evaluations on POPE, MME, and AMBER do not provide corresponding analyses of the three stability components, despite the different generation regimes involved. Consequently, the existing results demonstrate useful empirical performance but do not establish that the proposed stability-aware selection is the mechanism responsible for improved visual grounding.
Minor Comments
- The results are not uniformly improved across model–metric pairs. For instance, on CHAIR, InstructBLIP's Ci changes from 13.3 (greedy) to 14.0 (SAKED); on POPE, InternVL3's F1 changes from 0.9074 to 0.9027. These exceptions should be discussed rather than interpreted as failures of the entire method.
- The larger/recent-model experiments in Appendix Table 6 compare SAKED only with vanilla decoding. Comparisons with competitive decoding methods at those scales would better support the generalization claim.
- The limited role of CTSS on short answers, acknowledged in the supplement, merits task-specific ablations on POPE or MME.
- Section 4.6 reports average Ci/Cs of 10.0/35.7, whereas Table 4 reports 10.2/36.1 for the complete method. This discrepancy should be reconciled.
Questions for authors
- How predictive are KSS and the three individual scores of hallucination at the token level on the five primary LVLMs, relative to confidence- or entropy-based alternatives? What are the sample sizes and variances underlying Figure 6?
- With the decoding correction and computational budget fixed, how does KSS-based layer selection compare with simpler layer-selection policies? How much benefit specifically comes from the least-stable-layer contrast?
- Among the tokens changed by SAKED, how many correspond to corrected hallucinations versus newly introduced errors? How do caption length and object coverage change relative to the baselines?