- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies hallucinations in LVLMs by decomposing visual encoder representations into sparse features using sparse autoencoders. It identifies always-on and image-specific features and proposes Contrastive Neuron Steering, which uses differences between clean and perturbed image representations to adjust image-specific features while compensating activation changes through always-on features.
Critical strengths
-
The paper provides an interpretable analysis of hallucination through sparse visual features, and the qualitative interventions make the connection between individual visual concepts and model outputs relatively easy to inspect.
-
The distinction between always-on and image-specific neurons provides a simple way to characterize different activation patterns in the visual representation, and the paper connects this analysis to the design of CNS.
-
CNS is a training-free intervention with relatively low inference overhead, and the experiments cover multiple LVLMs and both hallucination-specific and general multimodal benchmarks.
Critical weaknesses
-
The motivation and novelty are not sufficiently positioned against recent work. The paper motivates its contribution by arguing that existing hallucination studies mainly focus on decoding or language-side representations, while the role of internal visual representations remains underexplored. However, several recent works have already investigated hallucination directly from the visual or internal-representation perspective. For example, HIT, VHR, VISTA, SHARP, and ILVAD similarly analyze internal visual signals, attention, or hidden representations and intervene on them to reduce hallucination [1-5]. Therefore, studying hallucination from the visual/internal-representation perspective itself is not a new contribution. The authors should discuss these findings in relation to their hypothesis and method better. The overlap is even stronger with recent SAE-based works. Prior studies have already shown that SAE features in vision-language models are interpretable and steerable [6,7], while Steering LVLMs via Sparse Autoencoder for Hallucination Mitigation and SAVE directly use SAE feature steering to mitigate hallucination [8,9]. The novelty of this paper therefore appears to lie more specifically in the clean-perturbed, image-conditioned SAE correction and the always-on neuron compensation mechanism, rather than in SAE-based hallucination analysis or visual feature intervention in general. But these are not new things either. You may find similar findings in SAE works quite often.
-
The causal evidence for the proposed hallucination mechanism is not fully convincing. Table 1 shows that zeroing image-specific neurons substantially hurts POPE performance, but this mainly demonstrates that these neurons encode useful image information; it does not establish that their abnormal activation specifically causes hallucination.
-
CNS is also conceptually close to visual contrastive decoding. Both methods use a clean image and a corrupted view and push the model away from the corrupted visual signal; CNS mainly differs by applying the contrast in SAE feature space, restricting it to image-specific features, and adding ANC. The paper should more clearly isolate the contribution of sparse feature decomposition from the underlying clean-versus-corrupted contrastive principle.
[1] Che L, Liu TQ, Jia J, Qin W, Tang R, Pavlovic V. Hallucinatory image tokens: a training-free EAZY approach to detecting and mitigating object hallucinations in LVLMs. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE/CVF; 2025:21635-21644.
[2]He J, Zhu K, Guo H, et al. Cracking the code of hallucination in LVLMs with vision-aware head divergence. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics; 2025:3488-3501.
[3]Li Z, Shi H, Gao Y, et al. The hidden life of tokens: reducing hallucination of large vision-language models via visual information steering. In: Proceedings of the 42nd International Conference on Machine Learning. Vol 267. Proceedings of Machine Learning Research; 2025:35799-35819.
[4] Wu J, Ding Y, Liu G, et al. SHARP: steering hallucination in LVLMs via representation engineering. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics; 2025:14346-14361.
[5] Xie Y, Hua Z, Wang R, Ng WWY, Wang X, Jia Y. Finding the correct visual evidence without forgetting: mitigating hallucination in LVLMs via inter-layer visual attention discrepancy. In: Proceedings of the 43rd International Conference on Machine Learning. Vol 306. Proceedings of Machine Learning Research; 2026:139471-139490.
[6] Lou H, Li C, Ji J, Yang Y. SAE-V: interpreting multimodal models for enhanced alignment. In: Proceedings of the 42nd International Conference on Machine Learning. Vol 267. Proceedings of Machine Learning Research; 2025:40344-40360.
[7] Pach M, Karthik S, Bouniot Q, Belongie S, Akata Z. Sparse autoencoders learn monosemantic features in vision-language models. Adv Neural Inf Process Syst. 2025;38.
[8] Hua Z, He J, Yao Z, et al. Steering LVLMs via sparse autoencoder for hallucination mitigation. In: Findings of the Association for Computational Linguistics: EMNLP 2025. Association for Computational Linguistics; 2025:10808-10828.
[9] Park S, Yoo S, Mok J, Yoon S. SAVE: sparse autoencoder-driven visual information enhancement for mitigating object hallucination. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. IEEE/CVF; 2026:7935-7944.
Questions for authors
See above.