- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes HT-SCD, a training-free decoding framework for mitigating object hallucinations in large vision-language models. The method first detects potentially hallucinated object words using gradient-weighted similarity between object-word representations and visual-token representations, and then rolls back the generated sentence when a hallucination is detected. The predictive distributions associated with detected hallucinated objects are subsequently used as negative signals for contrastive regeneration. The method is evaluated on MSCOCO-CHAIR and AMBER across three LVLMs.
Critical strengths
-
The paper studies an important and practically relevant problem, i.e., reducing object hallucinations without additional training or external detection models.
-
The proposed method shows fairly strong results on the reported hallucination benchmarks. In particular, HT-SCD improves CHAIR-related metrics consistently across three LVLMs.
-
I also appreciate that the authors include several ablations on the detector and the choice of negative branch.
Critical weaknesses
-
The paper is not sufficiently polished. The Introduction is very short, and the problem setup and notation are not clearly established before the method is introduced.
-
I am not convinced by the novelty. Contrastive decoding is already a very crowded direction in this field, including methods based on internal states and different negative branches. HT-SCD seems to mainly change how the negative signal is constructed and when regeneration is triggered, and I think the authors need to justify more clearly what is fundamentally new here.
-
The method description is difficult to follow. The interaction between hallucination detection, sentence-level rollback, hallucination trajectories, and regeneration is not explained cleanly, and I found the exact algorithm hard to reconstruct from the text.
-
The preliminary analysis is quite scattered. The transition from perturbation, to intermediate-layer observations, to gradient attribution is not very well motivated. I also think this section should better position the method against closely related work such as Devils in Middle Layers [1] and Hallucinatory Image Tokens / EAZY [2].
-
The experimental evaluation is incomplete. The main results focus on hallucination benchmarks, but there is no evaluation showing that the intervention preserves general visual understanding or reasoning ability. The formatting of Tables 2 and 3 is also confusing because only the authors' method is bolded, rather than consistently highlighting the best results.
-
The efficiency analysis is limited to one model and one hardware setting.
[1] Jiang Z, Chen J, Zhu B, Luo T, Shen Y, Yang X. Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2025:25004-25014. doi:10.1109/CVPR52734.2025.02328.
[2] Che L, Liu TQ, Jia J, Qin W, Tang R, Pavlovic V. Hallucinatory image tokens: A training-free EAZY approach to detecting and mitigating object hallucinations in LVLMs. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2025:21635-21644. doi:10.1109/ICCV51701.2025.02009.
Questions for authors
-
Could the authors clarify what exactly constitutes a “hallucination trajectory”? From the current description, it appears to be mainly the collection or aggregation of logits at detected hallucinated object positions, and I am not fully clear why this should be viewed as a trajectory.
-
How the object tokens are distinguished from the other tokens?
-
How sensitive is the method to the hallucination detection threshold and the selected intermediate layer? Are these hyperparameters shared across models, or tuned separately for each model?
-
Since rollback and regeneration directly modify the generation process, have the authors evaluated whether the method hurts general VQA, reasoning, or caption quality beyond hallucination-specific metrics?
-
What does boldface in Tables 2 and 3 denote? Does it indicate the best result, or simply the proposed method?