- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes HSS-DR, a framework for detecting and mitigating object hallucinations in large vision-language models (LVLMs). The authors observe that grounded and hallucinated object mentions exhibit different spatial spectral patterns in visual hidden states, particularly in late decoder layers. Based on this observation, they train a lightweight detector using spectral features, response-map statistics, and generation context to identify potentially hallucinated object mentions. When a high-risk mention is detected, the method performs constrained local regeneration and selects a lower-risk continuation using short lookahead evaluation. Experiments across six LVLMs show consistent hallucination reductions relative to vanilla decoding, while largely preserving general multimodal capabilities.
Critical strengths
-
An interesting observation about spectral signatures of object hallucinations. The paper identifies differences in the spatial frequency distributions of hidden visual representations between grounded and hallucinated object mentions. The authors further validate these observations across data splits, within-image comparisons, and different model architectures. This provides an interesting perspective on hallucination detection using internal representations.
-
A practical detection-and-repair framework evaluated across multiple architectures. HSS-DR combines lightweight hallucination detection with localized regeneration while keeping the LVLM backbone frozen. The experiments demonstrate consistent reductions in object hallucination across six models, suggesting that the framework can be adapted to different visual-token architectures.
Critical weaknesses
-
Weak connection between the hypothesis and the proposed mechanism. The paper hypothesizes that grounded objects exhibit more spatially coherent responses, but the connection between spatial coherence and the observed spectral differences is not directly validated. Moreover, the lack of an end-to-end ablation removing spectral features leaves their unique contribution to hallucination reduction unclear.
-
Limited evidence for comparative cross-model advantages. While HSS-DR consistently reduces hallucinations across six LVLMs, the evaluation lacks comparisons with established methods such as PAI and OPERA across different architectures. Since cross-model effectiveness is a key contribution, it is important to investigate whether these existing methods exhibit similar improvements or face limitations in cross-model generalization. Without such comparisons, the claimed cross-model advantage remains insufficiently substantiated.
Questions for authors
-
Can the authors provide direct evidence that the observed spectral differences reflect spatial coherence rather than global response statistics (e.g., the DC component)? Additionally, how does removing spectral features affect the final hallucination reduction when the repair procedure remains unchanged?
-
How does HSS-DR's cross-model effectiveness compare with established methods such as PAI and OPERA under matched evaluation settings? Do these methods exhibit similarly consistent improvements across different architectures, and if not, what limitations distinguish them from HSS-DR? If direct comparisons are infeasible, please clarify the reasons.
3 comments
Thank you for the additional experiments and clarifications. While I still have some reservations about the interpretation of the spectral signatures, the new end-to-end ablation and comparisons with PAI/OPERA substantially address my main concerns. I have therefore updated my rating to 3.
Thank you very much for your thoughtful follow-up and for taking the time to carefully consider our additional experiments and clarifications. We sincerely appreciate your constructive feedback and recognition of our efforts. Your comments have been very valuable in helping us improve our work, and we are grateful for your time and consideration throughout the review process.
We sincerely thank the reviewer for the careful evaluation and thoughtful comments. We appreciate the valuable feedback and would like to provide the following clarifications and additional experimental results.
W1. Spectral Hypothesis and End to End Ablation
DC component and spatial arrangement. We conducted additional analyses on 299 images with 1,915 grounded and 353 hallucinated mentions. The table reports AUROC using the low frequency energy ratio, with the DC energy ratio evaluated separately.
After removing DC, the remaining frequency components still provide useful information for distinguishing grounded from hallucinated mentions. Their discriminative ability decreases further when spatial positions are shuffled, highlighting the importance of spatial arrangement.
We also evaluated the frequency features separately across 20 object categories. After removing DC, the average AUROC reaches 0.6666, compared with 0.5052 after spatial shuffling. These results further support the value of spatial frequency information beyond global response statistics.
End to end ablation. We conducted an additional ablation on LLaVA-1.5 using the same 500 CHAIR images. The repair procedure, search budget, and training settings were kept unchanged.
Notably, the spectral feature block alone achieves a CHAIRs of 33.80%, approaching the performance of the full method. Removing spectral features increases CHAIRs from 32.40% to 37.20% and CHAIRi from 9.11% to 10.27%, while Recall slightly decreases from 76.45% to 75.99%.
These results demonstrate that spectral features contribute to hallucination mitigation beyond the repair procedure itself.
W2. Cross Model Comparisons with PAI and OPERA
We conducted additional comparisons with PAI and OPERA on LLaVA-1.5 and MiniGPT-4, using the same 500 CHAIR images and evaluation metrics.
HSS-DR achieves the lowest CHAIRs and CHAIRi on both models. On LLaVA-1.5, CHAIRs is 32.40%, compared with 36.80% for PAI and 44.60% for OPERA. On MiniGPT-4, HSS-DR further reduces CHAIRs to 26.80%, outperforming PAI (34.00%) and OPERA (29.20%).
These results provide additional evidence that HSS-DR achieves effective hallucination reduction across different model architectures.