- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies hallucination mitigation in large vision-language models. It identifies Post-Steer Content-Suppression: existing activation-steering methods may reduce the likelihood of genuine objects alongside hallucinated ones. The proposed training-free method, VES, constructs steering directions from image regions, weights them according to the evolving generation context, and combines them with a precomputed visual steering direction. The authors evaluate the method through token-level logit analysis, linear probes, ablations, and experiments on three models across CHAIR, POPE, MME, and AMBER. VES generally reduces hallucination while maintaining competitive object coverage.
Critical strengths
- Clear motivation supported by substantial preliminary analysis. The paper goes beyond presenting a steering heuristic by first investigating a limitation of existing approaches: suppressing hallucinated content may also suppress signals associated with genuine visual content.
- Broad empirical validation against strong baselines, with a favorable balance between hallucination reduction and content preservation. The experimental study covers three VLM families and four benchmarks overall, including CHAIR, POPE, MME, and AMBER, and compares VES with a substantial set of competitive methods, such as VCD, VISTA, DeCo, VTI, and DMAS. The generally strong results across different evaluation settings provide convincing evidence of the method’s practical effectiveness.
Critical weaknesses
- PSD is calculated from raw logit changes averaged across genuine and hallucinated token positions. However, a decrease in a token’s raw logit does not necessarily imply a decrease in its generation probability. A position-dependent additive shift applied to all vocabulary logits leaves the output distribution unchanged while changing the suppression score. Consequently, PSD alone does not establish that steering selectively suppresses genuine content or increases its omission rate.
- The ablation studies only conducted on a 200 images subset of CHAIR.
- LLaVA 1.5 and Qwen2-VL are somewhat outdated. Evaluating the method with more recent models would make the results more convincing.
- There is a typo on line 795 of the paper: “Table ??” appears to be an unresolved table reference.
Questions for authors
- See Weakness 1.
- Could the authors conduct more extensive ablation studies to better assess the contribution of each component?
- Could the authors evaluate VES on more recent models, such as LLaVA 1.6 and Qwen3-VL/Qwen3.5, to further validate its effectiveness?
- Is the Qwen2-VL MME total 721.0 or 717.7? There is a discrepancy in Line 732 and Appendix B1.