- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes ResOT, which mitigates object hallucination by repairing hallucinationrelated representations rather than directly suppressing them. ResOT confines the intervention to the residual subspace and uses Gaussian optimal transport to move representations toward a faithful target with minimum expected squared displacement. A likelihood-based gate further adjusts how much of this correction is applied during decoding. Evaluation on LLaVA-1.5, mPLUG-Owl2, and InstructBLIP covers hallucination, caption quality, multimodal capabilities, calibration robustness, and inference cost.
Critical strengths
-
The repair formulation is well motivated and supported by informative ablations. The method recognizes that hallucination-associated directions can also contain useful information, and replaces component removal with distribution-based correction. Table 4 helps separate the contributions of localization and repair: OT in the residual subspace outperforms OT in random or Nullu-style subspaces, while covariance-aware transport improves over mean-only alignment. These comparisons provide evidence that the gains depend on both where and how the intervention is applied.
-
The empirical evaluation provides substantial evidence for the effectiveness of ResOT. ResOT is combined with LLaVA-1.5, mPLUG-Owl2, and InstructBLIP and conducted experiments on multiple benchmarks. The results demonstrate that ResOT reduces object hallucination while improving image caption quality and multimodal performance.
Critical weaknesses
-
The fitted distribution and gate are validated mainly on complete descriptions, while intervention operates on generation prefixes. Section 3.1 fits the map using final-position states after complete faithful or hallucinated captions. Online repair instead acts at every next-token prediction position. The separability, alignment, and displacement analyses in Appendix D largely reuse complete-description representations, so they do not establish that the likelihood ratio identifies states needing repair during free-running generation. The paper acknowledges this shift in Appendix E, p. 21. This matters because OT optimality applies to its fitted source distribution; it does not establish that transporting an arbitrary prefix state improves grounding.
-
The robustness of the advantage over the strongest competing methods is not fully established. Some leading results have small margins: LLaVA-1.5 POPE F1 improves from 86.46 with HulluEdit to 86.54, while mPLUG-Owl2 CHAIRs improves from 0.314 with VTI to 0.302. The main comparisons lack paired confidence intervals or significance analyses.
Questions for authors
-
Can you evaluate how the actual likelihood gate relates to grounded versus hallucinated object predictions during free-running generation? A focused analysis of gate values at object-generating positions, including their behavior across generation positions, would help connect the complete-description calibration analyses to the online intervention.
-
Could you report paired image-level uncertainty for the principal CHAIR and POPE comparisons against the closest baselines, keeping questions from the same image together when resampling? Could you also assess whether the rankings persist when the closest representation baselines receive comparable validation tuning on the same image split?
-
Please report the final selected shared rank (r=k) and global repair strength (\alpha) for each backbone. Appendix B already specifies the search procedure and the per-layer 95th-percentile correction threshold. Could you additionally provide the exact norm-clipping formula and clarify whether clipping is applied before or after multiplication by the global strength and likelihood gate?
3 comments
Thank you for the detailed experiments and explainations.
The prefix replay analysis substantially addresses my concern about whether the likelihood gate remains informative on freely generated prefixes.
The experiment of statistical significance and robustness of comparisons under comparable hyperparameter tuning is clear and informative.
The revised appendix also fully answers my questions about the selected hyperparameters and clipping order.
Overall, the rebuttal strengthens the paper, and I am maintaining my rating of 3 (weak acceptance).
We are glad that our responses have addressed your concerns!
Thank you again for your positive assessment and continued support.
Your careful review and constructive feedback have helped us strengthen the empirical analysis and clarify the method.
We thank the reviewer for the positive assessment of our work and the constructive suggestions. We address your questions below.
We acknowledge the reviewer’s concern about this potential distribution shift. We address this from two perspectives:
Setting. We randomly sample 100 COCO images and generate captions using Vanilla. We use GPT-6 Sol, followed by manual verification, to identify hallucinated spans. Keeping the generated prefixes unchanged, we evaluate the likelihood gate and repair magnitude immediately before each span.
Across all evaluated token-position ranges, hallucinated spans receive higher average gate values and repair magnitudes than other spans.
We thank the reviewer for this suggestion. We repeat the comparisons using five random seeds. The experimental results are presented in the table below.
To assess the robustness of our comparisons with the strongest baselines, we conduct additional experiments below. We perform 10,000 paired image-level bootstrap resamples over 500 images. The experimental results are presented in the table below.
To further examine the fairness of our comparisons, we conduct additional experiments on an independent validation set of 200 images, comprising 3,600 POPE-style questions. We fine-tune the hyperparameters of VTI, HulluEdit, and ResOT on the same validation set. The experimental results are presented in the table below.
Overall, while ResOT does not consistently outperform the strongest baselines on every individual metric, its main contribution lies in achieving a favorable balance among hallucination reduction, multimodal capability preservation, and generation quality. Our extensive experimental results across multiple benchmarks further support this claim.
We apologize for the confusion. We have added the relevant clarifications in Lines 891–901.
Finally, we thank the reviewer again for their time and thoughtful feedback. We hope our responses have addressed your concerns, and we would be happy to clarify any remaining questions!
[1]. Liu S, Ye H, Zou J Y. Reducing hallucinations in large vision-language models via latent space steering. ICLR, 2025.
[2]. Li Z, Shi H, Gao Y, et al. The hidden life of tokens: Reducing hallucination of large vision-language models via visual information steering. ICML, 2025.