- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper develops Visual Evidence-guided Latent Correction (VE-LC), aiming to reduce hallucinations in large vision-language models (LVLMs). VE-LC is based on a monitoring and adaptation framework and incorporates the Latent Visual Correction (LVC) technique, which guides the subsequent reasoning process to better align with visual evidence. Furthermore, the method includes Local Hallucination Correction Supervision (LHCS), a mechanism that combines weighted correction signals with a contrastive learning objective. Experimental results across three LVLM architectures and six benchmarks demonstrate that the method significantly reduces hallucinations.
Critical strengths
-
This paper examines the critical issue of visual hallucinations in LVLMs. This problem is important and undermines the reliability and credibility of the responses generated by the models, particularly in real-world application scenarios.
-
The method is well motivated. The theoretical foundation is robust, and the explanation is clear. After identifying significant limitations in existing automatic self-calibration methods, the authors present a rational and effective solution, i.e., the use of latent correction based on visual evidence.
-
The structure of this paper is clear, and the presentation is appropriate. The logical organization is rigorous and the writing is accessible, enabling readers to easily understand the proposed method and experimental results.
Critical weaknesses
-
The novelty is limited. VE-LC seems to be a simple combination of existing works, i.e., OPERA (CVPR 24), VTI (ICLR 25), and MemVR (ICML 25).
-
The experimental comparisons may be somewhat unfair. VE-LC is training-based, while most baselines in Table 1 and Table 6 are training-free. The authors should divide the baselines into training-based/free categories. For comparisons with training-free approaches, the authors are encouraged to use the no-training variant of VE-LC.
-
Table 6 shows that VE-LC incurs 4.38× inference overhead compared to the base model. This additional overhead may undermine the utility of VE-LC in real-world applications. Moreover, since VE-LC involves the training phase, the computational complexity of this phase should also be computed and demonstrated.
-
Some recent works should be more clearly compared or discussed [1,2,3,4,5].
-
Minor: Line 247, modify EXPERIMENTS 4,500 to EXPERIMENTS.
References:
[1] Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering, ICLR 25.
[2] Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models, ICML 25.
[3] Hallucination-aware Intermediate Representation Edit in Large Vision-Language Models, ICLR 26.
[4] Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decoding, CVPR 26.
[5] Understanding and Mitigating Hallucinations in Multimodal Chain-of-Thought Models, CVPR 26.
Questions for authors
See weaknesses. I would like to raise my scores if the authors can effectively address my concerns.
3 comments
Thanks for the detailed responses. After carefully reading the rebuttal, I still think the novelty of this paper is somewhat limited. Despite this, the authors' additional comparisons (W2), experiments (W3) and discussions (W4) effectively addressed my concerns. Please add these results in the revised PDF, and I'm adjusting my rating to 3.
Thank you for your careful consideration of our responses and your constructive feedback. We will incorporate the additional comparisons, experimental results, and related discussions into the revised manuscript.
Response to Reviewer A8ye: Novelty, Fair Comparisons, and Computational Cost
We thank the reviewer for the constructive feedback and positive assessment of our motivation and framework.
W1. Novelty and correction mechanism. Our contribution is how visual evidence is used after rollback, not rollback or visual reuse alone. OPERA uses attention-based penalties and retrospection-allocation; VTI [1] applies precomputed visual/textual steering directions; MemVR [2] reinjects visual information through FFNs when uncertainty is high. VE-LC instead computes a correction signal at the current post-rollback prefix.
Specifically, LVC recursively updates continuous hidden states over the retained image-and-text KV cache and converts the normalized before/after logit difference into a residual for local regeneration (Eqs. 2-3). Visual alignment supervision encourages these states to retain correction-relevant evidence (Eq. 1). LHCS further emphasizes edited tokens and contrasts visually supported/unsupported candidates (Eqs. 4-8). Thus, the distinction lies in the construction and application of the correction signal and its local supervision.
A new controlled diagnostic supports this motivation: given a confirmed error location, we roll back and suppress the hallucinated object and its synonyms. Among 149 LLaVA-v1.5-7B cases, only 27 (18.12%) achieve visually consistent recovery; 121 (81.21%) still contain hallucinations, and one (0.67%) degrades. This shows that suppressing the identified concept alone does not reliably restore visually grounded continuations; it is not a direct comparison against VE-LC.
W2. Comparison fairness. We agree that training requirements must be explicit. We add AMP and POVID as training-based baselines, and MemVR and VTI as training-free baselines. Selected results on LLaVA-v1.5-7B are below. "Training" means additional method-specific training.
VE-LC improves both AMBER CHAIR and HalRate over all four added baselines, but its lower Cover reflects a hallucination-coverage trade-off. We do not claim uniform superiority: MemVR scores 73.75 on MMBench versus 73.73 for VE-LC. The reported VE-LC uses trained adapters, despite a frozen backbone. A no-additional-training VE-LC result is not included here; the cross-category comparisons therefore do not isolate the effects of training.
W3. Inference and training cost. We agree that 4.38× latency versus the base model is substantial and limits latency-sensitive deployment. In our timing experiments, 57% of responses did not trigger correction; this avoids rollback and latent correction, but not verification overhead. All online verification, rollback, correction, and regeneration are included in latency.
Under the evaluated configurations, VE-LC reduces mean latency from REVERSE's 58.90 s to 16.62 s (71.8%) and achieves 3.61× its output throughput, with similar output lengths (115.24 vs. 113.05 tokens). "Relatively low overhead" refers only to these comparisons, not to the base model. Appendix D specifies single-A40 inference timing with batch size 1.
For training, the base model is frozen and approximately 159.91M LoRA parameters are optimized. The recorded schedule comprises 2,500 initial, 375 contrastive, and 500 replay-training steps. The final training loop took approximately 6 h. This loop-level measurement should not be confused with a complete stage-wise accounting of training cost.
W4. Additional related work. Beyond the comparisons with [1,2] above, HIRE [3] selectively edits intermediate representations using an Editor and Router. We have not included HIRE results because the required pretrained Editor/Router checkpoints were not available to us. LEAD [4] uses entropy-triggered continuous embeddings and visual anchors; [5] identifies divergent-thinking steps and intervenes in MCoT decoding. These are relevant representation/decoding interventions, whereas VE-LC focuses on local regeneration after detected errors and rollback, using a recursively computed logit residual and correction-specific supervision. Reference numbers follow the review.
W5. Typographical error. Thank you for identifying this error. The heading at Line 247 should read "EXPERIMENTS"; the extraneous "4,500" will be removed.