acceptodds
Under review as a conference paper at ICLR 2027

HT-SCD: Self-Contrastive Decoding via Hallucination Trajectories for Mitigating Object Hallucinations in LVLMs

Abstract

Large vision-language models (LVLMs) demonstrate strong capabilities in image captioning, visual question answering, and multimodal reasoning, yet remain prone to object hallucinations. Existing mitigation methods often require additional training or external detectors, or incur substantial GPU memory overhead. We introduce HT-SCD, a training-free self-contrastive decoding method that uses hallucination trajectories to detect and mitigate object hallucinations online during autoregressive generation, without external models. Whereas conventional contrastive decoding methods construct a generic hallucination-prone negative branch in advance, HT-SCD first identifies specific hallucinations in the generated content and then uses the resulting hallucination trajectories to perform targeted rollback and correction through contrastive decoding. HT-SCD addresses the limited specificity and high computational cost of conventional contrastive decoding, and substantially outperforms existing contrastive decoding methods on MSCOCO-CHAIR and AMBER-G, two benchmarks for hallucination evaluation in open-ended visual generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-Contrastive Decoding, Training-Free Hallucination Mitigation, Large Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_dngVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes HT-SCD, a training-free decoding framework for mitigating object hallucinations in large vision-language models. The method first detects potentially hallucinated object words using gradient-weighted similarity between object-word representations and visual-token representations, and then rolls back the generated sentence when a hallucination is detected. The predictive distributions associated with detected hallucinated objects are subsequently used as negative signals for contrastive regeneration. The method is evaluated on MSCOCO-CHAIR and AMBER across three LVLMs.

Critical strengths
  1. The paper studies an important and practically relevant problem, i.e., reducing object hallucinations without additional training or external detection models.

  2. The proposed method shows fairly strong results on the reported hallucination benchmarks. In particular, HT-SCD improves CHAIR-related metrics consistently across three LVLMs.

  3. I also appreciate that the authors include several ablations on the detector and the choice of negative branch.

Critical weaknesses
  1. The paper is not sufficiently polished. The Introduction is very short, and the problem setup and notation are not clearly established before the method is introduced.

  2. I am not convinced by the novelty. Contrastive decoding is already a very crowded direction in this field, including methods based on internal states and different negative branches. HT-SCD seems to mainly change how the negative signal is constructed and when regeneration is triggered, and I think the authors need to justify more clearly what is fundamentally new here.

  3. The method description is difficult to follow. The interaction between hallucination detection, sentence-level rollback, hallucination trajectories, and regeneration is not explained cleanly, and I found the exact algorithm hard to reconstruct from the text.

  4. The preliminary analysis is quite scattered. The transition from perturbation, to intermediate-layer observations, to gradient attribution is not very well motivated. I also think this section should better position the method against closely related work such as Devils in Middle Layers [1] and Hallucinatory Image Tokens / EAZY [2].

  5. The experimental evaluation is incomplete. The main results focus on hallucination benchmarks, but there is no evaluation showing that the intervention preserves general visual understanding or reasoning ability. The formatting of Tables 2 and 3 is also confusing because only the authors' method is bolded, rather than consistently highlighting the best results.

  6. The efficiency analysis is limited to one model and one hardware setting.

[1] Jiang Z, Chen J, Zhu B, Luo T, Shen Y, Yang X. Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2025:25004-25014. doi:10.1109/CVPR52734.2025.02328.

[2] Che L, Liu TQ, Jia J, Qin W, Tang R, Pavlovic V. Hallucinatory image tokens: A training-free EAZY approach to detecting and mitigating object hallucinations in LVLMs. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2025:21635-21644. doi:10.1109/ICCV51701.2025.02009.

Questions for authors
  1. Could the authors clarify what exactly constitutes a “hallucination trajectory”? From the current description, it appears to be mainly the collection or aggregation of logits at detected hallucinated object positions, and I am not fully clear why this should be viewed as a trajectory.

  2. How the object tokens are distinguished from the other tokens?

  3. How sensitive is the method to the hallucination detection threshold and the selected intermediate layer? Are these hyperparameters shared across models, or tuned separately for each model?

  4. Since rollback and regeneration directly modify the generation process, have the authors evaluated whether the method hurts general VQA, reasoning, or caption quality beyond hallucination-specific metrics?

  5. What does boldface in Tables 2 and 3 denote? Does it indicate the best result, or simply the proposed method?

Reviewer_i2cEOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

HT-SCD proposes a training-free approach to mitigating object hallucinations in large vision-language models. After generating a sentence, it detects potentially hallucinated object mentions using gradient-weighted similarity between intermediate textual and visual representations. When hallucinations are detected, it rolls back to the sentence boundary and regenerates using contrastive decoding, with cached logits from hallucinated positions serving as the negative signal. Experiments evaluate detection, hallucination mitigation, localization, and computational overhead across three models.

Critical strengths
  1. The method reduces object hallucination across three model architectures and two benchmarks. Transferring hyperparameters from MSCOCO to AMBER-G without retuning provides useful evidence of robustness.

  2. Table 4 keeps the detector and rollback procedure fixed while comparing alternative correction strategies, including resampling, token-only penalties, and other contrastive branches. These comparisons support the usefulness of the cached distributional signal.

Critical weaknesses
  1. The conceptual novelty is limited relative to established approaches. The overall pipeline combines several familiar ideas. Generation rollback is already used in OPERA, and hallucination detection through intermediate image–text embedding similarity is central to ContextualLens. Training-free contrastive correction is also well established in VCD.

  2. HT-SCD contributes specific modifications: single-layer gradient weighting for detection and a negative branch constructed from cached logits at detected hallucinated positions. These distinctions are real, but their scope appears incremental; the paper does not sufficiently establish a new decoding principle or a substantially new understanding of hallucination.

  3. A full logit vector at a hallucinated position contains preferences for many tokens, including potentially valid alternatives and context-dependent continuations. The paper applies this historical vector throughout regeneration, even as the prefix changes. It remains unclear what makes this vector a transferable signal of hallucination, and when subtracting it might suppress useful content.

Questions for authors
  1. What information in the cached logit vectors is specifically associated with hallucination, beyond the identity and context of the flagged object?

  2. How often does sentence-wide contrastive correction remove valid details or introduce other factual errors? Can the authors evaluate the factuality and informativeness of the complete regenerated descriptions?

Reviewer_4ebMOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

HT-SCD is a training-free decoding loop for LVLMs: generate a sentence, detect hallucinated objects, roll back, and regenerate. Object words are identified by vocabulary matching and scored by a weighted cosine similarity between the layer-16 hidden state of the object word, hwh_w, and the visual hidden states viv_i. Each visual token is weighted by Ii=∥∂P/∂hin,il∥2I_i = \lVert \partial P / \partial h^{l}_{\mathrm{in},i} \rVert_2, where PP is the target-word probability read out at layer 16 through the final norm and LM head (logit lens), with backpropagation restricted to that single layer. A word is flagged as hallucinated if ssim<τs_{\mathrm{sim}} < \tau. The sentence is then rolled back and regenerated with Lnew=(1+λ)Lnorm−λLhalluL_{\mathrm{new}} = (1+\lambda) L_{\mathrm{norm}} - \lambda L_{\mathrm{hallu}}, where LhalluL_{\mathrm{hallu}} is the sum of the logit vectors at the flagged positions. The paper reports large CHAIR and AMBER-G reductions on LLaVA-1.5-7B, Qwen2.5-VL-7B and InstructBLIP-7B, a detection AUROC of 81.2, and 1.53x decoding latency.

Critical strengths
  1. Clear motivation. Tying the negative branch to the hallucination the model actually produced, rather than to a generic perturbation, is a sensible idea and is stated clearly.
  2. Consistent gains. Hallucination reductions hold across three architectures and two benchmarks, and the AMBER-G results reuse the COCO hyperparameters without retuning.
  3. Useful ablations. Table 4 swaps negative branches inside a fixed framework (VCD, SID, AGLA, Resample, TokenOnly, RandomPos), which fairly isolates the correction signal. Resample (38.8) vs. HT-SCD (29.4) shows that the contrastive term adds substantially beyond rollback alone.
  4. Candid analysis. The cost analysis (Table 8) and the failure cases (Appendix F) are honest and informative.
Critical weaknesses

W1. Overlap with GACD.

GACD [1] scores each visual token by the norm of a first-order (Taylor) gradient, links visual tokens to previously generated nouns, and decodes with z^m=(1+αm)zm⋆−αmzmo\hat{z}_m = (1+\alpha_m) z^{\star}_m - \alpha_m z^{o}_m. This is the same functional form as the HT-SCD update, with a negative branch tied to the model's own earlier object mentions. GACD's first-order Taylor justification is essentially the paper's Appendix B. GACD is neither cited nor compared against.

W2. The necessity of gradient attribution is not established.

The paper argues against attention using raw attention weights (Tables 1 and 6), not the attention contribution [2, 3]. Because both the paper's perturbation and its gradient are restricted to a single layer, the contribution is equivalent to perturbation in that setting.

Let xj=N(hj)x_j = \mathrm{N}(h_j), zjh=xjWVhWOhz^{h}_j = x_j W^{h}_V W^{h}_O, aph=∑jApjhzjha^{h}_p = \sum_j A^{h}_{pj} z^{h}_j, and up=hp+∑haphu_p = h_p + \sum_h a^{h}_p at the prediction position p=t−1p = t-1 of layer ll. Since only position pp of layer ll is read out, a visual token ii affects the output only through row pp of the attention.

Occlusion equals contribution (exact). Removing the value of token ii gives

Δup=−∑hApihzih=−ci,∥Δup∥2=Ci,\Delta u_p = -\sum_h A^{h}_{pi} z^{h}_i = -c_i, \qquad \lVert \Delta u_p \rVert_2 = C_i,

which is the attention contribution of [2], also used for visual tokens in LVLMs by [3].

Same ranking at the output. To first order, Δhout,pl≈(I+JFFN)Δup\Delta h^{l}_{\mathrm{out},p} \approx (I + J_{\mathrm{FFN}}) \Delta u_p and ΔP≈⟨γ,Δup⟩\Delta P \approx \langle \gamma, \Delta u_p \rangle with γ=∂P/∂up\gamma = \partial P / \partial u_p. Both maps depend only on pp and are shared by all visual tokens, so perturbation ranks tokens by CiC_i (or by ⟨ci,γ⟩\langle c_i, \gamma \rangle).

The gradient is also attention-gated. Differentiating through the same layer gives

∂P∂hi=JN(hi)⊤∑hApih[WVhWOh,γ+⟨zih−aph,γ⟩,κih],\frac{\partial P}{\partial h_i} = J_{\mathrm{N}}(h_i)^{\top} \sum_h A^{h}_{pi} \Big[ W^{h}_V W^{h}_O \\, \gamma + \langle z^{h}_i - a^{h}_p, \gamma \rangle \\, \kappa^{h}_i \Big],

where κih\kappa^{h}_i is the gradient of the attention score spihs^{h}_{pi} with respect to xix_i. Every term carries the factor ApihA^{h}_{pi}.

Raw attention ignores ∥zih∥\lVert z^{h}_i \rVert and therefore over-scores attention sinks (high attention, small value) [2, 3], so its weak results say nothing about CiC_i. In Tables 6 and 7, perturbation matches or beats the gradient (PG-Box 0.564 vs. 0.536; R@5% 0.462 vs. 0.367) and is rejected only for its cost (45.9 s per object). By the identity above, single-layer occlusion is closed-form and costs a single forward pass.

W3. The correction signal is over-broad.

LhalluL_{\mathrm{hallu}} is a fixed set of logit vectors taken from the noun slots that produced the flagged words. During regeneration,

pnew(y)∝exp⁡((1+λ)Lnorm(y)−λLhallu(y))p_{\mathrm{new}}(y) \propto \exp\big( (1+\lambda) L_{\mathrm{norm}}(y) - \lambda L_{\mathrm{hallu}}(y) \big)

is applied at every step of the regenerated sentence, with no plausibility constraint such as the adaptive plausibility constraint of VCD [4]. Each of these vectors ranks every token that was plausible in that slot, including objects actually present in the image, and the penalty grows linearly with the number of flagged words. The paper's own results suggest that this matters:

  • TokenOnly (penalizing only the flagged word) has higher recall than HT-SCD: 77.1 vs. 75.9 (base 76.1).
  • Recall drops on Qwen (58.9 to 55.8) and InstructBLIP (73.6 to 71.8), and AMBER Cover drops on all three models.
  • Figure 6: recall falls from 76.1 to 70.8 as λ\lambda grows.
  • Appendix F.3: cups and plates are present in the image, yet neither survives the rollbacks.

Detector false positives (Appendix F.2) make this worse, since the negative branch can then contain the distribution of a real object. The gap between TokenOnly and the full negative could come from suppressing synonyms and co-occurring hallucinations (desirable) or from suppressing grounded alternatives (undesirable), and the current experiments cannot distinguish the two.

W4. No general-purpose benchmarks.

The evaluation covers only hallucination benchmarks, so it is unclear whether rollback and contrastive regeneration harm general capability. Results on general benchmarks such as MME [5], MMBench [6], MM-Vet [7] or SEED-Bench [8] are needed.

Questions for authors
  1. LhalluL_{\mathrm{hallu}} is underspecified: which position (t−1t-1?), which layer (the final one?), and how are object words that span several tokens handled? The same multi-token question applies to idtarget\mathrm{id}_{\mathrm{target}} in PP.
  2. Do the negatives accumulate across the nn regeneration attempts, or are they replaced? The oscillation in Figure 10 suggests that they are replaced.
  3. Why should the cosine similarity between a text hidden state and visual hidden states measure grounding? τ\tau is 0.19 for LLaVA but 0.58 for Qwen, which suggests that the score is not calibrated. How was 0.58 chosen?
  4. The attribution is taken at position t−1t-1, but the similarity uses hwh_w at position tt. Why this mismatch, and do the results change when a single consistent position is used?
  5. How does HT-SCD compare with GACD [1] under the same models and benchmarks?
  6. Within the same detector, how do uniform weights, raw attention ApiA_{pi}, the attention contribution CiC_i [2, 3], and the gradient compare on detection (Table 1), mitigation (Table 4) and latency? Does closed-form single-layer occlusion match the localization of the gradient (Tables 6 and 7)?
  7. How many ground-truth objects present in the original sentence are lost after regeneration? How does HT-SCD perform with top-kk or plausibility-filtered negatives [4], and with results split by detector true vs. false positives?

References

[1] S. Wang et al. Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection. CVPR, 2026.

[2] G. Kobayashi, T. Kuribayashi, S. Yokoi, and K. Inui. Attention is Not Only a Weight: Analyzing Transformers with Vector Norms. EMNLP, 2020.

[3] S. Jha and J. Kim. CAPA: Contribution-Aware Pruning and FFN Approximation for Efficient Large Vision-Language Models. Findings of ACL, 2026.

[4] S. Leng et al. Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding. CVPR, 2024.

[5] C. Fu et al. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv:2306.13394, 2023.

[6] Y. Liu et al. MMBench: Is Your Multi-modal Model an All-around Player? ECCV, 2024.

[7] W. Yu et al. MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities. ICML, 2024.

[8] B. Li et al. SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension. CVPR, 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.