acceptodds
Withdrawn from ICLR 2027

Beyond the Current Prediction: Tracking Decoding Trajectories for Object Hallucination Detection in LVLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large vision-language models (LVLMs) can confidently generate objects that are plausible under the generated context but absent from the image. Existing training-free object hallucination detectors mainly score each generated object using signals available at its current generation step. However, autoregressive generation can obscure uncertainty revealed earlier in the decoding process. At each step, the model assigns a probability to the generated token, but once the token is selected, this probability is no longer retained in subsequent decoding. Later predictions instead condition on the selected token itself. Consequently, a token generated with low confidence can shape the subsequent context and make a later hallucinated object appear locally confident despite uncertainty observed earlier in the generation trajectory. Motivated by this observation, we ask whether retaining uncertainty signals from preceding decoding steps can improve object hallucination detection. We introduce History-Aware Risk Tracking (HART), a simple training-free method that estimates hallucination risk using both the current object prediction and uncertainty observed throughout the preceding generation process. Specifically, HART considers both the current object prediction and uncertainty accumulated over preceding decoding steps, with greater weight assigned to more recent steps. We use negative log-likelihood as a simple token-level measure of prediction uncertainty. This formulation retains information that would otherwise no longer be directly available as generation proceeds, without requiring additional training or auxiliary generation. Experiments across multiple LVLMs on MSCOCO and Pascal VOC show that incorporating decoding history improves over current-step scoring and achieves strong object hallucination detection performance compared with existing training-free methods. Our results show that uncertainty from preceding decoding steps provides a complementary signal for object hallucination detection beyond the current object prediction alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision language model, object hallucination detection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gtEQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper points out that existing training-free object hallucination detection methods for LVLMs mainly rely on signals from the current generation step. To address this, the authors define a historical uncertainty term, computed as an exponentially decayed weighted average of the NLLs of preceding tokens multiplied by (t−1), and combine it with the NLL of the object’s first token to detect object hallucinations.

Critical strengths
  • The method is simple and efficient, it requires no additional training or auxiliary generation and can be updated online with constant memory.
  • The evaluation covers a wide range of models across diverse architectures and scales.
  • The motivating analysis showing that uncertainty in earlier generated tokens is associated with hallucination is insightful.
Critical weaknesses

The core contribution is unclear.

  • The proposed score consists of three components: (1) the current NLL, (2) the decoding position, and (3) the history uncertainty. The ablation in Table 4 is insufficient to determine the contribution of each component, and a substantial portion of the performance appears to come from the position term.

The experiments do not sufficiently support the main claim.

  • The ℓ + log(t−1) variant is missing from the ablation in Table 4, so the main claim is not sufficiently supported.
  • In Figures 2 and 6, the history-augmented risk appears to be a simple vertical shift of the instantaneous risk.
Questions for authors
  • Could the authors also evaluate the method on recent LVLMs such as Qwen3.8?
  • Comparing HART with the position-only baseline on short responses would help verify the core contribution.
Reviewer_AZr2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes History-Aware Risk Tracking (HART), a training-free method for detecting object hallucinations in large vision language models. HART combines uncertainty from preceding decoding steps with the current object token's negative log-likelihood to keep more information in the process. The method does not require auxiliary generation or model training. Experiments across eight LVLMs on MSCOCO and Pascal VOC demonstrate consistent improvements over several training-free baselines. Additional ablations investigate the contributions of current and historical uncertainty, decoding position, and decay parameters.

Critical strengths
  • HART offers a practical way to use decoding history in hallucination detection without additional training, auxiliary generation or access to intermediate model representations. It follows a clear motivation and is kept simple.
  • The evaluation covers eight LVLMs across different architectures and scales, and compares against six other methods
Critical weaknesses
  • My main concern is in Table 4, where the position only baseline achieves 83.73/85.40 AUROC, compared to 86.13/85,98 for HART. Since the HART formulation explicitly includes a position dependent term through the mass-preserving normalization, this might be the most important factor for the demonstrated performance improvements (supported by the substantial performance drop when normalizing by prefix length). While the authors investigate this issue, a baseline jointly combining current NLL and decoding position is missing. Consequently, it remains unclear how much additional predictive information the historical uncertainty actually provides.

  • Despite evaluating a diverse range of LVLMs, the experiments are restricted to image-description generation on two datasets using a single fixed prompt. Since HART accumulates uncertainty across preceding tokens, its effectiveness may depend on response length, generation style, and the structure of the generated text. It is therefore unclear whether the observed improvements generalize to different prompts, shorter responses, or other LVLM tasks such as visual question answering. This is partially mentioned in Section 5.4, but especially an evaluation of robustness across different prompts would have been insightful.

Questions for authors
  • Could the authors clarify the discrepancy between the NLL results in Tables 2 and 3?
  • How sensitive is HART to response lengths? Does its advantage persist across different generation settings?
  • Could the authors report confidence intervals (or another indicator for significance) for the improvements over the strongest baselines?
Reviewer_sE8cOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This work proposes a simple training-free method to estimate hallucination risk with current object prediction and uncertainty for LVLMs. Uncertainty was predicted with neg log-likelihood. Experiments demonstrates its benefit over LVLMs on MSCOCO, Pascal VOC.

Critical strengths

The observation and the idea for the remedy in this manuscript look reasonable and simple (in a good way).

Critical weaknesses

I have major concerns on this manuscript as below:

  1. This work did not cite recent works on hallucinations and unfortunately, some of them are critical that can potentially undermine the novelty of this current work significantly.
  2. This work was only evaluated on limited datasets with limited metrics for object hallucinations. Recent works may guide the authors regarding it.
  3. Observation - proposed method logic may require further analysis. The observation in this work could be a single reason, but may not be comprehensive. I also recommend to investigate recent works below. There are a number of different reasons and causes that were discovered by other works and this work did not even cite them. Thus, in its current form, it is hard to justify the contribution of this work over prior arts.
Questions for authors

Please see the above comments. I put some recent refs here. Note that this is not even a complete list. The first work is the most related one in my view, which is more comprehensive than the current manuscript:

  1. Zhongxing Xu et al., Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decoding, CVPR 2026
  2. Tsung-Han Wu et al., Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling, NeurIPS 2025
  3. Jinhao Duan et al., TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Intervention, ICCV 2025
  4. Yixiong Fang et al., Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding, NeurIPS 2025
  5. Wenzheng Song et al., When Diffusion Language Models Hesitate: Detecting and Correcting Visual Hallucinations via Confidence Fluctuation, ICML 2026
  6. Hao Fang et al., Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs, NeurIPS 2025
  7. Hoigi Seo et al., On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Models, NeurIPS 2025

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.