acceptodds
Under review as a conference paper at ICLR 2027

Pre-Generation Hallucination Probing Works Only Under Output-Space Constraint

Abstract

Probes that read a vision-language model's hidden states before any token is generated have been reported to predict hallucination with AUROC up to 0.93. These results come mostly from yes/no questions, while object hallucination is usually measured in free-form captions. We test whether the same probes work in both settings by holding the model, image, queried object and sampling fixed and changing only the instruction. The endpoint is whether the model asserts one object that two annotators verified to be absent. For each query we estimate the model's propensity to hallucinate from 16 sampled decodes, score the probe on an independent decode, and report recovery, the probe's gain in AUROC over chance divided by the gain of an oracle that knows each query's propensity. Across three vision-language models and three decoding seeds, probes recover 84 to 97% under a forced yes/no format but only −14 to 6% in free captions, with upper 95% bounds of 0 to 22%. A permutation test, a sweep over layers and token positions, and nonlinear probes all support this null. Naming the object in an otherwise open prompt restores part of the signal in one of the three models. Under forced choice the signal appears soon after the object's name, before the prompt ends, and rises sharply in the middle layers, which suggests that the probe reads an answer the model has already settled on. A human audit also shows that a CHAIR-style labeler has a precision of 0.24 on the objects it flags.

Primary area:
interpretability and explainable AI
Keywords:
hallucination, vision-language models, linear probing, interpretability, object hallucination, evaluation validity, multimodal models, hidden state analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6XZqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper ask whether hallucination probes that works before a yes/no answer still work before free-form caption. To compare the two output format, it keep the images, the verified-absent objects, and the frozen models the same. Across three VLMs, the probes, tested on held-out images, does well on forced-choice answers but stay close to chance on free captions. The authors then look into this gap with intermediate prompts, permutation test, and alternative probes.

Critical strengths

1.The paired comparison get at an important mismatch in how these models is evaluated. Because the images and absent targets stays fixed, the gap between forced and free formats tell us more than comparing two unrelated benchmark would. Using human verification also mean the incomplete COCO annotations aren't treated as ground truth. 2.The main result is backed by raw AUROC, permutation test, and several readout control, so it don't rest entirely on recovery metric the authors propose. The image-level oracle and the prompts that names the object also deals with the worry that a generic caption prompt never tell the model which target is being scored.

Critical weaknesses

1.The authors admits the current matcher make mistakes with negation, especially in conditions where the object is named. Unless those output are audited, it's hard to say how much of the gap in the intermediate formats come from the output being more open-ended and how much comes from labeling error. This mostly matter for how the gap is explained. 2.Each seed have only 21 to 35 positive outcome, and every probe are trained on just one sampled outcome per query. The paper is careful about its claims, but it don't fully rule out the possibility that signal is there and just hard to learn from so few positive label, rather than missing altogether. The 16-decode propensity estimates the authors already have would be direct way to check this.

Questions for authors

1.Could you audit the current matcher on outputs that was actually scored, especially in named-object conditions? And if you fix the errors it make, does that changes any of the comparisons between format? 2.Suppose the probe is trained to predict empirical propensity from the 16 decode, but evaluation still use the independent outcome decode. Does performance on free captions still stays close to chance in that case?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6XZq: matcher audit and propensity-trained probes

Thank you for the positive review.

Q1 and W1. We audited the matcher on the scored outputs, with two annotators who did not see which rule had flagged each item. As you suspected, it counted negations after the object name as events in the named formats (precision 8-14%, and 12% in free captions). We relabeled every free-form output with a validated LLM judge, and in the named formats both annotators checked every positive label (3,059 mentions). Correcting the labels changes the intermediate formats but not the main gap. On the same decodes, the one-sentence answer goes from 20% to 87% (Qwen2.5-VL) and from 78% to 94% (InternVL3), and the named caption from 6% to 63% and from 6% to 42%. Forced choice (84-97%) and the free caption (11-18%) stay far apart, with paired refit intervals above zero in all three models. Both target naming and output openness contribute to the gap, and the revised Sections 4.2 and 4.3 will measure each.

Q2 and W2. Trained on the 16-decode propensity and scored on the independent decode, the probe reaches 18-35% in free captions and 96-98% under forced choice, so few positives do not explain the gap. Two annotators also verified 868 more absent pairs, which raises free-caption events to 38-48 per seed with the same result.

Reviewer_cvNCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether pre-generation hallucination probes, previously shown to perform well on yes/no questions, generalize to open-ended image descriptions. The authors construct a controlled evaluation using 300 COCO images and 1,182 verified image–absent-object pairs, varying only the instruction format while keeping the model, image, target object, and sampling procedure fixed. They evaluate linear probes trained on pre-generation hidden states across six prompting formats and three vision-language models, using an oracle-normalized AUROC metric to account for differences in hallucination rates. The results show strong probe performance under forced-choice questions (84–97% recovery) but substantially weaker performance for free captions (−14% to 6%). Additional analyses examine different layers, token positions, nonlinear probes, and statistical significance.

Critical strengths

The paper is well organized, and the main findings are easy to follow. The paper addresses an important limitation of existing hallucination probes: whether their strong performance on constrained yes/no evaluations generalizes to open-ended generation.

Critical weaknesses

The central conclusion is not fully disentangled from target-object information availability. Under the generic free-caption prompt, the pre-generation hidden state is identical for all target objects associated with the same image. Consequently, the probe has no explicit information about which object is being evaluated. This introduces a fundamental difference from the forced-choice setting beyond the generation format itself. The authors partially address this through target-aware prompting, where performance remains substantially below forced choice. However, the results are mixed across models, and the authors acknowledge potential labeling issues in this condition. Furthermore, the image-level analysis uses a score averaged over target objects, apparently including the evaluated target itself. A leave-one-target-out analysis and more reliable labeling of the target-aware condition would help determine whether the observed degradation reflects the inherent difficulty of predicting hallucinations during open-ended generation or the absence of target-specific information.


Statistical uncertainty and labeling reliability require further validation. The free-caption setting contains relatively few positive hallucination events (21–35 per seed), making the estimates sensitive to individual examples. The reported confidence intervals are based on fixed out-of-fold predictions and do not fully account for variability from probe fitting or oracle-ceiling estimation. This raises concerns about the reliability of the quantitative bounds, particularly given the negative recovery values observed in some layer analyses. Additionally, the reported human audit showing a precision of 0.24 concerns an earlier labeling method rather than the matcher used for the main experiments. Since the central finding depends on identifying relatively rare hallucination events in free captions, validating the final labeling procedure is important.

Questions for authors
  1. Could the authors evaluate a leave-one-target-out version of the image-level score, excluding the evaluated object from the propensity average? How does probe performance compare against this baseline? This would clarify whether the image-level signal captures general hallucination propensity or benefits from information about the specific target.

  2. How robust are the free-caption results to stricter object matching, such as word-boundary matching or LLM-based verification? Could the authors provide a human audit of the final matcher used in the main experiments, particularly for the generic and target-aware caption conditions?

  3. How do the confidence intervals change when probe fitting is repeated within each bootstrap sample, rather than reweighting fixed predictions? Could the authors also account for uncertainty in the oracle ceiling?

  4. Under forced-choice prompting, how does a probe compare with directly using the model’s probability of answering “yes” at the final pre-generation token? This would help establish the additional value of hidden-state probing over a simpler baseline.

  5. Have the authors considered training probes to predict hallucination propensity rather than a single sampled binary outcome? Would using the estimated propensity, including an image-level target for generic captions, change the main conclusions?

4 comments
AuthorsOpenReview ↗

Response to Reviewer cvNC: target information, label validation and refit intervals

Thank you for the careful review. We ran all the analyses you suggested. The numbers below use the paper's own decodes and corrected labels (Q2).

W1 and Q1 (target information). You are right that the image-level score in Section 4.3 included the evaluated target. With the target left out, that score drops to AUROC 0.44-0.54 with the paper's labels and 0.53-0.63 with corrected labels, which is the level our free-caption probe reaches (0.54-0.62). Once the probe gets target information, the signal comes back. The forced-choice state of the same image and object predicts that object's hallucination in the free caption at 72-76% recovery in all three models, while object identity alone gives 12-30%. Naming the object in the prompt also restores much of it: 87-94% for the one-sentence answer and 42-63% for the named caption (Qwen2.5-VL, InternVL3). The free caption therefore fails mainly because its prefill state carries no target information, and a long open output adds a further loss. We will rewrite Sections 4.2 and 4.3 around this in the revised PDF.

W2 and Q2 (labels). Two annotators audited the matcher on the scored outputs. It over-counted events (precision 12% in free captions and 8-14% in the named formats), for example through substring hits such as "cat" in "indicating" and negations after the object name. We relabeled all free-form outputs with an LLM judge validated against the annotators (precision 87% in free captions), and in the named formats both annotators checked every positive label (3,059 mentions, agreement 96-100%). Forced-choice labels are parsed answers and do not change.

W2 and Q3 (events and intervals). Two annotators verified 868 more absent pairs against the images (kappa 0.71), which raises free-caption events to 38-48 per seed. Refit intervals that also resample the ceiling: forced choice 84% [31, 98], 97% [94, 99] and 96% [86, 98]; free caption 18% [-13, 31], 11% [-26, 40] and 11% [-23, 36]; paired difference [18, 101], [57, 123] and [58, 120] points.

Q4 (P(yes)). Under forced choice, P(yes) recovers 98-100% (probe 84-97%). This supports Section 5.3: the forced format makes the model settle its answer while it reads the prompt, and the probe reads that answer. A caption prompt leaves no such answer pending, which is why the result does not carry over.

Q5 (propensity). Trained on the 16-decode propensity, the probe reaches 96-98% under forced choice and 18-35% in free captions, so the gap remains. Because the state is the same for all targets of an image, this regression already fits the image-level propensity you mention.

Thank you for the rebuttal. The refitted intervals and propensity-trained probes address my statistical questions, and the gap between forced choice and free captioning remains.

However, the new results substantially change the paper’s central claim. The image-level score drops to near chance when the evaluated target is excluded, while explicitly naming the target recovers much of the signal. This suggests that missing target information, rather than the output format itself, explains the gap. The current title, abstract, and Section 4.3 do not reflect this finding. Moreover, the corrected labels affect all free-form results, requiring substantial revisions to the reported experiments.

I find the new explanation convincing, but it differs from the central claim of the submitted paper. These changes require a revised manuscript and further review, rather than clarification through rebuttal alone. I therefore maintain my rating of 2.

AuthorsOpenReview ↗

Revision reflects the new finding and revised pdf upload

Thank you for the follow-up, and we are glad you find the new explanation convincing. We have uploaded a revised manuscript that states it (changes in blue).

The central result holds, and its explanation now has two parts. With corrected labels, probes recover 84-97% under forced choice and 11-18% in free captions, and the paired differences (66 to 86 points) exclude zero in every model. A free caption combines a long open output with a prompt that names no target, and each of the two costs a large part of the signal:

Recovery (%), corrected labelsTarget namedOutputQwen2.5-VLLlama-3.2-VInternVL3
Forced choiceyesyes/no849796
One sentenceyesshort878194
Named captionyeslong634742
Free captionnolong181111
Free caption, read from the forced-choice stateyeslong767273

Naming the target raises recovery from 11-18% to 42-63% (paired difference 32 to 45 points), and with the target named, a long output lowers it from 81-94% to 42-63% (24 to 51 points; for Qwen2.5-VL the interval starts at 0). The last row shows that a state about the object predicts the free-caption hallucination itself, 58 to 62 points better than the caption's own state on the same events.

What changed in the manuscript. The new title is "Pre-Generation Hallucination Probes Need a Target: Yes/No Answers versus Free Captions", and the abstract and introduction state the two-part result. Sections 4.2 and 4.3 are rewritten around the table above, including the leave-one-target-out score. All free-form results, including Section 5, use the validated labels, Table 1 shows each corrected value next to the submitted one, and the text now reports a small free-caption signal rather than none. The label audits are in Section 5.4 and Appendix B.

Scope. The protocol, images, models, probes and statistics are unchanged, and so are the forced-choice and template results. Relabeling needed the decode texts, so the one-sentence and named-caption cells use the first seed's decodes, regenerated with the original settings, and the free caption uses caption runs whose texts were stored, on the 2,050 verified pairs. The named-late variant, which we did not regenerate, is removed.

Since your remaining concern was the need for a revised manuscript, we would be grateful if you could look at it and reconsider your rating.

I appreciate the authors' efforts in preparing the revision. The new claim is better supported, but substantial changes to the title, central claim, labels, target set, and decodes warrant a full review. Some issues remain - Table 1 compares different query sets, and the claim that most of the gap comes from missing targets is not fully supported. I therefore maintain my rating of 2 but encourage the authors to resubmit the revised paper to another top-tier conference.

Reviewer_vBr4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies on pre-generation hallucination probes, which read a VLM’s hidden states before any token is generated. These probes have been reported to predict hallucination with AUROC up to 0.93. However, these results come mostly from yes/no questions, while object hallucination is usually measured in free-form captions. The authors conduct comprehensive experiments to test the probes in free-form captions. They observe that: (1) Across three vision-language models and three decoding seeds, probes recover 84 to 97% under a forced yes/no format but only −14 to 6% in free captions, with upper 95% bounds of 0 to 22%. A permutation test, a sweep over layers and token positions, and nonlinear probes all support this null. Naming the object in an otherwise open prompt restores part of the signal in one of the three models. (2) Under forced choice the signal appears soon after the object’s name, before the prompt ends, and rises sharply in the middle layers, which suggests that the probe reads an answer the model has already settled on. (3) A human audit also shows that a CHAIR-style labeler has a precision of 0.24 on the objects it flags.

Critical strengths
  • The paper is well-written with clear figures and tables.

  • The authors conduct comprehensive experiments on pre-generation hallucination probes, showing the failure of these probes in free captions.

  • They also do a few ablation and control explorations, such as a permutation test, a sweep over layers and token positions, and nonlinear probes.

Critical weaknesses
  • I appreciate that the authors conduct comprehensive experiments to show the failure of pre-generation hallucination probes, but the paper mainly focuses on observations, lacking in some deep analysis or insights.

    • First, why do these probes fail in free-form captions? The authors mainly show the failure but do little in explaining why this happens. A possible reason is that these probes only read the prefill token, the position from which the first output token is generated. VLMs may generate quite a few tokens before the target object. Thus, it might be very hard for the probe to predict future objects across a long token sequence. If we can understand the reasons behind, maybe we can design a better probe.

    • Second, given the failure of these probes in free captions, what should we do next? I think hallucination is very popular topic in LLMs/VLMs. There are many different methods and some of them are still useful in free captions. Therefore, I feel the easiest way is just to use another hallucination detection or mitigation method. This makes me feel the key observation is rather limited in the scope, now that we have so many alternatives.

  • The evaluation is based on 300 images from COCO val2017. I'm not sure if 300 images are enough to get robust evaluation results.

Questions for authors

I'm not an expert in hallucination detection so I appologize if I misunderstand or miss something important. I hope the authors can clarify the impact of the paper, for example: Are the pre-generation hallucination probes very important or popular? Or do most of people believe that these probes can work in free captions? What impact will the paper make on the practice in hallucination detection?

2 comments
AuthorsOpenReview ↗

Response to Reviewer vBr4: why probes fail on captions, what next, and sample size

Thank you for the review and the clear questions.

Why do the probes fail on free captions? We tested your explanation together with a second one. A generic caption prompt names no object, so the probe sees the same state for every object of an image and cannot tell which one will be hallucinated. When we give it a state about the object (the forced-choice prompt for the same image and object), it predicts that object's hallucination in the free caption at 72-76% recovery in all three models. Your explanation also holds in part: with the object named, a one-sentence answer keeps 87-94% of the signal and a long caption 42-63%. Missing target information is the main cause, and longer open outputs add to it.

What next? This gives a practical recipe. One prefill pass per candidate object ("Is there a {object} in the image?") predicts its hallucination in the caption before any text is written, and even the model's own P(yes) from this pass reaches 68-75% without training. Unlike post-hoc detectors, it can act before the error appears.

Are 300 images enough? The forced-versus-free gap is much larger than its uncertainty: paired bootstrap intervals that refit the probe are [18, 101], [57, 123] and [58, 120] points. Two annotators also verified 868 more absent pairs (2,050 in total), and on 2,000 images (Qwen2.5-VL, rule-based labels) the free-caption result stays small (12%).

Impact. Pre-generation probes are attractive because they are cheap and act before generation. HALP (Kogilathota et al., 2026) reports AUROC up to 0.93 across eight VLMs, but mostly on yes/no questions, while object hallucination is measured in captions. Our paper shows that this success does not transfer to captions, explains why, and shows how to make such probes work there.

Thank the authors for their response! My concerns have been addressed so I increased my score to 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers. Following their suggestions, we (1) audited the labels of the free-form formats with two annotators and relabeled them with a validated LLM judge, with both annotators checking every positive label in the named formats, (2) added refit bootstrap intervals, paired differences, 868 more verified-absent pairs, a P(yes) baseline and propensity-trained probes, and (3) tested the role of target information directly.

The main result holds: forced choice recovers 84-97% and the free caption 11-18%, and the paired difference is above zero in all three models. The new analyses also explain it. The generic caption prompt carries no information about the target, and a prefill state about the object predicts its hallucination in the free caption at 72-76% recovery. The revised PDF, which we will upload during the discussion, rewrites Sections 4.2 and 4.3 around this and uses the corrected labels for all free-form results, including Section 5. It also corrects the reported kappa of the target verification (0.77, raw agreement 99.6%) and the matcher description in Appendix B. The OpenReview title now matches the PDF. Results for the named formats of Llama-3.2-V will follow during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.