- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper ask whether hallucination probes that works before a yes/no answer still work before free-form caption. To compare the two output format, it keep the images, the verified-absent objects, and the frozen models the same. Across three VLMs, the probes, tested on held-out images, does well on forced-choice answers but stay close to chance on free captions. The authors then look into this gap with intermediate prompts, permutation test, and alternative probes.
Critical strengths
1.The paired comparison get at an important mismatch in how these models is evaluated. Because the images and absent targets stays fixed, the gap between forced and free formats tell us more than comparing two unrelated benchmark would. Using human verification also mean the incomplete COCO annotations aren't treated as ground truth. 2.The main result is backed by raw AUROC, permutation test, and several readout control, so it don't rest entirely on recovery metric the authors propose. The image-level oracle and the prompts that names the object also deals with the worry that a generic caption prompt never tell the model which target is being scored.
Critical weaknesses
1.The authors admits the current matcher make mistakes with negation, especially in conditions where the object is named. Unless those output are audited, it's hard to say how much of the gap in the intermediate formats come from the output being more open-ended and how much comes from labeling error. This mostly matter for how the gap is explained. 2.Each seed have only 21 to 35 positive outcome, and every probe are trained on just one sampled outcome per query. The paper is careful about its claims, but it don't fully rule out the possibility that signal is there and just hard to learn from so few positive label, rather than missing altogether. The 16-decode propensity estimates the authors already have would be direct way to check this.
Questions for authors
1.Could you audit the current matcher on outputs that was actually scored, especially in named-object conditions? And if you fix the errors it make, does that changes any of the comparisons between format? 2.Suppose the probe is trained to predict empirical propensity from the 16 decode, but evaluation still use the independent outcome decode. Does performance on free captions still stays close to chance in that case?
Response to Reviewer 6XZq: matcher audit and propensity-trained probes
Thank you for the positive review.
Q1 and W1. We audited the matcher on the scored outputs, with two annotators who did not see which rule had flagged each item. As you suspected, it counted negations after the object name as events in the named formats (precision 8-14%, and 12% in free captions). We relabeled every free-form output with a validated LLM judge, and in the named formats both annotators checked every positive label (3,059 mentions). Correcting the labels changes the intermediate formats but not the main gap. On the same decodes, the one-sentence answer goes from 20% to 87% (Qwen2.5-VL) and from 78% to 94% (InternVL3), and the named caption from 6% to 63% and from 6% to 42%. Forced choice (84-97%) and the free caption (11-18%) stay far apart, with paired refit intervals above zero in all three models. Both target naming and output openness contribute to the gap, and the revised Sections 4.2 and 4.3 will measure each.
Q2 and W2. Trained on the 16-decode propensity and scored on the independent decode, the probe reaches 18-35% in free captions and 96-98% under forced choice, so few positives do not explain the gap. Two annotators also verified 868 more absent pairs, which raises free-caption events to 38-48 per seed with the same result.