- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether visual-prefill hidden states, formed before a VLM observes the question, can predict subsequent answering failures. It introduces a reference-based geometric method that measures the angular distance between test representations and a “success reference cloud” constructed from usually correct samples. Experiments across six VLMs and eight benchmarks reveal strong task dependence: success–failure separation is pronounced on ScienceQA and ChartQA but weak on POPE and CV-Bench, and emerges within the first quarter of the visual sequence across multiple layers. The authors validate these findings using repeated-response majority labels, Hedges’ , image-group bootstrapping, comparisons with HALP, alternative distance metrics, deduplication analyses, and exploratory representation interventions.
Critical strengths
This paper presents a careful and relatively broad empirical study showing that hidden-state geometry during the visual prefill stage can distinguish downstream successes from failures on some tasks; evaluation across six VLMs and eight benchmarks is a clear strength.
Critical weaknesses
-
Limited methodological contribution and empirical gains: The method essentially constructs a reference bank from successful samples and computes standard angular distance/kNN. The comparison against HALP is restricted to ScienceQA and ChartQA on Qwen3-VL. Compared to HALP, AUROC improvements are merely 0.003 and 0.051, with 95% confidence intervals for both differences containing zero. Moreover, the geometric method additionally uses reference samples with correctness labels and features from multiple layers, thus failing to demonstrate a clear advantage over existing latent-state probes.
-
Core explanation lacks mechanistic evidence: The paper attributes task differences to the "incompressibility" of visual information, but neither operationalizes nor quantifies this concept, nor validates it through controlled experiments. Crucially, after directly reducing the target angular distance, no errors were corrected and no correct answers were altered across 64 ScienceQA inputs and 960 responses. Therefore, the observed geometric distance may merely be a dataset-specific statistical correlate rather than a behaviorally meaningful deficiency in visual representations.
Questions for authors
After matching the number of correctness labels, reference samples, and model layers, does the proposed geometric method still outperform HALP or other latent-state probes? If not, what is its substantive contribution over existing methods?