acceptodds
Under review as a conference paper at ICLR 2027

Measuring Alignment Collapse: A Training-Free Framework for Object Hallucination in Vision-Language Models

Abstract

Object-existence hallucination is a major failure mode of vision-language models, yet it is usually treated as a detection problem: design a scorer and report its AUROC. We argue for a different framing—measurement: what should be recorded from a cross-modal alignment matrix so that grounding failure becomes decidable? We cast object-existence hallucination as cross-modal alignment collapse: the patch–token similarity matrix of a frozen CLIP encoder fails to concentrate its evidence on the queried object. Reading that collapse requires two complementary families. The singular-value spectrum is a maximal invariant under patch- and token-orthogonal transformations—it exhausts every position-free reading, but is provably blind to which token columns carry the evidence. The Patch–Object Alignment Score (POAS) supplies the positional information the spectrum cannot. Conversely, POAS is inert to background spectral complexity. Together, the two families give a block-diagonal response in the idealized model, so the pair identifies both underlying parameters while either family alone leaves one unidentified. We instantiate this as , a training-free measurement that maps an (image, object query) pair to a deterministic 168-dimensional descriptor (90 spectral, 78 POAS) using a frozen CLIP-ViT-L/14. 's coordinates are constructed without any label; a classifier is a separate downstream consumer, trained on a small labeled set and kept apart from the measurement. On POPE, achieves AUROC against for the strongest training-free baseline (paired bootstrap ), and on a discriminative CHAIR construction. Ablations align with the theory: removing the spectral family costs 1.97 AUROC points; removing POAS costs 0.82. This work targets only object-existence hallucination; attribute and relation hallucinations remain open.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.