acceptodds
Under review as a conference paper at ICLR 2027

Measuring Alignment Collapse: A Training-Free Framework for Object Hallucination in Vision-Language Models

Abstract

Object-existence hallucination is a major failure mode of vision-language models, yet it is usually treated as a detection problem: design a scorer and report its AUROC. We argue for a different framing—measurement: what should be recorded from a cross-modal alignment matrix so that grounding failure becomes decidable? We cast object-existence hallucination as cross-modal alignment collapse: the patch–token similarity matrix of a frozen CLIP encoder fails to concentrate its evidence on the queried object. Reading that collapse requires two complementary families. The singular-value spectrum is a maximal invariant under patch- and token-orthogonal transformations—it exhausts every position-free reading, but is provably blind to which token columns carry the evidence. The Patch–Object Alignment Score (POAS) supplies the positional information the spectrum cannot. Conversely, POAS is inert to background spectral complexity. Together, the two families give a block-diagonal response in the idealized model, so the pair identifies both underlying parameters while either family alone leaves one unidentified. We instantiate this as , a training-free measurement that maps an (image, object query) pair to a deterministic 168-dimensional descriptor (90 spectral, 78 POAS) using a frozen CLIP-ViT-L/14. 's coordinates are constructed without any label; a classifier is a separate downstream consumer, trained on a small labeled set and kept apart from the measurement. On POPE, achieves AUROC against for the strongest training-free baseline (paired bootstrap ), and on a discriminative CHAIR construction. Ablations align with the theory: removing the spectral family costs 1.97 AUROC points; removing POAS costs 0.82. This work targets only object-existence hallucination; attribute and relation hallucinations remain open.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Object Hallucination Detection, Vision-Language Models, Training-Free Measurement, Cross-Modal Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7LRyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents an interesting perspective and promising empirical findings. However, the central theoretical claims contain inconsistencies, while the experimental comparisons lack sufficient fairness and generalization validation. Furthermore, the manuscript exhibits noticeable signs of insufficiently reviewed AI-assisted writing, including repetitive and overstated claims, inconsistencies between theoretical statements and proofs, and questionable figure interpretations. Although these issues do not establish that the figures were AI-generated, they raise concerns about the authors' quality control and careful verification of the manuscript. Overall, these shortcomings substantially undermine my confidence in the reliability and scientific rigor of this work.

Critical strengths
  1. The paper introduces Alignment Collapse Measurement (ACM), which characterizes object grounding through the structure of patch–token similarity matrices. The distinction between spectral information and object-specific positional alignment provides a potentially useful perspective beyond conventional scalar similarity scores.
Critical weaknesses
  1. The theoretical analysis contains several inconsistencies that undermine the paper's main contributions. In Theorem 1, the column-permuted matrix remains nonzero, contradicting the definition of object presence in Eq. (2) while being assigned the opposite label.
  2. ACM relies on a supervised gradient-boosted classifier, whereas the primary baselines use scalar similarity scores without comparable downstream classification capacity. The substantial performance drop when using a linear classifier further suggests that the classifier contributes significantly to the reported gains.
Questions for authors
  1. How do the authors reconcile the definition of object presence in Eq. (2), where (M\neq 0), with the opposite labels assigned to the two nonzero matrices constructed in Theorem 1?
  2. How is the discrete parameter (\kappa) continuously parameterized to justify the Jacobian and inverse function theorem in Theorem 2?
  3. Can the authors report image-disjoint cross-validation results and compare ACM against baselines using matched prompt budgets and comparable downstream classifiers?
Reviewer_oG5tOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes an alignment collapse measurement for object-existence hallucination. The method integrates a nominal spectral feature family with a positional Patch–Object Alignment Score family, yielding a total of 168 features (90 spectral and 78 POAS).

Critical strengths

The paper focuses on object-existence hallucination in vision–language models and takes the image together with the queried object as the unit of analysis, thereby offering a relatively concrete and operational perspective on the problem. Its principal strength lies in proposing a feature descriptor for object-level hallucination that has potential practical value. The feature design exhibits a degree of novelty and incorporates complementary sources of information. ACM combines spectral features with position-sensitive POAS features, which respectively capture the global similarity structure and the local alignment information associated with object tokens; template pairs are further used to extract statistics such as mean, dispersion, and contrast. Ablation results likewise indicate that both feature families contribute to predictive performance.

Critical weaknesses

W1: The central identifiability argument contains substantive errors of definition and proof.

For example, Equation (2) supplies a coherent alignment-collapse definition of object-existence failure. Presence is defined by a nonzero matrix M, yet the column-mismatch proof assigns opposite labels to two nonzero matrices M. The label definition and the proof construction are therefore inconsistent.

W2: Baseline attribution and evaluation protocol are insufficiently specified.

The manuscript correctly distinguishes label-free coordinates from the supervised consumer. The handling of baselines, however, requires clarification. Section 4 states that every benchmark number uses the same small classifier, whereas the abstract describes the comparator as a training-free baseline and Section 4.2 refers to scalar readings. I therefore do not assume that the baselines were left untrained. The paper should specify exactly which baseline coordinates, templates, trained consumers, and tuning budgets produced each reported row. Random-feature and shuffled-label controls do not resolve this attribution question.

The cross-validation protocol states five-fold stratified splitting but does not explicitly indicate grouping by image. This matters because multiple object queries share each image: the CHAIR construction alone comprises 2,994 pairs over 500 images. Without documented image-disjoint splits, the reported scores do not establish generalization to unseen visual content.

Questions for authors

The current submission presents a theoretical necessity result that is not rigorously established by its definitions and proofs. Furthermore, the evaluation leaves several critical questions unresolved, particularly regarding image-level independence, matched supervision, and genuine cross-dataset transferability. These deficiencies collectively undermine both the theoretical foundation and the empirical interpretation of the work. Consequently, a weak rejection would understate the extent of revision required.

Reviewer_ip8sOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper propose a method, namely ACM, that measures object hallucination, using calculations based on word(token)-patch alignment scores from an image-text matching model (CLIP). Given an image and a queried object, the authors compute 168 scalar statistics from the token–patch similarity matrix, including position-aware and position-free ones. By aggregating these values by either a linear model or a gradient boosting tree (GBT), trained on existing hallucination annotations, they achieve an combined hallucination measure. Empirically, this hallucination measure aligns well with existing hallucination metrics like POPE and CHAIR, above its compared baselines.

Critical strengths
  1. The paper provides rationales and empirical evidences on what a scalar score might discard.

  2. Reasonable experiments. The reported experimental results undergo large batch of statistical cross-validations, such as multi-seed repeats, multi-fold cross-validation, and many ablations including a separate linear vs. GBT as final score aggregator.

Critical weaknesses

Weakness 1: No validation on real MLLM responses. The paper is framed as measuring object hallucination in VLMs, yet no VLM appears in any experiment. The experiments on POPE and CHAIR are conducted on object-level: the textual half of the S (the token-patch similarity matrix, for brevity) calculation is just templated captions from exist-or-not object annotation from POPE or COCO (the CHAIR occasion).

What is evaluated is only CLIP-based object-presence classification on COCO, not the detection of hallucinated content in model outputs. To be practically usable, further experiments on detecting hallucinations and/or revealing hallucination patterns of real VLM/MLLM's response is necessary.

Weakness 2: Most of the patch–token matrix carries no object-specific information. S is computed between image patches and short fixed templates (“a photo of a {object}” / “a photo without a {object}”), in which only the object noun (and "without”) is query-specific; the remaining columns are only template words, and the similarities with these words are less informative about the hallucinated-or-not decision.

The paper’s own statistics also show the resulting degeneracy: the effective rank is only about 3.4, the top singular value accounts for about 49% of the singular-value mass. Most of the 168 descriptors are thus computed over columns with no object-specific content words, leaving much redundancy in it.

Weakness 3: 168 redundant coordinates, collapsed again, by a supervised classifier. The descriptor has 168 coordinates, several undefined in the paper (e.g., “hallucination drive intensity/ratio”), with no guidance on which matter.

The ablations (Table 4) only show small effects: removing the entire POAS family lowers AUROC only from 0.9591 to 0.9509, and removing the entire spectral family only to 0.9394. Either family alone thus already reaches 0.94–0.95, indicating that the two families largely duplicate each other rather than being complementary as claimed, and that many of the 168 coordinates are redundant.

More importantly, the final score comes from a supervised gradient-boosted classifier, i.e., the descriptor is collapsed into one scalar again, trained by labels from existing datasets like POPE. This undercuts the central claim that collapsing the alignment matrix into a single score is the problem: comparing against baselines that each feed a single scalar to the same classifier confounds “a richer reading of S” with “more features plus supervision”. A fair test, such as a label-free aggregation, or a simple feature selection strategy of the original 168 ACM scores that beats the compared baselines can help to further validate the effectiveness of the ACM descriptor.

Weakness 4: The orthogonal-invariance argument is unmotivated and justifies little. The completeness claim takes “position-free” to mean invariance under the full orthogonal transformation group O(256)×O(77) (S'=USV^T), under which the singular values are indeed the maximal invariant. However, arbitrary orthogonal maps mix patches and tokens into linear combinations with no physical meaning. Intuitively, the more natural symmetry here is permutation (as "position-free" indicates). Under permutations the spectrum remains invariant but is far from maximal: the maximal invariant is S up to row/column reordering, which includes many non-spectral statistics such as the maximum entry, the entry histogram, or the sorted row maxima. The spectrum proposed in the paper is only a small part of permutation-invariant (thus "position-free") information S gives.

The conceptual meaning, and the indication of the spectrum of S and scores calculated from them on real data is also unclear and not discussed in the paper, which can help understanding the design of the proposed ACM metric.

Questions for authors
  1. How does ACM perform on hallucinations in actual MLLM outputs? How does ACM compares with previous hallucination metrics (e.g., the compared baseline, F-CLIPScore or GLSim), on real data? Can ACM detect hallucinated objects in such responses, or reveal hallucination patterns of different MLLMs?

  2. Since only few words (e.g., object words and "without") in a text used to calculate S is query-specific in the templates, how much of ACM’s signal actually comes from the object-token columns rather than the template-word columns? For instance, how do the results change when S is restricted to the object-token columns? Given the effective rank of about 3.4 and the top singular value carrying about 49% of the mass, what information do the spectral descriptors capture beyond the object columns?

  3. Could the authors indicate which coordinates matter? Since either family (spectral and POAS) alone already reaches 0.94–0.95 AUROC in Table 4, what evidence supports their claimed complementarity? Does a label-free aggregation of the ACM coordinates, or a small subset chosen by a simple feature-selection strategy, still possible and outperform the baselines?

  4. Why is “position-free” defined by invariance under arbitrary orthogonal transformations rather than permutations? Under permutations, the spectrum is only a small part of the invariant information (e.g., the maximum entry or the entry histogram are also invariant); what justifies choosing the spectrum specifically? And what do the spectrum and the scores derived from it (e.g., effective rank) indicate conceptually?

Please also refer to weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.