acceptodds
Under review as a conference paper at ICLR 2027

When Modality Goes Missing: The Spurious Alignment Trap in Multimodal Large Language Models

Abstract

Multimodal Large Language Models (MLLMs) are generally assumed to degrade predictably when visual input is missing or corrupted, yet existing evaluations focus mainly on downstream performance drops, leaving the internal geometric degradation of representation spaces largely unexplored. In this work, we conduct a subspace geometric analysis of MLLM inference under visually degraded inputs. By measuring the effective rank and principal angles between vision and text subspaces across Transformer layers, we reveal a phenomenon termed the **Spurious Alignment Trap**: when visual input degrades to a blank image, the vision subspace loses dimensionality sharply, while conventional angle-based alignment metrics may counterintuitively indicate improved cross-modal alignment. To distinguish genuine alignment from this illusion, we introduce Gaussian noise images as a diagnostic control. This contrast suggests that the angles under blank conditions are artifacts of dimensional collapse. More importantly, further analysis reveals that under collapse, such metrics decouple from cross-modal semantic alignment and instead reflect the model's modality coupling structure and the landing direction of collapse. Building on these observations, we propose the **Rank–Angle Diagnostic (RAD)** to detect whether a sample's visual input has degraded during inference. Experiments on LLaVA-1.5 and Qwen2.5-VL across VQA and Image Captioning show that blank inputs consistently induce vision-subspace collapse and suppress the expansion of effective rank in deeper layers; across three types of degradation (15%–50% area masking, Gaussian blur, and complete degradation), RAD reliably detects complete degradation, while its sensitivity to partial degradation varies across architectures. Experiments on SmolVLM further validate the coupling–direction relationship on a third model. Our work reveals when and why angular alignment metrics become unreliable, while RAD-based detection requires neither labels nor downstream evaluation. Code will be released.

Primary area:
interpretability and explainable AI
Keywords:
Multimodal Large Language Models, Missing Modality, Cross-Modal Alignment, Subspace Geometry

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wneHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the geometry of multimodal language model representations under degraded visual inputs. It reports that blank images reduce the effective rank of visual representations, while the minimum vision–text principal angle may indicate stronger alignment despite the loss of visual semantics. The authors interpret architecture-dependent angular responses through modality coupling and the orientation of the collapsed subspace. They propose the Rank–Angle Diagnostic (RAD), combining effective rank and angular alignment, to detect visual degradation. Experiments cover VQA and image captioning with LLaVA-1.5 and Qwen2.5-VL, with additional SmolVLM analysis and detection evaluations in the appendix.

Critical strengths

1.The paper raises a useful diagnostic concern about interpreting representation alignment. The observation that apparently improved geometric alignment can coexist with the loss of visual semantics is relevant to interpreting MLLMs. It motivates examining representational dimensionality alongside angular similarity, rather than treating a smaller angle as sufficient evidence of meaningful alignment.

2.The observations extend beyond a single model or aggregate plot. The paper includes different architectures, blank/noise controls, per-sample analyses, and additional detection experiments. The severe-masking results on image captioning are promising. These results give the empirical observation substance and make the proposed diagnostic direction worth investigating, although they do not fully establish the mechanism or the detector’s additional value.

Critical weaknesses
  1. The proposed coupling mechanism does not follow from Eq. (8). Eq. (8) is a generic norm inequality. It does not involve PR, and it does not model how the subspace changes under degradation. θmin depends on the full operator P_T Q_v, including the cross-terms between components, not only on the per-component norms that PR summarises. The statement on p.8 (l.420–422) that cos θmin "reduces to max_i‖P_T v_i‖" when mass concentrates on "one or a few" components holds only for exactly one component. Counterexample: take T=span(e1), v1=0.8e1+0.6e2 and v2=0.6e1−0.8e2. Then the projection norms are 0.8 and 0.6, and PR≈1.85 under Eq. (6), but cos θmin=1, not 0.8. So the PR-to-direction story on p.8 is a hypothesis, not an "algebraic basis". The SmolVLM check in App. I is one binary prediction on one model, which is too little to call the relationship "predictive". Please remove the equality claim and test the hypothesis with controlled interventions, such as rotating or reweighting components with the text reference held fixed, or across many more models.

  2. Shared mean and anisotropy are not separated from alignment. By construction the analysis uses the uncentered matrix X^T X/(m−1) = C + m/(m−1)·μμ^T. Its eigenvalues are therefore not "variances", contrary to App. A (l.762–770); App. A's own point (3) concedes that the mean direction is kept. A blank image produces near-identical vision tokens. The collapsed subspace is then dominated by their mean, and hidden states of LLMs are known to share dominant directions: anisotropy, massive activations, and attention-sink tokens. None of this literature, nor the modality-gap literature, is discussed. The paper's own figures point the same way. For Qwen, the global effective rank (Fig. 2b, Fig. 9b: ≈1.1–1.3 at early layers) is far below the vision-only effective rank (Fig. 3b, Fig. 11: ≈10–24). This is consistent with a few high-norm tokens dominating the uncentered spectrum. Please repeat the key results with centering, with per-token normalization, and with the top shared directions removed. Please also add mismatched image–text controls, to show whether θmin measured semantic alignment even under full input.

  3. The null test does not discriminate the claimed mechanism. The null draws uniformly random r-dimensional subspaces of R^d. Against a 10-dimensional text basis in d≈4k dimensions, the null necessarily concentrates near 0.07. Any structured representation will reject it, and indeed the full-input observations in Fig. 5 lie in the same extreme tail as the blank ones. The test therefore says nothing specific about collapse or coupling. The construction also differs from the main analysis. The text subspace is the pooled top-10 basis (App. E), while the main text uses span(T). The null dimension is ceil(mean eRank), and it is not stated whether the observed vision bases are truncated to that same r. A structure-preserving null would be informative, for example subspaces drawn from the full-input vision spectrum, or a permutation of the text reference.

  4. RAD's angle term contributes almost nothing, and the gains over trivial baselines are narrow.

  • In Fig. 4, θmin ranges over roughly 4°–20°, so cos θmin ∈ [0.94, 0.998], and RAD ≈ log eRank. In LLaVA, blank inputs raise cos θmin, which works against detection.
  • Table 1 already shows that eRank alone separates the conditions. No rank-only or angle-only ablation is reported.
  • Blank is detected perfectly by raw pixel standard deviation (B1: 1.00/1.00 in Tables 5–6).
  • The text claims RAD "clearly outperforms all baselines in the moderate-to-severe range". This does not hold for VQA mask30: B3 scores 0.78/0.22, against RAD's best of 0.74/0.06.
  • For VQA mask50, TPR@5%FPR is only 0.04–0.05 at L8/L16, yet p.9 calls mask50 "separable from full modality at every layer". Paired ordering (Table 2) is not deployable separability.
  • The caption mask50 comparison should read 0.98 vs. 0.67 (B2), not 0.64. Please add same-layer log-rank-only and angle-only ablations, and restate the claims accordingly.
  1. The detection evaluation is narrow and lacks uncertainty estimates. Tables 5–6 report only LLaVA layers (L8–L31); there is no Qwen AUROC/TPR. App. F explains Qwen's insensitivity as "architectural robustness rather than a failure of the RAD mechanism" using effective rank itself, which is circular without task-accuracy evidence. All degradations are synthetic (black boxes, σ=4 blur, constant gray). Full-semantic-loss noise is absent from the detection tables. VizWiz has naturally degraded images with quality annotations, which would be a far more convincing test. The trajectory-level rule on p.9 is evaluated only as independent per-layer threshold sweeps, with no held-out calibration split. Everything uses one seed (42) and 500 samples, and Figs. 4, 5, 7 and Table 2 report no confidence intervals. Several angular effects that the mechanism rests on are differences of 1–3°.
Questions for authors
  1. Are the observed vision bases in Fig. 5 truncated to the same r=ceil(eRank) as the null? What rank is retained for S_V and S_T in Fig. 4?
  2. Do the Spurious Alignment Trap and the LLaVA/Qwen difference survive centering, per-token normalization, or removing the top shared (mean or massive-activation) directions?
  3. For Qwen2.5-VL with dynamic resolution, do full, blank (fixed 336×336) and noise inputs yield the same number of visual tokens? If not, 4. how is the dependence of eRank on token count controlled?
  4. What are the AUROC/TPR of log-eRank alone at the same layers, and of RAD on Gaussian-noise inputs and on Qwen?
  5. Can you report the variance of θmin and RAD across samples and seeds, together with task accuracy under each degradation, to separate model robustness from detector insensitivity?
Reviewer_et3tOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the geometry of multimodal representations under visual degradation. It finds that blank image inputs cause visual representations to collapse, while angle-based metrics can suggest better alignment. The authors analyze this behavior across models and further propose the Rank–Angle Diagnostic (RAD) to detect visual degradation without task labels.

Critical strengths
  1. The paper raises a interesting concern: geometric proximity should not be directly interpreted as semantic alignment when representations collapse.

  2. The noise controls, matched-rank random-subspace tests, and third-model validation provide useful evidence beyond the main observations.

  3. RAD offers a simple internal diagnostic that does not require evaluating generated answers.

Critical weaknesses
  1. The relationship between RAD and language bias is unclear. Appendix K shows that low RAD can occur alongside fluent outputs without visual support, but this does not establish whether lower RAD indicates greater reliance on language priors.

  2. RAD’s practical value needs clearer support. Blank images are also detected almost perfectly by pixel statistics, while performance on mild degradation is weak and sensitivity varies across models. On VQA with 50% masking, the reported AUROC reaches 0.90, but detection at a 5% false-positive rate is only 31% (from Table 6).

  3. (My major concern) The scope of the conclusions is limited by model and dataset coverage. Although Qwen2.5-VL is included, the study does not systematically examine models with ealy fusion stage/native multimodal pretraining (e.g., Qwen 3.5 and later series). Models with earlier multimodal fusion may show different behavior. VQA is also evaluated only on 500 VizWiz samples, leaving open whether the findings depend on this dataset’s image quality and question distribution.

Questions for authors

Please see the Weaknesses section above. I am open to raising my score if my main concern is adequately addressed.

Reviewer_fyhXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the geometric behavior of multimodal large language models (MLLMs) under missing or corrupted visual inputs. By analyzing effective rank and principal angles, the authors identify a phenomenon termed the Spurious Alignment Trap, where dimensional collapse may lead to misleading cross-modal alignment measurements. Based on these observations, they propose the Rank–Angle Diagnostic (RAD), a metric combining effective rank and angular correlation to detect visual degradation. Experiments are conducted primarily on LLaVA-1.5 and Qwen2.5-VL, with additional geometric analysis on SmolVLM.

Critical strengths
  1. The paper investigates an interesting problem from a representation geometry perspective, providing insights into how visual degradation affects internal representations of MLLMs.
  2. The observation that dimensional collapse can produce misleading alignment measurements is interesting. The Gaussian noise controls and random-subspace tests provide supporting evidence for this phenomenon.
Critical weaknesses
  1. Both effective rank and principal angles are well-established techniques. The main methodological contribution, RAD, is a relatively simple combination of these existing metrics, introduced in a short subsection. The technical novelty appears limited.
  2. Insufficient mathematical justification for RAD. The formulation RAD=log⁡(r)⋅c\mathrm{RAD}=\log(r)\cdot c appears heuristic. The paper does not explain why this particular combination is preferable to numerous alternatives. Although Pareto consistency is established, this property is satisfied by many other monotonic combinations. More importantly, when r1>r2r_1>r_2 but c1<c2c_1<c_2, it is unclear whether RAD provides a meaningful comparison between the two representations.
  3. Most figures report results at only four selected Transformer layers. Since the paper's central argument concerns the evolution of representation geometry across layers, evaluating all layers would provide substantially stronger evidence and help rule out potential selection bias.
  4. Unclear cross-model comparability. It remains unclear whether the absolute values of RAD are comparable across different architectures, model sizes, or layers. This raises concerns about the generalizability and calibration requirements of the proposed diagnostic.
  5. The experiments rely on VizWiz-VQA and Flickr30K, which are relatively old datasets. Evaluations on more recent and diverse benchmarks would strengthen the empirical evidence.
  6. The main experiments involve only two approximately 7B-scale models, LLaVA-1.5 and Qwen2.5-VL. Although SmolVLM is additionally studied, it is mainly used to validate the geometric observations rather than RAD's detection performance. Experiments on newer, larger, and more diverse MLLMs are needed to establish the generality of the findings.
  7. The evaluated corruptions are primarily synthetic, including rectangular masking, Gaussian blur, and blank images. These settings may not adequately represent realistic visual degradation. Moreover, RAD performs poorly under mild corruption, particularly on Qwen2.5-VL. Appendix H also shows that simple pixel statistics can already perfectly distinguish blank images. These results raise concerns about the practical advantages of RAD.
Questions for authors

See the weakness section.

Comments on the paper

2 comments

No reply from the authors are seen.

yes, nothing so far!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.