- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the geometry of multimodal language model representations under degraded visual inputs. It reports that blank images reduce the effective rank of visual representations, while the minimum vision–text principal angle may indicate stronger alignment despite the loss of visual semantics. The authors interpret architecture-dependent angular responses through modality coupling and the orientation of the collapsed subspace. They propose the Rank–Angle Diagnostic (RAD), combining effective rank and angular alignment, to detect visual degradation. Experiments cover VQA and image captioning with LLaVA-1.5 and Qwen2.5-VL, with additional SmolVLM analysis and detection evaluations in the appendix.
Critical strengths
1.The paper raises a useful diagnostic concern about interpreting representation alignment. The observation that apparently improved geometric alignment can coexist with the loss of visual semantics is relevant to interpreting MLLMs. It motivates examining representational dimensionality alongside angular similarity, rather than treating a smaller angle as sufficient evidence of meaningful alignment.
2.The observations extend beyond a single model or aggregate plot. The paper includes different architectures, blank/noise controls, per-sample analyses, and additional detection experiments. The severe-masking results on image captioning are promising. These results give the empirical observation substance and make the proposed diagnostic direction worth investigating, although they do not fully establish the mechanism or the detector’s additional value.
Critical weaknesses
-
The proposed coupling mechanism does not follow from Eq. (8). Eq. (8) is a generic norm inequality. It does not involve PR, and it does not model how the subspace changes under degradation. θmin depends on the full operator P_T Q_v, including the cross-terms between components, not only on the per-component norms that PR summarises. The statement on p.8 (l.420–422) that cos θmin "reduces to max_i‖P_T v_i‖" when mass concentrates on "one or a few" components holds only for exactly one component. Counterexample: take T=span(e1), v1=0.8e1+0.6e2 and v2=0.6e1−0.8e2. Then the projection norms are 0.8 and 0.6, and PR≈1.85 under Eq. (6), but cos θmin=1, not 0.8. So the PR-to-direction story on p.8 is a hypothesis, not an "algebraic basis". The SmolVLM check in App. I is one binary prediction on one model, which is too little to call the relationship "predictive". Please remove the equality claim and test the hypothesis with controlled interventions, such as rotating or reweighting components with the text reference held fixed, or across many more models.
-
Shared mean and anisotropy are not separated from alignment. By construction the analysis uses the uncentered matrix X^T X/(m−1) = C + m/(m−1)·μμ^T. Its eigenvalues are therefore not "variances", contrary to App. A (l.762–770); App. A's own point (3) concedes that the mean direction is kept. A blank image produces near-identical vision tokens. The collapsed subspace is then dominated by their mean, and hidden states of LLMs are known to share dominant directions: anisotropy, massive activations, and attention-sink tokens. None of this literature, nor the modality-gap literature, is discussed. The paper's own figures point the same way. For Qwen, the global effective rank (Fig. 2b, Fig. 9b: ≈1.1–1.3 at early layers) is far below the vision-only effective rank (Fig. 3b, Fig. 11: ≈10–24). This is consistent with a few high-norm tokens dominating the uncentered spectrum. Please repeat the key results with centering, with per-token normalization, and with the top shared directions removed. Please also add mismatched image–text controls, to show whether θmin measured semantic alignment even under full input.
-
The null test does not discriminate the claimed mechanism. The null draws uniformly random r-dimensional subspaces of R^d. Against a 10-dimensional text basis in d≈4k dimensions, the null necessarily concentrates near 0.07. Any structured representation will reject it, and indeed the full-input observations in Fig. 5 lie in the same extreme tail as the blank ones. The test therefore says nothing specific about collapse or coupling. The construction also differs from the main analysis. The text subspace is the pooled top-10 basis (App. E), while the main text uses span(T). The null dimension is ceil(mean eRank), and it is not stated whether the observed vision bases are truncated to that same r. A structure-preserving null would be informative, for example subspaces drawn from the full-input vision spectrum, or a permutation of the text reference.
-
RAD's angle term contributes almost nothing, and the gains over trivial baselines are narrow.
- In Fig. 4, θmin ranges over roughly 4°–20°, so cos θmin ∈ [0.94, 0.998], and RAD ≈ log eRank. In LLaVA, blank inputs raise cos θmin, which works against detection.
- Table 1 already shows that eRank alone separates the conditions. No rank-only or angle-only ablation is reported.
- Blank is detected perfectly by raw pixel standard deviation (B1: 1.00/1.00 in Tables 5–6).
- The text claims RAD "clearly outperforms all baselines in the moderate-to-severe range". This does not hold for VQA mask30: B3 scores 0.78/0.22, against RAD's best of 0.74/0.06.
- For VQA mask50, TPR@5%FPR is only 0.04–0.05 at L8/L16, yet p.9 calls mask50 "separable from full modality at every layer". Paired ordering (Table 2) is not deployable separability.
- The caption mask50 comparison should read 0.98 vs. 0.67 (B2), not 0.64. Please add same-layer log-rank-only and angle-only ablations, and restate the claims accordingly.
- The detection evaluation is narrow and lacks uncertainty estimates. Tables 5–6 report only LLaVA layers (L8–L31); there is no Qwen AUROC/TPR. App. F explains Qwen's insensitivity as "architectural robustness rather than a failure of the RAD mechanism" using effective rank itself, which is circular without task-accuracy evidence. All degradations are synthetic (black boxes, σ=4 blur, constant gray). Full-semantic-loss noise is absent from the detection tables. VizWiz has naturally degraded images with quality annotations, which would be a far more convincing test. The trajectory-level rule on p.9 is evaluated only as independent per-layer threshold sweeps, with no held-out calibration split. Everything uses one seed (42) and 500 samples, and Figs. 4, 5, 7 and Table 2 report no confidence intervals. Several angular effects that the mechanism rests on are differences of 1–3°.
Questions for authors
- Are the observed vision bases in Fig. 5 truncated to the same r=ceil(eRank) as the null? What rank is retained for S_V and S_T in Fig. 4?
- Do the Spurious Alignment Trap and the LLaVA/Qwen difference survive centering, per-token normalization, or removing the top shared (mean or massive-activation) directions?
- For Qwen2.5-VL with dynamic resolution, do full, blank (fixed 336×336) and noise inputs yield the same number of visual tokens? If not, 4. how is the dependence of eRank on token count controlled?
- What are the AUROC/TPR of log-eRank alone at the same layers, and of RAD on Gaussian-noise inputs and on Qwen?
- Can you report the variance of θmin and RAD across samples and seeds, together with task accuracy under each degradation, to separate model robustness from detector insensitivity?
No reply from the authors are seen.