acceptodds
Under review as a conference paper at ICLR 2027

The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Model

Abstract

Contrastive Language–Image Pretraining (CLIP) representations form a semantic embedding space governed by cosine similarity, reflecting an intrinsic hyperspherical geometry. However, existing probabilistic interpretations typically rely on Gaussian assumptions, which fail to capture this directional and multimodal structure. We propose a principled density model for the CLIP latent space based on Mixtures of von Mises–Fisher (MovMF) distributions defined on the unit hypersphere. Using the Expectation–Maximization (EM) algorithm, we efficiently learn a probabilistic model in which each mixture component corresponds to a coherent semantic concept. This formulation yields a closed-form likelihood naturally aligned with hyperspherical geometry, enabling accurate and interpretable density estimation. Empirically, our model significantly improves long-tailed and out-of-distribution detection and provides a natural semantic decomposition, representing each embedding as a sparse probabilistic combination of interpretable concepts. These results suggest that CLIP latent space is more faithfully characterized as a hyperspherical semantic mixture rather than an isotropic Gaussian, establishing a simple and geometrically consistent probabilistic framework for modeling and understanding multimodal representations.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
CLIP, vision language models, latent density estimation, representation interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HDk5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MovMF-CLIP, a probabilistic framework that models the CLIP latent space as a mixture of von Mises–Fisher (MovMF) distributions on the unit hypersphere. It first applies whitening to remove global anisotropy, then normalizes embeddings and fits a MovMF via EM. Each mixture component corresponds to a semantic prototype, enabling density estimation, OOD detection, semantic decomposition, and stabilization under iterative generative drift. Contributions include: (1) a geometric re-interpretation of CLIP latent space arguing that unimodal Gaussian modeling is inconsistent with its directional and multimodal structure; (2) a principled MovMF-based density model with closed-form likelihood; (3) empirical validation on long-tailed fairness, OOD detection, semantic decomposition, and iterative generation stability. Results show improvements over W-CLIP and other baselines, e.g., reducing FPR95 from 67.76% to 48.00% on MS-COCO, achieving 13× faster inference in semantic decomposition, and better semantic retention in iterative CLIP-UnCLIP loops.

Critical strengths

Geometric consistency and novelty. The paper correctly identifies that CLIP embeddings live on a hypersphere and that cosine similarity implies directional semantics. Modeling them with a mixture of vMF distributions is a principled and elegant choice that directly respects this geometry. This is consequential because it addresses a fundamental mismatch in prior probabilistic interpretations (e.g., W-CLIP’s Gaussian assumption).

Critical weaknesses
  1. Limited ablation on whitening and mixture components. The paper attributes gains to hyperspherical mixture modeling, but the whitening step is also a key component. Without an ablation that uses whitening followed by a single vMF (or a Gaussian after whitening), it is unclear how much of the improvement comes from the mixture versus the calibration. This is consequential because it affects the claimed novelty of the mixture modeling.

  2. Dependence on reference statistics and fixed K. The whitening transformation is estimated from a reference dataset, and K is fixed. In real-world deployment, the target distribution may shift, and the optimal K may vary. The paper does not discuss how sensitive the method is to these choices, nor does it explore adaptive alternatives (e.g., Bayesian nonparametrics). This limits the practical robustness and generality of the approach.

Questions for authors
  1. Could you provide an ablation that isolates the effect of whitening from the MovMF mixture? Specifically, how does whitening + a single vMF or whitening + a Gaussian compare to the full MovMF-CLIP?

  2. How sensitive is the method to the choice of the reference dataset used for whitening? What happens if the target domain differs significantly from the reference?

  3. The number of mixture components K is fixed. Have you considered automatic selection methods (e.g., hierarchical Dirichlet processes)? How does performance degrade if K is misspecified?

Reviewer_WXbXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper introduces a probabilistic framework for modeling CLIP embeddings using a Mixture of von Mises-Fisher distributions on the unit hypersphere. The proposed method first applies covariance-based whitening and hyperspherical normalization, followed by EM-based estimation of multiple directional mixture components. The resulting model provides explicit likelihood estimates for OOD detection and posterior responsibilities for interpretable semantic decomposition. Experiments on MS-COCO and OpenImages show improvements in OOD detection and long-tail robustness.

Critical strengths
  • The paper presents a nice approach to modeling CLIP representations using hyperspherical mixture distributions. The proposed method provides explicit likelihood estimates and posterior assignments while remaining computationally efficient. The OOD detection improvements look substantial.

  • The proposed method enables semantic decomposition without requiring auxiliary decoders or iterative optimization. The improvements in semantic relevance and computational efficiency are encouraging. The applications to concept intervention and iterative generative stability demonstrate the potential versatility of the learned representations.

  • Additional comparisons with GMM using the same number of mixture components provide evidence for the contributions.

Critical weaknesses
  • The primary comparison between a single Gaussian and a mixture of vMF distributions simultaneously changes the distribution family and model capacity. Without comparisons against capacity-matched Gaussian mixtures, unwhitened vMF mixtures, or simpler clustering alternatives, it remains unclear whether the improvements arise specifically from hyperspherical geometry or merely from multimodal modeling.

  • Interpreting mixture posterior responsibilities as faithful semantic decomposition is not theoretically guaranteed.

  • It is unclear whether separate subsets were used for whitening, mixture fitting, hyperparameter selection. If the mixture model is evaluated on the same samples used for density estimation, the gains may partially reflect in-sample fitting, particularly for rare semantic concepts.

  • The semantic decomposition evaluation does not clearly establish whether the captions used to construct prototype keywords overlap with those used as ground-truth annotations.

Questions for authors
  • Could the authors clarify which images and captions are used for whitening, EM fitting, prototype keyword generation, hyperparameter selection, and final evaluation? Are these datasets strictly separated?

  • Could the authors provide results on an entirely held-out evaluation set to eliminate concerns about potential in-sample fitting or information leakage?

  • Can the authors provide controlled ablations that isolate the contributions of whitening, multimodality, and directional density modeling?

  • How does the method effectively separate semantic rarity from low likelihood when the mixture weights remain proportional to component frequencies? Could the authors compare empirical and uniform mixture weights and report head/tail rejection rates using a common OOD threshold?

  • Were the full-ID and tail-only FPR95 results computed using the same threshold-selection protocol?

  • Could the authors provide quantitative concept localization or intervention results beyond qualitative visualizations?

  • To what extent can the posterior-weighted mixture centers reconstruct the original CLIP embeddings while preserving their semantic information? Could the authors report quantitative reconstruction errors, posterior sparsity, and downstream prediction consistency?

Reviewer_pjhrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes decomposing the (whitened, normalized) embedding space of CLIP into a mixture of vMF distributions fitted using EM. After fitting, they continue to show several use cases for this probablistic model such as OOD detection and image editing. They also present mainly qualitative evidence that the different components have semantic meaning.

Critical strengths
  1. The paper is clearly written, easy to understand and follow.

  2. The authors present a simple method for probablistic interpretation of multi-modal embedding spaces. Some of the results are promising and expanding on them could be interesting.

  3. The experiments in the paper present gains of the method compared to previous ones.

Critical weaknesses
  1. Novelty - The claim that a vMF mixture is better at modeling an embedding space vs a single gaussian is trivial. Even if the embedding space is a vMF (which CLIP's clearly isn't due to the modality gap), a mixture model can still learn a single component assigning πi=0\pi_i = 0 to all other components. I think this type of work is more suited for a workshop rather than the main paper track.

  2. Missing Experiments and Ablations - Several key experiments and ablations are missing:

    1. The role of whitening isn't properly ablated. How does the whitened space perform on retrieval tasks for example?
    2. Semantic correspondence of the different components is underexplored, and mainly relies on qualitative analysis. Maybe adding results on semantic segmentation as in figure 3? This could ground the results in numbers rather than images.
    3. Generally the experiments are on very few models and datasets, and on a single 'distribution' (MS-COCO). Results in Table 2 don't show significant improvement over NegLabel
    4. Some of the evaluation tasks are unimodal and don't take advantage of the multi-modal space. Where is the comparison to unimodal methods?
Questions for authors
  1. Table 1 - Don't we want our likelihood to penalize rare concepts?

  2. Could you elaborate on the procedure of extracting keywords from a mixture component (lines 902-903) and on the procedure producing figure 3?

  3. Is semantic relevance a task that is usually associated with the need for fast inference? Faster inference is always nice but not always relevant.

  4. The method seems generic to any embedding space, and indeed some of the experiments are unimodal. Could you elaborate on why this is tailored for CLIP-like models rather than a method that can apply to CLIP as well as other models?

  5. Regarding the CLIP-unCLIP experiment - I don't fully understand what this experiment is supposed to simulate, could you please provide clearer motivation for it?

  6. It seems that most of these tasks could be performed with a GMM in the unnormalized embedding space (or in the whitened + unnormalized space). What is the role of normalization? This could also a missing ablation.

  7. Could you provide empirical support that the whitening procedure helps "...obtain a geometrically consistent representation that isolates semantic structure from nuisance anisotropy"? (lines 153-154) If this is indeed true, wouldn't it induce a better embedding space for retrieval tasks as well?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.