acceptodds
Under review as a conference paper at ICLR 2027

GapDIM: Dimension Pruning for Modality Gap Reduction in Vision-Language Models

Abstract

Contrastive Vision-Language Models (VLMs) such as CLIP and SigLIP exhibit a modality gap, whereby embeddings from different modalities occupy separate regions of the shared latent space. Although this fragmentation does not necessarily prevent strong retrieval performance, it limits the use of pretrained embeddings for out-of-distribution or group-wise tasks such as clustering. Existing methods to reduce the modality gap typically require retraining with additional losses or architectural changes, while post-hoc alternatives often degrade retrieval by disrupting the instance-wise alignment learned during pretraining. In this paper, we ask whether the portion of the gap that disrupts the multimodal organization of the representation space can be removed selectively. We observe that the gap is concentrated in a few coordinates of the native bases of pretrained models and that these coordinates exhibit low participation in the dominant centered subspaces. We first separate these two geometric phenomena and relate them through a gap-leverage bound. Based on this analysis, we propose Gap-Importance DIMension pruning (GapDIM), a post-hoc, training-free method that identifies coordinates with high modality gap and low subspace participation and selectively prunes them from the embedding space. GapDIM thereby reduces the modality gap while preserving the discriminative structure of pretrained representations. Across multiple VLM backbones and datasets, GapDIM generally preserves retrieval performance while reducing modality separation and improving joint clustering in most configurations. These results suggest that the largest part of the modality gap is concentrated in a small set of low-participation embed- ding coordinates, enabling a training-free calibration strategy.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Learning, Modality Gap, Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8M75OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work studies the modality gap within contrastive vision‑language models like CLIP and SigLIP. Existing post‑hoc mitigation approaches including rigid translation and global standardization can alleviate the gap yet break instance‑level cross‑modal alignment, which drastically degrades retrieval performance. The authors find that under 3% of native basis coordinates bear most of the modality gap, and these coordinates show low leverage and participation within each modality’s dominant centered subspaces. Building on a derived gap‑leverage theoretical bound that constrains coordinates with high gap energy by their leverage in modality‑dominant subspaces, this paper proposes GapDIM, a lightweight training‑free post‑hoc dimension‑pruning method that eliminates high‑gap, low‑participation coordinates while keeping encoder weights unchanged. Validated over Flickr30k, MSCOCO and LLaVA‑CC3M with diverse backbones, GapDIM maintains or marginally lifts zero‑shot retrieval metrics, and achieves gains in cross‑modal clustering (up to +9 V‑measure) as well as zero‑shot image‑free captioning performance.

Critical strengths
  1. A targeted, training-free solution with preserved cross-modal alignment: Unlike existing post-hoc approaches that systematically degrade retrieval performance (often by multiple points of R@1), GapDIM provides a lightweight coordinate-pruning rule that leaves the remaining discriminative space intact, preserving pairwise alignment while significantly mitigating modality collapse

  2. Solid geometric grounding and empirical validation: The observation that high-gap coordinates decouple from dominant variance directions is supported by theoretical analysis (the gap-leverage bound), biplot visualizations, controlled synthetic experiments, and cross-dataset/cross-model transfer ablations.

Critical weaknesses
  1. Loose theoretical bound and idealized assumptions: The core motivation hinges on an orthogonality condition between the gap vector and the principal centered subspace. However, the paper's own empirical check reveals substantial overlap (ρX≈0.635,ρY≈0.575\rho_X \approx 0.635, \rho_Y \approx 0.575) and an upper bound that is quite loose (maximum tightness ratio ≤0.194\le 0.194), making the theory more of a loose geometric envelope rather than an exact predictive mechanism for real-world embeddings

  2. Dependence on supervised calibration (TPE on R@1): In its default form, GapDIM tunes its thresholds (τimp,τgap\tau_{imp}, \tau_{gap}) using TPE specifically to maximize validation R@1. While this ensures retrieval performance does not drop, it borders on an empirical search that directly protects the evaluation metric, whereas the unsupervised alternative (GMM) suffers an observable performance drop in both retrieval and clustering.

Questions for authors
  1. Leakage and generalization in threshold tuning: In the default setup, TPE optimizes τimp\tau_{imp} and τgap\tau_{gap} directly against R@1 on a calibration subset. When evaluating zero-shot retrieval on downstream test splits, how sensitive is the pruned coordinate mask to variations in the calibration split, and does optimizing solely for R@1 artificially constrain how much of the gap can actually be closed?

  2. Interaction with LayerNorm / final L2 normalization: Pruning coordinates alters the Euclidean norm before the final unit normalization. Does re-normalizing the sliced vectors induce minor geometric distortions in downstream cosine similarities?

  3. Clustering pseudo-label validity: The multimodal clustering benchmark relies on ImageNet-derived pseudo-labels generated by a ConvNeXt classifier. Have the authors evaluated clustering on datasets with ground-truth categorical labels (e.g., ImageNet zero-shot classification) to eliminate potential classifier bias?

  4. Generalization across projection types: Does the concentration of gap energy in a few native coordinates hold equally well for projection heads with non-linear activations versus standard linear projections across other multimodal architectures (e.g., modern open-source Vision LLMs using MLP/Q-Former projectors)? It would be valuable to evaluate whether GapDIM-filtered vision embeddings benefit multi-modal large language models (MLLMs) during inference when bridging visual tokens into LLM representation spaces.

4 comments

On Weakness 1. Thanks for giving us the opportunity to clarify this point. We would like to clarify that neither orthogonality nor the tightness of this bound is required for GapDIM to work. We would like to clarify the role of our theoretical results and provide additional experimental evidence.

First, Proposition 1 explains why modality gap and subspace participation can be decoupled: a modality-dependent offset changes the centroids but cancels when computing centered representations. Proposition 2 provides a general geometric bound, valid also in the non-orthogonal case, rather than a relation used to determine which coordinates to prune. Indeed, GapDIM does not rely on either proposition to construct its mask, which is computed directly from the empirical gap and leverage scores.

Second, we further investigate the properties that make selective pruning effective, using MSCOCO with ViT-B/32. Following Zhang et al., we average paired image-text differences to suppress pair-specific noise, stack the mean-gap vectors in G\mathbf{G}, and compute its SVD. We observe that:

  1. The gap has a dominant direction: the leading right-singular vector u\mathbf u explains 98.41% of the squared magnitude of the mean-gap vectors, with effective rank 1.03.
  2. The gap is concentrated in the original coordinates: the five largest coordinates of u\mathbf u account for 53.71% of its squared norm, despite representing less than 1% of the embedding dimensions.
  3. A substantial portion of this gap has low subspace participation: the five coordinates removed by GapDIM contain 47.70% of the squared centroid-gap energy, but account for only 0.560%/0.467% of the total text/image leverage mass. These measurements directly characterize the coordinates selected by GapDIM, without relying on the theoretical bound.

Third, we also compare GapDIM against projecting out the entire gap direction u\mathbf u, using z′=z−(z⊤u)u\mathbf z'=\mathbf z-(\mathbf z^\top\mathbf u)\mathbf u, and against removing five random coordinates.

InterventionGap removedR@1V-measure
Pretrained0.00%36.45%61.96%
Full gap-direction projection99.95%33.39%84.79%
GapDIM (5 coordinates)47.70%36.67%71.07%
Random 5-coordinate removal1.32%36.37%60.88%

Interestingly, removing almost the entire gap improves clustering but reduces retrieval by 3.06 R@1 points. This is consistent with the Reviewer's observation: the gap direction overlaps with the dominant subspaces, so removing it entirely can also affect discriminative information. This is why, in contrast, GapDIM removes nearly half of the gap energy by pruning only five coordinates, improving V-measure by 9.11 points while preserving R@1. Random pruning achieves neither result.

We believe this distinction is central to our contribution: the modality gap need not be orthogonal to the dominant subspaces for selective pruning to be effective. What matters is that a substantial part of the gap is concentrated in a few original coordinates with limited subspace participation. GapDIM identifies and removes this part rather than suppressing the entire gap.

On Weakness 2. We present the TPE as the best solution, as it directly targets the objective. However, the effectiveness of the proposed strategy is quite flexible and robust. Indeed, we perform cross-dataset and cross-model transfer experiments to test the robustness of the choice across different datasets, models and under a distribution shift, thus beyond the calibration setting. We detail these experiments in Appendix F.1 and F.2 and report the results here:

MethodDimRMGL2MR@1R@10V-measure
Pretrained5120.800.7436.4573.3261.96
GapDIM5070.770.5336.6773.1471.07
GapDIM thr. transf.5080.770.5336.5773.2062.56

Performance is mostly preserved even if thresholds are transferred from another dataset, with still (even though slight) improvements with respect to the pretrained model. Therefore, even if GapDIM performs better if thresholds are computed on the same dataset, it is also robust to threshold transfer.

Moreover, we also report the other scenario: cross-model threshold transfer.

MethodDimRMGL2MR@1R@10V-measure
Pretrained12800.770.6046.9880.7960.65
GapDIM12650.750.5047.5580.8968.56
GapDIM thr. transf.12670.750.5147.3580.8167.14

GapDIM mostly preserves the improvement behavior with respect to the pretrained model. Therefore, although the threshold search for the specific dataset-model obtains best results, also transferring thresholds to different datasets or models mostly preserves GapDIM improvements, going beyond the single-threshold empirical search.

In addition, GMM offers a fully unsupervised alternative: although weaker on retrieval, it still improves clustering over the pretrained model and is therefore suitable for group-wise tasks. We will clarify this supervision-performance trade-off.

Answer to Question 1. We perform two ablations in Appendix F, as discussed in the previous answer. These experiments show that the calibration procedure is not strictly dependent on a dataset or a model and that the thresholds can be transferred across backbones with different embedding dimensions and across diverse datasets.

Regarding the relation between the R@1 metric and the amount of gap that gets closed, we perform an additional ablation with different objectives (R@1, V-meas and weighted variants) on MSCOCO/ViT-B32.

Calibration objectiveRemoved dimsR@1R@10L2M
Pretrained036.45%73.32%0.736
R@1 (Table 1)536.67%73.14%0.533
0.75 R@1 + 0.25 V2234.60%70.82%0.384
0.5 R@1 + 0.5 V2732.88%69.19%0.333
0.25 R@1 + 0.75 V8728.97%64.09%0.227
V-measure4033.02%68.75%0.334

Therefore, the R@1 objective is the only that preserves retrieval performance while reducing the modality gap that are GapDIM objective. Changing the objective impacts the gap reduction and the number of pruned coordinates, as including the V-measure in the objective more consistently reduces the gap but drastically downgrades retrieval up to -8 points, making the model less suitable for this downstream task.

We have included this ablation in the revised paper.

Answer to Question 2. To show the effect of the mentioned normalization after the pruning procedure, we run an ablation and we report the results here:

DatasetBackbonePost-pruning L2Kept dimsR@1R@10V-measureL2M
MSCOCOViT-B/32no50736.6773.1471.070.533
MSCOCOViT-B/32yes50736.0572.0364.430.591
MSCOCOSigLIP2 ViT-B/16no75450.6583.0962.490.878
MSCOCOSigLIP2 ViT-B/16yes75449.9782.5159.590.937
Flickr30kViT-B/32no50857.1186.6136.010.582
Flickr30kViT-B/32yes50855.8086.9240.690.617
Flickr30kSigLIP2 ViT-B/16no75373.5994.9433.770.487
Flickr30kSigLIP2 ViT-B/16yes75371.0994.2433.030.593

Post-pruning L2 normalization is not neutral: it lowers R@1 in all four settings, always increases L2M, and has a model-dependent effect on V-measure. Rescaling each pruned vector therefore changes both retrieval rankings and clustering geometry.

GapDIM retains no post-pruning normalization because it consistently preserves retrieval and is more robust in clustering. We add this interesting ablation in the revised paper.

Answer to Question 3. According to the Reviewer's suggestion, we add a multimodal clustering experiment on the ImageNet-1K validation set using its ground-truth category labels, thus removing pseudo-labels entirely. GapDIM used frozen Table 1 thresholds, so it is a completely zero-shot scenario.

MethodDimsGround-truth V-measure
Pretrained51264.31
GapDIM50777.14

GapDIM improves V-measure by 12.83 points on average, confirming that GapDIM improvements are not caused by classifier bias or pseudo-labels.

Answer to Question 4.

We would like to clarify that GapDIM, like all other methods that reduce the modality gap, including ReAlign, Embedding shift, and I0T (among post-hoc methods, but many other exist as in-training strategies), are tailored to contrastively and jointly trained multimodal models, as the modality gap is a specific artifact of those models. Therefore, their applicability does not depend on the structure of the projector itself, but rather on the training dynamics used for learning.

Nevertheless, to further investigate the effect of nonlinearity when embeddings are projected into a text decoder, we performed an additional experiment for the captioning task. We consider two different projectors, the original nonlinear one with Linear-Tanh-Linear, and an additional linear one composed of Linear-Identity-Linear. The experiment follows the exact same settings as Appendix I. We report here the results obtained:

MapperMethodBLEU-1BLEU-4CIDErSPICE
Non-linear (Tanh)Pretrained53.7312.5432.5410.28
Non-linear (Tanh)GapDIM55.1313.3140.3711.77
Linear (Identity)Pretrained52.4312.0333.3110.59
Linear (Identity)GapDIM54.4013.0740.1511.74

Thus, GapDIM continues to provide a benefit over the pretrained model regardless of whether the projector contains or not nonlinearities. We have included this experiment in the revised paper.

AuthorsOpenReview ↗

Deadline Approaching Reminder

Dear Reviewer, as the deadline for the rebuttal period is approaching, we would like to remind that we have posted our rebuttal and revised the manuscript. We have provided answers to the questions and we have addressed the concerns raised in the weaknesses.

We would greatly appreciate any feedback, remaining concerns, or questions. We would be happy to provide further clarification before the deadline.

Reviewer_gJkHOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the modality gap in contrastive vision-language models, aiming to reduce modality separation while preserving cross-modal retrieval performance. The authors observe that a small number of embedding dimensions account for a substantial portion of the modality gap while exhibiting low participation in the dominant PCA subspaces. Based on this observation, they propose GapDIM, a training-free method that selectively prunes dimensions using coordinate-wise gap and leverage scores. Experiments across multiple CLIP and SigLIP backbones and datasets show that GapDIM reduces the modality gap, generally preserves retrieval performance, and improves cross-modal clustering in most configurations.

Critical strengths
  1. The paper is generally well-organized and clearly written.

  2. Combining modality gap scores with PCA leverage scores for dimension selection represents an incremental methodological contribution.

Critical weaknesses
  1. Limited novelty of the main insights. Most of the paper's central observations have already been established in prior work. [1] investigated the separation between modality-specific offsets and sample-dependent representations. [2] showed that a few embedding dimensions dominate the modality gap and that removing these dimensions can substantially reduce it. [3] demonstrated that reducing the modality gap benefits clustering, while its effects on retrieval are more limited or inconsistent. Consequently, the paper offers limited new insights beyond existing findings.

  2. Limited methodological novelty. The proposed method largely combines existing techniques. Gap-based dimension removal was already explored in [2], post-hoc embedding-level gap reduction was investigated in [4], and PCA leverage scores are a well-established measure of subspace participation [5]. The main methodological addition is combining gap scores with leverage scores for dimension selection. However, the experiments do not adequately establish the necessity of this combination, particularly through comparisons with gap-only pruning under matched pruning budgets. Therefore, the methodological contribution remains incremental and insufficiently justified.

[1] Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data. ICLR 2024.

[2] Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models. ICLR 2025.

[3] Closing the Modality Gap Aligns Group-Wise Semantics. ICLR 2026.

[4] I0T: Embedding Standardization Method Towards Zero Modality Gap. ACL 2025.

[5] Fast Approximation of Matrix Coherence and Statistical Leverage. Drineas et al., arXiv preprint, 2012.

Questions for authors
  1. Table 3 provides ablations of the two pruning criteria, but the number of pruned dimensions is not controlled. Could the authors compare gap-only pruning and GapDIM across multiple matched pruning budgets, using identical hyperparameter tuning conditions, to better establish the additional benefit of the leverage score?

  2. Equation (10) removes dimensions with either low importance or high gap, whereas the paper repeatedly describes pruning dimensions with both high gap and low importance. Could the authors clarify this discrepancy and justify the adopted selection criterion?

3 comments

Clarifications about the novelty (W1 and W2). We discuss these papers in the related work section and throughout the paper, clearly highlighting their limitations and GapDIM novelty relative to them. The Reviewer's comment notes that GapDIM discusses recent literature and addresses a relevant topic. GapDIM contains considerable novelty wrt the papers that the Reviewer highlighted, as we discuss in the related work section and briefly discuss again here.

We would like to clarify that:

  • Wrt [1], we provide a theoretical derivation for this, and further add the discussion and the link between the gap and the importance of each dimension. Therefore, GapDIM brings novel theoretical elements and insights wrt [1], significantly extending their analysis.

  • [2] Tested dimension removal in a simple experiment with a very vanilla method (just removing topK dimensions ordered by gap) and provides an extremely synthetic scenario (two embeddings with 5 elements each and removing the first dimension without any theoretical rule). On the contrary, GapDIM does so in a structured evaluation setup of real-world scenarios and in multiple downstream tasks. Also, [2] reports that ``while removing the largest contributors closes the gap, it leads to degraded performance", which is exactly the motivation of our work, which is proposing a method capable of reducing the gap while preserving performance.

  • Regarding [3], we explicitly wrote the findings of this paper in our related work section, and we do not claim anything new about this. Nevertheless, [3] does it by retraining the encoders from scratch, while GapDIM is a more flexible approach working post-hoc and adaptable to a plethora of existing pretrained models. Therefore, we confirm what [3] claims, and we extend it and verify it also in post-hoc scenarios, which is a novel finding.

  • Of course ``post-hoc embedding-level gap reduction was investigated in [4]", GapDIM is framed in a literature branch that tries to reduce the gap post-hoc (thoroughly discussed in the related work section). Nevertheless, existing methods, as we show extensively throughout our paper, consistently downgrade task performance while reducing the gap. Moreover, I0T is not coordinate-wise division by standard deviations or covariance whitening. It also does not prune coordinates. This means that GapDIM is considerably different and novel wrt I0T and that I0T, like all the other methods discussed, are just comparisons that GapDIM already takes into account (and outperforms) in the experiments.

  • Regarding ``PCA leverage scores are a well-established measure of subspace participation [5]", yes, indeed we built GapDIM on this known theory while revisiting this concept in the multimodal scenario and linking it to other measures like the gap for a completely novel task, which is reducing the modality gap while preserving (or improving) downstream task performance of vision-language models.

Overall, all these papers (that we have clearly discussed throughout our paper) provide the motivation for GapDIM and demonstrate that our method is built on solid and established literature. GapDIM is indeed a novel method capable of reducing the gap while preserving (or improving) performance in downstream tasks and it does so by pruning dimensions with high gap or low importance, which is a considerably novel pruning method with respect to the existing literature. The empirical validation confirms that existing literature fails to preserve the retrieval performance of pretrained models, thus affecting the effective usage of these methods for downstream tasks. Conversely, GapDIM can be effectively used in downstream tasks (including retrieval, clustering, and captioning) preserving (or slightly improving) retrieval and captioning while consistently improving clustering.

Answer to Question 1 and part of W2. Thanks for the valuable suggestion. Following the Reviewer's suggestion, we compare gap-only and GapDIM on MSCOCO with ViT-bigG-14 with the same pruning budget. Gap-only removes the top-kk gap coordinates. Both methods use the same fit data and test split.

Pretrained: R@1 46.99, V-measure 60.65, L2M 0.601, while the proposed GapDIM removes coordinates and obtains R@1 47.55, V-measure 68.56, L2M 0.51 (corresponding to 15 pruned coordinates).

RemovedGap-only R@1GapDIM R@1Gap-only VGapDIM VGap-only L2MGapDIM L2M
146.9747.1161.9265.000.5910.592
247.0347.0357.7057.700.5810.582
447.0947.0962.2062.200.5630.563
846.5447.1967.7167.150.5250.540
1546.7547.5568.5068.560.4810.510
1646.5246.9770.9063.030.4760.499
3246.2046.5477.5277.390.4250.451

Gap-only sometimes closes more centroid gap, as expected from its objective, but GapDIM is more robust in retrieval, even under the same budget.

Therefore, leverage scores provide an additional benefit over modality-gap scores alone. Under identical pruning budgets, GapDIM preserves R@1 at least as well as gap-only across all tested budgets and both backbones. Gap-only generally achieves lower L2M and occasionally higher V-measure because it directly maximizes gap removal, but it can also discard coordinates relevant to retrieval. Instead, by retaining coordinates with only low gap and high importance, GapDIM provides a better trade-off between reducing the modality gap and preserving retrieval information, highlighting the importance of exploiting both the measures to improve performance over the pretrained model. Such a novel combination also consistently distinguishes GapDIM from previous methods that only rely on the gap reduction and, indeed, crucially downgrades retrieval performance, hurting the effectiveness of pretrained models in downstream tasks.

Answer to Question 2. The reported default GapDIM results use Eq. (10). The gap and participation thresholds impose two requirements on the retained set, so a coordinate is removed when either requirement fails. We revised the method description to make this explicit.

AuthorsOpenReview ↗

Approaching Deadline Reminder

Dear Reviewer, as the deadline for the rebuttal period is approaching, we would like to remind that we have posted our rebuttal and revised the manuscript. We have provided answers to the questions and we have addressed the concerns raised in the weaknesses.

We would greatly appreciate any feedback, remaining concerns, or questions. We would be happy to provide further clarification before the deadline.

Reviewer_5UYzOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper addresses the modality gap in contrastive vision-language models, where image and text embeddings occupy different regions of the shared representation space, potentially degrading cross-modal clustering performance. The authors observe that much of this gap is concentrated in a small number of embedding coordinates and propose GapDIM, a training-free method that selectively prunes these dimensions. The method combines coordinate-wise modality gap measurements with leverage scores derived from SVD to estimate each dimension's participation in the dominant representation subspaces, aiming to reduce modality separation while preserving discriminative information. Experiments across multiple VLM backbones and datasets show that GapDIM generally improves clustering performance while largely maintaining the original cross-modal retrieval accuracy.

Critical strengths

Clear motivation and practical relevance. The paper is generally well organized and clearly identifies an important limitation of existing modality gap reduction methods: improving clustering performance often comes at the cost of degraded cross-modal retrieval. Addressing this trade-off through a lightweight, training-free approach is practically meaningful, particularly for large pretrained VLMs.

Critical weaknesses
  1. Insufficient theoretical justification and inconsistency in the pruning criterion. The paper assumes that coordinates with high modality gap and low subspace leverage can be safely removed without compromising semantic information. However, leverage scores derived from SVD only measure participation in dominant variance subspaces, which does not necessarily reflect semantic importance. Moreover, Proposition 2 establishes a geometric bound between gap energy and leverage scores, but provides no guarantee regarding semantic preservation or retrieval performance after pruning. The paper also lacks a convincing justification for hard coordinate pruning over less destructive alternatives, such as soft reweighting or partial gap correction. More critically, Eq. (10) retains coordinates satisfying both high importance and low gap, meaning that the actual pruning rule removes coordinates with either low importance or high gap, rather than only those satisfying both conditions as repeatedly claimed. This inconsistency directly challenges the proposed mechanism and its theoretical motivation.

  2. Insufficient experimental rigor and overstated empirical conclusions. The authors optimize the pruning thresholds using R@1 through Optuna, but the main text does not clearly establish whether threshold tuning uses an independent validation set or the evaluation data. It is also unclear whether competing post-hoc methods receive comparable hyperparameter tuning budgets, raising concerns about the fairness of retrieval comparisons. Moreover, the retrieval improvements over pretrained models are generally marginal, while clustering gains are often substantially smaller than those of existing gap-reduction methods. The claimed consistent clustering improvement is also contradicted by Table 1, where V-measure decreases from 30.10 to 25.83 for ViT-bigG/14 on LLaVA-CC3M. Without a controlled retrieval–clustering trade-off analysis and a clearer evaluation protocol, the empirical evidence is insufficient to substantiate the claimed superiority of GapDIM.

Questions for authors
  1. Coordinate-system dependence. The observation that the modality gap is concentrated in a few embedding coordinates appears to depend on the native coordinate basis. Under a shared orthogonal rotation of image and text embeddings, pairwise cosine similarities and retrieval performance remain unchanged, while the coordinate-wise distribution of the modality gap may change substantially. How robust is GapDIM to such transformations? Does the proposed method exploit a specific property of the native embedding basis, or does it reveal a more general geometric characteristic of multimodal representations?

  2. Potential error in Eq. (2). The denominator of the InfoNCE loss appears to exclude the positive pair through the condition (j \ne i). However, the standard CLIP InfoNCE objective includes the positive pair in the softmax denominator. Could the authors confirm whether this is a typographical error or an intentional modification of the objective?

  3. Mathematical terminology in the SVD formulation. In Section 3.2, the paper states that (\sigma_i) is the i-th eigenvalue of (A_c). However, (\sigma_i) should denote the i-th singular value rather than an eigenvalue, particularly since (A_c) is generally rectangular. Could the authors clarify and correct this notation?

5 comments

Point-to-point answer to weakness 1 (part 1).

On participation and semantics. We would like to clarify that GapDIM does not assume that low-leverage coordinates are semantically irrelevant. Our rationale is more specific: leverage measures how strongly each original coordinate participates in the dominant centered subspaces, providing a principled way to assess its contribution to the principal representation structure.

The dominant singular subspace provides the optimal low-rank approximation of the centered representations, and previous work has linked dominant spectral components to task-relevant information [3,4]. This motivates our use of the dominant subspace as a proxy for the principal representation structure, without assuming a direct equivalence between variance and semantics.

More importantly, the relationship between leverage and subspace participation is mathematically precise. From Eqs. (7)–(9), the leverage of coordinate jj for modality MM is

ℓM,j=∣WM⊤ej∣22.\ell_{M,j}=|W_M^\top e_j|_2^2.

For any vector sMs_M belonging to the dominant subspace span⁡(WM)\operatorname{span}(W_M), Cauchy–Schwarz gives

∣(sM)j∣2≤ℓM,j∣sM∣22.|(s_M)j|^2\leq \ell_{M,j}|s_M|_2^2.

Thus, if ℓM,j=0.01\ell_{M,j}=0.01, removing coordinate jj discards at most 1% of the energy of any vector lying in that subspace. Low leverage therefore provides a bound on the coordinate's contribution to the dominant subspace, rather than a guarantee of semantic irrelevance. This is consistent with the established use of leverage scores in feature selection and low-rank approximation [1,2].

We further examine this relationship on MSCOCO/ViT-B/32. The five coordinates removed by GapDIM account for 47.70% of the squared modality-gap energy, but only 0.560%/0.467% of the total text/image leverage mass. This provides direct evidence that a substantial portion of the gap can be removed while affecting only a small fraction of the dominant-subspace participation.

These observations are also consistent with the downstream results in Table 1: GapDIM improves V-measure from 61.96% to 71.07%, while maintaining retrieval performance (R@1: 36.45% to 36.67%). This supports the practical use of leverage as a criterion for selective pruning, without requiring it to be a direct measure of semantic content.

In summary, GapDIM does not equate dominant variance with semantic importance. Instead, it combines a mathematically grounded measure of subspace participation with the coordinate-wise modality gap. The resulting criterion allows us to identify coordinates whose removal reduces modality separation while largely preserving the dominant representation structure, consistently with the downstream results.

[1] C. Boutsidis, P. Drineas, and M. W. Mahoney, Unsupervised Feature Selection for the kk-means Clustering Problem, NeurIPS, 2009.

[2] M. W. Mahoney and P. Drineas, CUR Matrix Decompositions for Improved Data Analysis, PNAS, 2009.

[3] J. Guo et al., Spectral Disentanglement and Enhancement: A Dual-domain Contrastive Framework for Representation Learning, WWW, 2026.

[4] F. Gentile et al., From Weights to Concepts: Data-Free Interpretability of CLIP via Singular Vector Decomposition, CVPR, 2026.

On hard mask. We thank the Reviewer for raising this point. To assess whether hard pruning is preferable to less aggressive alternatives, we implemented a soft reweighting variant of GapDIM. Specifically, we replace the binary mask with continuous weights obtained through a sigmoid relaxation of the same gap and leverage thresholds. A temperature TT controls the smoothness of the weights, with smaller values approaching hard pruning.

We evaluate this alternative on MSCOCO with ViT-B/32:

MethodTTR@1V-measureL2M
Pretrained–36.4561.960.736
Soft0.5035.3068.060.296
Soft0.2035.1469.640.423
Soft0.1035.8062.760.491
Soft0.0136.1366.690.525
Hard GapDIM036.6771.070.533

The results show that soft reweighting does not improve the retrieval–clustering trade-off over hard GapDIM. Although some soft configurations achieve lower L2M, they consistently reduce retrieval performance. The best results on both downstream tasks are recovered only when the weights become effectively binary. These results support our choice of hard pruning. It preserves pretrained retrieval performance while improving clustering, requires no additional temperature calibration, and actually reduces embedding dimensionality. We do not claim that hard pruning is universally preferable, but within our evaluated setting, softer alternatives do not provide an improvement over the proposed approach.

Point-to-point answer to weakness 1 (part 2).

Selection rule. The reported default GapDIM results use Eq. (10). The gap and participation thresholds impose two requirements on the retained set, so a coordinate is removed when either requirement fails. We revised the method description to make this explicit.

Retrieval preservation. Proposition 2 characterizes gap geometry. We can give a separate, task-specific guarantee for retrieval that allows the discarded coordinates to vary. Let PP project onto the retained coordinates and Q=I−PQ=I-P. For a unit-normalized query qq and candidates p,np,n with nonzero retained norms, suppose the original score margin satisfies

\big|(Qq)^\top Q(p-n)\big| +\|Pq\|\big|\|Pp\|-\|Pn\|\big|.$$ Then $p$ remains above $n$ under cosine similarity after pruning. The first term measures the removed coordinates contribution to the candidate score difference. The second accounts for cosine renormalization. If the inequality holds against every competitor for an initially correct query, its top-1 result is preserved. This separates what leverage measures from what determines retrieval. Leverage quantifies participation in dominant centered directions. Ranking stability depends on the removed candidate differences relative to the original margins. On held-out data, GapDIM keeps R@1 within 0.6 percentage points of the pretrained model in all 15 settings and improves V-measure in 14. The unimodal linear probes in Appendix I.2 also retain accuracy on the tested MSCOCO classification task, which is separate from the retrieval objective used to calibrate the mask. Appendix F.4 provides a complementary control for centroid correction. GapDIM-ST uses the same coordinate selection and then centers and normalizes the retained embeddings. This further reduces the gap, while R@1 falls relative to GapDIM in 14 of 15 settings and ties in one. These results support the design choice to remove a concentrated source of gap and preserve the retained coordinates.

Point-to-point answer to weakness 2. We would like to thank the Reviewer for giving us the opportunity to clarify these points.

Threshold tuning. Due to the limited space, we include details on the threshold tuning in Appendix D.1 and H.1, which specify that we randomly select 10,000 training samples to estimate the gap and participation scores and calibrate the thresholds. More precisely, for each dataset, we randomly sample 10,000 examples from the training split to estimate the gap and participation scores. We select the thresholds by maximizing R@1 on those training examples over 100 Optuna trials. We then freeze the mask and evaluate it on the separate validation or test split.

On comparison methods. As the Reviewer correctly pointed out, competing methods generally achieve lower RMG/L2M than GapDIM in Table 1; thus, they behave exactly as expected, and they are not weakened with respect to their stated gap-reduction objective. Therefore, they are not affected by an unfair comparison. Their lower retrieval reflects the geometric trade-off induced by their complete distributional transformations, whereas GapDIM is specifically designed to identify an operating point that reduces the gap without disrupting the pretrained ranking structure. Beyond the different method itself, the GapDIM objective substantially differs from comparison objectives, as GapDIM deliberately targets retrieval preservation/improvement while reducing the gap, while others only care about the gap reduction, with consequent retrieval degradation.

On the improvements. Our goal is to take pretrained VLM models and reduce the gap while preserving downstream tasks performance, especially retrieval. As the Reviewer highlighted, GapDIM generally preserves retrieval performance and slightly improves it in some scenarios. Therefore, our objective and results differ substantially from other methods that focus on gap reduction but reduce retrieval performance as a side effect. We believe this is the fundamental difference between our work and the others: applying GapDIM on models allows us to perform tasks without one threatening the other. Even if we reduce the gap slightly less (and thus achieve lower clustering improvement) than comparison methods, GapDIM preserves or improves performance on the crucial retrieval downstream task while improving clustering wrt the pretrained model.

Therefore, applying GapDIM on pretrained VLMs does not imply accepting a trade-off in retrieval and clustering performance as it happens instead in comparison methods, since it can preserve (and slightly improve the first one), while improving the second one.

Nevertheless, we further propose a standardized version of GapDIM in Appendix F.4, which promotes gap reduction and clustering performance, aligned with comparison methods. GapDIM-ST presents a trade-off between retrieval and clustering performance instead, similarly to comparison methods: it obtains very high clustering performance at the cost of consistently reducing retrieval performance, like Embedding Shift, ReAlign and I0T. We report some GapDIM-ST results on MSCOCO here (full tab in App F.4):

BackboneMethodRMG ↓L2M ↓R@1 ↑R@10 ↑V-measure ↑
ViT-B/32Pretrained0.800.7436.4573.3261.96
ViT-B/32GapDIM0.770.5336.6773.1471.07
ViT-B/32GapDIM-ST0.710.0234.4569.8084.55
ViT-bigG/14Pretrained0.770.6046.9980.8060.65
ViT-bigG/14GapDIM0.750.5147.5580.9068.56
ViT-bigG/14GapDIM-ST0.700.0244.0577.6790.38
ViT-B/16-SigLIP2Pretrained0.871.0451.0683.4257.98
ViT-B/16-SigLIP2GapDIM0.850.8850.6583.0962.49
ViT-B/16-SigLIP2GapDIM-ST0.710.0446.8380.2386.34

Retrieval-clustering trade-off. Thanks for the valuable suggestion! We provide here a direct comparison to evaluate the effectiveness of GapDIM with respect to the comparison methods. We report a count on how many times each method outperforms the pretrained model in Tab.1 according to R@1 and V-measure, and both metrics together. Results are shown in the table:

MethodR@1V-measureAll the three
Embedding Shift0/1515/150/15
ReAlign0/1515/150/15
I0T0/1515/150/15
GapDIM10/1514/1510/15

As it is clear from the table, GapDIM is the only method preserving or improving R@1 and R@10, thus preserving the performance in the scope for which the VLM has been trained, while also improving the performance in 14 out of 15 cases in clustering. Overall, GapDIM is the only method among all the comparisons capable of improving both metrics in most of the cases.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.