- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work studies the modality gap within contrastive vision‑language models like CLIP and SigLIP. Existing post‑hoc mitigation approaches including rigid translation and global standardization can alleviate the gap yet break instance‑level cross‑modal alignment, which drastically degrades retrieval performance. The authors find that under 3% of native basis coordinates bear most of the modality gap, and these coordinates show low leverage and participation within each modality’s dominant centered subspaces. Building on a derived gap‑leverage theoretical bound that constrains coordinates with high gap energy by their leverage in modality‑dominant subspaces, this paper proposes GapDIM, a lightweight training‑free post‑hoc dimension‑pruning method that eliminates high‑gap, low‑participation coordinates while keeping encoder weights unchanged. Validated over Flickr30k, MSCOCO and LLaVA‑CC3M with diverse backbones, GapDIM maintains or marginally lifts zero‑shot retrieval metrics, and achieves gains in cross‑modal clustering (up to +9 V‑measure) as well as zero‑shot image‑free captioning performance.
Critical strengths
-
A targeted, training-free solution with preserved cross-modal alignment: Unlike existing post-hoc approaches that systematically degrade retrieval performance (often by multiple points of R@1), GapDIM provides a lightweight coordinate-pruning rule that leaves the remaining discriminative space intact, preserving pairwise alignment while significantly mitigating modality collapse
-
Solid geometric grounding and empirical validation: The observation that high-gap coordinates decouple from dominant variance directions is supported by theoretical analysis (the gap-leverage bound), biplot visualizations, controlled synthetic experiments, and cross-dataset/cross-model transfer ablations.
Critical weaknesses
-
Loose theoretical bound and idealized assumptions: The core motivation hinges on an orthogonality condition between the gap vector and the principal centered subspace. However, the paper's own empirical check reveals substantial overlap () and an upper bound that is quite loose (maximum tightness ratio ), making the theory more of a loose geometric envelope rather than an exact predictive mechanism for real-world embeddings
-
Dependence on supervised calibration (TPE on R@1): In its default form, GapDIM tunes its thresholds () using TPE specifically to maximize validation R@1. While this ensures retrieval performance does not drop, it borders on an empirical search that directly protects the evaluation metric, whereas the unsupervised alternative (GMM) suffers an observable performance drop in both retrieval and clustering.
Questions for authors
-
Leakage and generalization in threshold tuning: In the default setup, TPE optimizes and directly against R@1 on a calibration subset. When evaluating zero-shot retrieval on downstream test splits, how sensitive is the pruned coordinate mask to variations in the calibration split, and does optimizing solely for R@1 artificially constrain how much of the gap can actually be closed?
-
Interaction with LayerNorm / final L2 normalization: Pruning coordinates alters the Euclidean norm before the final unit normalization. Does re-normalizing the sliced vectors induce minor geometric distortions in downstream cosine similarities?
-
Clustering pseudo-label validity: The multimodal clustering benchmark relies on ImageNet-derived pseudo-labels generated by a ConvNeXt classifier. Have the authors evaluated clustering on datasets with ground-truth categorical labels (e.g., ImageNet zero-shot classification) to eliminate potential classifier bias?
-
Generalization across projection types: Does the concentration of gap energy in a few native coordinates hold equally well for projection heads with non-linear activations versus standard linear projections across other multimodal architectures (e.g., modern open-source Vision LLMs using MLP/Q-Former projectors)? It would be valuable to evaluate whether GapDIM-filtered vision embeddings benefit multi-modal large language models (MLLMs) during inference when bridging visual tokens into LLM representation spaces.
4 comments
On Weakness 2. We present the TPE as the best solution, as it directly targets the objective. However, the effectiveness of the proposed strategy is quite flexible and robust. Indeed, we perform cross-dataset and cross-model transfer experiments to test the robustness of the choice across different datasets, models and under a distribution shift, thus beyond the calibration setting. We detail these experiments in Appendix F.1 and F.2 and report the results here:
| Method | Dim | RMG | L2M | R@1 | R@10 | V-measure |
|---|---|---|---|---|---|---|
| Pretrained | 512 | 0.80 | 0.74 | 36.45 | 73.32 | 61.96 |
| GapDIM | 507 | 0.77 | 0.53 | 36.67 | 73.14 | 71.07 |
| GapDIM thr. transf. | 508 | 0.77 | 0.53 | 36.57 | 73.20 | 62.56 |
Performance is mostly preserved even if thresholds are transferred from another dataset, with still (even though slight) improvements with respect to the pretrained model. Therefore, even if GapDIM performs better if thresholds are computed on the same dataset, it is also robust to threshold transfer.
Moreover, we also report the other scenario: cross-model threshold transfer.
| Method | Dim | RMG | L2M | R@1 | R@10 | V-measure |
|---|---|---|---|---|---|---|
| Pretrained | 1280 | 0.77 | 0.60 | 46.98 | 80.79 | 60.65 |
| GapDIM | 1265 | 0.75 | 0.50 | 47.55 | 80.89 | 68.56 |
| GapDIM thr. transf. | 1267 | 0.75 | 0.51 | 47.35 | 80.81 | 67.14 |
GapDIM mostly preserves the improvement behavior with respect to the pretrained model. Therefore, although the threshold search for the specific dataset-model obtains best results, also transferring thresholds to different datasets or models mostly preserves GapDIM improvements, going beyond the single-threshold empirical search.
In addition, GMM offers a fully unsupervised alternative: although weaker on retrieval, it still improves clustering over the pretrained model and is therefore suitable for group-wise tasks. We will clarify this supervision-performance trade-off.
Answer to Question 1. We perform two ablations in Appendix F, as discussed in the previous answer. These experiments show that the calibration procedure is not strictly dependent on a dataset or a model and that the thresholds can be transferred across backbones with different embedding dimensions and across diverse datasets.
Regarding the relation between the R@1 metric and the amount of gap that gets closed, we perform an additional ablation with different objectives (R@1, V-meas and weighted variants) on MSCOCO/ViT-B32.
| Calibration objective | Removed dims | R@1 | R@10 | L2M |
|---|---|---|---|---|
| Pretrained | 0 | 36.45% | 73.32% | 0.736 |
| R@1 (Table 1) | 5 | 36.67% | 73.14% | 0.533 |
| 0.75 R@1 + 0.25 V | 22 | 34.60% | 70.82% | 0.384 |
| 0.5 R@1 + 0.5 V | 27 | 32.88% | 69.19% | 0.333 |
| 0.25 R@1 + 0.75 V | 87 | 28.97% | 64.09% | 0.227 |
| V-measure | 40 | 33.02% | 68.75% | 0.334 |
Therefore, the R@1 objective is the only that preserves retrieval performance while reducing the modality gap that are GapDIM objective. Changing the objective impacts the gap reduction and the number of pruned coordinates, as including the V-measure in the objective more consistently reduces the gap but drastically downgrades retrieval up to -8 points, making the model less suitable for this downstream task.
We have included this ablation in the revised paper.
Answer to Question 2. To show the effect of the mentioned normalization after the pruning procedure, we run an ablation and we report the results here:
| Dataset | Backbone | Post-pruning L2 | Kept dims | R@1 | R@10 | V-measure | L2M |
|---|---|---|---|---|---|---|---|
| MSCOCO | ViT-B/32 | no | 507 | 36.67 | 73.14 | 71.07 | 0.533 |
| MSCOCO | ViT-B/32 | yes | 507 | 36.05 | 72.03 | 64.43 | 0.591 |
| MSCOCO | SigLIP2 ViT-B/16 | no | 754 | 50.65 | 83.09 | 62.49 | 0.878 |
| MSCOCO | SigLIP2 ViT-B/16 | yes | 754 | 49.97 | 82.51 | 59.59 | 0.937 |
| Flickr30k | ViT-B/32 | no | 508 | 57.11 | 86.61 | 36.01 | 0.582 |
| Flickr30k | ViT-B/32 | yes | 508 | 55.80 | 86.92 | 40.69 | 0.617 |
| Flickr30k | SigLIP2 ViT-B/16 | no | 753 | 73.59 | 94.94 | 33.77 | 0.487 |
| Flickr30k | SigLIP2 ViT-B/16 | yes | 753 | 71.09 | 94.24 | 33.03 | 0.593 |
Post-pruning L2 normalization is not neutral: it lowers R@1 in all four settings, always increases L2M, and has a model-dependent effect on V-measure. Rescaling each pruned vector therefore changes both retrieval rankings and clustering geometry.
GapDIM retains no post-pruning normalization because it consistently preserves retrieval and is more robust in clustering. We add this interesting ablation in the revised paper.
Answer to Question 3. According to the Reviewer's suggestion, we add a multimodal clustering experiment on the ImageNet-1K validation set using its ground-truth category labels, thus removing pseudo-labels entirely. GapDIM used frozen Table 1 thresholds, so it is a completely zero-shot scenario.
| Method | Dims | Ground-truth V-measure |
|---|---|---|
| Pretrained | 512 | 64.31 |
| GapDIM | 507 | 77.14 |
GapDIM improves V-measure by 12.83 points on average, confirming that GapDIM improvements are not caused by classifier bias or pseudo-labels.
Answer to Question 4.
We would like to clarify that GapDIM, like all other methods that reduce the modality gap, including ReAlign, Embedding shift, and I0T (among post-hoc methods, but many other exist as in-training strategies), are tailored to contrastively and jointly trained multimodal models, as the modality gap is a specific artifact of those models. Therefore, their applicability does not depend on the structure of the projector itself, but rather on the training dynamics used for learning.
Nevertheless, to further investigate the effect of nonlinearity when embeddings are projected into a text decoder, we performed an additional experiment for the captioning task. We consider two different projectors, the original nonlinear one with Linear-Tanh-Linear, and an additional linear one composed of Linear-Identity-Linear. The experiment follows the exact same settings as Appendix I. We report here the results obtained:
| Mapper | Method | BLEU-1 | BLEU-4 | CIDEr | SPICE |
|---|---|---|---|---|---|
| Non-linear (Tanh) | Pretrained | 53.73 | 12.54 | 32.54 | 10.28 |
| Non-linear (Tanh) | GapDIM | 55.13 | 13.31 | 40.37 | 11.77 |
| Linear (Identity) | Pretrained | 52.43 | 12.03 | 33.31 | 10.59 |
| Linear (Identity) | GapDIM | 54.40 | 13.07 | 40.15 | 11.74 |
Thus, GapDIM continues to provide a benefit over the pretrained model regardless of whether the projector contains or not nonlinearities. We have included this experiment in the revised paper.
Deadline Approaching Reminder
Dear Reviewer, as the deadline for the rebuttal period is approaching, we would like to remind that we have posted our rebuttal and revised the manuscript. We have provided answers to the questions and we have addressed the concerns raised in the weaknesses.
We would greatly appreciate any feedback, remaining concerns, or questions. We would be happy to provide further clarification before the deadline.
On Weakness 1. Thanks for giving us the opportunity to clarify this point. We would like to clarify that neither orthogonality nor the tightness of this bound is required for GapDIM to work. We would like to clarify the role of our theoretical results and provide additional experimental evidence.
First, Proposition 1 explains why modality gap and subspace participation can be decoupled: a modality-dependent offset changes the centroids but cancels when computing centered representations. Proposition 2 provides a general geometric bound, valid also in the non-orthogonal case, rather than a relation used to determine which coordinates to prune. Indeed, GapDIM does not rely on either proposition to construct its mask, which is computed directly from the empirical gap and leverage scores.
Second, we further investigate the properties that make selective pruning effective, using MSCOCO with ViT-B/32. Following Zhang et al., we average paired image-text differences to suppress pair-specific noise, stack the mean-gap vectors in G, and compute its SVD. We observe that:
Third, we also compare GapDIM against projecting out the entire gap direction u, using z′=z−(z⊤u)u, and against removing five random coordinates.
Interestingly, removing almost the entire gap improves clustering but reduces retrieval by 3.06 R@1 points. This is consistent with the Reviewer's observation: the gap direction overlaps with the dominant subspaces, so removing it entirely can also affect discriminative information. This is why, in contrast, GapDIM removes nearly half of the gap energy by pruning only five coordinates, improving V-measure by 9.11 points while preserving R@1. Random pruning achieves neither result.
We believe this distinction is central to our contribution: the modality gap need not be orthogonal to the dominant subspaces for selective pruning to be effective. What matters is that a substantial part of the gap is concentrated in a few original coordinates with limited subspace participation. GapDIM identifies and removes this part rather than suppressing the entire gap.