acceptodds
Under review as a conference paper at ICLR 2027

Alignment-Preserving Low-Dimensional Projection for Efficient Vision-Language Retrieval

Abstract

Vision-language models produce high-dimensional embeddings that enable effec- tive cross-modal retrieval but incur substantial storage and similarity search costs at scale. Naive post-hoc dimensionality reduction alleviates these costs but may substantially distort the cross-modal geometry required for retrieval. We introduce Alignment-Preserving Low-Dimensional Projection (ALP), a lightweight post-hoc framework that maps frozen vision-language representations into a compact shared embedding space using modality-specific projections. We further introduce ALP+KD, which distills the cross-modal similarity distribution of the original representation into the compressed space. On our primary CLIP retrieval benchmark, ALP+KD compresses 512- dimensional embeddings to 128 dimensions while improving Mean Recall@1 from 0.4389 ±0.0025 to 0.4727 ±0.0046 across three random seeds. At the same dimensionality, PCA achieves 0.3295 ±0.0032 and a parameter-matched Matryoshka-style baseline achieves 0.4543 ±0.0035. The compact embeddings also provide practical systems benefits. On a 500K- vector FAISS database, ALP+KD reduces raw vector storage by 75%, accel- erates exact inner-product search by approximately 2.9×, and reduces index- construction time by 4.02×. Zero-shot cross-dataset and cross-backbone exper- iments further indicate that most of the original retrieval structure is retained outside the primary training setting. These results show that explicitly preserv- ing cross-modal similarity geometry can provide a favorable accuracy–efficiency trade-off for multimodal retrieval

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, multimodal retrieval, embedding compression, representation learning, knowledge distillation, efficient retrieval, CLIP

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_XKAzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ALP - alignment preserving projection for reducing the embedding dimensions of a CLIP-like model. The method essentially trains linear layer projections on top of both modality encoders, distilling the original teacher scores. ALP is evaluated against COCO with PCA and Matryoshka representation learning as a baselines.

Critical strengths

Performing an empirical study on various vector compression techniques would be beneficial for the community, so given a substantial extension this paper has a good potential.

Critical weaknesses
  1. This paper introduces nothing novel. Compressing the embedding through linear projection is a widely used technique in practice across all over areas of information retrieval, as well as logit-level distillation from the original model.

  2. While an empirical study of various vector compression techniques could be beneficial to the community, the current study is very narrow:

  • The method was examined only for CLIP which is now very outdated. The study should be extended to other models such as SigLIP, MetaClip, DFN and SOTA clip-like models such as SigLIP2, PE etc.
  • The evaluation should be extended to more benchmarks, covering (possibly) multilingual retrieval (XM3600), zero-shot image classification (ImageNet and its variants) etc.
  • In practice, the vector is usually quantized before saved in an ANN index, and very rarely saved in FP32. For instance, 8-bit quantisation which is very common achieves the same effect in terms of memory. Such a study should cover the performance in conjunction with vector quantisation methods.
Questions for authors

See above.

Reviewer_2T9rOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Alignment-Preserving Low-Dimensional Projection (ALP) and its relational distillation extension (ALP+KD) for post-hoc compression of frozen vision-language model (VLM) embeddings. Using PCA-initialized, modality-specific linear projections (from 512D to 128D), the model optimizes a symmetric InfoNCE loss alongside a KL-divergence distillation loss over cross-modal similarity matrices. Experiments on a custom 18.7K-image subset of MS-COCO show that ALP+KD outperforms unsupervised PCA and a post-hoc Matryoshka Representation Learning (MRL) baseline, while reducing memory footprint by 75% and speeding up exact FAISS retrieval by roughly 2.9x.

Critical strengths

Practical problem formulation: Post-hoc dimensionality reduction for frozen multimodal embeddings without re-running large-scale pretraining is practically relevant for resource-constrained retrieval deployment.

Critical weaknesses

Lack of methodological novelty and missing related work: The proposed method is essentially linear probing on top of frozen CLIP features combined with standard symmetric InfoNCE and relational similarity distillation. Applying KL divergence over teacher-student similarity matrices in contrastive learning is already well-established in prior VLM compression literature—such as TinyCLIP (Wu et al., ICCV 2023), CLIP-KD (Yang et al., CVPR 2024), and DistillVLM—none of which are cited or discussed. Moreover, using unsupervised PCA as the primary classical baseline is a weak strawman; the paper overlooks classical supervised cross-modal projection methods like Canonical Correlation Analysis (CCA / Deep CCA) or Procrustes alignment, which explicitly preserve cross-modal covariance in closed form.

Flawed and non-standard experimental evaluation: Unfair comparison with zero-shot CLIP, Non-standard benchmarks, Missing ablation studies.

Questions for authors

What is the retrieval performance of a 512D-to-512D linear projection trained with the exact same alignment and distillation objective on your COCO split? Without this baseline, how can you separate the benefit of supervised in-domain adaptation from the effect of dimensionality compression?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.