- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper observes the setting of post-hoc transformations applied to the embedding spaces of multi-modal models. In this setting the paper shows that while such transformations supposedly improve geometry of the embedding space via decreasing the modality gap and linear separability, these improvements do not capture the decrease in performance. The paper suggests instead analysing this decrease via the interaction of retrieval margins and variance applied to the space by the post-hoc transformation. They show that even in an unlabeled setting this correlates with the decrease in performance.
Critical strengths
The paper presents a straightforward and intuitive claim that the embedding space margins, together with how different transformations affect them, effect the performance of models after the transformation is applied.
Critical weaknesses
-
Presentation - I apologize for the bluntness but this paper is not humanly readable. The point of figures like figure 3 is that we won't have to read a long list of numbers such as section 3, yet both the section and the figure are not clear, as is the main takeaway from them. Quantities such as , are not mathematically defined in the main paper. The same goes for exposure. The tables are very difficult to read as many of the quantities are not defined (What is MAE in table 1? What is Severity? etc.). The description of the tables in section 5 mostly does not supplement the figures but details the numbers in them.
-
Novelty and Significance - The flipside of the strength of this paper is that it seems pretty obvious that retrieval sensitivity is dictated by margins. The paper doesn't propose any method to overcome this (margin perserving transformations for example), and while the claim that this isn't captured in other geometric measured is true, it hasn't been made anywhere as far as I know.
-
Setting - this might be a result of the trouble I have from reading the paper but from my understanding, the post-hoc alignment methods used are for unpaired encoders, yet the paper uses them mainly for CLIP-style models. For the first type of models the margin isn't defined since cross-modal retrieval isn't possible before applying the post-hoc method. On the other hand, for CLIP models the margin is defined but I'm not sure that the claim that geometry is a tell of performance is well grounded. At the very least, the paper does not do a good job of supporting this claim.
Questions for authors
-
Has it been claimed that the geometric improvements induced by post-hoc transformations correlated with performance? From my knowledge usually both are reported, and the claim that the gap does not correlate with performance has been presented in the past (figure 3 in [1], table 1 in [2]).
-
Does drop in performance correlate between datasets (mscoco, flickr) and transformations? This would also enable label free prediction if so.
-
Could you please plainly present the way you calculate the results of the forcasting method presented in the paper? Given the state of the current manuscript I suggest presenting it as pseudo code. Additionally, is there any way to see the scatter plots from which the correlations are calculated? I suggest adding at least a standard deviation when averaging over correlations over many model pairs.
-
What is the relevance of compression to the paper?
-
Many of the used models are trained as dual encoders (CLIP variants, SigLIP, etc) but many of the methods applied are for unpaired unimodal encoders. Could you please present presedence for this?
[1] "Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning" [2] "Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models"
6 comments
Response (2/3) to Reviewer Pdsv
W2 / Q1. Has it been claimed that the geometric improvements induced by post-hoc transformations correlated with performance? Yes. CKA predicts which encoder pairs align best and correlates with their retrieval accuracy (Maniparambil et al., 2025), mutual-nearest-neighbour alignment predicts downstream performance (Huh et al., 2024), shared-space alignment correlates with it (Yi et al., 2026), and closing the gap is reported to improve retrieval and clustering (Eslami & de Melo, 2024; Grassucci et al., 2026). We have now noted this in the related work (Sec. 7).
What these measures fail to capture had not been shown before. The works [1] and [2] cited by the reviewer study the modality gap alone. [1] shifts CLIP embeddings along the gap vector and finds that "modifying the modality gap can improve zero-shot performances for downstream tasks" (their Table 1), in a direction that depends on the task. [2] finds that "a larger modality gap has mild positive correlation with downstream performance" (their Takeaway 1), a Kendall's of at most 0.28 across 98 models and none significant on MS COCO (their Table 1), and that "simple post-hoc approaches can close the modality gap but do not improve performance" (their Takeaway 3). Over the 29 MSCOCO encoder pairs, ambient Recall@1 and separability likewise fail to order the pairs by their loss under compression ( and , not significant), while our risk score orders them at (, App. E, "Dependence on the encoder pair"). Our paper differs in four respects (App. A):
- it covers all the standard measures, two of which are provably blind to an orthogonal map that removes 33–49% of Recall@1 (Sec. 4);
- it studies practical transformations, over 1,932 configurations;
- it measures the effect of a transformation on each model, where [2] compares different models, a comparison that [2] notes is confounded by model, embedding and dataset size (their Takeaway 1), while comparing each model before and after a transformation holds these fixed;
- its decisive margins, read at the transformation's scale, predict which queries break and how much is lost before the transformation is applied (Sec. 5).
[1] is cited in Secs. 1, 2 and 7 and in App. A, and we have now cited [2] in Sec. 1, Sec. 7 and App. A, which discusses both in detail.
It seems pretty obvious that retrieval sensitivity is dictated by margins. Beyond the intrinsic vulnerability of small margins, the paper answers two open questions: 1) at what scale a margin counts as small, and 2) whether that scale belongs to the transformation or to the target. Reading the gaps at the transformation's threshold raises their correlation with the loss from 0.24 to 0.89 on MSCOCO compression (Tab. 2). We have now stated both answers in the introduction (Sec. 1), and our response to Reviewer 9din (W1/Q1) details their consequences (not drawn in prior works).
The paper doesn't propose any method to overcome this (margin perserving transformations for example). The conclusions of the original manuscript named this as the next step (lines 485–486), but we have now derived one from our account. Our query-weighted OPQ (QOPQ) loses less Recall@1 than the best of Product Quantization (PQ) and Optimized PQ (OPQ) in three of four settings and than ScaNN anisotropic quantization in 109 of 112 comparisons, at equal bytes (Sec. 6 and App. K).
Response (3/3) to Reviewer Pdsv
W3 / Q5. Alignment maps on jointly pretrained pairs (paired encoders) make up 340 of the 1,932 configurations. The jointly pretrained pairs enter mainly through the compressions (1,020 configurations), and the independently pretrained pairs (unpaired encoders) through the compressions applied after an alignment map (572) (Sec. 2).
For the first type of models the margin isn't defined since cross-modal retrieval isn't possible before applying the post-hoc method. We agree, and the original manuscript states it (lines 105–107): on these pairs the map creates the ordering, and the paper measures the compressions applied afterwards.
Many of the used models are trained as dual encoders (CLIP variants, SigLIP, etc) but many of the methods applied are for unpaired unimodal encoders. Could you please present presedence for this? Post-hoc maps on jointly pretrained spaces are established practice. Yi et al. (2026) evaluate post-hoc transformations of CLIP on MSCOCO retrieval, the works cited by the reviewer shift CLIP embeddings along the gap vector post hoc ([1]) and close the gap of CLIP and SigLIP post hoc ([2]), both paired encoders, and Nukrai et al. (2022), Qiu et al. (2024), Gao et al. (2024) and ImageBind map or adapt frozen CLIP spaces. Although with a different aim, the convergence of representations, Balestriero & LeCun (2026) also re-align a CLIP encoder (paired encoders) post hoc. We have now listed these precedents in Sec. 2 and App. A.
For CLIP models the margin is defined but I'm not sure that the claim that geometry is a tell of performance is well grounded. The paper's claim is that aggregate geometric measures do not tell what a transformation costs retrieval: "Those measures do not register the cost" (line 42 of the original manuscript). The view that they do is held in prior work (see W2 / Q1), and the paper tests it and finds that it does not hold: Recall@1 falls in 1,760 of 1,932 configurations while the measures move in either direction (Sec. 3).
Q2. Does drop in performance correlate between datasets (mscoco, flickr) and transformations? Yes, it does. We have now measured it: the same checkpoint under the same transformation loses nearly the same on MSCOCO and Flickr30k ( over 432 cells), as our account predicts, since the transformation's scale and the checkpoint's margins are both stable across datasets (Sec. 5, "Losses agree across datasets", and App. H).
This would also enable label free prediction if so. It does, wherever the transformed checkpoint has been evaluated on another labelled dataset ( within a transformation). The paper's forecast covers the case in which no transformed evaluation exists.
Q3. Could you please plainly present the way you calculate the results of the forcasting method presented in the paper? We have now added the forecast as pseudocode (Algorithm 1, Sec. 5). It sets a threshold from reference pairs, takes the share of the target's queries whose top-two gap falls below it, and predicts the loss from the reference average and the target's percentile among the reference risks. The target is never transformed.
Is there any way to see the scatter plots from which the correlations are calculated? We have now added them (Fig. 3). They also show why two design choices of the submitted forecast are needed: the relation saturates, hence the percentile, and it differs between transformations, hence a reference average and a threshold for each transformation.
I suggest adding at least a standard deviation when averaging over correlations over many model pairs. The original manuscript gives 95% confidence intervals for every correlation, from a bootstrap over encoder pairs, and all of them exclude zero, so every correlation is statistically significant (App. G, Tab. 14). We have now also added standard deviations to the averaged correlations (Tab. 1 and Sec. 5).
Q4. Compression is central to the paper for four reasons.
- It is a common post-hoc transformation of deployed embeddings.
- It is where the standard measures look safest: random projection to rank 256 costs 0.55% of separability and 38.2% of Recall@1 (Sec. 4).
- Its perturbation is tractable, which now yields the closed form for random projection (Eq. 5).
- It is where our account prescribes a method, QOPQ (Sec. 6).
Response (1/3) to Reviewer Pdsv
We thank the reviewer for the careful reading and for the candid feedback on the presentation, which guided a substantial rewrite of the paper. Where a point concerns the original manuscript, we first restate what the relevant section, figure or table was meant to convey, so that each answer is self-contained, and then describe how the revision addresses it.
W1. The point of figures like figure 3 is that we won't have to read a long list of numbers such as section 3, yet both the section and the figure are not clear, as is the main takeaway from them. Sec. 3 of the original manuscript makes one point: post-hoc transformations remove Recall@1 while the standard measures do not register it, and separability understates the loss by a median factor of 18 (lines 172–189 of the original manuscript). Fig. 2 shows that encoder pairs differ in how much of their gaps lies below the threshold, which is what the risk score measures. We have now closed Sec. 3 more explicitly with this takeaway and added the scatter of risk against measured loss (Fig. 3) as requested in Q3.
Quantities such as δ∗, σT, are not mathematically defined in the main paper. The same goes for exposure. We have now defined all of them in Sec. 4: the threshold (Definition 1), the severity σT and the exposure (Definition 2), linked by δ∗=σTF−1(0.5+κ) (Proposition 1).
The tables are very difficult to read as many of the quantities are not defined (What is MAE in table 1? What is Severity? etc.). MAE is the mean absolute error of the predicted relative loss. "Severity" in Tab. 1 of the original manuscript denoted the reference average, while Sec. 4 used the same word for σT. We have now reserved "severity" for σT throughout, renamed the column "Ref. avg." and defined it in the caption of Tab. 1 as the mean loss of the transformation on the reference pairs, and defined every other column in the captions of Tabs. 1–3.
The description of the tables in section 5 mostly does not supplement the figures but details the numbers in them. Each paragraph of Sec. 5 opened with its message, for example "Neither can separate targets" (line 369 of the original manuscript). We have now made this structure explicit: the main result closes with a takeaway, Fig. 3 shows what Tabs. 1 and 2 quantify, and Algorithm 1 gives the forecast step by step.