acceptodds
Under review as a conference paper at ICLR 2027

Mind the Margin: When Cross-Modal Geometry Misleads Retrieval

Abstract

Post-hoc transformations of pretrained cross-modal embeddings, such as compressions and alignment maps, change retrieval in ways the standard measures of cross-modal structure do not register. Across encoder pairs, three datasets, four modalities and transformations, alignment maps close the modality gap by – on MSCOCO while removing – of Recall@, and CKA and canonical correlation stay exactly unchanged under an orthogonal map that removes of it. We explain the loss through the comparisons that decide retrieval. A transformation perturbs the similarity difference between two candidates, and we read off the ambient margin below which orderings become unreliable, the order-preservation threshold. Under a location–scale model, the expected loss depends on the transformation only through this scale, its severity, and on the target only through its top-two similarity gaps, its exposure. We show supporting evidence for the account. The threshold is a property of the transformation, its median stable to within from to gallery items, and estimated without labels on reference encoder pairs it orders the losses of the compressions at to on three datasets. Read against it, a target's gaps order targets under a fixed compression at a mean of on MSCOCO and on MSR-VTT, and order encoder pairs by their mean loss at , where ambient accuracy and separability do not. Together they forecast a transformation's cost before it is applied, from an unlabelled retrieval log, for settings in which the target's embeddings are out of reach.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
cross-modal retrieval, embedding transformation, retrieval performance prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PdsvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper observes the setting of post-hoc transformations applied to the embedding spaces of multi-modal models. In this setting the paper shows that while such transformations supposedly improve geometry of the embedding space via decreasing the modality gap and linear separability, these improvements do not capture the decrease in performance. The paper suggests instead analysing this decrease via the interaction of retrieval margins and variance applied to the space by the post-hoc transformation. They show that even in an unlabeled setting this correlates with the decrease in performance.

Critical strengths

The paper presents a straightforward and intuitive claim that the embedding space margins, together with how different transformations affect them, effect the performance of models after the transformation is applied.

Critical weaknesses
  1. Presentation - I apologize for the bluntness but this paper is not humanly readable. The point of figures like figure 3 is that we won't have to read a long list of numbers such as section 3, yet both the section and the figure are not clear, as is the main takeaway from them. Quantities such as δ∗\delta^*, σT\sigma_T are not mathematically defined in the main paper. The same goes for exposure. The tables are very difficult to read as many of the quantities are not defined (What is MAE in table 1? What is Severity? etc.). The description of the tables in section 5 mostly does not supplement the figures but details the numbers in them.

  2. Novelty and Significance - The flipside of the strength of this paper is that it seems pretty obvious that retrieval sensitivity is dictated by margins. The paper doesn't propose any method to overcome this (margin perserving transformations for example), and while the claim that this isn't captured in other geometric measured is true, it hasn't been made anywhere as far as I know.

  3. Setting - this might be a result of the trouble I have from reading the paper but from my understanding, the post-hoc alignment methods used are for unpaired encoders, yet the paper uses them mainly for CLIP-style models. For the first type of models the margin isn't defined since cross-modal retrieval isn't possible before applying the post-hoc method. On the other hand, for CLIP models the margin is defined but I'm not sure that the claim that geometry is a tell of performance is well grounded. At the very least, the paper does not do a good job of supporting this claim.

Questions for authors
  1. Has it been claimed that the geometric improvements induced by post-hoc transformations correlated with performance? From my knowledge usually both are reported, and the claim that the gap does not correlate with performance has been presented in the past (figure 3 in [1], table 1 in [2]).

  2. Does drop in performance correlate between datasets (mscoco, flickr) and transformations? This would also enable label free prediction if so.

  3. Could you please plainly present the way you calculate the results of the forcasting method presented in the paper? Given the state of the current manuscript I suggest presenting it as pseudo code. Additionally, is there any way to see the scatter plots from which the correlations are calculated? I suggest adding at least a standard deviation when averaging over correlations over many model pairs.

  4. What is the relevance of compression to the paper?

  5. Many of the used models are trained as dual encoders (CLIP variants, SigLIP, etc) but many of the methods applied are for unpaired unimodal encoders. Could you please present presedence for this?

[1] "Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning" [2] "Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models"

6 comments
AuthorsOpenReview ↗

Response (1/3) to Reviewer Pdsv

We thank the reviewer for the careful reading and for the candid feedback on the presentation, which guided a substantial rewrite of the paper. Where a point concerns the original manuscript, we first restate what the relevant section, figure or table was meant to convey, so that each answer is self-contained, and then describe how the revision addresses it.

W1. The point of figures like figure 3 is that we won't have to read a long list of numbers such as section 3, yet both the section and the figure are not clear, as is the main takeaway from them. Sec. 3 of the original manuscript makes one point: post-hoc transformations remove Recall@1 while the standard measures do not register it, and separability understates the loss by a median factor of 18 (lines 172–189 of the original manuscript). Fig. 2 shows that encoder pairs differ in how much of their gaps lies below the threshold, which is what the risk score measures. We have now closed Sec. 3 more explicitly with this takeaway and added the scatter of risk against measured loss (Fig. 3) as requested in Q3.

Quantities such as δ∗\delta^{*}, σT\sigma_T, are not mathematically defined in the main paper. The same goes for exposure. We have now defined all of them in Sec. 4: the threshold (Definition 1), the severity σT\sigma_T and the exposure (Definition 2), linked by δ∗=σTF−1(0.5+κ)\delta^{*}=\sigma_T F^{-1}(0.5+\kappa) (Proposition 1).

The tables are very difficult to read as many of the quantities are not defined (What is MAE in table 1? What is Severity? etc.). MAE is the mean absolute error of the predicted relative loss. "Severity" in Tab. 1 of the original manuscript denoted the reference average, while Sec. 4 used the same word for σT\sigma_T. We have now reserved "severity" for σT\sigma_T throughout, renamed the column "Ref. avg." and defined it in the caption of Tab. 1 as the mean loss of the transformation on the reference pairs, and defined every other column in the captions of Tabs. 1–3.

The description of the tables in section 5 mostly does not supplement the figures but details the numbers in them. Each paragraph of Sec. 5 opened with its message, for example "Neither can separate targets" (line 369 of the original manuscript). We have now made this structure explicit: the main result closes with a takeaway, Fig. 3 shows what Tabs. 1 and 2 quantify, and Algorithm 1 gives the forecast step by step.

AuthorsOpenReview ↗

Response (2/3) to Reviewer Pdsv

W2 / Q1. Has it been claimed that the geometric improvements induced by post-hoc transformations correlated with performance? Yes. CKA predicts which encoder pairs align best and correlates with their retrieval accuracy (Maniparambil et al., 2025), mutual-nearest-neighbour alignment predicts downstream performance (Huh et al., 2024), shared-space alignment correlates with it (Yi et al., 2026), and closing the gap is reported to improve retrieval and clustering (Eslami & de Melo, 2024; Grassucci et al., 2026). We have now noted this in the related work (Sec. 7).

What these measures fail to capture had not been shown before. The works [1] and [2] cited by the reviewer study the modality gap alone. [1] shifts CLIP embeddings along the gap vector and finds that "modifying the modality gap can improve zero-shot performances for downstream tasks" (their Table 1), in a direction that depends on the task. [2] finds that "a larger modality gap has mild positive correlation with downstream performance" (their Takeaway 1), a Kendall's τ\tau of at most 0.28 across 98 models and none significant on MS COCO (their Table 1), and that "simple post-hoc approaches can close the modality gap but do not improve performance" (their Takeaway 3). Over the 29 MSCOCO encoder pairs, ambient Recall@1 and separability likewise fail to order the pairs by their loss under compression (ρ=0.31\rho=0.31 and 0.330.33, not significant), while our risk score orders them at ρ=0.755\rho=0.755 (p<10−5p<10^{-5}, App. E, "Dependence on the encoder pair"). Our paper differs in four respects (App. A):

  1. it covers all the standard measures, two of which are provably blind to an orthogonal map that removes 33–49% of Recall@1 (Sec. 4);
  2. it studies practical transformations, over 1,932 configurations;
  3. it measures the effect of a transformation on each model, where [2] compares different models, a comparison that [2] notes is confounded by model, embedding and dataset size (their Takeaway 1), while comparing each model before and after a transformation holds these fixed;
  4. its decisive margins, read at the transformation's scale, predict which queries break and how much is lost before the transformation is applied (Sec. 5).

[1] is cited in Secs. 1, 2 and 7 and in App. A, and we have now cited [2] in Sec. 1, Sec. 7 and App. A, which discusses both in detail.

It seems pretty obvious that retrieval sensitivity is dictated by margins. Beyond the intrinsic vulnerability of small margins, the paper answers two open questions: 1) at what scale a margin counts as small, and 2) whether that scale belongs to the transformation or to the target. Reading the gaps at the transformation's threshold raises their correlation with the loss from 0.24 to 0.89 on MSCOCO compression (Tab. 2). We have now stated both answers in the introduction (Sec. 1), and our response to Reviewer 9din (W1/Q1) details their consequences (not drawn in prior works).

The paper doesn't propose any method to overcome this (margin perserving transformations for example). The conclusions of the original manuscript named this as the next step (lines 485–486), but we have now derived one from our account. Our query-weighted OPQ (QOPQ) loses less Recall@1 than the best of Product Quantization (PQ) and Optimized PQ (OPQ) in three of four settings and than ScaNN anisotropic quantization in 109 of 112 comparisons, at equal bytes (Sec. 6 and App. K).

AuthorsOpenReview ↗

Response (3/3) to Reviewer Pdsv

W3 / Q5. Alignment maps on jointly pretrained pairs (paired encoders) make up 340 of the 1,932 configurations. The jointly pretrained pairs enter mainly through the compressions (1,020 configurations), and the independently pretrained pairs (unpaired encoders) through the compressions applied after an alignment map (572) (Sec. 2).

For the first type of models the margin isn't defined since cross-modal retrieval isn't possible before applying the post-hoc method. We agree, and the original manuscript states it (lines 105–107): on these pairs the map creates the ordering, and the paper measures the compressions applied afterwards.

Many of the used models are trained as dual encoders (CLIP variants, SigLIP, etc) but many of the methods applied are for unpaired unimodal encoders. Could you please present presedence for this? Post-hoc maps on jointly pretrained spaces are established practice. Yi et al. (2026) evaluate post-hoc transformations of CLIP on MSCOCO retrieval, the works cited by the reviewer shift CLIP embeddings along the gap vector post hoc ([1]) and close the gap of CLIP and SigLIP post hoc ([2]), both paired encoders, and Nukrai et al. (2022), Qiu et al. (2024), Gao et al. (2024) and ImageBind map or adapt frozen CLIP spaces. Although with a different aim, the convergence of representations, Balestriero & LeCun (2026) also re-align a CLIP encoder (paired encoders) post hoc. We have now listed these precedents in Sec. 2 and App. A.

For CLIP models the margin is defined but I'm not sure that the claim that geometry is a tell of performance is well grounded. The paper's claim is that aggregate geometric measures do not tell what a transformation costs retrieval: "Those measures do not register the cost" (line 42 of the original manuscript). The view that they do is held in prior work (see W2 / Q1), and the paper tests it and finds that it does not hold: Recall@1 falls in 1,760 of 1,932 configurations while the measures move in either direction (Sec. 3).

Q2. Does drop in performance correlate between datasets (mscoco, flickr) and transformations? Yes, it does. We have now measured it: the same checkpoint under the same transformation loses nearly the same on MSCOCO and Flickr30k (ρ=0.957\rho=0.957 over 432 cells), as our account predicts, since the transformation's scale and the checkpoint's margins are both stable across datasets (Sec. 5, "Losses agree across datasets", and App. H).

This would also enable label free prediction if so. It does, wherever the transformed checkpoint has been evaluated on another labelled dataset (ρ=0.71\rho=0.71 within a transformation). The paper's forecast covers the case in which no transformed evaluation exists.

Q3. Could you please plainly present the way you calculate the results of the forcasting method presented in the paper? We have now added the forecast as pseudocode (Algorithm 1, Sec. 5). It sets a threshold from reference pairs, takes the share of the target's queries whose top-two gap falls below it, and predicts the loss from the reference average and the target's percentile among the reference risks. The target is never transformed.

Is there any way to see the scatter plots from which the correlations are calculated? We have now added them (Fig. 3). They also show why two design choices of the submitted forecast are needed: the relation saturates, hence the percentile, and it differs between transformations, hence a reference average and a threshold for each transformation.

I suggest adding at least a standard deviation when averaging over correlations over many model pairs. The original manuscript gives 95% confidence intervals for every correlation, from a bootstrap over encoder pairs, and all of them exclude zero, so every correlation is statistically significant (App. G, Tab. 14). We have now also added standard deviations to the averaged correlations (Tab. 1 and Sec. 5).

Q4. Compression is central to the paper for four reasons.

  • It is a common post-hoc transformation of deployed embeddings.
  • It is where the standard measures look safest: random projection to rank 256 costs 0.55% of separability and 38.2% of Recall@1 (Sec. 4).
  • Its perturbation is tractable, which now yields the closed form for random projection (Eq. 5).
  • It is where our account prescribes a method, QOPQ (Sec. 6).
Reviewer_xkuoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the effect of post-hoc transformations on cross-modal retrieval. It argues that common global measures of cross-modal structure, such as modality gap, true-pair cosine, CKA, and canonical correlation, may fail to reflect retrieval degradation caused by transformations. The paper introduces an order-preservation threshold δ∗\delta^*, the margin scale at which the probability of preserving candidate orderings exceeds a fixed criterion. Under a location-scale perturbation model, transformation severity is captured by the perturbation scale, and target-specific vulnerability by the distribution of retrieval-critical margins. Based on these factors, the paper proposes a label-free risk score and a two-term predictor of retrieval cost, using an untransformed retrieval log and thresholds calibrated on reference encoder pairs; a fully label-free variant replaces reference losses with the share of changed top-1 decisions. The empirical study evaluates 16 transformations (12 compressions and 4 alignment maps) on MSCOCO, Flickr30k, and MSR-VTT, covering 85 jointly pretrained encoder pairs/retrieval directions plus 13 independently pretrained aligned units (98 evaluation units). It reports that margin-based quantities correlate with Recall@1 degradation and can outperform reference-average and query-performance baselines in several forecasting settings.

Critical strengths
  1. The paper identifies an important failure mode of standard cross-modal diagnostics. The evidence that modality gap, true-pair cosine, CKA, and canonical correlation can move favorably, remain nearly unchanged, or change inconsistently while retrieval Recall@1 decreases is compelling, especially for post-hoc alignment maps.
  2. The margin-based explanation is simple, interpretable, and empirically useful. Retrieval decisions are determined by local score differences rather than aggregate embedding statistics. The proposed top-two similarity gap, read against the transformation-specific order-preservation threshold, provides a natural target-specific exposure measure, and the experiments show meaningful correlations across multiple datasets and transformation families.
Critical weaknesses
  1. The theoretical status of the order-preservation threshold is underspecified. The paper defines δ∗\delta^* as a characteristic margin at which the preservation probability exceeds a fixed criterion, but the criterion itself, its dependence on the candidate distribution, and its statistical estimation error are not sufficiently formalized. The location-scale perturbation model is assumed, and it is unclear when real transformations satisfy the required symmetry, scale, or independence assumptions. Consequently, the theory supports a useful heuristic account, but not yet a general guarantee.
  2. The forecasting claims depend on reference calibration and may be fragile under distribution shift. The predictor uses transformation-specific thresholds and a reference-average severity term estimated from other encoder pairs. Although cross-dataset transfer is reported, performance varies considerably: for example, the within-setting rank correlations for alignment are only 0.500, 0.451, and 0.301 on MSCOCO, Flickr30k, and MSR-VTT, respectively. The paper should more clearly distinguish cases where the method reliably forecasts loss from cases where it offers little improvement over the reference average. The current evidence does not establish robustness to substantially different encoder families, gallery distributions, or transformation parameters.
Questions for authors
  1. What statistical uncertainty is associated with the estimated threshold? Please report confidence intervals or bootstrap variability for δ^∗\hat{\delta}^*, the risk score, and the reported rank correlations.
  2. The location-scale model appears to be an assumption. Can the authors quantify how well the empirical perturbation distributions satisfy the assumed symmetry and scale relationship across transformation families?
  3. Why does the margin-based forecast perform substantially worse for Flickr30k alignment and MSR-VTT alignment than for compression? Can the authors characterize the failure cases and explain whether they arise from reference-target mismatch, non-location-scale perturbations, or alignment-specific effects?
  4. How are the 98 encoder pairs selected, and how independent are the encoder pairs from the reference and target sets? If multiple checkpoints share the same pretraining corpus or architecture, correlations across configurations may inflate effective sample size and rank correlations.
  5. Can the authors compare against stronger transformation-aware baselines, such as directly estimating the distribution of score perturbations from a small transformed subsample or using empirical top-K stability? The paper includes a transformed-subsample comparison, but a more systematic budget-matched comparison would clarify the practical advantage of the proposed method.
4 comments
AuthorsOpenReview ↗

Response (1/2) to Reviewer xkuo

We thank the reviewer for the thorough and constructive review and the positive assessment. The reviewer's questions map onto concrete checks, and we answer each of them below.

W1. Each element is now formalised or measured.

  • The criterion. The threshold is read where order preservation first exceeds 0.5+κ0.5+\kappa, with κ=0.05\kappa=0.05 (Sec. 4). We have now written it in closed form under the location–scale model, δ∗=σTF−1(0.5+κ)\delta^*=\sigma_T F^{-1}(0.5+\kappa), so that its dependence on the criterion is explicit (Proposition 1). The criterion sets the scale of the threshold and leaves the forecast essentially unchanged: a threefold change of the threshold moves the correlation of risk and loss by at most 0.016 under compression and 0.037 under alignment (App. G, Tab. 19).
  • Its dependence on the candidate distribution. Under Proposition 1 the threshold is set by the scale of the transformation's perturbation (Sec. 4). For random projection, we have now derived exactly how the candidates enter: the threshold is proportional to the mean candidate distance, δ∗≈0.126dˉ/k\delta^*\approx 0.126 \bar d/\sqrt{k} (Sec. 4, Eq. 5).
  • Its statistical estimation error. Please see Q1.
  • When real transformations satisfy the assumptions. Please see Q2.
  • A general guarantee. We have now stated the corollaries for the full failure event of a query, under any joint law of its perturbations, and bounded the failure probability on both sides by the query's own margins (Sec. 4, Eq. 4, and App. F). Under the location–scale model, the corollaries are therefore guarantees, not heuristics.

Q1. All three quantities are precise relative to the effects the paper reports.

  • The rank correlations. The paper gives 95% confidence intervals for every setting and depth from a cluster bootstrap, which resamples whole encoder pairs, and every one excludes zero (App. G, Tab. 14). We have now summarised them in Sec. 5 and added standard deviations to Tab. 1.
  • The threshold and the risk score. We have now bootstrapped both over queries (App. G, Tab. 15). The threshold varies by about 4%, and by under 1% for quantization, and the risk score's sampling noise is smaller than the differences between targets, so the score separates them beyond its noise.

Q2. Yes. We have now measured each part of the assumption for all 16 transformations on the 29 MSCOCO pairs (App. F, Tab. 11). Every compression, Procrustes and least squares give a centred, near-normal perturbation, and for every transformation except ASIF its scale is proportional to the candidate distance. The departures are ASIF, with heavy tails, and the decisive gaps of least squares and low-rank PCA, which shift downwards. These are exactly the cases where the forecast is weaker (Q3). We have now also tested the model with no free parameter: its closed form for random projection matches the measured threshold (Sec. 4, Eq. 5, and App. F, Tab. 10). These measurements extend the test of symmetry in the submitted paper, where the measured preservation at vanishing margin stays within 0.005 of the 0.5 that symmetry requires (App. F, "On the assumption").

Q3 / W2. The three correlations the reviewer cites are those of the risk score alone. Added to the reference average, as in the full forecast, the risk score raises the rank correlation under alignment on all three datasets (Sec. 5, "Alignment maps", and Tab. 1).

Can the authors characterize the failure cases and explain whether they arise from reference-target mismatch, non-location-scale perturbations, or alignment-specific effects? They arise from non-location-scale perturbations. Procrustes satisfies the model, and the margins order its losses (Fig. 3b). ASIF is heavy-tailed with losses that barely vary across targets, and least squares shifts the decisive gaps, so for these maps the reference average carries the forecast (App. F and App. G, Tab. 17). Reference-target mismatch is not the cause: we have now excluded every reference that shares the target's corpus or architecture, and the alignment correlation is unchanged (App. G, Tab. 16).

The paper should more clearly distinguish cases where the method reliably forecasts loss from cases where it offers little improvement over the reference average. The margins improve on the reference average when the transformation fixes a small share of the loss variance, a share computed from the reference runs alone, before the target is transformed (Sec. 5 and Tab. 24). We have now made this rule explicit as Step 5 of Algorithm 1.

AuthorsOpenReview ↗

Response (2/2) to Reviewer xkuo

Q4. We chose the pairs to cover as much diversity as public models allow: 12 pretraining corpora, five architecture families, softmax and sigmoid contrastive losses, and two pairs trained from scratch (App. B). The confidence intervals already treat each encoder pair as a unit, resampling whole pairs, so that the many configurations of one pair do not inflate the effective sample size (Tab. 14).

If multiple checkpoints share the same pretraining corpus or architecture, correlations across configurations may inflate effective sample size and rank correlations. We thank the reviewer for raising this point, and we have now tested it: the references of each MSCOCO target exclude every pair that shares its pretraining corpus (13 groups) or its architecture (5 groups). If shared provenance inflated the correlations, this would lower them. They stay at ρ=0.887\rho=0.887 and 0.8890.889, against 0.8870.887 with only the target pair held out, and the full forecast keeps its advantage over the reference average (Sec. 5 and App. G, Tab. 16). The test therefore finds no inflation from shared corpora or architectures.

Q5. Can the authors compare against stronger transformation-aware baselines, such as directly estimating the distribution of score perturbations from a small transformed subsample or using empirical top-K stability? We have now compared the forecast with directly estimating the target's own threshold from a transformed subsample, matched in budget from 25 items to the whole target, on the twelve compressions of all three datasets. The compression is fitted on the whole target, which favours the baseline (Sec. 5, "Transforming the target", and App. I, Tab. 27). Our threshold, calibrated on reference pairs without transforming any item of the target, is more accurate on MSCOCO at every budget, even against the whole target transformed. It is on par on Flickr30k, and more accurate on MSR-VTT up to 50 transformed items. Proposition 1 predicts this: the threshold belongs to the transformation, so the median over references is the better estimate. With nothing of the target transformed, our forecast therefore matches or beats a baseline that transforms up to 50 of its items on all three datasets, and the whole target on MSCOCO.

A more systematic budget-matched comparison would clarify the practical advantage of the proposed method. The forecast serves the settings in which no item of the target can be transformed, and there every subsample baseline is undefined: alignment maps chosen before the target's pairs exist, transformations not yet built, and logs that expose similarities only (Sec. 1 and "Scope and limitations"). At the level of single queries, we have now also compared the gap with RaBitQ's per-query error bound, which exists only once the index is built. The gap matches its AUROC within 0.005, and to catch every broken query it flags 71–86% of the correctly answered ones, against 91–95% for RaBitQ's overlap rule (App. H, Tab. 22).

W2. The current evidence does not establish robustness to substantially different encoder families, gallery distributions, or transformation parameters.

  • Encoder families. Thresholds transfer to the tri-modal MSR-VTT models and to independently pretrained pairs, both absent from the references (Sec. 5 and Tab. 14), and we have now added the hold-out of Q4.
  • Gallery distributions. Thresholds calibrated on MSCOCO rank the compression losses on Flickr30k and MSR-VTT at ρ=0.772\rho=0.772 and 0.8490.849 (Sec. 5, "Transfer", and Tab. 14). We have now also grown the gallery to 100,000 items. The threshold moves by at most 8% under compression and 3.2% under alignment, and it keeps ordering the losses (Sec. 5, "Gallery size", and App. H).
  • Transformation parameters. The study covers five ranks each for PCA and random projection, two quantization levels and six PQ and OPQ codes (Sec. 2 and App. B), and the closed form now covers every rank of random projection.

Together, we believe these results provide evidence of robustness to substantially different encoder families, gallery distributions and transformation parameters.

Thank you for the authors' detailed response; after review, I am maintaining my original score.

AuthorsOpenReview ↗

Response to positive assessment by Reviewer xkuo

We thank the reviewer for their time and the positive assessment, and remain open to address any further point that can help raise the score.

Reviewer_9dinOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how the performance of cross-modal retrieval changes under compression and post-hoc alignment transformations. By combining ranking perturbation thresholds calibrated using a reference model with the target model’s candidate similarity margins before transformation, the method predicts both the risk and magnitude of performance degradation without requiring target labels or applying the transformation to the target model.

Critical strengths

The paper connects similarity margins between candidates to ranking perturbations caused by transformations and uses reference models for calibration, enabling the target system to assess the risk of performance degradation without requiring target relevance labels or actually applying the transformations. This provides a practical basis for screening compression and post hoc alignment methods in advance. In addition, the paper presents extensive experiments that support the effectiveness of the proposed approach.

Critical weaknesses

The paper provides a useful empirical study of retrieval degradation under compression and alignment transformations. However, its conceptual contribution primarily consists of connecting candidate similarity margins to ranking instability and using reference calibration to construct a practical predictor. While this offers a useful diagnostic perspective, the paper does not sufficiently establish what substantive understanding emerges beyond the general observation that small margins are vulnerable to perturbations. The claimed contributions therefore appear modest relative to the breadth of the empirical evaluation.

The theoretical analysis does not fully account for changes in the strongest negative. For an initially correct query, the analysis considers the positive candidate and its strongest negative before transformation. However, a negative originally ranked below this competitor can subsequently overtake the positive, even when the ordering of the original pair is preserved. Consequently, reversal of that fixed pair is sufficient but not necessary for retrieval failure, making the “exactly when” statement in the corollary proof in Appendix F too strong. Although the appendix acknowledges that loss depends on the joint perturbations across candidates, this observation does not resolve the gap in the proof. This limits the theoretical justification for connecting pairwise order preservation to overall retrieval degradation.

Questions for authors

What is the main conceptual insight beyond combining a margin based confidence signal with calibration on reference models? Please distinguish the new findings from established principles and explain why they constitute a substantive contribution to understanding retrieval robustness.

Appendix F states that a correctly answered query becomes incorrect exactly when its margin against the original strongest negative becomes nonpositive. How does this account for another negative overtaking the positive while the original pair retains its ordering? Does the argument require an additional assumption, or should the corollaries be weakened?

Given the variation in estimated thresholds across encoder pairs, what justifies treating their median as an appropriate threshold for an unseen target? Does the current evaluation establish transfer to unseen model families, or primarily to other checkpoints within represented families?

Under what conditions does the target specific risk correction reliably improve on the reference average loss? Can those conditions be identified before observing the target’s transformed performance, so that users can determine when the additional correction is beneficial?

Section 5 describes the coefficient β as fitted within each transformation, whereas Appendix I describes fitting it after centering within transformations and pooling across transformations. Which procedure produced the reported results, and how are the target and its associated observations excluded from this fitting process?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 9din

We thank the reviewer for the precise and constructive review.

W1 / Q1. Beyond the intrinsic vulnerability of small margins, two questions remained open: 1) at what scale a margin counts as small, and 2) whether that scale belongs to the transformation or to the target. The paper answers both. 1) A margin is small below the transformation's order-preservation threshold δ∗=σTF−1(0.5+κ)\delta^*=\sigma_T F^{-1}(0.5+\kappa), set by the scale σT\sigma_T of its perturbation on margins, and 2) this scale belongs to the transformation alone, so it can be estimated on reference pairs and applied to any target. We have now stated both answers in the introduction (Sec. 1). Together, we believe they make a substantive contribution to understanding retrieval robustness, with four consequences not drawn in prior works:

  1. Margins say little until they are read at the transformation's scale. Read alone, the top-1 gaps order the losses weakly, and read against the transformation's threshold they order them better in all six settings (Tab. 2). The information lies in the margin read at the specific transformation's scale.
  2. The scale is predictable without any of the target's data. We have now shown that it stays within 8% from 500 to 100,000 gallery items, so the account also predicts how a transformation's cost grows with the collection (Sec. 5 and App. H), and derived it in closed form for random projection, within a median 1.8–4.1% of the measured value (Sec. 4, Eq. 5).
  3. The scale prescribes better transformations. We have now derived query-weighted OPQ (QOPQ) from our account, which outperforms PQ, OPQ and ScaNN at equal bytes in most settings (Sec. 6 and App. K), a new method that mitigates margin vulnerability.
  4. The standard measures are blind by construction. CKA and canonical correlation are exactly invariant to an orthogonal map that removes 33–49% of Recall@1 (Sec. 4 and App. F).

W2 / Q2. We thank the reviewer for spotting this error, which we have now fixed (Sec. 4 and App. F). The location–scale model now covers all the margins of a query jointly, Δ=σTZ\Delta=\sigma_T Z with the joint law of ZZ fixed by the transformation, so a correctly answered query fails exactly when some competitor overtakes the correct item. Both corollaries now cover this full failure event, and their conclusions still hold: a stronger transformation breaks no fewer queries, and a target with larger margins breaks no more. The failure probability is now bounded on both sides by the query's own margins: 1−F(m⋆/σT)≤Pr⁡[failure]≤min⁡(1,∑j≠+[1−F(mj/σT)]).1-F(m^\star/\sigma_T) \le \Pr[\text{failure}] \le \min\left(1, \sum_{j\neq +}\left[1-F(m_j/\sigma_T)\right]\right). No reported number changes, since the forecast calibrates the loss on reference pairs (Sec. 5).

Q3. Theory supports the median. Under Proposition 1 the threshold belongs to the transformation, so the thresholds of different encoder pairs estimate one quantity, and their median estimates it robustly (Sec. 4). For random projection, we have now also derived the residual spread: the closed form ties each pair's threshold to its mean candidate distance, within −23% to +25% of the median across the 29 MSCOCO pairs (Tab. 10). The median is therefore the appropriate estimate for an unseen target.

Does the current evaluation establish transfer to unseen model families, or primarily to other checkpoints within represented families? It establishes transfer to unseen families. Thresholds calibrated on MSCOCO image–text pairs transfer to tri-modal MSR-VTT models and to independently pretrained pairs, two families absent from the references (Sec. 5 and Tab. 14). We have now also excluded every reference sharing the target's pretraining corpus or architecture, and the forecast is unchanged (Tab. 16).

Q4. Yes, they can. The correction helps when the transformation fixes a small share of the loss variance, so that targets differ and their margins tell them apart. Across the 26 settings its gain grows as this share falls (ρ=−0.51\rho=-0.51), and the share is known from the reference runs alone (Sec. 5 and Tab. 24). Added to the reference average, the correction raises the rank correlation in 19 of the 26 settings and lowers the Recall@1 error in 9 of them, and at Recall@5 it lowers the error in every compression setting (Tab. 30). We have now made this rule Step 5 of Algorithm 1.

Q5. The reported results use the pooled procedure of App. I: a single slope β\beta, fitted after centring the reference losses within each transformation. "Fitted within each transformation" in Sec. 5 of the original manuscript was a wording error, which we have now corrected (Sec. 5).

How are the target and its associated observations excluded from this fitting process? The target's whole group (e.g. every retrieval direction of one MSR-VTT model) is excluded from every reference quantity: δ^∗\hat\delta^*, F^T\hat F_T, LˉT\bar L_T and β\beta (caption of Tab. 1). We have now also stated this in Sec. 5 and Algorithm 1.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: summary of the revision

We thank the reviewers for their careful reading, assessment of the paper and constructive feedback.

The reviewers recognise that the paper "identifies an important failure mode of standard cross-modal diagnostics", with evidence that is "compelling, especially for post-hoc alignment maps" (xkuo). They find the margin-based explanation "simple, interpretable, and empirically useful" (xkuo) and the claim "straightforward and intuitive" (Pdsv), and they note that the forecast "provides a practical basis for screening compression and post hoc alignment methods in advance", supported by "extensive experiments" (9din).

The weaknesses and questions fall into five groups:

  1. what the account adds beyond the vulnerability of small margins, and how it relates to prior work on the modality gap (9din, Pdsv);
  2. the theory: a gap in the proof of the corollaries, and the formal status of the threshold and of the location–scale assumption (9din, xkuo, Pdsv);
  3. the statistical robustness of the forecast: uncertainty, independence of encoder pairs, unseen model families, gallery size and agreement across datasets (9din, xkuo, Pdsv);
  4. the forecast itself: when its correction helps, how it is fitted, and how it compares with transformation-aware baselines (9din, xkuo);
  5. the presentation and the setting: definitions, pseudocode, scatter plots, and the use of compressions and of post-hoc maps on jointly pretrained encoders (Pdsv).

Every one of them has been addressed. The manuscript has been revised accordingly, with every change in red, and new analyses strengthen both the account and the method. For example:

  • the account now states precisely what it adds to prior work: the scale at which a margin counts as small, which belongs to the transformation, with a closed form for random projection;
  • the account is prescriptive: it yields a new query-weighted product quantization that retains more retrieval accuracy than standard quantizers at the same memory cost in most settings;
  • the corollaries now hold for the full failure event of a query, and the failure probability is bounded on both sides by the query's own margins;
  • every correlation of the forecast is statistically significant, and it is unchanged when the references exclude every encoder pair that shares the target's pretraining corpus or architecture.

For ease of reading, the responses are structured and target precise points. Also, everything done during the rebuttal appears in the responses to the reviewers in the form "We have now …". We have tried to keep the responses concise without sacrificing the necessary detail and precision. We thank the reviewers in advance for reading our responses and the revised manuscript, and we remain open to address any further point they may raise. We also thank the area chairs and program chairs for overseeing the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.