- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
I have reviewed this paper for a previous conference. I've failed to notice any substantial changes to the main paper and therefore post my previous review below:
The paper claims that the embedding spaces of different multi-modal models can be aligned using a single orthogonal transformation. This transformation can be calculated using a small number of anchors from one modality and then applied to either one of the modalities (“transfers across modalities”). Experimentally, the paper validates these claims by comparing different mappings, ablating the number of anchors and computing various performance metrics of the aligned modalities on different datasets. The authors complement the above with theoretical claims showing that if multi-modal models agree on the kernel matrix w.r.t. a set of anchors, then an orthogonal transformation can map between the embeddings of different models.
Critical strengths
Overall this is a well written paper with interesting theoretical claims and empirical evidence. The theorems showing that multi-modal models converge to a similar kernel, even when trained on different subsamples of the same distribution, is a very nice finding. Additionally, the empirical results showing that different multi-modal models trained on different datasets have similar (mostly class-level) similarity structures. This is an interesting result relevant to current research on multi-modal models, that might also suggest why many of them fail on unimodal tasks (since they learn similar unimodal kernels).
Critical weaknesses
The main weakness is that the main paper mainly uses metrics such as performance on classification datasets to measure kernel agreement, and the only measure presented in the main paper (CKA) is unconvincing. Since there are many methods to measure this directly (such as nearest neighbor agreement, etc). I think that the paper would benefit from posing a question such as when do multimodal models align (answer: when they converge to their global minimum, even when trained on different subsampling of the entire training data), and investigating what happens in practice (partial alignment, mean shift is needed, etc). See fuul comments below.
Metrics
If I understand correctly, the goal of measuring performance (accuracy and recall) is a testament to the compatibility of local neighborhood structure between models. However, this is just an implicit measure. I would suggest explicitly measuring the neighborhood overlap \ nearest neighbor Jaccard coefficient between the image embeddings of CLIP models [2] if this is indeed the claim. Since cosine similarity is invariant to rotations I would expect “aligned image – aligned text” accuracy and retrieval to stay similar to before applying Q (figures 3,6), since the only change that could happen in this metric is due to the mean shift (and not rotation). Therefore, the more interesting metrics are aligned vs non-aligned (e.g. “aligned image – text”). However, these are not presented in the main paper at all.
Empirical Evidence for Theory
From my understanding, the main contribution of the paper is that multi-modal models converge to the same cross-modal kernel function, along with empirical evidence. However, the empirical part of the paper supports these claims only by proxy (as discussed above, and see comment below). Since a big part of the theory focuses on subsampling from a fixed distribution, it would be interesting and perhaps more convincing to see some experiments testing this claim in a controlled setting. Currently most of the empirical verification of this theorem is for models trained on e.g. LAION vs OpenAI for which the weights are unknown , therefore possibly very close between the two models.
Usefulness of the method
In order to use the method for re-embedding, one would need to have 2 trained multi-modal models (, ), re-embed the dataset using , calculate , keep , meaning the true cost save of this method is just re-embedding using , which I would argue is not that much since re-embedding with is still required. The larger concern is that there is no evidence in the main paper (and only little in the appendix) that this method can be used to improve Model B using an encoder from Model A (contrary to [1] for example). So the claim that "it supports model upgrades without re-encoding while keeping the embedding geometry intact" is questionable - does it really support upgrades (rather than just switchability)?
Furthermore, there seems to be some circular logic in this claim: "Since it preserves inner products, it supports model upgrades without re-encoding while keeping the embedding geometry intact" If inner products are preserved between two models, wouldn't that nullify the need for upgrade (since the models agree on inner products, therefore performance)?
Minor Comments
Abstract claims implications for privacy but this isn’t discussed in the main paper.
Figure 2: Where in the figure do you see that the unimodal kernel isn’t preserved? Nearest neighbors between modalities seem the same between the two models. The CKA alignment numbers don’t seem so high as claimed.
Section 5.2 - From the first paragraph it isn't clear what is your contribution and what is background.
Figure 3 - what does After Q measure in (c)? <Qf,Q^Tg>? If Q is orthogonal why shouldn't this be close to model A's performance (up to error from the mean shift)
Generally, while there is a single result on FLICKR30k, the main paper rarely presents results on dense prediction tasks, and the existing results, mainly on Oxford Pets, suggest that the consistency between models is at the class level rather than the instance level (as can be seen in figure 7, or figure 59).
[1] Jang, Young Kyun, and Ser-nam Lim. "Towards cross-modal backward-compatible representation learning for vision-language models.", 2025
[2] "Hierarchical nucleation in deep neural networks", Doimo et al, NeurIPS 2020
Questions for authors
-
The theory assumes that the models converge to their global minimum for kernel agreement, is this indeed the case?
-
The role of adding \ subtracting the modality mean is unclear, could you please explain how and why it contributes to the method?
-
Are there any other ways to measure kernel agreement except performance?
-
"It turns out that agreement on a finite set of anchors, such as a downstream image-text dataset, is enough" - from the theorems this is enough to align on , is it also enough to align on the population level? Under what conditions if so?
9 comments
(Q7) Our approach supports model upgrades rather than just model switchability
Under exact alignment, both models have identical cross-modal scores, with resolving their coordinate mismatch. In practice, alignment is approximate, and mixed-system performance can differ from either model. Indeed, Table 1 shows that achieves 84.9%, outperforming native Model A (82.4%). Thus, our method enables encoder upgrades without recomputing stored embeddings, although performance gains are not guaranteed. We have further clarified the implications for model upgrades in Section 7.2 (lines 381-383).
(Q8) Unimodal kernel preservation in Figure 2
Fig. 2 does not test unimodal-kernel preservation. It was intended to illustrate strong agreement of the multimodal kernel. In the revised manuscript, we have revised the text and caption to avoid implying that it confirms or refutes PRH at the unimodal level. Further, under exact population minimization, our theory yields and with . Hence, unimodal kernels are preserved automatically, and aligned images coincide exactly with their target representations.
(Q9) Clarity of contributions in Section 5.2
We thank the reviewer for this suggestion. While the optimal contrastive score's connection to PMI is established, our contribution is proving that distinct training distributions can yield the same optimal score under explicit curation assumptions. Sections 5.3–5.4 then establish when kernel agreement implies a shared orthogonal map. We have made these contributions clearer in Section 5.2 in lines 218-223
(Q10) Findings only work for classification benchmarks
We respectfully disagree that our findings are limited to class-level alignment. Beyond classification, we demonstrate fine-grained image–text retrieval on Flickr30k (Figure 5) and MS-COCO (Appendix E.6, Table 5). On MS-COCO, achieves 48.0%/27.8% Recall@1, approaching native Model B's 51.8%/30.3%. These results provide direct evidence of instance-level cross-modal alignment. To emphasize this finding, we have moved the fine-grained retrieval results to Section 7.3 (lines 395–406) in the revised manuscript.
To further strengthen this claim, we now evaluate data efficiency for instance-level retrieval on Flickr30k (Appendix E.7, Fig. 19). With only 2,500 anchor images, fewer than 10% of the 29,000-image training set, unseen image-to-text Recall@1 reaches 68.9%, compared with 71.0% when is fit on the full training set. Thus, a small calibration set recovers most of the full-data retrieval performance, showing that generalizes beyond its fitting examples.
(Q11) Why shouldn't be close to model A's performance
Indeed, for pure , $\langle Qf,Qg\rangle=\langle f,g\rangle$ exactly. Figure 3(c) in the original manuscript, however, used modality-specific mean correction, which changed cross-modal scores. Table 1 in the revised manuscript (Table 6 in the original manuscript) shows that centering improves mixed-encoder accuracy from 80.2% to 84.9% and 77.6% to 82.6%, while text alignment cosine increases from 0.287 to 0.822. Jointly mapped accuracy remains close to Model A, as expected. We have clarified this in Section 7.6 (lines 455–468).
(Q12) Assumes that the models converge to their global minimum for kernel agreement
We agree that pretrained models need not reach the population optimum. Our theory establishes kernel agreement under Bayes-optimal learning and the stated distributional assumptions. Crucially, Section 7 independently validates the predicted alignment across real pretrained models, without assuming global optimality.
(Q13) Role of mean shift
Mean correction accounts for a systematic offset between source and target embedding distributions. For a fixed orthogonal matrix , the optimal constant translation under squared-error alignment is
Thus, the centered transformation is . This adjusts the first moment but does not identify the orthogonal correspondence. The distinction is supported by Table 1 in the revised manuscript (Table 6 in the original manuscript). Mean shift alone fails to recover useful mixed-encoder accuracy, whereas pure already achieves substantial transfer. Centering further improves pointwise agreement and some downstream metrics. We have expanded the discussion and ablations on mean correction in Section 7.6 (lines 455–468).
(Q14) Other ways to measure kernel agreement except performance?
We respectfully disagree that kernel agreement is evaluated only indirectly. Figure 2(c) already measures agreement between the native multimodal kernels using Centered Kernel Alignment (CKA), without fitting or applying . CKA is a well-established metric for comparing learned representation geometry (Kornblith et al., ICML 2019), with strong theoretical foundations in kernel comparison (Cortes et al., JMLR 2012). Its invariance to orthogonal transformations makes it particularly appropriate for our setting, where embedding coordinate systems are arbitrary.
Beyond CKA, as detailed in response Q3, we now also evaluate nearest-neighbor overlap using Jaccard similarity to assess local geometric compatibility. On Caltech-101, top-10 overlap increases from 0.4% before alignment to 62.8% after , exceeding the 49.6% within-model baseline. This demonstrates substantial coordinate mismatch before alignment and strong recovery of target-model neighborhoods after alignment. We have added these results to Section 7.3 (lines 401–411), complementing CKA's rotation-invariant assessment of kernel agreement.
(Q15) Conditions under which agreement on a finite set of anchors imply agreement at population level
Finite anchor agreement alone does not guarantee population-level alignment. Assumption 5.2 establishes global alignment from agreement along anchor-induced kernel sections, subject to the stated rank conditions. Crucially, Proposition C.27 extends this to purely finite anchor–anchor agreement, provided the anchors sufficiently cover the representation space and are well-conditioned.
We hope the additional revisions to the main manuscript address the reviewer's concerns. We would be grateful if the reviewer could reconsider their assessment in light of these changes.
We thank the reviewer for their continued engagement and recognition of our contributions. We have moved key results into the main paper and clarified their implications. Revisions are highlighted in magenta.
We appreciate the reviewer’s continued engagement and would like to clarify the substantial changes since the previous submission. These include MS-COCO instance-level retrieval (E.6), matched–mismatched alignment statistics showing over 4σ separation (E.5), calibration ablations across 12 configurations (E.7), and controls isolating orthogonality from mean correction (E.14). We also added Proposition C.27, establishing finite-anchor guarantees under representation-space coverage.
We recognize that several of these additions were previously concentrated in the appendix. To make their significance clearer, we have now moved mixed-encoder evaluations, fine-grained retrieval, and mean-correction ablations into the main paper (Sections 7.2–7.3 and 7.6, lines 381–384, 399–411, and 455–468). These constitute substantive extensions to both the theoretical and empirical contributions.
We thank the reviewer for highlighting this and agree that mixed-encoder performance is the more direct test of compatibility with the target model. We evaluated this in Appendix E.3 and Figures 11–15(g,h) of the original manuscript by pairing aligned source embeddings with native target embeddings. In response to the reviewer’s suggestion, we have moved these evaluations to the main paper of the revised manuscript (Sections 7.2–7.3, lines 360-365, 381–384, 399–411 and 427-428).
We had reported ⟨Qf(x),Qg(y)⟩ to show that mapping both modalities with Q preserves Model A's decision geometry, even allowing the aligned representations to outperform Model B while remaining natively compatible (Fig. 3(c) from the original manuscript). For a pure orthogonal map, jointly mapped embeddings preserve inner products exactly i.e. ⟨Qf(x),Qg(y)⟩=⟨f(x),g(y)⟩. However, Figure 3(c) uses modality-specific mean correction, which introduces translations and can therefore change cross-modal scores. We have clarified this distinction in Section 7.6.
We thank the reviewer for this suggestion. We additionally evaluate neighborhood overlap using Jaccard similarity under three settings. “Within-model” compares NkA(xi) with NkB(yi), measuring the intrinsic agreement between the two models’ local geometries. “Unaligned” compares the cross-model neighborhood NkA→B(xi) with the target neighborhood NkB(yi), while After Q repeats this comparison using NkQ(A)→B(Qxi). The near-zero unaligned scores show that the embedding spaces are not directly compatible, while after alignment with Q, the overlap increases significantly, even exceeding the intrinsic agreement, demonstrating that Q faithfully recovers the target model's local semantic neighborhoods. We have added neighborhood-overlap results in Section 7.3 (lines 401-411).
ViT-B/32 (OpenAI)-ViT-B/16 (OpenAI) on Caltech
We thank the reviewer for this suggestion. Controlled subsampling under the assumptions of Theorem 5.1 would provide a valuable empirical test of the theoretical prediction. Motivated by the same question, we investigate whether the predicted alignment persists even when these idealized assumptions are not guaranteed. Our experiments address precisely this challenge using independently trained models with different datasets, architectures, and optimization procedures. This provides evidence of practical robustness beyond the theorem's guaranteed setting.
We respectfully disagree. Figure 6(a) demonstrates that Q can be calibrated using images from just 10–15 anchor classes, with further evidence in Appendix E.7. Only this small subset requires encoding with f~. Once Q is learned, existing f-embeddings and new g-embeddings can both be mapped using Q, without requiring either target encoder. This is precisely the computational advantage of our approach.
We agree that the abstract's privacy statement is insufficiently supported by the main paper and have removed this claim from the abstract line 28