acceptodds
Under review as a conference paper at ICLR 2027

Orthogonal Maps Are Sufficient to Align Multimodal Contrastive Models

Abstract

As models and data scale, independently trained networks often learn similar representations. Yet establishing similarity is weaker than identifying an explicit correspondence between representation spaces, especially for multimodal models, where consistency must hold both within modalities and across the learned coupling between modalities. We therefore ask whether two independently trained multimodal contrastive models, with encoders and , trained on different data distributions and with different architectures, exhibit a systematic geometric relationship between their embedding spaces. Across model families including CLIP, SigLIP, FLAVA and CLAP, we find that this relationship is well approximated by a single orthogonal transformation, i.e., there exists an orthogonal map , with , such that for paired images (audio) . Strikingly, the same map also aligns the text encoders, so that for text inputs . Theoretically, we prove that if two models agree on the multimodal kernel over a small finite anchor set, meaning , then their image (audio) and text representations must be related by a single orthogonal map. We validate this premise, showing strong qualitative and quantitative kernel agreement. This geometric alignment enables backward-compatible model upgrades without costly re-embedding, and has implications for the privacy of learned representations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Convergence, Multimodal Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fxphOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

I have reviewed this paper for a previous conference. I've failed to notice any substantial changes to the main paper and therefore post my previous review below:

The paper claims that the embedding spaces of different multi-modal models can be aligned using a single orthogonal transformation. This transformation can be calculated using a small number of anchors from one modality and then applied to either one of the modalities (“transfers across modalities”). Experimentally, the paper validates these claims by comparing different mappings, ablating the number of anchors and computing various performance metrics of the aligned modalities on different datasets. The authors complement the above with theoretical claims showing that if multi-modal models agree on the kernel matrix w.r.t. a set of anchors, then an orthogonal transformation can map between the embeddings of different models.

Critical strengths

Overall this is a well written paper with interesting theoretical claims and empirical evidence. The theorems showing that multi-modal models converge to a similar kernel, even when trained on different subsamples of the same distribution, is a very nice finding. Additionally, the empirical results showing that different multi-modal models trained on different datasets have similar (mostly class-level) similarity structures. This is an interesting result relevant to current research on multi-modal models, that might also suggest why many of them fail on unimodal tasks (since they learn similar unimodal kernels).

Critical weaknesses

The main weakness is that the main paper mainly uses metrics such as performance on classification datasets to measure kernel agreement, and the only measure presented in the main paper (CKA) is unconvincing. Since there are many methods to measure this directly (such as nearest neighbor agreement, etc). I think that the paper would benefit from posing a question such as when do multimodal models align (answer: when they converge to their global minimum, even when trained on different subsampling of the entire training data), and investigating what happens in practice (partial alignment, mean shift is needed, etc). See fuul comments below.

Metrics

If I understand correctly, the goal of measuring performance (accuracy and recall) is a testament to the compatibility of local neighborhood structure between models. However, this is just an implicit measure. I would suggest explicitly measuring the neighborhood overlap \ nearest neighbor Jaccard coefficient between the image embeddings of CLIP models [2] if this is indeed the claim. Since cosine similarity is invariant to rotations I would expect “aligned image – aligned text” accuracy and retrieval to stay similar to before applying Q (figures 3,6), since the only change that could happen in this metric is due to the mean shift (and not rotation). Therefore, the more interesting metrics are aligned vs non-aligned (e.g. “aligned image – text”). However, these are not presented in the main paper at all.

Empirical Evidence for Theory

From my understanding, the main contribution of the paper is that multi-modal models converge to the same cross-modal kernel function, along with empirical evidence. However, the empirical part of the paper supports these claims only by proxy (as discussed above, and see comment below). Since a big part of the theory focuses on subsampling from a fixed distribution, it would be interesting and perhaps more convincing to see some experiments testing this claim in a controlled setting. Currently most of the empirical verification of this theorem is for models trained on e.g. LAION vs OpenAI for which the weights ua,vau_a,v_a are unknown , therefore possibly very close between the two models.

Usefulness of the method

In order to use the method for re-embedding, one would need to have 2 trained multi-modal models (f,gf,g, f~,g~\tilde{f}, \tilde{g}), re-embed the dataset using ff, calculate QQ, keep Qf,g~Qf, \tilde{g}, meaning the true cost save of this method is just re-embedding using gg, which I would argue is not that much since re-embedding with ff is still required. The larger concern is that there is no evidence in the main paper (and only little in the appendix) that this method can be used to improve Model B using an encoder from Model A (contrary to [1] for example). So the claim that "it supports model upgrades without re-encoding while keeping the embedding geometry intact" is questionable - does it really support upgrades (rather than just switchability)?

Furthermore, there seems to be some circular logic in this claim: "Since it preserves inner products, it supports model upgrades without re-encoding while keeping the embedding geometry intact" If inner products are preserved between two models, wouldn't that nullify the need for upgrade (since the models agree on inner products, therefore performance)?

Minor Comments

Abstract claims implications for privacy but this isn’t discussed in the main paper.

Figure 2: Where in the figure do you see that the unimodal kernel isn’t preserved? Nearest neighbors between modalities seem the same between the two models. The CKA alignment numbers don’t seem so high as claimed.

Section 5.2 - From the first paragraph it isn't clear what is your contribution and what is background.

Figure 3 - what does After Q measure in (c)? <Qf,Q^Tg>? If Q is orthogonal why shouldn't this be close to model A's performance (up to error from the mean shift)

Generally, while there is a single result on FLICKR30k, the main paper rarely presents results on dense prediction tasks, and the existing results, mainly on Oxford Pets, suggest that the consistency between models is at the class level rather than the instance level (as can be seen in figure 7, or figure 59).

[1] Jang, Young Kyun, and Ser-nam Lim. "Towards cross-modal backward-compatible representation learning for vision-language models.", 2025

[2] "Hierarchical nucleation in deep neural networks", Doimo et al, NeurIPS 2020

Questions for authors
  1. The theory assumes that the models converge to their global minimum for kernel agreement, is this indeed the case?

  2. The role of adding \ subtracting the modality mean is unclear, could you please explain how and why it contributes to the method?

  3. Are there any other ways to measure kernel agreement except performance?

  4. "It turns out that agreement on a finite set of anchors, such as a downstream image-text dataset, is enough" - from the theorems this is enough to align on ΩX/Y\Omega_{X / Y}, is it also enough to align on the population level? Under what conditions if so?

9 comments

We thank the reviewer for their continued engagement and recognition of our contributions. We have moved key results into the main paper and clarified their implications. Revisions are highlighted in magenta.

(Q1) Substantial changes to the paper from previous submission

We appreciate the reviewer’s continued engagement and would like to clarify the substantial changes since the previous submission. These include MS-COCO instance-level retrieval (E.6), matched–mismatched alignment statistics showing over 4σ4\sigma separation (E.5), calibration ablations across 12 configurations (E.7), and controls isolating orthogonality from mean correction (E.14). We also added Proposition C.27, establishing finite-anchor guarantees under representation-space coverage.

We recognize that several of these additions were previously concentrated in the appendix. To make their significance clearer, we have now moved mixed-encoder evaluations, fine-grained retrieval, and mean-correction ablations into the main paper (Sections 7.2–7.3 and 7.6, lines 381–384, 399–411, and 455–468). These constitute substantive extensions to both the theoretical and empirical contributions.

(Q2) Evaluations that combine mapped source embeddings with native target embeddings in main text

We thank the reviewer for highlighting this and agree that mixed-encoder performance is the more direct test of compatibility with the target model. We evaluated this in Appendix E.3 and Figures 11–15(g,h) of the original manuscript by pairing aligned source embeddings with native target embeddings. In response to the reviewer’s suggestion, we have moved these evaluations to the main paper of the revised manuscript (Sections 7.2–7.3, lines 360-365, 381–384, 399–411 and 427-428).

We had reported ⟨Qf(x),Qg(y)⟩\langle Qf(x),Qg(y)\rangle to show that mapping both modalities with QQ preserves Model A's decision geometry, even allowing the aligned representations to outperform Model B while remaining natively compatible (Fig. 3(c) from the original manuscript). For a pure orthogonal map, jointly mapped embeddings preserve inner products exactly i.e. ⟨Qf(x),Qg(y)⟩≠⟨f(x),g(y)⟩\langle Qf(x), Qg(y) \rangle \neq \langle f(x), g(y) \rangle. However, Figure 3(c) uses modality-specific mean correction, which introduces translations and can therefore change cross-modal scores. We have clarified this distinction in Section 7.6.

(Q3) Measuring neighborhood overlap

We thank the reviewer for this suggestion. We additionally evaluate neighborhood overlap using Jaccard similarity under three settings. “Within-model” compares NkA(xi)N_k^A(x_i) with NkB(yi)N_k^B(y_i), measuring the intrinsic agreement between the two models’ local geometries. “Unaligned” compares the cross-model neighborhood NkA→B(xi)N_k^{A\rightarrow B}(x_i) with the target neighborhood NkB(yi)N_k^B(y_i), while After QQ repeats this comparison using NkQ(A)→B(Qxi)N_k^{Q(A)\rightarrow B}(Qx_i). The near-zero unaligned scores show that the embedding spaces are not directly compatible, while after alignment with QQ, the overlap increases significantly, even exceeding the intrinsic agreement, demonstrating that QQ faithfully recovers the target model's local semantic neighborhoods. We have added neighborhood-overlap results in Section 7.3 (lines 401-411).

ViT-B/32 (OpenAI)-ViT-B/16 (OpenAI) on Caltech

Methodk=1k=5k=10k=20
Within-model34.542.449.651.5
Unaligned0.20.30.40.6
after QQ47.956.362.865.4

(Q4) Testing theory in a controlled setting with subsampled distributions

We thank the reviewer for this suggestion. Controlled subsampling under the assumptions of Theorem 5.1 would provide a valuable empirical test of the theoretical prediction. Motivated by the same question, we investigate whether the predicted alignment persists even when these idealized assumptions are not guaranteed. Our experiments address precisely this challenge using independently trained models with different datasets, architectures, and optimization procedures. This provides evidence of practical robustness beyond the theorem's guaranteed setting.

(Q5) Cost saving is mainly not embedding gg

We respectfully disagree. Figure 6(a) demonstrates that QQ can be calibrated using images from just 10–15 anchor classes, with further evidence in Appendix E.7. Only this small subset requires encoding with f~\tilde f. Once QQ is learned, existing ff-embeddings and new gg-embeddings can both be mapped using QQ, without requiring either target encoder. This is precisely the computational advantage of our approach.

(Q6) Abstract claims implications for privacy but this isn’t discussed in the main paper.

We agree that the abstract's privacy statement is insufficiently supported by the main paper and have removed this claim from the abstract line 28

(Q7) Our approach supports model upgrades rather than just model switchability

Under exact alignment, both models have identical cross-modal scores, with QQ resolving their coordinate mismatch. In practice, alignment is approximate, and mixed-system performance can differ from either model. Indeed, Table 1 shows that ⟨Qf,g~⟩\langle Qf, \tilde g \rangle achieves 84.9%, outperforming native Model A (82.4%). Thus, our method enables encoder upgrades without recomputing stored embeddings, although performance gains are not guaranteed. We have further clarified the implications for model upgrades in Section 7.2 (lines 381-383).

(Q8) Unimodal kernel preservation in Figure 2

Fig. 2 does not test unimodal-kernel preservation. It was intended to illustrate strong agreement of the multimodal kernel. In the revised manuscript, we have revised the text and caption to avoid implying that it confirms or refutes PRH at the unimodal level. Further, under exact population minimization, our theory yields f~=Qf\tilde f=Qf and g~=Qg\tilde g=Qg with Q⊤Q=IQ^\top Q=I. Hence, unimodal kernels are preserved automatically, and aligned images coincide exactly with their target representations.

(Q9) Clarity of contributions in Section 5.2

We thank the reviewer for this suggestion. While the optimal contrastive score's connection to PMI is established, our contribution is proving that distinct training distributions can yield the same optimal score under explicit curation assumptions. Sections 5.3–5.4 then establish when kernel agreement implies a shared orthogonal map. We have made these contributions clearer in Section 5.2 in lines 218-223

(Q10) Findings only work for classification benchmarks

We respectfully disagree that our findings are limited to class-level alignment. Beyond classification, we demonstrate fine-grained image–text retrieval on Flickr30k (Figure 5) and MS-COCO (Appendix E.6, Table 5). On MS-COCO, ⟨Qf,g~⟩\langle Qf,\tilde g\rangle achieves 48.0%/27.8% Recall@1, approaching native Model B's 51.8%/30.3%. These results provide direct evidence of instance-level cross-modal alignment. To emphasize this finding, we have moved the fine-grained retrieval results to Section 7.3 (lines 395–406) in the revised manuscript.

To further strengthen this claim, we now evaluate data efficiency for instance-level retrieval on Flickr30k (Appendix E.7, Fig. 19). With only 2,500 anchor images, fewer than 10% of the 29,000-image training set, unseen image-to-text Recall@1 reaches 68.9%, compared with 71.0% when QQ is fit on the full training set. Thus, a small calibration set recovers most of the full-data retrieval performance, showing that QQ generalizes beyond its fitting examples.

(Q11) Why shouldn't ⟨Qf,Qg⟩\langle Qf, Qg \rangle be close to model A's performance

Indeed, for pure QQ, $\langle Qf,Qg\rangle=\langle f,g\rangle$ exactly. Figure 3(c) in the original manuscript, however, used modality-specific mean correction, which changed cross-modal scores. Table 1 in the revised manuscript (Table 6 in the original manuscript) shows that centering improves mixed-encoder accuracy from 80.2% to 84.9% and 77.6% to 82.6%, while text alignment cosine increases from 0.287 to 0.822. Jointly mapped accuracy remains close to Model A, as expected. We have clarified this in Section 7.6 (lines 455–468).

(Q12) Assumes that the models converge to their global minimum for kernel agreement

We agree that pretrained models need not reach the population optimum. Our theory establishes kernel agreement under Bayes-optimal learning and the stated distributional assumptions. Crucially, Section 7 independently validates the predicted alignment across real pretrained models, without assuming global optimality.

(Q13) Role of mean shift

Mean correction accounts for a systematic offset between source and target embedding distributions. For a fixed orthogonal matrix QQ, the optimal constant translation under squared-error alignment is

tm=μmB−QμmAt_m=\mu_m^B-Q\mu_m^A

Thus, the centered transformation is Φm(z)=Qz+tm\Phi_m(z)=Qz+t_m. This adjusts the first moment but does not identify the orthogonal correspondence. The distinction is supported by Table 1 in the revised manuscript (Table 6 in the original manuscript). Mean shift alone fails to recover useful mixed-encoder accuracy, whereas pure QQ already achieves substantial transfer. Centering further improves pointwise agreement and some downstream metrics. We have expanded the discussion and ablations on mean correction in Section 7.6 (lines 455–468).

(Q14) Other ways to measure kernel agreement except performance?

We respectfully disagree that kernel agreement is evaluated only indirectly. Figure 2(c) already measures agreement between the native multimodal kernels using Centered Kernel Alignment (CKA), without fitting or applying QQ. CKA is a well-established metric for comparing learned representation geometry (Kornblith et al., ICML 2019), with strong theoretical foundations in kernel comparison (Cortes et al., JMLR 2012). Its invariance to orthogonal transformations makes it particularly appropriate for our setting, where embedding coordinate systems are arbitrary.

Beyond CKA, as detailed in response Q3, we now also evaluate nearest-neighbor overlap using Jaccard similarity to assess local geometric compatibility. On Caltech-101, top-10 overlap increases from 0.4% before alignment to 62.8% after QQ, exceeding the 49.6% within-model baseline. This demonstrates substantial coordinate mismatch before alignment and strong recovery of target-model neighborhoods after alignment. We have added these results to Section 7.3 (lines 401–411), complementing CKA's rotation-invariant assessment of kernel agreement.

(Q15) Conditions under which agreement on a finite set of anchors imply agreement at population level

Finite anchor agreement alone does not guarantee population-level alignment. Assumption 5.2 establishes global alignment from agreement along anchor-induced kernel sections, subject to the stated rank conditions. Crucially, Proposition C.27 extends this to purely finite anchor–anchor agreement, provided the anchors sufficiently cover the representation space and are well-conditioned.

We hope the additional revisions to the main manuscript address the reviewer's concerns. We would be grateful if the reviewer could reconsider their assessment in light of these changes.

Reviewer_ybQ6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the geometry between independently trained image/text and audio/text contrastive encoders. It argues that one orthogonal Procrustes map, learned from paired examples of a single modality, can also align the other modality. A conditional identifiability result is developed from multimodal-kernel agreement, with evaluations on CLIP, SigLIP, FLAVA, and CLAP pairs.

Critical strengths
  1. Transfer to the modality not used to fit the rotation is the most interesting result. The cross-dataset, reverse-direction, and audio-text experiments make this a useful empirical finding beyond applying Procrustes to two image encoders.
  2. The theory distinguishes linear identifiability from orthogonal identifiability and states the additional spanning requirement. The approximate and low-dimensional extensions help clarify when cross-modal transfer should be expected.
Critical weaknesses
  1. The finite-anchor claim is stronger than the main theorem establishes. Assumption 5.2 requires agreement between anchors and every input in the opposite domain, not merely agreement on a finite anchor matrix. Proposition C.27 adds a coverage condition, while the stability bounds depend on anchor conditioning. These qualifications are not reflected in the claim about a small set of dd anchors in Section 7.3.
  2. Several prominent results use aligned-image-to-aligned-text accuracy. For an uncentered orthogonal map, ⟨Qf,Qg⟩=⟨f,g⟩\langle Qf,Qg\rangle=\langle f,g\rangle automatically, so this metric cannot establish alignment with the target model. Mixed-encoder results and centering controls are provided, but should carry the main argument. The centered implementation also uses modality-specific target means, which qualifies the claim that text-side information is unnecessary.
Questions for authors
  1. Can you distinguish finite-matrix agreement from anchor-section agreement in the main claims, and report whether measured kernel errors and anchor conditioning yield informative bounds for the evaluated models?
  2. How are target text means obtained, and do they include evaluation texts? What are the main mixed-encoder results when no target text statistics are used?

Post-rebuttal update: I read the revised PDF and the authors' response (see my comment below for details). Moving the mixed-encoder results and the centering ablation (Table 1) into the main text clarifies the role of pure Q, and the training-split mean calculation is now clear. The distinction between Assumption 5.2 and Proposition C.27 is also clearer in the revision. My main questions on the experimental setup are addressed, and I am keeping my score at 3.

3 comments

We appreciate the reviewer's positive assessment, particularly their recognition of cross-modal transfer as a key contribution and the value of our generalization experiments. We have further clarified the finite-anchor guarantees and strengthened the mixed-encoder evaluations in response to their suggestions, as detailed below, and updated the manuscript with corresponding revisions

(Q1) Regarding finite-anchor guarantees

Assumption 5.2 requires kernel agreement between a finite set of anchors and all inputs of the opposite modality, not merely agreement on a finite anchor matrix. We have clarified this distinction throughout the revised manuscript, including the Introduction, Sections 5.3–5.4, Appendix A, and revised the abstract and main claims accordingly.

Importantly, Proposition C.27 establishes a separate guarantee for finite anchor–anchor agreement. When the anchors sufficiently cover the representation spaces, finite kernel error ε\varepsilon yields approximate kernel-section agreement with errors bounded by ε+ρX\varepsilon+\rho_X and ε+ρY\varepsilon+\rho_Y, where ρX,ρY\rho_X,\rho_Y quantify coverage. Together with the conditioning and spanning assumptions in Appendix C.6, this yields approximate alignment guarantees. Our few-anchor experiments in Section 7.4 demonstrate it’s empirical generalization.

(Q2) Evaluations that combine mapped source embeddings with native target embeddings in main text

We thank the reviewer for highlighting this and agree that mixed-encoder performance is the more direct test of compatibility with the target model. We evaluated this in Appendix E.3 and Figures 11–15(g,h) of the original manuscript by pairing aligned source embeddings with native target embeddings. In response to the reviewer’s suggestion, we have moved these evaluations to the main paper of the revised manuscript (Sections 7.2–7.3, lines 360-365, 381–384, 399–411 and 427-428).

We had reported ⟨Qf(x),Qg(y)⟩\langle Qf(x),Qg(y)\rangle to show that mapping both modalities with QQ preserves Model A's decision geometry, even allowing the aligned representations to outperform Model B while remaining natively compatible (Fig. 3(c) from the original manuscript). For a pure orthogonal map, jointly mapped embeddings preserve inner products exactly i.e. ⟨Qf(x),Qg(y)⟩≠⟨f(x),g(y)⟩\langle Qf(x), Qg(y) \rangle \neq \langle f(x), g(y) \rangle. However, Figure 3(c) uses modality-specific mean correction, which introduces translations and can therefore change cross-modal scores. We have clarified this distinction in Section 7.6.

(Q3) Distinguish image-only fitting of (Q) from a complete pipeline that requires no text-side information.

Table 6 in the original manuscript already isolates the contribution of QQ. On Caltech-101, the two mixed-encoder accuracies are 80.2% and 77.6% without mean correction, compared with 84.9% and 82.6% with correction. Mean shift alone gives 0.5% and 1.2%. Thus Q recovers substantial cross-model compatibility without text-mean correction, while the translation provides an additional, measurable benefit. For fixed Q, the least-squares optimal translation is μmB−QμmA\mu_m^B − Q\mu_m^A, which removes the mean alignment residual.

In light of the reviewer’s feedback, we have moved these results to Table 1 in the main paper and expanded its discussion in Section 7.6 (lines 453–477).

Below, we further test whether image statistics alone suffice for mean correction. On Oxford Pets, applying image-derived translations to both modalities yields substantially lower text alignment (0.356 vs. 0.790) and weaker performance (72.2% vs. 76.2%) than modality-specific correction. This confirms that text-specific statistics are essential for effective text alignment.

Mean correctioncos⁡(Qf,f~)\cos(Qf,\tilde f)cos⁡(Qg,g~)\cos(Qg,\tilde g)⟨Qf,g~⟩\langle Qf,\tilde g\rangle⟨f~,Qg⟩\langle\tilde f,Qg\rangle⟨Qf,Qg⟩\langle Qf,Qg\rangle
None (pure QQ)0.5910.34568.771.983.9
Image means for both modalities0.8890.35665.472.270.2
Modality-specific means0.8890.79068.476.275.2

We hope these additions and clarifications address the reviewer's remaining concerns and would appreciate their reconsideration of the score.

Reviewer_ybQ6OpenReview ↗

Review

Thanks for the detailed response and the revisions to the paper.

Moving the mixed-encoder results and the centering ablation into the main text (Table 1 and Sections 7.2-7.6) makes the contribution of the pure orthogonal map Q much clearer. On Caltech-101, Q without mean centering still reaches 80.2% and 77.6% accuracy across the two mixed-encoder directions, which addresses my main empirical question. Thank you also for clarifying in Section 7.1 that the modality means are computed only on the training split. The new Oxford Pets control in the rebuttal is also informative and shows that using image means for both modalities leaves a clear gap on the text side (0.356 vs. 0.790 cosine similarity).

On the theory side, the revised text now does a better job of separating anchor-section agreement (Assumption 5.2) from finite anchor-anchor agreement under coverage (Proposition C.27). While the paper still does not plug in the empirical kernel errors and conditioning constants to check how tight the stability bounds are in practice, the few-anchor empirical results in Section 7.4 are already interesting on their own.

Overall, the finding that a single orthogonal map fit on one modality transfers directly to the other modality remains the main strength of the paper. The revisions address my questions on the experimental setup, and I will keep my score at 3 .

We thank the reviewer for carefully examining our revisions and for their thoughtful follow-up. We are pleased that the new mixed-encoder results and theoretical clarifications address your main questions. If our revisions have satisfactorily addressed your concerns, we would be grateful if you could consider increasing your score further.

Reviewer_KpPiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether two independently trained multimodal contrastive models, with different settings, exhibit a systematic geometric relationship between their embedding spaces. They show an orthogonal map relationship, which enables backward-compatible model upgrades.

Critical strengths

The motivation for finding such a geometric relationship is interesting. The paper provides theoretical justification.

Critical weaknesses
  1. The implemented transformation differs materially from a single orthogonal map. The principal experiments use a shared orthogonal matrix together with modality-specific mean corrections. Table 6 shows that matched text cosine similarity increases from 0.287 without mean correction to 0.822 with correction, indicating that this component substantially affects a central result. The authors should clarify how text-side means are obtained and distinguish image-only fitting of QQ from a complete pipeline that requires no text-side information. Whether these means are re-estimated during cross-dataset evaluation should also be explicitly stated.

  2. Jointly transforming both source modalities is not an independent test of cross-model alignment. For a pure orthogonal map, ⟨Qf(x),Qg(y)⟩=⟨f(x),g(y)⟩\langle Qf(x),Qg(y)\rangle=\langle f(x),g(y)\rangle holds identically. Consequently, aligned-image–aligned-text performance primarily reflects preservation of source-model geometry, not successful alignment with the target model. The main text should prioritize evaluations that combine mapped source embeddings with native target embeddings and clearly distinguish these two types of evidence.

  3. The connection between finite-anchor theory and small-sample calibration is insufficiently established. Assumption 5.2 requires agreement of anchor-induced kernel sections throughout the domain, rather than only agreement among finitely many anchors. The paper should more clearly distinguish conditional identifiability from finite-sample generalization and investigate whether the relevant kernel errors, coverage terms, and conditioning constants make the approximate guarantees informative in the evaluated settings.

  4. A theory-matched linear baseline would make the comparison more informative. Theorem 5.3 naturally suggests applying AA to images and A−⊤A^{-\top} to text. Although this is not a shared-matrix method, it can be constructed without separately fitting text correspondences. Comparing against a regularized version of this baseline would help determine whether orthogonality is essential to the underlying correspondence or primarily beneficial as a stable inductive constraint.

  5. Under unit normalization, Theorem 5.5 appears to force all target text embeddings into Im⁡(Q)\operatorname{Im}(Q). How is this compatible with the assumed invertibility of G~\widetilde G when d~>d\widetilde d>d? In addition, do measured kernel errors and anchor-conditioning statistics yield non-vacuous stability guarantees?

Questions for authors

see weakness.

4 comments

We thank the reviewer for their thoughtful assessment and for recognizing the motivation and theoretical foundations of our work. We appreciate the constructive suggestions, particularly regarding the role of mean correction, mixed-encoder evaluations, and the scope of our theoretical guarantees. We address each concern below and have revised the manuscript accordingly, with changes highlighted in magenta.

(Q1) Clarify how text-side means are obtained and whether they are re-estimated during cross-dataset evaluation

We thank the reviewer for the opportunity to elucidate this procedure. We fit QQ using paired image embeddings only. When applying QQ to text, we also correct the mean offset: ΦT(z)=Q(z−μTA)+μTB\Phi_T(z)=Q(z-\mu_T^A)+\mu_T^B, where μTA\mu_T^A and μTB\mu_T^B are computed from the training image/texts using the source (A) and target (B) encoders, respectively. In cross-dataset evaluation, we keep QQ fixed but recompute the modality-specific means using the new dataset’s training split. Thus, the rotation transfers without refitting. We have updated the manuscript to further clarify this distinction in Section 7.1 (lines 337–339).

(Q2) Distinguish image-only fitting of (Q) from a complete pipeline that requires no text-side information.

Table 6 in the original manuscript already isolates the contribution of QQ. On Caltech-101, the two mixed-encoder accuracies are 80.2% and 77.6% without mean correction, compared with 84.9% and 82.6% with correction. Mean shift alone gives 0.5% and 1.2%. Thus Q recovers substantial cross-model compatibility without text-mean correction, while the translation provides an additional, measurable benefit. For fixed Q, the least-squares optimal translation is μmB−QμmA\mu_m^B − Q\mu_m^A, which removes the mean alignment residual.

In light of the reviewer’s feedback, we have moved these results to Table 1 in the main paper and expanded its discussion in Section 7.6 (lines 453–477).

Below, we further test whether image statistics alone suffice for mean correction. On Oxford Pets, applying image-derived translations to both modalities yields substantially lower text alignment (0.356 vs. 0.790) and weaker performance (72.2% vs. 76.2%) than modality-specific correction. This confirms that text-specific statistics are essential for effective text alignment.

Mean correctioncos⁡(Qf,f~)\cos(Qf,\tilde f)cos⁡(Qg,g~)\cos(Qg,\tilde g)⟨Qf,g~⟩\langle Qf,\tilde g\rangle⟨f~,Qg⟩\langle\tilde f,Qg\rangle⟨Qf,Qg⟩\langle Qf,Qg\rangle
None (pure QQ)0.5910.34568.771.983.9
Image means for both modalities0.8890.35665.472.270.2
Modality-specific means0.8890.79068.476.275.2

(Q3) Evaluations that combine mapped source embeddings with native target embeddings in main text

We thank the reviewer for highlighting this and agree that mixed-encoder performance is the more direct test of compatibility with the target model. We evaluated this in Appendix E.3 and Figures 11–15(g,h) of the original manuscript by pairing aligned source embeddings with native target embeddings. In response to the reviewer’s suggestion, we have moved these evaluations to the main paper of the revised manuscript (Sections 7.2–7.3, lines 360-365, 381–384, 399–411 and 427-428).

We had reported ⟨Qf(x),Qg(y)⟩\langle Qf(x),Qg(y)\rangle to show that mapping both modalities with QQ preserves Model A's decision geometry, even allowing the aligned representations to outperform Model B while remaining natively compatible (Fig. 3(c) from the original manuscript). For a pure orthogonal map, jointly mapped embeddings preserve inner products exactly i.e. ⟨Qf(x),Qg(y)⟩≠⟨f(x),g(y)⟩\langle Qf(x), Qg(y) \rangle \neq \langle f(x), g(y) \rangle. However, Figure 3(c) uses modality-specific mean correction, which introduces translations and can therefore change cross-modal scores. We have clarified this distinction in Section 7.6.

(Q4) Agreement on finite-samples versus kernel sections

Proposition C.27 in Appendix C.6.3 bridges finite anchor–anchor agreement and approximate kernel-section agreement under a representation-space coverage condition. Combined with the stated conditioning and diversity assumptions, it yields the approximate alignment guarantees. We have clarified this distinction throughout the revised manuscript, including the Introduction, Sections 5.3–5.4 and Appendix, and further emphasized the role of Proposition C.27 and its conditions. Establishing whether the bounds are numerically informative for the evaluated models remains a separate empirical question for future work.

(Q5) Applying A−1A^{-1} to text instead of an orthogonal map

We thank the reviewer for this great suggestion! Following Theorem 5.3, we fit a linear map AA on paired images and apply its exact or regularized inverse transpose to text. Although AA aligns the images well, its inverse transpose amplifies directions with small singular values and reduces classification accuracy to chance. Regularization helps, but QQ performs better without an inverse to regularize. We have included this theory-matched baseline in Appendix E.13 (lines 3743–3756, Table 7).

Oxford Pets zero-shot accuracy (%) for OpenAI ViT-B/32 to LAION ViT-B/32, using native target images and mapped source text.

Image mapText map⟨Qf,g~⟩\langle Qf,\tilde g\rangle⟨f~,Qg⟩\langle\tilde f,Qg\rangle⟨Qf,Qg⟩\langle Qf,Qg\rangle
QQQQ68.476.275.2
AAA−⊤A^{-\top}64.12.82.7
AARegularized A−⊤A^{-\top} (best tested)64.156.955.8

(Q6) Theorem 5.5 under unit normalization

We thank the reviewer for identifying this apparent inconsistency. We agree that under unit normalization, Theorem 5.5 implies g~(y)=Qg(y)\tilde g(y)=Qg(y), so when d~>d\tilde d>d, the target text embeddings lie in Im⁡(Q)\operatorname{Im}(Q) and G~\tilde G must be singular. Crucially, Theorem 5.3 does not require G~\tilde G to be invertible. We have clarified its assumption to f~(x)∈col⁡(G~)\tilde f(x)\in\operatorname{col}(\tilde G), which is sufficient for the pseudoinverse argument, and explicitly stated the singularity of G~\tilde G in the unequal-dimensional case (Sections 5.3–5.4, lines 263–287; Appendix C.5).

Appendix C.6 provides stability bounds in terms of kernel errors, anchor conditioning, and representation-space coverage. While our experiments demonstrate strong alignment, evaluating these bounds numerically requires estimating worst-case quantities beyond the measured benchmarks. We therefore leave a detailed assessment of their tightness for future work.

We hope these additional experiments and clarifications address the reviewer's concerns regarding empirical compatibility and theoretical guarantees, and would be grateful if the reviewer could reconsider their assessment.

We wanted to follow up on our response and revised manuscript. If our revisions address your main concerns, we would be grateful if you could reconsider your assessment accordingly.

With the November 18 deadline approaching, we wanted to follow up on our response and revised manuscript. If our revisions satisfactorily resolve your concerns, we would appreciate your consideration of a score increase.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.