Orthogonal Maps Are Sufficient to Align Multimodal Contrastive Models

As models and data scale, independently trained networks often learn similar representations. Yet establishing similarity is weaker than identifying an explicit correspondence between representation spaces, especially for multimodal models, where consistency must hold both within modalities and across the learned coupling between modalities. We therefore ask whether two independently trained multimodal contrastive models, with encoders $(f,g)$ and $(\tilde{f},\tilde{g})$, trained on different data distributions and with different architectures, exhibit a systematic geometric relationship between their embedding spaces. Across model families including CLIP, SigLIP, FLAVA and CLAP, we find that this relationship is well approximated by a single orthogonal transformation, i.e., there exists an orthogonal map $Q$, with $Q^{\top }Q=I$, such that $\tilde{f}(x)\approx Qf(x)$ for paired images (audio) $x$. Strikingly, the same map also aligns the text encoders, so that $\tilde{g}(y)\approx Qg(y)$ for text inputs $y$. Theoretically, we prove that if two models agree on the multimodal kernel over a small finite anchor set, meaning $⟨f(x),g(y)⟩\approx ⟨\tilde{f}(x),\tilde{g}(y)⟩$, then their image (audio) and text representations must be related by a single orthogonal map. We validate this premise, showing strong qualitative and quantitative kernel agreement. This geometric alignment enables backward-compatible model upgrades without costly re-embedding, and has implications for the privacy of learned representations.