acceptodds
Under review as a conference paper at ICLR 2027

Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data

55 views

Abstract

Multimodal representations enable zero-shot classification and retrieval, but aligning independently trained models usually requires large amounts of paired data. Yet, the Platonic Representation Hypothesis suggests that models trained on different modalities may converge spontaneously toward a shared representation geometry. But then, do we even need paired examples for cross-modal alignment? Remarkably, we show that paired examples are unnecessary for coarse cross-modal alignment. Our simple Wasserstein Procrustes method with a coarse geometric initialization aligns two disjoint embedding sets by estimating a single orthogonal map without seeing any pairs. Across datasets, modalities, and unimodal models, we show that we can consistently align independently trained representations without pairs, and standard geometric alignment metrics accurately predict when this is possible. Nevertheless, we can naturally benefit from paired examples. In the very few-pair regime, our method substantially outperforms existing ones, while staying competitive with pair-based methods with more added examples. Finally, we demonstrate that the resulting alignments can enable text-to-image generation without paired examples. These results show that independently trained models often share enough geometry to establish cross-modal correspondence with little or no paired data.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
unpaired alignment, cross-modal alignment, representation geometry, multimodal representation learning, few-pair learning, platonic representation hypothesis, optimal transport

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WAniOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether independently trained representations from different modalities can be aligned without any paired cross-modal examples. The proposed method builds on Wasserstein Procrustes alignment and introduces a coarse geometric initialization: the two embedding spaces are independently clustered, cluster centers are matched by maximizing their geometric similarity, and the resulting coarse correspondence is converted into sample-level pseudo-pairs before Procrustes refinement. The same framework can also incorporate a small number of paired examples when available. The authors evaluate the method across multiple vision-language encoders and datasets, including a cross-dataset setting with no hidden sample-level correspondence, as well as several scientific and biomedical modality pairs. The results show that shared geometry can recover non-trivial coarse cross-modal alignment, substantially improve alignment in the low-pair regime, and support downstream tasks including zero-shot classification, retrieval, and text-to-image generation. Additional analyses suggest that the shared structure is primarily coarse rather than fine-grained.

Critical strengths

S1. Interesting idea. The paper studies an interesting and important problem: how far cross-modal alignment can be achieved from representation geometry alone, without paired data.

S2. Comprehensive experiments. The empirical evaluation is comprehensive, covering many model combinations, datasets, and modalities, including useful cross-dataset and downstream evaluations.

Critical weaknesses

W1. The paper shows that shared geometry enables coarse alignment, but it is less clear when geometry is sufficient to uniquely identify the semantically correct correspondence. In symmetric or near-symmetric cases, multiple permutations may achieve similar geometric objectives. Is the success mainly due to real representation geometries being sufficiently asymmetric, or is the method robust to such ambiguity? A controlled analysis of this issue would be helpful.

Questions for authors

Q1. In cases where the geometry admits multiple equivalent or near-equivalent permutations, how does the proposed method select the semantically correct alignment? Are such ambiguities rare in real representation spaces, or is the method robust to them? Some analysis or controlled experiments on this issue would help clarify the limits of the approach.

Q2. The method appears to rely on the two unpaired sets having reasonably similar semantic coverage and distributions. How sensitive is the method to mismatched semantic distributions, for example when some semantic groups are missing on one side or their frequencies differ substantially?

5 comments
AuthorsOpenReview ↗

Response to Reviewer WAni (1/3): W1+Q1 (synthetic)

We are glad that the reviewer finds our problem “interesting and important” and recognizes our “comprehensive experiments”. We thank the reviewer for raising two important questions about the limitations of our approach: whether symmetries in the embedding geometry can lead to ambiguous cluster correspondences, and whether differences in semantic class frequencies between modalities can impair alignment. Interestingly, real representations are not symmetric on coarse scales, and symmetries mostly occur between similar classes. However, our method is sensitive to substantial distribution mismatches, and we have added an explicit discussion of this in our paper.

W1+Q1: “Is the success mainly due to real representation geometries being sufficiently asymmetric, or is the method robust to such ambiguity?”

Real embedding spaces are typically not symmetric at the coarse level. Fine-level near-symmetries are common, but they mainly swap semantically similar classes like a baseball bat and a glove and therefore preserve the coarse alignment. Our method has no rule for breaking ties. It averages the cluster matchings across 3030 restarts, so under exact symmetry, no method that uses only geometry can pick the right matching. We perform three synthetic and one MS COCO experiment to investigate symmetries.

  1. Global symmetry: We study how symmetry affects cluster matching by interpolating between a perfect symmetric space of 3030 cluster centers (t=0t = 0, a regular simplex) and random normalized centers (t=1t = 1). The parameter tt therefore controls the degree of symmetry. We independently add Gaussian noise to the centers in each space, where std\text{std} controls the noise magnitude. Thus, std=0\text{std} = 0 means identical centers in both spaces, while larger values make the spaces increasingly different. We then independently rotate both spaces and evaluate alignment using FOSCTTM (0.50.5 corresponds to chance), averaged over five seeds. Further details are provided in Appendix F.

    Table R5. FOSCTTM on the simplex toy example.

    std\text{std}  symmetry tt000.010.010.030.030.10.10.30.311
    000.511±0.0300.511 \pm 0.0300.501±0.0220.501 \pm 0.0226.6×10−5±3.9×10−56.6 \times 10^{-5} \pm 3.9 \times 10^{-5}3.5×10−5±1.6×10−53.5 \times 10^{-5} \pm 1.6 \times 10^{-5}4.4×10−5±3.9×10−54.4 \times 10^{-5} \pm 3.9 \times 10^{-5}6.9×10−5±8.1×10−56.9 \times 10^{-5} \pm 8.1 \times 10^{-5}
    0.030.030.501±0.0110.501 \pm 0.0110.504±0.0200.504 \pm 0.0204.8×10−1±2.2×10−24.8 \times 10^{-1} \pm 2.2 \times 10^{-2}3.7×10−5±2.0×10−53.7 \times 10^{-5} \pm 2.0 \times 10^{-5}5.5×10−5±3.6×10−55.5 \times 10^{-5} \pm 3.6 \times 10^{-5}6.8×10−5±3.2×10−56.8 \times 10^{-5} \pm 3.2 \times 10^{-5}
    0.10.10.483±0.0260.483 \pm 0.0260.485±0.0330.485 \pm 0.0335.2×10−1±2.9×10−25.2 \times 10^{-1} \pm 2.9 \times 10^{-2}4.4×10−1±4.9×10−24.4 \times 10^{-1} \pm 4.9 \times 10^{-2}8.2×10−5±3.8×10−58.2 \times 10^{-5} \pm 3.8 \times 10^{-5}9.4×10−5±5.6×10−59.4 \times 10^{-5} \pm 5.6 \times 10^{-5}
    0.30.30.517±0.0330.517 \pm 0.0330.499±0.0660.499 \pm 0.0664.8×10−1±4.3×10−24.8 \times 10^{-1} \pm 4.3 \times 10^{-2}5.2×10−1±3.6×10−25.2 \times 10^{-1} \pm 3.6 \times 10^{-2}4.8×10−1±5.9×10−24.8 \times 10^{-1} \pm 5.9 \times 10^{-2}8.8×10−4±1.8×10−48.8 \times 10^{-4} \pm 1.8 \times 10^{-4}

    When the centers are perfectly symmetric, the method performs randomly. Introducing a shared symmetric component that is not symmetric allows the solver to recover the correct matching, even when that component is relatively small. As independent noise increases, less symmetry is required.

  2. Mirror symmetry: Here, we consider a less symmetric setting in which clusters are mirrored, so that each cluster has only one symmetric counterpart.

    Table R6. FOSCTTM on the mirror toy example.

    std\text{std}  symmetry tt000.010.010.030.030.10.10.30.311
    000.330±0.3050.330 \pm 0.3050.401±0.2220.401 \pm 0.2220.401±0.2220.401 \pm 0.2220.202±0.2750.202 \pm 0.2752.1×10−1±2.8×10−12.1 \times 10^{-1} \pm 2.8 \times 10^{-1}5.4×10−5±3.0×10−55.4 \times 10^{-5} \pm 3.0 \times 10^{-5}
    0.030.030.301±0.2720.301 \pm 0.2720.201±0.2710.201 \pm 0.2710.301±0.2720.301 \pm 0.2720.302±0.2750.302 \pm 0.2757.6×10−5±2.7×10−57.6 \times 10^{-5} \pm 2.7 \times 10^{-5}9.2×10−5±4.5×10−59.2 \times 10^{-5} \pm 4.5 \times 10^{-5}
    0.10.10.301±0.2720.301 \pm 0.2720.250±0.2850.250 \pm 0.2850.131±0.2820.131 \pm 0.2820.505±0.0180.505 \pm 0.0181.0×10−1±2.3×10−11.0 \times 10^{-1} \pm 2.3 \times 10^{-1}1.2×10−4±3.6×10−51.2 \times 10^{-4} \pm 3.6 \times 10^{-5}
    0.30.30.147±0.0770.147 \pm 0.0770.514±0.2320.514 \pm 0.2320.408±0.2910.408 \pm 0.2910.284±0.2590.284 \pm 0.2594.2×10−3±3.1×10−34.2 \times 10^{-3} \pm 3.1 \times 10^{-3}1.3×10−3±1.6×10−41.3 \times 10^{-3} \pm 1.6 \times 10^{-4}

    At exact symmetry, runs can recover either the correct map or its mirror: for t=0t = 0, 88 of 1515 runs select the mirror solution. This directly demonstrates that the method does not reliably resolve a geometrically equivalent but semantically incorrect correspondence.

AuthorsOpenReview ↗

Response to Reviewer WAni (2/3): W1+Q1 (hierarchical, MS COCO)

  1. Hierarchical symmetry: We create six coarse groups that are not symmetric, each containing five symmetric subclusters.

    Table R7. FOSCTTM on the hierarchical toy example.

    std\text{std}  symmetry tt000.010.010.030.030.10.10.30.311
    000.073±0.0120.073 \pm 0.0120.074±0.0110.074 \pm 0.0110.056±0.0150.056 \pm 0.0150.022±0.0100.022 \pm 0.0100.008±0.0070.008 \pm 0.0070.018±0.0110.018 \pm 0.011
    0.030.030.075±0.0130.075 \pm 0.0130.076±0.0170.076 \pm 0.0170.073±0.0190.073 \pm 0.0190.046±0.0040.046 \pm 0.0040.014±0.0050.014 \pm 0.0050.017±0.0140.017 \pm 0.014
    0.10.10.077±0.0080.077 \pm 0.0080.083±0.0100.083 \pm 0.0100.104±0.0480.104 \pm 0.0480.062±0.0190.062 \pm 0.0190.058±0.0140.058 \pm 0.0140.029±0.0240.029 \pm 0.024
    0.30.30.176±0.2130.176 \pm 0.2130.076±0.0140.076 \pm 0.0140.078±0.0140.078 \pm 0.0140.127±0.1190.127 \pm 0.1190.194±0.2510.194 \pm 0.2510.069±0.0100.069 \pm 0.010

    Across the tested settings, FOSCTTM remains below 0.20.2, and 8080–100%100\% of incorrect cluster matches remain within the correct coarse group. This shows how local symmetries can preserve useful coarse alignment despite ambiguity at finer levels.

  2. MS COCO: On MS COCO, we cannot change the amount of symmetry, but we know the class of every image. So we ran the matching step of our method on the average embeddings of each class, for all 2121 vision-language model pairs.

    Table R8. Matching quality on MS COCO using class labels and joint clustering.

    ClassesMatched correctlyRuns where a wrong matching scores at least as well as the true one
    1212 superclasses92%±19%92\% \pm 19\%24%24\%
    8080 categories (subsets of 3030)90%±15%90\% \pm 15\%43%43\%
    Joint clustering into 3030 clusters85%±19%85\% \pm 19\%51%51\%

    At the coarse scale, the geometry identifies the correspondence. For 1616 of the 2121 model pairs, all 1212 superclasses are matched correctly, and on average, 92%92\% are. Random subsets of 3030 of the 8080 categories are matched with an average accuracy of 90%90\%, and 3030 joint clusters with 85%85\%. At this finer scale, however, a wrong matching scores at least as well as the true one in 43%43\% (categories) and 51%51\% (joint clusters) of the runs. These wrong matchings usually swap similar classes, e.g., a baseball bat and a glove, a pizza and a sandwich, or a motorcycle and a bicycle, and swapping a category with its closest neighbor changes the objective by only about 0.0010.001. Altogether, this is similar to the hierarchical toy example above, where symmetries exist mainly between nearby groups and the coarse structure is largely not symmetric.

More details about both experiments are in the new Appx. F. We also include it in the discussion in Sec. 5.

AuthorsOpenReview ↗

Response to Reviewer WAni (3/3): Q2

Q2: “How sensitive is the method to mismatched semantic distributions?”

Our method is sensitive to both unequal class frequencies and missing semantic groups. This assumption is used during initialization, where we search for 1-to-1 matchings in the QAP, and during readout and refinement, where we use a 1-to-1 Hungarian matching. This can also be observed empirically on MS COCO using its superclasses. We subsample 10k samples per side and vary the caption-side class frequencies or remove selected classes. We measure distribution mismatch using total variation (TV).

Different frequencies: If p∈[0,1]Cp \in [0, 1]^C is the empirical frequency of the CC classes, we can change the frequency on the text side by sampling the classes proportionally to pβp^\beta. When β\beta is smaller than 11, the groups are closer to uniformly distributed, and β>1\beta > 1 makes the large groups larger.

Table R9. FOSCTTM for different semantic frequencies.

Caption sharesTVFOSCTTM, mean of 33 pairs
β=1\beta = 1, baseline0.000.000.103±0.0100.103 \pm 0.010
β=0.5\beta = 0.5, more even0.210.210.282±0.0560.282 \pm 0.056
β=2\beta = 2, large groups larger0.280.280.441±0.0790.441 \pm 0.079
β=0.25\beta = 0.25, close to uniform0.320.320.380±0.0300.380 \pm 0.030

Missing classes: Here, we remove one or more classes from one side.

Table R10. FOSCTTM for missing classes.

Missing on one sideTVFOSCTTM, mean of 33 pairs
Food captions0.050.050.146±0.0280.146 \pm 0.028
Animal captions0.140.140.178±0.0380.178 \pm 0.038
Animal images0.140.140.161±0.0180.161 \pm 0.018
Vehicle captions0.160.160.268±0.0350.268 \pm 0.035
Person captions (the largest group)0.340.340.331±0.0620.331 \pm 0.062
Animal, food, vehicle and sports captions0.360.360.582±0.0110.582 \pm 0.011

Both experiments show that mismatched semantic distributions can degrade alignment. Small changes or the removal of a small group can still yield meaningful alignment, whereas substantial mismatches can bring performance close to the random baseline. When groups are missing or frequencies differ on one side, 1-to-1 matching assigns samples from one group to the other, and orthogonal Procrustes then fits those mismatched pairs.

We added these experiments in Appx. C.7, and we discuss the semantic distribution together with data quality in Sec. 5. Maybe unbalanced transport could help to tackle this in the future.


We thank the reviewer for raising these interesting questions. Our experiments reveal that real embeddings have surprisingly few coarse-level symmetries, which further explains why our method succeeds at coarse alignment. They also motivate future work on unbalanced mapping to address mismatched semantic distributions.

Reviewer_cfWKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a method to align embedding spaces of different pre trained frozen models trained on different modalities, in the case where no (or very few) paired data is available. They use the Wasserstein-Procrustes alignment, which jointly finds the transport plan between two embedding spaces and the best orthogonal transformation. They then use the orthogonal map to align the embedding spaces of many vision and text models, as well as models trained on different domains such as neuroscience and genomics. They show that their methods is able to recover coarse cross-model alignment for many model combinations without any supervised pairs, and benefits from additional ground truth pairs (<1000).

Critical strengths
  • Aligning models without paired data is potentially valuable in domains where the amount of paired data is limited, such as in biology and neuroscience. The paper shows this by not limiting its experiments to images and text and demonstrates that the method is effective on genomics and fMRI data.
  • The paper is very clearly presented, with highlighted findings and color codes making it easy to follow.
  • The empirical results strongly support the claims: in the vision-text setting, the method achieves strong coarse alignment without any paired examples, outperforming several baselines using paired supervision.
Critical weaknesses
  • The paper focuses on coarse alignment, and the evaluation of the learned alignment relies primarily on FOSCTTM and class-level zero-shot accuracy. While FOSCTTM measures the normalized rank of the true image–caption match, reporting retrieval Recall@1 and Recall@5 would clarify the ability of the model to distinguish more fine-grained details.
Questions for authors
  • Restricting the transformation to an orthogonal alignment seems restrictive. If the two embedding distributions differ in shape, the transformation may limit alignment. Can you compare it to an unrestricted linear map within the same pipeline? Additionally, could a mapping derived from the learned transport plan improve alignment? Additionally, could a barycentric projection derived from the transport plan improve alignment?
4 comments
AuthorsOpenReview ↗

Response to Reviewer cfWK (1/2): W1, Q1

We appreciate the reviewer’s positive assessment of our work, describing it as “valuable” and “clearly presented”, and noting that our “empirical results strongly support the claims”. We thank the reviewer for helping us improve the manuscript by suggesting additional evaluations of recall and unrestricted linear maps, beyond the orthogonal maps considered in the original submission. We address both suggestions below.

W1: “Reporting retrieval Recall@1 and Recall@5 would clarify the ability of the model to distinguish more fine-grained details”

We added the requested recall evaluation metrics. As expected, the recall is far above random but well below CLIP, suggesting a coarse alignment. We use the CLIP protocol on 1k1\mathrm{k} and 5k5\mathrm{k} images from the MS COCO Karpathy test split, with 55 captions per image. None of the test images are used in training, and an image counts as correct at k if one of its 55 captions is among the k nearest.

Table R3. Recall@1 and Recall@5 on MS COCO.

ImagesMethodImage to text R@1Image to text R@5Text to image R@1Text to image R@5
5k5\mathrm{k}ours, mean of 2121 model pairs3.3±0.53.3 \pm 0.59.2±1.49.2 \pm 1.42.5±0.42.5 \pm 0.47.9±1.17.9 \pm 1.1
5k5\mathrm{k}ours, best pair (DINOv2 G/14, Qwen3-8B)9.4±1.19.4 \pm 1.124.1±2.624.1 \pm 2.66.1±0.76.1 \pm 0.717.8±2.017.8 \pm 2.0
5k5\mathrm{k}CLIP ViT-L/14 (400M pairs)56.356.379.579.536.636.661.161.1
5k5\mathrm{k}random0.020.020.10.10.020.020.10.1
1k1\mathrm{k}ours, mean of 2121 model pairs8.9±1.38.9 \pm 1.320.2±2.720.2 \pm 2.77.0±1.07.0 \pm 1.019.2±2.419.2 \pm 2.4
1k1\mathrm{k}ours, best pair (DINOv2 G/14, Qwen3-8B)22.8±2.522.8 \pm 2.547.2±4.547.2 \pm 4.515.6±1.815.6 \pm 1.838.8±3.738.8 \pm 3.7
1k1\mathrm{k}random0.10.10.50.50.10.10.50.5

All quantitative retrieval results are substantially above random. However, even our best model pair remains considerably below CLIP. This is expected because CLIP is explicitly trained to align image and text representations, whereas our method operates on independently trained models and data without paired supervision. It confirms that our method recovers coarse semantic correspondences rather than precise sample-level matches. For example, an image may be mapped to captions describing a similar scene without identifying its own caption.

We have added the complete results, including also Recall@10, to the new Appx. C.6 and discuss the findings in Sec. 4.1.

Q1: “Can you compare it to an unrestricted linear map within the same pipeline?”

As suggested, we additionally tested unrestricted maps in the same pipeline. However, we found that they perform worse than when constrained to orthogonal maps, as in our setting. Intuitively, embedding spaces could be less isometric in the cross-modal than in the unimodal one. Our new evaluation, allowing us to reach this conclusion, consists of replacing the orthogonal transformations with linear maps from least-squares and ridge regression with three regularization strengths λ=a⋅tr⁡(X⊤X)/d\lambda = a \cdot \operatorname{tr}(\mathbf{X}^\top \mathbf{X}) / d. Everything else stays the same, and each row is the mean over all 2121 model pairs and 55 seeds on MS COCO.

Table R4. Unrestricted linear maps versus orthogonal maps.

MapFOSCTTM ↓\downarrow (0.50.5 is chance)CIFAR-10 top-1 in % ↑\uparrow
Linear, least squares0.184±0.0190.184 \pm 0.01945.2±3.945.2 \pm 3.9
Linear, ridge with a=0.01a = 0.010.179±0.0180.179 \pm 0.01845.9±3.445.9 \pm 3.4
Linear, ridge with a=0.1a = 0.10.171±0.0170.171 \pm 0.01747.4±3.8\mathbf{47.4 \pm 3.8}
Linear, ridge with a=1a = 10.160±0.0200.160 \pm 0.02047.0±3.847.0 \pm 3.8
Orthogonal (ours)0.154±0.020\mathbf{0.154 \pm 0.020}47.4±3.7\mathbf{47.4 \pm 3.7}

The unrestricted maps perform worse, although stronger regularization improves their performance. The initialization doesn’t find perfect pseudo-pairs, so some of the pseudo-pairs are wrong initially and in each refinement step. An unrestricted map can fit these incorrect correspondences, whereas the orthogonality constraint limits this overfitting and preserves the geometry. With real pairs, that problem goes away, and the linear map reaches similar performance but only at 1,0001,000 pairs (cf. Fig. 5). We added the comparison in the new Appx. A.2.3 and mention it in Sec. 3.

AuthorsOpenReview ↗

Response to Reviewer cfWK (2/2): Q2

Q2: “Could a barycentric projection derived from the transport plan improve alignment?”

Indeed, our read-out is derived directly from the transport plan via a barycentric projection. Given a transport plan T\mathbf{T}, the barycentric projection maps X\mathbf{X} to the weighted mean of its targets, Yˉ∝TY\bar{\mathbf{Y}} \propto \mathbf{T}\mathbf{Y}. Therefore, the orthogonal Procrustes map fitted to the barycentric projection is exactly polar⁡(X⊤Yˉ)=polar⁡(X⊤TY)\operatorname{polar}(\mathbf{X}^\top \bar{\mathbf{Y}}) = \operatorname{polar}(\mathbf{X}^\top \mathbf{T} \mathbf{Y}).

In Appx. A.2.1, we compare a direct read-out using the averaged low-rank transport plan with the updated transport plan with one conditional gradient step on the Gromov-Wasserstein objective. The updated transport plan performs better on all three benchmarks, and we use it in our method. For clarification, we added one sentence to App. A.2.1 that makes the link between the direct read-out and the barycentric projection explicit.


We appreciate the suggestion to evaluate alignment beyond FOSCTTM. The additional retrieval results and comparisons with unrestricted linear maps help clarify what our method achieves and where its alignment remains coarse.

Dear authors, thank you for the clarifications and additional experiments, which make the contribution clearer. My main concerns have been addressed, and my assessment remains positive. I will maintain my score.

Thank you for your response and positive assessment. It’s great to hear that the additional experiments and clarifications addressed your main concerns.

Reviewer_kiqEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3(+2)
Soundness
2
Read review
Paper summary

This submission works on the alignment of features of multimodal representations, by proposing to use the Wasserstein Procrustes method to align feature space without using pairs. The proposed approach was applied to align across datasets, modalities and unimodal models. It shows that the proposed method outperforms existing methods, and competitive with pair-based methods. The proposed alignment method is applied to text-to-image generation without paired examples.

Critical strengths
  1. The investigation of alignment of features across different models/modalities is an interesting research topic. This paper utilizes Wasserstein Procrustes method to align the features, even without any given paired features.

  2. The proposed alignment method was demonstrated to outperform the compared alignment methods, and the alignment is further applied to text-to-image generation tasks.

Critical weaknesses

I have several concerns on the claims of novelty, and the comparisons with related works.

  1. The alignment of feature space across models/modalities is not a novel task, as also mentioned in the paper. This is not the first work to align feature distributions across modalities/models. But in the abstract, there might be misunderstanding that this is the first work to do this task and the finding (paired examples are unnecessary) is totally new.

  2. The method takes the question "Can two independently trained modalities be aligned from their embedding geometry alone, without observing a single corresponding pair?" as the key motivation in introduction. The optimal transport can be naturally applied to this task without using pairs, and there are previous OT-based methods for this task. The method should well position the contribution of this work in the context of related works.

  3. To the key methodology for alignment, the paper utilized the Wasserstein Procrustes alignment, relying on an orthogonal map and a transport for alignment. First, the Wasserstein Procrustes alignment method is not a novel method, the paper should clearly present the novel design in this alignments approach. Second, the formulations in Eq. (1) is jointly optimized for W and T. Is a global W sufficient to align the feature distributions? Moreover, why not introduce entropy regularization in the Eq. (1) to achieve more robust probabilistic alignment?

  4. More OT-based baseline methods should be compared, e.g., the Gromov-Wasserstein Optimal Transport, though the initialization of the proposed approach is related to GW-OT.

  5. The coarse geometric initialization relies on the cluster initialization of mini-vec2vec, which is a compared method. It seems that this cluster alignment is essential to achieve good alignment by the proposed Wasserstein Procrustes alignment. How about the computational overhead of these additional operations? Moreover, since the proposed approach relies on the compared method mini-vec2vec, whether the comparisons are fair?

Questions for authors

As presented above, I appreciate the investigation of the shared geometry of feature space across modalities. However, I have concerns on the positioning of novelty of this approach considering the previous methods for this task, the methodology novelty and its reliance on a compared method for model initialization.

5 comments
AuthorsOpenReview ↗

Response to Reviewer kiqE (1/3): W1, W2, E3.1, W4

We are happy to read that the reviewer recognizes unpaired alignment across modalities as an “interesting research topic”. Furthermore, we thank them for pointing out where we could clarify our connection to existing unpaired alignment methods. To make the connection more explicit and improve the accessibility of our presentation, we clarify how optimization-based unpaired alignment relates to optimal transport (OT). Although this connection is established in the literature, its relevance to our formulation may not be immediately apparent to readers outside this specific line of work. Therefore, we have made this connection more explicit and added several OT-based comparisons. We address each concern below.

W1+W3.1: "This is not the first work to align feature distributions across modalities/models." and "the Wasserstein Procrustes alignment method is not a novel method"

Neither unpaired alignment nor Wasserstein-Procrustes is new, and we do not claim either as a contribution of this work. As discussed in the related work section, unpaired alignment has been studied extensively within individual modalities, for example, for translating word embeddings between languages, aligning different language models, and mapping biological data.

Our main contribution is to show that the geometry of independently trained representations across different modalities is sufficiently similar to be aligned without paired data. Prior to our work, the mere possibility of successful unpaired alignment across modalities was a major open question, because models across modalities are usually trained on different data, with completely different losses and architectures.

While combining Wasserstein Procrustes with Gromov-Wasserstein (GW) is a common approach, our ablations show that standard GW initializations do not work well in our cross-modal setting (see Appx. A.2.1). We therefore initialize Wasserstein Procrustes by solving multiple low-rank GW problems with a strong QAP solver, following the cluster-matching strategy of mini-vec2vec. Perhaps the most convincing argument is the following: no other existing method accurately aligns across modalities, whereas our method succeeds, as shown by the significantly superior results (see the table in W2+W4).

We have added this clarification to Sec. 1 of the revised manuscript

W2+W4: “The method should well position the contribution of this work in the context of related works.” and “More OT-based baseline methods should be compared”

To make our work more self-contained, we present the connection between OT and unpaired alignment more clearly. OT is the basis of many unpaired alignment methods. Optimization-based methods mostly build on Wasserstein Procrustes, Gromov-Wasserstein (GW), or both, while adversarial methods are not directly based on OT. In our work, we compare with the most recent methods in each group, vec2vec (adversarial) and mini-vec2vec (optimization-based). As a clarification, we further emphasize in Sec. 2 that optimization-based methods typically build on OT.

As suggested, we added four OT-based baselines: GW with cosine distances, SCOT+ (Baker et al., 2026), augmented GW (AGW; Baker et al., 2026), and a Wasserstein Procrustes method (InvarOT; Alvarez-Melis et al., 2019). We evaluate all approaches on the unpaired MS COCO setting, using all 2121 vision-language model combinations and five seeds. All methods use the same 10k10\mathrm{k} samples per side. GW hyperparameters are tuned using their respective objectives on 1k1\mathrm{k} samples.

Table R1. Comparison of various OT methods on unpaired MS COCO.

MethodFOSCTTM ↓\downarrow (0.50.5 is chance)CIFAR-10 top-1 ↑\uparrow
vec2vec0.477±0.0200.477 \pm 0.02010.4±1.010.4 \pm 1.0
GW (cosine distances)0.437±0.0200.437 \pm 0.02013.6±1.513.6 \pm 1.5
InvarOT0.351±0.0240.351 \pm 0.02417.2±2.817.2 \pm 2.8
SCOT+0.319±0.0290.319 \pm 0.02928.6±4.528.6 \pm 4.5
AGW0.316±0.0280.316 \pm 0.02828.4±3.328.4 \pm 3.3
mini-vec2vec0.223±0.0120.223 \pm 0.01229.9±0.929.9 \pm 0.9
ours0.154±0.020\mathbf{0.154 \pm 0.020}47.4±3.7\mathbf{47.4 \pm 3.7}

Plain GW performs close to random. SCOT+ improves on GW by using k-NN graph distances, while AGW adds a co-optimal transport term. InvarOT performs better than plain GW but remains below SCOT+ and AGW. All four newly tested methods perform worse than both mini-vec2vec and our approach. We include these new results as a table in the new Appx. C.5, and Sec. 4.1 refers to it.

AuthorsOpenReview ↗

Response to Reviewer kiqE (2/3): W3.2, W3.3

W.3.2: “Is a global W sufficient to align the feature distributions?”

Our experiments show that, perhaps remarkably, a single global orthogonal map is sufficient to recover coarse semantic alignment across the model pairs evaluated. We also evaluate unrestricted linear maps in response to cfWK’s suggestion (see the response to cfWK, Q1). Despite their greater flexibility, these maps perform worse, suggesting that the embedding spaces are better related by an approximate isometry and that the orthogonality constraint provides useful regularization in the unpaired setting.

W.3.3: “Why not introduce entropy regularization in the Eq. (1) to achieve more robust probabilistic alignment”

As suggested, we tested the suggested entropy regularization: it does not improve alignment. Eq. 1 can be extended with entropic regularization, and the resulting entropic linear assignment problem can be classically solved with the Sinkhorn algorithm instead of Hungarian matching in the read-out and refinement. We vary the regularization and batch size on NQ, SNARE-seq, and MS COCO.

Table R2. Entropic versus exact assignment during read-out and refinement.

read-outrefinement
Solverε\varepsilonbbNQSNARE-seqMS COCONQSNARE-seqMS COCO
Sinkhorn0.10.110k10\mathrm{k}6.0×10−3±2.9×10−36.0 \times 10^{-3} \pm 2.9 \times 10^{-3}0.219±0.0330.219 \pm 0.0330.082±0.0190.082 \pm 0.0192.7×10−5±2.7×10−62.7 \times 10^{-5} \pm 2.7 \times 10^{-6}0.209±0.035\mathbf{0.209 \pm 0.035}0.039±0.0190.039 \pm 0.019
Sinkhorn0.10.120k20\mathrm{k}5.2×10−3±2.7×10−35.2 \times 10^{-3} \pm 2.7 \times 10^{-3}0.219±0.0330.219 \pm 0.0330.082±0.0190.082 \pm 0.0192.5×10−5±2.2×10−62.5 \times 10^{-5} \pm 2.2 \times 10^{-6}0.209±0.035\mathbf{0.209 \pm 0.035}0.038±0.0180.038 \pm 0.018
Sinkhorn0.10.140k40\mathrm{k}4.8×10−3±2.7×10−34.8 \times 10^{-3} \pm 2.7 \times 10^{-3}0.219±0.0330.219 \pm 0.0330.081±0.0180.081 \pm 0.0182.4×10−5±3.6×10−62.4 \times 10^{-5} \pm 3.6 \times 10^{-6}0.209±0.035\mathbf{0.209 \pm 0.035}0.037±0.0180.037 \pm 0.018
Sinkhorn0.030.0310k10\mathrm{k}2.0×10−3±1.2×10−32.0 \times 10^{-3} \pm 1.2 \times 10^{-3}0.216±0.032\mathbf{0.216 \pm 0.032}0.081±0.0190.081 \pm 0.0191.8×10−6±2.5×10−71.8 \times 10^{-6} \pm 2.5 \times 10^{-7}0.209±0.035\mathbf{0.209 \pm 0.035}0.036±0.0420.036 \pm 0.042
Sinkhorn0.030.0320k20\mathrm{k}1.7×10−3±1.2×10−31.7 \times 10^{-3} \pm 1.2 \times 10^{-3}0.216±0.032\mathbf{0.216 \pm 0.032}0.080±0.0190.080 \pm 0.0191.4×10−6±2.8×10−71.4 \times 10^{-6} \pm 2.8 \times 10^{-7}0.209±0.035\mathbf{0.209 \pm 0.035}0.024±0.019\mathbf{0.024 \pm 0.019}
Sinkhorn0.030.0340k40\mathrm{k}1.5×10−3±1.1×10−31.5 \times 10^{-3} \pm 1.1 \times 10^{-3}0.216±0.032\mathbf{0.216 \pm 0.032}0.079±0.0180.079 \pm 0.0181.2×10−6±1.6×10−71.2 \times 10^{-6} \pm 1.6 \times 10^{-7}0.209±0.035\mathbf{0.209 \pm 0.035}0.033±0.0390.033 \pm 0.039
Sinkhorn0.010.0110k10\mathrm{k}5.5×10−4±4.0×10−4\mathbf{5.5 \times 10^{-4} \pm 4.0 \times 10^{-4}}0.216±0.033\mathbf{0.216 \pm 0.033}0.078±0.0190.078 \pm 0.0192.7×10−6±4.5×10−72.7 \times 10^{-6} \pm 4.5 \times 10^{-7}0.210±0.0330.210 \pm 0.0330.032±0.0410.032 \pm 0.041
Sinkhorn0.010.0120k20\mathrm{k}4.7×10−4±3.8×10−4\mathbf{4.7 \times 10^{-4} \pm 3.8 \times 10^{-4}}0.216±0.033\mathbf{0.216 \pm 0.033}0.078±0.0190.078 \pm 0.0191.3×10−6±2.3×10−71.3 \times 10^{-6} \pm 2.3 \times 10^{-7}0.210±0.0330.210 \pm 0.0330.030±0.0380.030 \pm 0.038
Sinkhorn0.010.0140k40\mathrm{k}4.2×10−4±3.6×10−4\mathbf{4.2 \times 10^{-4} \pm 3.6 \times 10^{-4}}0.216±0.033\mathbf{0.216 \pm 0.033}0.077±0.0190.077 \pm 0.0199.7×10−7±6.1×10−8\mathbf{9.7 \times 10^{-7} \pm 6.1 \times 10^{-8}}0.210±0.0330.210 \pm 0.0330.029±0.0380.029 \pm 0.038
Sinkhorn0.0030.00310k10\mathrm{k}1.7×10−4±1.3×10−4\mathbf{1.7 \times 10^{-4} \pm 1.3 \times 10^{-4}}0.216±0.034\mathbf{0.216 \pm 0.034}0.076±0.0210.076 \pm 0.0214.6×10−6±9.4×10−74.6 \times 10^{-6} \pm 9.4 \times 10^{-7}0.210±0.0330.210 \pm 0.0330.031±0.0410.031 \pm 0.041
Sinkhorn0.0030.00320k20\mathrm{k}1.4×10−4±1.0×10−4\mathbf{1.4 \times 10^{-4} \pm 1.0 \times 10^{-4}}0.216±0.034\mathbf{0.216 \pm 0.034}0.076±0.0210.076 \pm 0.0211.5×10−6±1.9×10−71.5 \times 10^{-6} \pm 1.9 \times 10^{-7}0.210±0.0330.210 \pm 0.0330.031±0.0410.031 \pm 0.041
Sinkhorn0.0030.00340k40\mathrm{k}1.2×10−4±1.0×10−4\mathbf{1.2 \times 10^{-4} \pm 1.0 \times 10^{-4}}0.216±0.034\mathbf{0.216 \pm 0.034}0.075±0.021\mathbf{0.075 \pm 0.021}9.4×10−7±9.7×10−8\mathbf{9.4 \times 10^{-7} \pm 9.7 \times 10^{-8}}0.210±0.0330.210 \pm 0.0330.034±0.0420.034 \pm 0.042
Hungarian-10k10\mathrm{k}2.8×10−4±1.8×10−4\mathbf{2.8 \times 10^{-4} \pm 1.8 \times 10^{-4}}0.216±0.034\mathbf{0.216 \pm 0.034}0.076±0.0230.076 \pm 0.0237.7×10−6±1.4×10−67.7 \times 10^{-6} \pm 1.4 \times 10^{-6}0.209±0.034\mathbf{0.209 \pm 0.034}0.027±0.0290.027 \pm 0.029

We observe that sufficiently small regularization approaches the performance of Hungarian matching. However, it does not yield a meaningful, consistent improvement beyond the standard deviation across seeds. Increasing the batch size also changes the results only marginally.

Thus, entropy regularization introduces an additional undesired hyperparameter without providing a significant performance benefit in our experiments. We report the complete results and discuss this trade-off in Appx. A.2.1.

AuthorsOpenReview ↗

Response to Reviewer kiqE (3/3): W5

W5.1: “How about the computational overhead of these additional operations?”

Our initialization is cheap compared to the rest of the method. On MS COCO, our initialization takes approximately 99 minutes for all sequential restarts together on a CPU, compared with approximately 3737 minutes for a full run. The 3030 initialization restarts are independent and can be parallelized. Appx. A.1 now reports these additional timings.

W5.2: “[Is] the comparisons [to mini-vec2vec] fair?”

While our method indeed builds on the cluster initialization of mini-vec2vec, we change the QAP solver, read-out, and refinement. This change is fundamental, as without it alignment performance drops significantly. Our manuscript has already compared the components individually in Appx. A.2.1. In the ablations on MS COCO,

  • the QAP solver improves FOSCTTM from 0.0920.092 to 0.0760.076,
  • the read-out improves FOSCTTM from 0.0850.085 to 0.0760.076, and
  • the refinement improves FOSCTTM from 0.0460.046 to 0.0200.020.

These ablations show that the improvements are not simply inherited from mini-vec2vec’s initialization. Each of our modified components contributes to the final performance. On our full MS COCO benchmark, mini-vec2vec reaches an FOSCTTM of 0.2230.223, whereas our method reaches 0.1540.154.

Our method also replaces several heuristic steps and hyperparameters with a more direct optimization procedure. We show that the cluster initialization solves a low-rank GW problem, providing a principled interpretation within the Wasserstein Procrustes framework. It uses four robust and untuned hyperparameters that remain fixed across our experiments. This is particularly useful in the unpaired setting, where reliable hyperparameter selection is difficult.


We thank the reviewer for helping to improve our manuscript, which has led us to better clarify the contribution of our work: not a new OT formulation, but an effective approach to cross-modal alignment that makes Wasserstein Procrustes work in this setting by identifying a suitable low-rank GW initialization, inspired by mini-vec2vec’s cluster matching with a better QAP solver.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal and Additional Findings

We thank all three reviewers for their thoughtful feedback. We are delighted to read that the unpaired alignment across modalities is an “interesting research topic” (kiqE), that our work addresses an “interesting and important problem” (WAni), that our empirical evaluation is “comprehensive” (WAni), and that our “empirical results strongly support the claims” (cfWK). The comments helped us clarify the contribution and investigate additional aspects of our approach, leading to several new findings. Below, we summarize the discussion of the main points raised by the reviewers, along with the main findings from new experiments suggested by the reviewers. All changes are marked in blue in the revised manuscript.

  1. Our contributions and novelty (kiqE): As mentioned by the reviewer, neither unpaired alignment nor Wasserstein Procrustes are indeed new on their own and they have both been studied extensively within individual modalities. We do not claim otherwise: our paper acknowledges this prior work, and we have further clarified the distinction in the revision. Our main contribution is to show that it is possible to align, without any example pairs, independently trained representations from different modalities. To our knowledge, this had been conjectured but had not previously been demonstrated in practice. Making Wasserstein Procrustes effective in this setting requires more than applying it off the shelf. We formulate mini-vec2vec’s coarse initialization as low-rank Gromov-Wasserstein problems and solve them with a stronger QAP solver. This yields a better initialization than existing Wasserstein Procrustes initializations, and each of our components (initialization, read-out, and refinement) outperforms its mini-vec2vec counterpart while being simpler. As requested by kiqE, we compare against four additional OT-based methods. On our unpaired MS COCO benchmark, all four newly tested methods perform worse than ours (FOSCTTM: 0.3160.316–0.4370.437 OT baselines vs. 0.1540.154 ours), see new Appx. C.5.
  2. Recall and unrestricted linear mapping (cfWK): As requested, we added Recall@1 and Recall@5 on the MS COCO Karpathy test split. Our method is much better than the random performance 0.02%0.02\%, but far below the 56.3%56.3\% of CLIP ViT-L/14. This is consistent with our results throughout the paper, which show that our method recovers coarse cross-modal structure but does not reliably capture fine-grained sample-level correspondences. We also compare orthogonal and unrestricted linear maps and clarify the barycentric projection used in our read-out (Appxs. A.2.1, A.2.3, C.6).
  3. Conditions for successful alignment (WAni): As requested, we further investigate failure cases under exact geometric symmetries and substantial semantic-distribution mismatch. On MS COCO, the coarse structures in the embedding spaces tend to lack symmetry, and near-symmetries mostly swap similar classes, so the coarse alignment stays correct. Missing classes and different class frequencies can degrade the alignment performance. We explicitly state this limitation in our discussion (Sec. 5, Appxs. C.7, F).

We have address every single concern raised by reviewers and incorporated all changes, including clarifications and additional experiments, into the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.