- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies whether independently trained representations from different modalities can be aligned without any paired cross-modal examples. The proposed method builds on Wasserstein Procrustes alignment and introduces a coarse geometric initialization: the two embedding spaces are independently clustered, cluster centers are matched by maximizing their geometric similarity, and the resulting coarse correspondence is converted into sample-level pseudo-pairs before Procrustes refinement. The same framework can also incorporate a small number of paired examples when available. The authors evaluate the method across multiple vision-language encoders and datasets, including a cross-dataset setting with no hidden sample-level correspondence, as well as several scientific and biomedical modality pairs. The results show that shared geometry can recover non-trivial coarse cross-modal alignment, substantially improve alignment in the low-pair regime, and support downstream tasks including zero-shot classification, retrieval, and text-to-image generation. Additional analyses suggest that the shared structure is primarily coarse rather than fine-grained.
Critical strengths
S1. Interesting idea. The paper studies an interesting and important problem: how far cross-modal alignment can be achieved from representation geometry alone, without paired data.
S2. Comprehensive experiments. The empirical evaluation is comprehensive, covering many model combinations, datasets, and modalities, including useful cross-dataset and downstream evaluations.
Critical weaknesses
W1. The paper shows that shared geometry enables coarse alignment, but it is less clear when geometry is sufficient to uniquely identify the semantically correct correspondence. In symmetric or near-symmetric cases, multiple permutations may achieve similar geometric objectives. Is the success mainly due to real representation geometries being sufficiently asymmetric, or is the method robust to such ambiguity? A controlled analysis of this issue would be helpful.
Questions for authors
Q1. In cases where the geometry admits multiple equivalent or near-equivalent permutations, how does the proposed method select the semantically correct alignment? Are such ambiguities rare in real representation spaces, or is the method robust to them? Some analysis or controlled experiments on this issue would help clarify the limits of the approach.
Q2. The method appears to rely on the two unpaired sets having reasonably similar semantic coverage and distributions. How sensitive is the method to mismatched semantic distributions, for example when some semantic groups are missing on one side or their frequencies differ substantially?
5 comments
Response to Reviewer WAni (2/3): W1+Q1 (hierarchical, MS COCO)
-
Hierarchical symmetry: We create six coarse groups that are not symmetric, each containing five symmetric subclusters.
Table R7. FOSCTTM on the hierarchical toy example.
symmetry Across the tested settings, FOSCTTM remains below , and – of incorrect cluster matches remain within the correct coarse group. This shows how local symmetries can preserve useful coarse alignment despite ambiguity at finer levels.
-
MS COCO: On MS COCO, we cannot change the amount of symmetry, but we know the class of every image. So we ran the matching step of our method on the average embeddings of each class, for all vision-language model pairs.
Table R8. Matching quality on MS COCO using class labels and joint clustering.
Classes Matched correctly Runs where a wrong matching scores at least as well as the true one superclasses categories (subsets of ) Joint clustering into clusters At the coarse scale, the geometry identifies the correspondence. For of the model pairs, all superclasses are matched correctly, and on average, are. Random subsets of of the categories are matched with an average accuracy of , and joint clusters with . At this finer scale, however, a wrong matching scores at least as well as the true one in (categories) and (joint clusters) of the runs. These wrong matchings usually swap similar classes, e.g., a baseball bat and a glove, a pizza and a sandwich, or a motorcycle and a bicycle, and swapping a category with its closest neighbor changes the objective by only about . Altogether, this is similar to the hierarchical toy example above, where symmetries exist mainly between nearby groups and the coarse structure is largely not symmetric.
More details about both experiments are in the new Appx. F. We also include it in the discussion in Sec. 5.
Response to Reviewer WAni (3/3): Q2
Q2: “How sensitive is the method to mismatched semantic distributions?”
Our method is sensitive to both unequal class frequencies and missing semantic groups. This assumption is used during initialization, where we search for 1-to-1 matchings in the QAP, and during readout and refinement, where we use a 1-to-1 Hungarian matching. This can also be observed empirically on MS COCO using its superclasses. We subsample 10k samples per side and vary the caption-side class frequencies or remove selected classes. We measure distribution mismatch using total variation (TV).
Different frequencies: If is the empirical frequency of the classes, we can change the frequency on the text side by sampling the classes proportionally to . When is smaller than , the groups are closer to uniformly distributed, and makes the large groups larger.
Table R9. FOSCTTM for different semantic frequencies.
| Caption shares | TV | FOSCTTM, mean of pairs |
|---|---|---|
| , baseline | ||
| , more even | ||
| , large groups larger | ||
| , close to uniform |
Missing classes: Here, we remove one or more classes from one side.
Table R10. FOSCTTM for missing classes.
| Missing on one side | TV | FOSCTTM, mean of pairs |
|---|---|---|
| Food captions | ||
| Animal captions | ||
| Animal images | ||
| Vehicle captions | ||
| Person captions (the largest group) | ||
| Animal, food, vehicle and sports captions |
Both experiments show that mismatched semantic distributions can degrade alignment. Small changes or the removal of a small group can still yield meaningful alignment, whereas substantial mismatches can bring performance close to the random baseline. When groups are missing or frequencies differ on one side, 1-to-1 matching assigns samples from one group to the other, and orthogonal Procrustes then fits those mismatched pairs.
We added these experiments in Appx. C.7, and we discuss the semantic distribution together with data quality in Sec. 5. Maybe unbalanced transport could help to tackle this in the future.
We thank the reviewer for raising these interesting questions. Our experiments reveal that real embeddings have surprisingly few coarse-level symmetries, which further explains why our method succeeds at coarse alignment. They also motivate future work on unbalanced mapping to address mismatched semantic distributions.
Response to Reviewer WAni (1/3): W1+Q1 (synthetic)
We are glad that the reviewer finds our problem “interesting and important” and recognizes our “comprehensive experiments”. We thank the reviewer for raising two important questions about the limitations of our approach: whether symmetries in the embedding geometry can lead to ambiguous cluster correspondences, and whether differences in semantic class frequencies between modalities can impair alignment. Interestingly, real representations are not symmetric on coarse scales, and symmetries mostly occur between similar classes. However, our method is sensitive to substantial distribution mismatches, and we have added an explicit discussion of this in our paper.
W1+Q1: “Is the success mainly due to real representation geometries being sufficiently asymmetric, or is the method robust to such ambiguity?”
Real embedding spaces are typically not symmetric at the coarse level. Fine-level near-symmetries are common, but they mainly swap semantically similar classes like a baseball bat and a glove and therefore preserve the coarse alignment. Our method has no rule for breaking ties. It averages the cluster matchings across 30 restarts, so under exact symmetry, no method that uses only geometry can pick the right matching. We perform three synthetic and one MS COCO experiment to investigate symmetries.
Global symmetry: We study how symmetry affects cluster matching by interpolating between a perfect symmetric space of 30 cluster centers (t=0, a regular simplex) and random normalized centers (t=1). The parameter t therefore controls the degree of symmetry. We independently add Gaussian noise to the centers in each space, where std controls the noise magnitude. Thus, std=0 means identical centers in both spaces, while larger values make the spaces increasingly different. We then independently rotate both spaces and evaluate alignment using FOSCTTM (0.5 corresponds to chance), averaged over five seeds. Further details are provided in Appendix F.
Table R5. FOSCTTM on the simplex toy example.
When the centers are perfectly symmetric, the method performs randomly. Introducing a shared symmetric component that is not symmetric allows the solver to recover the correct matching, even when that component is relatively small. As independent noise increases, less symmetry is required.
Mirror symmetry: Here, we consider a less symmetric setting in which clusters are mirrored, so that each cluster has only one symmetric counterpart.
Table R6. FOSCTTM on the mirror toy example.
At exact symmetry, runs can recover either the correct map or its mirror: for t=0, 8 of 15 runs select the mirror solution. This directly demonstrates that the method does not reliably resolve a geometrically equivalent but semantically incorrect correspondence.