acceptodds
Under review as a conference paper at ICLR 2027

Representation Alignment from a Few Paired Data via Distilled Optimal Transport

Abstract

Aligning two independently pretrained unimodal encoders typically demands millions of paired examples. In this paper, we study the regime where only – pairs are available, while unpaired data on either side is abundant. Our method expresses every representation by its similarities to the paired anchors, which places both modalities in a common coordinate system; matches the two point clouds with a single entropic optimal transport problem solved over the entire pool, whose marginal constraints counteract the hubness that defeats nearest-neighbor matching; and distills the resulting plan into two lightweight encoders, so that the correspondence becomes inductive. We additionally put this marginal constraint in relation with the one-step hub corrections of the bilingual lexicon induction literature. Pairing is required only for the anchors: the unpaired collections may be drawn independently from the two marginals, so that no item outside the anchor set needs to have both of its modalities observed. We evaluate on image-text and molecule-text alignment, where our method outperforms existing approaches at the same anchor budget on Flickr30k, MS-COCO and ChEBI-20. Code is available at https://anonymous.4open.science/r/AnchorDOT-B6CF/.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Learning, Unpaired data, Relative representations, Optimal Transport, Hubness reduction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xUiPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes AnchorDOT, a weakly supervised cross-modal representation alignment method from unimodal encoders. The key idea is to first leverage aligned cross-modal pairs as anchors to represent unlabelled samples with anchor-relative representations, and then use OT to align unlabelled cross-modal samples by using the relative representation distances as the transport cost. Using OT-based pseudo-labels across modalities, AnchorDOT trains two projection heads that project representations from two unimodal encoders into the same embedding space via a CLIP-style soft-label InfoNCE loss. The proposed AnchorDOT achieves promising results across image-text and molecule-text alignment tasks.

Critical strengths
  • The paper aim to address cross-modal alignment with unimodal encoders and scarce supervision pairs, which is practical and important problem where alignment across different modalities or domains are hard to obtain.
  • The overall architecture of the proposed AnchorDOT is intuitive and easy to understand. AnchorDOT consist of three main steps: anchor-relative representation with orthonormalized similarity, a global OT for pseudo labeling with block-wise computation, and two inductive encoders following a CLIP-style contrastive learning process, which are all clearly explained and well-motivated.
  • The empirical evaluation is well-structured and thorough. AnchorDOT is tested on two cross-modality alignment tasks: image-text and molecule-text, and shows consistent outperformance over the SOTA baselines. The authors also evaluate AnchorDOT comprehensive across various dimensions, including matching quality, hubness, and efficiency. Additional study regarding different design choices of AnchorDOT is also comprehensive, such as orthonormalization, alternative OT costs, and hyperparameter sensitivity.
Critical weaknesses

My biggest concern is the novelty of the proposed method. AnchorDOT essentially leverage OT for pseudo-labeling unlabeled cross-modality pairs and then following a CLIP-style contrastive training of two linear projection heads that project unimodal representation into a shared embedding space. Similar ideas of OT-based pseudo-labeling combined with CLIP-style contrastive cross-modal alignment has been explore in OTTER [1] and SAT [2], making the overall novelty of AnchorDOT incremental. The additional heuristics, such as anchor-relative representation with orthonormalized similarity, while simple and intuitive, has been explored extensively in other related works of vision-language learning, such as RELIT[3] and CCA.

[1] Wu, Bichen, et al. "Data efficient language-supervised zero-shot recognition with optimal transport distillation." arXiv preprint arXiv:2112.09445 (2021).

[2] Mishra, Shambhavi, et al. "Semantic Anchor Transport: Robust Test-Time Adaptation for Vision-Language Models." arXiv preprint arXiv:2411.17002 (2024).

[3] Chen, Chi, et al. "Weakly supervised vision-and-language pre-training with relative representations." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Questions for authors
  • The definition of the training wall-clock time provided in Table 7 is unclear. Is it end-to-end runtime that included the OT computation, or only training time on the student model side?
  • How can AnchorDOT be extended to continual training setting where OT-based pseudo labels are continually updated using the newly arrived samples? How to enable efficient OT-based labling?
2 comments
AuthorsOpenReview ↗

Response to Reviewer xUiP (Part 1 of 2)

We thank the reviewer for the constructive feedback, and for highlighting the practical importance of the problem, the intuitive and well-motivated design of AnchorDOT, and the thorough evaluation. We first address the novelty concern, then the two questions. Figure and table numbers refer to the revised PDF.

Regarding the novelty concern

"My biggest concern is the novelty of the proposed method. AnchorDOT essentially leverage OT for pseudo-labeling unlabeled cross-modality pairs and then following a CLIP-style contrastive training of two linear projection heads that project unimodal representation into a shared embedding space. Similar ideas of OT-based pseudo-labeling combined with CLIP-style contrastive cross-modal alignment has been explore in OTTER [1] and SAT [2], making the overall novelty of AnchorDOT incremental. The additional heuristics, such as anchor-relative representation with orthonormalized similarity, while simple and intuitive, has been explored extensively in other related works of vision-language learning, such as RELIT[3] and CCA."

We agree that OT-based pseudo-labeling and relative representations have antecedents, which Section 2 discusses (OTTER, ASIF, RELIT). The setting differs, however: OTTER computes in-batch OT on noisy web-collected image–text pairs, with teacher encoders trained on paired data, and SAT assigns test images to class-text embeddings of CLIP, an already aligned model. In both, a reliable cross-modal similarity exists by construction. Between independently trained encoders, none exists and the anchors provide it.

This is where we identified two failure modes of anchor-relative representations (as used by ASIF and RELIT), each corrected by one component of AnchorDOT:

  1. Hubness. Nearest-neighbor matching on anchor-relative profiles, as performed by ASIF and RELIT, is dominated by hubs (Section 3.2). Despite their low cost, anchor-relative representations cannot be used as is, and we developed the global OT coupling to correct this. Hubness corrections exist in retrieval, but they are applied at inference time, in an already aligned space. In our setting no aligned space exists yet: the matching is performed at training time and becomes the supervision. In a new experiment (Figure 3), encoders distilled from nearest-neighbor targets reach 0.68 R@10 on Flickr30k, against 0.79 with our converged coupling, whose encoders are also two to four times less hub-dominated (N₁₀ skewness 0.39/0.55 against 1.67/1.11). No correction is needed at inference.
  2. Spectrum squaring. We show (Proposition 1) that the plain profile squares the spectrum of the embedding covariance, a distortion that, to our knowledge, had not been identified for anchor-relative representations, and we correct it by orthonormalizing the anchor matrix. A new ablation (Appendix C.3.4) confirms that this is what limits the plain profile: at fixed OT and distillation, the profile without orthonormalization does not outperform a ridge CCA fitted on the same anchors (0.750 vs. 0.757 R@10 on Flickr30k), while the orthonormalized profile does (0.795). Orthonormalization itself is a standard operation, and CCA goes further by whitening each modality. Proposition 1 places the three on one scale: the plain profile squares the spectrum (γ=0\gamma = 0), orthonormalization preserves its shape (γ=1\gamma = 1), and whitening flattens it (γ=2\gamma = 2), relying on directions poorly estimated from NN anchors. The γ\gamma selected on held-out data (Appendix D) lies between these two extremes, which is why our cost outperforms both.

We believe this identification of the failure modes of anchor-relative representations, and of a targeted correction for each, goes beyond a combination of existing parts.

AuthorsOpenReview ↗

Response to Reviewer xUiP (Part 2 of 2)

Q1. Wall-clock time (Table 8)

"The definition of the training wall-clock time provided in Table 7 is unclear. Is it end-to-end runtime that included the OT computation, or only training time on the student model side?"

The times in Table 9 (Table 7 in the submission) are end-to-end for both methods. For AnchorDOT, they include the relative representations, the OT coupling and the distillation. We have clarified the caption.

Q2. Extension to continual training

"How can AnchorDOT be extended to continual training setting where OT-based pseudo labels are continually updated using the newly arrived samples? How to enable efficient OT-based labling?"

We have not evaluated our method to continual training, but we believe AnchorDOT could be extended to it, because the coupling is entirely determined by the dual potentials (u,v)(u, v), which take O(n+m)\mathcal{O}(n+m) memory. We distinguish two cases, depending on whether the new samples are unpaired or paired.

Case 1: new unpaired samples. The anchors are unchanged, so the relative representation of a new sample only costs NN similarities, and the cost matrix is unchanged for existing samples (with ε\varepsilon kept fixed). The potential of a new sample can then be obtained in closed form by the entropic c-transform, unew=εlog⁡μnew−εlog⁡∑jexp⁡((vj−Cnew,j)/ε)u_{\text{new}} = \varepsilon \log \mu_{\text{new}} - \varepsilon \log \sum_{j} \exp\big((v_j - C_{\text{new},j})/\varepsilon\big), at O(m)\mathcal{O}(m) cost, without re-solving the problem. Since adding samples perturbs the marginals (1/n1/n becomes 1/(n+Δn)1/(n+\Delta n)), a few Sinkhorn iterations would be needed to restore them, warm-started from the existing potentials. Good initializations are known to accelerate Sinkhorn substantially (Thornton & Cuturi, 2023; Amos et al., 2023), and we expect the previous potentials, which should remain close to the new solution when Δn≪n\Delta n \ll n, to be a particularly good one. The targets of the new chunks would then follow in closed form from the updated potentials (Eq. 9), as in the current pipeline.

Case 2: new paired samples. New pairs extend the anchor set, which changes the profile and its orthonormalization, hence the cost for all samples: the coupling would have to be re-solved. We expect a warm start from the previous potentials to remain useful, since adding a few pairs to N=500N = 500 should change the cost only slightly, but this remains to be verified. The new pairs would also directly enter the anchor chunk as supervision.

Again, we thank the reviewer for the constructive feedback, and welcome any further discussion.

Reviewer_6ZwZOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method called AnchorDOT for the following situation: one needs to connect a pretrained image encoder and a pretrained text encoder, but the number of matched image-caption pairs, NN, is very small. In this paper, NN is only about 100∼500100\sim 500, whereas the number of unmatched images and captions is much larger. AnchorDOT has three steps. First, every image is described by how similar it is to each of the NN matched images (the anchors), and every caption by how similar it is to each of the NN matched captions. Because the k-th anchor image and the kk-th anchor caption show the same content, images and captions now share a common set of coordinates. Second, the authors pair the unmatched images with the unmatched captions in these coordinates using entropic optimal transport. This procedure assigns every image and every caption an equal share of matches, so that no single caption becomes the best match for many images. Third, the resulting soft pairings serve as training targets for two small encoders, one for each modality, so that new images and captions can be embedded without the anchors. With 500 anchor pairs, the method outperforms ASIF, CSA and SUE on Flickr30k, MS-COCO and ChEBI-20, and performs on par with SOTAlign while training much faster. Additional experiments show that the unmatched images and captions need not come from the same underlying pairs, and that optimal transport reduces hubness more than simpler corrections such as CSLS.

Critical strengths

The main strength of the paper is that it addresses a very practical problem. Matched image–text pairs are often scarce, while unmatched images and texts are plentiful, and this imbalance is common in real applications. As the authors point out, creating a large number of matched pairs is very expensive in many domains, such as medical imaging. The proposed method clearly outperforms most existing methods, is tied with SOTAlign, and trains several times faster.

The authors handle hubness carefully. They start from the same similarity scores and compare four ways of pairing items, that is, plain nearest neighbor, two common corrections (inverted softmax and CSLS), and their optimal transport method. Hubness is measured by N10N_{10} skewness, which shows how unevenly items appear in each other’s top-10 lists. Because the scores are identical across the four methods, any difference comes only from the pairing rule. This shows clearly what is gained by requiring every image and every caption to receive an equal share of matches. The detailed experiment also shows that the inverted softmax, a method designed to reduce hubness, actually increases it in one direction. This result is useful on its own.

The experiments are run carefully. The authors run every competing method again under the same conditions, so the comparison is fair. Each result is the average of five runs with different random seeds, and the standard deviations are reported. The settings of their own method are chosen using a small set of anchor pairs held back for this purpose, not using the test data (Appendix D). The overlap experiment is also well designed.

Critical weaknesses

The abstract states that the method outperforms existing approaches, but most of the differences from SOTAlign are smaller than one standard deviation over five seeds. On MS-COCO, SOTAlign is ahead at R@1. In addition, the test set has only 400 pairs, so a single query changes the score by 0.25 points, and it is not the standard Flickr30k 1K or MS-COCO 5K test set. What the results do support is that the method matches SOTAlign while training several times faster.

The cost used by the proposed method seems to be essentially a linear mapping learned from the anchor pairs. The authors criticize SOTAlign for relying on a linear alignment, but their own cost appears to rely on one as well. It would help to run the same pipeline with a standard linear mapping as the cost, to show where the gain actually comes from.

The unmatched images and captions are two halves of the same dataset, so they contain the same kinds of content in the same proportions. The method gives every item an equal share of matches, which fits this case well. With data collected separately, for example with more dog images than dog captions, this rule would force wrong matches. Since this setting is not tested, the claim should be limited to pools drawn from the same source.

Only one small image encoder and one small text encoder are used, and the zero-shot test has just 700 images from 7 classes. Results with larger encoders and a larger zero-shot test would show whether the conclusions hold more generally.

Questions for authors

I find the proposed method practical and reasonable, and I think the paper is of sufficient quality for acceptance. The following questions are relatively minor, and if they are addressed, I would be happy to raise my score.

The authors claim that the proposed method outperforms SOTAlign, but the two methods’ scores are quite similar. As noted in the weakness section, it would be useful if the authors could make this comparison more rigorous statistically, for example with a paired test across seeds or with results on the standard test sets. I think the stronger selling point of the proposed method is that it performs similarly to SOTAlign while training much faster.

CSA scores almost at chance (0.9 R@1 on Flickr30k), even though a simple contrastive model trained on the same 500 pairs reaches 22. CCA usually needs regularization when the number of pairs is close to the feature dimension, and if CSA was run without it, the comparison would not be fair. Why is CSA’s performance so poor, and was its CCA regularized?

Section 3.2 states that γ∈[0,2]\gamma \in [0,2], that γ=1\gamma = 1 preserves the shape of the spectrum, and that whitening corresponds to γ=2\gamma = 2. Appendix D restricts γ∈[0,1]\gamma \in [0, 1] and states that each modality is whitened independently. Could the authors clarify which γ\gamma range is actually searched and whether whitening is applied?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 6ZwZ (Part 1 of 4)

We thank the reviewer for the positive assessment and the precise questions, and for highlighting the practical relevance of the problem, the careful design of the hubness comparison and the fairness of the experimental protocol. We first answer the three questions, then the remaining weaknesses. Figure and table numbers refer to the revised PDF.

Q1. Comparison with SOTAlign (and first weakness)

"The abstract states that the method outperforms existing approaches, but most of the differences from SOTAlign are smaller than one standard deviation over five seeds. On MS-COCO, SOTAlign is ahead at R@1. In addition, the test set has only 400 pairs, so a single query changes the score by 0.25 points, and it is not the standard Flickr30k 1K or MS-COCO 5K test set."

"[...] it would be useful if the authors could make this comparison more rigorous statistically, for example with a paired test across seeds or with results on the standard test sets. I think the stronger selling point of the proposed method is that it performs similarly to SOTAlign while training much faster."

We agree that the two methods are tied at R@1 on image–text, and we have revised the abstract, Section 4 and the conclusion. As the reviewer suggests, the abstract now states that AnchorDOT "matches or exceeds all baselines while training 3 to 15× faster than the strongest one" (end-to-end times, including the OT computation, Table 9).

As suggested, we compared the two methods with paired tests (new Appendix C.10, Table 12). The std in Tables 5–7 measures variation across seeds, and each seed redraws the anchors, the unpaired pool and the test split. Within a seed, both methods share all three, so a paired comparison removes this shared variation. We kept the 400-pair test split of SUE's protocol (Section 4). To increase statistical power, we ran McNemar tests over the 2,000 query–seed pairs (5 seeds × 400 test queries), where a single query moves the recall by 0.05 points instead of 0.25. Differences are in recall points (AnchorDOT − SOTAlign), p-values in parentheses:

Flickr I2TFlickr T2ICOCO I2TCOCO T2IChEBI M2TChEBI T2M
R@1+0.95 (0.32)+0.85 (0.39)−1.45 (0.12)−0.60 (0.56)+2.30 (0.005)+2.25 (0.008)
R@5+2.90 (<10⁻³)+3.00 (<10⁻³)+0.95 (0.29)+1.25 (0.17)+2.65 (0.002)+3.60 (<10⁻⁴)
R@10+3.05 (<10⁻⁴)+3.85 (<10⁻⁶)+0.85 (0.20)+2.00 (0.004)+3.45 (<10⁻⁴)+2.45 (0.004)

Bold differences are significant and survive a Holm–Bonferroni correction within each dataset. On Flickr30k, the two methods are tied at R@1, and AnchorDOT is significantly better at R@5 and R@10. On ChEBI-20, it is significantly better in all six columns. On MS-COCO, the two methods are tied, except at T2I R@10. SOTAlign is never significantly better.

We also ran paired t-tests over the 5 seeds, which account for training variability but have less power. They lead to the same conclusions. On Flickr30k, AnchorDOT is significantly better at R@5 and R@10 in both directions (p ≤ 0.036) and tied at R@1 (p = 0.17 and 0.42). On MS-COCO, the only significant difference is at T2I R@10 (p = 0.022). On ChEBI-20, AnchorDOT is significantly better in four of the six columns (p ≤ 0.048), and the two remaining differences go in the same direction (p = 0.065 and 0.089).

We have also re-run the methods on the standard Flickr30k 1K test set (new Appendix C.8): the conclusions are unchanged.

AuthorsOpenReview ↗

Response to Reviewer 6ZwZ (Part 2 of 4)

Q2. CSA's performance

"CSA scores almost at chance (0.9 R@1 on Flickr30k), even though a simple contrastive model trained on the same 500 pairs reaches 22. CCA usually needs regularization when the number of pairs is close to the feature dimension, and if CSA was run without it, the comparison would not be fair. Why is CSA's performance so poor, and was its CCA regularized?"

Thank you for catching this. The reviewer's diagnosis is correct. CSA, as published, fits an unregularized CCA (Li et al., 2025, Eq. 3) and is validated with 5,000 to 35,000 pairs. With N = 500 pairs, the 384-dimensional image and text features together span more dimensions (768) than the centered pairs (499). Unregularized CCA then finds at least 269 directions that have a correlation of exactly 1 on the anchors but carry no signal on new data. We re-ran all experiments involving CSA with a ridge-regularized CCA, with its ridge selected by the same held-out procedure as our γ\gamma (Appendix D), and updated Tables 1, 2 and 5–7. For ease of reference, Recall@k (%), mean over 5 seeds:

DatasetCSAI2T/M2T R@1R@5R@10T2I/T2M R@1R@5R@10
Flickr30kunregularized0.903.155.600.953.556.10
regularized20.2045.3057.8515.8039.1551.75
MS-COCOunregularized1.305.559.401.355.708.95
regularized19.2053.9571.9015.5546.9065.60
ChEBI-20unregularized0.201.102.550.652.454.40
regularized12.9534.9049.2011.0033.8548.40

With regularization, CSA becomes comparable to the contrastive baseline trained on the anchors alone, and outperforms it on MS-COCO. It remains 10 to 20 points below AnchorDOT at R@1, and 7 to 19 points below SOTAlign. The conclusions of the paper are thus unchanged.

Q3. Range of γ\gamma

"Section 3.2 states that γ ∈ [0, 2], that γ = 1 preserves the shape of the spectrum, and that whitening corresponds to γ = 2. Appendix D restricts γ ∈ [0, 1] and states that each modality is whitened independently. Could the authors clarify which γ range is actually searched and whether whitening is applied?"

We apologize for this inconsistency, and thank the reviewer for catching it. The search is over γ∈[0,1]\gamma \in [0, 1], and no whitening is applied. Although γ\gamma is defined on [0, 2], for a small number of pairs such as N=500N = 500, values of γ\gamma above 1 amplify the smallest eigenvalues of MXM_X, which are poorly estimated from NN anchors, and add noise to the profile. This effect disappears as NN increases (participation ratio, Appendix C.1). We therefore restrict the hyperparameter search to γ∈[0,1]\gamma \in [0, 1] at N=500N = 500. The mention of whitening in Appendix D was incorrect, since γ≤1\gamma \leq 1 corresponds at most to the orthonormalization of the anchor matrix. We have corrected Appendix D accordingly.

AuthorsOpenReview ↗

Response to Reviewer 6ZwZ (Part 3 of 4)

Remaining weaknesses (1 of 2)

"The cost used by the proposed method seems to be essentially a linear mapping learned from the anchor pairs. The authors criticize SOTAlign for relying on a linear alignment, but their own cost appears to rely on one as well. It would help to run the same pipeline with a standard linear mapping as the cost, to show where the gain actually comes from."

Linearity of the cost. We agree that our cost is built linearly. SOTAlign shares this structure, as its KLOT term matches the student's OT plan to that of a CCA teacher. The two methods differ in the cost. In practice, CCA is ill-posed at our budget: with N=500N = 500 pairs and dx+dy=768>N−1d_x + d_y = 768 > N - 1, unregularized CCA finds at least 269 directions of correlation 1 that carry no signal (Q2). A ridge is therefore required, as in SOTAlign. Without it, CCA rescales each eigendirection of the anchor covariance to unit variance, which amplifies the directions of smallest variance, the ones NN anchors estimate least reliably (Appendix C.1). The ridge, which adds λ\lambda to each variance ss, only caps this amplification. In contrast, anchor coordinates need no whitening, since coordinate kk refers to the same pair in both modalities, and orthonormalization weights each direction by s1−γs^{1-\gamma} with γ≤1\gamma \le 1, which never favors directions of small variance (an isometry at γ=1\gamma = 1, Section 3.2). As suggested, we ran the same pipeline with a ridge CCA as the cost, its λ\lambda selected as for γ\gamma, keeping OT and distillation fixed (new Appendix C.3.4, Table 4, Flickr30k, mean ± std over 3 seeds):

CostR@1R@5R@10
Ridge CCA0.316 ± 0.0060.632 ± 0.0050.757 ± 0.005
Profile, no orthonormalization0.301 ± 0.0270.631 ± 0.0390.750 ± 0.029
Profile + orthonormalization (ours)0.346 ± 0.0140.679 ± 0.0210.795 ± 0.013

The orthonormalized profile improves on ridge CCA by 3.0 / 4.7 / 3.8 points at R@1 / R@5 / R@10, while with the ridge-CCA cost our pipeline only matches SOTAlign (0.757 vs. 0.759 R@10, Table 1). The gain over SOTAlign thus comes from the orthonormalized anchor-relative cost, and solving the coupling once, rather than at every training step, makes training 3–15× faster (Table 8). We have revised the presentation of SOTAlign accordingly in the introduction and related work, which now attribute the difference to the cost rather than to the linearity of SOTAlign’s alignment.

"The unmatched images and captions are two halves of the same dataset, so they contain the same kinds of content in the same proportions. The method gives every item an equal share of matches, which fits this case well. With data collected separately, for example with more dog images than dog captions, this rule would force wrong matches. Since this setting is not tested, the claim should be limited to pools drawn from the same source."

Pools from different sources. We agree that this setting was not tested in the submission. As suggested, we replaced a fraction p of the unpaired text pool by MS-COCO captions (Flickr30k images, anchors and test split, new experiment in Appendix C.5.2, Figure 14). AnchorDOT degrades gracefully (recall@10 0.795, 0.770 and 0.728 at p = 0, 0.5 and 1), at the same rate as SOTAlign (0.780, 0.757, 0.712), and stays well above the contrastive baseline (0.600): the unpaired data still helps, and the global marginal constraint does not make it more fragile. Since Flickr30k and MS-COCO captions are fairly close, we have qualified the claim in Section 1 and Section 5. We also agree that with strongly different proportions, such as more dog images than dog captions, uniform marginals would force some wrong matches. Unbalanced OT, which relaxes the marginals, is the natural extension (Section 5).

Reviewer_d4c3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to align two independently pretrained, frozen unimodal encoders using only small scale paired examples (N ≈ 100–500), together with abundant unpaired data from each modality. The proposed approach has three stages. First, every embedding is represented by its similarity profile to the N paired anchors, with a γ-controlled anchor orthonormalization, which places both modalities in a shared N-dimensional coordinate space. Second, a single entropic OT problem is solved over the entire unpaired pool in this space, and its marginal constraints are used to reduce the hubness that affects nearest-neighbor matching. Third, the resulting transport plan is distilled into two lightweight encoders through a soft-target contrastive loss, so inference requires neither the anchors nor OT. The main contribution is combining anchor-relative representations, global OT matching, and distillation to recover cross-modal correspondences from unpaired pools that need not contain any hidden pairing. Experiments cover image–text retrieval on Flickr30k and MS-COCO and molecule–text retrieval on ChEBI-20at N = 500. The paper also reports zero-shot ImageNet classification, ablations on anchor budget, pool size, and pool overlap, and a hubness comparison against nearest neighbor, inverted softmax, and CSLS.

Critical strengths
  • Clear writing and well-positioned problem. The paper is easy to follow, and the low-resource setting (N ≤ 500, disjoint unpaired pools) is practically relevant. The related work is thorough and states how the method differs from each prior approach.

  • Simple and efficient pipeline. The method requires no anchors or OT at inference. The OT is solved once, and chunks are recovered in closed form from the dual potentials. Training is reported to be 3–15× faster than SOTAlign.

  • Useful analysis of hubness and pool composition. The comparison of nearest neighbor, inverted softmax, CSLS, and full Sinkhorn on the same cost is informative. The overlap experiment (o ∈ [0, 1]) shows the method does not rely on hidden pairing in the pool.

Critical weaknesses
  • Limited novelty; the method is a combination of existing components. Each stage is taken from prior work: anchor-relative representations (Moschella et al., 2022; ASIF), Sinkhorn coupling as a hubness correction (Pan et al., 2025), and distilling an OT plan as soft contrastive targets (OTTER, S-CLIP). The γ-orthonormalization is a standard whitening/Löwdin interpolation, and Proposition 1 is a straightforward linear-algebra result. The relation to inverted softmax and CSLS is also largely known. The overall contribution is mainly an engineering combination, which feels incremental for ICLR.

  • Marginal gains over the strongest baseline. Against SOTAlign, R@1 is tied: ours is ahead by 0.9 on Flickr30k and behind by 0.6–1.4 on MS-COCO, and the authors acknowledge this. The R@5/R@10 gains (+0.9 to +3.9) fall within or close to the reported std (≈2–3 points). Zero-shot classification is also mixed. With a 400-pair test set, these differences are hard to interpret without significance tests.

  • Ablations do not isolate the contribution of OT after distillation. a) The hubness comparison (Fig. 4) is done only on the transductive coupling, before distillation. A control is needed where nearest-neighbor, CSLS, or inverted-softmax pseudo-labels are distilled with the same encoders and loss. This would show whether the full Sinkhorn coupling matters for the final encoders. b) There is no comparison with a simple alternative cost: a linear map (e.g., Procrustes/CCA) fitted on the N anchors, followed by OT in that space. This would show whether the anchor-relative representation is necessary.

Questions for authors

see the weakness sec.

5 comments
AuthorsOpenReview ↗

Response to Reviewer d4c3 (Part 1 of 2)

We thank the reviewer for the precise suggestions, and for highlighting the paper's clarity, the method's simplicity and efficiency, and the usefulness of the hubness and pool-composition analyses. As suggested, we ran both requested ablations. Figure and table numbers below refer to the revised PDF. We start with the ablations, since our answer on novelty builds on them.

1. Ablations (Weakness 3)

"Ablations do not isolate the contribution of OT after distillation."

"a) The hubness comparison (Fig. 4) is done only on the transductive coupling, before distillation. A control is needed where nearest-neighbor, CSLS, or inverted-softmax pseudo-labels are distilled with the same encoders and loss."

(a) We thank the reviewer for the suggestion. We have now added new distillation experiments of nearest-neighbor, CSLS and inverted-softmax targets with the same encoders, loss, anchor chunk and hyperparameters as ours (Flickr30k, I2T / T2I). Figure 3 is updated accordingly.

For ease of reference, we summarize the results below (I2T / T2I, mean over 5 seeds, best in bold).

TargetsN₁₀ skewness ↓R@10 ↑
Nearest neighbor1.67 / 1.110.68 / 0.67
CSLS1.03 / 0.720.76 / 0.75
Inverted softmax1.11 / 0.720.76 / 0.75
Converged OT (ours)0.39 / 0.550.79 / 0.79

We found that the encoders inherit the hubness of their targets, and those distilled from our coupling are the least hub-dominated and the most accurate in both directions. One-step corrections recover most of the recall gap to nearest neighbor, and enforcing both marginals to convergence adds a further 3–4 points.

"b) There is no comparison with a simple alternative cost: a linear map (e.g., Procrustes/CCA) fitted on the N anchors, followed by OT in that space. This would show whether the anchor-relative representation is necessary."

We agree. In the updated manuscript, we have now added a comparison with a ridge CCA fitted on the anchors, with OT and distillation unchanged. We use CCA because SUE, CSA and SOTAlign all rely on it, and SOTAlign found it a better linear teacher than Procrustes. Its regularization parameter is selected with the same held-out procedure as our γ\gamma (Appendix D). Results are in the new Appendix C.3.4 (Table 4, Flickr30k, mean ± std over 3 seeds).

CostR@1R@5R@10
Ridge CCA0.316 ± 0.0060.632 ± 0.0050.757 ± 0.005
Profile, no orthonormalization0.301 ± 0.0270.631 ± 0.0390.750 ± 0.029
Profile + orthonormalization (ours)0.346 ± 0.0140.679 ± 0.0210.795 ± 0.013

To answer the reviewer's question directly: the plain anchor-relative profile is comparable to CCA. However, the orthonormalized profile does outperform ridge CCA by 3.0 / 4.7 / 3.8 points at R@1 / R@5 / R@10, with half the std of the plain profile across seeds.

2. Novelty (Weakness 1)

"Limited novelty; the method is a combination of existing components. [...] The overall contribution is mainly an engineering combination, which feels incremental for ICLR."

Thank you for raising this point. Each building block indeed relies on existing methods, which we cite. What the paper adds is the identification of two key limitations of existing anchor-relative methods (ASIF, RELIT) at a few hundred anchors, and a targeted correction for each:

  1. Hubness. ASIF and RELIT match anchor-relative profiles by nearest neighbor, and point 1(a) shows that the resulting hubness carries over into trained encoders. Hubness corrections, including the Sinkhorn normalization of Pan et al. (2025), are applied at inference time, in an already aligned space. In our setting no aligned space exists yet: the matching is performed at training time and becomes the supervision. OTTER and S-CLIP also distill an OT plan, but their similarity is available by construction (teachers trained on pairs, or images compared in CLIP space). Between independently trained unimodal encoders no such similarity exists, and the anchor-relative coordinates supply it.
  2. Spectrum squaring. We show (Proposition 1) that the plain profile squares the spectrum of the embedding covariance, a distortion that, to our knowledge, had not been identified for anchor-relative representations, and we correct it by orthonormalizing the anchor matrix, which turns a cost no better than CCA into one that outperforms it (point 1(b)).

We believe this identification of the failure modes of anchor-relative representations, and of a targeted correction for each, is the core novelty of the work.

AuthorsOpenReview ↗

Response to Reviewer d4c3 (Part 2 of 2)

3. Gains over SOTAlign (Weakness 2)

"Against SOTAlign, R@1 is tied [...] The R@5/R@10 gains (+0.9 to +3.9) fall within or close to the reported std (≈2–3 points). Zero-shot classification is also mixed. With a 400-pair test set, these differences are hard to interpret without significance tests."

We agree the methods are tied at R@1 on image–text, and we revised the abstract, Section 4 and the conclusion accordingly. The abstract now states that our method "matches or exceeds all baselines while training 3 to 15× faster than the strongest one".

We additionally conducted statistical tests. The std in Tables 5–7 measures variation across seeds, and each seed redraws the anchors, the unpaired pool and the test split. Within a seed, both methods share all three, a paired comparison thus removes this shared variation. We compared SOTAlign and our method query by query with McNemar tests over the 2,000 query–seed pairs (5 seeds × 400 queries), reported in the new Appendix C.10 (Table 12). Differences are in recall points (AnchorDOT − SOTAlign), with p-values in parentheses:

Flickr I2TFlickr T2ICOCO I2TCOCO T2IChEBI M2TChEBI T2M
R@1+0.95 (.32)+0.85 (.39)−1.45 (.12)−0.60 (.56)+2.30 (.005)+2.25 (.008)
R@5+2.90 (<.001)+3.00 (<.001)+0.95 (.29)+1.25 (.17)+2.65 (.002)+3.60 (<.001)
R@10+3.05 (<.001)+3.85 (<.001)+0.85 (.20)+2.00 (.004)+3.45 (<.001)+2.45 (.004)

Bold differences are significant and survive a Holm–Bonferroni correction within each dataset. On Flickr30k, the two methods are tied at R@1, and AnchorDOT is significantly better at R@5 and R@10. On ChEBI-20, it is significantly better in all six columns. On MS-COCO, the two methods are tied, except at T2I R@10. Overall, SOTAlign is never significantly better. Paired t-tests over the 5 seeds, which account for training variability but have less power, lead to the same conclusions (Table 12).

Finally, AnchorDOT trains 3–15× faster than SOTAlign, end to end including the OT coupling (Table 9, Table 7 in the submission).

We thank the reviewer again for these suggestions, which strengthened the paper, and welcome any further questions.

Thanks the additional experiments. The new distillation ablation, the ridge-CCA cost comparison, and the paired significance tests address my experimental concerns.

However, my core concern about novelty remains, and this is also shared shared by Reviewer xUiP. In the rebuttal, the author reframes the contribution as identifying two failure modes, but the components themselves are known: hubness correction via OT/CSLS, anchor orthonormalization, and OT-based distillation. The new ablation experiment also shows that one-step corrections already recover most of the gap. Thus, I consider the contribution as a well-executed but incremental combination of existing components. I therefore maintain my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall response

We sincerely thank all reviewers for their careful reading and constructive suggestions. Their feedback has been incorporated into the revised manuscript, where all revised text appears in red.

We are happy that the reviewers found (i) the problem practical and important, since paired multimodal data is scarce while unimodal data is abundant [d4c3, 6ZwZ, xUiP], (ii) the paper clear and easy to follow [d4c3, 6ZwZ, xUiP], (iii) the pipeline simple, intuitive and efficient, requiring neither anchors nor OT at inference and training several times faster than SOTAlign [d4c3, 6ZwZ, xUiP], (iv) the hubness analysis informative [d4c3, 6ZwZ], (v) and the experiments careful and fair [6ZwZ, xUiP].

To address specific concerns and incorporate reviewer feedback:

  • Reviewer d4c3: We have conducted both suggested ablations: distilling nearest-neighbor, CSLS and inverted-softmax targets (updated Figure 3, Fig. 4 in the submission), and replacing our cost with a ridge CCA fitted on the anchors (new Appendix C.3.4). We have clarified our contribution as the identification of two failure modes of anchor-relative representations, hubness and spectrum squaring, each corrected by one component of AnchorDOT. We have added paired statistical tests against SOTAlign (new Appendix C.10) and revised the abstract and conclusion accordingly.
  • Reviewer 6ZwZ: We have added paired statistical tests (McNemar tests over queries and t-tests over seeds, new Appendix C.10), results on the standard Flickr30k 1K test set (new Appendix C.8), re-run CSA with a regularized CCA (Tables 1, 2 and 5–7), and clarified the range of γ in Appendix D. We have also added the suggested comparison with a linear cost (new Appendix C.3.4) and revised the presentation of SOTAlign in Sections 1 and 2 accordingly, a new experiment with unpaired pools drawn from different datasets (Appendix C.5.2, new Figure 14), with the corresponding claim qualified in Sections 1 and 5, results with larger encoders (new Appendix C.9), and zero-shot classification on CIFAR-100 (new Appendix C.5.1).
  • Reviewer xUiP: We have clarified in the caption of Table 9 (Table 7 in the submission) that the times are end-to-end, discussed how AnchorDOT could be extended to continual training, and clarified our positioning with respect to OTTER, SAT, RELIT and CCA.

All other points are addressed in the individual responses below. Again, we deeply appreciate the feedback and welcome any further discussion.

Area_Chair_tppqOpenReview ↗

Please respond to the authors

Dear reviewers,

Thank you for your service.

The authors have provided their response to your initial reviews. As we approach the end of the author-reviewer discussion period, could you please respond to the authors’ rebuttal (if you haven’t done so already) and communicate the extent to which your concerns have been addressed, and whether you have any further questions and concerns?

Thanks, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.