- Rating
- 3
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes AnchorDOT, a weakly supervised cross-modal representation alignment method from unimodal encoders. The key idea is to first leverage aligned cross-modal pairs as anchors to represent unlabelled samples with anchor-relative representations, and then use OT to align unlabelled cross-modal samples by using the relative representation distances as the transport cost. Using OT-based pseudo-labels across modalities, AnchorDOT trains two projection heads that project representations from two unimodal encoders into the same embedding space via a CLIP-style soft-label InfoNCE loss. The proposed AnchorDOT achieves promising results across image-text and molecule-text alignment tasks.
Critical strengths
- The paper aim to address cross-modal alignment with unimodal encoders and scarce supervision pairs, which is practical and important problem where alignment across different modalities or domains are hard to obtain.
- The overall architecture of the proposed AnchorDOT is intuitive and easy to understand. AnchorDOT consist of three main steps: anchor-relative representation with orthonormalized similarity, a global OT for pseudo labeling with block-wise computation, and two inductive encoders following a CLIP-style contrastive learning process, which are all clearly explained and well-motivated.
- The empirical evaluation is well-structured and thorough. AnchorDOT is tested on two cross-modality alignment tasks: image-text and molecule-text, and shows consistent outperformance over the SOTA baselines. The authors also evaluate AnchorDOT comprehensive across various dimensions, including matching quality, hubness, and efficiency. Additional study regarding different design choices of AnchorDOT is also comprehensive, such as orthonormalization, alternative OT costs, and hyperparameter sensitivity.
Critical weaknesses
My biggest concern is the novelty of the proposed method. AnchorDOT essentially leverage OT for pseudo-labeling unlabeled cross-modality pairs and then following a CLIP-style contrastive training of two linear projection heads that project unimodal representation into a shared embedding space. Similar ideas of OT-based pseudo-labeling combined with CLIP-style contrastive cross-modal alignment has been explore in OTTER [1] and SAT [2], making the overall novelty of AnchorDOT incremental. The additional heuristics, such as anchor-relative representation with orthonormalized similarity, while simple and intuitive, has been explored extensively in other related works of vision-language learning, such as RELIT[3] and CCA.
[1] Wu, Bichen, et al. "Data efficient language-supervised zero-shot recognition with optimal transport distillation." arXiv preprint arXiv:2112.09445 (2021).
[2] Mishra, Shambhavi, et al. "Semantic Anchor Transport: Robust Test-Time Adaptation for Vision-Language Models." arXiv preprint arXiv:2411.17002 (2024).
[3] Chen, Chi, et al. "Weakly supervised vision-and-language pre-training with relative representations." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
Questions for authors
- The definition of the training wall-clock time provided in Table 7 is unclear. Is it end-to-end runtime that included the OT computation, or only training time on the student model side?
- How can AnchorDOT be extended to continual training setting where OT-based pseudo labels are continually updated using the newly arrived samples? How to enable efficient OT-based labling?
2 comments
Response to Reviewer xUiP (Part 2 of 2)
Q1. Wall-clock time (Table 8)
"The definition of the training wall-clock time provided in Table 7 is unclear. Is it end-to-end runtime that included the OT computation, or only training time on the student model side?"
The times in Table 9 (Table 7 in the submission) are end-to-end for both methods. For AnchorDOT, they include the relative representations, the OT coupling and the distillation. We have clarified the caption.
Q2. Extension to continual training
"How can AnchorDOT be extended to continual training setting where OT-based pseudo labels are continually updated using the newly arrived samples? How to enable efficient OT-based labling?"
We have not evaluated our method to continual training, but we believe AnchorDOT could be extended to it, because the coupling is entirely determined by the dual potentials , which take memory. We distinguish two cases, depending on whether the new samples are unpaired or paired.
Case 1: new unpaired samples. The anchors are unchanged, so the relative representation of a new sample only costs similarities, and the cost matrix is unchanged for existing samples (with kept fixed). The potential of a new sample can then be obtained in closed form by the entropic c-transform, , at cost, without re-solving the problem. Since adding samples perturbs the marginals ( becomes ), a few Sinkhorn iterations would be needed to restore them, warm-started from the existing potentials. Good initializations are known to accelerate Sinkhorn substantially (Thornton & Cuturi, 2023; Amos et al., 2023), and we expect the previous potentials, which should remain close to the new solution when , to be a particularly good one. The targets of the new chunks would then follow in closed form from the updated potentials (Eq. 9), as in the current pipeline.
Case 2: new paired samples. New pairs extend the anchor set, which changes the profile and its orthonormalization, hence the cost for all samples: the coupling would have to be re-solved. We expect a warm start from the previous potentials to remain useful, since adding a few pairs to should change the cost only slightly, but this remains to be verified. The new pairs would also directly enter the anchor chunk as supervision.
Again, we thank the reviewer for the constructive feedback, and welcome any further discussion.
Response to Reviewer xUiP (Part 1 of 2)
We thank the reviewer for the constructive feedback, and for highlighting the practical importance of the problem, the intuitive and well-motivated design of AnchorDOT, and the thorough evaluation. We first address the novelty concern, then the two questions. Figure and table numbers refer to the revised PDF.
Regarding the novelty concern
We agree that OT-based pseudo-labeling and relative representations have antecedents, which Section 2 discusses (OTTER, ASIF, RELIT). The setting differs, however: OTTER computes in-batch OT on noisy web-collected image–text pairs, with teacher encoders trained on paired data, and SAT assigns test images to class-text embeddings of CLIP, an already aligned model. In both, a reliable cross-modal similarity exists by construction. Between independently trained encoders, none exists and the anchors provide it.
This is where we identified two failure modes of anchor-relative representations (as used by ASIF and RELIT), each corrected by one component of AnchorDOT:
We believe this identification of the failure modes of anchor-relative representations, and of a targeted correction for each, goes beyond a combination of existing parts.