- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes unimo-x, a framework for learning multimodal representations from predominantly unpaired data using cross-modal self-supervision. The method first establishes cross-modal mappings through limited paired training and then exploits unpaired observations through cycle-consistent reconstruction. To prevent the model from encoding hidden information in generated intermediate representations, the authors introduce a teacher-anchoring loss that constrains student predictions toward those of an EMA teacher. The framework is implemented on the 4M backbone and evaluated on image generation, captioning, retrieval, and caption-based classification
Critical strengths
Relevant research problem. Learning from unpaired multimodal data is an important problem, particularly in domains where collecting aligned observations is expensive or impractical.
Meaningful analysis of shortcut learning. The paper investigates steganographic shortcuts in cycle-consistent learning, where successful reconstruction does not necessarily imply meaningful cross-modal alignment. The proposed diagnostics, including semantic drift, intermediate substitution, and semantic fidelity, provide useful evidence of this phenomenon.
Reasonable experimental improvements. The method achieves notable improvements in image generation with fewer paired samples. The authors also investigate different pairing ratios, introduce a no-cycle baseline, and evaluate unpaired data from another corpus.
Critical weaknesses
-
Missing comparisons with modern synthetic pairing approaches. Generating synthetic captions using pretrained vision-language models, such as BLIP and recent MLLM-based captioning methods, is an established approach to reducing dependence on paired image-text data. However, the paper does not compare against this practical alternative. Notably, CommonCatalog already contains BLIP-2-generated captions, which the authors discard to construct unpaired data. Training the same backbone directly on these synthetic pairs would be an important baseline to establish whether cycle-based learning offers additional benefits.
-
Although unimo-x is presented as a general framework for unpaired multimodal learning, it relies on pretrained CLIP representations that already encode extensive image-text alignment knowledge.
-
Thus, there is a mismatch between motivation and experimental validation. The Introduction emphasizes biological and medical applications, where heterogeneous modalities are often independently collected and difficult to align. However, experiments are limited to the mature image-text domain, with unpaired samples largely constructed by removing modalities from existing paired datasets. This does not adequately reflect the distribution shifts, heterogeneous measurements, or missingness mechanisms in the motivating applications.
-
Limited methodological novelty over closely related work. The framework closely resembles existing cycle-consistent multimodal learning methods, particularly ITIT and DoraCycle. Key components, including paired pretraining, cross-modal reconstruction, and EMA teacher-student learning, have been explored previously. The main addition, teacher-anchoring loss, is conceptually similar to established pseudo-labeling and consistency regularization. Its novelty should be better justified through comparisons with alternative intermediate supervision objectives.
Questions for authors
see the weakness
2 comments
response
I thank the authors for their response and additional experiments. The DINOv2 results partially address my concern about reliance on CLIP. However, I still have two main concerns.
First, EMA-based teacher-student learning is a well-established technique for handling noisy co-response, as seen in vision-language pretraining methods such as ALBEF. Its novelty over existing consistency regularization methods remains unclear.
Second, I agree with the authors that applications where reliable cross-modal generators are unavailable are worth studying. However, the experiments are still limited to standard image-text datasets, where powerful pretrained encoders and generators are readily available. I would like to see results on real heterogeneous multimodal data, such as medical imaging or another domain where reliable cross-modal generators are unavailable. This would provide stronger support for the motivation and generality of the method.
I therefore maintain my original rating.
Overall response: We thank the reviewer for the thoughtful feedback and for recognizing the value of our shortcut analysis. The comments helped us strengthen the experimental evidence and clarify our contribution. Below, we address synthetic pairing, pretrained supervision, experimental scope, and novelty.
W1: Missing comparisons using synthetic data pairings.
Training on BLIP-2 captions would provide a useful complementary baseline, and we do not establish superiority over synthetic pairing. Our experiments instead test whether unpaired observations improve training across corpora when their correspondences are unavailable. We therefore retain the limited paired CC12M subset but withhold CommonCatalog’s image - caption correspondences, using disjoint image-only and caption-only observations.
Our method generates intermediates through an EMA teacher initialized from the available paired training and updated alongside the student. It therefore does not require an additional pretrained cross-modal generator for the target modalities - an assumption motivated by applications where reliable generators of corresponding measurements may not be available.
W2: unimo-x relies on pretrained CLIP representations that already encode extensive image-text alignment knowledge.
To test this dependence, we replaced CLIP with DINOv2 features learned without text supervision. We compared cycle training with the corresponding paired-only baseline under the same 25% paired setting, late-fork schedule, and training-token budget.
Cycle training reduces FID from 60.43 to 51.52 and improves CLIPScore and CIDEr, while retrieval remains slightly lower (Table R2; Appendix §B.4, Table 7). Although absolute performance is below the CLIP-based configuration, these results show that cycle gains do not require text-supervised pretraining of the bridge encoder.
Table R2: COCO-based replacing CLIP with DINOv2 embeddings.
W3: There is a mismatch between motivation and experimental validation.
We acknowledge that image - caption experiments do not capture the full complexity of biological measurements, batch effects, and missingness. Biology motivates our focus on sparse data pairings, while image–text provides a controlled setting to examine cycle benefits, steganographic shortcuts, teacher anchoring, EMA, and training schedules in depth.
CommonCatalog provides an initial test across different data distributions, but does not replace biological validation. We have clarified this distinction between motivation and experimental scope in the Introduction.
W4: Limited novelty relative to ITIT and DoraCycle.
Our contribution is to characterize and mitigate a failure of cross-modal reconstruction: reconstruction can improve while dense intermediates lose semantic fidelity. Teacher anchoring constrains this behavior while preserving reconstruction gradients through intermediate predictions.
ITIT compares two different cycle settings. In its half-cycle setting, it treats generated intermediates as fixed inputs, blocking direct reconstruction gradients into their predictions. Its full-cycle setting retains these gradients and achieves better results. Our experiments show that this gradient path, however, introduces steganography when cycles contain consecutive dense visual representations, such as CLIP -> RGB. Teacher anchoring preserves full-cycle learning while suppressing these observed shortcuts. This extends the investigation beyond ITIT’s image - text cycles toward settings with multiple dense modalities, motivated by connecting different spatial biological measurements.
DoraCycle uses LoRA to adapt a frozen pretrained Show-o model, assuming existing image–text alignment and generation capabilities. We instead train the shared backbone from random initialization with limited paired data, while retaining pretrained modality-specific components. Both methods use EMA and differentiable intermediates, but DoraCycle adjusts gradient directions to address conflicts between cycle objectives; we directly constrain intermediate predictions to suppress reconstruction shortcuts.
Appendix A.3 also reports exploratory comparisons with token corruption, low-pass filtering, and distance preservation, which did not eliminate the observed artifacts in the tested configurations. These observations motivated directly constraining intermediate predictions through teacher anchoring.