acceptodds
Under review as a conference paper at ICLR 2027

Unpaired Multimodal Learning with Cross-Modal Self-Supervision

Abstract

Multimodal learning typically relies on paired observations, yet many domains, including biology and medicine, provide abundant unimodal data with only sparse and potentially imperfect pairing. We introduce unimo-x, a general learning framework for training multimodal Transformers on highly unpaired corpora. Our approach uses teacher-anchored cross-modal self-supervision to turn unpaired observations into training signals, complementing paired supervision while mitigating steganographic shortcuts that can undermine consistency-based learning. We instantiate unimo-x in 4M and evaluate it across several datasets and tasks. Unpaired observations improve image generation and caption semantics over paired-only training.With 25%, 17.5%, or 10% paired samples, unimo-x recovers substantial benefits of paired supervision, from competitive image-generation quality with four times fewer pairs to stronger caption representations than paired-only training even at 10% pairing. These findings point to a practical route to multimodal learning in which sparse paired examples guide training on much larger unimodal corpora.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal foundation models, unpaired multimodal learning, cross-modal self-supervision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pM2QOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes unimo-x, a framework for learning multimodal representations from predominantly unpaired data using cross-modal self-supervision. The method first establishes cross-modal mappings through limited paired training and then exploits unpaired observations through cycle-consistent reconstruction. To prevent the model from encoding hidden information in generated intermediate representations, the authors introduce a teacher-anchoring loss that constrains student predictions toward those of an EMA teacher. The framework is implemented on the 4M backbone and evaluated on image generation, captioning, retrieval, and caption-based classification

Critical strengths

Relevant research problem. Learning from unpaired multimodal data is an important problem, particularly in domains where collecting aligned observations is expensive or impractical.

Meaningful analysis of shortcut learning. The paper investigates steganographic shortcuts in cycle-consistent learning, where successful reconstruction does not necessarily imply meaningful cross-modal alignment. The proposed diagnostics, including semantic drift, intermediate substitution, and semantic fidelity, provide useful evidence of this phenomenon.

Reasonable experimental improvements. The method achieves notable improvements in image generation with fewer paired samples. The authors also investigate different pairing ratios, introduce a no-cycle baseline, and evaluate unpaired data from another corpus.

Critical weaknesses
  1. Missing comparisons with modern synthetic pairing approaches. Generating synthetic captions using pretrained vision-language models, such as BLIP and recent MLLM-based captioning methods, is an established approach to reducing dependence on paired image-text data. However, the paper does not compare against this practical alternative. Notably, CommonCatalog already contains BLIP-2-generated captions, which the authors discard to construct unpaired data. Training the same backbone directly on these synthetic pairs would be an important baseline to establish whether cycle-based learning offers additional benefits.

  2. Although unimo-x is presented as a general framework for unpaired multimodal learning, it relies on pretrained CLIP representations that already encode extensive image-text alignment knowledge.

  3. Thus, there is a mismatch between motivation and experimental validation. The Introduction emphasizes biological and medical applications, where heterogeneous modalities are often independently collected and difficult to align. However, experiments are limited to the mature image-text domain, with unpaired samples largely constructed by removing modalities from existing paired datasets. This does not adequately reflect the distribution shifts, heterogeneous measurements, or missingness mechanisms in the motivating applications.

  4. Limited methodological novelty over closely related work. The framework closely resembles existing cycle-consistent multimodal learning methods, particularly ITIT and DoraCycle. Key components, including paired pretraining, cross-modal reconstruction, and EMA teacher-student learning, have been explored previously. The main addition, teacher-anchoring loss, is conceptually similar to established pseudo-labeling and consistency regularization. Its novelty should be better justified through comparisons with alternative intermediate supervision objectives.

Questions for authors

see the weakness

2 comments

Overall response: We thank the reviewer for the thoughtful feedback and for recognizing the value of our shortcut analysis. The comments helped us strengthen the experimental evidence and clarify our contribution. Below, we address synthetic pairing, pretrained supervision, experimental scope, and novelty.

W1: Missing comparisons using synthetic data pairings.

Training on BLIP-2 captions would provide a useful complementary baseline, and we do not establish superiority over synthetic pairing. Our experiments instead test whether unpaired observations improve training across corpora when their correspondences are unavailable. We therefore retain the limited paired CC12M subset but withhold CommonCatalog’s image - caption correspondences, using disjoint image-only and caption-only observations.

Our method generates intermediates through an EMA teacher initialized from the available paired training and updated alongside the student. It therefore does not require an additional pretrained cross-modal generator for the target modalities - an assumption motivated by applications where reliable generators of corresponding measurements may not be available.

W2: unimo-x relies on pretrained CLIP representations that already encode extensive image-text alignment knowledge.

To test this dependence, we replaced CLIP with DINOv2 features learned without text supervision. We compared cycle training with the corresponding paired-only baseline under the same 25% paired setting, late-fork schedule, and training-token budget.

Cycle training reduces FID from 60.43 to 51.52 and improves CLIPScore and CIDEr, while retrieval remains slightly lower (Table R2; Appendix §B.4, Table 7). Although absolute performance is below the CLIP-based configuration, these results show that cycle gains do not require text-supervised pretraining of the bridge encoder.

Table R2: COCO-based replacing CLIP with DINOv2 embeddings.

Training settingFID ↓CLIPScore ↑CIDEr ↑T2I R@1 ↑
Paired-only60.4317.410.0864.64
Cycle51.5217.890.0934.60

W3: There is a mismatch between motivation and experimental validation.

We acknowledge that image - caption experiments do not capture the full complexity of biological measurements, batch effects, and missingness. Biology motivates our focus on sparse data pairings, while image–text provides a controlled setting to examine cycle benefits, steganographic shortcuts, teacher anchoring, EMA, and training schedules in depth.

CommonCatalog provides an initial test across different data distributions, but does not replace biological validation. We have clarified this distinction between motivation and experimental scope in the Introduction.

W4: Limited novelty relative to ITIT and DoraCycle.

Our contribution is to characterize and mitigate a failure of cross-modal reconstruction: reconstruction can improve while dense intermediates lose semantic fidelity. Teacher anchoring constrains this behavior while preserving reconstruction gradients through intermediate predictions.

ITIT compares two different cycle settings. In its half-cycle setting, it treats generated intermediates as fixed inputs, blocking direct reconstruction gradients into their predictions. Its full-cycle setting retains these gradients and achieves better results. Our experiments show that this gradient path, however, introduces steganography when cycles contain consecutive dense visual representations, such as CLIP -> RGB. Teacher anchoring preserves full-cycle learning while suppressing these observed shortcuts. This extends the investigation beyond ITIT’s image - text cycles toward settings with multiple dense modalities, motivated by connecting different spatial biological measurements.

DoraCycle uses LoRA to adapt a frozen pretrained Show-o model, assuming existing image–text alignment and generation capabilities. We instead train the shared backbone from random initialization with limited paired data, while retaining pretrained modality-specific components. Both methods use EMA and differentiable intermediates, but DoraCycle adjusts gradient directions to address conflicts between cycle objectives; we directly constrain intermediate predictions to suppress reconstruction shortcuts.

Appendix A.3 also reports exploratory comparisons with token corruption, low-pass filtering, and distance preservation, which did not eliminate the observed artifacts in the tested configurations. These observations motivated directly constraining intermediate predictions through teacher anchoring.

Reviewer_pM2QOpenReview ↗

response

I thank the authors for their response and additional experiments. The DINOv2 results partially address my concern about reliance on CLIP. However, I still have two main concerns.

First, EMA-based teacher-student learning is a well-established technique for handling noisy co-response, as seen in vision-language pretraining methods such as ALBEF. Its novelty over existing consistency regularization methods remains unclear.

Second, I agree with the authors that applications where reliable cross-modal generators are unavailable are worth studying. However, the experiments are still limited to standard image-text datasets, where powerful pretrained encoders and generators are readily available. I would like to see results on real heterogeneous multimodal data, such as medical imaging or another domain where reliable cross-modal generators are unavailable. This would provide stronger support for the motivation and generality of the method.

I therefore maintain my original rating.

Reviewer_zRTAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes unimo-x to train a shared multimodal Transformer with a small paired set and a larger unpaired set. An EMA teacher generates intermediate modalities, while a student predicts them and reconstructs the observed input through a cycle with an anchoring loss.

Critical strengths
  1. Learning from mostly unpaired data is a useful problem.

  2. The teacher-anchoring objective addresses a concrete shortcut in cycle training. The paper measures changes in intermediate representations rather than relying only on final task scores.

  3. The experiments cover several paired fractions and include paired-only, no-cycle, and cross-corpus controls.

Critical weaknesses
  1. The gains are not consistent across tasks. At 25% pairing, Table 1 shows FID improving from 45.15 to 40.90, while CIDEr falls from 0.171 to 0.135 and text-to-image Recall@1 falls from 6.38 to 5.86.

  2. The main teacher-anchoring ablation uses an early training setting, while the best results use a later setting. Also, the image-to-caption-to-image cycle has no anchoring loss, and the shortcut diagnostics focus on the caption-starting cycle. It remains unclear which part of the cycle causes the main gains.

  3. The generality of the framework is not yet shown. Experiments use one 4M backbone and image-caption data, while the third stream is CLIP features derived from the image. Frozen CLIP also brings prior image-text supervision, so the reported paired fractions do not describe all paired supervision used.

  4. The 150B-token budget counts gradient-bearing tokens but excludes teacher sampling. The paper gives approximate run time, but not a matched total-compute comparison across methods or variation over repeated runs. This limits the strength of the efficiency and small-difference claims.

Questions for authors

Please address the above weaknesses

3 comments

Overall response: We thank the reviewer for the constructive assessment and thoughtful feedback. Below, we address the task-dependent gains, anchoring at the late fork, dependence on CLIP supervision, and the additional computational cost of cycle training.

W1: The gains are not consistent across tasks. At 25% pairing, Table 1 shows FID improving, while CIDEr falls and T2I Recall@1 falls

We agree that the gains are task-dependent. Our new cycle-direction ablation (Appendix §B.5, Table 8) makes this trade-off explicit. The caption-starting cycle achieves CIDEr 0.173, compared with 0.101 for the image-starting cycle. Combining both achieves the lowest FID (40.90), but intermediate CIDEr (0.135) and lower retrieval Recall@1 than either direction alone. These results locate the retrieval decrease in the combined-cycle setting, although they do not establish its underlying cause.

CIDEr measures lexical agreement with COCO reference captions and can therefore penalize semantically correct descriptions that differ in wording or specificity. CC12M contains web-derived captions that can differ substantially from COCO references in wording and specificity. We now clarify this limitation of CIDEr in its description in the updated Appendix §A.5. Retrieval scores likewise reflect transfer from CC12M to COCO and measure instance-level image-text matching rather than lexical agreement.

Table 3 complements CIDEr with caption-based classification, which improves across all tested paired fractions. At 25% pairing, MiniLM accuracy rises from 13.14% to 17.82%, indicating better preservation of object-class information. Table 1 separately reports CLIPScore for image-text alignment in image generation.

W2: Anchoring is evaluated only at the early fork, omitted in the image-starting cycle, and the contribution of each cycle is unclear.

Late-fork steganography. To align the anchoring analysis with our main training setting, we repeated the ablation at the late fork. Figure 7 shows the same qualitative pattern as the early-fork experiment: without anchoring, caption reconstruction increasingly benefits from student intermediates c2c_2 over substituted teacher intermediates c1c_1, while student-teacher drift grows and the student CLIP intermediates lose fidelity to the ground-truth image representation.

Caption steganography. In the caption -> (CLIP -> image) -> caption cycle, a short caption is mapped first into a dense spatial token grid of CLIP from which we then generate RGB images. These high-capacity representations offer substantial freedom to encode the source caption through private token patterns that support reconstruction without preserving meaningful visual content. Our steganography analysis demonstrates this shortcut behavior. Teacher anchoring constrains this freedom by penalizing deviations from the teacher’s predictions.

Image -> caption -> image cycles instead compress the visual input into a caption of at most 64 discrete tokens, processed through a frozen T5 encoder. This creates a substantially narrower channel, with continued paired supervision further constraining the image - text mapping. This capacity difference motivates concentrating explicit anchoring on the dense visual intermediates where we observe shortcuts. While we don’t claim that a text bottleneck is inherently immune to steganography, we have not observed comparable shortcut behavior in this direction.

Gains of cycle. To further analyze which part of the cycles contribute to which parts, we have added two additional experiments in Appendix §B.5. Using the late-fork setting, we a) only use the caption -> (CLIP -> image) -> caption and b) only the image -> caption -> image cycle to disentangle which cycle introduces the main performance improvements. As shown in Table R1 (Table 8), the ablations show complementary effects. The caption-starting cycle improves CIDEr, while the image-starting cycle reduces FID, consistent with their respective reconstruction objectives. Combining both cycles further improves FID, but yields lower CIDEr than the caption-starting cycle alone. These results clarify each cycle’s contribution and reveal a trade-off between caption agreement and image-generation quality.

Table R1: Effect of cycle direction at 25% pairing with the same late-fork schedule and training-token budget.

Training settingFID ↓CLIPScore ↑CIDEr ↑T2I R@1 ↑
Paired-only45.1518.870.1716.38
Image-starting cycle41.2619.080.1016.70
Caption-starting cycle45.2218.580.1736.62
Both cycles40.9018.960.1355.86

W3: The generality of the framework is not yet shown. The experiments rely on 4M and image-derived CLIP embeddings which introduces prior image-text supervision.

We chose 4M as a general any-to-any multimodal architecture representing a family of multimodal encoder-decoder models. Its shared Transformer supports multiple modalities and prediction directions, allowing us to evaluate image generation, captioning, retrieval, and caption-based classification within one backbone. Our contribution concerns the cross-modal training strategy, rather than task-specific architectural changes.

Our paired fractions describe the training corpus, not the supervision inherited from pretrained components, as clarified in §4.1. To assess the dependence on CLIP’s pretrained alignment and the benefit of cycles, we ran a full training run in which we replaced the CLIP representation with DINOv2 features removing text-supervised pretraining from the intermediate image representations. As shown in Table R2 and Appendix §B.4, under the same 25% paired setting and late-fork schedule, cycle training still improves image generation over the corresponding paired-only DINOv2 baseline. Although absolute performance is lower with DINOv2 in our experiments, the benefit of cycle training persists without CLIP’s inherited image - text alignment.

Table R2: COCO-based replacing CLIP with DINOv2 embeddings.

Training settingFID ↓CLIPScore ↑CIDEr ↑T2I R@1 ↑
Paired-only60.4317.410.0864.64
Cycle51.5217.890.0934.60

W4. The comparison between cycle and no cycles does not account for additional compute overhead.

We highlight the difference between our reporting gradient-bearing based metric in our experimental section to computational expense in §4.1: “Runs are compared by gradient-bearing training tokens, including student-predicted intermediates. This excludes teacher sampling and does not equate total computational cost.”. We now explicitly mention the computational overhead of about 2.7x FLOPs in Section A.3, and added Figure 4 to the appendix which further reflects the total-compute expense difference between the paired and cycle-added runs.

We would like to emphasize, however, that we do not claim computational efficiency but paired data efficiency.

Thank you for the additional experiments and clarifications. I also read another reviewer's comment and the corresponding response. Despite the interesting and promising topic of using unpaired data, this manuscript does not fully demonstrate its potential compared with paired data. This paper needs a careful and rigorous discussion to convince readers. I would like to maintain the score.

Reviewer_S29kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to learn a unified multimodal model from predominantly unpaired multimodal data, i.e. samples in which only one modality is observed. Given a small number of paired examples and a significantly large number of unpaired examples, the paper proposes a framework based on consistency-based learning and self-supervised learning to learn a unified multimodal model by turning the unpaired data into a training signal for cross-modal generation. The paper identifies steganographic shortcuts and provides an anchoring loss for mitigating these shortcuts. The authors train their proposed framework on CC12M dataset and evaluate it on COCO and ImageNet, showing that their method improves cross-modal generation over the paired-only and no-cycle baselines. The paper also evaluates the generated captions on ImageNet classification task to measure how much of the object-class information they preserve.

Critical strengths
  1. Steganographic Shortcuts Analysis - Steganographic shortcuts are a known failure mode in cycle-consistency based learning. The authors diagnosed and studied the presence of steganographic shortcuts in the caption →\rightarrow (CLIP →\rightarrow RGB) →\rightarrow caption cycle using a. A cosine-distance-based measure of teacher-student semantic drift, b. The change in reconstruction CE when the student's intermediate is replaced with the teacher's, and c. The semantic fidelity of the intermediate representation. These assess the effectiveness of the proposed anchoring loss well.
  2. Empirical Evaluation - Experiments compare against several baselines and against versions of the proposed framework with its main components removed (the cycle objective, the EMA teacher, and the anchoring loss), effectively evaluating the contribution of each of the components to the framework.
Critical weaknesses
  1. Image-derived CLIP as a modality - The paper uses three modality streams: captions, RGB images, and image-derived CLIP representations. Generation in both the directions, i.e., caption →\rightarrow CLIP →\rightarrow RGB and RGB →\rightarrow CLIP →\rightarrow caption, uses this Image-derived CLIP as an intermediate modality. Because the original CLIP model is trained contrastively on large-scale image-text pairs, this image-derived CLIP acts as a semantic bridge for the unpaired images. This understates the cross-modal supervision available to the model. It is not clear if the method would work without such a bridge, since generation is an important part of the framework. The paper also motivates the method with biomedical domains where no such bridge usually exists, but it does not test this setting. An ablation with only caption and RGB streams would show how much the gains depend on CLIP being used as the third modality.
  2. Unclear comparison against other steganography mitigation methods - The paper motivates teacher anchoring as the other mitigation methods failed, but the details of the results are missing. Section A.3 reports that the three mitigation methods (token replacement with embedding noise, low-pass filtering, and a distance-preservation loss) did not prevent the shortcut, but provides no numbers beyond a distance correlation of 0.98. None of the diagnostics used to evaluate anchoring are reported for these methods. Reporting the same diagnostics and downstream metrics for these alternatives under the same setting as anchoring would show whether/extent to which anchoring is the better strategy over others.
  3. Single-seed results - All the reported results seem to be with only a single-seed since the variance across seeds or error bars are all missing. Many of the reported differences are small enough that seed-to-seed variation could account for them.
Questions for authors
  1. In Table 1, paired-only training at 25% pairing reaches a higher CIDEr (0.171) than fully paired training (0.144). Can you provide some clarification on why this could be?] - I would expect the fully paired (100%) setting to reach a higher CIDEr than paired-only training at 25% pairing. If this gap is due to seed-to-seed variance, other small differences in Table 1 may also fall within noise

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. Both recognized the relevance of learning from predominantly unpaired data and the value of diagnosing steganographic shortcuts. Their main concerns involved inconsistent gains across tasks, the alignment between our anchoring ablations and main training setting, dependence on CLIP’s pretrained supervision, computational cost, and the framework’s scope and novelty.

We conducted four additional training runs and expanded the manuscript accordingly:

  • Late-fork teacher-anchoring ablation (§B.3, Figure 7). To address zRTA’s concern that the original anchoring analysis used an early fork, we repeated the analysis under the late-fork setting used for our main results. Without anchoring, the same shortcut patterns emerge, supporting the role of the proposed anchoring loss at both early and late training stages.
  • Two individual cycle-direction ablations (§B.5, Table 8). In response to zRTA’s questions about inconsistent task gains and which cycle contributes to them, we trained variants using only the image-starting or caption-starting cycle. The image-starting cycle achieves lower FID, whereas the caption-starting cycle achieves higher CIDEr. Combining both yields the lowest observed FID but lower retrieval Recall@1, making the task trade-offs explicit.
  • Testing dependence on pretrained image-text alignment (§B.4, Table 7). To address the concern raised by both reviewers about implicit image-text supervision, we replaced CLIP with DINOv2 features learned through image-only self-supervision. Cycle training reduces FID from 60.43 to 51.52 and improves CLIPScore and CIDEr over the corresponding paired-only baseline. These results show that cycle benefits persist without text-supervised pretraining of the image representation encoder.

We now more clearly state the computational overhead (§A.3, Figure 4) introduced through cycles in response to zRTA. The added plot reports FID against total training compute, including teacher sampling.

Finally, we clarified the interpretation of CIDEr (§A.5) and the distinction between biological motivation and image-text validation in the Introduction. In response to pM2Q, we explain the different assumptions behind external synthetic pairing and our EMA-generated intermediates, and clarify our contribution relative to ITIT and DoraCycle.

We hope these additions and clarifications address the reviewers’ concerns and support reconsideration of their assessments. Detailed responses are provided under each review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.