acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Multimodal (Mis)Alignment for Vision-Language Representation Learning

Abstract

Vision-language models (VLMs) commonly learn by aligning image and text representations, achieving strong performance on zero-shot classification and image-text retrieval. However, this paradigm is sensitive to imperfect image-text correspondence and can overlook information that is complementary rather than shared across modalities. We investigate whether multimodal representation learning can provide a more robust alternative while retaining strong cross-modal representations. For the first time, we systematically assess CoMM as a vision-language representation learner, studying its behavior across image-text alignment quality and pretraining scale. We find that CoMM is substantially more robust to misaligned pretraining pairs than alignment-based objectives such as CLIP, while learning strong visual and multimodal representations. At larger scale, however, CoMM lags behind CLIP on standard cross-modal tasks, revealing a trade-off between robustness to misalignment and efficient learning of explicit image-text alignment. To bridge this gap, we introduce CoMM-VL, which augments CoMM's multimodal objective with an explicit image-text alignment loss within the same fused representation space. CoMM-VL substantially improves zero-shot classification and image-text retrieval over CoMM and outperforms established alignment-based VLMs, including CLIP, SigLIP, and TIPS, while preserving CoMM's robustness to misaligned data and strong multimodal representations. Across small- and large-scale vision-language datasets, these results show that explicit cross-modal alignment and multimodal interaction can be complementary rather than competing objectives. Code and pretrained weights will be open-sourced.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Multimodal Learning, Data Misalignment

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_4o71OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the relationship between image–text correspondence quality, pretraining scale, and vision–language representations. It proposes CoMM-VL, which adds an alignment loss between image-only and text-only representations in CoMM’s fusion space to improve cross-modal matching while retaining multimodal representations. Evaluation uses CC12M and LAION-COCO pretraining and covers classification, retrieval, multimodal probing, and transfer to frozen language models. The central result is a substantial improvement in cross-modal performance over CoMM, supported by CC12M ablations and competitive large-scale results.

Critical strengths
  1. A useful empirical contribution with substantial gains. On CC12M, CoMM-VL improves CoMM’s ImageNet zero-shot accuracy from 33.6% to 41.8% and mean image-to-text retrieval from 33.7% to 50.1%. This addresses a concrete limitation of CoMM, rather than relying on marginal gains on a single benchmark. The breadth of classification, retrieval, multimodal probing, and frozen-LLM transfer is the main reason I lean toward acceptance.
  2. Relevant ablations support the design at medium scale. Table 4 tests fusion depth and alignment location. Increasing fusion depth alone reaches 35.1% zero-shot accuracy, compared with 41.8% for CoMM-VL, while CoMM and CoMM-VL both reach 64.1% with the fused MM readout. Together with the parameter-count control, this provides useful evidence that the modification improves cross-modal performance while retaining useful fused representations. The method is simple, but the empirical trade-off is informative.
Critical weaknesses
  1. Correspondence quality is not fully separated from other data properties. The three equal-sized subsets are selected by pretrained-model similarity. This controls sample count but may also change category, source, image quality, or caption properties; low similarity need not mean incorrect correspondence. The observations on those subsets remain useful, but attributing the differences specifically to correspondence quality requires validation or narrower wording. This limits the strength of the paper’s explanatory claim and keeps my recommendation at weak acceptance.
  2. Large-scale objective and representation attribution remain incomplete. The CC12M controls are valuable, but LAION-COCO also involves changes to fusion depth, token pooling, and text masking, with masking disabled for CoMM-VL. In addition, multimodal probes compare concatenated native features with CoMM-VL’s fused features, and large-scale visual probes use different readout locations. These comparisons can assess complete systems, but do not fully isolate the new loss or native-encoder quality. A matched large-scale control and clearer interpretation would materially strengthen the paper.
Questions for authors
  1. What validates the similarity splits as levels of actual correspondence quality? Please provide key distribution statistics and an independently checked sample, or explicitly restrict the conclusions to the similarity-based subsets.
  2. Please list the actual fusion depth, pooling, and masking settings for CoMM and CoMM-VL at each scale. Is there a LAION-COCO control that holds the CoMM-VL configuration fixed and removes only the added alignment loss?
  3. Can you provide matched-readout CoMM and CoMM-VL results at large scale, or clearly distinguish complete-system claims from claims about native encoders? The existing CC12M concat/fused controls already help at that scale.
  4. Section 5.1 describes synthetic COCO-style captions for LAION-COCO, while Appendix Table 5 says original web alt-text. Which field was used, and was it the same across methods? A table typo would be readily fixable; different training fields could materially affect the comparison.
3 comments

We thank the reviewer for their detailed and constructive feedback, and for acknowledging the usefulness of our empirical contribution, the breadth of the evaluation and the relevance of the ablations. Please find below our responses to your concerns. We hope they adequately address the points you raised and will encourage you to reconsider your rating.

W1 and Q1. Similarity splits and correspondence quality.

We agree that selecting subsets by the similarity of a pretrained model controls the number of samples, but that other factors may still vary across subsets. We address this in two ways: statistics of the splits, and a controlled experiment in which only the correspondence changes.

Statistics of the splits. We computed distribution statistics of the three splits on 3,657 pairs from three CC12M shards. More precisely, we measured the image-text similarity under PE-Core and under a second model (CLIP ViT-L/14), the caption length and style (captions with more than one sentence, or with a stock-image keyword such as "stock photo" or "vector illustration"), the image size, the aesthetic score (LAION aesthetic predictor; higher is better), and the source website of each image.

LowAlignMediumAlignHighAlign
PE-Core similarity (mean)0.1980.2500.296
CLIP ViT-L/14 similarity (mean)0.2300.2710.305
Caption length (median words)111415
Multi-sentence captions (%)19.125.630.5
Captions with a stock-image keyword (%)5.78.713.3
Image size (median megapixels)0.480.510.58
Aesthetic score (mean)5.025.014.79
Source (%)
Pinterest14.217.410.7
Dreamstime12.212.512.5
Slideplayer0.52.57.8
Booking.com6.10.80.2

The second model agrees with the splits: about two thirds of the LowAlign and HighAlign pairs fall in the same third of its own similarity ranking, and only 7% of the LowAlign pairs fall in its top third. The splits differ only slightly in caption length, image size and aesthetic score. Caption style differs more: HighAlign captions more often span several sentences and more often carry the wording of stock-image websites. Their two most frequent sources are the same, but the splits are not identical in content: the largest differences are on two smaller sources, with more presentation slides in HighAlign (Slideplayer) and more hotel listings in LowAlign (Booking.com). Splits based on similarity therefore cannot fully separate correspondence from other properties of the data.

Controlled experiment. To address this concern, we designed an experiment in which only the correspondence changes: on the HighAlign split, we shuffle a fraction pp of the captions among samples.

HighAlign (p=0%p=0\%)ZSLPMM-LPi2tt2iShuffled (p=25p=25%)ZSLPMM-LPi2tt2iShuffled (p=50p=50%)ZSLPMM-LPi2tt2i
CLIP14.048.958.820.113.2CLIP4.434.755.84.63.4CLIP1.928.754.52.11.4
CoMM17.366.961.019.912.4CoMM7.962.560.06.83.5CoMM3.059.259.72.51.2
CoMM-VL23.069.162.728.119.7CoMM-VL14.762.359.813.49.5CoMM-VL7.656.257.45.93.8

The three methods separate clearly as soon as captions are shuffled, as on the similarity splits. On the visual and multimodal probes, CLIP degrades sharply (34.7% then 28.7% linear probe), while CoMM and CoMM-VL remain far above it: CoMM-VL is on par with CoMM at p=25%p=25\% (62.3% vs. 62.5% linear probe, 59.8 vs. 60.0 MM-LP) and within 3.0 and 2.3 points of it at p=50%p=50\%. On cross-modal tasks, CoMM-VL is the most robust of the three at both levels: it reaches 14.7% zero-shot accuracy at p=25%p=25\% (7.9% for CoMM, 4.4% for CLIP) and 7.6% at p=50%p=50\% (3.0% and 1.9%).

This shows that degrading the correspondence alone, with every other property held fixed, is enough to reproduce the differences between methods observed on the similarity splits. We see the two experiments as complementary: the splits reflect the partial mismatches found in real web data but vary several properties at once, while shuffling isolates the correspondence, at the cost of a more extreme form of misalignment.

We will clarify the paper accordingly: the three subsets will be described as splits by image-text similarity, used as a proxy for correspondence quality, and the conclusion that robustness to imperfect correspondence drives the differences will rest on both the splits and the controlled experiment. We will add both analyses to the appendix.

W2. Large-scale attribution of the loss and of the representation.

We agree that the LAION-COCO results compare complete configurations: in the paper, CoMM follows its original recipe (pooled tokens, 1-layer fusion, text masking), while CoMM-VL uses the configuration selected in our ablations. We address the two parts of this concern below: a matched control for the loss (Q2) and matched readouts (Q3).

Q2. Settings per method and scale, and a LAION-COCO control without the alignment loss.

The settings of the CoMM and CoMM-VL models of the main results table are:

SettingCoMM (CC12M)CoMM-VL (CC12M)CoMM (LAION-COCO)CoMM-VL (LAION-COCO)
Fusion inputpooledpooledpooledunpooled
Fusion layers1212
Text maskingp=0.15p=0.15p=0.15p=0.15p=0.15p=0.15p=0p=0

We will separate the two methods in Table 5 of the appendix accordingly. To isolate the alignment loss from these settings, we ablate them below.

Fusion input and text masking. We trained CoMM and CoMM-VL on LAION-COCO for 20k steps with a 1-layer fusion module, varying the fusion input and the text masking one at a time.

MethodFusion inputText maskingZSi2tMM-LP
CoMMpooledp=0.15p=0.1525.519.669.6
CoMMpooledp=0p=031.630.768.4
CoMMunpooledp=0.15p=0.1526.622.568.7
CoMMunpooledp=0p=029.728.468.7
CoMM-VLpooledp=0.15p=0.1537.141.368.8
CoMM-VLpooledp=0p=039.143.168.6
CoMM-VLunpooledp=0.15p=0.1539.546.568.4
CoMM-VLunpooledp=0p=040.048.468.6

In a matched configuration (unpooled tokens, p=0p=0), adding the alignment loss raises zero-shot accuracy from 29.7% (CoMM) to 40.0% (CoMM-VL), and image-to-text R@1 from 28.4% to 48.4%. This gap persists over the full 100k-step schedule: trained in the same configuration, CoMM reaches 43.1% zero-shot accuracy, slightly above the 40.2% of the configuration reported in the paper, but still far below CoMM-VL (58.1%).

Finally, removing text masking improves both models in zero-shot and retrieval: with pooled tokens, zero-shot accuracy goes from 25.5% to 31.6% for CoMM and from 37.1% to 39.1% for CoMM-VL. CoMM without masking nevertheless remains far below CoMM-VL (31.6% vs. 39.1%). We keep text masking for CoMM in the paper, as in its original recipe, where it also gives its best multimodal probe (69.6).

Fusion depth. On CC12M, the ablation of the paper (Table 4) already varies the fusion depth, but we agree that its presentation could be improved, as it does not make this comparison clear. We will update its layout in the revised version of the paper, and restate it below, with pooled tokens and text masking for both models:

CC12M, 50k stepsFusion layersZSi2tMM-LP
CoMM133.633.764.1
CoMM235.133.763.1
CoMM-VL141.850.164.1
CoMM-VL241.551.063.7

At equal depth, CoMM-VL is ahead of CoMM by 8.2 and 6.4 zero-shot points, while a second layer changes either model by at most 1.5 points. We further ran this ablation on LAION-COCO for 20k steps, each model in its configuration of the main table:

LAION-COCO, 20k stepsFusion layersZSi2tMM-LP
CoMM125.519.669.6
CoMM226.721.469.6
CoMM-VL140.048.468.6
CoMM-VL241.749.769.1

There too, the second layer adds little: 1.2 zero-shot points for CoMM and 1.7 for CoMM-VL. The same holds when CoMM-VL is trained for the full 100k steps: the 2-layer model reported in the paper reaches 59.0% zero-shot accuracy, against 58.1% with a single layer. In the main results, we keep the single fusion layer of the original CoMM recipe for CoMM, and use two layers for CoMM-VL, where the second layer brings a small additional gain.

Taken together, these ablations show that the gains of CoMM-VL over CoMM come primarily from the alignment loss; we will present them clearly in the revised paper, in the ablation table of the main text and in the appendix.

Q3. Matched readouts at large scale.

We provide matched-readout results for CoMM and CoMM-VL on LAION-COCO at 100k steps, for both the multimodal probe and the ImageNet linear probe.

Multimodal probe. We compare the "concat" readout (the concatenation of the image and text encoder outputs taken before the fusion module) with the "fused" readout (the 512-d output of the fusion module).

Multimodal probeConcatFused
CoMM67.267.5
CoMM-VL64.067.0

CoMM-VL benefits from the fused readout (+3.0 points), since its concatenated readout goes through the backbone CLS token, which is not trained when unpooled tokens are fused. CoMM performs similarly with both readouts (67.2 and 67.5). Whatever the readout of CoMM, CoMM-VL stays within 0.5 points of it, while outperforming it by 18.8 points in zero-shot accuracy on LAION-COCO.

Linear probe. We compare the native CLS token of the backbone, the fused image representation, and the mean-pooled patch tokens of the backbone, a native readout that does not depend on the CLS token.

ImageNet linear probeNative CLSMean-pooled patchesFused
CLIP75.068.8--
CoMM73.766.569.1
CoMM-VL56.669.275.7

With matched readouts, CoMM-VL is ahead of CoMM on the fused representation (75.7% vs. 69.1%) and on the mean-pooled patch tokens (69.2% vs. 66.5%), while CoMM is ahead on the native CLS token (73.7% vs. 56.6%). The fused readout of CoMM-VL gives the best accuracy overall, above the native CLS token of CoMM and of CLIP (75.0%), although it is lower-dimensional (512-d vs. 768-d). This is expected, since CoMM-VL is optimized in the fused space. Its low native-CLS accuracy only reflects that this token is not trained when unpooled tokens are fused: the mean-pooled patch tokens, which do not depend on it, show that its backbone features are not degraded (69.2%, also above CLIP at 68.8%).

We will clarify the readout used for each model in the paper, and report these ablations in the appendix.

Q4. Synthetic captions (Sec. 5.1) or original alt-text (Table 5).

Thank you for raising this. Appendix Table 5 is correct: all models are trained on the original web alt-text of LAION-COCO, not on its synthetic captions. We will correct the sentence in Sec. 5.1.

Reviewer_e8J8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary. This paper discussed an issue about alignment-based VLMs. These type of models assume task-relevant information is shared across image and text. In reality it is fragile under misaligned web pairs, blind to complementary (unique/synergistic) information. This paper proposes CoMM-VL, a CoMM loss plus symmetric InfoNCE (CLIP-style) between masked-modality representations in the fused space, at negligible extra cost.

Contributions: (1) Controlled misalignment study: CC12M split into Low/Medium/High tertiles (3M pairs each, Perception Encoder G14 cosine) — CoMM far more robust than CLIP/SLIP/TIPS as alignment drops (CLIP −26% ImageNet linear probe vs CoMM on LowAlign). (2) Diagnosis: CoMM strong on visual/multimodal probes but collapses on cross-modal tasks at LAION-COCO scale (zero-shot 40.2, i2t/t2i 39.6/25.1 vs CLIP's 57.1, 63.9/46.1). (3) CoMM-VL closes it: CC12M 33.6/33.7/24.0 → 41.8/50.1/36.5 (best in table) while matching CoMM's multimodal probe and topping linear probe; at 402M-pair scale best on all three zero-shot suites + linear probe, parity with TIPS on retrieval; POPE/LLM-bridge gains (+14.4 accuracy over CLIP with Llama).

Validation. All six baselines are retrained from scratch. Ablations: alignment-loss placement, fusion depth, parameter-matched deeper-CLIP control (+10.2M params vs fusion's 7.0M) — CoMM-VL still wins all seven metrics.

Critical strengths

Controlled comparison is rare in VLM papers. It uses the same data, architecture and optimizer. The pair count is fixed at 3M so alignment quality is the only variable.

Critical weaknesses

A headline claim on misalignment robustness is never directly tested. The split experiments evaluate CoMM only. The explicit alignment loss could reintroduce the sensitivity CoMM avoided; "preserving robustness" rests on inference.

Questions for authors

N/A

2 comments

We thank the reviewer for their careful and constructive feedback, and for acknowledging the value of our controlled comparison, where all models share the same data, architecture and optimizer. Please find below our response to your concern. We hope it adequately addresses the point you raised and will encourage you to reconsider your rating.

W1. Robustness of CoMM-VL to misalignment is not directly tested.

Thank you for pointing this out. We agree that the paper did not test this claim directly enough: CoMM-VL does appear on the three alignment splits in Figure 1 (left), but only on some tasks. We now report the full results of CoMM-VL trained on the LowAlign, MediumAlign and HighAlign splits of CC12M (tables below).

ZSLowMed.HighLPLowMed.HighMM-LPLowMed.High
CLIP8.614.214.0CLIP39.946.248.9CLIP56.157.658.8
CoMM18.325.417.3CoMM66.168.966.9CoMM59.661.561.0
CoMM-VL17.425.323.0CoMM-VL60.868.869.1CoMM-VL59.861.862.7
i2t R@1LowMed.Hight2i R@1LowMed.High
CLIP5.112.220.1CLIP3.78.413.2
CoMM8.817.619.9CoMM6.512.212.4
CoMM-VL10.421.328.1CoMM-VL7.115.419.7

The alignment loss does not reintroduce CLIP's sensitivity to misalignment. On LowAlign, CoMM-VL keeps CoMM's multimodal probe (59.8 vs. 59.6) and zero-shot accuracy (17.4 vs. 18.3), and improves retrieval (10.4 vs. 8.8 i2t, 7.1 vs. 6.5 t2i), whereas CLIP is far below on all four (56.1, 8.6, 5.1 and 3.7). On MediumAlign and HighAlign, CoMM-VL matches or outperforms CoMM on every metric. On the multimodal probe, this holds under both readouts: CoMM-VL is at or above CoMM on all three splits, whether the fused or the concatenated unimodal representations are probed.

The one exception is the ImageNet linear probe on LowAlign, where CoMM-VL loses 5.3 points with respect to CoMM (60.8 vs. 66.1). It still remains 20.9 points above CLIP (39.9), so most of CoMM's robustness is preserved, but not all of it.

We will add these results and this discussion to Sec. 5: CoMM-VL preserves CoMM's robustness on cross-modal and multimodal tasks, and largely preserves it on the visual linear probe.

Thank you for the additional experiments. These directly addresses my main concern.

I would maintain my rating of 3.

Reviewer_3qrxOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks into the effectiveness of a multimodal contrastive learning approach named CoMM that aligns fused image-text representations instead of the conventional contrastive loss using individual modalities. The authors study the robustness of this approach to image-text misalignment and find that CoMM learns stronger visual and multimodal representations, compared to CLIP, SigLIP, TIPS etc. However, they show that the approach works well for multimodal tasks and linear probing but does not do well on zero-shot classification and retrieval. The authors propose CoMM-VL, to tackle this, which introduces an additional cross-modal alignment objective. The authors show this preserves the strengths of CoMM and becomes better at ZS and retrieval. They further show that CoMM-VL consistently outperforms or matches strong vision-language baselines across diverse downstream tasks, with being the best method under misaligned pairs.

Critical strengths
  1. Studies foundational VLMs under misaligned pairs of data, which is a practical and useful study as it helps developers to develop models where real data pairs are aligned partially.
Critical weaknesses
  1. Showing numbers on fine-grained retrieval on DOCCI and IIW would be a nice addition to see how well this approach works on fine-grained retrieval. Since the method is robust to misalignment, it would be good to have numbers on robust COCO (ROCOCO) retrieval as well.

  2. CoMM-VL has modified both the loss terms and the type of tokens in comparison to the CoMM baseline. Specifically, it uses the extra alignment loss and the fusion input is unpooled tokens when trained on the LAION-COCO dataset. There are also minor changes e.g., in text masking (p=0 vs. 0.15) and attention normalization (Sec. 5.1, Table 5). This makes it impossible to attribute the improvement to the proposed loss. Could you report all four combinations, {CoMM, CoMM-VL} × {pooled, unpooled}, on LAION-COCO with the remaining settings matched? At least CoMM with unpooled tokens can give insights on how much of the gap the architecture closes on its own.

  3. Section 5.1 says the authors use 2-layer transformer for fusion module for CoMM-VL and not for CoMM, which might put CoMM at a disadvantage. Could the authors explain the reason behind this. And what is the reason for using text masking = 0 for LAION-COCO for CoMM-VL?

  4. In appendinx A.1.3, you have written that 
 “The visual probe reads the native CLS token, except for CoMM-VL on LAION-COCO: that model fuses unpooled tokens, which leaves its backbone CLS untrained, so its row is probed in the fused space instead.”

Could you additionally report a linear probe on native backbone features, such as mean-pooled patch tokens?

Questions for authors
  1. In CoMM, the loss term involving z_I^M and z_T^M and the fused representation z is already there, so in figure 4, why is this loss term only shown in CoMM-VL? What I understand is in CoMM-VL, the only extra term is between z_I^M and z_T^M.

  2. CoMM is introduced what it is in the abstract.

  3. Could you make the captions self-contained for tables and figures, briefly mentioning what is the key takeaway, that would greatly enhance the comprehensibility.

3 comments

We thank the reviewer for their detailed and constructive feedback, and for acknowledging the practical relevance of studying vision-language models under image-text misalignment. Please find below our responses to your concerns. We hope they adequately address the points you raised and will encourage you to reconsider your rating.

W1. Fine-grained retrieval (DOCCI, IIW) and robust retrieval (RoCOCO).

Thank you for these suggestions!

Evaluation on DOCCI and IIW. We have now evaluated CoMM-VL and the baselines, trained on CC12M and LAION-COCO, on DOCCI and ImageInWords (IIW). CoMM-VL is best or tied-best at both scales. On CC12M, it is best on all four metrics. On LAION-COCO, it is best or on par on three of the four; on IIW-500 image-to-text, it is only 2 points behind the best model, SigLIP.

ModelDOCCI i2tDOCCI t2iIIW-500 i2tIIW-500 t2i
CC12M, 50k steps
CLIP26.524.952.452.2
SigLIP22.621.250.646.6
SLIP35.134.765.466.2
TIPS38.638.468.670.2
CoMM22.223.151.857.8
CoMM-VL40.740.373.872.0
LAION-COCO, 100k steps
CLIP61.059.090.487.4
SigLIP61.960.092.488.8
SLIP61.959.390.089.6
TIPS63.362.290.888.6
CoMM34.330.472.269.0
CoMM-VL63.362.790.489.8

Evaluation on RoCOCO. We evaluated on RoCOCO following the authors' scripts: the COCO test gallery is extended with altered captions (one noun replaced) for image-to-text retrieval, and with altered images (mixup or patch) for text-to-image retrieval.

ModelCOCO i2tAltered captionsCOCO t2iAltered images
CC12M, 50k steps
CLIP22.915.015.08.9
SigLIP20.913.413.57.8
SLIP32.522.321.313.1
TIPS35.224.323.914.4
CoMM23.115.216.57.7
CoMM-VL37.726.526.216.4
LAION-COCO, 100k steps
CLIP49.636.933.221.2
SigLIP50.735.934.121.8
SLIP50.538.432.822.3
TIPS52.040.135.022.5
CoMM29.121.617.99.5
CoMM-VL52.036.934.823.5

On altered images, CoMM-VL obtains the best R@1 at both scales.

On altered captions, the results are more nuanced. On CC12M, CoMM-VL obtains the best R@1. On LAION-COCO, the models are close: TIPS is best on average, and CoMM-VL remains within 3.2 points of it. Per-variant results will be reported in the appendix.

We will add DOCCI and IIW-500 to the retrieval results of the main paper, and the RoCOCO results and their analysis to a new appendix section.

W2. Loss and fusion input change together on LAION-COCO.

We agree that the original results make it difficult to disentangle the contributions of the different modifications. In the paper, CoMM follows its original recipe (pooled tokens, text masking), while CoMM-VL uses the configuration selected in our ablations. To clarify the impact of each change, the table below reports CoMM and CoMM-VL trained on LAION-COCO for 20k steps with a 1-layer fusion module, varying the fusion input and the text masking one at a time.

MethodFusion inputText maskingZero-shoti2t R@1MM-LP
CoMMpooledp=0.15p=0.1525.519.669.6
CoMMpooledp=0p=031.630.768.4
CoMMunpooledp=0.15p=0.1526.622.568.7
CoMMunpooledp=0p=029.728.468.7
CoMM-VLpooledp=0.15p=0.1537.141.368.8
CoMM-VLpooledp=0p=039.143.168.6
CoMM-VLunpooledp=0.15p=0.1539.546.568.4
CoMM-VLunpooledp=0p=040.048.468.6

CoMM-VL consistently outperforms CoMM with both fusion inputs (+7.5 and +10.3 zero-shot points), while the fusion input changes the zero-shot accuracy of either model by less than 2 points. The performance gains therefore cannot be attributed to unpooled tokens alone. The same holds over the full 100k-step schedule: adding the alignment loss to the unpooled model raises zero-shot accuracy from 43.1% to 58.1%.

Removing text masking improves both models in zero-shot and retrieval, yet CoMM without masking remains far below CoMM-VL (31.6% vs. 39.1% zero-shot). We kept masking for CoMM, as in its original recipe, where it also gives the best multimodal probe (69.6).

Attention normalization is required for CoMM with unpooled tokens, as training otherwise collapses to chance. CoMM-VL does not collapse without it (39.5% zero-shot), but we enable it for all unpooled models so that both methods share the same architecture.

We will add these results and this discussion to the appendix.

W3. 2-layer fusion for CoMM-VL only, and no text masking on LAION-COCO.

Thank you for raising this point. We agree that the paper could better motivate these two choices, which may seem to put CoMM at a disadvantage. They do not explain the gap between the two models: when both are trained with exactly the same settings on LAION-COCO (1-layer fusion, unpooled tokens, no text masking, attention normalization, 20k steps), CoMM reaches 29.7% ImageNet-1k zero-shot accuracy and CoMM-VL 40.0% (see the W2 table above). We discuss each choice below.

Fusion depth. The paper already ablates the fusion depth on CC12M (Table 4): a second fusion layer brings little to CoMM (33.6% to 35.1% zero-shot, with lower linear and multimodal probes), and CoMM-VL remains far ahead of CoMM at equal fusion depth (41.5% vs. 35.1% with two layers). We agree that this table is hard to read in its current form, and we will restructure it with explicit columns for the fusion depth and the placement of the alignment loss.

We further ran this ablation on LAION-COCO for 20k steps (table below): a second layer adds only 1.2 zero-shot points to CoMM and 1.7 to CoMM-VL, far less than the gain brought by the alignment loss.

MethodFusion layersZero-shoti2t R@1MM-LP
CoMM125.519.669.6
CoMM226.721.469.6
CoMM-VL140.048.468.6
CoMM-VL241.749.769.1

We therefore use two layers for CoMM-VL, where the second layer brings a small additional gain, and keep the single layer of the original CoMM recipe for CoMM.

Text masking. Masking is part of CoMM's original recipe, and we use it (p=0.15p=0.15) by default in all experiments. On LAION-COCO, the W2 table above shows that it lowers CoMM-VL's zero-shot accuracy and retrieval (37.1% vs. 39.1% and 41.3 vs. 43.1 R@1 with pooled tokens), while its multimodal probe stays similar (68.8 vs. 68.6), so we disabled it for our final LAION-COCO model. We did not repeat this ablation on CC12M, where CoMM-VL keeps the default setting.

In both ablations, the same changes are applied to CoMM, and CoMM-VL stays well ahead in every matched configuration. We will motivate both choices in Sec. 5.1.

W4. Linear probe on native backbone features for CoMM-VL on LAION-COCO.

Thank you for the suggestion. We ran the linear probe on the mean-pooled patch tokens of the backbone (table below). CoMM-VL reaches 69.2%, above both CoMM (66.5%) and CLIP (68.8%), which shows that its backbone features are not degraded. Its low native-CLS accuracy (56.6%) only reflects that this token is not trained when unpooled tokens are fused.

ModelNative CLSMean-pooled patchesFused zIMz_I^M
CLIP75.068.8--
CoMM73.766.569.1
CoMM-VL56.669.275.7

The table also reports the native and fused readouts of both models. The paper uses the better readout of each (in bold): the native CLS token for CoMM and the fused CLS token for CoMM-VL. CoMM-VL's fused readout outperforms CoMM's native one, although it is lower-dimensional (512-d vs. 768-d), and it is expected that its best features lie in the fused space, where the model is optimized.

We will add this table and discussion to the appendix.

Q1. Loss terms shown in Figure 4.

Thank you for this careful reading; you are right. The loss term between zIMz_I^M, zTMz_T^M and the fused representation zz is already part of CoMM, and the only new term in CoMM-VL is the alignment between zIMz_I^M and zTMz_T^M. The CoMM panel of Figure 4 is reproduced from the original CoMM illustrations, which do not draw this term explicitly. To avoid this confusion, we will clarify the caption of Figure 4: CoMM-VL keeps all CoMM loss terms and only adds the cross-modal alignment between zIMz_I^M and zTMz_T^M.

Q2. Introducing CoMM in the abstract.

Thank you for raising this. We will revise the abstract to introduce CoMM before assessing it.

Q3. Self-contained captions.

Thank you for the suggestion. We will revise the captions of all tables and figures of the main paper so that each is self-contained and states its key takeaway.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their constructive feedback, which has helped us strengthen the paper. The reviewers valued the practical relevance of studying vision-language models under image-text misalignment (R-3qrx), the controlled comparison in which all models are retrained under the same conditions (R-e8J8), and the gains of CoMM-VL on cross-modal tasks at no cost for its visual and multimodal representations (R-3qrx, R-e8J8, R-4o71). The main concerns have been addressed through additional experiments, clarifications, and planned updates for the revised version.

  • Contribution of the alignment loss: We ablated the fusion input, the text masking and the fusion depth on LAION-COCO and CC12M. In every matched configuration, CoMM-VL remains far ahead of CoMM (e.g., 58.1% vs. 43.1% zero-shot accuracy on LAION-COCO with identical settings), which shows that the gains come primarily from the alignment loss. These ablations will be added to the appendix, and the ablation table of the main paper (Table 4) will be restructured for clarity.
  • Robustness of CoMM-VL to misalignment: We trained and evaluated CoMM-VL on the three alignment splits of CC12M. It preserves CoMM's robustness on cross-modal and multimodal tasks, and largely preserves it on the visual linear probe. These results will be added to Section 5.
  • Validity of the alignment splits: We agree that factors other than correspondence quality may confound the analysis on the splits. We computed several statistics for each split and found mostly small differences, but we cannot exclude differences in content. We therefore designed a controlled experiment in which 50% of the captions of the HighAlign split are shuffled, so that only the image-text correspondence changes: it reproduces the differences between methods observed on the splits. Both analyses will be added to the appendix, and the splits will be described as based on image-text similarity, used as a proxy for correspondence quality.
  • Matched readouts: We evaluated CoMM and CoMM-VL under matched readouts on LAION-COCO, for both the linear and the multimodal probes. The fused readout is always the best one for CoMM-VL, whereas the best readout of CoMM depends on the task: for the linear probe, it is the native CLS token and not the fused representation. These results will be added to the appendix, and the readout used for each model will be stated more clearly.
  • Additional retrieval benchmarks: We extended the evaluation to long-caption retrieval (DOCCI, IIW-500) and robust retrieval (RoCOCO), where CoMM-VL is the best or among the best models at both scales. DOCCI and IIW-500 will be added to the retrieval results of the main paper, and RoCOCO to the appendix.
  • Clarifications: We will report the settings of CoMM and CoMM-VL separately (Table 5) and motivate them in Section 5.1, introduce CoMM in the abstract, clarify the caption of Figure 4, and make the captions of all tables and figures self-contained. We also confirm that all models are trained on the original alt-text of LAION-COCO.

We sincerely appreciate the reviewers' constructive feedback, which has greatly improved the clarity, scope and completeness of this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.