- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the relationship between image–text correspondence quality, pretraining scale, and vision–language representations. It proposes CoMM-VL, which adds an alignment loss between image-only and text-only representations in CoMM’s fusion space to improve cross-modal matching while retaining multimodal representations. Evaluation uses CC12M and LAION-COCO pretraining and covers classification, retrieval, multimodal probing, and transfer to frozen language models. The central result is a substantial improvement in cross-modal performance over CoMM, supported by CC12M ablations and competitive large-scale results.
Critical strengths
- A useful empirical contribution with substantial gains. On CC12M, CoMM-VL improves CoMM’s ImageNet zero-shot accuracy from 33.6% to 41.8% and mean image-to-text retrieval from 33.7% to 50.1%. This addresses a concrete limitation of CoMM, rather than relying on marginal gains on a single benchmark. The breadth of classification, retrieval, multimodal probing, and frozen-LLM transfer is the main reason I lean toward acceptance.
- Relevant ablations support the design at medium scale. Table 4 tests fusion depth and alignment location. Increasing fusion depth alone reaches 35.1% zero-shot accuracy, compared with 41.8% for CoMM-VL, while CoMM and CoMM-VL both reach 64.1% with the fused MM readout. Together with the parameter-count control, this provides useful evidence that the modification improves cross-modal performance while retaining useful fused representations. The method is simple, but the empirical trade-off is informative.
Critical weaknesses
- Correspondence quality is not fully separated from other data properties. The three equal-sized subsets are selected by pretrained-model similarity. This controls sample count but may also change category, source, image quality, or caption properties; low similarity need not mean incorrect correspondence. The observations on those subsets remain useful, but attributing the differences specifically to correspondence quality requires validation or narrower wording. This limits the strength of the paper’s explanatory claim and keeps my recommendation at weak acceptance.
- Large-scale objective and representation attribution remain incomplete. The CC12M controls are valuable, but LAION-COCO also involves changes to fusion depth, token pooling, and text masking, with masking disabled for CoMM-VL. In addition, multimodal probes compare concatenated native features with CoMM-VL’s fused features, and large-scale visual probes use different readout locations. These comparisons can assess complete systems, but do not fully isolate the new loss or native-encoder quality. A matched large-scale control and clearer interpretation would materially strengthen the paper.
Questions for authors
- What validates the similarity splits as levels of actual correspondence quality? Please provide key distribution statistics and an independently checked sample, or explicitly restrict the conclusions to the similarity-based subsets.
- Please list the actual fusion depth, pooling, and masking settings for CoMM and CoMM-VL at each scale. Is there a LAION-COCO control that holds the CoMM-VL configuration fixed and removes only the added alignment loss?
- Can you provide matched-readout CoMM and CoMM-VL results at large scale, or clearly distinguish complete-system claims from claims about native encoders? The existing CC12M concat/fused controls already help at that scale.
- Section 5.1 describes synthetic COCO-style captions for LAION-COCO, while Appendix Table 5 says original web alt-text. Which field was used, and was it the same across methods? A table typo would be readily fixable; different training fields could materially affect the comparison.
3 comments
W2. Large-scale attribution of the loss and of the representation.
We agree that the LAION-COCO results compare complete configurations: in the paper, CoMM follows its original recipe (pooled tokens, 1-layer fusion, text masking), while CoMM-VL uses the configuration selected in our ablations. We address the two parts of this concern below: a matched control for the loss (Q2) and matched readouts (Q3).
Q2. Settings per method and scale, and a LAION-COCO control without the alignment loss.
The settings of the CoMM and CoMM-VL models of the main results table are:
| Setting | CoMM (CC12M) | CoMM-VL (CC12M) | CoMM (LAION-COCO) | CoMM-VL (LAION-COCO) |
|---|---|---|---|---|
| Fusion input | pooled | pooled | pooled | unpooled |
| Fusion layers | 1 | 2 | 1 | 2 |
| Text masking |
We will separate the two methods in Table 5 of the appendix accordingly. To isolate the alignment loss from these settings, we ablate them below.
Fusion input and text masking. We trained CoMM and CoMM-VL on LAION-COCO for 20k steps with a 1-layer fusion module, varying the fusion input and the text masking one at a time.
| Method | Fusion input | Text masking | ZS | i2t | MM-LP |
|---|---|---|---|---|---|
| CoMM | pooled | 25.5 | 19.6 | 69.6 | |
| CoMM | pooled | 31.6 | 30.7 | 68.4 | |
| CoMM | unpooled | 26.6 | 22.5 | 68.7 | |
| CoMM | unpooled | 29.7 | 28.4 | 68.7 | |
| CoMM-VL | pooled | 37.1 | 41.3 | 68.8 | |
| CoMM-VL | pooled | 39.1 | 43.1 | 68.6 | |
| CoMM-VL | unpooled | 39.5 | 46.5 | 68.4 | |
| CoMM-VL | unpooled | 40.0 | 48.4 | 68.6 |
In a matched configuration (unpooled tokens, ), adding the alignment loss raises zero-shot accuracy from 29.7% (CoMM) to 40.0% (CoMM-VL), and image-to-text R@1 from 28.4% to 48.4%. This gap persists over the full 100k-step schedule: trained in the same configuration, CoMM reaches 43.1% zero-shot accuracy, slightly above the 40.2% of the configuration reported in the paper, but still far below CoMM-VL (58.1%).
Finally, removing text masking improves both models in zero-shot and retrieval: with pooled tokens, zero-shot accuracy goes from 25.5% to 31.6% for CoMM and from 37.1% to 39.1% for CoMM-VL. CoMM without masking nevertheless remains far below CoMM-VL (31.6% vs. 39.1%). We keep text masking for CoMM in the paper, as in its original recipe, where it also gives its best multimodal probe (69.6).
Fusion depth. On CC12M, the ablation of the paper (Table 4) already varies the fusion depth, but we agree that its presentation could be improved, as it does not make this comparison clear. We will update its layout in the revised version of the paper, and restate it below, with pooled tokens and text masking for both models:
| CC12M, 50k steps | Fusion layers | ZS | i2t | MM-LP |
|---|---|---|---|---|
| CoMM | 1 | 33.6 | 33.7 | 64.1 |
| CoMM | 2 | 35.1 | 33.7 | 63.1 |
| CoMM-VL | 1 | 41.8 | 50.1 | 64.1 |
| CoMM-VL | 2 | 41.5 | 51.0 | 63.7 |
At equal depth, CoMM-VL is ahead of CoMM by 8.2 and 6.4 zero-shot points, while a second layer changes either model by at most 1.5 points. We further ran this ablation on LAION-COCO for 20k steps, each model in its configuration of the main table:
| LAION-COCO, 20k steps | Fusion layers | ZS | i2t | MM-LP |
|---|---|---|---|---|
| CoMM | 1 | 25.5 | 19.6 | 69.6 |
| CoMM | 2 | 26.7 | 21.4 | 69.6 |
| CoMM-VL | 1 | 40.0 | 48.4 | 68.6 |
| CoMM-VL | 2 | 41.7 | 49.7 | 69.1 |
There too, the second layer adds little: 1.2 zero-shot points for CoMM and 1.7 for CoMM-VL. The same holds when CoMM-VL is trained for the full 100k steps: the 2-layer model reported in the paper reaches 59.0% zero-shot accuracy, against 58.1% with a single layer. In the main results, we keep the single fusion layer of the original CoMM recipe for CoMM, and use two layers for CoMM-VL, where the second layer brings a small additional gain.
Taken together, these ablations show that the gains of CoMM-VL over CoMM come primarily from the alignment loss; we will present them clearly in the revised paper, in the ablation table of the main text and in the appendix.
Q3. Matched readouts at large scale.
We provide matched-readout results for CoMM and CoMM-VL on LAION-COCO at 100k steps, for both the multimodal probe and the ImageNet linear probe.
Multimodal probe. We compare the "concat" readout (the concatenation of the image and text encoder outputs taken before the fusion module) with the "fused" readout (the 512-d output of the fusion module).
| Multimodal probe | Concat | Fused |
|---|---|---|
| CoMM | 67.2 | 67.5 |
| CoMM-VL | 64.0 | 67.0 |
CoMM-VL benefits from the fused readout (+3.0 points), since its concatenated readout goes through the backbone CLS token, which is not trained when unpooled tokens are fused. CoMM performs similarly with both readouts (67.2 and 67.5). Whatever the readout of CoMM, CoMM-VL stays within 0.5 points of it, while outperforming it by 18.8 points in zero-shot accuracy on LAION-COCO.
Linear probe. We compare the native CLS token of the backbone, the fused image representation, and the mean-pooled patch tokens of the backbone, a native readout that does not depend on the CLS token.
| ImageNet linear probe | Native CLS | Mean-pooled patches | Fused |
|---|---|---|---|
| CLIP | 75.0 | 68.8 | -- |
| CoMM | 73.7 | 66.5 | 69.1 |
| CoMM-VL | 56.6 | 69.2 | 75.7 |
With matched readouts, CoMM-VL is ahead of CoMM on the fused representation (75.7% vs. 69.1%) and on the mean-pooled patch tokens (69.2% vs. 66.5%), while CoMM is ahead on the native CLS token (73.7% vs. 56.6%). The fused readout of CoMM-VL gives the best accuracy overall, above the native CLS token of CoMM and of CLIP (75.0%), although it is lower-dimensional (512-d vs. 768-d). This is expected, since CoMM-VL is optimized in the fused space. Its low native-CLS accuracy only reflects that this token is not trained when unpooled tokens are fused: the mean-pooled patch tokens, which do not depend on it, show that its backbone features are not degraded (69.2%, also above CLIP at 68.8%).
We will clarify the readout used for each model in the paper, and report these ablations in the appendix.
Q4. Synthetic captions (Sec. 5.1) or original alt-text (Table 5).
Thank you for raising this. Appendix Table 5 is correct: all models are trained on the original web alt-text of LAION-COCO, not on its synthetic captions. We will correct the sentence in Sec. 5.1.
We thank the reviewer for their detailed and constructive feedback, and for acknowledging the usefulness of our empirical contribution, the breadth of the evaluation and the relevance of the ablations. Please find below our responses to your concerns. We hope they adequately address the points you raised and will encourage you to reconsider your rating.
W1 and Q1. Similarity splits and correspondence quality.
We agree that selecting subsets by the similarity of a pretrained model controls the number of samples, but that other factors may still vary across subsets. We address this in two ways: statistics of the splits, and a controlled experiment in which only the correspondence changes.
Statistics of the splits. We computed distribution statistics of the three splits on 3,657 pairs from three CC12M shards. More precisely, we measured the image-text similarity under PE-Core and under a second model (CLIP ViT-L/14), the caption length and style (captions with more than one sentence, or with a stock-image keyword such as "stock photo" or "vector illustration"), the image size, the aesthetic score (LAION aesthetic predictor; higher is better), and the source website of each image.
The second model agrees with the splits: about two thirds of the LowAlign and HighAlign pairs fall in the same third of its own similarity ranking, and only 7% of the LowAlign pairs fall in its top third. The splits differ only slightly in caption length, image size and aesthetic score. Caption style differs more: HighAlign captions more often span several sentences and more often carry the wording of stock-image websites. Their two most frequent sources are the same, but the splits are not identical in content: the largest differences are on two smaller sources, with more presentation slides in HighAlign (Slideplayer) and more hotel listings in LowAlign (Booking.com). Splits based on similarity therefore cannot fully separate correspondence from other properties of the data.
Controlled experiment. To address this concern, we designed an experiment in which only the correspondence changes: on the HighAlign split, we shuffle a fraction p of the captions among samples.
The three methods separate clearly as soon as captions are shuffled, as on the similarity splits. On the visual and multimodal probes, CLIP degrades sharply (34.7% then 28.7% linear probe), while CoMM and CoMM-VL remain far above it: CoMM-VL is on par with CoMM at p=25% (62.3% vs. 62.5% linear probe, 59.8 vs. 60.0 MM-LP) and within 3.0 and 2.3 points of it at p=50%. On cross-modal tasks, CoMM-VL is the most robust of the three at both levels: it reaches 14.7% zero-shot accuracy at p=25% (7.9% for CoMM, 4.4% for CLIP) and 7.6% at p=50% (3.0% and 1.9%).
This shows that degrading the correspondence alone, with every other property held fixed, is enough to reproduce the differences between methods observed on the similarity splits. We see the two experiments as complementary: the splits reflect the partial mismatches found in real web data but vary several properties at once, while shuffling isolates the correspondence, at the cost of a more extreme form of misalignment.
We will clarify the paper accordingly: the three subsets will be described as splits by image-text similarity, used as a proxy for correspondence quality, and the conclusion that robustness to imperfect correspondence drives the differences will rest on both the splits and the controlled experiment. We will add both analyses to the appendix.