- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
this paper is working on multimodal dataset distillation. Instead of cluster different modality separately and then pair image and text clusters by a post-hoc assignment, the paper proposes a framework CM2 to make cross-modal correspondence part of mode discovery rather than a step after. the core idea of the work is cross-modal swapped prediction: the image of a pair must predict the mode assignment of its caption and vice versa, with assignments balanced over the K slots, so that index k comes to denote one concept in both modalities. The paper introduces mode anchoring, a scale-free term that regresses each mode onto the rawscale soft centroid of the samples the transport plan assigns to. And tested on 2 benchmarks.
Critical strengths
multimodal dataset distillation is an important research problem. This paper uses cross modal swapped assignments to jointly optimize both modalities. The paper has proven that cosine based optimization doesn't constrain the magnitude or position of learned modes in the original embedding space.
Critical weaknesses
each module of this idea isn't novel, for example the cross modality learning, the contrastive learning and swapped assignment are not novel, . It's like a combination of difference existing approaches. The method relies on a retrained CLIP encoder and a retrained decoder. Have you tried other encoders?
Questions for authors
I think the goal is to demonstrate the distilled dataset can be used to train multi-modal and achieve competitive performance on downstream tasks without significantly degrading performance compared to training on the full dataset. From this point of view, I think the experiments are insufficient. but The experiments are only tested on image text retrieval on Flickr30K and MS-COCO. It's not clear if the distilled datasets generalizable enough to other vision language tasks or model architectures. I suggest either test on more datasets like CC3M, DOCCI, SugarCrepe or train more model architectures like SigLiP 2, BLIP-2.
2 comments
feedback
Thank you for the detailed responses and additional experiments. While my concern regarding methodological novelty remains. I will maintain my original score.
Response to Reviewer f9a1
Thank you for the review and helpful suggestions. We address your questions about the generalizability of the distilled data and clarify the contributions of CM² below.
Question: Do the distilled sets generalize to other datasets, tasks and architectures?
Short answer: yes for datasets and architectures. CM² keeps its lead in all three new tests: across datasets, on SigLIP 2 and CLIP image towers, and with a trained image backbone. Other tasks are covered below.
Cross-dataset transfer. We trained on one dataset's distilled set, selected the checkpoint on that same dataset's validation split, and tested once on the other dataset (NF-ResNet50, K = 500, 5 seeds).
CM² leads PDS by 2.95 points when transferring from Flickr30K to MS-COCO and by 4.80 points in the reverse direction.
Other architectures. We used a frozen SigLIP 2 ViT-B/16 vision tower, as you suggested, and the CLIP ViT-L/14 image tower (Flickr30K, checkpoint selected on validation, 5 seeds):
Training the backbone. We also trained the NF-ResNet50 image tower jointly with the text projection, instead of keeping it frozen. Its learning rate is selected on validation from {0, 1e-4, 1e-3, 1e-2}, with the same grid for both methods, over 3 seeds.
CM² leads PDS in all three settings with a trainable image tower.
What we did not run. We could not distill CC3M within the response period. DOCCI's descriptions are long paragraphs that exceed the 77-token context of the CLIP text encoder in which our text modes are defined, so it would need a different text encoder; we will consider it in our future work. In this protocol, SugarCrepe would mostly measure the frozen CLIP text encoder rather than the distilled set. BLIP-2 has no dual-encoder retrieval head that could be trained on K pairs.
Weakness: "Have you tried other encoders?"
Short answer: yes. Sec. 4.5 and Table 3 change the generator, and also the encoder, without changing the objective or any hyperparameter (Flickr30K, K = 100).
With different decoder which is Karlo and accepts the same CLIP ViT-L/14 embeddings, we reuse the learned mode bank without re-optimizing or rescaling it. CM² outperforms PDS by 4.44 mean recall points on NF-ResNet50 and 2.94 points on ViT-T/16, as shown in Table 3. In the Karlo ablation, removing anchoring reduces mean recall on NF-ResNet50 from 29.61 to 1.60 (Table 11). We also changed both the encoder and the generator, learning modes with OpenCLIP ViT-H/14 and rendering them with Stable Diffusion 1.5 via IP-Adapter. In this setting, CM² scores 32.35 vs. 25.30 for PDS on NF-ResNet50 and 20.93 vs. 17.16 on ViT-T/16 (Table 3).
Weakness: Novelty of the individual modules
CM² combines established tools in a distinct distillation formulation: it learns two index-aligned mode banks that together form the distilled dataset.
A different formulation. Swapped assignment and contrastive learning are established tools, typically used to train an encoder, after which the prototypes are discarded. In CM², the encoders are frozen, and the two mode banks are the only learned parameters. The banks form the distilled dataset, with the number of modes equal to the budget. Pairing is built into the objective: swapped prediction makes index k represent the same concept in both banks, whereas PDS matches the modalities after the fact. At K = 1000, mismatched pairs drop from 7.2% to 0.5% (Flickr30K, Table 6).
The components work together. Removing only the contrastive term from the full method drops mean recall from 45.24 to 18.26 (Flickr30K, K = 500, NF-ResNet50, averaged over three seeds; details in our response to Reviewer uShW). Without this term, anchored modes drift together. Joint k-means over the same features also trails CM² in all eight tested settings.
A component with no counterpart in the methods discussed above. Those prototype-learning methods do not render their prototypes as images. Mode anchoring enables decoding with a different generator: under Karlo, mean recall is 29.61 with anchoring and 1.60 without it (Table 11).