acceptodds
Under review as a conference paper at ICLR 2027

CM: Cross-Modal Mode Learning for Generative Multimodal Dataset Distillation

Abstract

Multimodal dataset distillation (MDD) compresses a large image-text corpus into a small synthetic set on which a vision-language model can be trained. Generative MDD methods cluster each modality separately and then pair image and text clusters by a post-hoc assignment; nothing in two independent clusterings forces cluster to denote the same concept in both modalities, and the non-differentiable matching cannot correct this. We present CM, which makes cross-modal correspondence part of mode discovery rather than a step after it. CM learns image and text modes, one pair per slot of the distillation budget, in a frozen vision-language embedding space. Its core is cross-modal swapped prediction: the image of a pair must predict the mode assignment of its caption and vice versa, with assignments balanced over the slots, so that index comes to denote one concept in both modalities. A mode-level contrastive term aligns same-index modes and keeps the two banks distinct, and a mode-anchoring term ties each mode to the raw-scale centroid of the samples it owns, so the learned modes are decoded directly and are themselves the distilled set. CM requires no offline clustering, no post-hoc matching and no generator fine-tuning. It raises cross-modal index agreement by 30-85% over post-hoc matching and reduces mismatched image-caption pairs from 7.2% to 0.5% on Flickr30K at the largest budget. Across two benchmarks, four budgets and two evaluation backbones unseen during distillation, it improves mean retrieval recall over the strongest generative MDD baseline in all 16 settings, by +3.6 on Flickr30K and +2.61 on MS-COCO on average.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Dataset Distillation, Vision-Language Models, Cross-Modal Alignment, Dataset Condensation, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_f9a1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

this paper is working on multimodal dataset distillation. Instead of cluster different modality separately and then pair image and text clusters by a post-hoc assignment, the paper proposes a framework CM2 to make cross-modal correspondence part of mode discovery rather than a step after. the core idea of the work is cross-modal swapped prediction: the image of a pair must predict the mode assignment of its caption and vice versa, with assignments balanced over the K slots, so that index k comes to denote one concept in both modalities. The paper introduces mode anchoring, a scale-free term that regresses each mode onto the rawscale soft centroid of the samples the transport plan assigns to. And tested on 2 benchmarks.

Critical strengths

multimodal dataset distillation is an important research problem. This paper uses cross modal swapped assignments to jointly optimize both modalities. The paper has proven that cosine based optimization doesn't constrain the magnitude or position of learned modes in the original embedding space.

Critical weaknesses

each module of this idea isn't novel, for example the cross modality learning, the contrastive learning and swapped assignment are not novel, . It's like a combination of difference existing approaches. The method relies on a retrained CLIP encoder and a retrained decoder. Have you tried other encoders?

Questions for authors

I think the goal is to demonstrate the distilled dataset can be used to train multi-modal and achieve competitive performance on downstream tasks without significantly degrading performance compared to training on the full dataset. From this point of view, I think the experiments are insufficient. but The experiments are only tested on image text retrieval on Flickr30K and MS-COCO. It's not clear if the distilled datasets generalizable enough to other vision language tasks or model architectures. I suggest either test on more datasets like CC3M, DOCCI, SugarCrepe or train more model architectures like SigLiP 2, BLIP-2.

2 comments
AuthorsOpenReview ↗

Response to Reviewer f9a1

Thank you for the review and helpful suggestions. We address your questions about the generalizability of the distilled data and clarify the contributions of CM² below.

Question: Do the distilled sets generalize to other datasets, tasks and architectures?

Short answer: yes for datasets and architectures. CM² keeps its lead in all three new tests: across datasets, on SigLIP 2 and CLIP image towers, and with a trained image backbone. Other tasks are covered below.

Cross-dataset transfer. We trained on one dataset's distilled set, selected the checkpoint on that same dataset's validation split, and tested once on the other dataset (NF-ResNet50, K = 500, 5 seeds).

Flickr30K set → MS-COCO testMS-COCO set → Flickr30K test
PDS12.72±0.1527.04±0.16
CM²15.67±0.1531.84±0.16

CM² leads PDS by 2.95 points when transferring from Flickr30K to MS-COCO and by 4.80 points in the reverse direction.

Other architectures. We used a frozen SigLIP 2 ViT-B/16 vision tower, as you suggested, and the CLIP ViT-L/14 image tower (Flickr30K, checkpoint selected on validation, 5 seeds):

Image towerKRandom real pairsPDSCM²
SigLIP 2 ViT-B/1610039.9954.3162.59
SigLIP 2 ViT-B/1650071.2876.4179.47
CLIP ViT-L/1410030.2843.8752.53
CLIP ViT-L/1450063.3569.3173.19

Training the backbone. We also trained the NF-ResNet50 image tower jointly with the text projection, instead of keeping it frozen. Its learning rate is selected on validation from {0, 1e-4, 1e-3, 1e-2}, with the same grid for both methods, over 3 seeds.

PDS, frozenPDS, trainableCM², frozenCM², trainableMargin, trainable
Flickr30K K=10024.70±0.2225.54±0.0630.43±0.1131.78±0.24+6.24
Flickr30K K=50040.89±0.2441.90±0.2745.19±0.4746.04±0.40+4.14
MS-COCO K=50019.82±0.0521.02±0.1923.69±0.1526.48±0.11+5.46

CM² leads PDS in all three settings with a trainable image tower.

What we did not run. We could not distill CC3M within the response period. DOCCI's descriptions are long paragraphs that exceed the 77-token context of the CLIP text encoder in which our text modes are defined, so it would need a different text encoder; we will consider it in our future work. In this protocol, SugarCrepe would mostly measure the frozen CLIP text encoder rather than the distilled set. BLIP-2 has no dual-encoder retrieval head that could be trained on K pairs.

Weakness: "Have you tried other encoders?"

Short answer: yes. Sec. 4.5 and Table 3 change the generator, and also the encoder, without changing the objective or any hyperparameter (Flickr30K, K = 100).

With different decoder which is Karlo and accepts the same CLIP ViT-L/14 embeddings, we reuse the learned mode bank without re-optimizing or rescaling it. CM² outperforms PDS by 4.44 mean recall points on NF-ResNet50 and 2.94 points on ViT-T/16, as shown in Table 3. In the Karlo ablation, removing anchoring reduces mean recall on NF-ResNet50 from 29.61 to 1.60 (Table 11). We also changed both the encoder and the generator, learning modes with OpenCLIP ViT-H/14 and rendering them with Stable Diffusion 1.5 via IP-Adapter. In this setting, CM² scores 32.35 vs. 25.30 for PDS on NF-ResNet50 and 20.93 vs. 17.16 on ViT-T/16 (Table 3).

Weakness: Novelty of the individual modules

CM² combines established tools in a distinct distillation formulation: it learns two index-aligned mode banks that together form the distilled dataset.

A different formulation. Swapped assignment and contrastive learning are established tools, typically used to train an encoder, after which the prototypes are discarded. In CM², the encoders are frozen, and the two mode banks are the only learned parameters. The banks form the distilled dataset, with the number of modes equal to the budget. Pairing is built into the objective: swapped prediction makes index k represent the same concept in both banks, whereas PDS matches the modalities after the fact. At K = 1000, mismatched pairs drop from 7.2% to 0.5% (Flickr30K, Table 6).

The components work together. Removing only the contrastive term from the full method drops mean recall from 45.24 to 18.26 (Flickr30K, K = 500, NF-ResNet50, averaged over three seeds; details in our response to Reviewer uShW). Without this term, anchored modes drift together. Joint k-means over the same features also trails CM² in all eight tested settings.

A component with no counterpart in the methods discussed above. Those prototype-learning methods do not render their prototypes as images. Mode anchoring enables decoding with a different generator: under Karlo, mean recall is 29.61 with anchoring and 1.60 without it (Table 11).

Reviewer_f9a1OpenReview ↗

feedback

Thank you for the detailed responses and additional experiments. While my concern regarding methodological novelty remains. I will maintain my original score.

Reviewer_uShWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CM², a generative multimodal dataset distillation framework that jointly optimizes image and text prototypes through cross-modal swapped prediction, mode-level contrastive learning, and mode anchoring. Experiments on Flickr30K and MS-COCO demonstrate consistent improvements over existing multimodal dataset distillation methods across different distillation budgets and downstream backbones.

Critical strengths
  1. The research problem is practically relevant: The paper discusses the limitations of existing generative multimodal dataset distillation methods, including insufficient cross-modal interaction in prototype-based approaches and the lack of explicit distillation budget constraints in diffusion fine-tuning approaches. The motivation is clear and addresses a meaningful research problem.

  2. The method offers potential computational efficiency advantages: CM² optimizes prototypes in the embedding space of a pretrained vision-language model, without requiring offline clustering, post-hoc matching, or generator fine-tuning. This design may help reduce computational overhead.

Critical weaknesses
  1. The methodological novelty and necessity of the core components are not sufficiently clear: The proposed method largely builds on existing cross-modal prototype learning techniques, but its key differences from prior work are not clearly explained. Moreover, the ablation results show that some components provide only marginal improvements, making it difficult to assess their individual contributions and necessity.

  2. The paper lacks clarity in presentation and rigor in experimental evaluation: The method description and figures are not sufficiently clear, making the relationships between different modules difficult to understand. In addition, there are concerns regarding the selection of baseline methods and the evaluation protocol, which may affect the reliability of the experimental conclusions.

Questions for authors
  1. Jointly learning image-text prototypes and using cross-modal consistency to guide clustering have been explored in previous studies. This paper mainly applies existing swapped prediction and contrastive learning techniques to multimodal dataset distillation. The authors should further clarify the key differences between CM² and existing cross-modal prototype learning methods.

  2. The experimental analysis mainly focuses on comparisons with PDS. Although the authors explain the differences in evaluation protocols across existing methods, I would still suggest including comparisons with other recent generative multimodal dataset distillation methods under a unified evaluation protocol to better demonstrate the advantages of the proposed method.

  3. The evaluation protocol selects the best checkpoint based on test-set performance and then reports the results on the same test set. This practice is methodologically questionable and may lead to overly optimistic performance estimates.

  4. The ablation study in Table 2 is not clearly organized, making it difficult to identify the individual contributions of different components. Moreover, some components provide only marginal improvements. For example, on MS-COCO with ViT-T, the full method outperforms Swap Only by just 0.02 points, which does not provide convincing evidence of the additional contributions of these components.

2 comments
AuthorsOpenReview ↗

Response to Reviewer uShW (part 1)

Thank you for taking the time to review our paper and provide helpful feedback. We answer your four questions in order and cover the two weaknesses within them.

Question 1 (and Weakness 1): How CM² differs from cross-modal prototype learning

CM² learns two index-aligned mode banks that form the distilled dataset, so image–text pairing is learned with the modes rather than repaired afterward. Swapped prediction and contrastive learning are established tools and we can later add related work on SwAV and cross-modal prototype learning. CM² differs in three ways.

1. The prototypes are the output. In the prototype-learning methods discussed here, prototypes train an encoder and are then discarded. In CM², the encoders are frozen, and the two mode banks are the only learned parameters. Together, they form the distilled dataset, with K modes per modality.

2. Pairing is learned, not matched afterward. Each slot forms an image–text pair, so index k must represent the same concept in both banks. PDS establishes this correspondence through post-hoc matching. CM²’s swapped-prediction objective learns it directly: index agreement is 0.47–0.74 for CM² versus 0.27–0.57 for PDS (Tables 6–7), and at K = 1000, mismatched pairs drop from 7.2% to 0.5%. The contrastive term is also important: without �0��0�, mean recall drops from 45.24 to 18.26 on Flickr30K with NF-ResNet50 at K = 500, averaged over three seeds (Question 4). Joint k-means over the same features trails CM² in all eight tested settings, by 1.22–5.67 points.

3. The modes are decodable. The prototype-learning methods discussed above do not render their prototypes as images. Mode anchoring supports decoding with a different generator: with Karlo, mean recall is 29.61 with anchoring and 1.60 without it (Table 11).

Question 2 (and Weakness 2): Comparison with other generative MDD methods

Short answer: we could not place EDGE or Jeong et al. (2026) under our protocol, because both evaluate with a BERT text encoder that cannot read the CLIP-space text our method distills. We did check that our comparison with PDS does not depend on this one protocol.

Table 1 includes every method that has been evaluated under the shared protocol of Choi et al. (2026): TESLA-VL, LoRS and PDS. EDGE and Jeong et al. (2026) report results with an NFNet image encoder and a frozen BERT text encoder. CM² and PDS store the text side of each pair as a CLIP text vector, so under a BERT-based protocol we would have to replace it with a retrieved caption, and the evaluation would no longer measure the distilled text. Evaluating them inside our protocol would require their distilled sets. EDGE's public code does not yet include its training configurations, fine-tuned diffusion checkpoints or distilled sets (its roadmap lists these as forthcoming), and a reimplementation without them risks understating the method, so we did not include it.

On the reliability of the protocol itself (Weakness 2), we repeated the PDS comparison under three harder settings, and CM² stays ahead in all of them. With a trainable NF-ResNet50 backbone the margin is +6.24 / +4.14 / +5.46 (Flickr30K K = 100, Flickr30K K = 500, MS-COCO K = 500). With a frozen SigLIP 2 tower that played no part in distillation, CM² reaches 79.47 against 76.41 (Flickr30K, K = 500). When trained on one dataset and tested on the other, it reaches 15.67 against 12.72 (Flickr30K → MS-COCO) and 31.84 against 27.04 (MS-COCO → Flickr30K). Question 3 below addresses checkpoint selection.

Question 3 (and Weakness 2): Checkpoint selection on the test set

Short answer: We re-ran the PDS and CM² comparisons from Table 1 and Appendix D using validation-based checkpoint selection. CM² still leads in all 16 settings.

We agree with this concern. The practice comes from the released PDS evaluation loop, which we reused for comparability and disclosed in Sec. 4.1 and Appendix C. Below, the checkpoint is selected on the Karpathy validation split and test is scored once, with the same distilled sets and 5 downstream seeds.

Validation-selected, K = 100 / 300 / 500 / 1000PDSCM²
Flickr30K, NF-ResNet5024.52 / 37.28 / 41.02 / 44.5130.73 / 41.09 / 45.59 / 49.44
Flickr30K, ViT-T/1617.48 / 24.19 / 28.20 / 31.3119.97 / 27.54 / 30.94 / 33.21
MS-COCO, NF-ResNet5010.43 / 17.67 / 19.74 / 22.6713.62 / 20.92 / 23.84 / 26.47
MS-COCO, ViT-T/168.21 / 12.16 / 13.74 / 16.1410.41 / 14.43 / 15.82 / 17.70

Averaged over the eight settings per dataset, CM² leads PDS by 3.75 points on Flickr30K and 2.81 points on MS-COCO. If needed, in the revised paper, we can report the validation-selected results for PDS and CM² and clearly label the evaluation protocol for the previously published baselines.

AuthorsOpenReview ↗

Response to Reviewer uShW (part 2)

Response to Reviewer uShW (part 2)

Question 4 (and Weakness 1): Organization of Table 2, and the size of the component gains

Short answer: the components are not additive. The rung that was missing from Table 2, the full method without L_CM, drops from 45.24 to 18.26 on Flickr30K with NF-ResNet50, averaged over three seeds, so L_CM is essential once anchoring is used.

You are right that some single cells of Table 2 are small, including the MS-COCO ViT-T/16 cell you point to. Table 2 lacked the rung that removes only L_CM from the full method, so we ran it over three distillation seeds (K = 500, NF-ResNet50 / ViT-T/16):

Flickr30Kseed 42seed 1seed 2
Swap only43.31 / 29.90
Swap + L_anc (no L_CM)17.91 / 14.1418.65 / 14.7618.21 / 14.24
Full (Swap + L_CM + L_anc)45.61 / 30.4145.11 / 31.1145.00 / 30.29
MS-COCOseed 42seed 1seed 2
Swap only20.98 / 15.86
Swap + L_anc (no L_CM)13.50 / 11.0313.86 / 11.2513.55 / 10.89
Full (Swap + L_CM + L_anc)23.80 / 15.8823.46 / 16.4022.83 / 15.77

Averaged over three seeds, removing L_CM lowers mean recall by 26.98 points on Flickr30K and 9.73 on MS-COCO with NF-ResNet50, and by 16.22 and 4.96 points, respectively, with ViT-T/16. The bank diagnostics show why. Anchoring pulls each mode toward the average of the data it owns, and without the negatives of L_CM the modes drift together. On Flickr30K, only 37% of the image modes remain in use (all of them for the full method), cross-modal index agreement falls from 0.47 to 0.08, and the mean cosine between image modes rises from 0.10 to 0.69. L_CM keeps the modes distinct and paired, and anchoring makes them decodable. At seed 42, the full method improves over Swap Only with NF-ResNet50 by 2.30 points on Flickr30K and 2.82 points on MS-COCO. For anchoring itself, Table 9 shows a gain in 15 of 16 paired comparisons and in 8 of 8 independent repeats. We will reorganize Table 2 as a grid with one column per component and add these diagnostics.

Weakness 2: Clarity of presentation

If needed, we can redraw Fig. 2 so that the two banks, the swapped codes and the anchoring target appear as separate steps. We can also add an algorithm box, and restructure Table 2 as described above.

Reviewer_g5YfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CM² (Cross-Modal Mode Learning), a generative approach to multimodal dataset distillation (MDD). Existing generative MDD methods independently cluster image and text representations and subsequently establish their correspondence through post-hoc matching, potentially losing cross-modal dependencies. To address this limitation, CM² jointly learns image and text modes in a frozen CLIP embedding space. Specifically, cross-modal swapped prediction encourages consistent mode assignments, while a mode-level contrastive loss aligns corresponding modes. Additionally, mode anchoring regularizes the learned modes toward the original embedding distribution, enabling direct image synthesis without post-processing. Experiments on Flickr30K and MS-COCO demonstrate consistent performance improvements.

Critical strengths
  1. Clear motivation: It is well-motivated and directly addresses the limitations of independently clustering image and text representations followed by post-hoc matching. The proposed framework provides a reasonable approach to preserving cross-modal correspondence during mode learning.

  2. Comprehensive experiments: CM² consistently outperforms existing generative MDD baselines across different datasets, distillation budgets, and evaluation backbones. The ablation studies also demonstrate the effectiveness of the proposed components.

Critical weaknesses
  1. Limited evaluation of dataset distillation beyond feature alignment: My primary concern is whether the evaluation protocol adequately assesses the effectiveness of the proposed method as a dataset distillation approach. As I understand it, both the image and text encoders are frozen, and only a linear text projection layer is trained. Under this setting, the downstream task primarily involves aligning pretrained representations rather than learning new representations from the distilled dataset. This raises the question of whether generating synthetic training data is necessary, or whether directly estimating an effective projection matrix could be a simpler alternative. Since the primary goal of dataset distillation is to enable effective model training with a compact dataset, the current evaluation may not sufficiently demonstrate the usefulness of CM² for training or fine-tuning representation models.

  2. Lack of comparison with simple joint clustering approaches: A straightforward alternative to independently clustering image and text embeddings would be to concatenate the paired representations and perform joint k-means clustering. This would naturally incorporate cross-modal information while avoiding post-hoc matching. I believe joint k-means should be evaluated under the same experimental protocol to clarify whether the improvements arise from the proposed optimization objectives or simply from incorporating cross-modal information during clustering.

  3. Insufficient justification for balanced codes as pseudo-labels: The swapped prediction objective uses Sinkhorn-based balanced codes as pseudo-labels. However, uniformly distributing assignment mass across modes does not necessarily guarantee that the assignments accurately capture the underlying semantic structure. Since these pseudo-labels directly supervise mode learning, further justification or empirical analysis of their quality and robustness would strengthen the paper.

Questions for authors
  1. Sensitivity to pseudo-label construction: This question may be related to weakness 3. Have the authors considered alternative methods for constructing pseudo-labels without Sinkhorn-based balanced assignments? Additionally, how sensitive is the performance to the entropy regularization parameter (ϵ\epsilon) in Eq. (3)?

  2. Evaluation protocol for LoRS in Table 1: Is the image encoder also frozen when evaluating LoRS in Table 1? If so, this setting may not fully reflect the benefits of LoRS, which was originally designed to support training of the image encoder. Could the authors clarify whether LoRS was re-evaluated under the same protocol and discuss the fairness of this comparison?

  3. Evaluation using the CLIP image encoder: What happens when the downstream image encoder is the same CLIP image encoder used during distillation? In particular, how does the performance compare with the original pretrained CLIP model without additional training (zero-shot) ?

  4. Evaluation with trainable encoders: Could the authors evaluate CM² and compare with baselines in the setting where the image or text encoders are trainable rather than frozen? This would more directly assess the effectiveness of the distilled dataset for representation learning and help determine whether CM² maintains its advantages over existing distillation methods in a more general training setting. This question may be related to weakness 1.

3 comments
AuthorsOpenReview ↗

Response to Reviewer g5Yf (Part 1)

Thank you for the careful review. Your main concern is whether a frozen-backbone evaluation shows that the distilled data is useful for training. We ran new experiments for each of your questions. The answers follow your order, and the related weaknesses are addressed inside them; the joint k-means comparison (Weakness 2) comes last.

Question 1 (and Weakness 3): Pseudo-labels without Sinkhorn, and sensitivity to ε

Short answer: without balancing, the mode bank collapses and recall drops by about 10 points. ε is stable between 0.02 and 0.05, and our default of 0.05 lies in that range; at ε = 0.1 the codes become too flat and training fails.

We replaced the Sinkhorn codes with plain softmax codes (same ε, no balancing) and varied ε, keeping everything else fixed (Flickr30K, K = 500, one distillation seed). "Slots used" is the fraction of the K modes that are the nearest mode of at least one training pair. "Small slots" are modes holding less than half the average number of pairs.

Pseudo-labelsNF-RN50ViT-T/16Slots usedSmall slots
Softmax codes, no balancing34.4121.020.6900.998
Sinkhorn, ε = 0.0243.8630.441.0000.192
Sinkhorn, ε = 0.05 (ours)45.6130.411.0000.026
Sinkhorn, ε = 0.110.968.490.6560.894

Without balancing, 31% of slots are unused and 99.8% hold fewer than half the average number of pairs. Mean recall falls to 34.41 / 21.02, a drop of 11.20 / 9.39 points relative to ε = 0.05. Increasing ε from 0.02 to 0.05 raises mean recall by 1.75 points on NF-RN50 and lowers it by 0.03 points on ViT-T/16. At ε = 0.1 the codes no longer separate the modes, and the bank collapses much as it does without balancing. We used the SwAV default of 0.05 throughout without tuning it, and we will add this table and the working range to the paper.

On Weakness 3: balance alone does not guarantee that the codes are semantic, so we measured this directly. Of the caption pairs that describe the same image, 53.2% fall in the same CM² text mode, against 35.5% for PDS and 37.4% for joint k-means (Flickr30K, K = 500). Equipartition suits distillation for a simple reason: each mode becomes one distilled pair and the downstream model weights all pairs equally. Equal mass per mode therefore makes every pair stand for the same share of the corpus. Our optimization is full-batch, so the constraint holds over the whole corpus, not over a mini-batch.

Question 2: Evaluation protocol for LoRS in Table 1

Short answer: yes, the image encoder is frozen for LoRS as well. Those numbers are taken from Choi et al. (2026), and we agree that this setting does not show LoRS at its best.

The LoRS and TESLA-VL entries (†) are reported by Choi et al. (2026, Table 1); we did not re-run them. As described there (Sec. 4.2), both baselines are distilled with a pretrained NFNet image encoder following Xu et al. (2024), with the text encoder replaced by the CLIP text encoder, which improves both (their Table 12). They are then evaluated on frozen ResNet and ViT backbones. LoRS's own cross-architecture study shows the same sensitivity: at 499 pairs, its mean recall falls from 31.58 on NFNet to 15.65 on ResNet (Xu et al., 2024, Table 4). We will state this in the caption of Table 1. For our own method, the fairness question is addressed by the trainable-backbone experiment under Question 4.

Question 3: The CLIP image encoder downstream, and zero-shot CLIP

Short answer: With the CLIP image encoder, CM² leads PDS by 8.66 points at K = 100 and 3.88 points at K = 500. Zero-shot CLIP (87.56) is far above every distilled set.

Flickr30K, frozen image tower, trained text projection, checkpoint selected on validation, 5 seeds:

Image towerKRandom real pairsPDSCM²
CLIP ViT-L/1410030.2843.8752.53
CLIP ViT-L/1450063.3569.3173.19
SigLIP 2 ViT-B/1610039.9954.3162.59
SigLIP 2 ViT-B/1650071.2876.4179.47
Zero-shot CLIP ViT-L/14 (no training)–––87.56

Zero-shot CLIP uses its own text tower, which was aligned with the image tower on 400M pairs. Here, the text projection starts from random initialization and sees only K pairs, so it cannot recover that alignment. The CLIP row is also partly circular, since both methods find their modes in this encoder. We therefore added SigLIP 2, which played no part in distillation; its margin (+8.28 at K = 100, +3.06 at K = 500) is close to the one with CLIP.

AuthorsOpenReview ↗

Response to Reviewer g5Yf (Part 2)

Question 4 (and Weakness 1): Trainable encoders and the need for synthetic data

Short answer. With a trainable image backbone, CM² leads PDS by 6.24, 4.14, and 5.46 points across the three settings below. Synthetic pairs outperform their nearest real pairs in every setting.

Trainable backbone. We trained NF-ResNet50 jointly with the text projection. The CLIP text encoder stayed frozen because the distilled text is stored as CLIP vectors. For both methods, we selected the backbone learning rate on validation from {0, 1e-4, 1e-3, 1e-2} and averaged results over 3 seeds.

PDS, frozenPDS, trainableCM², frozenCM², trainableMargin, trainable
Flickr30K K=10024.70±0.2225.54±0.0630.43±0.1131.78±0.24+6.24
Flickr30K K=50040.89±0.2441.90±0.2745.19±0.4746.04±0.40+4.14
MS-COCO K=50019.82±0.0521.02±0.1923.69±0.1526.48±0.11+5.46

CM² leads PDS in all three settings with either a frozen or trainable image tower. The NFNet + BERT protocol of LoRS and EDGE would require substituting retrieved captions for the distilled text because BERT cannot read CLIP-space vectors.

Is synthesis needed? We used Table 1’s frozen-backbone setup with validation-based checkpoint selection and varied only the training set. “Random real” uses K randomly sampled training pairs (3 draws). “Nearest real” preserves each method’s mode selection by replacing each distilled pair with the real training pair closest to its two modes in CLIP space.

Flickr30KKRandom realPDS nearest realPDSCM² nearest realCM²
NF-ResNet5010015.1118.6224.5922.3230.51
NF-ResNet5050031.8735.9140.7937.1745.40
ViT-T/161009.4512.2617.5015.1820.17
ViT-T/1650021.1625.7227.9427.9030.96
MS-COCOKRandom realPDS nearest realPDSCM² nearest realCM²
NF-ResNet501006.318.6710.449.8713.65
NF-ResNet5050015.1718.2419.7319.1623.65
ViT-T/161004.706.348.197.4310.40
ViT-T/1650010.7813.3413.8713.9115.86

CM²’s nearest-real subsets outperform PDS’s in all 8 settings, by 0.57–3.70 points. For the same modes, synthetic pairs outperform nearest real pairs in all 16 comparisons, with a maximum gain of 8.23 points for CM² on Flickr30K (NF-ResNet50, K = 500). Random subsets score lower throughout. For Flickr30K at K = 100, Choi et al. (2026, Table 2) report 12.52 for k-center and 15.10 for herding.

Fitting the projection directly requires the full corpus for each backbone. We can reuse the same distilled set across NF-ResNet50 and ViT-T/16 (Table 1) and across SigLIP 2 and CLIP ViT-L/14 (Question 3). Storing images as pixels gives all these backbones a common input.

Weakness 2: Comparison with joint k-means

Short answer. Joint k-means performs close to PDS, and CM² beats it in all 8 settings.

In Choi et al. (2026, Table 18), joint clustering beats separate clustering in one of four ResNet comparisons. The joint vs. separate scores are 25.27 vs. 24.73 and 37.18 vs. 37.57 on Flickr30K, and 11.28 vs. 11.67 and 17.08 vs. 18.00 on MS-COCO (K = 100 and 300, respectively). CM² scores 30.97, 41.14, 13.67, and 21.00 in the same order. In our implementation, we concatenate unit-normalized image and text embeddings, cluster them with k-means, and split each centroid into image and text modes at the original scale. We keep PDS’s filtering, decoder, and decoder seed fixed.

Mean recall (NF-RN50 / ViT-T)F30K K=100F30K K=500COCO K=100COCO K=500
PDS (separate k-means + Hungarian)24.82 / 17.6140.91 / 28.0010.50 / 8.2519.75 / 13.79
Joint k-means25.30 / 15.9741.19 / 28.4411.25 / 8.4820.49 / 14.66
CM²30.97 / 20.0745.61 / 30.4113.67 / 10.4623.80 / 15.88

On Flickr30K at K = 500, cross-modal index agreement rises from 0.295 for PDS to 0.381 for joint k-means and 0.527 for CM². Joint k-means also yields more similar image modes (mean cosine 0.73–0.84 vs. 0.07–0.21 for CM²). Without a balance constraint, 7–13% of joint clusters hold less than half the average mass, compared with 1–3% for CM². We will add the joint k-means row to Table 2 and the diagnostics to Tables 6–7.

Thank you for the additional clarifications and experiments. However, I still have concerns regarding the evaluation framework and the effectiveness of CM² for dataset distillation.

First, I do not fully understand the statement: "The NFNet + BERT protocol of LoRS and EDGE would require substituting retrieved captions for the distilled text because BERT cannot read CLIP-space vectors." LoRS [1] has already evaluated a trainable NFNet image encoder with a frozen CLIP text encoder in Appendix D.2 (Table 9). On Flickr30K with (K=100), MTT-VL and LoRS achieve mean recall scores of 47.6 and 54.9, respectively. These results are significantly higher than those reported for CM² with a trainable image backbone (31.78). This means that CM² is less effective for learning better representations and is only beneficial for aligning already well-trained representations using a small amount of data.

However, the main motivation for dataset distillation is to reduce the huge computational cost of model training, whereas learning only a linear projection between frozen representations is a relatively inexpensive and simple task. Therefore, I am not fully convinced of the significance of this work. I think showing improvements over zero-shot CLIP may provide stronger evidence of its practical value.

Thank you for the authors' responses, which have addressed my other concerns to some extent. However, since the issue I consider most critical remains unresolved, I have decided to maintain my original score.

[1] Xu, Yue, et al. "Low-rank similarity mining for multimodal dataset distillation." ICML 2024.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion deadline (November 18, AoE) approaches, please review the authors’ rebuttal if you have not already done so and post a follow-up confirming that you have read it and indicating whether your main concerns have been addressed.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.