acceptodds
Under review as a conference paper at ICLR 2027

Cross-Modal Correspondence Matters: Unified Prototype-Guided Data Synthesis for Multimodal Dataset Distillation

Abstract

Multimodal dataset distillation aims to construct compact yet informative image–text datasets that preserve essential knowledge from large-scale datasets for efficient cross-modal learning. Recent prototype-guided approaches typically construct visual and textual prototypes separately, primarily preserving modality-specific structures while insufficiently capturing the joint dependency that defines image–text correspondence. In this paper, we analyze this limitation from an information-theoretic perspective and introduce UPSyn, a Unified Prototype-guided Synthesis framework that designed to adequately preserve cross-modal information from the original dataset. Specifically, we construct two cross-modal common spaces from correlation and geometry perspectives, and jointly cluster the paired data by integrating the structural evidence from both spaces, thereby producing informative prototypes that better maintain image–text dependency. Additionally, we leverage a large language model to aggregate and summarize the entities surrounding each textual prototype into a refined prototype, providing semantically rich guidance for subsequent distilled data generation. Extensive experiments on Flickr30K and MS-COCO demonstrate that our method consistently outperforms coreset-based, pixel-optimization-based and generative-based baselines in bidirectional retrieval, while retaining strong cross-architecture generalization with an efficient training-free paradigm.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Multimodal Dataset Distillation, Vision-Language Learning, Cross-Modal Representation Learning, Prototype-Guided Data Synthesis, Canonical Correlation Analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_J6PuOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper propose a multimodal dataset distillation method. The method is a prototype guided data synthesis method, which address the existing prototype data synthesis methods' bottleneck: image and prototypes are built independently. By unifying pair prototype with RCCA, WP and unified prototype discovery, the synthesized data could keep the correspondence of the original image and text pairs.

Critical strengths
  1. The paper is well-motivated. The bottleneck in existing work lies in the prototype selection, where image and text prototypes are selected independently.
  2. The paper smartly leverages RCCA, and WP to get the image-text pair feature which can be used as prototypes. The technique of this method sounds.
Critical weaknesses
  1. PDS and UPSyn both are prototype guided data synthesis method with pretrained unLIP, but in Table 5 there is a huge gap between two methods. Could authors explain the reason of this?
  2. In the original dataset like Flickr or COCO, each image is paired with 5 captions. How does the method process with image-text pairs? For example, does the method randomly select one caption or use other strategies?
Questions for authors

In table 4, it shows best-k is dependent on model architectures rather than dataset itself. Also, performance doesn't increase along with k increase. Could author provide any explanation on these?

3 comments
AuthorsOpenReview ↗

Response to Reviewer J6Pu

We sincerely appreciate your valuable and insightful suggestions as well as your high recognition of our work. It is an honor to respond to your concerns below.

(W1) Computational efficiency gap between PDS and UPSyn.

UPSyn differs from PDS primarily in prototype construction and the number of image generation steps, which are the main reasons for the difference in computational cost.

(1) Prototype construction. PDS clusters images and texts separately and then performs one-to-one prototype matching using the Hungarian algorithm. UPSyn directly constructs shared cross-modal prototypes through the RCCA and WP spaces and the consensus mechanism, reducing the need for explicit matching after independent clustering. Table 5 reports a preprocessing time of 620.8 s for PDS and 567.1 s for UPSyn (417.1 s without RCR).

(2) Image generation. The PDS results retain the 100-step diffusion generation setting from its original protocol, whereas UPSyn uses only 20 steps, as stated in lines 339–340 of the experimental details. This results in more efficient sampling.

Despite using fewer generation steps, UPSyn outperforms PDS in retrieval performance on both ResNet and ViT in Table 1, demonstrating a favorable trade-off between distillation quality and computational efficiency.

(W2) How does the method process with five captions per image?

We apologize for the lack of clarity. For each image in Flickr30K and MS-COCO, we pair each of its five captions with the corresponding image to form a separate image–text pair. All pairs participate in cross-modal prototype discovery; we do not randomly select a single caption. For example, the Flickr30K training set contains 29,000 images, each associated with 5 captions, yielding 29,000 × 5 = 145,000 image–text pairs. Clustering is performed at the image–text pair level. We have clarified this data processing procedure in lines 322–323 of the revised manuscript to remove ambiguity.

(Q1) Optimal Top-KK and non-monotonic performance.

Table 4 shows that ResNet performs best at K=10K=10, whereas ViT performs best at K=5K=5. This may reflect differences in how visual backbones respond to semantic detail and redundancy in the synthetic data. Increasing KK introduces more complementary information but may also include less relevant or conflicting captions, affecting LLM aggregation and subsequent image generation. Top-KK therefore reflects a trade-off between semantic coverage and prototype consistency; larger values are not necessarily better.

To further examine this, we compare Nearest Top-5 (Ours) with three additional strategies on Flickr30K (100 pairs, K=5K=5), using ViT for downstream evaluation: random selection within the cluster (Random-Cluster), random selection from the local Top-50 captions (Random-Local), and MMR (λ=0.7\lambda=0.7) [1], which balances relevance and diversity. Each strategy selects 5 captions.

MethodIR@1 ↑IR@5 ↑IR@10 ↑TR@1 ↑TR@5 ↑TR@10 ↑
Random-Cluster5.6 ± 0.218.5 ± 0.327.4 ± 0.46.0 ± 0.718.3 ± 0.426.6 ± 0.8
Random-Local6.5 ± 0.220.8 ± 0.130.1 ± 0.26.7 ± 0.220.3 ± 0.329.0 ± 0.5
MMR6.5 ± 0.119.4 ± 0.228.4 ± 0.37.0 ± 0.319.3 ± 0.327.4 ± 0.5
Nearest Top-5 (Ours)7.3 ± 0.322.4 ± 0.532.1 ± 0.37.4 ± 0.521.7 ± 0.431.4 ± 0.8

The results show that prioritizing semantic relevance between captions and prototypes is more beneficial for distillation than pursuing diversity alone. Although random selection within a cluster increases caption diversity, it may also introduce semantic ambiguity or incompatible information, affecting the accuracy of LLM aggregation and subsequent image generation and thereby reducing distillation quality. Thus, the benefits of RCR depend on appropriate candidate selection as well as multi-caption aggregation. Consequently, a larger K does not necessarily yield better results: as more captions are included, the likelihood of introducing weakly relevant or conflicting semantics increases, which can offset the gains from additional semantic coverage.

[1] The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries. SIGIR 1998.

Thank authors for the effort during rebuttal. While some limitations remain, the empirical improvements and the clarification of the work are sufficient to justify my original assessment. Therefore, I maintain my score of 3 (Weak Accept).

AuthorsOpenReview ↗

Response to Reviewer J6Pu

Thank you for your thoughtful feedback and continued engagement throughout the discussion. We sincerely appreciate your time and consideration.

Reviewer_hUBVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new prototype-guided data synthesis framework for multimodal dataset distillation. Similar to existing prototype-guided work, the proposed method uses pretrained CLIP embeddings and an unCLIP generator to synthesize compact image-text datasets. However, previous prototype-guided work typically discovers prototypes separately for images and text, whereas the proposed method jointly discovers prototypes by explicitly considering cross-modal correspondence.

Critical strengths
  • The paper presents an interesting method that attempts to explicitly model cross-modal correspondence during prototype discovery through: 1) RCCA to identify correlated directions. 2) WP to align feature geometry. and 3) Consensus clustering to integrate evidence from both spaces.

  • An LLM-based caption refinement module is presented to enrich the textual descriptions associated with text prototypes before passing them to unCLIP to generate synthetic image-text pairs.

  • Experiments have been conducted on Flickr30K and MS-COCO, where promising results have been obtained. The effectiveness of each component is also evaluated through ablation studies.

Critical weaknesses
  • Although the proposed method is training-free and more efficient than existing optimization-based methods such as MDM, it is still necessary to compare UPSyn with MDM in terms of both accuracy and computational efficiency to help readers better understand the trade-offs. MDM is particularly relevant, as both methods focus on image-text retrieval and cross-architecture generalization.

  • The LLM-based caption refinement module has some limitations. Although the ablation study demonstrates that RCR improves retrieval performance, the necessity of using an LLM is not sufficiently justified. The observed improvements may simply result from richer or longer textual descriptions rather than the LLM's ability to aggregate semantic information. Comparisons against simpler caption combination or summarization strategies, together with image-text consistency measurements, would help clarify RCR's contribution. Furthermore, using an external LLM introduces an additional source of variability, which can affect the reproducibility and long-term robustness of UPSyn.

  • The evaluation focuses exclusively on bidirectional image-text retrieval using Flickr30K and MS-COCO. It remains unclear whether the proposed method generalizes to other downstream tasks, such as image classification or segmentation.

  • Some important related work is missing such as: 1) Dataset Distillation via Vision-Language Category Prototype (ICCV 2025). 2) Correspondence Coverage Matters for Multi-Modal Dataset Distillation (AAAI 2026).

Questions for authors
  • Could the authors discuss the limitations and potential failure cases of the proposed method, particularly when image-text pairs are noisy or mismatched, clusters contain semantically diverse samples, or LLM-based caption refinement and image generation introduce inconsistencies between the synthesized images and texts?

  • Since the proposed method identifies representative clusters of paired image-text samples, could the authors clarify whether synthetic generation provides meaningful advantages over directly selecting representative real image-text pairs? Although the paper includes coreset selection baselines, these methods use different selection strategies. A comparison under identical cluster assignments would help isolate and justify the benefit of synthetic image generation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer hUBV (1/4)

We sincerely appreciate your valuable and insightful suggestions. Our responses are provided below.

(W1) Comparisons with MDM in terms of performance and computational efficiency.

To further evaluate UPSyn's performance and efficiency advantages over optimization-based multimodal dataset distillation, we add a comparison with MDM alongside other generative methods. We run MDM on Flickr30K with 100 distilled pairs, using BERT text features and 3,000 distillation iterations, and evaluate downstream performance with ResNet-50.

MethodPreproc. (s) ↓Synth./Distill. (s/pair) ↓IR@1 ↑IR@5 ↑IR@10 ↑TR@1 ↑TR@5 ↑TR@10 ↑
EDGE194,190.011.52.9 ± 0.210.6 ± 0.116.9 ± 0.33.7 ± 0.411.2 ± 0.516.7 ± 0.5
MDM271800.070.23.3 ± 0.212.1 ± 0.419.4 ± 0.45.7 ± 0.519.4 ± 0.227.9 ± 0.5
PDS620.834.27.9 ± 0.325.8 ± 0.437.3 ± 0.310.2 ± 0.328.2 ± 0.939.0 ± 0.3
UPSyn w/o RCR417.17.110.5 ± 0.229.9 ± 0.641.8 ± 0.214.1 ± 0.632.3 ± 0.343.5 ± 0.1
UPSyn w/ RCR567.17.711.3 ± 0.131.1 ± 0.942.2 ± 0.214.9 ± 0.234.4 ± 0.344.0 ± 0.8

UPSyn substantially outperforms MDM across all retrieval metrics, improving IR@1/5/10 by 8.0, 19.0, and 22.8 percentage points and TR@1/5/10 by 9.2, 15.0, and 16.1 percentage points, respectively. In terms of computational efficiency, UPSyn requires only 7.7 s per generated pair, whereas MDM's amortized distillation time is 70.2 s/pair, approximately 9.1 times as long. UPSyn also avoids the additional cost of training expert trajectories, providing a more favorable trade-off between performance and computational efficiency.

(W2) The LLM-based caption refinement module has some limitations.

RCR integrates shared semantics and complementary details from multiple relevant captions into a coherent scene description, rather than merely increasing text length. To assess the need for an LLM, we compare three alternatives: Concat-Full, which directly concatenates the Top-5 captions; Concat-Match, which truncates the concatenation to approximately match the length of the LLM-generated description; and Extractive, which combines complete captions under a length constraint. We also evaluate image–text consistency. These experiments use Flickr30K with 100 distilled pairs and ViT as the downstream evaluation architecture.

MethodIR@1IR@5IR@10TR@1TR@5TR@10Image–text prototype cosine (×100)Image–generated caption cosine (×100)Image → generated caption R@1Generated caption → image R@1
Concat-Full6.5 ± 0.120.9 ± 0.230.3 ± 0.37.7 ± 0.421.6 ± 0.730.5 ± 0.429.827.093.0%95.0%
Concat-Match6.8 ± 0.121.4 ± 0.130.5 ± 0.26.7 ± 0.320.4 ± 0.229.9 ± 0.429.926.592.0%91.0%
Extractive6.7 ± 0.220.8 ± 0.130.1 ± 0.47.1 ± 0.221.3 ± 0.429.4 ± 0.529.927.497.0%97.0%
UPSyn w/ RCR7.3 ± 0.322.4 ± 0.532.1 ± 0.37.4 ± 0.521.7 ± 0.431.4 ± 0.830.128.299.0%96.0%

UPSyn w/ RCR achieves better overall retrieval performance than simple concatenation and extractive strategies. For image–text consistency, RCR achieves the highest image–text prototype similarity (30.1), image–generated caption similarity (28.2), and image-to-caption R@1 (99.0%), indicating that it better preserves the semantic correspondence between generated images and text. These results show that LLM refinement helps improve semantic consistency between generated images and their actual captions, beyond the effect of longer text inputs.

AuthorsOpenReview ↗

Response to Reviewer hUBV (2/4)

(W3) Limited evaluation across downstream tasks.

To assess the transferability of distilled data to other downstream tasks, we add an ImageNet image classification transfer experiment. We train image–text alignment projection layers using 100 distilled image–text pairs generated from Flickr30K by PDS, UPSyn w/o RCR, and full UPSyn, respectively, while freezing the pre-trained NF-ResNet50 and ViT-Ti/16 visual encoders. We then evaluate classification on 500 images from 10 classes in the ImageNet validation set, considering two candidate label spaces: the 10 selected classes and all 1000 classes. Class text representations are generated by the frozen CLIP encoder, and ImageNet labels are used only for evaluation. The results are shown below.

Candidate classesMethodResNet Top-1 ↑ResNet Top-5 ↑ViT Top-1 ↑ViT Top-5 ↑
10-wayPDS54.1 ± 3.194.4 ± 1.559.0 ± 0.694.0 ± 0.8
10-wayUPSyn w/o RCR71.0 ± 1.894.9 ± 0.455.4 ± 0.693.9 ± 1.0
10-wayUPSyn w/ RCR73.5 ± 0.897.0 ± 1.364.8 ± 0.796.4 ± 0.6
1000-wayPDS6.5 ± 1.219.7 ± 1.51.9 ± 0.69.2 ± 1.8
1000-wayUPSyn w/o RCR9.6 ± 1.428.0 ± 1.41.9 ± 0.511.8 ± 1.4
1000-wayUPSyn w/ RCR10.2 ± 1.332.7 ± 1.52.0 ± 1.115.6 ± 1.3

UPSyn w/ RCR outperforms PDS in both the 10-way and 1000-way classification settings. In the 10-way setting, Top-1 accuracy improves by 19.4 and 5.8 percentage points for ResNet and ViT, respectively. In the more challenging 1000-way setting, Top-5 accuracy improves by 13.0 and 6.4 percentage points, respectively. These results indicate that the image–text data distilled by UPSyn can support cross-task transfer to classification.

These findings provide preliminary evidence of the transfer potential of UPSyn-distilled data for image classification, suggesting that the preserved cross-modal correspondence can support generalization across tasks. UPSyn primarily targets image–text retrieval, whereas image segmentation additionally requires pixel-level spatial information and region-level semantic modeling. Future work could use UPSyn's synthetic data for cross-modal pretraining or feature adaptation of visual encoders, together with segmentation heads and pixel-level supervision, to explore dense prediction tasks such as image segmentation.

(W4) Missing relevant studies.

Both studies are closely related to our work. We have added citations to Dataset Distillation via Vision-Language Category Prototype (ICCV 2025) and Correspondence Coverage Matters for Multi-Modal Dataset Distillation (AAAI 2026) in the Related Work section of the revised manuscript, at lines 716 and 731, respectively, to provide a more complete review and discussion of related work.

AuthorsOpenReview ↗

Response to Reviewer hUBV (3/4)

(Q1) Limitations and potential failure cases.

Noisy image–text pairs, within-cluster semantic heterogeneity, and semantic deviations during generation can affect distillation quality. UPSyn can mitigate these issues through low-similarity pair filtering, dual-space consensus clustering, and prototype-guided generation. To further examine its effectiveness and limitations, we add experiments on noise robustness, cluster coherence, and image–text consistency.

(1) Noise and mismatched pairs. In UPSyn's original pipeline, we first use CLIP cosine similarity to filter out 20% of the image–text pairs as potential mismatches, reducing noise. We then construct complementary cross-modal common spaces using RCCA and WP, and apply a consensus penalty to suppress potentially incorrect cluster assignments supported by only one space, improving the robustness of cross-modal prototype discovery. We further inject 10% and 20% image–text mismatch noise into the Flickr30K training set and compare PDS and UPSyn under the same filtering procedure, using ViT as the downstream evaluation architecture.

Injected noiseMethodIR@1 ↑IR@5 ↑IR@10 ↑TR@1 ↑TR@5 ↑TR@10 ↑
0%PDS6.8 ± 0.319.2 ± 0.328.5 ± 0.46.6 ± 0.517.5 ± 0.526.9 ± 0.5
0%UPSyn7.3 ± 0.322.4 ± 0.532.1 ± 0.37.4 ± 0.521.7 ± 0.431.4 ± 0.8
10%PDS6.3 ± 0.118.3 ± 0.127.4 ± 0.26.3 ± 0.417.4 ± 0.424.2 ± 0.6
10%UPSyn7.2 ± 0.221.1 ± 0.230.3 ± 0.26.7 ± 0.318.4 ± 0.626.7 ± 0.7
20%PDS4.7 ± 0.216.6 ± 0.425.4 ± 0.24.8 ± 0.214.0 ± 0.722.2 ± 0.3
20%UPSyn5.8 ± 0.219.0 ± 0.328.6 ± 0.35.4 ± 0.317.1 ± 0.426.0 ± 0.2

UPSyn maintains higher retrieval performance than PDS across all noise conditions. Moreover, UPSyn's pre-filtering reduces the injected noise levels of 10% and 20% to 0.06% and 1.24%, respectively, effectively reducing interference from mismatched samples. This demonstrates the role of pre-filtering in improving data quality and robustness.

(2) Within-cluster semantic heterogeneity. RCR further mitigates the risk of substantial semantic differences among samples within a cluster during image generation. Specifically, we retrieve the Top-KK captions most relevant to the text prototype to reduce interference from irrelevant semantics. We then use the LLM to refine these captions, with prompts that prioritize shared semantics and avoid combining conflicting or unsupported details, producing a more coherent and explicit description for image generation. The additional experiments in (W2) show that LLM-RCR achieves an image–generated caption cosine similarity of 0.282, exceeding direct concatenation (0.270) and extractive combination (0.274), while preserving prototype-level image–text consistency. This indicates that the strategy helps mitigate the effects of semantic heterogeneity on image generation.

(3) Semantic inconsistency in LLM refinement and image generation. RCR aggregates shared semantics from multiple captions in the prototype neighborhood. Carefully designed prompts prioritize shared semantics and avoid combining conflicting information, yielding a more coherent and explicit description for image generation. This description is combined with the image prototype to guide generation and reduce interference from irrelevant captions. We further evaluate the consistency of the generated image–text pairs using frozen CLIP:

MethodImage–text prototype cosine ↑ (×100)Image–generated caption cosine ↑ (×100)
UPSyn w/o RCR29.826.9
UPSyn w/ RCR30.128.2

RCR increases the cosine similarity between images and generated captions from 0.269 to 0.282, while largely preserving prototype-level consistency. LLM-RCR achieves the highest image–generated caption similarity, indicating that semantic refinement helps improve the consistency of the generation conditions. These experiments show that UPSyn's design can mitigate noise, clustering bias, and generation inconsistency to some extent. We will include these analyses in the revised manuscript to clarify the method's scope of applicability.

AuthorsOpenReview ↗

Response to Reviewer hUBV (4/4)

(Q2) Does synthesis offer meaningful advantages over directly selecting representative real image–text pairs?

Our motivation is that a single real image–text pair represents only a local instance within a cluster, whereas image and text prototypes constructed from multiple samples can integrate broader shared semantic information. Prototype-guided synthesis is therefore expected to preserve cluster-level semantics more fully than selecting a single real sample. To examine this, we compare the following three real-sample selection strategies using exactly the same 100 prototype clusters, with ResNet-50 for downstream evaluation:

  1. Cluster Medoid: Select the most representative real image–text pair within each cluster, i.e., the sample closest to the cluster center, and retain its original image and corresponding caption.

  2. Caption Top-1: Select the real caption with the highest CLIP similarity to the text prototype and pair it with its corresponding original image.

  3. Real Top-1 + Prototype Text: Select the real image closest to the image prototype and pair it with the cluster's aggregated CLIP text prototype instead of its original caption.

MethodIR@1 ↑IR@5 ↑IR@10 ↑TR@1 ↑TR@5 ↑TR@10 ↑
Cluster Medoid6.34 ± 0.0820.01 ± 0.2230.31 ± 0.437.20 ± 0.2620.12 ± 0.5029.86 ± 0.71
Caption Top-15.85 ± 0.1318.51 ± 0.2529.03 ± 0.177.26 ± 0.2019.90 ± 0.4329.34 ± 0.55
Real Top-1 + Prototype Text6.37 ± 0.1420.95 ± 0.2732.18 ± 0.318.14 ± 0.4222.30 ± 0.4631.42 ± 0.57
UPSyn w/o RCR10.5 ± 0.229.9 ± 0.641.8 ± 0.214.1 ± 0.632.3 ± 0.343.5 ± 0.1
UPSyn w/ RCR11.3 ± 0.131.1 ± 0.942.2 ± 0.214.9 ± 0.234.4 ± 0.344.0 ± 0.8

The synthetic data clearly outperform all real-sample selection strategies in the downstream evaluation. Even without RCR, UPSyn improves over the best real-sample selection strategy. These results show that the gains do not arise solely from selecting representative clusters: prototype-guided synthesis further improves downstream learning from the distilled data, demonstrating the practical value of the synthesis stage.

Reviewer_14Z4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes UPSyn, a generative approach to Multimodal Dataset Distillation (MDD). Existing prototype-based methods independently cluster image and text features and subsequently match their prototypes, potentially failing to preserve the joint dependencies between the two modalities. To address this limitation, the authors construct two cross-modal common representation spaces using Regularized Canonical Correlation Analysis (RCCA) and Whitening–Procrustes (WP), which emphasize cross-modal correlation and geometric structure, respectively. The proposed method then performs joint clustering by integrating similarity information from both spaces through consensus refinement. Additionally, Region-Aware Caption Refinement (RCR) leverages an LLM to aggregate captions surrounding each textual prototype, producing semantically enriched descriptions for synthetic data generation. Experiments on Flickr30K and MS-COCO demonstrate consistent improvements over existing baselines.

Critical strengths
  1. Clear motivation and well-defined problem. The paper is well-motivated and addresses a meaningful limitation of existing prototype-based MDD approaches. In particular, it explicitly considers the joint dependency between image and text representations, which may not be adequately preserved when the two modalities are independently clustered and subsequently matched.

  2. Comprehensive experimental evaluation. The proposed method consistently improves retrieval performance across different datasets, distillation budgets, and evaluation architectures. Furthermore, the ablation studies demonstrate the contributions of individual components, including RCCA, WP, consensus refinement, and RCR, providing empirical support for the effectiveness of the proposed framework.

Critical weaknesses
  1. Insufficient justification for the dual-space design. Although the proposed combination of RCCA and WP demonstrates empirical improvements, the motivation for using these two particular methods is not sufficiently clear. Both methods aim to construct cross-modal common representations, and it remains unclear why combining them is preferable to constructing a single, more effective common representation space. Furthermore, the paper provides limited theoretical justification for why these two spaces should capture complementary information that is particularly beneficial for dataset distillation. A more detailed analysis of their complementary roles and comparisons with alternative common-space construction methods would strengthen the methodological justification.

  2. Limited justification for the caption selection strategy in RCR. RCR selects the Top-KK nearest captions to each textual prototype and aggregates them using an LLM. However, it is unclear why nearest-neighbor retrieval is the most appropriate strategy for selecting captions to be aggregated. Alternative strategies that explicitly consider semantic diversity or coverage might produce more informative descriptions. Even a random caption selection baseline within each cluster could help clarify whether the observed improvements arise from the nearest-neighbor selection strategy or simply from aggregating multiple captions. Additional comparisons would better establish the effectiveness of the proposed caption refinement procedure.

Questions for authors
  1. Necessity of whitening. Both RCCA and WP involve whitening operations. However, whitening alters the geometry of the original CLIP representation space, potentially affecting semantic relationships that may be useful for clustering. Could the authors provide further justification for the necessity of whitening, particularly in WP? An ablation study comparing WP with Procrustes alignment without whitening, or with partial whitening, would be informative.

  2. Evaluation using a CLIP image encoder. The current evaluation uses ResNet-50 and ViT-Ti/16 as image encoders while keeping the text encoder frozen. It would be interesting to evaluate the distilled datasets using a pretrained CLIP image encoder as well. In particular, how does the performance obtained by training or fine-tuning the CLIP image encoder on the distilled dataset compare with that of the original frozen CLIP model without additional training?

  3. Choice of VAE features for D4M and MGD3. In the multimodal adaptations of D4M and MGD3, the authors retain VAE-based image features while introducing CLIP text features. Given that these representations are not explicitly aligned, this design may disadvantage these baselines in cross-modal retrieval tasks. Could the authors clarify why CLIP image features were not considered for these methods? Using CLIP image representations instead of VAE features for D4M and MGD3 would provide a fairer comparison with PDS and UPSyn.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 14Z4 (1/3)

We sincerely thank you for your positive assessment of our work and for your constructive feedback. We have carefully considered your comments and address each of your concerns below.

(W1) Insufficient justification for the dual-space (RCCA and WP) design.

Our design is motivated by two considerations:

  1. Extracting shared information between images and text enables semantically rich and consistent prototypes, as demonstrated by prior research [1] highlighting its importance and effectiveness. We therefore use CCA as the foundation for capturing cross-modal correlation and select RCCA based on the comparison of CCA variants in Appendix Table 11.

  2. Emphasizing cross-modal correlation alone may downweight weakly correlated directions or geometric structure useful for clustering. WP complements this perspective: regularized whitening and orthogonal Procrustes alignment reduce cross-modal covariance differences while preserving geometric relationships in the whitened space without additional weighting by canonical correlations.

The two spaces do not extract independent information; their different directional weighting and geometric constraints provide complementary clustering evidence for prototype discovery.

We also provide experimental evidence:

  1. Figure 5 (left) shows that WP is more favorable for image retrieval, whereas RCCA is more favorable for text retrieval.

  2. Figure 5 (right) reports agreement between their clustering results, with ARI = 0.445 and NMI = 0.695.

  3. We further find that the Top-10 nearest-neighbor overlap between the two spaces is approximately 51%.

  4. Table 3 provides ablations of RCCA alone, WP alone, and their combination.

These results indicate substantial differences in their representations and clustering structures, which provide complementary, rich semantic information for the prototypes and distilled data and improve downstream performance.In highly compressed dataset distillation, the cluster partition directly determines the representativeness and cross-modal consistency of the prototypes used for synthesis. RCCA and WP provide correlation-oriented and geometry-oriented evidence, respectively, while consensus refinement suppresses assignments with insufficient support in an individual space, improving prototype quality. Appendix Table 10 also shows that simply increasing the common-space dimensionality does not consistently improve performance. Our design therefore does not claim that two spaces are theoretically superior to any single space; it achieves more effective prototype compression in the evaluated settings through complementary representations and consensus.

(W2) Limited justification for the caption selection strategy in RCR.

We apologize for the insufficient explanation of RCR. Since UPSyn jointly uses image and text prototypes to guide synthesis, RCR must enrich textual semantics while maintaining cross-modal relevance. As shown in Table 4, too few captions provide limited additional information, whereas excessive aggregation may introduce semantics that deviate from the image prototype.

To validate our nearest-neighbor selection, we compare Nearest Top-5 with Random-Cluster, Random-Local (Top-50), and MMR (λ=0.7\lambda=0.7) [1] on Flickr30K (100 pairs, K=5K=5), using ViT for evaluation.Each strategy selects 5 captions.

MethodIR@1 ↑IR@5 ↑IR@10 ↑TR@1 ↑TR@5 ↑TR@10 ↑
Random-Cluster5.6 ± 0.218.5 ± 0.327.4 ± 0.46.0 ± 0.718.3 ± 0.426.6 ± 0.8
Random-Local6.5 ± 0.220.8 ± 0.130.1 ± 0.26.7 ± 0.220.3 ± 0.329.0 ± 0.5
MMR6.5 ± 0.119.4 ± 0.228.4 ± 0.37.0 ± 0.319.3 ± 0.327.4 ± 0.5
Nearest Top-5 (Ours)7.3 ± 0.322.4 ± 0.532.1 ± 0.37.4 ± 0.521.7 ± 0.431.4 ± 0.8

Nearest Top-5 achieves the best performance across all six metrics, indicating that prioritizing prototype relevance is more effective than introducing additional diversity. We further visualize a representative basketball-themed cluster in Appendix Table 13, where Nearest Top-5 achieves the highest image-prototype cosine similarity both before (29.71) and after LLM aggregation (28.34), supporting its ability to preserve cross-modal semantic consistency in this example.

These quantitative and qualitative analyses have been added to Tables 12–13 of the revised manuscript, further justifying our nearest-neighbor selection strategy.

[1] Image Clustering with External Guidance. ICML 2024.

[2] The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries. SIGIR 1998.

AuthorsOpenReview ↗

Response to Reviewer 14Z4 (2/3)

(Q1) Necessity of whitening.

Whitening is intended to reduce cross-modal covariance and scaling differences before prototype discovery, rather than to preserve the original geometry exactly. Our response to (W1) explains the rationale for RCCA and WP and provides evidence of their complementarity.

Whitening serves different roles in the two spaces. In RCCA, regularized covariance normalization is intrinsic to estimating canonical correlations and identifying correlated cross-modal directions. In WP, whitening balances the contributions of different feature directions, followed by orthogonal Procrustes alignment, thereby providing complementary information from weakly correlated directions or geometric structure useful for clustering.

To further assess the necessity of whitening in WP, we introduce a whitening-strength parameter β\beta, which controls the degree of whitening through eigenvalue scaling λ~−β/2\widetilde{\lambda}^{-\beta/2}, and compare three configurations:

  • β=0\beta=0: no whitening (PCA + Procrustes).
  • β=0.5\beta=0.5: partial whitening.
  • β=1\beta=1: full whitening.

On Flickr30K with 100 distilled pairs, we evaluate WP alone (WP-only) and the complete RCCA+WP+Consensus prototype discovery framework without RCR, using ResNet-50 for downstream evaluation.

Methodβ\betaIR@1 ↑IR@5 ↑IR@10 ↑TR@1 ↑TR@5 ↑TR@10 ↑
WP-only0.08.1 ± 0.225.5 ± 0.337.6 ± 0.410.0 ± 0.228.9 ± 0.739.4 ± 0.4
WP-only0.59.4 ± 0.426.9 ± 0.337.7 ± 0.310.6 ± 0.628.7 ± 0.639.4 ± 1.0
WP-only1.09.8 ± 0.728.9 ± 0.341.0 ± 0.111.6 ± 0.229.3 ± 0.140.4 ± 0.4
UPSyn w/o RCR0.09.8 ± 0.328.8 ± 0.440.8 ± 0.312.5 ± 0.431.6 ± 0.642.6 ± 0.5
UPSyn w/o RCR0.59.5 ± 0.327.8 ± 0.239.8 ± 0.214.0 ± 0.531.7 ± 0.542.0 ± 0.4
UPSyn w/o RCR1.010.5 ± 0.229.9 ± 0.641.8 ± 0.214.1 ± 0.632.3 ± 0.343.5 ± 0.1

These results show that whitening improves cross-modal prototype discovery through covariance normalization. Full whitening achieves the best overall performance among the evaluated configurations. We have added these ablations to Table 9, lines 1016–1025 of the revised manuscript, further clarifying the role and trade-offs of whitening.

(Q2) Evaluation using a CLIP image encoder.

To examine whether the distilled data can effectively adapt pre-trained vision–language models, we conduct an additional experiment with pre-trained CLIP ViT-L/14 on Flickr30K.

Specifically, we fine-tune the pre-trained CLIP image encoder and its original projection layer using only 100 distilled image–text pairs, while freezing the original text encoder and text projection layer.

MethodIR@1 ↑IR@5 ↑IR@10 ↑TR@1 ↑TR@5 ↑TR@10 ↑
Frozen CLIP67.088.893.286.597.999.1
UPSyn FT73.2 ± 0.392.2 ± 0.195.7 ± 0.187.8 ± 0.598.0 ± 0.299.3 ± 0.2

FT denotes fine-tuning. Fine-tuning on UPSyn-distilled data improves IR@1 from 67.0% to 73.2%, a gain of 6.2 percentage points. These results show that UPSyn-distilled data can both train other visual backbones and improve the retrieval performance of pre-trained CLIP. We have added this experiment to Table 13, lines 1101–1106 of the revised manuscript to further examine the effectiveness of the distilled data for adapting pre-trained vision–language models.

AuthorsOpenReview ↗

Response to Reviewer 14Z4 (3/3)

(Q3) Choice of VAE features for D4M and MGD3.

D4M and MGD3 were originally developed for image classification rather than multimodal tasks. Both perform clustering in the VAE latent space: D4M uses the resulting prototypes to initialize diffusion in Stable Diffusion v1.5, while MGD3 uses them to guide reverse diffusion in DiT. Accordingly, our original adaptations retain their VAE encoders and decoders, rather than the CLIP encoder and unCLIP decoder used in UPSyn, and separately incorporate CLIP text features. Directly replacing VAE latents with CLIP features would introduce incompatibility with the generation space and would not preserve the original generation mechanisms.

We further use CLIP image and text features for prototype discovery and matching. CLIP image features are used only to determine initial cluster membership; prototypes are then constructed in the VAE latent space from the assigned members to maintain compatibility with the original diffusion frameworks. All other core generation mechanisms of D4M and MGD3 remain unchanged. These experiments are conducted on Flickr30K (100 pairs), using ResNet-50 for downstream evaluation.

MethodIR@1 ↑IR@5 ↑IR@10 ↑TR@1 ↑TR@5 ↑TR@10 ↑
D4M-VAE1.3 ± 0.15.8 ± 0.29.8 ± 0.31.4 ± 0.25.6 ± 0.59.5 ± 0.3
D4M-CLIP6.1 ± 0.119.7 ± 0.329.9 ± 0.17.7 ± 0.921.9 ± 0.631.0 ± 0.5
MGD3-VAE2.6 ± 0.110.0 ± 0.217.2 ± 0.53.4 ± 0.211.6 ± 0.317.5 ± 0.4
MGD3-CLIP6.0 ± 0.219.5 ± 0.229.8 ± 0.17.9 ± 0.322.8 ± 0.932.4 ± 0.4
PDS7.9 ± 0.325.8 ± 0.437.3 ± 0.310.2 ± 0.328.2 ± 0.939.0 ± 0.3
UPSyn11.3 ± 0.131.1 ± 0.942.2 ± 0.214.9 ± 0.234.4 ± 0.344.0 ± 0.8

The additional experiments show that CLIP image features are more compatible with CLIP text representations than VAE image features, improving the retrieval performance of these baselines. UPSyn retains a clear advantage under the evaluation architectures even against these stronger CLIP-based adaptations. We have added this experiment to Table 14, lines 1141-1147 of the revised manuscript and explicitly distinguish the original baselines from the new adaptations.

Comments on the paper

2 comments

Dear Reviewers,

As the discussion deadline (November 18, AoE) approaches, please review the authors’ rebuttal if you have not already done so and post a follow-up confirming that you have read it and indicating whether your main concerns have been addressed.

Best, AC

AuthorsOpenReview ↗

A respectful request for discussion

Dear Reviewers,

Thank you for your time and valuable feedback. We have carefully addressed your concerns in our responses. We understand that you have a busy schedule. As the discussion period is approaching its end, we would greatly appreciate it if you could review our responses and let us know if any questions remain. We would be happy to provide further clarification.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.