- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper propose a multimodal dataset distillation method. The method is a prototype guided data synthesis method, which address the existing prototype data synthesis methods' bottleneck: image and prototypes are built independently. By unifying pair prototype with RCCA, WP and unified prototype discovery, the synthesized data could keep the correspondence of the original image and text pairs.
Critical strengths
- The paper is well-motivated. The bottleneck in existing work lies in the prototype selection, where image and text prototypes are selected independently.
- The paper smartly leverages RCCA, and WP to get the image-text pair feature which can be used as prototypes. The technique of this method sounds.
Critical weaknesses
- PDS and UPSyn both are prototype guided data synthesis method with pretrained unLIP, but in Table 5 there is a huge gap between two methods. Could authors explain the reason of this?
- In the original dataset like Flickr or COCO, each image is paired with 5 captions. How does the method process with image-text pairs? For example, does the method randomly select one caption or use other strategies?
Questions for authors
In table 4, it shows best-k is dependent on model architectures rather than dataset itself. Also, performance doesn't increase along with k increase. Could author provide any explanation on these?
3 comments
Thank authors for the effort during rebuttal. While some limitations remain, the empirical improvements and the clarification of the work are sufficient to justify my original assessment. Therefore, I maintain my score of 3 (Weak Accept).
Response to Reviewer J6Pu
Thank you for your thoughtful feedback and continued engagement throughout the discussion. We sincerely appreciate your time and consideration.
Response to Reviewer J6Pu
We sincerely appreciate your valuable and insightful suggestions as well as your high recognition of our work. It is an honor to respond to your concerns below.
(W1) Computational efficiency gap between PDS and UPSyn.
UPSyn differs from PDS primarily in prototype construction and the number of image generation steps, which are the main reasons for the difference in computational cost.
(1) Prototype construction. PDS clusters images and texts separately and then performs one-to-one prototype matching using the Hungarian algorithm. UPSyn directly constructs shared cross-modal prototypes through the RCCA and WP spaces and the consensus mechanism, reducing the need for explicit matching after independent clustering. Table 5 reports a preprocessing time of 620.8 s for PDS and 567.1 s for UPSyn (417.1 s without RCR).
(2) Image generation. The PDS results retain the 100-step diffusion generation setting from its original protocol, whereas UPSyn uses only 20 steps, as stated in lines 339–340 of the experimental details. This results in more efficient sampling.
Despite using fewer generation steps, UPSyn outperforms PDS in retrieval performance on both ResNet and ViT in Table 1, demonstrating a favorable trade-off between distillation quality and computational efficiency.
(W2) How does the method process with five captions per image?
We apologize for the lack of clarity. For each image in Flickr30K and MS-COCO, we pair each of its five captions with the corresponding image to form a separate image–text pair. All pairs participate in cross-modal prototype discovery; we do not randomly select a single caption. For example, the Flickr30K training set contains 29,000 images, each associated with 5 captions, yielding 29,000 × 5 = 145,000 image–text pairs. Clustering is performed at the image–text pair level. We have clarified this data processing procedure in lines 322–323 of the revised manuscript to remove ambiguity.
(Q1) Optimal Top-K and non-monotonic performance.
Table 4 shows that ResNet performs best at K=10, whereas ViT performs best at K=5. This may reflect differences in how visual backbones respond to semantic detail and redundancy in the synthetic data. Increasing K introduces more complementary information but may also include less relevant or conflicting captions, affecting LLM aggregation and subsequent image generation. Top-K therefore reflects a trade-off between semantic coverage and prototype consistency; larger values are not necessarily better.
To further examine this, we compare Nearest Top-5 (Ours) with three additional strategies on Flickr30K (100 pairs, K=5), using ViT for downstream evaluation: random selection within the cluster (Random-Cluster), random selection from the local Top-50 captions (Random-Local), and MMR (λ=0.7) [1], which balances relevance and diversity. Each strategy selects 5 captions.
The results show that prioritizing semantic relevance between captions and prototypes is more beneficial for distillation than pursuing diversity alone. Although random selection within a cluster increases caption diversity, it may also introduce semantic ambiguity or incompatible information, affecting the accuracy of LLM aggregation and subsequent image generation and thereby reducing distillation quality. Thus, the benefits of RCR depend on appropriate candidate selection as well as multi-caption aggregation. Consequently, a larger K does not necessarily yield better results: as more captions are included, the likelihood of introducing weakly relevant or conflicting semantics increases, which can offset the gains from additional semantic coverage.
[1] The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries. SIGIR 1998.