acceptodds
Withdrawn from ICLR 2027

Resolving Semantic Degeneracy in Multimodal Dataset Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion-based methods for dataset distillation (DD) cluster images into prototypes and decode them with diffusion models into a synthetic set, becoming a prominent DD paradigm due to their training-free, architecture-agnostic design. Existing methods transfer this pipeline to multimodal dataset distillation (MDD) on paired image–text data by clustering each modality separately at the target budget and keep the DD decode process unchanged. However, per-modality clustering ignores cross-modal interactions and collapses distinct pairs, budget-sized clustering merges fine-grained semantics into meaningless prototypes, and standard diffusion guidance renders samples from distinct prototypes indistinguishable. We term these failures semantic degeneracy and propose DPRG (Distinction-Preserving Representation and Guidance). On the representation side, we assign each pair jointly and over-cluster before selecting back to the budget, keeping semantically distinct pairs in separate prototypes. On the guidance side, we denoise each target against its hardest image- and text-side alternatives, keeping distinct prototypes apart during synthesis. Extensive experiments on Flickr30K and MS-COCO demonstrate that DPRG outperforms existing baselines across all evaluated distillation budgets and downstream architectures.

Primary area:
generative models
Keywords:
Dataset distillation, Diffusion model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gkSxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DPRG (Distinction-Preserving Representation and Guidance), a multimodal dataset distillation framework that addresses semantic degeneracy in prototype-based diffusion distillation. The method combines joint image-text clustering, over-clustering with diversity-based prototype selection, and hard-negative Triple CFG to improve semantic distinctness during synthetic data generation. Experiments on four datasets demonstrate consistent improvements in cross-modal retrieval across different distillation budgets and downstream architectures.

Critical strengths
  1. Relevant problem formulation. The paper identifies semantic degeneracy as a potential limitation of prototype-based multimodal dataset distillation, where semantically distinct image-text pairs may be merged during clustering or become indistinguishable during generation. This is a meaningful concern for downstream contrastive learning.

  2. Coherent methodological design. DPRG addresses semantic distinctness during both prototype discovery and image generation. Joint pair clustering, over-clustering with coverage selection, and hard-negative Triple CFG form a reasonably well-motivated framework.

  3. Consistent empirical improvements. The proposed method improves mean recall across all 20 dataset-budget-backbone configurations. For example, on Flickr30K with 100 synthetic pairs and NF-ResNet50, DPRG improves mean recall from PDS 24.7% to 30.2%.

  4. Comprehensive component ablations. The experiments evaluate different combinations of the three proposed components and demonstrate their complementary behavior. In particular, over-clustering becomes beneficial when combined with pair coupling, providing useful evidence about the interaction between representation construction and prototype selection.

  5. Computational efficiency. DPRG achieves improved retrieval performance with a relatively small increase in dataset construction cost compared with PDS. This makes the framework practically attractive for settings where repeated multimodal model training is expensive.

  6. Evidence of transferability. The experiments consider NF-ResNet50 and ViT downstream architectures, alongside an alternative SigLIP and FLUX.1-dev configuration. These results provide evidence that the method is not entirely dependent on one specific image encoder and diffusion generator combination.

  7. Additional analysis of semantic distinctness. The retrieval-margin experiments report improvements of approximately 1.6 to 2.5 times over PDS, while the qualitative results illustrate increased diversity among generated image-text pairs. These findings provide supporting evidence for the proposed motivation, although they do not fully validate local semantic coherence.

Critical weaknesses
  1. Insufficient coverage of closely related multimodal dataset distillation methods. The related work does not adequately position DPRG relative to recent advances in joint multimodal representation construction and diversity preservation. In particular, MDM (CVPR 2026) already employs joint image-text embedding clustering for synthetic pair initialization, while EDGE (NeurIPS 2025) explicitly addresses insufficient diversity among generated multimodal samples through a diversity loss. MDM is not properly cited or discussed, and although EDGE appears in the experimental comparisons, its conceptual relationship to DPRG is not sufficiently examined. These omissions weaken the novelty assessment and the positioning of the proposed contributions.

  2. Insufficient differentiation from existing joint clustering and diversity-preserving techniques. Pair coupling and over-clustering are presented as important mechanisms for preserving semantic distinctions, but their relationship to the joint embedding clustering used in MDM is not clearly established. Similarly, Triple CFG aims to prevent semantically indistinguishable generated samples, which is closely related to the diversity preservation problem already investigated by EDGE. Although the optimization mechanisms differ, the paper should explicitly clarify what conceptual and methodological advances DPRG offers beyond these existing approaches.

  3. Unclear generality of semantic degeneracy across MDD paradigms. The paper primarily studies semantic degeneracy in prototype-based diffusion distillation, without establishing whether similar phenomena occur in distribution-matching, kernel-based, or trajectory-matching methods. This is particularly relevant because existing methods such as MDM and EDGE already incorporate mechanisms related to cross-modal alignment and sample diversity. The paper should clarify whether semantic degeneracy is a general limitation of multimodal dataset distillation or a specific failure mode of the prototype-based pipeline.

  4. Limited direct evidence for local semantic coherence. The evaluation primarily relies on retrieval performance, retrieval margins, and qualitative visualizations. These measures do not directly establish whether local semantic neighborhoods or fine-grained cross-modal relationships are preserved. Stronger analyses of neighborhood consistency, semantic cluster preservation, and local alignment would be necessary to substantiate the central claims.

  5. Restricted downstream evaluation. The main experiments assess cross-modal retrieval with frozen encoders and a trained projection head. Although this follows established evaluation protocols, it is insufficient to demonstrate that the distilled datasets retain broader semantic information or support general multimodal representation learning.

  6. Incomplete characterization of backbone dependence. The experiments with SigLIP and FLUX.1-dev demonstrate transfer beyond the default CLIP and unCLIP configuration, but both settings still rely on pretrained aligned image-text representations. It remains unclear whether DPRG's improvements depend primarily on strong pretrained joint embedding spaces or whether the proposed method remains effective with weaker representations.

  7. Potential conflict between diversity and distribution fidelity. Farthest-first prototype selection prioritizes semantic separation, potentially favoring sparse or atypical regions of the embedding space. The paper does not sufficiently examine whether this introduces distributional bias or sacrifices common semantic modes.

Questions for authors
  1. Relationship to previous MDD works. (i) MDM (CVPR 2026) already uses clustering in a joint image-text embedding space for synthetic data initialization. How does DPRG's pair coupling differ conceptually and empirically from this approach? Could the authors provide a controlled comparison or clarify the additional benefits of their representation construction strategy? (ii) EDGE (NeurIPS 2025) explicitly addresses insufficient diversity among generated multimodal samples through a diversity loss. How does DPRG's proposed semantic degeneracy differ from the diversity problem identified in EDGE? What advantages does retrieval-based Triple CFG provide over existing diversity-preserving objectives, apart from downstream retrieval performance?

  2. Literature completeness and novelty. Could the authors expand the related work to discuss MDM and the diversity-preserving components of EDGE in greater detail? Given the overlap in motivation and certain design principles, a clearer account of the distinctions from these methods is important for evaluating novelty.

  3. Generality of semantic degeneracy. Does semantic degeneracy also arise in distribution-matching, kernel-based, and trajectory-matching multimodal dataset distillation approaches? If it is primarily a limitation of prototype-based diffusion methods, what properties of this pipeline make it particularly susceptible to semantic collapse?

  4. Direct evaluation of semantic coherence. Can the authors provide quantitative evaluations of local neighborhood preservation, semantic cluster purity, or fine-grained cross-modal consistency? These would help distinguish improved semantic structure preservation from increased separation between synthetic samples.

  5. Evaluation beyond retrieval. Can the authors demonstrate improvements on additional multimodal tasks, such as image-text matching, compositional understanding, or attribute-level discrimination? Such evaluations would provide stronger support for the claim that the method preserves meaningful semantic distinctions.

  6. Backbone dependence. How sensitive is DPRG to the quality and alignment of the pretrained image-text representation? Experiments using encoders with different capacities or alignment quality would help clarify the extent to which the reported improvements depend on CLIP-like pretrained representations.

Reviewer_oTktOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed a training-free method for diffusion-based multimodal dataset distillation. It argues that transferring the cluster-then-generate recipe of PDS to image–text data causes “semantic degeneracy.” DPRG makes three changes: It jointly clusters interpolated, normalized CLIP image and text features. It over-clusters to ρM candidates, filters them by distinct-image support, and selects M by farthest-first coverage. It applies “Triple CFG,” which pushes the positive prediction away from image-side and text-side hard negatives mined within the distilled set, during the first half of denoising. Under a frozen-encoder, projection-head protocol, DPRG improves R_mean over PDS in all 20 settings on Flickr30K, MS-COCO, Flickr8K, and Pascal1K. The gains carry over to SigLIP + FLUX.1-dev.

Critical strengths

The method is simple and cheap. The problem framing is clear that without class labels, discovering semantic modes is itself the challenge. Each component maps onto an identified failure mode.

Critical weaknesses
  1. Pair coupling is feature interpolation, and over-clustering with farthest-first selection is k-center greedy (Sener & Savarese, 2018). Triple CFG is negative-prompt guidance with two negatives. The closest prior work is DIPSY (BMVC 2025), which already uses CFG with separate positive and negative image conditions, drawing negatives from CLIP-similar classes to make synthetic training data more discriminative. Neither DIPSY nor the broader negative-guidance literature is discussed, and the negative-mining strategy itself is never ablated. It would also be nice to include D3HR, HierAmp, and IGD.
  • Taming Diffusion for Dataset Distillation with High Representativeness (D³HR). ICML 2025.
  • HierAmp: Coarse-to-Fine Autoregressive Amplification for Generative Dataset Distillation. CVPR 2026.
  • Influence-Guided Diffusion for Dataset Distillation. ICLR 2025.
  1. The ablation is fragile, which indicates these components may not be helpful all the time. Table 9 shows that Pair alone gives only +0.3–0.4 in three of four settings. Adding Pair on top of PDS + Triple CFG hurts at M=300 (NF: 37.1 → 36.9; ViT: 24.2 → 22.0). Triple CFG gives no gain with Pair-only at M=300/ViT (22.0 → 22.0). This contradicts the claim that it helps “under every state.”
  2. Generation quality and fidelity are unverified. Qualitative evidence is limited to Fig. 7 and Fig. 12. Fig. 12 shows noticeable artifacts, such as malformed faces and limbs and unnaturally vivid colors, and no PDS samples are shown anywhere. Please provide matched-seed side-by-side samples for PDS and DPRG, fidelity metrics (CLIP score, cosine similarity to the prototype, FID/KID), and the protocol used to select the Fig. 12 samples.
Questions for authors
  1. How does Triple CFG differ from standard negative-prompt guidance and from DIPSY, beyond using two negatives? Please ablate hardest negatives against random and similarity-sampled negatives.
  2. Table 9 shows Pair hurting on top of PDS + Triple CFG at M=300, and Triple CFG giving no gain under Pair-only at M=300/ViT. How does this fit the claim that Triple CFG helps “under every state”? Are these differences within seed variance?
  3. Were the Fig. 12 samples selected at random or curated?
  4. Do the artifacts in Fig. 12 also appear with Triple CFG off, using the same seeds? Please provide matched-seed PDS vs. DPRG samples and fidelity metrics (CLIP score, FID/KID).
Reviewer_iizyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DPRG addresses semantic degeneracy in multimodal dataset distillation through pair-level feature coupling, over-clustering with coverage-based prototype selection, and bidirectional hard-negative diffusion guidance. It uses a frozen encoder-generator pipeline to construct compact image-text datasets, which are evaluated on downstream retrieval tasks.

Critical strengths
  1. DPRG improves mean retrieval performance over PDS across all 20 reported settings. Experiments with an alternative encoder-generator combination provide additional evidence of portability.

  2. The component analysis is useful. The ablations suggest that pair coupling and over-clustering complement each other, offering insights into how coherent pair assignments support finer semantic discovery.

Critical weaknesses
  1. The contribution relative to prior work could be articulated more clearly. RepBlend already discusses the conflict between compression and discriminability, while ProCo uses correspondence-aware clustering and structural regularization to improve coverage. Although DPRG adopts different mechanisms, the paper could more clearly explain what methodological insight is added. The comparison with a clustering-only variant of ProCo also provides limited evidence regarding the full method. The efficiency claims should similarly be contextualized against PDS, which already provides much of the speed advantage over optimization-based approaches.

  2. Both the image and text encoders remain frozen, and only a shallow text projection head is trained. Claims about the utility of the distilled data should therefore be scoped to this setting, as its effectiveness when the encoders are updated has not been established.

  3. Additional ablation could be included to support the design choices. Please compare linear pair fusion with concatenated image-text features using the same clustering and generation pipeline. For Triple CFG, please include comparisons with random-negative guidance and a tuned native CFG baseline. These experiments would help establish the specific benefits of linear fusion and hard-negative selection.

Questions for authors

1.What is the main methodological advance beyond existing work on discriminability and correspondence coverage?

2.Do the improvements persist when a lightweight adapter or a small portion of the encoder is trained? Supporting experiments in either setting would help clarify how broadly the paper’s conclusions apply.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.