acceptodds
Withdrawn from ICLR 2027

Post-Distillation Hybrid Dataset Curation via Differentiated Valuation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Dataset distillation produces compact synthetic training sets, while coreset selection preserves real examples with natural instance-level variation. This complementarity motivates post-distillation Hybrid Dataset Curation (HDC), where a compact training subset is constructed from a fixed distilled set and real candidates. However, curating such hybrid subsets is challenging because both sources play different roles in training. Existing strategies either score both sources with the same agnostic criterion or determine the real-distilled composition before sample valuation, which limits adaptive hybrid subset construction. To address these limitations, we propose Differentiated Hybrid Curation (DHC), a framework for post-distillation HDC based on differentiated valuation. DHC first constructs a balanced and semantically aligned real-distilled candidate pool to make cross-source comparison more tractable. It then introduces the Differentiated Identity Score (DIS), which compares candidate responses under distilled and real experts to provide a sample-wise source-aware valuation signal. Finally, DHC combines DIS with gradient-based representativeness and diversity-aware updates to construct class-balanced hybrid subsets without prescribing a fixed real-distilled ratio. Experiments on ImageWoof, ImageNette, and ImageNet-1K show that DHC improves over strong coreset selection and dataset distillation baselines in most settings and remains competitive with hybrid curation baselines.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Post-Distillation Hybrid Dataset Curation, Coreset Selection, Dataset Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FywsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Coreset selection and dataset distillation each have their own strengths and limitations. Hybrid Dataset Curation (HDC) therefore attempts to combine representative real samples with synthetic samples. Source-agnostic HDC approaches select from a combined pool of real and synthetic samples, which can result in too many real samples being selected. Fixed-ratio approaches instead use a preset hyperparameter to determine the proportion of the two types of samples, limiting the adaptability of the selected subset. This paper proposes Differentiated Hybrid Curation (DHC). DHC first builds a candidate pool with balanced numbers of real and synthetic samples using nearest-neighbor matching in CLIP feature space. It then defines a source-aware score, DIS, based on the difference in gradient norms between expert models trained separately on the two types of data. Finally, it greedily selects a subset by combining a gradient-based representativeness score with a diversity-aware penalty.

Critical strengths
  • The authors are transparent about the less favorable results on ImageNette and provide a detailed appendix.

  • IGD_NNr is an interesting byproduct of this work. At low IPC, it outperforms most existing coreset selection baselines in the reported experiments.

Critical weaknesses

1. Limited evaluation across IPC settings.

The scope of the evaluation is relatively limited. The authors only evaluate ImageNette and ImageWoof at different IPC settings. On ImageNet-1K, they evaluate the method only at IPC 10. This is not sufficient evidence to establish the effectiveness of the method across different IPC settings, since some methods perform well at low IPC but not at higher IPC. For example, IGD_NNr performs well at low IPC but sometimes fails to outperform Random at higher IPC. Therefore, the current results showing that DHC outperforms other methods at low IPC do not necessarily mean it will perform well at higher IPC. To make the evaluation on ImageNet-1K more convincing, I would suggest conducting more experiments at different IPC settings. Moreover, based on the current results, DHC fails to outperform CCFS in three of the four evaluated settings on ImageNette.

2. Potential limitations of DIS.

Since the two experts used to select samples are trained only on real or synthetic data, respectively, they may have lower gradient norms on their own training data. Meanwhile, evaluating samples from the other source is an out-of-distribution scenario, which may result in higher gradient norms. Therefore, I suspect that DIS mainly reflects which source the samples come from, rather than their individual utility. One way to investigate this is to train each expert on only a portion of its corresponding data and then compare the gradient norms of held-out samples from the same source with those of samples from the other source.

Questions for authors

Why do the authors consider DHC's selection strategy adaptive when its hyperparameters are tuned separately for each IPC setting? What is the practical advantage of this approach over tuning the real-to-synthetic ratio in CCFS?

Reviewer_coWGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies post-distillation hybrid dataset curation: selecting a compact training subset from fixed distilled images and labeled real examples. The proposed Differentiated Hybrid Curation (DHC) first constructs a balanced candidate pool by pairing distilled images with unique real nearest neighbors in CLIP feature space. Its main contribution, the Differentiated Identity Score (DIS), contrasts sample gradient norms under experts trained on the distilled and real candidate sets. DHC combines DIS with gradient alignment to the full real dataset and iterative penalties for semantic redundancy.

Experiments cover ImageWoof, ImageNette, and ImageNet-1K. DHC improves over CCFS across all ImageWoof ResNet-18 settings and reaches 46.71% on ImageNet-1K, compared with CCFS’s 45.44%, while results on ImageNette are mixed. The appendix provides additional component analyses, default hyperparameters, computational costs, and cross-architecture evaluation.

Critical strengths

The method demonstrates useful empirical improvements in several relevant settings. On ImageWoof with ResNet-18, DHC improves over CCFS by 3.60, 3.24, 2.00, and 2.56 percentage points across the four budgets. The ImageNet-1K result provides additional evidence beyond the two ten-class datasets. These results support the practical usefulness of the overall pipeline.

The paper includes meaningful controlled analyses. Figure 6 compares selectors on the same balanced hybrid pool, while Table 6 varies the real candidate set while retaining the DHC selector. These experiments help distinguish candidate-pool construction from subsequent selection. The additional reporting of default hyperparameters, runtime, and architecture transfer also strengthens the empirical assessment.

Critical weaknesses
  1. The benefit of sample-wise DIS needs a clearer comparison. I am not fully convinced that DIS adds useful information beyond choosing an appropriate preference for real or distilled samples. Figures 3 and 11 already show that adjusting this preference for each budget can improve selection. Figure 7 includes fixed preferences for both sources, but it is unclear whether their strength was tuned separately for each budget.

    Tables 9 and 10 help explain the behavior of DIS, but they do not clearly show that ranking samples within each source improves selection. Since each expert is trained on one of the candidate sources, the separation could partly reflect which samples the expert was trained on.

    This is important because sample-wise valuation is the main contribution.In Table 3, the full method outperforms the representativeness-plus-diversity variant by 0.20, 0.92, and 0.72 percentage points at IPC 100, 300, and 500, respectively. The comparison with CCFS is also mixed, with DHC performing worse in three of four ImageNette settings. I would like to see a comparison against a source preference tuned for each budget, using the same candidate pool and selection procedure.

  2. Some important experimental details are missing. It is unclear how α, β, and CCFS’s distillation fraction were selected, and whether an independent validation set was used. Appendix D.1 provides useful results with default parameters, but does not explain how the reported configurations were chosen.

    I also could not find sufficient details about the training schedules and checkpoint selection for the two DIS experts and the shared gradient model. These choices directly affect the scores. Please also clarify whether the component ablations use the full method’s hyperparameters or are tuned separately.

Questions for authors
  1. Could you compare DIS with a simple preference for real or distilled samples, tuned separately for each budget using validation data? This should use the same candidate pool and the same representativeness and diversity terms. Another useful check would be to shuffle DIS values within each class and source and see whether performance changes. This would help clarify how much the individual sample scores contribute beyond the source preference.

    Relatedly, DIS seems connected to sample-importance measures in coreset construction, including sensitivity, and more directly to gradient-norm scores such as GraNd. Classical sensitivity measures a sample’s worst-case relative contribution to the objective, while DIS compares gradient norms under two source-specific experts. Could you clarify this relationship and explain what additional information the proposed contrast provides over existing gradient-norm importance scores?

  2. How were DHC’s α and β and CCFS’s distillation fraction chosen? Which data split was used for tuning, and were the test sets excluded from this process? Please also provide the ImageNet-1K hyperparameters, which are missing from Table 5.

  3. Could you provide the training details for the two DIS experts and the shared gradient model, including initialization, training duration, and checkpoint selection? Are the gradients in Equations 3 and 4 computed using all model parameters or only specific layers? These choices seem important when comparing gradient norms across separately trained models.

  4. Table 5 sets β = 0 for ImageWoof ConvNet-6 at IPC 100, so I would expect the diversity update to have no effect if the other settings remain the same. However, Table 3 reports 61.28 for representativeness alone and 62.40 for representativeness plus diversity. Were the hyperparameters tuned separately for these variants, or were other settings changed?

Reviewer_U7VCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims to construct a compact hybrid training subset from a fixed distilled set and the real training set without fixing the real–distilled ratio in advance. Instead, real and distilled candidates are ranked jointly, and the ratio is determined by which samples are selected. DHC first builds a balanced pool comprising distilled images and their nearest real neighbors in CLIP space. Each candidate is scored by a Differentiated Identity Score (DIS), the difference between its gradient norms under an expert trained on the real neighbors and one trained on the distilled set, combined with a gradient-matching representativeness score. Samples are then selected class-wise in a greedy manner, applying a CLIP-similarity penalty to promote diversity. The method is evaluated on ImageWoof, ImageNette (ConvNet-6, ResNet-18), and ImageNet-1K using DiT-IGD distilled sets.

Critical strengths

The problem of HDC has received relatively little attention from the community, and addressing this task offers a fresh perspective.

Critical weaknesses
  1. The core premise that real and distilled samples play different roles is not justified. It is also unclear why source-composition bias is a problem that must be addressed. Intuitively, a good score should select good samples regardless of their source, so accounting for the source should not be necessary. The evidence in Sec. 2.2 does not change this view. Fig. 3 shows only that adding a constant source offset to Forgetting gains about 1%, with the offset chosen on test accuracy, which indicates that one difficulty score is miscalibrated across sources rather than that the sources play different roles. Also, difficulty scores are known to undervalue easy, prototype-like samples at small budgets [A]. Thus, the second requirement of post-distillation HDC (Sec. 2.3, line 192) is unconvincing, and I am not convinced that a source-aware signal is needed.

  2. DIS mainly reflects which expert was trained on each sample, i.e., its source, so the composition is effectively set by α\alpha. The two experts are trained on exactly the candidates being scored, so a sample's gradient norm is small under the expert that saw it and large under the other. DIS thus acts like the constant source offset that the paper itself argues is insufficient, and with a 50/50 pool the per-setting α\alpha effectively determines the real–distilled ratio, much like CCFS's ratio search. Empirically, gains over CCFS are small and mixed. CCFS wins 3 of 4 ImageNette settings and ImageWoof ConvNet-6 IPC 500, and many differences are within one standard deviation.

Questions for authors
  • What are the results when DIS is computed with cross-fitted experts that never saw the scored sample, and with ratio-matched random or Forgetting selection within the pool?

  • Would a well-calibrated source-agnostic score match DHC, e.g., Forgetting or wGM standardized separately per source?

  • In Fig. 2b, moving from the raw pool to a balanced pool changes both the real–distilled ratio and the pool size (e.g., at IPC 100, selecting 100 of ~1,000 vs. 100 of 200 candidates per class), so Forgetting picks far less extreme samples. Could you verify whether the gain comes from the source balance or from the smaller pool?

  • How were α\alpha and β\beta selected, and was a validation split used?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.