acceptodds
Under review as a conference paper at ICLR 2027

In Search of Lost Consistency in Dataset Distillation

Abstract

Dataset Distillation (DD) condenses large datasets into compact synthetic ones, promising comparable performance at a fraction of the training cost. This would make hyperparameter optimization and neural architecture search feasible even for increasingly large foundation models. Recent DD methods report impressive accuracy, yet a critical question remains: Does dataset distillation truly deliver on its promises? We argue that accuracy alone is insufficient; equally crucial is consistency, the extent to which distilled data rank hyperparameters or architectures as the full dataset does. To study this, we introduce the Dataset Distillation Testbed (DataDistillBed), which spans 13 DD methods, five datasets up to ImageNet-1K, and diverse hyperparameter configurations and neural architectures. Our results show that DD delivers on its promises only in part. At matched training budgets, distilled data do not yet replace the full dataset. As proxies for search, however, they rank training hyperparameters largely as the full dataset does (median Spearman correlation 0.82 at 50 images per class), though architectures less faithfully (0.52). Case studies that use DD to retrace the design of ConvNeXt and EfficientNet show its practicality and limitations. We further analyze how consistency relates to accuracy, compression ratio, training cost, and individual hyperparameters. Building on these findings, we design a search strategy that uses distilled data as a cheap first stage and matches the strongest full-data baselines at a third to half the cost. Because accuracy depends strongly on training hyperparameters, we also propose an evaluation protocol that compares DD methods under shared, randomly drawn training configurations and reports consistency alongside accuracy. We hope that the release of DataDistillBed will catalyze more rigorous, reproducible, and practically effective DD methods.

Primary area:
datasets and benchmarks
Keywords:
Consistency, Dataset Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ee2cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces DataDistillBed, a benchmark for evaluating whether distilled datasets preserve the model rankings obtained from full-data training. It argues that ranking consistency is an important complement to accuracy when distilled data are used for hyperparameter optimization or neural architecture search. The benchmark covers 13 dataset distillation methods, five datasets, multiple IPC levels, and both training-hyperparameter and architecture search spaces, involving more than 280,000 training runs. The results show that training-hyperparameter rankings are generally better preserved than architecture rankings, and that accuracy and consistency can diverge. The paper further analyzes the sources of consistency, studies architectural changes through ConvNeXt and EfficientNet case studies, and proposes a multi-stage search strategy combining distilled-data screening with full-data evaluation.

Critical strengths
  1. The paper addresses an important practical question through a broad empirical evaluation. Evaluating whether distilled datasets preserve model-selection decisions is valuable beyond conventional accuracy comparisons. The shared candidate configurations, coverage of multiple methods and datasets, and combination of global and top-k metrics provide a useful foundation for a standardized benchmark.
  2. The analysis offers meaningful insights beyond an accuracy leaderboard. The distinction between accuracy and consistency, the weaker preservation of architecture rankings, and the differing behavior of optimization and regularization choices expose practical limitations of current DD methods. The ConvNeXt and EfficientNet case studies make these limitations concrete.
  3. The paper connects its observations to downstream search. The staged-search experiments, including live experiments on newly sampled configurations with separate selection and evaluation subsets, provide encouraging evidence that distilled data can be useful as an initial screening mechanism. The proposed reporting protocol also encourages more transparent accounting of training configurations and tuning effort.
Critical weaknesses
  1. Reproduction discrepancies undermine the benchmark’s reliability.
    Table 2 reports substantial gaps, such as FreD on CIFAR-100 at IPC 50 (31.61% reproduced vs. 47.8% reported). The paper does not clearly distinguish unsuccessful reproduction from changes introduced by the unified evaluation protocol, leaving uncertainty about whether method comparisons reflect DD quality or reproduction issues.

  2. The hyperparameter recommendations are not sufficiently supported.
    Table 3 includes nonstandard behavior, such as constant warm-up reducing the learning rate throughout training and ASGD weight decay depending on no_bias_decay. These choices may affect the observed dominance of learning rate and optimizer. Moreover, Figure 9’s decomposition is underspecified, and low contribution to global correlation does not necessarily imply poor preservation of a hyperparameter’s effect.

  3. The practical acceleration claim lacks essential controls and cost accounting.
    The ladder is not compared with a matched random-real-subset ladder, despite random subsets being strong references. Teacher-matched controls are also missing for the ResNet pipeline. Distillation costs are excluded, while teacher inference and evaluation overhead are insufficiently documented, leaving the DD-specific, end-to-end savings unclear.

  4. Global consistency does not establish reliable selection among strong candidates.
    Top-k recovery is weak in some settings, and ImageNette correlations decrease substantially after failed configurations are excluded. NAS results additionally rely on 30-epoch reference rankings and fixed augmentation settings. The live experiments are encouraging but limited, so broader evidence of validation-based selection and competitive-candidate discrimination is needed.

Questions for authors

referring to the weakness.

4 comments
AuthorsOpenReview ↗

Response to critical weakness 1

We sincerely thank the reviewer for the thoughtful and constructive comments, as well as for recognizing the practical significance of our work and its insights beyond conventional accuracy comparisons.

W1 Reproduction discrepancies undermine the benchmark’s reliability...

A1 The large CIFAR-100 discrepancies for FreD come from our checkpoint-selection procedure, not from the unified evaluation protocol. Fixing this narrows the accuracy gap considerably and improves consistency across most HPO tasks. For NAS, the corrected data likewise improve the architecture-ranking correlation.

  1. Reproduction diagnosis. The official FreD CIFAR-100 distilled datasets at IPC 10 and 50 were not available, so we regenerated them using the authors' pipeline. To unify the evaluation protocol, our original experiments evaluated the last checkpoints instead of the best-accuracy checkpoints. After switching checkpoint selection, FreD-TM reaches 47.53% at IPC 50 and 38.73% at IPC 10 under our protocol, versus the reported 47.8% and 42.7%.
  2. We re-evaluated ranking consistency on the re-selected dataset.
  • HPO: Spearman's ρ\rho increases in 16 of the 20 IPC/budget settings, with values from 0.779 to 0.905. For example, at IPC 50 and 350 epochs, ρ\rho rises from 0.813 to 0.883.
  • NAS: Spearman's ρ\rho rises from 0.4320.432 to 0.5030.503 at IPC 10 and from −0.166-0.166 to 0.4140.414 at IPC 50 (200 epochs).

Table R3a. FreD-TM CIFAR-100 HPO consistency before and after regeneration (Spearman ρ\rho, 100 configurations × 3 trial seeds). ρ\rho is computed within each trial seed by pairing candidates by identity and then averaged over the three trial seeds; each budget has 299–300 paired candidates. "Before" is the submitted run, "after" is the regenerated artifact.

Budget (epochs)IPC 10: ρ\rho before → afterIPC 50: ρ\rho before → after
500.773 → 0.7790.830 → 0.824
1000.799 → 0.7930.886 → 0.886
1500.817 → 0.8240.877 → 0.900
2000.845 → 0.8310.873 → 0.905
2500.848 → 0.8470.849 → 0.904
3000.851 → 0.8540.831 → 0.887
3500.836 → 0.8580.813 → 0.883
4000.830 → 0.8560.795 → 0.876
4500.820 → 0.8480.769 → 0.873
5000.807 → 0.8450.756 → 0.849
  1. NAS consistency. The correlations improve, changing from 0.432/−0.166 to 0.503/0.414 at IPC 10/50.

We will update the affected FreD results in Table 2, the main consistency table, Figure 4, and the supplementary material.

Table R3b. FreD-TM NAS rank correlation on CIFAR-100 (200 proxy epochs). ρ\rho is computed within each augmentation setting of a trial seed and averaged over the three seeds, as in the main consistency table. This rerun updates the statistic only.

IPCρ\rho, beforeρ\rho, after
100.4320.503
50−0.1660.414
AuthorsOpenReview ↗

Response to critical weakness 2

W2.1. The hyperparameter recommendations are not sufficiently supported...

A2.1 We believe two implementation details need further clarification and we will correct Table 3 and document these behaviors.

  • ASGD : With no_bias_decay=True, the sampled weight decay is applied through parameter groups. With no_bias_decay=False, the sampled global weight decay is not passed, and ASGD's default lambd remains active.
  • Constant warm-up: The milestone is m = max(1, ratio × epochs). An integer milestone restores the base learning rate; a non-integer milestone keeps the 0.01 multiplier in the next schedule. All enabled cases in the audited 26-condition CIFAR subset have non-integer milestones.

To evaluate their potential impact, we conducted a retrospective candidate-exclusion test over 26 CIFAR method/dataset/pipeline conditions at IPC 50 and 350 epochs (78 trials). Importantly, this is a sensitivity analysis based on candidate exclusion, not a substitute for retraining under corrected implementations. We retain the original full-data oracle and recompute the proxy's top-10 shortlist among the remaining candidates. The detailed results are reported in Table :

Candidates excluded (either side)Oracle excludedMean lossMedianTrials above 2 points
None0 / 780.320.000 / 78
ASGD with omitted sampled decay0 / 780.320.000 / 78
All ASGD0 / 780.320.000 / 78
Constant warm-up, non-integer milestone8 / 780.460.004 / 78
All ASGD and non-integer constant warm-up8 / 780.430.004 / 78

Excluding the affected candidates has only a modest impact on shortlist quality, with mean loss remaining below 0.5 percentage points and Spearman's ρ\rho remaining positive across all conditions.

W2.2. Figure 9’s decomposition is underspecified...

A2.2 Both accuracies are converted to ranks, so ρ\rho is the correlation of two rank vectors. Since the hyperparameters are drawn independently (Table 3), ρ\rho splits into one term per hyperparameter (the covariance between its effect on full data and its effect on the proxy), a learning-rate × optimizer term, and a higher-order remainder. Each term is estimated out of fold and corrected by a permutation null. Because the constant warm-up rescales the learning rate, the learning-rate term uses the effective rate. Together with the optimizer and their interaction, it accounts for 64%/70% of observed ρ\rho on ConvNet/ResNet-18; the 70%/77% in the paper additionally counts the warm-up switch as its own term, which overlaps with the effective rate. We will add this specification. We agree that a low share alone does not imply poor preservation: weight decay and label smoothing barely affect full-data accuracy in this space, whereas augmentation does and the proxy misjudges it (up to 6 points lost when the proxy chooses it, Section 6.1). We will replace the "optimization side but not the regularization side" interpretation with the narrower conclusion that this signal supports coarse screening while final choices are confirmed on full data.

AuthorsOpenReview ↗

Response to critical weakness 3

W3.1 The ladder is not compared with a matched random-real-subset ladder...

A3.1 We have added matched random-real-subset ladder controls and evaluated both ResNet-18/WMDD and teacher-free ConvNet/MTT at B=5/15B=5/15. For ResNet-18, the distilled-data and random-real-subset ladders share the same SRe2L teacher, five configuration pools, 5,000/5,000 selection/evaluation split, selection rule, and search budgets. As shown in Table R3e, WMDD achieves 74.59%/75.40%, compared with 74.65%/75.67% for random-real-subset screening. Similarly, MTT achieves 59.77%/60.09%, compared with 59.70%/60.09% for random-real-subset screening. Both proxy-based ladders outperform the evaluated full-data random-search and successive-halving baselines under the same full-run-equivalent search budgets.

Table R3e. Mean held-out final selection accuracy (%) at B=5/15B=5/15. Each cell reports B=5B=5 and B=15B=15. Random-real results average three subset seeds; WMDD uses the repeat above. The two ResNet proxy ladders share the SRe2L teacher; the full-data columns are the paper's random-search and successive-halving baselines.

PipelineFull-data random searchFull-data successive halvingDistilled ladderRandom-real ladder
ResNet-18 (SRe2L shared by the two proxy ladders)68.70 / 71.7272.14 / 74.5874.59 / 75.4074.65 / 75.67
ConvNet, teacher-free52.64 / 55.9955.30 / 58.1059.77 / 60.0959.70 / 60.09

The largest difference in mean held-out accuracy between the distilled and random-real ladders is 0.27 percentage points across the four comparisons. Both improve on the evaluated full-data searches, supporting reduced-data screening followed by full-data refinement under the tested budgets.

W3.2 Teacher-matched controls are also missing for the ResNet pipeline.

A3.2 In this rebuttal, we reran WMDD in the same staged-search framework using the released training code. The rerun reached 74.59% and 75.40% at B=5B=5 and B=15B=15, close to the submitted 74.12% and 74.85%, respectively. The WMDD and random-real ladders in Table R3e use the same SRe2L teacher and KD protocol.

W3.3 Distillation costs are excluded...

A3.3 We clarify that our reported efficiency reflects search-stage savings given an available distilled dataset and teacher, rather than end-to-end savings including one-time distillation. Search budgets include evaluation on 5,000 images at each selection step. In a representative IPC-10, 1,000-image, 100-epoch run, online KD adds 1.91 s of wall-clock time and 18.1% to training-step time. The matched ResNet-18 ladders share the teacher and KD protocol.

AuthorsOpenReview ↗

Response to critical weakness 4

W4 Global consistency does not establish reliable selection among strong candidates.

A4 We tested selection directly on held-out data. In five ResNet-18 pools of 300 configurations, we used separate 5,000-image sets for selection and evaluation. Relative to the best held-out score among the 100-epoch candidates reached by either proxy ladder in each pool, the distilled ladder's mean regret was 1.16 percentage points at B=5B=5 and 0.36 at B=15B=15. It also achieved higher mean held-out accuracy than full-data random search and successive halving at both budgets (Table R3e).

On ImageNette, restricting the pool from 300 candidates to the 214 viable ones preserves each ranking's top 10 and gives the same NDCG@10, Overlap@10, and MRR@10. Within this subset, ρv\rho_{\mathrm{v}} ranges from 0.218 to 0.387 at IPC 1 and from 0.499 to 0.653 at IPC 10. Our NAS result applies to the evaluated NATS-Bench topology space, with a shared training recipe, fixed main optimization hyperparameters, and a 30-epoch full-data reference.

Reviewer_1pSVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a new benchmark for dataset distillation (DD), challenging the assumption that accuracy alone is a sufficient measure. They introduce DataDistillBed, which covers 13 DD methods, five datasets (CIFAR-10 to ImageNet-1K), and two search spaces: training hyperparameters and NATS-Bench architectures. They found empirically that distilled datasets can be used as a proxy for search (even if the model does not reach the same performance as using the full dataset). In more detail, it ranks training hyperparameters largely similar to the full dataset, while architectures are ranked not as good as using the full dataset. Two case studies on ImageNet-1K retrace the design of ConvNeXt and EfficientNet; distilled data follows the ConvNeXt roadmap until the last step (changing from BatchNorm to LayerNorm), where it fails.

Critical strengths
  • The authors argue that a search proxy does not need to match the model's accuracy on the full dataset. In contrast, it must rank candidates as the full dataset would. They turn this into a lightweight training strategy: candidates are evaluated on distilled data, which is light, and only the best are trained on the full dataset. This pipeline matches the strongest full-data baselines at about a third of the compute. This is consequential because it gives DD a concrete, measurable use in HPO, beyond accuracy metrics.

  • A large set of experiments. In this paper, the authors presented results of more than 280K trained networks across small- and large-scale datasets, plus two case studies on ImageNet-1K, making the findings more likely to generalize.

Critical weaknesses
  • A known limitation of distilled datasets is poor generalization across architectures. In the hyperparameter search space, the evaluated network matches the distillation or teacher architecture (ConvNet, or ResNet-18 with a ResNet-18 teacher), so the reported consistency is an in-architecture result. When the architecture changes, consistency drops substantially.

  • All search spaces and case studies are convolutional. The ConvNeXt case study already shows distilled data failing when LayerNorm replaces BatchNorm. This drop could be caused by the prior of Decoupling Distilled Datasets (SRe2L, CVDD, G-VBSM, SCDD) approaches, which leverage the statistics collected by BatchNorm layers.

Questions for authors
  • How does this limitation affect the proxy search when the hyperparameters are intended for a different target architecture? For example, the distilled dataset is obtained by a Convolutional Network and evaluated on a ResNet-18

  • Decoupling Distilled Dataset approaches rely on soft-labeling, for example, the relabel step in SRe2L. What is the effect if soft-labelling is not present or is different teachers?

  • Do your claims hold in other use case which are not based on convolutional networks?. Vision Transformers are a good example for evaluating the claims because you can use the training recipe of DeiT III (Revenge of the ViT) as a search space. The results can help to generalize your claim or find opportunities for improving your benchmark.

3 comments
AuthorsOpenReview ↗

Response to question 1

We thank the reviewer for the constructive suggestions on architecture generalization and supervision. We first test cross-architecture HPO, then separate the effects of removing soft labels and changing their source, and finally state the scope of our non-CNN claims.

>>> W1 & Q1

A known limitation of distilled datasets is poor generalization across architectures. In the hyperparameter search space, the evaluated network matches the distillation or teacher architecture (ConvNet, or ResNet-18 with a ResNet-18 teacher), so the reported consistency is an in-architecture result. When the architecture changes, consistency drops substantially.

How does this limitation affect the proxy search when the hyperparameters are intended for a different target architecture? For example, the distilled dataset is obtained by a Convolutional Network and evaluated on a ResNet-18

>>> A1 Useful HPO ranking transfers to the tested ConvNet-to-ResNet-18 settings, although its strength varies by method and direction. We directly evaluated ConvNet-distilled data with a ResNet-18 proxy against the ResNet-18 full-data ranking, and the reverse direction with a ConvNet proxy, on CIFAR-10/100 at IPC 50 and 350 epochs.

  1. Experimental setting. The experiment uses the paper's pairing protocol and covers 16 cells, 100 configurations × 3 trial seeds, and 4,800 completed runs; 46 low-accuracy runs are retained. The proxy and full-data target architectures match, while their relation to the data-construction architecture changes.

  2. ConvNet-to-ResNet-18. DSA and MTT achieve ρ\rho = 0.816/0.834 on CIFAR-10 and 0.857/0.901 on CIFAR-100. Strong global agreement therefore does not require the evaluation architecture to match the data-construction architecture in these settings.

    Table R2a. ConvNet-distilled data with a ResNet-18 proxy and target (CIFAR-10/100, IPC 50, 350 epochs). ρ\rho is the Spearman correlation between the proxy ranking and the full-data ranking of the target architecture, computed within each trial seed and averaged over three seeds.

    Distilled dataProxy → targetCIFAR-10 ρ\rhoCIFAR-100 ρ\rho
    DSA (ConvNet)ResNet-18 → ResNet-180.8160.857
    MTT (ConvNet)ResNet-18 → ResNet-180.8340.901
  3. Reverse direction. Agreement remains positive but is weaker and more variable (Table R2b): CVDD/G-VBSM give ρ\rho = 0.526/0.429 on CIFAR-10 and 0.628/0.728 on CIFAR-100 with hard labels, rising to 0.573/0.532 and 0.674/0.740 with their own teachers. SRe2L also improves with KD.

    Table R2b. Reverse direction: ResNet-18-distilled data with a ConvNet proxy and target. Same code, protocol, budgets and statistics as Table R2a.

    Distilled dataProxy supervisionCIFAR-10 ρ\rhoCIFAR-100 ρ\rho
    CVDDhard label0.5260.628
    G-VBSMhard label0.4290.728
    SRe2Lhard label0.5290.708
    CVDDKD0.5730.674
    G-VBSMKD0.5320.740
    SRe2LKD0.6470.793

The supported conclusion is useful transfer in the tested settings, not a universal cross-architecture guarantee. Because the two directions also use different DD methods, the directional gap is not an architecture-only effect. Practical use should validate consistency with the intended target architecture and supervision protocol.

AuthorsOpenReview ↗

Response to question 2

Q2. What is the effect if soft-labelling is not present or is different teachers?

A2 Removing teacher soft labels sharply lowers accuracy but leaves the global ranking signal. Changing the teacher largely preserves configuration ordering.

  • No soft labels. We compare hard-label and KD training on the same distilled images, configurations, seeds, and full-data reference across 34 matched CNN cells. Only teacher-derived supervision is removed; other sampled choices, including mixup, stay fixed.

    Table R2c. Effect of removing teacher-derived soft labels.

    QuantityResult
    Mean Δρ\Delta\rho after removing the teacher (hard label − KD)+0.03
    Cells with higher / lower ρ\rho after removing the teacher23 / 11
    Hard-label vs. KD ranking correlation at IPC 500.75-0.91
    Best CIFAR-10 accuracy at IPC 50, KD → hard label60-71% → 30-42%

    Removing soft labels hurts proxy accuracy much more than global ranking. In these settings, proxy accuracy and ranking consistency are therefore different properties.

  • Different teachers. On CIFAR-10 IPC-50 SRe2L images, we vary only the teacher checkpoint across five teachers. The 300 configurations, seed, images, ResNet-18 student, KD protocol, 350-epoch budget, and code are fixed.

    Table R2d. Effect of changing the teacher checkpoint.

    QuantityResult
    ρ\rho against the ResNet-18 full-data ranking0.652-0.730
    Hard-label baseline0.730
    Pairwise Spearman correlation between teacher-induced rankings0.940-0.975
    Mean pairwise Spearman correlation0.952

    Teacher identity changes absolute ρ\rho by about 0.08, but pairwise agreement is high and configuration ordering is largely preserved. The positive-consistency conclusion holds in this controlled setting.

AuthorsOpenReview ↗

Response to question 3

W2 & Q3. Do your claims hold in other use case which are not based on convolutional networks?

A3 We limit the positive HPO claims to the evaluated CNN pipelines; non-CNN generalization is not established. We will qualify the abstract, introduction, and conclusion accordingly. BatchNorm-statistics dependence is a plausible but untested explanation for the ConvNeXt failure. A ViT/DeiT-III study is a useful extension, not evidence in this submission; we will pursue non-convolutional evaluation and direct tests of this hypothesis as future work.

Reviewer_DaDnOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work identifies a key flaw in Dataset Distillation (DD): test accuracy alone fails to validate distilled datasets as reliable proxies for HPO and NAS. We propose ranking consistency as a new metric and build DataDistillBed, a unified large-scale benchmark encompassing 13 DD methods and 5 datasets (including ImageNet-1K), with more than 280k training runs. We show accuracy and consistency are decoupled: distilled data ranks optimization hyperparameters (e.g., learning rate, optimizer) reasonably well, but poorly captures regularization effects from weight decay, augmentation, and normalization (BN/LN). We further present a two-stage HPO search pipeline and a standardized reporting protocol for DD evaluations.

Critical strengths

It identifies a prevalent misconception in the field: that high fidelity guarantees good surrogate ranking performance for HPO/NAS tasks. The research motivation is of strong practical significance. Large-scale standardized experiments yield counterintuitive conclusions valuable to the community. The proposed staged search pipeline and evaluation protocol offer actionable guidance for practitioners.

Critical weaknesses

This is an empirical benchmarking work and does not introduce a novel dataset distillation algorithm. Its incremental contributions need to be more clearly justified when compared with existing benchmarks such as DC-Bench and DD-Ranking. Confounding factors exist in the experiments: ConvNets and ResNet-KD follow different training pipelines, and direct cross-architecture comparison can be misleading. Filtering out non-converged configurations may introduce bias to ranking metrics. The open-source release does not include the full distilled datasets, raising the barrier to reproducibility. The failure case of ConvNeXt is only presented as an observation, without mechanistic ablation studies. Some conclusions are over-generalized and should be qualified with explicit scope constraints. It would also be beneficial to refer to studies in AutoML.

Questions for authors

When you exclude non‑converged runs, what exact filtering criteria are applied? Quantitatively, how much do consistency metrics shift if you retain those failed configurations?

3 comments
AuthorsOpenReview ↗

Response to Questions

We sincerely thank the reviewer for the constructive feedback and for recognizing the practical significance of our study. We first clarify the candidate populations and their quantitative effects, then explain the evaluation gap relative to prior benchmarks.

Q1. When you exclude non‑converged runs, what exact filtering criteria are applied? Quantitatively, how much do consistency metrics shift if you retain those failed configurations?

>>> A1 There is no general convergence-based filter in our primary analysis. The question combines three distinct population rules: the primary paired population, the accuracy-based ρv\rho_{\mathrm{v}} viability diagnostic, and a separate four-pair ImageNet protocol exclusion. We define each rule and then quantify its effect below.

Population definitions.

  1. Primary paired population. Runs are paired by hyperparameter and trial seed under the same sampled settings, and both sides must record a finite final accuracy. Subject to the separate ImageNet protocol rule below, the primary ρ\rho, NDCG@10, Overlap@10, and MRR@10 use this full paired population. Runs that terminate after divergence remain included when they record a finite final accuracy.

  2. Viability diagnostic. Let CC be the number of classes. The 2/C2/C threshold is applied only to the last recorded full-data accuracy in the separate ρv\rho_v diagnostic; proxy accuracy is not used for this label. Thus, ρv\rho_v is an accuracy-based viability analysis, not a convergence filter, and it does not replace the primary ρ\rho.

  3. ImageNet protocol requirement. Full-data runs always use mixed precision, while proxy runs sample this setting. The ImageNet table excludes four pairs in which the proxy ran without mixed precision and only the full-data run diverged with non-finite loss.

Quantitative impact.

  • ImageNette. Across the three trial seeds, all 300 full-data runs finished 50 epochs with finite loss, and 214 met the viability threshold. We rank candidates within each seed and average the metrics over seeds. Across the eight method/IPC settings, ρ\rho ranges from 0.693 to 0.866 on all candidates and ρv\rho_{\mathrm{v}} from 0.218 to 0.653 on the viable subset. Each seed retains the same proxy and full-data top 10. The removed candidates also rank below every viable candidate on full data, so NDCG@10, Overlap@10 and MRR@10 are unchanged.

  • ImageNet. Pooling the 56 retained pairs across the three trial seeds gives ρ\rho = 0.649–0.675 and Overlap@10 = 0.70. Restoring the four protocol-mismatched pairs reduces ρ\rho to 0.419–0.440, lowers NDCG@10 by 0.22–0.30, lowers Overlap@10 to 0.50–0.60, and lowers MRR@10 by 0.016–0.031. Unlike ImageNette, ImageNet top-k recovery is therefore sensitive to these four pairs.

  • CIFAR/Tiny-ImageNet loss sensitivity. As an additional numerical-stability analysis, excluding pairs whose full-data loss becomes non-finite changes ρ\rho by at most 0.05 in 33/34 cells; median ρ\rho changes from 0.843 over all candidates to 0.859 over the numerically stable population.

We will revise “fail to train / do train” to state the operational 2/C2/C definition directly, identify ρv\rho_{\mathrm{v}} as a viability diagnostic rather than a convergence filter, and report the primary results, the viability diagnostic, and the effect of the four ImageNet exclusions separately.

AuthorsOpenReview ↗

Response to Critical Weakness: incremental contributions to existing benchmarks

W1.1. This is an empirical benchmarking work...

This is an empirical benchmarking work and does not introduce a novel dataset distillation algorithm. Its incremental contributions need to be more clearly justified when compared with existing benchmarks such as DC-Bench and DD-Ranking.

>>> A1.1 DataDistillBed asks whether a fixed distilled dataset preserves the ranking of training hyperparameters—an evaluation question not answered by DC-Bench or DD-Ranking. Accuracy under a fixed recipe is insufficient: a proxy can train an accurate model while favoring the wrong hyperparameters.

  • DC-Bench studies architecture-ranking agreement through NAS. DD-Ranking ranks DD methods using accuracy-derived measures that account for supervision and augmentation. Neither establishes whether a given distilled dataset preserves the ordering of candidate training configurations.

  • DataDistillBed fills this gap with a shared randomized protocol across 13 methods and five datasets, measuring global agreement and top-k recovery across budgets and search spaces. The distinction matters: TESLA can be more accurate than DSA yet preserve hyperparameter rankings less faithfully, and longer training can improve accuracy without improving consistency. The stronger HPO agreement but weaker architecture-ranking agreement in our settings further shows that evidence does not transfer automatically across search spaces.

Our contribution is an empirical basis for deciding when DD can support inexpensive screening and when full-data refinement remains necessary. We will make this distinction explicit in the introduction and related work.

AuthorsOpenReview ↗

Response to some other Critical Weakness

W1.2 Confounding factors exist in the experiments: ConvNets and ResNet-KD follow different training pipelines, and direct cross-architecture comparison can be misleading.

A1.2 Each HPO correlation compares two rankings obtained with the same evaluation architecture. For ConvNet, we rank the same hyperparameter configurations on distilled data and on full data using ConvNet for both; for ResNet-18, we repeat the comparison using ResNet-18 for both. These are separate HPO tasks with different training pipelines. We will clarify this in the table caption.

W1.4 The open-source release does not include the full distilled datasets, raising the barrier to reproducibility.

A1.4 The complete distilled-input collection is several GB, far beyond the 100 MB attachment limit. Many of the inputs are already publicly available in the original authors' own releases, which the reproducibility appendix uses directly; the remaining inputs were regenerated with the authors' official code. We will release the complete distilled inputs upon acceptance.

W1.5 The failure case of ConvNeXt is only presented as an observation, without mechanistic ablation studies.

A1.5 Thanks for this suggestion. The ConvNeXt result is a case-study observation, not a mechanistic result, and we do not present it as one. The two case studies (ConvNeXt and EfficientNet) are meant to illustrate where DD is practical and where its assumptions break — the abstract states this — so this work's substantive claims rest on the main consistency tables, not on this case study. We already bound it with "distilled data can guide structural search within the resolution and normalization regime used for distillation," and the the mechanism behind the BatchNorm→LayerNorm collapse — including whether decoupled methods rely on BatchNorm statistics — and non-convolutional generalization are left as future work.

W1.6 Some conclusions are over-generalized and should be qualified with explicit scope constraints.

A1.6 We thank the reviewer for pointing this out, which helps us state the scope more precisely. We have narrowed the claims where they are stated. The abstract, conclusion, and Section 6.1 now restrict the positive findings to the evaluated CNN pipelines, and the cost and architecture claims are bounded to the same pipelines: the abstract reports the search-stage saving given an available distilled dataset and teacher, and the architecture results use a 30-epoch full-data reference within a fixed hyperparameter setting.

W1.7 It would also be beneficial to refer to studies in AutoML.

A1.7 Thanks for the guidance and reference suggestions on AutoML. We already cite the standard AutoML search methods — random search, TPE, successive halving, and Hyperband — together with the tabular benchmarks HPOBench, HPO-B, and NATS-Bench, in our search evaluation, where distilled data serve as a lower-fidelity proxy (the multi-fidelity HPO setting).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.