- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces DataDistillBed, a benchmark for evaluating whether distilled datasets preserve the model rankings obtained from full-data training. It argues that ranking consistency is an important complement to accuracy when distilled data are used for hyperparameter optimization or neural architecture search. The benchmark covers 13 dataset distillation methods, five datasets, multiple IPC levels, and both training-hyperparameter and architecture search spaces, involving more than 280,000 training runs. The results show that training-hyperparameter rankings are generally better preserved than architecture rankings, and that accuracy and consistency can diverge. The paper further analyzes the sources of consistency, studies architectural changes through ConvNeXt and EfficientNet case studies, and proposes a multi-stage search strategy combining distilled-data screening with full-data evaluation.
Critical strengths
- The paper addresses an important practical question through a broad empirical evaluation. Evaluating whether distilled datasets preserve model-selection decisions is valuable beyond conventional accuracy comparisons. The shared candidate configurations, coverage of multiple methods and datasets, and combination of global and top-k metrics provide a useful foundation for a standardized benchmark.
- The analysis offers meaningful insights beyond an accuracy leaderboard. The distinction between accuracy and consistency, the weaker preservation of architecture rankings, and the differing behavior of optimization and regularization choices expose practical limitations of current DD methods. The ConvNeXt and EfficientNet case studies make these limitations concrete.
- The paper connects its observations to downstream search. The staged-search experiments, including live experiments on newly sampled configurations with separate selection and evaluation subsets, provide encouraging evidence that distilled data can be useful as an initial screening mechanism. The proposed reporting protocol also encourages more transparent accounting of training configurations and tuning effort.
Critical weaknesses
-
Reproduction discrepancies undermine the benchmark’s reliability.
Table 2 reports substantial gaps, such as FreD on CIFAR-100 at IPC 50 (31.61% reproduced vs. 47.8% reported). The paper does not clearly distinguish unsuccessful reproduction from changes introduced by the unified evaluation protocol, leaving uncertainty about whether method comparisons reflect DD quality or reproduction issues. -
The hyperparameter recommendations are not sufficiently supported.
Table 3 includes nonstandard behavior, such as constant warm-up reducing the learning rate throughout training and ASGD weight decay depending on no_bias_decay. These choices may affect the observed dominance of learning rate and optimizer. Moreover, Figure 9’s decomposition is underspecified, and low contribution to global correlation does not necessarily imply poor preservation of a hyperparameter’s effect. -
The practical acceleration claim lacks essential controls and cost accounting.
The ladder is not compared with a matched random-real-subset ladder, despite random subsets being strong references. Teacher-matched controls are also missing for the ResNet pipeline. Distillation costs are excluded, while teacher inference and evaluation overhead are insufficiently documented, leaving the DD-specific, end-to-end savings unclear. -
Global consistency does not establish reliable selection among strong candidates.
Top-k recovery is weak in some settings, and ImageNette correlations decrease substantially after failed configurations are excluded. NAS results additionally rely on 30-epoch reference rankings and fixed augmentation settings. The live experiments are encouraging but limited, so broader evidence of validation-based selection and competitive-candidate discrimination is needed.
Questions for authors
referring to the weakness.
4 comments
Response to critical weakness 2
W2.1. The hyperparameter recommendations are not sufficiently supported...
A2.1 We believe two implementation details need further clarification and we will correct Table 3 and document these behaviors.
- ASGD : With
no_bias_decay=True, the sampled weight decay is applied through parameter groups. Withno_bias_decay=False, the sampled global weight decay is not passed, and ASGD's default lambd remains active. - Constant warm-up: The milestone is
m = max(1, ratio × epochs). An integer milestone restores the base learning rate; a non-integer milestone keeps the 0.01 multiplier in the next schedule. All enabled cases in the audited 26-condition CIFAR subset have non-integer milestones.
To evaluate their potential impact, we conducted a retrospective candidate-exclusion test over 26 CIFAR method/dataset/pipeline conditions at IPC 50 and 350 epochs (78 trials). Importantly, this is a sensitivity analysis based on candidate exclusion, not a substitute for retraining under corrected implementations. We retain the original full-data oracle and recompute the proxy's top-10 shortlist among the remaining candidates. The detailed results are reported in Table :
| Candidates excluded (either side) | Oracle excluded | Mean loss | Median | Trials above 2 points |
|---|---|---|---|---|
| None | 0 / 78 | 0.32 | 0.00 | 0 / 78 |
| ASGD with omitted sampled decay | 0 / 78 | 0.32 | 0.00 | 0 / 78 |
| All ASGD | 0 / 78 | 0.32 | 0.00 | 0 / 78 |
| Constant warm-up, non-integer milestone | 8 / 78 | 0.46 | 0.00 | 4 / 78 |
| All ASGD and non-integer constant warm-up | 8 / 78 | 0.43 | 0.00 | 4 / 78 |
Excluding the affected candidates has only a modest impact on shortlist quality, with mean loss remaining below 0.5 percentage points and Spearman's remaining positive across all conditions.
W2.2. Figure 9’s decomposition is underspecified...
A2.2 Both accuracies are converted to ranks, so is the correlation of two rank vectors. Since the hyperparameters are drawn independently (Table 3), splits into one term per hyperparameter (the covariance between its effect on full data and its effect on the proxy), a learning-rate × optimizer term, and a higher-order remainder. Each term is estimated out of fold and corrected by a permutation null. Because the constant warm-up rescales the learning rate, the learning-rate term uses the effective rate. Together with the optimizer and their interaction, it accounts for 64%/70% of observed on ConvNet/ResNet-18; the 70%/77% in the paper additionally counts the warm-up switch as its own term, which overlaps with the effective rate. We will add this specification. We agree that a low share alone does not imply poor preservation: weight decay and label smoothing barely affect full-data accuracy in this space, whereas augmentation does and the proxy misjudges it (up to 6 points lost when the proxy chooses it, Section 6.1). We will replace the "optimization side but not the regularization side" interpretation with the narrower conclusion that this signal supports coarse screening while final choices are confirmed on full data.
Response to critical weakness 3
W3.1 The ladder is not compared with a matched random-real-subset ladder...
A3.1 We have added matched random-real-subset ladder controls and evaluated both ResNet-18/WMDD and teacher-free ConvNet/MTT at . For ResNet-18, the distilled-data and random-real-subset ladders share the same SRe2L teacher, five configuration pools, 5,000/5,000 selection/evaluation split, selection rule, and search budgets. As shown in Table R3e, WMDD achieves 74.59%/75.40%, compared with 74.65%/75.67% for random-real-subset screening. Similarly, MTT achieves 59.77%/60.09%, compared with 59.70%/60.09% for random-real-subset screening. Both proxy-based ladders outperform the evaluated full-data random-search and successive-halving baselines under the same full-run-equivalent search budgets.
Table R3e. Mean held-out final selection accuracy (%) at . Each cell reports and . Random-real results average three subset seeds; WMDD uses the repeat above. The two ResNet proxy ladders share the SRe2L teacher; the full-data columns are the paper's random-search and successive-halving baselines.
| Pipeline | Full-data random search | Full-data successive halving | Distilled ladder | Random-real ladder |
|---|---|---|---|---|
| ResNet-18 (SRe2L shared by the two proxy ladders) | 68.70 / 71.72 | 72.14 / 74.58 | 74.59 / 75.40 | 74.65 / 75.67 |
| ConvNet, teacher-free | 52.64 / 55.99 | 55.30 / 58.10 | 59.77 / 60.09 | 59.70 / 60.09 |
The largest difference in mean held-out accuracy between the distilled and random-real ladders is 0.27 percentage points across the four comparisons. Both improve on the evaluated full-data searches, supporting reduced-data screening followed by full-data refinement under the tested budgets.
W3.2 Teacher-matched controls are also missing for the ResNet pipeline.
A3.2 In this rebuttal, we reran WMDD in the same staged-search framework using the released training code. The rerun reached 74.59% and 75.40% at and , close to the submitted 74.12% and 74.85%, respectively. The WMDD and random-real ladders in Table R3e use the same SRe2L teacher and KD protocol.
W3.3 Distillation costs are excluded...
A3.3 We clarify that our reported efficiency reflects search-stage savings given an available distilled dataset and teacher, rather than end-to-end savings including one-time distillation. Search budgets include evaluation on 5,000 images at each selection step. In a representative IPC-10, 1,000-image, 100-epoch run, online KD adds 1.91 s of wall-clock time and 18.1% to training-step time. The matched ResNet-18 ladders share the teacher and KD protocol.
Response to critical weakness 4
W4 Global consistency does not establish reliable selection among strong candidates.
A4 We tested selection directly on held-out data. In five ResNet-18 pools of 300 configurations, we used separate 5,000-image sets for selection and evaluation. Relative to the best held-out score among the 100-epoch candidates reached by either proxy ladder in each pool, the distilled ladder's mean regret was 1.16 percentage points at and 0.36 at . It also achieved higher mean held-out accuracy than full-data random search and successive halving at both budgets (Table R3e).
On ImageNette, restricting the pool from 300 candidates to the 214 viable ones preserves each ranking's top 10 and gives the same NDCG@10, Overlap@10, and MRR@10. Within this subset, ranges from 0.218 to 0.387 at IPC 1 and from 0.499 to 0.653 at IPC 10. Our NAS result applies to the evaluated NATS-Bench topology space, with a shared training recipe, fixed main optimization hyperparameters, and a 30-epoch full-data reference.
Response to critical weakness 1
We sincerely thank the reviewer for the thoughtful and constructive comments, as well as for recognizing the practical significance of our work and its insights beyond conventional accuracy comparisons.
W1 Reproduction discrepancies undermine the benchmark’s reliability...
A1 The large CIFAR-100 discrepancies for FreD come from our checkpoint-selection procedure, not from the unified evaluation protocol. Fixing this narrows the accuracy gap considerably and improves consistency across most HPO tasks. For NAS, the corrected data likewise improve the architecture-ranking correlation.
Table R3a. FreD-TM CIFAR-100 HPO consistency before and after regeneration (Spearman ρ, 100 configurations × 3 trial seeds). ρ is computed within each trial seed by pairing candidates by identity and then averaged over the three trial seeds; each budget has 299–300 paired candidates. "Before" is the submitted run, "after" is the regenerated artifact.
We will update the affected FreD results in Table 2, the main consistency table, Figure 4, and the supplementary material.
Table R3b. FreD-TM NAS rank correlation on CIFAR-100 (200 proxy epochs). ρ is computed within each augmentation setting of a trial seed and averaged over the three seeds, as in the main consistency table. This rerun updates the statistic only.