acceptodds
Withdrawn from ICLR 2027

Profile Once, Distill Across Budgets: Budget–Checkpoint Alignment for Dataset Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

*Decoupled dataset distillation* improves scalability by using pretrained checkpoints for synthetic image recovery and labeling. Its effectiveness, however, depends strongly on checkpoint maturity, with different images-per-class (IPC) budgets favoring different checkpoints. Jointly searching over recovery and labeling checkpoints at each budget requires a separate post-evaluation for every candidate pair, creating substantial alignment overhead. We introduce **P**rofile-Once **A**lignment of **C**heckpoints (**PAC**), a plug-in framework for reducing checkpoint alignment costs across IPC budgets. **PAC** replaces recovery checkpoint search with a single curriculum recovery run that uses checkpoints from all maturity levels to generate a shared image bank at the largest budget. With these images fixed, only the labeling checkpoints need to be aligned with each IPC budget, reducing the joint recovery labeling search to labeling selection. To guide this selection across IPC budgets, we identify exposure-indexed rank transfer, which links labeling checkpoint rankings at matched cumulative synthetic data exposure, defined as IPC times the number of training epochs on synthetic data. This allows us to profile labeling checkpoints only at the largest budget by tracking model performance throughout training. For smaller budgets, we sample from the recovered images and select labeling checkpoints using reference rankings at epochs matching the target budget's full-training exposure. To further reduce profiling cost, we characterize critical exposures at which checkpoint preferences change and derive sufficient conditions for permanently eliminating candidates from further evaluation. The resulting selection procedure supports previously unseen budgets within the reference range without additional recovery or profiling. Experiments across five datasets, various IPC budgets, and model architectures establish new state-of-the-art performance. At IPC 1, integrating **PAC** with FADRM+ substantially improves top-1 accuracy by **12.4%** on CIFAR-100 and by **18.1%** on ImageNet-1K.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Dataset Distillation, Data-centric AI, Efficient Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XxMCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies checkpoint selection in decoupled dataset distillation, where the preferred maturity of recovery and labeling checkpoints depends on the IPC budget. The authors propose PAC. PAC first builds a checkpoint pool from accuracy milestones and forms local ensembles of consecutive checkpoints as candidates. It then recovers one shared image bank at the largest budget using a curriculum over all checkpoints, so that only labeling checkpoints need to be selected per budget. To avoid profiling every budget, the paper introduces exposure-indexed rank transfer. Labeling candidates are therefore profiled once at the reference budget, and the ranking at the matched epoch is read out for smaller budgets.

Critical strengths
  1. Reading candidate rankings from intermediate epochs of a single reference run is simple and easy to implement. It can produce selections for unseen budgets below the reference budget at no extra cost.
  2. The gains at low IPC are large and consistent across datasets and student architectures.
  3. The paper reports the reduction in alignment time relative to joint search and the effect of prefix elimination on profiling time.
Critical weaknesses
  1. The paper introduces many terms that are not standard in dataset distillation, such as maturity level, alignment, profiling, selection frontier, and readout. Some are never defined explicitly, and others are defined only implicitly or long after their first use.
  2. The conceptual novelty is limited. The finding that the preferred teacher training stage depends on the budget has been established by DATM and CheckpointKD. Sampling smaller IPC sets from a larger distilled set is common practice and is also used for the baselines in Table 1. Curriculum recovery over multiple teachers is a combination of existing components. The only new ingredient is the exposure matching rule, which is a heuristic.
  3. Exposure-indexed rank transfer, the core idea of the paper, is assumed rather than verified. Eq. (8) simply assumes that training at IPC 1 for 300 epochs is equivalent to training at IPC 50 for 6 epochs, which is the pairing the paper uses in Appendix B.1. The paper never checks whether candidate rankings actually agree under such matched settings across datasets. For the same reason, the transfer error δ_s in Corollary 1 is never measured, so the regret bound cannot be assessed.
  4. Theorem 1 restates its own assumption. It assumes the best candidate never shifts back to a less mature one, then concludes that less mature candidates can be safely removed. The theorem thus adds no insight beyond the assumption.
  5. PAC combines several components, including full-pool recovery, intermediate-checkpoint labeling, exposure matching, and prefix elimination. An ablation that adds each component to FADRM+ separately is missing.
  6. Most ablations and analyses (Tables 3 to 12) are run only on CIFAR-100 with ResNet-18, mostly from single runs without standard deviations. The evidence for rank transfer in Figures 2 and 3 does not state the dataset clearly.
  7. Eq. (6) does not specify how Π_s selects s images per class from the shared bank, for example randomly or by some criterion.
Questions for authors
  1. Were the 9 architectures in Table 2 each profiled separately, or do they reuse the selections from one architecture?
  2. In Table 13 the least mature window Ω1 is selected at IPC 1, the smallest budget tested. How sensitive are the IPC 1 results to the lower end of the milestone range?
  3. Table 16 lists “Number of ARC 4” without explanation. What does ARC refer to?
  4. In Figure 4, is profiling repeated at each stage as the number of classes grows?
Reviewer_ZFgeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose Profile-Once Alignment of Checkpoints (PAC), a plug-in framework for reducing the cost of selecting recovery and labeling checkpoints across IPC in decoupled dataset distillation. The method leverages prior work, mainly on trajectory matching (e.g., DATM), showing that early teacher checkpoints work better in smaller IPCs, whereas late checkpoints fit larger IPCs. PAC recovers a single shared image bank at the largest IPC via a curriculum over checkpoints of increasing maturity, and smaller IPCs reuse subsets of these images. To select the labeling checkpoints for each IPC, the authors assume that candidate rankings depend on cumulative exposure. The method is evaluated on five datasets (CIFAR-10, CIFAR-100, Tiny-ImageNet, ImageNette and ImageNet-1K) with ResNet-18, ResNet-101, and MobileNet-V2 as evaluation models, plus a cross-architecture study on ImageNet-1K.

Critical strengths
  • The full-pool curriculum recovery, which passes through teacher checkpoints in order of maturity to build a shared image bank. One recovery run serves all smaller IPCs by subsampling, which removes per-budget recovery. With labeling fixed, it improves over budget-matched recovery at all IPCs. This is consequential because it offers a cheap, reusable way to improve recovered images that could benefit other recovery-based methods.
Critical weaknesses
  • The FADRM+ results reported in Table 1 are lower than those published in the original paper. For example, CIFAR-100/ResNet-18/IPC=1 is reported at 19.0% vs. 40.6% (−21.6 pp), and Tiny-ImageNet/ResNet-101/IPC=1 at 17.9% vs. 41.9% (−24.0 pp). PAC is worse in 7 of the 24 experiments using the published results of FADRM+. The claim of consistent gains across settings is therefore not supported.

  • Existing decoupled methods use the final teacher for both recovery and labeling, so the exhaustive NJ^2 joint search is not a practical baseline. The evidence that the recovery checkpoint must depend on the budget comes from trajectory-matching works (Guo et al., 2024; Lee and Chung, 2024), not from decoupled methods; Qin et al. (2024) show budget-dependent labeling checkpoints. Simpler baselines are also missing: recovery with the final teacher and PAC’s selected checkpoint ensembles for labeling, and full-pool recovery with the final teacher for labeling. For instance, Table 12 (SRe²L + PAC) may provide initial evidence, but the paper does not specify whether PAC changes only the labeling. Finally, in Table 3, PAC outperforms exhaustive joint search on CIFAR-100, which should not be possible. This can only be possible because full-pool recovery lies outside the joint-search candidate set, which suggests that the gain comes mainly from the new recovery scheme rather than from checkpoint selection. As a result, the benefit of PAC’s main contribution, budget-aware labeler selection, is not totally supported.

Questions for authors
  • What are the reasons for the discrepancy between the published and reported FADRM+ results?

  • Please report results for the suggested baselines: final-teacher recovery with PAC’s selected labelers, and full-pool recovery with the final teacher as labeler. Does “SRe2L + PAC” in Table 12 change only the labeling, or also the recovery?

  • What is the overhead of computing the accuracies used for profiling (e.g., Figure 3)?

Reviewer_39vROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

This paper studies how to select teacher checkpoints in decoupled dataset distillation methods such as SRe2L and FADRM+. In these methods, pretrained checkpoints play two roles: they are used both to recover the synthetic images and to generate the soft labels. Because the preferred teacher maturity depends on the images-per-class (IPC) budget, aligning the recovery and labeling checkpoints separately for every budget would require NJ² student trainings.

The proposed method, PAC, recovers a single image bank at the largest budget (IPC 50) using a curriculum over checkpoints spanning all maturity levels, and subsamples this bank for smaller budgets. It then trains one student per labeling candidate at IPC 50. For a given target budget, PAC selects the candidate that leads at the reference epoch with matching cumulative exposure (IPC × epochs). Candidates that fall behind are terminated early by a margin-based rule, for which the authors provide guarantees under a monotone-argmax assumption (Theorem 1, Corollary 1).

PAC is implemented on top of FADRM+ and evaluated in three settings: five datasets with three student architectures, nine architectures at ImageNet-1K IPC 5, and SRe2L as a second host method. The reported gains at IPC 1 are substantial: +12.4 percentage points (pp) on CIFAR-100 and +18.1 pp on ImageNet-1K over FADRM+ with ResNet-18. On CIFAR-100, PAC matches or outperforms an exhaustive joint search while requiring roughly 1/19 of the alignment time.

Critical strengths

Strengths

  1. PAC replaces per-budget checkpoint search with a single, much cheaper procedure that performs at least as well in the reported comparison.

    • Prior work has documented that the preferred teacher maturity depends on the budget (Section 2), and aligning checkpoints separately for every budget would require NJ² student trainings.
    • PAC instead requires only one recovery run and one reference profile, and it integrates into existing pipelines (FADRM+ in Table 1; SRe2L in Table 12).
    • On CIFAR-100, it matches or exceeds exhaustive joint search (e.g., 57.7 vs. 55.7 at IPC 5) while reducing alignment time from 37.93 h to 1.96 h (Table 3).
    • The full pipeline improves over its host method: over FADRM+ across five datasets, three students, and nine architectures (Tables 1, 2), and over SRe2L (Table 12). The gains at IPC 1 are large (ResNet-18: 19.0 → 31.4 on CIFAR-100; 13.4 → 31.5 on ImageNet-1K).
    • Even though the source of these gains remains unclear (see Weakness 1), the cost reduction makes budget-specific selection practical, and the low-IPC gains are large enough to be meaningful for practitioners.
    • One caveat is that the comparison with joint search is reported only on CIFAR-100. Whether it is a like-for-like comparison also depends on whether both methods use the same recovery setup (Question 4).
  2. The selection rule is transparent, and the theoretical analysis is sound under its stated assumption.

    • Table 13 reports the full reference trajectories alongside the ground-truth selections, which makes the selection rule directly verifiable. I confirmed that applying Eq. (12) with η = 1.0 at the listed epochs exactly reproduces the reported selections.
    • I found Theorem 1 to be correct. It properly accounts for adaptive elimination, and its tightness is established in App. A.1.
    • The authors also openly acknowledge the limitations of the exposure bridge (App. A.2).
Critical weaknesses

Weaknesses

  1. The gains cannot yet be clearly attributed to the proposed checkpoint selection.

    • Relative to FADRM+, PAC changes several components simultaneously:
      • recovery (checkpoint pools drawn from four architectures; Table 16),
      • labeling (two-checkpoint ensembles with BSSL soft labels; App. C),
      • per-cell learning-rate settings S1–S4 (Tables 18–23).
    • Nevertheless, Section 4.1 states that the gains "directly demonstrate the effectiveness of our model selection technique."
    • According to the critical points reported in the paper, the selected window in many cells is Ω5, i.e., the two most mature checkpoints. This is the case for:
      • ResNet-18 at IPC 10/20/50 on CIFAR-100, Tiny-ImageNet, and ImageNet-1K (Tables 19, 21, 23);
      • ResNet-101 and MobileNet-V2 at ImageNet-1K IPC 5 (Table 23).
    • The gains in these cells (e.g., +5.8 pp and +7.3 pp in Table 2) are therefore difficult to attribute to selecting a less mature teacher for the given budget, which is the mechanism credited in Section 4.2.
    • A similar confound affects the recovery comparisons. I could not find the recovery iterations and architectures used for Joint Search and Matched. Without them, Tables 3 and 9 do not separate the benefit of spanning maturity levels from the benefit of using four architectures.
    • I also could not find an ablation that isolates the effect of selection within a fixed pipeline, as Tables 4, 6, and 9 each use a different reference configuration.
    • Consequently, the current evidence does not distinguish selection-driven gains from an alternative explanation: most of the improvement may come from the stronger recovery, ensemble labeling, and tuned training settings.
    • A fixed-pipeline ablation (Question 2) showing that selection accounts for a substantial share of the gain would largely resolve this concern.
  2. The key insight, exposure-indexed rank transfer, has limited empirical support, and its stated precondition is not satisfied in the experiments.

    • Oracle comparisons are reported only for CIFAR-100 (Tables 8, 13). There, all transitions occur by IPC 7 (Table 19), so the selection is constant for all larger budgets. Datasets whose transitions fall in the middle of the profile, such as CIFAR-10 and ImageNette with ResNet-18 (Tables 20, 22), would provide a more stringent test but are not evaluated (Question 3).
    • Even on CIFAR-100, at the epoch matched to IPC 3, the profile ranks Ω4 above Ω3, whereas the ground truth is Ω3 (Table 13, epoch 18). The reported 100% agreement therefore relies on η = 1.0, which is the agreement-maximizing value in Table 8.
    • The analysis assumes a fixed training protocol (Sections 3.2 and 3.3). However, ResNet-18 is trained with different settings at different IPCs, while profiling uses the IPC-50 setting (Table 19, App. C).
    • Moreover, matched exposure equalizes only the number of presentations. It does not equalize the number of distinct images, the number of repetitions per image, or the learning-rate state (Eq. 19).
Questions for authors

Questions

  1. Data used for selection and tuning. On which data are P̂_j(e), η, and the S1–S4 assignments evaluated: the test set or a held-out split? Were RDED, DELT, and FADRM+ run with the same S1–S4 assignments and soft-label protocol? Which baseline entries in Table 1 were re-run, and which were taken from the original papers?

  2. Isolating the effect of selection. Could the authors report a 2×2 comparison on CIFAR-100 with ResNet-18 at IPC 1, 10, and 50? The factors would be {FADRM+ recovery, PAC recovery} × {final-checkpoint labels, PAC-selected labels}, under a single fixed post-evaluation protocol, together with an "always Ω5" baseline. This would clarify how much of the gain over FADRM+ comes from checkpoint selection itself.

  3. Rank transfer beyond CIFAR-100 and under the deployed protocol. Could the authors report oracle regret on at least one dataset whose transitions fall mid-profile (e.g., CIFAR-10 or ImageNette with ResNet-18; Tables 20, 22)? Ideally, the ground truth would be obtained under the training protocol actually used at each IPC rather than the IPC-50 profiling setting.

  4. Recovery algorithm and related comparisons. Could the authors describe the recovery algorithm in more detail?

    • How are the checkpoints of the four architectures ordered?
    • How are the 2,000 iterations divided?
    • What does "Number of ARC" refer to?
    • Do Joint Search and Matched use the same recovery budget, iterations, and architectures as PAC?

    These details would help explain two results: why τ = 3 and τ = 1 differ at IPC 5 (56.7 vs. 57.7; Table 7), and why Matched outperforms Joint Search at IPC 1 (31.3 vs. 30.6; Tables 9 and 3).

Clarification

The final Ω5 entry of the reference profile (70.93, Table 13) appears to correspond to the same configuration as PAC at IPC 50 in Table 1 (71.5 ± 0.1). Could the authors explain this discrepancy (e.g., a different evaluation split, different seeds, or a different configuration)? Please also specify what ± denotes (e.g., standard deviation or standard error) and over how many runs it is computed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.