Profile Once, Distill Across Budgets: Budget–Checkpoint Alignment for Dataset Distillation
Abstract
*Decoupled dataset distillation* improves scalability by using pretrained checkpoints for synthetic image recovery and labeling. Its effectiveness, however, depends strongly on checkpoint maturity, with different images-per-class (IPC) budgets favoring different checkpoints. Jointly searching over recovery and labeling checkpoints at each budget requires a separate post-evaluation for every candidate pair, creating substantial alignment overhead. We introduce **P**rofile-Once **A**lignment of **C**heckpoints (**PAC**), a plug-in framework for reducing checkpoint alignment costs across IPC budgets. **PAC** replaces recovery checkpoint search with a single curriculum recovery run that uses checkpoints from all maturity levels to generate a shared image bank at the largest budget. With these images fixed, only the labeling checkpoints need to be aligned with each IPC budget, reducing the joint recovery labeling search to labeling selection. To guide this selection across IPC budgets, we identify exposure-indexed rank transfer, which links labeling checkpoint rankings at matched cumulative synthetic data exposure, defined as IPC times the number of training epochs on synthetic data. This allows us to profile labeling checkpoints only at the largest budget by tracking model performance throughout training. For smaller budgets, we sample from the recovered images and select labeling checkpoints using reference rankings at epochs matching the target budget's full-training exposure. To further reduce profiling cost, we characterize critical exposures at which checkpoint preferences change and derive sufficient conditions for permanently eliminating candidates from further evaluation. The resulting selection procedure supports previously unseen budgets within the reference range without additional recovery or profiling. Experiments across five datasets, various IPC budgets, and model architectures establish new state-of-the-art performance. At IPC 1, integrating **PAC** with FADRM+ substantially improves top-1 accuracy by **12.4%** on CIFAR-100 and by **18.1%** on ImageNet-1K.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.