Online Continual Dataset Distillation from Evolving Learning Trajectories

We study online continual dataset distillation, where a compact standalone dataset is constructed from a single-pass, non-stationary stream and later used to train a fresh model from scratch. Existing dataset distillation methods typically require repeated access to source data or offline expert trajectories, while prior continual-learning approaches mainly use condensed samples as replay for the learner being trained. Although stream data become unavailable after being processed, the learning trajectories they induce remain available as supervision for dataset construction. We exploit this supervision and propose BiTraC, which distills two levels of learning dynamics from a meta-continual learner. Inner trajectories capture current-task acquisition and supervise the synthesis of newly observed tasks. Outer trajectories provide complementary cross-task information to retrospectively refine earlier synthetic data. A progressive trajectory curriculum gradually incorporates later learning states during distillation. Experiments on Seq-CIFAR10, Seq-CIFAR100, and Seq-TinyImageNet show that the resulting datasets outperform retained real-data and online summarization baselines in most settings, remain effective at intermediate task boundaries, and transfer across downstream model architectures.