acceptodds
Withdrawn from ICLR 2027

From Local Condensation to Global Utility: Replay-Enhanced Continual Dataset Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Dataset distillation typically assumes that the entire training dataset is available simultaneously. In this work, we study *Continual Dataset Distillation* (CDD), a new setting in which class groups arrive sequentially and a synthetic memory must be constructed and maintained without retaining historical real samples. The resulting synthetic memory is evaluated by training fresh models on all observed classes, requiring synthetic subsets distilled at different steps to remain compatible as the label space expands. We identify a key failure mode in CDD, which we term *Local-Global Discrimination Collapse* (LGDC). Synthetic subsets distilled independently within individual class groups may remain effective for local classification, yet lose compatibility after aggregation for global classification. To address this issue, we propose *Replay-Enhanced Continual Dataset Distillation* (ReCDD), which replays historical synthetic samples during the distillation of new classes and refines the historical memory as the label space expands. To provide a consistent competition signal when future classes are unavailable, ReCDD introduces a non-semantic auxiliary reference that is discarded before evaluation. Experiments on four datasets and five representative methods spanning gradient and trajectory matching show that ReCDD improves final global accuracy in most evaluated settings. Further diagnostic analyses show that ReCDD reduces the local-global discrepancy and cross-step errors associated with LGDC.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Dataset Distillation, Dataset Condensation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DZFqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies Continual Dataset Distillation (CDD), where class groups arrive sequentially without access to past real data, and the maintained synthetic memory is evaluated by training a fresh classifier on all observed classes. It identifies Local–Global Discrimination Collapse (LGDC), measured by LGGap and CSGap, and proposes ReCDD, which replays and refines historical synthetic samples and adds a noise-initialized auxiliary class during condensation. ReCDD is applied to DC, DSA, MTT, FTD and DATM on MNIST, CIFAR-10/100 and Tiny ImageNet, and improves over sequential baselines in most settings.

Critical strengths

(1) The CDD setting is clearly defined, and comparing CSGap against a Joint control is a reasonable way to isolate cross-step interference.

(2) ReCDD serves as a plug-in for both gradient- and trajectory-matching methods, with consistent gains at IPC=10/50 on smaller datasets.

Critical weaknesses

(1) Limited novelty. LGDC is a restatement of inter-task confusion, a well-known issue in class-incremental learning: classes from different tasks are never trained together and are therefore confused at test time. CSGap likewise reuses the standard comparison between predicting with and without task identity. The method components also have clear precedents in works the paper itself cites: joint distillation of current data with previous memory in DDN (Liao et al., 2025), and refinement of stored exemplars across phases in Mnemonics (Liu et al., 2020). The paper does not explain how ReCDD differs from them methodologically.

(2) The experiments do not support the paper’s narrative. In the ablation (Table 4), the auxiliary reference alone reaches 45.4% on CIFAR-10 versus 45.6% for the full model, so replay and refinement, the core of the method, contribute little; the auxiliary reference itself is left unexplained. The diagnostics also overstate the problem: on CIFAR-10, Joint condensation already has a CSGap of about 36 pp versus about 42 pp for sequential distillation, so most cross-step confusion is inherent to the task. Figure 1 omits this Joint control. Finally, no competing method, such as DDN or LoDM (Yang et al., 2023), and no simple global fine-tuning baseline is compared.

Minor: The paper gives no background on the five base methods and omits backbone details. ConvNet appears only in the caption of Figure 3; its depth, the training schedule and cross-architecture results are not reported, which limits readability and reproducibility.

Questions for authors
  1. How does LGDC differ from inter-task confusion in class-incremental learning?
  2. Why does the auxiliary reference help, and why is Frozen replay about 10 pp worse than no replay?
  3. Figure 1 reports 78.0% global accuracy, well below the Table 3 mean for the same setting (85.5 ± 2.3%). Is it a single run or a selected class order?
  4. For MTT/FTD/DATM, how are expert trajectories built at each step, and are they retrained per step?
  5. Were sequential baselines tuned under the same budget as ReCDD? Some degrade at larger IPC. What is ReCDD’s computational overhead?
Reviewer_aj2gOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes 'replay-enhanced continual dataset distillation' to address the problem of local-global discrimination collapse. It replays and refines the synthetic memory to faithfully constructs all the historical label space. It also introduces a non-semantic auxiliary reference, which compensate the lack of unseen classes during early distillation phase.

Critical strengths

S1. Clear motivation : The motivation of LGDC is clear; locally condensed images are less discriminative than global one.

S2. Simple and intuitive : The proposed method is simple and easy to understand.

Critical weaknesses

W1. Scope of originality : Continual dataset distillation is already formulated before by [R1], while it limits to domain-incremental setting but shares a common problem; sequentially arriving datasets. The authors should clarify the scope of originality.

W2. Lack of comparison with existing CIL methods : There are many class-incremental learning methods, however, none of these is not compared or discussed properly.

W3. Increasing memory : The memory budget linearly increases according to the number of classes, which conflicts to the efficiency of DD.

W4. Lack of novelty : The method of Sec 4.1 indicates just involvement of replay samples which is conventional and naive in continual learning field.

W5. Lack of analysis : Analysis on proposed components, implementation details are not sufficient.

[R1] Asymmetric synthetic data update for domain incremental dataset distillation, ICLR 2026

Questions for authors

Q1. Can we expand this to domain-incremental setting? If so, comparison with [R1 at Weakness] under that setting would strengthen the superiority of the proposed method.

Q2. Can the authors compare some representative CIL methods which can be applied to CDD framework?

Q3. Can the authors provide concise analysis of the impact of learning rate decay?

Q4. In Tab 3, A_local and A_intra seems not strongly correlated with performance. For example, ReCDD-DC even reduces A_local but has better A_global. Can the authors explain why?

Q5. Can the authors visualize reference images and analyze the impact of KK?

Q6. Performance gain on trajectory matching families tend to be small, especially on Tiny-ImgNet IPC=10/50. Can we know why?

Please also refer to Weakness.

Reviewer_fQAaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies continual dataset distillation in a class-incremental stream, where historical real data cannot be retained and the accumulated synthetic memory is evaluated by training a fresh classifier over all observed classes. The authors identify Local-Global Discrimination Collapse (LGDC), in which independently condensed subsets retain local utility but become less effective after aggregation. They propose Replay-Enhanced Continual Dataset Distillation (ReCDD), which replays and refines historical synthetic samples and uses a non-semantic auxiliary category during condensation. Experiments across four datasets and five base condensation methods generally improve final global accuracy over naive sequential condensation, with additional diagnostics and component ablations.

Critical strengths

S1. Meaningful problem formulation. The paper clearly distinguishes continual dataset distillation from conventional continual learning: the objective is to maintain a reusable synthetic dataset whose utility is measured by training a fresh global classifier. This is a relevant and practically motivated setting.

S2. Broad empirical evaluation and useful diagnostics. The study covers four datasets, five condensation methods, and three IPC budgets. Separating local utility, global utility, and cross-step competition offers a useful diagnostic view of why independently condensed memories may not work well after aggregation.

Critical weaknesses

W1. Limited methodological novelty. ReCDD mainly combines synthetic-memory replay, cross-entropy refinement of historical samples, and an auxiliary optimization class. Although this combination is effective in several settings, the paper does not clearly establish a new distillation principle or sufficiently differentiate the method from closely related continual-memory approaches beyond the problem formulation and evaluation protocol.

W2. Limited matched comparisons and statistical evidence. The main baseline is naive sequential condensation, with no direct comparison to the closest continual synthetic-memory methods under the same fresh-classifier protocol. Most Joint results are imported from prior work, and only three class orders are evaluated without paired confidence intervals or significance tests. The reliability of several small or negative gains is therefore unclear.

W3. Insufficient evidence for the proposed mechanism and component generality. CSGap is partition-dependent and can also be nonzero for Joint condensation, so it does not establish that LGDC mitigation causes the gains. The ablation is also limited to DC at IPC=10 on two datasets under one fixed order.

Questions for authors

Q1. Can the authors compare ReCDD with the closest continual synthetic-memory methods, such as adapted versions of DDN, Remember the Past, or LoDM, under the same fresh-classifier evaluation and memory budget? If adaptation is not feasible, which incompatibilities prevent a matched comparison?

Q2. Can the authors report paired per-order differences or confidence intervals for ReCDD versus the sequential baselines, and reproduce the principal Joint references within the same implementation and evaluation pipeline?

Q3. How sensitive are LGDC and ReCDD to the number of classes per step? Would the conclusions remain unchanged under different task granularities or controls that preserve the additional auxiliary output and optimization budget without using the proposed auxiliary mechanism?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.