- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies online continual dataset distillation, aiming to construct a compact, standalone synthetic dataset from a single-pass, non-stationary data stream. Unlike prior dataset distillation methods that require repeated access to source data or offline expert trajectories, and unlike continual learning methods that use synthetic samples only as replay buffers, this work leverages the learning trajectories produced during meta-continual learning as the sole supervision signal.
The proposed method, BiTraC, distills two levels of learning dynamics: inner-loop trajectories, which capture current-task adaptation and guide the synthesis of new task data; and outer-loop trajectories, which provide cross-task information to retrospectively refine earlier synthetic samples. A progressive trajectory curriculum further stabilizes training by gradually exposing later learning states.
Experiments on Seq-CIFAR10, Seq-CIFAR100, and Seq-TinyImageNet demonstrate that the distilled datasets outperform real-data retention and online summarization baselines, remain effective at intermediate task boundaries, and transfer well across different downstream architectures.
Critical strengths
-
The paper addresses a meaningful problem by combining online continual learning with standalone dataset distillation, enabling new models to be trained from compact synthetic datasets without accessing the original learner.
-
BiTraC achieves consistent improvements across multiple benchmarks and storage budgets. Additional experiments on intermediate task performance, ablation studies, and cross-architecture generalization further support its effectiveness.
Critical weaknesses
- The main idea is to apply MTT-style trajectory matching to both the inner and outer loops of a meta-continual learner. While the combination is reasonable, the technical novelty beyond existing trajectory matching and meta-continual learning methods is not entirely clear. The paper would benefit from a more explicit discussion of what makes the proposed dual-level matching different from existing approaches.
- The paper argues that outer-loop matching helps preserve cross-task knowledge and improve historical synthetic data. However, Table 2 shows only a 0.84 percentage point improvement (49.27% to 50.11%) when outer-loop matching is added, and the ablation is limited to Seq-CIFAR10. It would be useful to see the same ablation on Seq-CIFAR100 and Seq-TinyImageNet, along with an analysis of how outer-loop matching affects the accuracy of previously learned classes.
- The cross-architecture experiments in Figure 3 only consider CNN-based models (ResNet-18, ConvNetD3, and PC-CNN). Since the paper aims to produce synthetic datasets that can train independent models, it would be interesting to see whether the distilled data also transfers to different architectures, such as Vision Transformers.
Questions for authors
- What is the main technical difference between the proposed dual-level trajectory matching and directly applying existing MTT methods to a meta-continual learner? Could the authors clarify which part of the proposed design is essential for online dataset distillation?
- In Table 2, outer-loop matching improves accuracy by only 0.84 percentage points on Seq-CIFAR10. Does it provide larger gains on Seq-CIFAR100 or Seq-TinyImageNet? More importantly, does it consistently improve the accuracy of previously learned classes?
- How well do the distilled datasets transfer to architectures beyond CNNs, such as Vision Transformers? Is the effectiveness of BiTraC dependent on the similarity between the architectures used for distillation and evaluation?
- How sensitive is BiTraC to the choice of the underlying meta-continual learning method? Appendix A.5 shows a noticeable performance gap between VR-MCL and La-MAML. Does this suggest that the quality of the distilled dataset largely depends on the continual learner used to generate the trajectories?
- BiTraC requires considerably more GPU memory than the compared baselines. How does its performance change under a more constrained memory or computation budget? Would the improvements remain significant when the methods are compared under similar resource constraints?
3 comments
4. Sensitivity to the continual learner.
BiTraC uses the learning trajectories generated by a meta-continual learner, so the choice of learner can affect the supervision available for distillation. To examine this dependence, we instantiate the same distillation procedure with two different learners, VR-MCL and La-MAML.
The two source learners differ substantially in final continual-learning accuracy, achieving 56.48% and 33.43%, respectively. However, their distilled datasets achieve 50.11% and 43.43%, a considerably smaller gap of 6.68 percentage points, compared with the 23.05-point difference between the source learners.
Furthermore, the dataset distilled from La-MAML trajectories outperforms random real-data retention (43.43% versus 41.13%), even though La-MAML itself achieves only 33.43%. This shows that useful training information can be extracted from the trajectories of a weaker continual learner and that distilled dataset quality is not simply determined by the source learner's final accuracy. The results also demonstrate that BiTraC can be applied to another meta-continual learner beyond VR-MCL.
5. Resource constraints.
We investigate whether BiTraC's accuracy advantage can be retained with lower construction-time resource requirements. Since GPU memory for differentiable trajectory matching and storage for recorded parameter states arise from different components, we evaluate them separately on Seq-CIFAR100 at IPC = 10.
First, reducing the historical-refinement context batch size from 128 to 32 lowers peak GPU memory during distillation by 58.6%, from 9.71 GB to 4.02 GB, while accuracy changes only from 24.04% to 23.92%. This indicates that the memory required for the student unroll can be substantially reduced with little effect on dataset quality.
Second, increasing the inner-trajectory recording interval from 2 to 8 meta-updates reduces the estimated peak stored-state footprint from 2.48 GB to 1.17 GB. Although accuracy decreases to 19.62%, BiTraC still exceeds the strongest competing method in the main comparison (15.52%) by 4.10 percentage points at the same final IPC.
We also report end-to-end construction time, including online learning and dataset construction. BiTraC takes 992.7 minutes, comparable to 988.8 minutes for DM under the same hardware and software environment. Together, these results quantify the resource requirements of BiTraC and show that its accuracy advantage persists under the tested reduced-resource configurations.
Dear Reviewer jgSq,
We sincerely appreciate the effort you’ve dedicated to providing constructive and insightful comments. We are eager to hear your feedback and to know whether our rebuttal properly addresses your concerns.
Thank you once again for your time and valuable response.
Best,
Authors
1. Difference from direct trajectory matching.
BiTraC uses the MTT endpoint-matching objective, but its main technical distinction lies in how trajectory supervision is used to construct and update synthetic data in a class-incremental stream.
Directly matching the online learner's outer trajectory treats its parameter evolution as a single source of supervision. In contrast, BiTraC separates the two levels of learning dynamics according to their roles. Inner trajectories capture adaptation to newly observed classes and supervise the synthesis of current-task samples. Outer trajectories reflect learning updates informed by both current data and replay, and are used to refine historical synthetic samples that were constructed before the new classes became available. During this refinement, the student trains on the accumulated synthetic dataset, while only historical samples are updated.
To evaluate the benefit of this separation, we compare BiTraC with direct online outer-trajectory matching under the same nominal iteration budget. BiTraC achieves 50.09% versus 48.45% on Seq-CIFAR10 (IPC = 50) and 24.31% versus 23.69% on Seq-CIFAR100 (IPC = 10). These results show that assigning distinct roles to inner and outer trajectories improves upon directly applying trajectory matching to the online outer trajectory.
2. Effect of historical refinement.
We add a 6,000-iteration inner-only control to distinguish the benefit of historical refinement from the additional matching iterations in the original comparison. We also repeat the structural ablation on Seq-CIFAR100 and introduce an all-sample refinement control to examine whether restricting outer-stage updates to historical samples is beneficial.
BiTraC uses 4,000 inner and 2,000 outer matching iterations for tasks where both stages apply, while the inner-only baseline spends all 6,000 iterations on current-task matching.
At the same nominal iteration budget, BiTraC improves over inner-only matching by 3.81 and 4.02 percentage points on Seq-CIFAR10 and Seq-CIFAR100, respectively. This shows that the improvement cannot be explained merely by increasing the number of matching iterations. Moreover, with the two-stage schedule fixed, refining only historical samples outperforms refining all samples by 0.44 and 1.04 points, respectively.
The purpose of historical refinement is to make previously distilled samples useful for the expanded classification problem, where old and new classes must be distinguished jointly. Accordingly, our primary evaluation measures joint Class-IL accuracy over all observed classes. The new ablations show that historical refinement improves this objective on both benchmarks, although the aggregate results do not separately establish the effect on each previously learned class.
3. Transfer to Vision Transformers.
Since BiTraC constructs synthetic data using ResNet-18 trajectories, an important question is whether the resulting dataset remains useful for models with different architectures. To examine transfer beyond CNNs, we evaluate independently initialized compact ViTs trained from scratch on the frozen synthetic datasets.
BiTraC achieves 36.16% on Seq-CIFAR10 (IPC = 50), compared with 34.81% for the strongest ViT baseline, and 15.68% on Seq-CIFAR100 (IPC = 10), compared with 10.24%. Because all synthetic datasets are constructed using ResNet-18 trajectories, these results show that BiTraC's advantage extends beyond the architecture used during distillation, including transfer from CNN-based trajectory matching to ViT evaluation.