acceptodds
Under review as a conference paper at ICLR 2027

Online Continual Dataset Distillation from Evolving Learning Trajectories

Abstract

We study online continual dataset distillation, where a compact standalone dataset is constructed from a single-pass, non-stationary stream and later used to train a fresh model from scratch. Existing dataset distillation methods typically require repeated access to source data or offline expert trajectories, while prior continual-learning approaches mainly use condensed samples as replay for the learner being trained. Although stream data become unavailable after being processed, the learning trajectories they induce remain available as supervision for dataset construction. We exploit this supervision and propose BiTraC, which distills two levels of learning dynamics from a meta-continual learner. Inner trajectories capture current-task acquisition and supervise the synthesis of newly observed tasks. Outer trajectories provide complementary cross-task information to retrospectively refine earlier synthetic data. A progressive trajectory curriculum gradually incorporates later learning states during distillation. Experiments on Seq-CIFAR10, Seq-CIFAR100, and Seq-TinyImageNet show that the resulting datasets outperform retained real-data and online summarization baselines in most settings, remain effective at intermediate task boundaries, and transfer across downstream model architectures.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Dataset Distillation, Online Coninual Learning, Class Incremental Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jgSqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies online continual dataset distillation, aiming to construct a compact, standalone synthetic dataset from a single-pass, non-stationary data stream. Unlike prior dataset distillation methods that require repeated access to source data or offline expert trajectories, and unlike continual learning methods that use synthetic samples only as replay buffers, this work leverages the learning trajectories produced during meta-continual learning as the sole supervision signal.

The proposed method, BiTraC, distills two levels of learning dynamics: inner-loop trajectories, which capture current-task adaptation and guide the synthesis of new task data; and outer-loop trajectories, which provide cross-task information to retrospectively refine earlier synthetic samples. A progressive trajectory curriculum further stabilizes training by gradually exposing later learning states.

Experiments on Seq-CIFAR10, Seq-CIFAR100, and Seq-TinyImageNet demonstrate that the distilled datasets outperform real-data retention and online summarization baselines, remain effective at intermediate task boundaries, and transfer well across different downstream architectures.

Critical strengths
  1. The paper addresses a meaningful problem by combining online continual learning with standalone dataset distillation, enabling new models to be trained from compact synthetic datasets without accessing the original learner.

  2. BiTraC achieves consistent improvements across multiple benchmarks and storage budgets. Additional experiments on intermediate task performance, ablation studies, and cross-architecture generalization further support its effectiveness.

Critical weaknesses
  1. The main idea is to apply MTT-style trajectory matching to both the inner and outer loops of a meta-continual learner. While the combination is reasonable, the technical novelty beyond existing trajectory matching and meta-continual learning methods is not entirely clear. The paper would benefit from a more explicit discussion of what makes the proposed dual-level matching different from existing approaches.
  2. The paper argues that outer-loop matching helps preserve cross-task knowledge and improve historical synthetic data. However, Table 2 shows only a 0.84 percentage point improvement (49.27% to 50.11%) when outer-loop matching is added, and the ablation is limited to Seq-CIFAR10. It would be useful to see the same ablation on Seq-CIFAR100 and Seq-TinyImageNet, along with an analysis of how outer-loop matching affects the accuracy of previously learned classes.
  3. The cross-architecture experiments in Figure 3 only consider CNN-based models (ResNet-18, ConvNetD3, and PC-CNN). Since the paper aims to produce synthetic datasets that can train independent models, it would be interesting to see whether the distilled data also transfers to different architectures, such as Vision Transformers.
Questions for authors
  1. What is the main technical difference between the proposed dual-level trajectory matching and directly applying existing MTT methods to a meta-continual learner? Could the authors clarify which part of the proposed design is essential for online dataset distillation?
  2. In Table 2, outer-loop matching improves accuracy by only 0.84 percentage points on Seq-CIFAR10. Does it provide larger gains on Seq-CIFAR100 or Seq-TinyImageNet? More importantly, does it consistently improve the accuracy of previously learned classes?
  3. How well do the distilled datasets transfer to architectures beyond CNNs, such as Vision Transformers? Is the effectiveness of BiTraC dependent on the similarity between the architectures used for distillation and evaluation?
  4. How sensitive is BiTraC to the choice of the underlying meta-continual learning method? Appendix A.5 shows a noticeable performance gap between VR-MCL and La-MAML. Does this suggest that the quality of the distilled dataset largely depends on the continual learner used to generate the trajectories?
  5. BiTraC requires considerably more GPU memory than the compared baselines. How does its performance change under a more constrained memory or computation budget? Would the improvements remain significant when the methods are compared under similar resource constraints?
3 comments

1. Difference from direct trajectory matching.

BiTraC uses the MTT endpoint-matching objective, but its main technical distinction lies in how trajectory supervision is used to construct and update synthetic data in a class-incremental stream.

Directly matching the online learner's outer trajectory treats its parameter evolution as a single source of supervision. In contrast, BiTraC separates the two levels of learning dynamics according to their roles. Inner trajectories capture adaptation to newly observed classes and supervise the synthesis of current-task samples. Outer trajectories reflect learning updates informed by both current data and replay, and are used to refine historical synthetic samples that were constructed before the new classes became available. During this refinement, the student trains on the accumulated synthetic dataset, while only historical samples are updated.

To evaluate the benefit of this separation, we compare BiTraC with direct online outer-trajectory matching under the same nominal iteration budget. BiTraC achieves 50.09% versus 48.45% on Seq-CIFAR10 (IPC = 50) and 24.31% versus 23.69% on Seq-CIFAR100 (IPC = 10). These results show that assigning distinct roles to inner and outer trajectories improves upon directly applying trajectory matching to the online outer trajectory.


2. Effect of historical refinement.

We add a 6,000-iteration inner-only control to distinguish the benefit of historical refinement from the additional matching iterations in the original comparison. We also repeat the structural ablation on Seq-CIFAR100 and introduce an all-sample refinement control to examine whether restricting outer-stage updates to historical samples is beneficial.

BiTraC uses 4,000 inner and 2,000 outer matching iterations for tasks where both stages apply, while the inner-only baseline spends all 6,000 iterations on current-task matching.

VariantSeq-CIFAR10, IPC = 50Seq-CIFAR100, IPC = 10
Inner only, 6,000 iterations46.28%20.29%
Direct online outer matching, 6,000 iterations48.45%23.69%
Two-stage matching, refine all samples49.65%23.27%
BiTraC, refine historical samples50.09%24.31%

At the same nominal iteration budget, BiTraC improves over inner-only matching by 3.81 and 4.02 percentage points on Seq-CIFAR10 and Seq-CIFAR100, respectively. This shows that the improvement cannot be explained merely by increasing the number of matching iterations. Moreover, with the two-stage schedule fixed, refining only historical samples outperforms refining all samples by 0.44 and 1.04 points, respectively.

The purpose of historical refinement is to make previously distilled samples useful for the expanded classification problem, where old and new classes must be distinguished jointly. Accordingly, our primary evaluation measures joint Class-IL accuracy over all observed classes. The new ablations show that historical refinement improves this objective on both benchmarks, although the aggregate results do not separately establish the effect on each previously learned class.


3. Transfer to Vision Transformers.

Since BiTraC constructs synthetic data using ResNet-18 trajectories, an important question is whether the resulting dataset remains useful for models with different architectures. To examine transfer beyond CNNs, we evaluate independently initialized compact ViTs trained from scratch on the frozen synthetic datasets.

BiTraC achieves 36.16% on Seq-CIFAR10 (IPC = 50), compared with 34.81% for the strongest ViT baseline, and 15.68% on Seq-CIFAR100 (IPC = 10), compared with 10.24%. Because all synthetic datasets are constructed using ResNet-18 trajectories, these results show that BiTraC's advantage extends beyond the architecture used during distillation, including transfer from CNN-based trajectory matching to ViT evaluation.

4. Sensitivity to the continual learner.

BiTraC uses the learning trajectories generated by a meta-continual learner, so the choice of learner can affect the supervision available for distillation. To examine this dependence, we instantiate the same distillation procedure with two different learners, VR-MCL and La-MAML.

The two source learners differ substantially in final continual-learning accuracy, achieving 56.48% and 33.43%, respectively. However, their distilled datasets achieve 50.11% and 43.43%, a considerably smaller gap of 6.68 percentage points, compared with the 23.05-point difference between the source learners.

Furthermore, the dataset distilled from La-MAML trajectories outperforms random real-data retention (43.43% versus 41.13%), even though La-MAML itself achieves only 33.43%. This shows that useful training information can be extracted from the trajectories of a weaker continual learner and that distilled dataset quality is not simply determined by the source learner's final accuracy. The results also demonstrate that BiTraC can be applied to another meta-continual learner beyond VR-MCL.


5. Resource constraints.

We investigate whether BiTraC's accuracy advantage can be retained with lower construction-time resource requirements. Since GPU memory for differentiable trajectory matching and storage for recorded parameter states arise from different components, we evaluate them separately on Seq-CIFAR100 at IPC = 10.

First, reducing the historical-refinement context batch size from 128 to 32 lowers peak GPU memory during distillation by 58.6%, from 9.71 GB to 4.02 GB, while accuracy changes only from 24.04% to 23.92%. This indicates that the memory required for the student unroll can be substantially reduced with little effect on dataset quality.

Second, increasing the inner-trajectory recording interval from 2 to 8 meta-updates reduces the estimated peak stored-state footprint from 2.48 GB to 1.17 GB. Although accuracy decreases to 19.62%, BiTraC still exceeds the strongest competing method in the main comparison (15.52%) by 4.10 percentage points at the same final IPC.

We also report end-to-end construction time, including online learning and dataset construction. BiTraC takes 992.7 minutes, comparable to 988.8 minutes for DM under the same hardware and software environment. Together, these results quantify the resource requirements of BiTraC and show that its accuracy advantage persists under the tested reduced-resource configurations.

Dear Reviewer jgSq,

We sincerely appreciate the effort you’ve dedicated to providing constructive and insightful comments. We are eager to hear your feedback and to know whether our rebuttal properly addresses your concerns.

Thank you once again for your time and valuable response.

Best,

Authors

Reviewer_VpWpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies online continual dataset distillation, aiming to construct a compact, standalone synthetic dataset from a single pass over a non-stationary data stream for training new models from scratch. The proposed method, BiTraC, uses two types of parameter trajectories generated by a meta-continual learner: inner-loop trajectories guide current-task distillation, while outer-loop trajectories refine previously synthesized images and their soft labels. A progressive trajectory curriculum gradually incorporates later learning states. Experiments on three class-incremental image classification benchmarks evaluate the quality of the resulting datasets.

Critical strengths

The paper addresses a clear problem: preserving and reusing training information from a single-pass data stream, and assessing the resulting synthetic datasets by training new models from scratch. The method uses inner- and outer-loop learning trajectories to construct new synthetic data and refine earlier samples as new tasks arrive. It achieves the highest mean accuracy in six of seven settings, and evaluations at intermediate task boundaries and across architectures further support the reusability of the distilled data.

Critical weaknesses

The proposed method relies on a two-level trajectory design. The inner-level design is similar to existing work, while the reported results suggest that the outer level provides only modest gains at the cost of a more complex optimization procedure and higher computational costs. The paper should also describe the method more concretely and include experiments on larger datasets and higher-resolution images.

  1. The method builds on existing trajectory matching and meta-continual learning algorithms. DATM has already explored learnable soft labels and progressively expanding trajectory sampling [1]. The main contribution therefore lies in using inner trajectories for current-task distillation and outer trajectories for historical refinement. Although Table 2 includes an inner-only variant, the paper does not directly compare the method with a simpler online trajectory-matching approach using the same learner and a comparable computational budget. Such a comparison would help clarify what this division adds beyond online trajectory supervision itself.

  2. In Table 2, historical refinement raises accuracy from 49.27% to 50.11%, a gain of 0.84 percentage points, but adds 2,000 optimization iterations. This comparison does not separate the benefit of cross-task supervision from that of the extra optimization. To assess the need for this design, it would help to compare it with an inner-only baseline under the same computational budget, and to compare updating only historical data with updating all synthetic data. The component ablation is also limited to Seq-CIFAR10 at IPC=50 and should be repeated in other settings.

  3. The comparisons use the same final IPC budget but do not match the memory and computation used to construct the datasets. On Seq-CIFAR100 at IPC=10, BiTraC requires 10.78 GB of peak GPU memory, compared with 2.55 GB for DM and 1.12 GB for LoDM. Appendix A.6 reports these costs, but does not examine how accuracy changes when trajectory storage, matching iterations, or student unroll length are reduced. Whether the performance advantage persists under comparable construction budgets therefore remains unclear.

  4. The paper could explain more clearly how the meta-continual learner’s inner and outer updates relate to the two stages of synthetic-data optimization. It provides the trajectory-matching objectives and student update equations, but should also specify the exact soft-label loss and how the optimization variables are parameterized. It should explicitly describe how gradients propagate through the unrolled student updates. More detailed pseudocode or a computation graph would clarify how the matching loss updates the synthetic images, soft labels, and, during current-task distillation, the learnable student learning rate.

  5. The evaluation covers only CIFAR-10, CIFAR-100, and TinyImageNet, providing limited evidence of effectiveness and scalability on larger datasets and higher-resolution images. Because the method stores model trajectories and differentiates through multiple student updates, its resource requirements may grow substantially with model size and image resolution. Experiments in more demanding settings, reporting both accuracy and construction costs, are needed to determine whether the reported advantages can be maintained.

[1] Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching. In The Twelfth International Conference on Learning Representations (ICLR), 2024.

Questions for authors

Please refer to the weakness.

3 comments

1. Novelty and direct online trajectory matching.

Conventional trajectory matching optimizes a synthetic dataset to reproduce updates from expert trajectories. BiTraC reuses the matching loss and learnable soft labels, but addresses an additional challenge in online class-incremental learning. New classes must be represented as they arrive, while previously distilled samples need to remain useful for the expanding classification problem.

We address these two requirements using different levels of online learning dynamics. Inner trajectories supervise the synthesis of samples for newly observed classes, while replay-informed outer trajectories supervise the refinement of historical images and soft labels. During outer matching, the temporary student trains on the accumulated synthetic dataset, but only historical samples are updated. Thus, the two stages differ not only in their trajectory sources but also in which synthetic data they optimize.

To test whether this separation is beneficial, we compare BiTraC with two single-stage baselines in the same VR-MCL setting. Each baseline uses 6,000 matching iterations on one trajectory type. BiTraC allocates 4,000 iterations to inner matching and 2,000 to outer matching for tasks where both stages apply.

VariantSeq-CIFAR10, IPC = 50Seq-CIFAR100, IPC = 10
Inner-only matching, 6,000 iterations46.28%20.29%
Direct online outer matching, 6,000 iterations48.45%23.69%
BiTraC, 4,000 inner + 2,000 outer iterations50.09%24.31%

BiTraC outperforms both single-stage baselines at the same nominal iteration budget on both benchmarks. These results support our central design choice of using current-task trajectories for new-class synthesis and replay-informed trajectories for historical refinement, rather than relying on either trajectory source alone.


2. Contribution of historical refinement.

The original comparison included additional outer-stage matching iterations, leaving open whether the improvement came from historical refinement or simply from more optimization. We therefore add two controls to distinguish the contribution of outer-stage supervision from the effect of iteration count and the choice of samples to update.

First, we increase the inner-only baseline to 6,000 iterations, matching the nominal iteration budget of BiTraC. Second, we introduce a two-stage control that uses the same 4,000 inner and 2,000 outer iterations but allows outer matching to refine both historical and current synthetic samples. We also repeat the ablation on Seq-CIFAR100 at IPC = 10.

VariantSeq-CIFAR10, IPC = 50Seq-CIFAR100, IPC = 10
Inner only, 4,000 iterations45.62%19.99%
Inner only, 6,000 iterations46.28%20.29%
Two-stage matching, refine all samples49.65%23.27%
BiTraC, refine historical samples50.09%24.31%

The results support two aspects of our design. First, allocating matching iterations to replay-informed historical refinement is more effective than spending the same nominal budget entirely on current-task matching. BiTraC improves accuracy over the 6,000-iteration inner-only baseline by 3.81 and 4.02 percentage points on the two benchmarks.

Second, with the two-stage schedule fixed, restricting outer-stage updates to historical samples further improves accuracy by 0.44 and 1.04 points, respectively, compared with refining all synthetic samples. This supports the proposed separation between synthesizing new-class samples and refining historical samples using cross-task information.

3. Construction budget.

We distinguish wall-clock construction time, peak GPU memory, and trajectory storage because they capture different aspects of construction cost.

On Seq-CIFAR100 at IPC = 10, BiTraC takes 992.7 minutes for end-to-end construction, compared with 988.8 minutes for DM and 858.5 minutes for LoDM under the same hardware and software environment. Thus, although BiTraC performs differentiable trajectory matching, its end-to-end construction time is comparable to DM.

The main additional resource requirement is GPU memory. BiTraC reaches a peak of 10.78 GB during construction, compared with 2.55 GB for DM and 1.12 GB for LoDM. To investigate whether this requirement can be reduced without substantially affecting dataset quality, we vary the historical-refinement context batch size. Reducing it from 128 to 32 lowers peak GPU memory during distillation from 9.71 GB to 4.02 GB, a 58.6% reduction, while accuracy changes only from 24.04% to 23.92%.

We separately examine trajectory storage by reducing the frequency of inner-trajectory recording. Increasing the recording interval from 2 to 8 meta-updates reduces the estimated peak stored-state footprint from 2.48 GB to 1.17 GB. Although accuracy decreases from 24.04% to 19.62%, it still exceeds the strongest competing result of 15.52% at the same final IPC.

These experiments characterize the construction-time costs of BiTraC and show that its accuracy advantage can be retained under the tested reduced-resource settings.


4. Optimization details.

Both stages use differentiable trajectory matching, but differ in their reference trajectories and optimization variables. At each matching iteration, a temporary student is initialized from a recorded trajectory state and unrolled for 12 synthetic-data training updates. The normalized endpoint error measures the difference between the resulting student parameters and the recorded target state, and gradients are propagated through all 12 updates.

During current-task distillation, these gradients optimize the newly synthesized images, their soft labels, and the student learning rate. During historical refinement, the student trains on both current and historical synthetic samples, but only historical images and labels are optimized. The student learning rate is held fixed at the value learned during current-task distillation.

We have expanded Appendix A.2 to make this procedure explicit. Eq. (5) specifies the soft-label cross-entropy loss, including the parameterization and initialization of synthetic labels. Algorithm 2 further details the student unroll, gradient computation, and variables updated at each stage, complementing the revised overview in Fig. 1.


5. Scalability.

Our evaluation covers progressively larger class-incremental streams, extending from Seq-CIFAR10 and Seq-CIFAR100 to Seq-TinyImageNet. The latter contains 200 classes across 20 tasks at 64 × 64 resolution, allowing us to evaluate BiTraC with more classes, a longer task sequence, and higher-resolution images than the CIFAR benchmarks. Across the three benchmarks, we consider seven dataset–budget combinations with 1, 10, and 50 images per class.

On Seq-TinyImageNet, BiTraC improves over the strongest competing method at both budgets, from 2.54% to 4.71% at IPC = 1 and from 12.02% to 13.02% at IPC = 10. These results show that its advantage extends beyond the smaller CIFAR streams.

For larger models and datasets, the principal construction costs arise from recording parameter trajectories and differentiating through student training updates. Trajectory storage scales with the number and size of recorded states, while GPU memory also depends on the activations retained during the student unroll. We quantify these costs in our resource analysis and evaluate the effect of reducing trajectory storage and GPU memory on distilled dataset quality.

Dear Reviewer VpWp,

We sincerely appreciate the effort you’ve dedicated to providing constructive and insightful comments. We are eager to hear your feedback and to know whether our rebuttal properly addresses your concerns.

Thank you once again for your time and valuable response.

Best,

Authors

Reviewer_kL1cOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes BiTraC that distills two different learning dynamics from a meta-continual learner for online continual dataset distillation. It exploits meta-continual learner's learning trajectory; inner loop trajectory distills the current data's knowledge while the outer one complements this with earlier data's knowledge.

Critical strengths

S1. The idea to use meta-continual learner's trajectory is interesting while naive extension may require extra storage cost for buffer, which is not negligible.

S2. Performance gain is noticeable compared to DM baseline, while many DD baselines are missing such as (Guo et al., 2024).

(Guo et al., 2024) Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching, ICLR 2024

Critical weaknesses

W1. Scope of the originality : while continual dataset distillation setup is practical, it was already formulated by (Oh & Sim, 2026) which assumes continually arriving datasets. However, the current manuscript does not distinguish this clearly.

W2. Figure readability : Fig 1 is hard to read since the texts are too small. I recommend the authors to re-draw it for visibility by hand.

W3. Storage cost : The proposed method saves replay examples for meta-continual learning and historical synthetic dataset, and further inner-outer trajectory parameters, which hinders the storage efficiency of DD.

W4. Motivation of historical refinement : Since the historical refinement was updated by its own dataset already, the reason to refine this with knowledge from replay memory is not clear.

W5. Motivation of progressive trajectory curriculum : It is unclear which problem curriculum strategy aims to deal with.

W6. Lack of comparsion : The method is constructed on TM baselines, however, only being compared to DM baselines which tend to have lower performance.

(Oh & Sim, 2026) Asymmetric synthetic data update for domain incremental dataset distillation, ICLR 2026

Questions for authors

Please refer to Weakness.

4 comments

W1. Scope of originality.

We cite Oh and Sim (2026) [1] as the closest prior formulation of distillation from sequentially arriving datasets. The revised Related Work clarifies three differences.

[1]. Minyoung Oh and Jae-Young Sim. Asymmetric Synthetic Data Update for Domain Incremental Dataset Distillation. (ICLR 2026).

Oh and Sim (2026)BiTraC
Research problemDistills successive domains that share a fixed label space.Distills a class-incremental stream in which each task introduces new classes.
Access to real dataSamples from the current domain at each distillation iteration.Traverses each task once and revisits only real examples retained in bounded buffers.
Synthetic dataset updatesMaintains a fixed-size synthetic set and uses asymmetric updates to balance stability and plasticity.Adds synthetic samples for each new class and refines historical images and soft labels as the label space grows.

These differences lead to a distinct technical challenge. In a class-incremental stream, previously distilled samples must remain useful as new classes are introduced, even though the original data are no longer available. Our contribution is to address this challenge by assigning distinct roles to two levels of online learning trajectories. Inner trajectories capture current-task adaptation and guide the synthesis of examples for new classes. Replay-informed outer trajectories reflect learning updates involving both current and past data, providing supervision for refining earlier synthetic samples in the expanded label space. This allows BiTraC to incorporate cross-task information that was unavailable when those samples were first distilled.


W2. Figure readability.

We have redesigned Fig. 1 with larger labels and a clearer separation of trajectory collection, current-task synthesis, and historical refinement. The revised figure also explicitly indicates which synthetic samples participate in training and which are updated during each matching stage.


W3. Storage cost.

We distinguish two types of storage costs, since the temporary resources required to construct a distilled dataset are different from the storage required to use it for subsequent training.

  • Construction-time working storage includes the real replay buffer, the current-task initialization buffer, and the recorded inner and outer parameter trajectories.
  • Persistent distilled dataset storage consists of the final synthetic images and their soft labels. This standalone dataset is all that a fresh model needs for subsequent training.

On Seq-CIFAR100 at IPC = 10, trajectory recording accounts for most of the construction-time storage. However, the trajectory pools and initialization buffer are released after each task, so their storage does not accumulate across the stream. Once construction is complete, subsequent training requires only the final 12.70 MB distilled dataset.

We further evaluate the trade-off between trajectory storage and dataset quality. Increasing the inner-trajectory recording interval from 2 to 8 meta-updates reduces the estimated peak stored-state footprint from 2.48 GB to 1.17 GB. Although accuracy decreases from 24.04% to 19.62%, it remains above the strongest competing method in Tab. 1 at 15.52%. Thus, BiTraC retains its accuracy advantage even with substantially reduced trajectory storage.

W4. Motivation for historical refinement.

Historical synthetic data are initially optimized for the classes available at their creation time. As new classes arrive, the classification problem changes, and previously distilled samples may no longer adequately capture the distinctions needed in the expanded label space. Simply retaining these samples would leave them optimized for an earlier classification problem.

BiTraC addresses this by using replay-informed outer trajectories, which reflect learning updates involving both old and new classes. These trajectories provide cross-task supervision that was unavailable during the original distillation of historical samples. During refinement, the temporary student trains on the accumulated synthetic dataset, while gradients update only historical images and their soft labels. Consequently, earlier synthetic samples can incorporate information from newly observed classes without modifying the newly synthesized samples.

The structural ablations support this design. Compared with the 6,000-iteration inner-only baseline, allocating matching iterations to historical refinement improves final Class-IL accuracy by 3.81 percentage points on Seq-CIFAR10 and 4.02 points on Seq-CIFAR100, under the same nominal matching-iteration budget.


W5. Motivation for the progressive trajectory curriculum.

The recorded trajectories span different stages of the learner's adaptation to a task. Because the learner's parameters and acquired knowledge evolve during this process, trajectories recorded at different stages can provide substantially different matching targets. Sampling from the entire trajectory pool from the beginning therefore exposes initially unoptimized synthetic samples to widely varying targets.

Our curriculum addresses this by starting with trajectories recorded earlier in the task and progressively introducing later trajectories as distillation proceeds. This gradually expands the range of learning dynamics that the synthetic data are trained to reproduce, while keeping the matching objective unchanged.

The ablation supports the value of this progression. On Seq-CIFAR100 at IPC = 10, with both matching stages and learned soft labels held fixed, enabling both curricula improves final Class-IL accuracy from 19.71% to 21.87%.


W6. Comparison with trajectory-matching baselines.

The key question is whether BiTraC benefits from its two-stage organization or simply from applying trajectory matching to an online learner. To examine this, we add a direct online trajectory-matching baseline in Table 2 and repeat the comparison on Seq-CIFAR100 in the appendix.

This baseline directly matches the online outer trajectory using 6,000 matching iterations. BiTraC instead allocates 4,000 iterations to current-task inner matching and 2,000 to historical outer matching for tasks where both stages apply. At the same nominal iteration budget, BiTraC improves final Class-IL accuracy from 48.45% to 50.09% on Seq-CIFAR10 (IPC = 50) and from 23.69% to 24.31% on Seq-CIFAR100 (IPC = 10).

Standard MTT and DATM rely on offline expert trajectories generated with repeated access to the original dataset, which does not satisfy our single-pass data-access constraint. Our direct online matching baseline provides a protocol-compatible comparison. Its results show that separating current-task synthesis from historical refinement is more effective than directly matching the online outer trajectory alone.

Dear Reviewer kL1c,

We sincerely appreciate the effort you’ve dedicated to providing constructive and insightful comments. We are eager to hear your feedback and to know whether our rebuttal properly addresses your concerns.

Thank you once again for your time and valuable response.

Best,

Authors

Thanks for detailed response. There still remains concerns about the novelty and the extend of evaluation.

  1. Since the lifelong learner assumes infinite number of tasks, the real data buffer should be retained to further distillation: weakens the effectivness of DD, specifically on storage.
  2. the novelty is incremental: the main distinction of this work is limiited to introduction of meta-continual learner trajectory, which makes inner matching straightforward. And the proposed progressive trajectory curriculum is a different form of trajectory selection of DATM without new insight.
  3. Comparsion of TM baselines: while the authors provide comparison with outerloop only performance (48.45 on CIFAR10, 23.69 on CIFAR100), these performance outperforms every methods on Tab 1, which suggests incomplete comparison. Furthermore, the performance of that the authors provide in the response is not same as that of the paper (24.31 vs 21.87, Seq-CIFAR100, IPC=10), degrading reliability.

Overall, I therefore maintain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Thank you for the constructive feedback. The table below summarizes the main changes in the revised manuscript and the reviewer concerns they address.

Reviewer concernRevision
kL1c (W1) — Distinction from prior continual distillation workExpanded Sec. 2 to distinguish our single-pass setting with newly arriving classes from Oh & Sim’s domain-incremental setting with a shared label space.
kL1c (W2, W4) — Figure readability and motivation for historical refinementRedrew Fig. 1 to show how current-task and replay-informed trajectories provide different supervision. The figure now shows that the outer stage trains the student on both current and historical synthetic data while updating only the historical data.
kL1c (W5), VpWp (#4) — Curriculum and optimization detailsExpanded Appendix A.2 with the soft-label loss (Eq. 5), label initialization and extension to new classes, and gradient propagation through the student updates. Algorithm 2 now spells out both matching procedures, including how each stage samples from the trajectories exposed by the curriculum.
kL1c (W6), VpWp (#1–2), jgSq (Q1–2) — Direct trajectory matching and the contribution of historical refinementAdded direct online trajectory matching, an inner-only control with the same nominal iteration budget, and a comparison of refining all samples versus only historical samples. The updated Seq-CIFAR10 results are in Table 2, and the new Seq-CIFAR100 results are in Appendix A.4, Table 4.
kL1c (W3), VpWp (#3), jgSq (Q5) — Construction storage and GPU memoryAdded resource ablations in Appendix A.7.3 (Tables 9–10). Recording inner trajectories less frequently reduces estimated peak construction storage from 2.48 to 1.17 GB. Reducing the refinement context batch lowers peak GPU memory during distillation from 9.71 to 4.02 GB, with a 0.12 percentage-point decrease in final Class-IL accuracy.
jgSq (Q3) — Transfer beyond CNNsAdded a Vision Transformer trained from scratch on the frozen datasets in Sec. 5.4 and Fig. 3. BiTraC exceeds the strongest ViT baseline by 1.35 percentage points on Seq-CIFAR10 and 5.44 points on Seq-CIFAR100.

Implementation clarification. For Seq-CIFAR100 at IPC 10, the new ablations restrict the online meta-loss for current-task examples to the current task’s classes. Restoring the original loss in the new full configuration reduced final Class-IL accuracy from 24.04% to 21.90% under the same downstream training initialization seeds, close to the originally reported 21.87%.

For Seq-CIFAR10 at IPC 50, the original inner-only baseline disabled outer trajectory matching but continued to expand historical soft targets as new classes arrived. The new inner-only baseline additionally freezes each task’s soft targets at construction and assigns zero probability to classes introduced later. The two rows therefore measure different controls. The new ablation measures the combined effect of disabling outer matching and freezing historical targets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.