acceptodds
Under review as a conference paper at ICLR 2027

Trajectory Replay: Continuous-Time Mixture Transport for Lifelong Generative Models

Abstract

Adapting pre-trained continuous-time generative models to sequentially incoming data without forgetting earlier tasks is a central challenge in lifelong learning. A leading strategy, generative replay, uses a generative model trained on historical tasks to augment its own continual update with synthetic samples from experience. However, it requires full trajectory simulations at every step that accumulate discretization errors and can degrade the historical distribution. Here, we propose *trajectory replay*, which bypasses sample generation by training an updated velocity field to match both a frozen historical velocity along its trajectories and the incoming task's velocity field, weighted by a stability-plasticity coefficient. Through the lens of continuous-time transport, we show the resulting optimum averages the two task velocities by their respective densities and generates the mixture of their probability paths. Combined with mean velocity distillation, trajectory replay accesses historical trajectories using only a few network evaluations per continual update. Empirically, it outperforms generative replay and distillation baselines on class-incremental image generation. In two biological case studies, trajectory replay extends generative perturbation models to new contexts and experimental batches across two data modalities while preserving treatment-effect prediction.

Primary area:
generative models
Keywords:
flow matching, mixture modeling, continual learning, generative replay, catastrophic forgetting

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_mPkDOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies continual learning for flow-matching generative models, aiming to learn new tasks while preserving earlier ones without replaying stored training samples. It is motivated by the cost of generating replay samples and the errors that can accumulate when these samples are used for later updates. The proposed trajectory replay method trains the new model to match a frozen historical model at intermediate points along its generation trajectories. It combines this supervision with learning from the incoming data. A MeanFlow extension reduces the sampling work needed to obtain these intermediate states. The paper provides a theoretical interpretation of how the update balances old and new tasks. Experiments cover synthetic distributions, class-incremental image generation, single-cell perturbation modeling, and microscopy batch streams, reporting improvements in sample quality, retention, and replay efficiency.

Critical strengths

I find the method intuitive and well motivated. It addresses both forgetting and the cost of replay by learning from intermediate teacher states instead of fully generated samples. The analysis explains how old and new tasks are balanced under ideal assumptions. The experiments support improved retention and sample quality across several settings, including when replay weights follow a simple rule without validation-based tuning.

The evaluation combines controlled simulations with image generation and biological applications, allowing the method to be assessed beyond a single benchmark. The reported runtime comparisons support a practical benefit over online generative replay. The MeanFlow experiments also show that trajectory replay can often use fewer sampling steps while maintaining similar performance. This is useful when teacher sampling is expensive, although the additional training overhead means MeanFlow is not always faster.

Critical weaknesses
  1. The current comparisons leave some uncertainty about what produces the gains. The method changes both how replay states are obtained and how the student learns from the teacher. However, the closest distillation baseline uses a different generative framework. The theory explains the intended mixture under ideal assumptions, but does not establish which change accounts for the practical improvement. The replay methods also allocate different proportions of each batch to old and new tasks. A comparison that controls these choices would make the contribution of trajectory replay clearer.

  2. The microscopy evaluation does not directly establish whether the model preserves the response to each perturbation. The reported scores compare image distributions after pooling across perturbations. These scores would not change if generated images were reassigned to different perturbations within the same experimental batch. This limits the evidence for preserving treatment responses, although the results still support improved retention of batch-level image distributions. The latest batch also shows slightly worse sample quality than generative replay in the reported comparison, despite better retention of older batches. This trade-off should be discussed alongside the overall improvement.

Questions for authors

1.Could you add a flow-matching version of generative distillation that generates teacher samples, adds noise back to them, and matches the teacher’s velocity predictions? Keeping the teacher, training setup, replay weighting, and old/new sample allocation matched would help isolate the benefit of sampling along the teacher’s trajectories. Please compare sample quality, retention, and actual training time under these matched conditions.

2.Could you provide an evaluation that checks whether generated microscopy images reflect the intended perturbations, beyond matching the overall image distribution? A measure that detects incorrect perturbation labels would address this concern. Otherwise, please narrow the microscopy conclusion to the distributional retention supported by the current evaluation.

3.Could you clarify which data are used to select the adaptive replay weight in the transcriptomics experiments? The methods refer to training strata, while the table captions refer to validation. Please explain whether these are separate data and what historical information must remain available. The paper already allows old-task validation data, so the question is which conclusions apply when that information is unavailable.

5 comments
AuthorsOpenReview ↗

Rebuttal comment 1

We thank mPkD for taking the time to review our submission. We appreciate the overall positive feedback and the valuable suggestions to improve the quality and breadth of our contribution. Changes are highlighted in red in the updated manuscript.

W1a. Clarification on batch allocation

Before presenting the additional experiments, we would like to clarify one point. Generative distillation already uses the same even old/new split with loss weighting as trajectory replay, so batch allocation differs only for generative replay, which we control for in W1b.

W1b / Q1. Isolating the source of the gains

We ran the requested controlled comparisons.

  • Matched flow-matching distillation. To remove the difference in generative framework, we implemented generative distillation in our flow-matching setup: full samples from the frozen teacher are renoised to time tt, and the student is regressed onto the teacher velocity at these states. Teacher, backbone, trainer, adaptive λ\lambda selection and 50/50 batch split are shared with trajectory replay, so the two methods differ only in how supervision states are obtained. Trajectory replay achieves lower final stream FID, which covers all tasks seen so far and thus reflects retention, on all three datasets (21.1 vs. 24.6 on Fashion-MNIST, 31.4 vs. 34.5 on CIFAR-10, 70.0 vs. 73.0 on ImageNet-100), with 1.3–1.5× fewer GPU hours per update (new Fig. 19).
  • Seed-level comparison. The error bars in Fig. 19 partly overlap because each seed defines a different task stream, so the spread across seeds mainly reflects differences in stream difficulty rather than in the methods. Compared on the same stream, trajectory replay has lower FID than matched distillation in all nine runs (three datasets × three seeds).
  • Matched batch allocation for generative replay. We additionally reran generative replay with the same 50/50 old/new batch split as trajectory replay, applying λ\lambda as a loss weight rather than as the fraction of the batch drawn from the teacher. On Fashion-MNIST, this leaves generative replay essentially unchanged (66.5 vs. 63.9 FID, 0.540 vs. 0.526 KLD). On CIFAR-10, it improves generative replay only moderately (78.9 vs. 92.4 FID, within standard errors, and 0.592 vs. 0.737 KLD). On both datasets, generative replay remains far from trajectory replay (22.1 and 38.6 FID, 0.068 and 0.237 KLD), so the allocation does not account for the gap (new Table 8).

Paper edits. We report the matched distillation comparison (new Fig. 19) and the batch-allocation control (Table 8) in App. D.3.2, and summarize the former in Sec. 4.3.

AuthorsOpenReview ↗

Rebuttal comment 2

W2a / Q2. Perturbation-level fidelity on RxRx1

We thank the reviewer for this suggestion. Our pooled evaluation follows Zhang et al. (2025) and Jones et al. (2026), generating one image per test condition so that generated and reference sets share the same perturbation-by-batch composition. We agree that pooled scores are invariant to reassigning images across perturbations, and so establish retention of batch-level distributions rather than of individual treatment responses.

We therefore compared the final checkpoints of trajectory replay and generative replay per perturbation (new Table 15). In each batch (U2OS-01, -02, -03), we generated images for the 100 evaluation perturbations and the negative control, and compared them with real test images of that batch in DINOv2 space, using two measures that collapse under reassignment:

  • Effect retrieval. Each perturbation's effect is its mean feature minus the batch's control mean. We report top-5 accuracy of retrieving the real effect among the 100 perturbations by cosine similarity, against a within-batch label-shuffle null (chance 0.05).
  • Conditional KID between each perturbation's generated images and real images of the same (matched) or other (mismatched) perturbations, averaged over perturbations. KID is unbiased at these small sample sizes (about 32 generated and 50 real images), unlike FID.

Both methods preserve perturbation-specific signal: retrieval lies far above the null for every seed (p<0.005p < 0.005), and generated images are always closer in KID to their own perturbation than to others. Trajectory replay, however, preserves it better on the older batches, with higher retrieval (0.40 vs. 0.30 on U2OS-01, 0.27 vs. 0.23 on U2OS-02) and lower KID (2.47 vs. 4.06 and 1.54 vs. 2.28), while the two methods are on par on the latest batch, U2OS-03 (0.40 vs. 0.38 retrieval, 1.76 vs. 1.75 KID).

Paper edits. We added the perturbation-level evaluation to App. D.6 (Table 15).

W2b. Trade-off on the latest batch

We agree this should be discussed. On the newest batch (U2OS-03), trajectory replay trails generative replay by 1–2 FID points (28.6–28.8 vs. 27.0–27.7; Table 14), while retaining earlier batches far better: about 30 FID points on U2OS-01 and 4 on U2OS-02, for a 5-point gain on the pooled stream. Since the newest data is available during the update, naive fine-tuning, generative replay and trajectory replay all fit it within a narrow range (26.0–28.9 FID, close to joint training's 26.6), whereas retention is where they differ. We view this small cost as the stability–plasticity trade-off inherent to replay. At the perturbation level, the two methods are on par on this batch (Table 15; see W2a).

Paper edits. We now discuss the latest-batch trade-off in App. D.6.2.

Q3. Data used for adaptive replay weight selection

We thank the reviewer for catching this inconsistency. Adaptive λ\lambda is selected on validation cells held out from each training stratum, i.e., cells from the same context–perturbation categories seen during training that are not used to train the model. The term "training strata" in App. D.5.1 and Table 9 refers to these categories rather than to the training cells themselves.

Regarding historical information, adaptive selection therefore requires held-out cells from past tasks, used only to score candidate weights and never for training. When no historical data is available, the equal-task schedule λk=k/(k+1)\lambda_k = k/(k+1) applies, as it requires no data at all, and the corresponding rows of Tables 11 and 12 show the results that hold in that setting.

Paper edits. We clarified in App. D.5.1 and Table 9 that selection uses held-out validation cells.

Dear Reviewer mPkD,

Thank you again for your detailed suggestions. Before the discussion period closes on November 18 (AoE), we wanted to briefly follow up. We ran the controlled comparisons you asked for (matched distillation in Fig. 19, batch allocation in Table 8) and a perturbation-level evaluation on RxRx1 (Table 15).

We would be grateful to hear whether this resolves your concerns, and are happy to clarify anything further.

The Authors

Reviewer_aMf8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Trajectory Replay, a continual learning (CL) method for continuous-time generative models that avoids full sample generation during replay by regressing an updated velocity field against (i) the frozen historical field along its own generative trajectories and (ii) the incoming task’s conditional velocity (CFM), weighted by a stability–plasticity parameter λ. The authors prove that the population optimum of this trajectory-level objective is a density-weighted average of task velocities that generates the mixture of their probability paths, and introduce a MeanFlow-based variant that enables replay using only a few network evaluations. Empirically, Trajectory Replay improves the fit–retention trade-off over generative replay/distillation baselines on synthetic mixtures, class-incremental image generation, and two biological applications, while reducing computational cost.

Critical strengths

1) Technical novelty and innovation: Poses continual generative replay as a trajectory-level regression problem over velocity fields, avoiding full sample generation and shifting replay to model-space supervision. Provides a clean optimality characterization: the minimizer is a density-weighted average of velocities that exactly transports the mixture of probability paths; connects to a weighted terminal KL objective via continuous-time likelihood control. Extends the analysis and algorithm to mean velocities (MeanFlow), yielding a practical sampler that amortizes historical trajectory access across updates with few NFEs.

2) Experimental rigor and validation: Multi-scale evaluation: controlled 2D mixtures, multi-task ring benchmark, class-incremental image generation on FMNIST/CIFAR-10/ImageNet-100, and two biologically relevant case studies (single-cell perturbation modeling and microscopy). Reports both fit (FID) and retention (class-mass KLD, energy distance) as well as domain-specific metrics in biology; includes runtime comparisons and analysis of solver vs. MeanFlow trade-offs.

3) Significance of contributions: Addresses a key bottleneck in generative CL for continuous-time models: expensive and error-prone replay via full trajectory simulation.

Critical weaknesses

1) Technical limitations or concerns: The main theoretical results assume conditional Gaussian paths with shared schedules (αt, σt), which narrows scope; while common in flow matching, extensions to broader settings (e.g., differing schedules across tasks, non-Gaussian conditionals) are deferred to the appendix or future work. Although the method “bypasses sample generation” at the terminal time, it still requires partial historical simulation (or a learned MeanFlow); the claim could be more carefully framed to avoid overstatement. Reliance on a frozen teacher for historical states means errors compound across long streams; mitigation strategies or bounds are mentioned but not quantified.

2) Experimental gaps or methodological issues: While diffusion-based baselines are included, the proposed method is instantiated primarily in the FM/ODE setting; a native stochastic (SDE) version of Trajectory Replay is not empirically validated.

3) Missing related work or comparisons: Recent mean-flow and joint trajectory/likelihood distillation works (e.g., Mean Flow Distillation; F2D2) are pertinent to the MeanFlow backend and could be discussed/compared more explicitly to position the contribution

Questions for authors
  1. How sensitive is training to regions where λpt(x)+(1−λ)p′t(x)λ p_t(x) + (1 − λ) p′_t(x) is very small? Did you observe numerical instabilities in v∗t’sv*_t’s density-weighted averaging, and do you apply any floor/clipping heuristics?

  2. Can you provide empirical results for a stochastic (SDE/diffusion) instantiation of Trajectory Replay, beyond diffusion-based baselines, to validate the “stochastic extension” claim?

  3. Does the theoretical mixture result still hold if the old and new tasks use different schedules (αt,σtαt, σt)? If not, can the method be adapted (e.g., reparameterizing time or using schedule-aligned conditionals)?

  4. How does the method behave when incoming tasks introduce new conditioning variables (e.g., new classes) vs. shifts within known conditions? Are there differences in stability–plasticity behavior?

5 comments
AuthorsOpenReview ↗

Rebuttal comment 1

We thank aMf8 for the review and the positive assessment of our contribution's novelty and quality. We address each point below, with changes in the revised manuscript highlighted in red.

W1a. Gaussian paths and shared schedules

  • Non-Gaussian and task-specific paths. We present our framework with conditional Gaussian paths to interpret the objective as a weighted terminal KL to the historical and incoming tasks (Thm. 3.2). However, the characterization of its optimum (Prop. 3.1) holds for arbitrary conditional paths and schedules, which can even differ across tasks. Its proof (App. A.6) uses neither the Gaussian structure nor the schedule, only a shared prior, positivity and the stated regularity, so any pair of probability paths can be chosen.
  • Schedules are shared by design in practice. The schedule of each continual update is a design choice and can always be set to match the historical model's. We do so in all experiments, so Thm. 3.2 applies directly. W1b covers the case where the historical schedule cannot be matched.

W1b / Q3. Restoring a shared schedule across updates

If the historical model was trained with a different Gaussian schedule (αs′,σs′)(\alpha_s^{\prime}, \sigma_s^{\prime}), the shared-family setting of Thm. 3.2 can be recovered in closed form. The key observation is that a noisy sample αtx1+σtx0\alpha_t x_1 + \sigma_t x_0 is determined, up to an overall scale, by its signal-to-noise ratio αt2/σt2\alpha_t^2/\sigma_t^2. Two schedules therefore describe the same distribution at matching noise levels, differing only by a rescaling and by the time at which that noise level is reached. Let s(t)s(t) be the time at which the historical schedule reaches the same signal-to-noise ratio as the update's schedule at time tt, and c(t)=αt/αs(t)′c(t) = \alpha_t/\alpha_{s(t)}^{\prime} the corresponding scale. The historical field b′b^{\prime} then converts to the update's schedule as bt(x)=c˙(t)c(t)x+c(t)s˙(t)bs(t)′(xc(t)),b_t(x) = \frac{\dot c(t)}{c(t)} x + c(t) \dot s(t) b_{s(t)}^{\prime}\left(\frac{x}{c(t)}\right), which generates the same historical distribution, so that Thm. 3.2 applies.

Paper edits. We added the derivation and assumptions in App. A.16.

W1c. "Bypassing sample generation"

We agree the phrasing can be more precise. Trajectory replay never generates complete samples to use as training data. It does, however, require partially simulating the historical model up to time tt, as acknowledged in our Limitations paragraph. This partial simulation needs nearly half the teacher evaluations of full generation in expectation (App. A.13) and yields end-to-end speedups of 1.22–1.65× over generative replay (Fig. 13).

Paper edits. We rephrased the abstract and introduction to state that trajectory replay avoids full sample generation by training on partially simulated historical trajectories.

W1d. Error compounding

  • Errors can accumulate across updates for any method that learns from a frozen teacher. They split into (i) sampler error, from simulating the historical model, and (ii) fitting error, since each update's mistakes are inherited by the next.
  • Trajectory replay reduces the first source. Generative replay trains on simulated samples, so their flaws become part of what the student learns. Trajectory replay instead queries the teacher at whichever state the simulation reaches, so a slightly displaced state still receives the teacher's own answer (App. A.13). This displacement is less than half that of full generation in expectation (Eq. 66).
  • Empirical quantification. Across updates, generative replay drifts while trajectory replay stays roughly level or matches real replay with stored data (Figs. 11, 14, 27a and 31a).
  • Open directions. Fitting error still accumulates, as for any teacher-based method. Regressing onto several retained past teachers is a promising direction for future work.

W1e. Bound on accumulation across updates

Let q1,…,qKq_1, \dots, q_K be the task distributions, pkp_k the terminal distribution of the model after kk tasks, and μk=1k∑j=1kqj\mu_k = \frac{1}{k}\sum_{j=1}^{k} q_j the ideal balanced mixture. Each update fits mk+1=λkpk+(1−λk)qk+1m_{k+1} = \lambda_k p_k + (1-\lambda_k) q_{k+1}, whose historical component is the learned pkp_k rather than the ideal μk\mu_k. Measuring the per-update error εk+1=TV(pk+1,mk+1)\varepsilon_{k+1} = \mathrm{TV}(p_{k+1}, m_{k+1}) and the accumulated error δk=TV(pk,μk)\delta_k = \mathrm{TV}(p_k, \mu_k) in total variation, with λk=k/(k+1)\lambda_k = k/(k+1) the triangle inequality gives δk+1≤εk+1+λkδk\delta_{k+1} \le \varepsilon_{k+1} + \lambda_k \delta_k, which unrolls to δK≤∑k=1KkKεk≤K+12max⁡kεk,\delta_K \le \sum_{k=1}^{K} \frac{k}{K} \varepsilon_k \le \frac{K+1}{2} \max_k \varepsilon_k, so accumulated error grows at most linearly in the number of tasks. The bound holds for any teacher-based method, which differ only through εk\varepsilon_k: for generative replay, it includes sample corruption that further training cannot remove.

Paper edits. We added the result and its proof in App. A.17.

AuthorsOpenReview ↗

Rebuttal comment 2

W2 / Q2. Stochastic trajectory replay

We implemented the stochastic instantiation of App. A.10. The teacher is rolled out with an Euler–Maruyama discretization of the marginal-preserving SDE, whose drift adds the score correction of Prop. A.1 to the teacher velocity, with diffusion coefficient gg (g=0g = 0 recovers the ODE). The rollout uses the same time grid as the ODE, so teacher cost is unchanged, and the student is trained with the unchanged objective at the SDE states.

  • Controlled mixture. As predicted by Prop. A.1, SDE replay recovers the prescribed mixture weight as accurately as the ODE, with errors between 0.01 and 0.03 for all solvers and all λ∈{0.1,…,0.9}\lambda \in \lbrace 0.1, \dots, 0.9 \rbrace (new Fig. 8, Table 3).
  • Images. With moderate noise (g=0.5g = 0.5), stochastic replay matches the ODE in FID on Fashion-MNIST (23.7 vs. 22.1), at a higher KLD, and improves on it on CIFAR-10 in both FID (34.6 vs. 38.6) and KLD (0.069 vs. 0.237). Larger noise (g=1.0g = 1.0) reduces quality on Fashion-MNIST (31.2 FID). All stochastic variants remain well ahead of generative replay (63.9 and 92.4 FID; new Table 7).

Paper edits. We added the controlled results to App. D.1 (new Fig. 8 and Table 3) and the image results to App. D.3.2 (new Table 7).

W3. Missing related work or comparisons

We thank the reviewer for the pointers. Mean-flow distillation methods convert a pretrained flow model into a few-step flow map, and F2D2 (Ai et al., 2026) jointly distills sampling and likelihood evaluation into a single flow map. Both target faster inference for a fixed model on a single task. In our setting, the mean-velocity model instead serves as a replay sampler that is updated across tasks: our MeanFlow backend is obtained by distilling the velocity field after the first task (App. D.3.1), and any improved flow-map distillation scheme, including these, could be used as a drop-in backend. We therefore view these works as complementary building blocks rather than direct competitors.

Paper edits. We now discuss mean-velocity distillation in the related work, positioning our mean-velocity models as replay samplers updated across tasks.

Q1. Sensitivity to low-density regions

  • The density-weighted average is never computed explicitly. It characterizes the minimizer of our objective (Prop. 3.1) but is not evaluated during training. The network is regressed onto the teacher velocity at states sampled from ptp_t and onto the conditional velocity at states sampled from pt′p_t^{\prime} (Eqs. 7 and 12), so no densities are evaluated and no division occurs.
  • No numerical instabilities, and no floor or clipping. Since there is no ratio to compute, there is nothing to stabilize. The weighting emerges implicitly from how often each region is sampled.
  • Low-density regions behave as in standard flow matching. They simply receive little supervision. Moreover, for t<1t < 1 both paths are convolved with Gaussian noise, so λpt+(1−λ)pt′\lambda p_t + (1-\lambda) p_t^{\prime} is strictly positive everywhere and the positivity assumption of Prop. 3.1 holds automatically.

Q4. New conditioning variables

Our experiments cover new conditioning values as well as shifts with mostly shared conditions, and trajectory replay performs well across them:

  • New modes without conditioning (Fashion-MNIST, CIFAR-10): new classes appear as new mass, and trajectory replay achieves the best non-joint FID and class-mass recovery (Table 1).
  • New conditioning values (class-conditional ImageNet-100): each task adds new labels, and trajectory replay again achieves the best non-joint FID (Table 1).
  • A new value of one variable, others shared (new cell lines, donors or experimental batches with a shared perturbation vocabulary; Sec. 4.4, App. D.6): on RxRx1, trajectory replay retains the oldest batch far better than generative replay (39.6 vs. 71.4 FID). The synthetic experiment of App. D.4 isolates this case.

A pure shift within known conditions is not covered exactly. There, by Prop. 3.1, the model learns a λ\lambda-weighted mixture of the old and new conditional distributions, so λ\lambda controls whether the update retains or overrides previous behavior. Regarding stability–plasticity, adapting only new condition embeddings cannot express a new context: the velocity field itself must change, which is where forgetting arises (App. D.5).

Dear Reviewer aMf8,

We hope our response has been helpful, and wanted to follow up before the discussion period ends on November 18 (AoE). Following your suggestions, we added the stochastic variant of trajectory replay (Fig. 8, Tables 3 and 7), the schedule conversion (App. A.16) and an error-accumulation bound (App. A.17). We would be grateful to hear whether this resolves your concerns, and are happy to clarify anything further.

The Authors

Reviewer_8qYzOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a flow-based adaptation algorithm of a base model for continual learning. The algorithm includes a training formulation and a new loss schedule that balances learning of new tasks and forgetting of old tasks. The authors evaluate the method on low dimensional examples, an established class-incremental image generation protocol and a biological application.

Critical strengths
  • The schedule of the λk\lambda_k seems to be a critical design choice, and was motivated well.
  • The extensive evaluation of the method is good and the appendix provides many details.
Critical weaknesses
  • It is unclear how the proposed method differs from previous work. Section 3 introduces the theory behind the method but does not make clear what is new. It is also not clear what is different in the resulting Algorithms (Appendix C) compared to the Algorithm 1 of the cited baseline of Masip et. al. 2025 aside from the flow versus diffusion model formulation and a different λ\lambda scheduling.
  • The authors emphasize how important the λ\lambda hyper parameter is for the balance of fitting a new task and forgetting an old task, hence the proposed equal-task/adaptive weights λk\lambda_k. It is even explicitly stated that a "constant is rarely appropriate, as task weights then decay geometrically with age rather than balancing". However, the generative distillation baseline (Masip et. al. 2025) is relatively close in performance to the proposed method, while having a constant λ\lambda across tasks and time. An ablation study for a constant schedule and/or a schedule-matched comparison to the baselines would be helpful to understand the contribution of each design choice.
  • The paper is motivated by the diffusion discretization error, where "Approximation errors compound along these trajectories, degrading replay fidelity and, in turn, subsequent CL updates.". The citation of Gao & Liu 2023 however says exactly the opposite in Appendix G: "To figure out the effect of diffusion steps, we conduct experiments and present the results with numbers of different diffusion steps in Table G.3. Overall, the gap of the final average accuracy and forgetting rate between different N is small". As this seems to be a main motivation for the paper, it needs to be shown that ODE discretization error really is an issue, especially with the vast amount of post-training samplers available that are optimized for few-step sampling. It is also not clear why one would train a standard flow matching model and then fix discretization error during the adaptation phase, for example through the mean flow, instead of directly training a flow model incentivized to learn straight trajectories.
Questions for authors
  • What is the marginal contribution of each design choice to the overall performance?
  • How important is the adaptivity of the task schedule?
  • How does the ODE discretization error affect the continual learning objective?
5 comments
AuthorsOpenReview ↗

Rebuttal comment 1

We thank 8qYz for the constructive feedback and the positive assessment of our experimental setup. We address each point below, with changes in the revised manuscript highlighted in red.

W1a. Differences from previous work

Both trajectory replay and generative distillation (Masip et al., 2025) regress the updated model onto a frozen teacher, but differ in where the supervision states come from:

  • Generative distillation generates clean samples by simulating the teacher from noise to data, re-noises them to time tt, and matches the student to the teacher at the re-noised inputs (L5, Alg. 1, Masip et al.).
  • Trajectory replay (ours) never generates clean samples. It simulates the teacher only up to time tt and matches velocities at the states reached there.

Both converge to the same λ\lambda-mixture in the idealized setting (Prop. 3.1). In practice, they differ in:

  1. Cost. Partial rollouts in trajectory replay need nearly half the teacher evaluations of full generation in expectation (App. A.13).
  2. Error. In both methods, solver error only changes where the student is supervised, since the target is always the teacher's prediction. Trajectory replay, however, simulates only up to time tt, so it accumulates less of this error and never depends on the quality of generated clean samples (App. A.13). Relying on such samples is costly: coarse discretization and few-step sampling degrade generative replay but not trajectory replay (Table 1, Figs. 17a and 20; see W3b).

Paper edits. We name generative distillation in the introduction, state the difference in the related work and Sec. 4.3, and add App. A.15 comparing the three replay strategies.

W1b. Additional contributions

Our work also contains novel theoretical contributions:

  • Replay as mixture transport. We formulate the continual update as transporting a λ\lambda-mixture of historical and incoming probability paths, which, to our knowledge, is new for continual learning of continuous-time generative models. Its minimizer is the density-weighted average field, which generates exactly this mixture for arbitrary paths (Prop. 3.1) and controls a weighted terminal KL (Thm. 3.2).
  • Continual updates of flow maps. We extend mixture-based continual learning to few-step MeanFlow models, showing that regressing weighted consistency residuals along both task paths (Eq. 11) recovers the mixture's mean velocity (Prop. 3.3), so that a single few-step model generates both historical and incoming data.

This justifies replaying only partial trajectories.

W1c. Experiment: matched setups

To also control for the generative framework, we implemented generative distillation in our flow-matching setup, with the same teacher, training setup, adaptive λ\lambda and batch split as trajectory replay. Trajectory replay still achieves lower FID on all three datasets (21.1 vs. 24.6 on Fashion-MNIST, 31.4 vs. 34.5 on CIFAR-10, 70.0 vs. 73.0 on ImageNet-100) with 1.3–1.5× fewer GPU hours per update. Since each seed is a different task stream, we also compared seed by seed: trajectory replay has lower FID in all nine runs.

Paper edits. We report the matched comparison in App. D.3.2 (new Fig. 19) and summarize it in Sec. 4.3.

W2a. The baseline's schedule

We apologize for the lack of clarity. Masip et al. (2025) do not use a constant weight, but a scaled version of our equal-task schedule: at task kk, they weight the replay and incoming terms by c⋅k/(k+1)c \cdot k/(k+1) and 1/(k+1)1/(k+1), so that c=1c = 1 recovers our schedule exactly. To reproduce the baseline results (Table 1), we use the same weighting with c=0.75c = 0.75, as in their work.

Paper edits. While our submission did not claim a constant schedule, we agree the weighting was unclear and now state it explicitly in Sec. 4.3 and App. A.15.

W2b. Experiment: baseline's schedule

To remove the schedule as a confound, we reran trajectory replay with the baseline's exact weighting (three seeds). Under identical settings, our model improves mean FID on CIFAR-10 (33.2 vs. 38.5) and is slightly behind on Fashion-MNIST (27.2 vs. 25.5). It substantially improves class balance on both datasets (KLD 0.077 vs. 0.290 on Fashion-MNIST, 0.109 vs. 0.230 on CIFAR-10), so it comes out ahead overall at the same schedule.

Paper edits. We report the schedule-matched comparison in App. D.3.2 (new Table 6).

W2c. Constant λ\lambda claim

A constant λ\lambda discounts older tasks geometrically with their age instead of balancing them (App. A.12). On the ring benchmark, no constant value reaches the balanced target (Fig. 10a). To further support this, we ablated constant λ\lambda on Fashion-MNIST (new Fig. 21): fixing it at 0.5 or 0.75 raises FID from 22.1 (equal-task) to about 30.5–33.5 and triples the class-mass KLD.

Paper edits. We report the constant-λ\lambda ablation on Fashion-MNIST in App. D.3.2 (Fig. 21).

AuthorsOpenReview ↗

Rebuttal comment 2

W2d. Similar performance to baseline

The two methods are comparable only on CIFAR-10 (38.5 vs. 38.6 FID under each method's own schedule, 38.5 vs. 33.2 under matched schedules; see W2b). Elsewhere, generative distillation is consistently behind: on ImageNet-100 it is on par with generative replay (85.5 vs. 88.4 FID) and far from trajectory replay (70.0), on Fashion-MNIST its class-mass KLD is more than four times higher (0.29 vs. 0.068), and on RxRx1 it is the weakest replay method on all metrics (46.4 vs. 16.8 pooled FID; App. D.6).

W3a. Motivation: Gao & Liu

Our statement describes a two-stage effect: (i) simulating full trajectories accumulates errors within each generated sample, and (ii) since these samples are used as training data, their errors are inherited by later updates. Gao & Liu is cited for the second stage, showing that reusing generated samples makes later samples drift from previous tasks (Sec. 3.2, Fig. 1). Their App. G only varies between 1000 and 4000 diffusion steps, where discretization error is negligible, and so does not speak to the few-step regime, where differences between methods are largest (new Fig. 20).

Paper edits. We revised the introduction and abstract to separate the two stages, citing Gao & Liu for the second.

W3b / Q3. Sampler error and ODE discretization

(i) Errors in generated samples degrade replay. We added a sweep over the number of ODE steps on Fashion-MNIST (new Fig. 20). Generative replay is highly sensitive to discretization (64 FID at the default 10 steps, 125 at 5). Trajectory replay instead remains stable and consistently better, and generative replay does not reach its quality even at 100 steps. This matches the mechanism in App. A.13: generative replay fits discretization error as training data, whereas trajectory replay only queries the teacher at intermediate states.

(ii) Few-step samplers do not circumvent it. A few-step MeanFlow map has larger sampling error than an ODE solve, making it a stress test for sampler error. Generative replay degrades with few-step sampling, whereas trajectory replay is essentially unchanged and matches its ODE run across replay budgets (Fig. 17a).

W3c. Straight trajectories

The MeanFlow backend is not a discretization fix: it only makes reaching intermediate states cheaper at the next update (Algorithm 2), and trajectory replay works equally with the ODE backend (Table 1). Straight-trajectory models would also make sampling cheaper, but would not remove the need for trajectory replay:

  • Straightness is not preserved across updates. Each update targets a mixture of tasks, whose marginal flow is generally curved even if each task's flow is straight. Re-straightening after every update, e.g. by reflow, requires fully simulating the model, precisely the cost replay seeks to avoid.
  • Residual error is still fit as data. Straightening reduces sampling error but never removes it, and generative replay trains on whatever is left. It stays behind trajectory replay even with many ODE steps or a few-step generator (Fig. 20, W3b(ii)).

Q1. Marginal contribution

  • Retention mechanism. Adapter and regularization baselines stay far behind replay (65–68 vs. 21.1 FID on Fashion-MNIST; Table 5).
  • Regression target. Regressing onto the teacher instead of fitting generated samples as data accounts for most of the gain: FID drops from 35.0 to 24.6 on Fashion-MNIST, 43.3 to 34.5 on CIFAR-10 and 88.4 to 73.0 on ImageNet-100 (generative replay vs. matched distillation; W1c).
  • Supervision states. Replaying partial trajectories instead of renoised samples further lowers FID (24.6 → 21.1, 34.5 → 31.4, 73.0 → 70.0) at 1.3–1.5× lower cost (W1c).
  • Batch allocation. Giving generative replay the same even old/new split does not close the gap (78.9 vs. 38.6 FID on CIFAR-10; new Table 8).
  • Schedule. The schedule alone does not explain the gap: under the baseline's exact weighting, trajectory replay still improves class balance (W2b), and a constant λ\lambda performs worse (W2c).
  • Replay backend. One-step MeanFlow leaves trajectory replay unchanged (70.0 → 70.7 FID on ImageNet-100) but degrades generative replay (88.4 → 151.0; Table 1).

Q2. Adaptivity

  • Trajectory replay. Adaptive weighting matches or improves on equal-task weighting on every dataset: 21.1 vs. 22.1 FID on Fashion-MNIST, 31.4 vs. 38.6 on CIFAR-10, 70.0 vs. 70.4 on ImageNet-100 (Table 1) and 16.8 vs. 17.2 on RxRx1 (Table 14).
  • Generative replay. Adaptivity matters much more: equal-task weighting raises FID from 35.0 to 63.9 on Fashion-MNIST and from 43.3 to 92.4 on CIFAR-10 (Table 1). The adaptive policy compensates for low-fidelity replay by down-weighting generated samples (Figs. 16 and 32a).

When all tasks should be equally represented, the equal-task weights are optimal (Prop. 3.1), and the weights selected for trajectory replay indeed closely track them (Figs. 10b, 16 and 32a).

Dear Reviewer 8qYz,

With the discussion period closing on November 18 (AoE), we wanted to check in on our response. To address your main concern about the comparison with Masip et al. (2025), we added a matched flow-matching version of generative distillation (W1c, Fig. 19) and a schedule-matched comparison (W2b).

We would be grateful to hear whether this resolves your concerns, and are happy to clarify anything further.

The Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

General rebuttal comment

Dear AC, SAC, and reviewers,

We would like to thank everyone for the time invested in handling our submission. We believe the suggestions and feedback provided during the reviewing phase have contributed to improving our work's clarity and comprehensiveness.

We addressed all comments in our individual responses and uploaded a revised manuscript, with changes highlighted in red. The main changes are summarized below.

New experiments

  • Matched flow matching distillation (8qYz, mPkD; App. D.3.2, Fig. 19): generative distillation implemented in our flow matching setup with identical teacher, training setup, weighting and batch split. Trajectory replay achieves lower FID on all three image benchmarks at 1.3–1.5× lower cost.
  • Schedule-matched comparison with generative distillation (8qYz; App. D.3.2, Table 6).
  • Constant-λ\lambda ablation on Fashion-MNIST (8qYz; App. D.3.2, Fig. 21).
  • ODE-step sweep for generative and trajectory replay (8qYz; App. D.3.2, Fig. 20).
  • Stochastic (SDE) trajectory replay on the two-mode mixture and image streams (aMf8; App. D.1 and D.3.2, Fig. 8, Tables 3 and 7).
  • Batch-allocation control for generative replay (mPkD; App. D.3.2, Table 8).
  • Perturbation-level evaluation on RxRx1 (mPkD; App. D.6, Table 15).

Text and theory

  • Explicit comparison of generative replay, generative distillation and trajectory replay (App. A.15), with the distinction and the baseline's weighting stated in the introduction, related work and Sec. 4.3 (8qYz, mPkD).
  • Closed-form conversion between Gaussian schedules (App. A.16) and a bound on error accumulation across updates (App. A.17) (aMf8).
  • Revised motivation, separating within-sample error from its propagation across updates (8qYz), and more precise framing of "avoiding full sample generation" (aMf8).
  • Discussion of mean-velocity distillation in the related work (aMf8), the latest-batch trade-off on RxRx1 (App. D.6.2), and the data used for adaptive λ\lambda selection (App. D.5.1) (mPkD).

We look forward to engaging with the reviewers during the discussion period.

The Authors

Area_Chair_nbyrOpenReview ↗

Review discussion

Dear reviewers,

the authors have responded to your reviews. Can you please check their responses and engage in a discussion with them to clarify any open questions or help them address your concerns. I also strongly recommend you check the reviews and answers of your fellow reviewers and take them into consideration during the discussion so that we can preferably reach a consensus at the end of the review period.

Thank you, your AC

AuthorsOpenReview ↗

Summary of the discussion phase

Dear AC, SAC and reviewers,

As the discussion period closes, we briefly summarize its outcome.

Reviewer 8qYz

Changes. We clarified the difference from generative distillation (App. A.15), added a matched flow matching version of generative distillation (Fig. 19), a schedule-matched comparison (Table 6), a constant-λ\lambda ablation (Fig. 21) and an ODE-step sweep (Fig. 20), and revised the motivation.

Outcome. The reviewer confirmed that these resolved their concerns and kindly raised their score.

Reviewer aMf8

Changes. We added a closed-form schedule conversion (App. A.16), a bound on error accumulation across updates (App. A.17), stochastic trajectory replay experiments (Fig. 8, Tables 3 and 7) and a discussion of mean-velocity distillation.

Outcome. The reviewer confirmed that these addressed their concerns.

Reviewer mPkD

Changes. We ran all requested controlled comparisons, namely a matched flow matching distillation baseline (Fig. 19) and a batch-allocation control (Table 8), added a perturbation-level evaluation on RxRx1 (Table 15), discussed the latest-batch trade-off (App. D.6.2) and clarified the data used for adaptive λ\lambda selection (App. D.5.1).

Outcome. The reviewer confirmed that these substantially addressed their concerns and raised their score.

We thank all reviewers and the AC for their time and constructive feedback.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.