acceptodds
Under review as a conference paper at ICLR 2027

Trajectory Replay In Sequential Flow Matching: Concentration and Coverage

Abstract

Recent work on sequential flow matching shows that the infinite-capacity empirical velocity field is a kernel regressor over stored transitions, approaching nearest-neighbour trajectory replay as its weights concentrate. We characterise when this occurs and identify which modelling choices control it. For Gaussian conditional paths, we characterise the onset of replay, its behaviour in high dimensions, and how it depends on the geometry of the stored transitions and the choice of conditional path. We further show that, under constant-in-flow-time covariance, the empirical field remains within the convex hull of the stored velocity targets, imposing a representation floor that cannot be overcome by reweighting. Experiments on chaotic systems validate the predicted concentration and coverage behaviour, while finite-capacity experiments show that neural networks can depart from the empirical minimiser and thereby mitigate replay. Together, these results distinguish replay concentration from target coverage, showing that controlling replay alone does not guarantee accurate prediction.

Primary area:
generative models
Keywords:
flow matching, sequential data, trajectory replay, kernel regression, high-dimensional concentration

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_feqKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper builds on a pivotal work of Lim et al. (2026), who showed that the infinite-capacity minimizer of the sequential flow matching objective is a Nadaraya-Watson kernel regressor over stored transitions, which can collapse to nearest-neighbor "trajectory replay".

For Gaussian conditional paths, this paper studies two questions: (i) concentration (when the posterior weights αj(t,x)\alpha_j(t,x) become one-hot) and (ii) target coverage (whether convex combinations of the stored targets can represent the true velocity).

For concentration, they show that for a query drawn from the path of window ii, the log-weight gap to window jj is exactly N(κij/2,κij)\mathcal{N}(\kappa_{ij}/2, \kappa_{ij}) with κij=∥μt(i)−μt(j)∥2/σt2\kappa_{ij} = \|\mu_t^{(i)} - \mu_t^{(j)}\|^2/\sigma_t^2 (Theorem 1), that replay onset requires κ\kappa of order log⁡M\log M (Proposition 1), and that since κ≈dVk(t)/σt2\kappa \approx d V_k(t)/\sigma_t^2, replay occurs as d→∞d \to \infty at a fixed bandwidth (Theorem 2). Coupling and interpolant only enter through Vk(t)V_k(t) (Proposition 2), the score objective has the same weights (Proposition 3), and a bandwidth σt2∝da\sigma_t^2 \propto d^a gives replay, non-degenerate and uniform regimes for a<1a<1, a=1a=1 and a>1a>1 (Theorem 3). An anisotropic covariance replaces the Euclidean distance with a Mahalanobis distance, and the equal-trace optimum Σ∝Γ1/2\Sigma \propto \Gamma^{1/2} replaces dd with an effective rank (Propositions 4 and 5). For coverage, when the covariance is constant in flow time, vt⋆(x)v_t^\star(x) lies in the convex hull of the stored targets, which gives an error floor Dt(x)D_t(x) that reweighting cannot remove (Theorem 4).

Experiments on Lorenz-63/96 with the closed-form estimator show that ESS curves collapse onto κ^\hat\kappa, that the onset scales with log⁡M\log M, that the hull floor bounds the best achievable error, and that anisotropic and isotropic kernels mostly perform alike at matched κ\kappa. A small MLP experiment shows that narrower networks deviate more from v⋆v^\star and can have lower off-support error.

Critical strengths
  1. The separation between concentration and coverage is an effective way to frame the problem. Theorem 4 and the remark that concentration is not itself a performance objective make a clearer point: methods that only reduce replay cannot fix a deficient target bank. This is a helpful perspective for the growing literature on sequential and multi-marginal flow matching.

  2. The characterization of replay onset is well-constructed and precise, and the synthetic experiments results agree. The exact Gaussian law of the log-weight gap gives a single parameter κ\kappa that absorbs the dimension, window order and interpolant, and the experiments confirm the predicted collapse, the d−1/2d^{-1/2} relative concentration, and the log scaling. The derivations are mathematically correct (e.g., Proposition 1(ii) follows a union bound with z≈2log⁡(n/δ)z \approx \sqrt{2\log(n/\delta)}, which gives Gj≥8log⁡(n/δ)G_j \ge 8\log(n/\delta)).

Critical weaknesses

1. The novelty is limited.

The main concentration result is essentially a known collapse result, and the paper does not position against it.

(a) Compared with Lim et al. (2026): that paper already derives the closed-form kernel-weighted minimizer (Lemma 1 in this paper), notes that it is a mixture of per-transition velocities, decomposes the drift into a velocity mixture and a score term, and states that the weights converge to hard nearest-neighbor assignments as the bandwidth goes to zero. What is effectively new in this submission is the quantitative part.

(b) Compared with Biroli et al. (2024): their collapse analysis studies the same object, namely the posterior weights of a Gaussian mixture centered at stored points, evaluated at noisy versions of a stored point. This is the same query law as in Theorem 1 and its corresponding Proposition 1. Using a Random Energy Model argument, they obtain a sharp collapse time tC=12log⁡(1+σ2/(n2/d−1))t_C = \frac{1}{2}\log\left(1 + \sigma^2/(n^{2/d} - 1)\right) (Equation 10). In the notation of this paper, the collapse condition becomes κC=2d(n2/d−1)\kappa_C = 2d(n^{2/d} - 1), which tends to 4log⁡n4\log n when log⁡n≪d\log n \ll d. This is the same log⁡M\log M onset as Proposition 1 and the same d/σ2d/\sigma^2 scaling as Theorems 2 and 3, with a sharp constant. Prop. 1 only places the transition between 2log⁡M2\log M (necessity) and 32log⁡(M/δ)32\log(M/\delta) (sufficiency), and the remark after Prop. 1 explicitly does not identify a critical constant, even though Fig. 2(c) plots a 4log⁡M4\log M reference line. Biroli et al. is cited only in a list of memorization papers, and this connection is not discussed.

(c) Key results are closely related to the immediate studies (Proposition 3, Theorem 4, and Proposition 5). What remains "novel" is a reasonable yet incremental extension. For a theoretical paper, I consider this the most important weakness.

2. The analysis covers the regime that is least relevant for inference, and the practical implications are not tested.

(a) All concentration results assume that the query is generated from a stored component. In this regime replay is the expected outcome. During inference, ODE trajectories move away from the training law, and this off-support regime is where replay vs. generalization matters. The authors acknowledge this critical distinction as a future work.

(b) The coverage result requires σt′=0\sigma_t' = 0. Many sequential FM use time-varying noise, so the floor does not apply to them. This includes the Gaussian bridge path of Lim et al. (2026), on which this paper builds upon.

(c) There is no forecasting experiment showing that the recommendations derived from the theory improve prediction, such as σt2∝d\sigma_t^2 \propto d (in Theorm 3, a=1a=1), effective-rank covariance shaping, or a larger target bank. All evidence comes from Lorenz-63/96 with closed-form estimators, plus one MLP experiment, in which, as the authors already acknowledge, capacity and optimization effects are not separated.

Weakness 1 determines my recommendation. Since the contribution is mainly theoretical, its novelty over Lim et al. (2026) and the diffusion collapse / memorization work has to be clearly established, and the current manuscript does not do this. Weakness 2 further limits the practical significance.

Questions for authors
  1. What exactly is new relative to Lim et al. (2026)? In particular, rewriting the collapse time of Biroli et al. (2024) in your notation gives κC=2d(n2/d−1)→4log⁡n\kappa_C = 2d(n^{2/d}-1) \to 4\log n, i.e., the same log⁡M\log M onset and d/σ2d/\sigma^2 scaling as Prop. 1 and Thm. 2, with a sharp constant. Which of your concentration results are not implied by, or are strictly stronger than, that analysis? And why does Prop. 1 not recover the sharp constant?

  2. Can any of the concentration results be extended, at least heuristically or empirically, to queries along generated ODE trajectories, e.g., by measuring ESS along actual sampling/rollout paths instead of training-law queries?

  3. Do the regimes predicted by Thm. 3 (specially this: σt2∝d\sigma_t^2 \propto d) or the Σ∝Γ1/2\Sigma \propto \Gamma^{1/2} shaping lead to better multi-step forecasts (rollout error, CRPS) for trained sequential FM models on Lorenz-96 or a real dataset?

  4. For commonly used time-varying paths (σt′≠0\sigma_t' \neq 0, e.g., Brownian bridge noise), v⋆v^\star is no longer restricted to the convex hull. Can you characterize the reachable set in this case, or show empirically how much the floor DtD_t is relaxed?

  5. In Fig. 5, can you separate capacity from optimization, e.g., by tracking ∥vθ−v⋆∥\|v_\theta - v^\star\| during training for both widths, or by adding wider networks and longer training? (This is just out of curiosity, and for the extension-ability.)

Suggestions: the main text is too dense and several figure labels are hard to read. Moving some propositions (e.g., Props. 2-3) to the appendix and adding a worked example relating κ\kappa to bandwidths used in practice would help.

References (note)

Lim, S. H., Lin, S., Mahoney, M. W., & Erichson, N. B. (2026). Is Flow Matching Just Trajectory Replay for Sequential Data?. arXiv preprint arXiv:2602.08318.

Biroli, G., Bonnaire, T., de Bortoli, V. et al. Dynamical regimes of diffusion models. Nat Commun 15, 9957 (2024). https://doi.org/10.1038/s41467-024-54281-3

4 comments
AuthorsOpenReview ↗

Response to Reviewer feqK

We thank the Reviewer for the detailed assessment and for recognising the distinction between concentration and target coverage. We particularly appreciate the feedback on off-support queries and the effects of time-varying covariance on coverage, both of these have strengthened our paper. We clarify some of the comments regarding novelty, scope, and interpretation of our results below; all changes to the paper are in blue text in the revised pdf.

1. Novelty relative to Lim et al. (2026) and Biroli et al. (2024) (W1 and Q1)

  • Lim et al. (2026) establish the kernel-regression representation of the minimiser and its nearest-neighbour limit as bandwidth vanishes. We build on this work by quantifying the dimension and memory bank dependence for the replay onset. We further characterise this replay dependence on window geometry and covariance specifically, finally showing that replay is not a sufficient factor for performance, introducing target coverage as a separate variable for consideration.

  • We agree that Biroli et al. (2024) establish a sharper threshold for Gaussian-mixture concentration. Fig 2(c) in our paper demonstrates logarithmic scaling, while the onset constants vary with bank geometry and the ESS criterion, and do not strictly follow the constant in Biroli et al. (2024). Our result is thus a broader but less sharp characterisation that suits the use case of sequential data. We have amended Prop. 1 from our work to position it as giving finite memory bank, geometry-dependent concentration bounds conditional on arbitrary stored path centres, allowing correlated and overlapping sequential windows without assuming independent Gaussian centres. We have also made this positioning more explicit in the related works.

  • Beyond this mechanism, Propositions 2-5 characterise the effects of multi-marginal interpolation, score regression and covariance geometry. Theorem 4 introduces the separate target-coverage constraint, demonstrating why preventing replay need not improve prediction. These constitute the main extensions beyond the above concentration analysis. We thank the reviewer for bringing this to our attention.

2. Off-support queries and inference (W2 and Q2)

  • The Gaussian law in Theorem 1 assumes a query generated from a stored component, but the posterior weights in Equation (1) remain well-defined at any query, including during inference.

  • To empirically check the effects on replay, we implemented the Reviewer's suggestion to evaluate ESS along generated trajectories of the closed-form field and both trained MLPs on Lorenz-96, from training-law, held-out and off-support initial states. The results are in Appendix B.8, Figure 9 in the revised paper. Training-law starts exhibit near-complete concentration, whereas held-out and off-support starts initially have higher ESS. During autoregressive rollouts, trajectories move progressively farther from the stored bank; network trajectories remain closer than those of the empirical minimiser and exhibit more concentrated weights. These observations support the distinction between weight concentration and accurate prediction off-support. They do not establish a universal off-support onset law, which would require characterising the generated query distribution.

  • For a theoretical starting point, we can assume that for a query (xx), which follows a distribution qtq_t, its expected log-weight gap (modified from Thm. 1) is the difference between the KL divergences from qtq_t to the two conditional components considered: Eq[Gij]=KL(qt∣∣pj)−KL(qt∣∣pi),E_q[G_{ij}]=KL(q_t||p_j)-KL(q_t||p_i), provided both divergences are finite. This provides a nice starting point for analysing concentration under generated query distributions, however, it does not itself establish replay along an ODE trajectory, since the distribution need not coincide with any training component. We leave the complete analysis of this to future work.

3. Multi-step forecasting performance (W2 and Q3)

We do not claim that Theorem 3's bandwidth regimes or Proposition 5's covariance necessarily improve multi-step forecasting. These results characterise and minimise concentration, respectively, not forecasting error. Indeed, Sections 6.2–6.3 show that indiscriminate averaging and poor target coverage can persist even after replay is mitigated. The theoretical results identify mechanisms for controlling replay, not universally optimal forecasting configurations. We will ensure this distinction is made more explicit in the revision.

4. Coverage under time-varying covariance (W2 and Q4)

The fixed convex-hull floor in Thm. 4 requires constant covariance wrt flow time. However, as the Reviewer correctly pointed out, noise schedules can vary with time. For a shared time-varying isotropic path, writing bt=σ˙t/σtb_t=\dot\sigma_t/\sigma_t we have ut(j)(x)=μ˙t(j)+bt(x−μt(j)).u_t^{(j)}(x)=\dot\mu_t^{(j)}+b_t(x-\mu_t^{(j)}). Consequently, Lemma 1 gives vt∗(x)∈btx+conv{μ˙t(j)−btμt(j):j=1,…,M}.v^*_t(x)\in b_tx+\mathrm{conv}\lbrace\dot\mu_t^{(j)}-b_t\mu_t^{(j)}:j=1,\dots,M\rbrace. Thus a translated, path-dependent convex-hull constraint remains. Unlike Theorem 4, the reachable set now depends on the noise schedule. We note that btb_t is unchanged when the whole schedule is rescaled i.e. σt↦sσt\sigma_t\mapsto s\sigma_t, so changing the bandwidth alone (the parameter controlling concentration) does not move the floor. This has been added as Corollary 1 to the main text.

5. Capacity versus optimization (Q5)

  • We conducted additional ablations across widths 64-1024, training budgets of 12.5k-200k steps and weight-decay strengths. The results are in Appendix B.7, Figure 8. Wider networks consistently approach the empirical minimiser more closely, and the observed width dependence persists across the tested budgets and regularisation settings. Wider networks also exhibit higher off-support error in the width sweep. These experiments establish robustness of the observed width dependence, though they do not prove a converged capacity-only effect. We thank the Reviewer for this comment as it has allowed us to strengthen the paper.
  • A natural theoretical formulation to separate width and training time would combine a mean-field limit with Wasserstein gradient-flow training dynamics, examining whether the width and training-time limits commute. This remains a separate theoretical question that we will focus on for future work.

6. Presentation

We appreciate the comment on density and figure readability. We have attempted to make the figures more readable. We will ensure that the paper is made more readable in further revisions. We have also noted the comment on including a table for kappa for commonly used bandwidth regimes, it is now in Appendix A.2, Table 1

References:

Thank you authors for very detailed (and timely!) response. I especially appreciate clarification of the novelty that I pointed out earlier, and after thorough reading, I believe the authors have rather clearly established a solid ground beyond the existing stream of literature.

With that said, I also appreciate the accommodation of my earlier readibility concerns.

I have therefore adjusted my assessment.

We thank the Reviewer for engaging with our response and updating their assessment.

Reviewer_shgHOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies when sequential flow matching relies almost entirely on a single stored training transition. It starts from a previously established result: with unlimited model capacity, the optimal solution combines stored conditional velocity targets using similarity weights. The authors analyze when these weights concentrate on one transition and how this depends on dimension, dataset size, interpolation, and noise. The paper also separates two possible problems: assigning poor weights to the available targets, and having targets that cannot produce the required answer regardless of their weights. Experiments on Lorenz systems examine these effects. An additional experiment compares neural networks with two different widths.

Critical strengths

The distinction between weighting and available targets is useful. Averaging more training examples does not necessarily improve prediction. If the stored targets cannot represent the required motion, changing their weights cannot fully solve the problem. The paper makes this distinction clear.

The main mathematical results have clearly stated assumptions. The analysis explains how distances between stored transitions and the noise level affect the weights. It also connects the onset of concentration to the number of stored transitions. The authors distinguish these results for the ideal solution from observations about trained networks.

The experiments examine the proposed explanations directly. Evaluating the closed-form solution avoids mixing the main concentration experiments with network training difficulties. The additional network experiment is useful because it shows that a trained model does not necessarily behave like the ideal solution.

Critical weaknesses

In Figure 4, the comparison matches the average concentration measure. Two settings with the same average can still assign very different weights to individual targets. Reporting effective sample size and the largest weight for the evaluated queries would help establish whether the comparison actually holds replay behaviour constant. Without this check, the remaining error differences cannot be cleanly separated from differences in weight concentration.

Appendix A.4 considers the optimal solution with positive noise and then allows the noise level to approach zero. This does not establish what a network trained directly without noise will learn away from the training paths. The appendix itself notes that predictions at those locations are not determined by the training objective. Its conclusion about the behaviour of methods trained on deterministic interpolants therefore needs further justification.

Questions for authors

See Critical Weaknesses section

1 comment
AuthorsOpenReview ↗

Response to Reviewer shgH

We thank the Reviewer for the constructive and positive assessment and for recognising the distinction between concentration and target coverage. All changes made to the revised pdf are in blue text. We address the concerns below:

1. Matched concentration in Fig. 4 (W1)

  • We agree that matching the average concentration parameter does not imply identical posterior weight distributions. After conducting the empirical test suggested by the Reviewer, we conclude that on well-conditioned systems, shape effects are largely inconsequential. However, on strongly anisotropic systems, there is residual error, and thus the effects of concentration cannot be separated cleanly. Fig. 4(a) matches average Mahalanobis separation, rather than ESS or the complete responsibility distribution.
  • Checking the ESS comparison for the queries shows that on relatively isotropic systems, matching average concentration already provides a good approximation: for Lorenz-96 (d = 64), all 48 comparisons lie within the ±10% error-ratio band under either matching criterion.
  • For strongly anisotropic systems, differences remain. On L96-F4-cos, the count increases from 19/48 under matched concentration to 29/48 under matched ESS. On L96-F4-bump, 17/29 valid ESS-matched comparisons lie within the band. The differing denominators reflect operating points where ESS matching is unavailable.
  • Thus, average concentration explains much of the observed behaviour in well-conditioned systems, but does not fully control posterior weights in strongly anisotropic settings. Residual prediction-error differences cannot be attributed exclusively to covariance geometry.

Table: Prediction-error agreement under matched concentration and ESS

Systemr_eff/dMedian absolute log ESS ratioMatched kappa (±10%)Matched ESS (±10%)
L96-640.920.0848/4848/48
L96-160.920.1447/4848/48
L96-64-F50.760.2248/4848/48
L630.820.0833/4839/47
L96-64-F4-cos0.321.6919/4829/48
L96-64-F4-bump0.060.3817/4817/29

Entries in the last two columns count comparisons whose error-ratio seed range lies entirely within ±10% of unity. Reduced denominators indicate cases where ESS matching was unavailable.

The table of reported ESS values has also been added to Appendix B.6, Table 2 in the revised manuscript. We thank the Reviewer for bringing this to our attention.

2. Deterministic interpolants in Appendix A.5 (W2)

We agree that the positive-noise limit and training directly with zero noise are mathematically distinct. Corollary 4 identifies the canonical zero-bandwidth limit of the Gaussian-smoothed empirical minimiser, not the unique behaviour of a network trained directly on deterministic paths. The discussion of deterministic-interpolant methods in Appendix A.5 is not intended as a theorem asserting trained networks must reproduce the limiting field; we have made this distinction explicit in the revision.

Reviewer_FVtTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper discussed trajectory replay in the unrestricted empirical minimisation of Flow Matching. Based on the kernel regression representation, the Gaussian law for pairwise log-weight gaps and bounds relating concentration to memory-bank size are derived, and then the dimension, bandwidth, and covariance geometry are studied. The experiments on dynamical systems provide empirical support, along with a study of finite-capacity DNNs.

Critical strengths
  1. The paper clearly explains why increasing bandwidth, or changing covariance, can reduce replay without resolving deficiencies in the velocity targets.
  2. The analysis connects several modelling choices within the framework.
  3. The experiments are well-scoped and well-designed.
Critical weaknesses
  1. The technical novelty is not high, as several of the results follow somewhat directly, such as from Gaussian likelihood ratios, tail bounds, or the convex combination in the kernel regression.
  2. The empirical connection with the trained models is also limited, as most experiments evaluate the closed-form estimator. The neural network experiments are there, but are limited in providing broader conclusions about practical sequential flow-matching models.
  3. In Figure 4, can the authors report the ESS or maximum posterior weights for the matched pairs?
  4. Can the authors clarify the relationship between pointwise selection of a stored target and trajectory replay under numerical integration?
  5. Can the authors clarify if the finite-capacity study includes a training-budget or regularisation control to assess the robustness of the observed width dependence?
Questions for authors

See weaknesses above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer FVtT

We thank the Reviewer for the positive assessment of the paper. We particularly appreciate the concern highlighting the robustness of the observed width dependence, as this has allowed us to strengthen the paper. Changes to the revised paper are highlighted in blue text. We address the specific concerns below:

1. Technical Novelty (W1)

We agree with the reviewer that mathematical tools such as Gaussian likelihood ratios and convex combinations are not novel methods. Our contribution lies in making use of these methods to formalise the phenomenon of trajectory replay, combining finite-bank, geometry-dependent replay bounds with an analysis of sequential modelling choices and the introduction of target coverage as a separate variable for consideration.

2. Practical considerations (W2)

We agree with the Reviewer's critique that the empirical work is not expansive; the primary motivation of the paper was to provide an in-depth study of a phenomenon that occurs in sequential flow matching and to better inform practical choices via mathematically rigorous theoretical considerations. We do not claim that these experiments establish general improvements in trained multi-step forecasting. We have added two experiments in this direction, namely concentration along generated rollouts of the closed-form field and trained MLPs (Appendix B.8, Figure 9) and width, training-budget and weight decay ablations (Appendix B.7, Figure 8)

3. Reporting ESS in Fig. 4 (W3)

We have reported the ESS comparison for Fig. (4) in a table in Appendix B.6, Table 2 (same table as in the response to Reviewer shgH). The results show that average concentration is a good proxy on relatively isotropic systems, but insufficient on strongly anisotropic ones. ESS matching reduces some apparent error differences, although residual differences remain. Fig. 4 should therefore be interpreted as testing the explanatory power of the concentration parameter, not as controlling the entire posterior weight distribution. We have ensured the discussion reflects this.

4. Distinction between pointwise selection of stored target and trajectory replay under numerical integration (W4)

  • Trajectory replay in our analysis refers to the local nearest-component selection mechanism of the empirical velocity field at a query: when a single posterior responsibility dominates, the field is approximately the corresponding stored conditional velocity target.
  • Numerical integration need not reproduce a complete stored trajectory exactly. The dominant component may change as the state and flow time evolve, and integration may consequently combine portions of different stored conditional paths. We hope this is a sufficient clarification.

5. Training budget and regularisation controls (W5)

  • As detailed in Section 6.4 and Appendix B, the two MLP widths are trained for 50000 steps using the same architecture family, optimiser, learning-rate schedule and batch size, across three seeds. The protocol in the original paper does not use weight decay or early stopping.
  • Based on Reviewer feqK and FVtT's comments, we included an ablation study in Appendix B.7 Figure 8. The ablation establishes that the observed width dependence persists across the tested training budgets and regularisation settings.
  • The width dependence persists across the tested configurations: wider networks approach the empirical minimiser more closely, while the width sweep shows higher off-support error for wider models. This supports the robustness of the observed trend, without establishing that optimisation and capacity effects are fully separable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.