- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper builds on a pivotal work of Lim et al. (2026), who showed that the infinite-capacity minimizer of the sequential flow matching objective is a Nadaraya-Watson kernel regressor over stored transitions, which can collapse to nearest-neighbor "trajectory replay".
For Gaussian conditional paths, this paper studies two questions: (i) concentration (when the posterior weights become one-hot) and (ii) target coverage (whether convex combinations of the stored targets can represent the true velocity).
For concentration, they show that for a query drawn from the path of window , the log-weight gap to window is exactly with (Theorem 1), that replay onset requires of order (Proposition 1), and that since , replay occurs as at a fixed bandwidth (Theorem 2). Coupling and interpolant only enter through (Proposition 2), the score objective has the same weights (Proposition 3), and a bandwidth gives replay, non-degenerate and uniform regimes for , and (Theorem 3). An anisotropic covariance replaces the Euclidean distance with a Mahalanobis distance, and the equal-trace optimum replaces with an effective rank (Propositions 4 and 5). For coverage, when the covariance is constant in flow time, lies in the convex hull of the stored targets, which gives an error floor that reweighting cannot remove (Theorem 4).
Experiments on Lorenz-63/96 with the closed-form estimator show that ESS curves collapse onto , that the onset scales with , that the hull floor bounds the best achievable error, and that anisotropic and isotropic kernels mostly perform alike at matched . A small MLP experiment shows that narrower networks deviate more from and can have lower off-support error.
Critical strengths
-
The separation between concentration and coverage is an effective way to frame the problem. Theorem 4 and the remark that concentration is not itself a performance objective make a clearer point: methods that only reduce replay cannot fix a deficient target bank. This is a helpful perspective for the growing literature on sequential and multi-marginal flow matching.
-
The characterization of replay onset is well-constructed and precise, and the synthetic experiments results agree. The exact Gaussian law of the log-weight gap gives a single parameter that absorbs the dimension, window order and interpolant, and the experiments confirm the predicted collapse, the relative concentration, and the log scaling. The derivations are mathematically correct (e.g., Proposition 1(ii) follows a union bound with , which gives ).
Critical weaknesses
1. The novelty is limited.
The main concentration result is essentially a known collapse result, and the paper does not position against it.
(a) Compared with Lim et al. (2026): that paper already derives the closed-form kernel-weighted minimizer (Lemma 1 in this paper), notes that it is a mixture of per-transition velocities, decomposes the drift into a velocity mixture and a score term, and states that the weights converge to hard nearest-neighbor assignments as the bandwidth goes to zero. What is effectively new in this submission is the quantitative part.
(b) Compared with Biroli et al. (2024): their collapse analysis studies the same object, namely the posterior weights of a Gaussian mixture centered at stored points, evaluated at noisy versions of a stored point. This is the same query law as in Theorem 1 and its corresponding Proposition 1. Using a Random Energy Model argument, they obtain a sharp collapse time (Equation 10). In the notation of this paper, the collapse condition becomes , which tends to when . This is the same onset as Proposition 1 and the same scaling as Theorems 2 and 3, with a sharp constant. Prop. 1 only places the transition between (necessity) and (sufficiency), and the remark after Prop. 1 explicitly does not identify a critical constant, even though Fig. 2(c) plots a reference line. Biroli et al. is cited only in a list of memorization papers, and this connection is not discussed.
(c) Key results are closely related to the immediate studies (Proposition 3, Theorem 4, and Proposition 5). What remains "novel" is a reasonable yet incremental extension. For a theoretical paper, I consider this the most important weakness.
2. The analysis covers the regime that is least relevant for inference, and the practical implications are not tested.
(a) All concentration results assume that the query is generated from a stored component. In this regime replay is the expected outcome. During inference, ODE trajectories move away from the training law, and this off-support regime is where replay vs. generalization matters. The authors acknowledge this critical distinction as a future work.
(b) The coverage result requires . Many sequential FM use time-varying noise, so the floor does not apply to them. This includes the Gaussian bridge path of Lim et al. (2026), on which this paper builds upon.
(c) There is no forecasting experiment showing that the recommendations derived from the theory improve prediction, such as (in Theorm 3, ), effective-rank covariance shaping, or a larger target bank. All evidence comes from Lorenz-63/96 with closed-form estimators, plus one MLP experiment, in which, as the authors already acknowledge, capacity and optimization effects are not separated.
Weakness 1 determines my recommendation. Since the contribution is mainly theoretical, its novelty over Lim et al. (2026) and the diffusion collapse / memorization work has to be clearly established, and the current manuscript does not do this. Weakness 2 further limits the practical significance.
Questions for authors
-
What exactly is new relative to Lim et al. (2026)? In particular, rewriting the collapse time of Biroli et al. (2024) in your notation gives , i.e., the same onset and scaling as Prop. 1 and Thm. 2, with a sharp constant. Which of your concentration results are not implied by, or are strictly stronger than, that analysis? And why does Prop. 1 not recover the sharp constant?
-
Can any of the concentration results be extended, at least heuristically or empirically, to queries along generated ODE trajectories, e.g., by measuring ESS along actual sampling/rollout paths instead of training-law queries?
-
Do the regimes predicted by Thm. 3 (specially this: ) or the shaping lead to better multi-step forecasts (rollout error, CRPS) for trained sequential FM models on Lorenz-96 or a real dataset?
-
For commonly used time-varying paths (, e.g., Brownian bridge noise), is no longer restricted to the convex hull. Can you characterize the reachable set in this case, or show empirically how much the floor is relaxed?
-
In Fig. 5, can you separate capacity from optimization, e.g., by tracking during training for both widths, or by adding wider networks and longer training? (This is just out of curiosity, and for the extension-ability.)
Suggestions: the main text is too dense and several figure labels are hard to read. Moving some propositions (e.g., Props. 2-3) to the appendix and adding a worked example relating to bandwidths used in practice would help.
References (note)
Lim, S. H., Lin, S., Mahoney, M. W., & Erichson, N. B. (2026). Is Flow Matching Just Trajectory Replay for Sequential Data?. arXiv preprint arXiv:2602.08318.
Biroli, G., Bonnaire, T., de Bortoli, V. et al. Dynamical regimes of diffusion models. Nat Commun 15, 9957 (2024). https://doi.org/10.1038/s41467-024-54281-3
4 comments
4. Coverage under time-varying covariance (W2 and Q4)
The fixed convex-hull floor in Thm. 4 requires constant covariance wrt flow time. However, as the Reviewer correctly pointed out, noise schedules can vary with time. For a shared time-varying isotropic path, writing we have Consequently, Lemma 1 gives Thus a translated, path-dependent convex-hull constraint remains. Unlike Theorem 4, the reachable set now depends on the noise schedule. We note that is unchanged when the whole schedule is rescaled i.e. , so changing the bandwidth alone (the parameter controlling concentration) does not move the floor. This has been added as Corollary 1 to the main text.
5. Capacity versus optimization (Q5)
- We conducted additional ablations across widths 64-1024, training budgets of 12.5k-200k steps and weight-decay strengths. The results are in Appendix B.7, Figure 8. Wider networks consistently approach the empirical minimiser more closely, and the observed width dependence persists across the tested budgets and regularisation settings. Wider networks also exhibit higher off-support error in the width sweep. These experiments establish robustness of the observed width dependence, though they do not prove a converged capacity-only effect. We thank the Reviewer for this comment as it has allowed us to strengthen the paper.
- A natural theoretical formulation to separate width and training time would combine a mean-field limit with Wasserstein gradient-flow training dynamics, examining whether the width and training-time limits commute. This remains a separate theoretical question that we will focus on for future work.
6. Presentation
We appreciate the comment on density and figure readability. We have attempted to make the figures more readable. We will ensure that the paper is made more readable in further revisions. We have also noted the comment on including a table for kappa for commonly used bandwidth regimes, it is now in Appendix A.2, Table 1
References:
- Biroli, G., Bonnaire, T., de Bortoli, V., & Mézard, M. (2024). Dynamical regimes of diffusion models. Nature Communications, 15, 9957. https://doi.org/10.1038/s41467-024-54281-3
- Lim, S. H., Lin, S., Mahoney, M. W., & Erichson, N. B. (2026). Is flow matching just trajectory replay for sequential data? arXiv:2602.08318. https://arxiv.org/abs/2602.08318
Thank you authors for very detailed (and timely!) response. I especially appreciate clarification of the novelty that I pointed out earlier, and after thorough reading, I believe the authors have rather clearly established a solid ground beyond the existing stream of literature.
With that said, I also appreciate the accommodation of my earlier readibility concerns.
I have therefore adjusted my assessment.
We thank the Reviewer for engaging with our response and updating their assessment.
Response to Reviewer feqK
We thank the Reviewer for the detailed assessment and for recognising the distinction between concentration and target coverage. We particularly appreciate the feedback on off-support queries and the effects of time-varying covariance on coverage, both of these have strengthened our paper. We clarify some of the comments regarding novelty, scope, and interpretation of our results below; all changes to the paper are in blue text in the revised pdf.
1. Novelty relative to Lim et al. (2026) and Biroli et al. (2024) (W1 and Q1)
Lim et al. (2026) establish the kernel-regression representation of the minimiser and its nearest-neighbour limit as bandwidth vanishes. We build on this work by quantifying the dimension and memory bank dependence for the replay onset. We further characterise this replay dependence on window geometry and covariance specifically, finally showing that replay is not a sufficient factor for performance, introducing target coverage as a separate variable for consideration.
We agree that Biroli et al. (2024) establish a sharper threshold for Gaussian-mixture concentration. Fig 2(c) in our paper demonstrates logarithmic scaling, while the onset constants vary with bank geometry and the ESS criterion, and do not strictly follow the constant in Biroli et al. (2024). Our result is thus a broader but less sharp characterisation that suits the use case of sequential data. We have amended Prop. 1 from our work to position it as giving finite memory bank, geometry-dependent concentration bounds conditional on arbitrary stored path centres, allowing correlated and overlapping sequential windows without assuming independent Gaussian centres. We have also made this positioning more explicit in the related works.
Beyond this mechanism, Propositions 2-5 characterise the effects of multi-marginal interpolation, score regression and covariance geometry. Theorem 4 introduces the separate target-coverage constraint, demonstrating why preventing replay need not improve prediction. These constitute the main extensions beyond the above concentration analysis. We thank the reviewer for bringing this to our attention.
2. Off-support queries and inference (W2 and Q2)
The Gaussian law in Theorem 1 assumes a query generated from a stored component, but the posterior weights in Equation (1) remain well-defined at any query, including during inference.
To empirically check the effects on replay, we implemented the Reviewer's suggestion to evaluate ESS along generated trajectories of the closed-form field and both trained MLPs on Lorenz-96, from training-law, held-out and off-support initial states. The results are in Appendix B.8, Figure 9 in the revised paper. Training-law starts exhibit near-complete concentration, whereas held-out and off-support starts initially have higher ESS. During autoregressive rollouts, trajectories move progressively farther from the stored bank; network trajectories remain closer than those of the empirical minimiser and exhibit more concentrated weights. These observations support the distinction between weight concentration and accurate prediction off-support. They do not establish a universal off-support onset law, which would require characterising the generated query distribution.
For a theoretical starting point, we can assume that for a query (x), which follows a distribution qt, its expected log-weight gap (modified from Thm. 1) is the difference between the KL divergences from qt to the two conditional components considered: Eq[Gij]=KL(qt∣∣pj)−KL(qt∣∣pi), provided both divergences are finite. This provides a nice starting point for analysing concentration under generated query distributions, however, it does not itself establish replay along an ODE trajectory, since the distribution need not coincide with any training component. We leave the complete analysis of this to future work.
3. Multi-step forecasting performance (W2 and Q3)
We do not claim that Theorem 3's bandwidth regimes or Proposition 5's covariance necessarily improve multi-step forecasting. These results characterise and minimise concentration, respectively, not forecasting error. Indeed, Sections 6.2–6.3 show that indiscriminate averaging and poor target coverage can persist even after replay is mitigated. The theoretical results identify mechanisms for controlling replay, not universally optimal forecasting configurations. We will ensure this distinction is made more explicit in the revision.