acceptodds
Under review as a conference paper at ICLR 2027

Walking the Score Manifold: Continuous-time Generative Dynamics on Learned Data Manifolds

Abstract

Generative modeling of time-dependent data is typically formulated on a discrete temporal grid, restricting supervision to the observed timestamps in the training data. We instead frame generation as continuous-time evolution on a learned data manifold. To this end, we leverage pretrained score-based models as geometric priors and learn a vector field that evolves data along score-induced interpolation paths. Because these dynamics follow manifold-respecting transitions, they support generation at arbitrary timestamps and temporal super-resolution beyond the discretization of the training data. Moreover, this geometric formulation allows us to train the vector field simulation-free through a regression objective. To improve long-horizon rollout robustness, we introduce an objective that promotes path-relative transverse exponential stability. While motivated by stability theory, it admits a practical interpretation as denoising score matching transverse to the interpolation path. Further, we extend the framework to a probabilistic setting that models a distribution over plausible future trajectories. We demonstrate the method on natural video and scientific dynamical data, including temporal super-resolution, PDE-based spatiotemporal fields, and molecular dynamics. Our results show that score-based priors provide a strong foundation for learning stochastic continuous-time generative dynamics.

Primary area:
generative models
Keywords:
Generative modeling, Continuous-time dynamics, Continuous-time generative models, Score-based models, Riemannian geometry, Temporal super-resolution

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_roauOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the challenge of continuous-time generative modeling for time-dependent data (e.g., video, physical simulations, molecular dynamics) when supervision is available only at sparse, discrete timestamps. The authors propose to structure continuous-time trajectories by constraining evolution to a learned data manifold induced by a pre-trained score-based model.

Critical strengths

The paper is well-structured and clearly written. The motivation for moving beyond Euclidean linear interpolation in continuous-time models is established early on. The introduction of path-relative transverse exponential stability (Section 3.2) directly addresses a fundamental vulnerability of simulation-free Neural ODEs (error accumulation).

Critical weaknesses

The framework relies on multiple pre-trained/frozen modules: a pre-trained score model, an amortized geodesic interpolator, a latent encoder/posterior, a history-conditioned prior, and the velocity network. This multi-stage setup adds significant training pipeline complexity compared to standard end-to-end continuous-time models.

Questions for authors

What is the computational overhead at inference time (sampling latency / NFE) compared to standard Flow Matching or unconstrained Neural ODE baselines

1 comment

Dear reviewer, we thank you for your feedback and for the positive overall assessment of our paper, especially regarding its motivation and presentation.

We have addressed your concerns through a new pipeline overview and a clarification of the computational requirements at inference. Details are provided below and incorporated into a revised version of the paper, with changes marked in blue. We also made several further additions, summarized below.

We believe these additions address your concerns and strengthen our evidence. We hope they will reinforce your positive assessment and support a higher rating. If any relevant concerns remain, we would greatly appreciate the opportunity to address them.

Weakness: Complex Pipeline

Thank you for pointing this out. We have added Figure 5 (Appendix D.1), which provides an overview of the complete pipeline and distinguishes the components required during training and inference. We hope that this improves accessibility of the paper.

Regarding the complexity itself, our framework builds on a pretrained score-based prior. We expect such models to become increasingly available as general-purpose priors, potentially removing the need to train this component from scratch. Furthermore, the interpolator amortizes the cost of finding geodesic-like paths under this prior and could potentially be replaced by more efficient geometric constructions.

Beyond these components, the architecture follows standard CVAE machinery, consisting of a latent prior, posterior, and velocity network. Moreover, the score model and interpolator are frozen during the final training stage. Thus, the additional complexity primarily concerns the construction of geometric supervision rather than the generative model itself.

Question: Computational Overhead

The score model, interpolator, and posterior are not required at inference time. Generation only involves sampling a latent from the history-conditioned prior and evolving the learned vector field (Figure 5, Appendix D.1).

Consequently, inference has essentially the same computational structure as standard Neural ODE or Flow Matching sampling. For h=0h=0, we solve an ODE using only the velocity network, while for h>0h>0, each finite-step update requires just one velocity-network evaluation, allowing direct rollouts without an ODE solver.

Thus, our geometric machinery introduces no additional per-step inference cost. As with standard Neural ODEs, the actual NFE depends on the solver, its tolerances, and the learned field. Figures 9–10 (Appendix E.2) report rollout quality against step size and NFE across direct, fixed-step, and adaptive integration, showing that our robustness improvements persist across these settings.

Further Additions

Beyond your specific concerns, we made several additions that further strengthen the geometric and stability contributions you highlighted.

First, Corollary 3.1 and the accompanying remark (Section 3.2, Appendix C.4) connect our stability theory more directly to the MSE-based training objective, without requiring a uniform approximation-error bound. Furthermore, we added video-level metrics (tOF, Ewarp, FVD), showing improved temporal consistency over Vid-ODE, including on KTH interpolation (Tables 5–6).

We also introduced two stochastic generative baselines, autoregressive Flow Matching and a latent Neural ODE. Our model achieves competitive long-horizon distributional performance while supporting continuous-time generation, and remains stable through four times its training horizon, whereas the latent ODE diverges (Section 4.4, Table 4).

Finally, a new experiment confirms finite-step self-consistency, with composed and direct steps differing by only 1.5–5% of the reference displacement for h≤0.5h\leq 0.5 (Appendix E.1, Table 16).

Reviewer_9yX2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a continuous-time generative dynamics framework that leverages pretrained score-based models as geometric priors. Rather than relying on conventional linear interpolation between observed states, the method constructs score-induced interpolation paths and learns a vector field through simulation-free velocity matching. To improve rollout robustness, the authors introduce a path-relative transverse stability objective that encourages perturbed trajectories to return toward their reference paths. A latent-variable formulation is further incorporated to model stochastic future trajectories.

The framework is evaluated on synthetic dynamical systems, PDE-governed spatiotemporal data, molecular dynamics, and natural video generation. The results demonstrate improved geometric plausibility and rollout stability in several settings, although the advantages in prediction accuracy and video generation are not consistent across all tasks.

Critical strengths
  1. Well-motivated integration of score-based geometry and continuous-time generative dynamics.

The paper presents an interesting framework that uses pretrained score models to construct manifold-aware interpolation paths for continuous-time generation. This formulation provides a principled alternative to conventional Euclidean interpolation and enables simulation-free training through local velocity matching. The connection between learned geometric priors and continuous-time dynamics is conceptually appealing and potentially applicable to different dynamical systems.

  1. Principled treatment of trajectory stability with supporting theoretical and empirical analyses.

The proposed path-relative transverse stability objective is a valuable contribution. The authors derive theoretical properties for an ideal vector field and establish bounds under approximation-error assumptions. The controlled experiments, particularly the two-dimensional example and Navier–Stokes stability analysis, provide useful evidence that the proposed transverse correction improves trajectory adherence and reduces accumulated rollout errors.

Critical weaknesses
  1. The connection between theoretical stability guarantees and the practical learned dynamics is insufficiently established.

The theoretical analysis provides exponential transverse stability guarantees for an ideal path-relative vector field. However, the corresponding guarantee for the learned vector field requires a uniform bound on the transverse approximation error.

The actual training procedure minimizes a sampled regression objective, which does not directly ensure that this uniform approximation-error condition holds throughout the relevant neighborhood of the reference trajectory. Consequently, the theoretical results do not automatically establish comparable stability properties for the learned model.

Although the two-dimensional and Navier–Stokes experiments provide encouraging evidence, these evaluations are relatively limited and do not fully establish whether the predicted stability behavior generalizes to more complex, high-dimensional dynamical systems.

The paper would benefit from a clearer distinction between guarantees for the ideal vector field and empirical properties of the learned dynamics, together with more systematic validation across perturbation magnitudes, trajectories, and model initializations.

  1. The experimental comparisons are insufficient to establish consistent advantages over existing continuous-time and stochastic generative models.

While the experiments cover several application domains, the comparative results are mixed, and some important baselines are missing.

First, on the KTH Actions benchmark, the proposed method improves over Vid-ODE in extrapolation but underperforms it on all three reported interpolation metrics (SSIM, PSNR, and LPIPS). This raises questions about the consistency of the claimed benefits of score-induced interpolation, particularly because generating plausible intermediate states is a central motivation of the paper.

Second, the Gray–Scott experiments reveal a trade-off between geometric plausibility and trajectory accuracy. In Table 3, Neural ODE achieves a lower relative L2 error (0.060 versus 0.115), whereas the proposed method achieves a better spectral plausibility score (0.335 versus 0.482). These results suggest that improved manifold adherence does not necessarily translate into more accurate recovery of the true intermediate dynamics. This distinction deserves greater emphasis.

Third, the stochastic long-horizon evaluation primarily compares the proposed probabilistic model against a deterministic Fourier Neural Operator (FNO). Since the deterministic baseline cannot represent multiple plausible futures, the reported distributional advantage partly reflects differences in modeling capacity rather than necessarily demonstrating the superiority of score-guided dynamics.

More competitive comparisons against stochastic continuous-time models, such as latent ODE-based or other probabilistic dynamics approaches, would strengthen the experimental conclusions. Similarly, the natural video evaluation relies primarily on Vid-ODE, leaving the performance relative to more recent generative approaches insufficiently explored.

Overall, the experiments demonstrate the feasibility of the proposed framework, but they do not yet provide sufficiently consistent evidence of its advantages in prediction accuracy, stochastic generation, and general-purpose continuous-time modeling.

Questions for authors
  1. Theoretical guarantees and learned dynamics

Proposition 3 assumes a uniform transverse approximation-error bound for the learned vector field. Could the authors clarify how this assumption relates to the actual regression-based training objective? Is there empirical evidence that the required condition approximately holds beyond the two-dimensional example?

  1. Video interpolation performance

In Table 5, the proposed method underperforms Vid-ODE on all three KTH interpolation metrics, despite the central motivation of improving intermediate-time generation through score-induced geometry. Could the authors explain this discrepancy and clarify under which conditions the geometric prior is expected to improve interpolation quality?

  1. Geometric plausibility versus trajectory accuracy

In the Gray–Scott experiments, Neural ODE achieves substantially lower relative L2 error, while the proposed method achieves better spectral plausibility. How should this trade-off be interpreted? In which applications would the authors prioritize score-induced geometric plausibility over direct trajectory accuracy?

  1. Stochastic generative baselines

The distributional rollout experiments primarily compare the proposed stochastic model against deterministic FNO. Could the authors explain the choice of this baseline? Are comparisons available against stochastic dynamical models capable of generating multiple plausible future trajectories under equivalent conditioning information?


If the author answers the above question, I will consider increasing my rating.

3 comments

Dear reviewer, we thank you for your careful feedback and for recognizing the motivation of our approach and the value of our stability analysis.

To summarize, we addressed your concerns through:

  • New theory connecting rollout stability to our training loss.
  • Additional video metrics.
  • New stochastic baselines: autoregressive Flow Matching and latent Neural ODE.
  • Clarification of geometric plausibility versus trajectory accuracy.

Details are provided below and incorporated into a revised version of the paper, with changes marked in blue. Further additions include expanded stability experiments, a full pipeline diagram, and minor clarifications to improve readability.

We believe the additional results and clarifications directly address the main concerns underlying your initial assessment. We hope this will allow you to reconsider your rating accordingly. If any relevant concerns remain, we would greatly appreciate the opportunity to address them.

Q1 and W1: Connecting stability with the loss

Proposition 3 assumes a uniform transverse-error bound, as in standard contraction theory, without linking it to training. Our new Corollary 3.1 goes further, bounding displacement through exponentially discounted errors along the rollout instead.

The accompanying remark connects this to MSE training. If expected squared field error along rollouts is bounded by LL, then E∥e(Yt)∥≤e−λ(t−t0)E∥e(Yt0)∥+L/λ.\mathbb E\|e(Y_t)\|\le e^{-\lambda(t-t_0)}\mathbb E\|e(Y_{t_0})\|+\sqrt L/\lambda. Our loss targets this error on perturbed training states, linking regression accuracy to expected rollout stability. Existing experiments support the intended contraction. All tested Navier–Stokes perturbations contract (rate −0.90 vs. −0.13, Table 1), and rollouts improve across solvers (Appendix E.2).

Q2: Interpolation performance

Vid-ODE leads on KTH frame-level interpolation metrics, despite visible artifacts such as duplicated limbs (Appendix E.4, Figure 13). We therefore added tOF, Ewarp, and FVD (Tables 5–6), which provide a more complete evaluation of temporal consistency and video quality. Even in interpolation, our method outperforms Vid-ODE on tOF and Ewarp, although Vid-ODE retains better FVD. This shows that frame-level reconstruction metrics do not fully capture the differences in temporal quality.

Extrapolation is particularly relevant to our objective, as it tests whether the learned dynamics can generate plausible continuous-time evolution from past observations, without a future endpoint to guide the trajectory. Here, our method outperforms Vid-ODE on all three video-level metrics, further supporting its ability to maintain coherent evolution over time.

Q3: Plausibility vs. Accuracy

Although single-trajectory fitting isolates intermediate behavior, we aim to generate plausible continuous-time trajectories, not just match observed states. With sparse observations, matching endpoints does not necessarily produce plausible intermediates, which our score-based prior supervises.

In molecular dynamics (Figure 3), Neural ODE fails to recover the intermediate trajectory and severely distorts molecular geometries. On Gray–Scott (Table 3), it follows the reference more closely, perhaps due to its inductive bias. Yet in both cases, trajectory fitting misses important characteristics of the data, reflected in unrealistic bond lengths in molecular dynamics and higher spectral error on Gray–Scott (0.482 vs. 0.335). Such characteristics matter for plausible generation, beyond matching observed states. The latent ODE further shows that good reconstruction need not translate to generative performance (Q4).

Q4 and W2: Experimental Scope

Choice of FNO. Deterministic FNO is a sanity check that our model adds variability beyond different initial states (Table 4).

Stochastic baselines. We added autoregressive FM (Kohl et al., 2026) for discrete-time and stochastic latent ODE (Chen et al., 2018 and Rubanova et al., 2019) for continuous-time (Table 4, Appendix D.6).

Gray–Scott generative rollouts (lower is better)

ModelTimeSW T=64T=64SW T=256T=256FVD T=256T=256
AR FMDiscrete0.690.790.61
Latent ODEContinuous10.72DivergedDiverged
OursContinuous0.980.750.74

At T=256T=256, four times the training horizon, ours is competitive with AR FM while generating in continuous-time. Latent ODE instead reconstructs well (L2 0.023) but performs poorly at generation (0.580, Table 12) and diverges. Together, these results demonstrate competitive distributional quality with stable continuous-time generation.

Scaling. Our contribution is the geometric framework and stability mechanism, not state-of-the-art video quality. Scaling needs larger datasets, models, and compute and might require further engineering, as emphasized in the revised Limitations (Section 6).

Thank you to the author for the detailed reply. The author answered my question, and I will increase my rating.

Thank you for positively assessing our rebuttal and reflecting that in your score. Are there any remaining concerns prohibiting a higher rating, which we could address?

Reviewer_x7F6OpenReview ↗
Rating
4
Confidence
2
Informativeness
3(+1)
Presentation
3(+1)
Soundness
3
Read review
Paper summary

The authors propose an interesting methodology on how to learn continuous-time dynamics models. First they train a diffusion-like model to learn the overall data manifold. After, then use this to learn a probability path (starting from a linear interpolation) between arbitrary examples, by penalizing generations that diverge from the manifold at random timepoints. This creates an "augmented" dataset of marginal samples from an (arbitrarily) fine grid of points. They then learn a dynamics model with an arbitrary step to walk on the data manifold.

Critical strengths
  • This is a very creative paper, I was not familiar with this line of work and enjoyed reading about it.
  • The method is grounded and makes good use of well established modelling techniques to learn manifolds and dynamics.
  • The pipeline is simple (although a bit long).
Critical weaknesses
  • Many training phases, a bit hard to follow initially.
  • Paper talks about uncovering the true underlying dynamics, but it seems the experiments don't provide enough evidence for this? Can the authors explicitly comment on what they think the evidence is?
  • LPIPS just measures if each image looks perceptually pleasing to human observers, but are there metrics which measure video consistency / temporal alignment between frames (given a reference)? Video LPIPS? It seems all metrics are frame-level metrics, not video metrics.
  • How consistent is the temporal dynamics model with itself, relative to composing steps? For example, if I do one step advancing τ\tau, or 2 steps advancing τ/2\tau/2 each, would it produce similar results?
  • Experiments are toy-ish settings, but I think this is ok for this type of paper and the pipeline is probably expensive to run on academic-level GPU budgets.
Questions for authors

See weaknesses section, please.

3 comments

Dear reviewer, we thank you for your feedback and for the positive overall assessment of our paper.

To summarize, we addressed your concerns through:

  • A new overview figure.
  • Clarification of intermediate dynamics recovery.
  • Additional video-level metrics.
  • An experiment evaluating the consistency of composed prediction steps.

Details are provided below and incorporated into a revised version of the paper, with changes marked in blue. Further additions include a stronger theoretical connection between regression error and rollout stability, new stochastic generative baselines, additional rollout stability evaluations, and minor clarifications to improve readability.

We believe these additions address your concerns and further strengthen the paper. We appreciate your strong support for our work and hope that the revisions reinforce your positive assessment. If any questions remain, we would be happy to address them.

W1: Training pipeline

Thank you for pointing this out. We added Figure 5 (Appendix D.1) to provide an overview of the full training and inference pipeline. Furthermore, Corollary 3.1 and its accompanying remark (Section 3.2, Appendix C.4) connect stability theory more explicitly to our training objective. Together, we hope that these additions make the method easier to follow.

W2: True Dynamics

We cannot guarantee recovery of the true intermediate dynamics from sparse observations alone, as already acknowledged in the Limitations (Section 6). Rather, our argument is that, when the dynamics are unknown, score-based geometry provides a prior for plausible intermediate evolution instead of leaving trajectories unconstrained. This is particularly clear in molecular dynamics (Figure 3), where linear interpolation and Neural ODE produce severely distorted intermediate configurations, while our method better preserves molecular geometry.

Importantly, additional dynamics knowledge can be incorporated when available. Adding a PDE residual to interpolator training (Appendix D.9, Table 19) reduces intermediate-state error by another 8.1× compared with score geometry alone. Thus, the prior supports plausible (frame-wise) evolution without known dynamics, while physical knowledge can further improve recovery.

W3: Video Metrics

Thank you for this suggestion. We have added tOF, Ewarp, and FVD (Appendix D.11) to evaluate temporal consistency.

KTH Actions — additional video-level metrics (Tables 5–6; lower is better)

SettingMethodtOF ↓Ewarp ↓FVD ↓
InterpolationVid-ODE1.1220.3847.8
Ours (linear)1.1000.36014.8
Ours1.0730.3389.1
ExtrapolationVid-ODE1.2270.283119.6
Ours (linear)1.1760.147117.9
Ours1.1530.16493.3

Crucially, these metrics change the picture even for KTH interpolation. Here, Vid-ODE leads on all frame-level metrics, despite visible artifacts such as duplicated limbs (Appendix E.4, Figure 13). Our method instead achieves better tOF and Ewarp, revealing temporal differences not captured by those frame-level scores, although Vid-ODE retains better FVD. In extrapolation, our score-based variant improves all three metrics over Vid-ODE. On Gray–Scott, score-induced paths also reduce tOF by 2.4× relative to linear interpolation (Table 2). Thus, the expanded evaluation provides further evidence for temporal coherence that frame-level reconstruction scores alone missed.

W4: Self-consistency

We tested two steps of size hh against one of size 2h2h in our 2D example (Section 4.1, Table 16 in Appendix E.1). For h≤0.5h\leq 0.5, composition errors are 1.5–5% of the displacement along the reference path. More importantly, in full-loop rollouts using step sizes h=0.05h=0.05–11, the trajectories return close to their intended endpoints, with final errors only 2–7% of the root-mean-square displacement from the start. Thus, finite-step updates are reasonably self-consistent even without explicit enforcement. A semigroup-consistency regularizer, as in shortcut models, could be a natural extension to enforce self-consistency explicitly.

W5: Scope

We agree that our main contribution is methodological: score-guided continuous-time generative dynamics, together with a stability mechanism for robustness. We see no fundamental methodological obstacle to scaling this approach, e.g. towards state-of-the-art video generation, but it requires larger data, models, and compute, and might involve further engineering. We are happy to acknowledge this more explicitly and have emphasized it in the Limitations (Section 6).

Thank you for addressing my questions. I've raised the informativeness and presentation to 3.

Thank you

Thank you for your constructive feedback and thoughtful engagement with our work. Your comments have helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.