acceptodds
Under review as a conference paper at ICLR 2027

Variational Streaming Flow: Probabilistic Forecasting in Physical Time

Abstract

Probabilistic forecasting is important for predicting complex dynamical systems because intrinsic randomness and incomplete observations can cause the same observed state to evolve into multiple plausible futures. While flow matching is a flexible approach for probabilistic forecasting, it is computationally expensive. Streaming flow (SF) reformulates this approach to model temporal evolution efficiently by learning a continuous velocity field directly in physical time. However, SF learns a deterministic velocity field. Thus, it provides only a single future trajectory for a given fixed initial state and observation history. To overcome this limitation, we introduce Variational Streaming Flow (VSF). Our approach learns a latent distribution that is conditioned on the dynamics of interest. In turn, this enables probabilistic forecasting. Importantly, we retain the computational efficiency of SF by generating in physical time. Across deterministic and stochastic dynamical systems, VSF demonstrates superior predictive accuracy and distributional fidelity. We demonstrate the advantage for both long-horizon rollouts exceeding 1,000 steps, and settings with bifurcating dynamics. Moreover, VSF can be integrated into existing Joint-Embedding Predictive Architecture (JEPA)-based world models as a plug-and-play predictor to improve temporal dynamics and goal-directed success rate in navigation, motion planning, and manipulation.

Primary area:
learning on time series and dynamical systems
Keywords:
Probabilistic forecasting, Generative models, Dynamical Systems, World Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KgSzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds on Streaming Flow (SF; Jiang et al., 2025), which forecasts a dynamical system by integrating a learned velocity field directly in physical forecast time ss. Because the SF velocity field is deterministic, it gives one future for a fixed history. The proposed Variational Streaming Flow (VSF) adds a latent variable, sampled once per forecast and kept fixed through the rollout. Training follows from a VAE recipe, with the reconstruction term a generalized energy score over MM latent samples. VSF is evaluated on

  • three deterministic chaotic systems (double pendulum, Rössler, Lorenz);
  • two "stochastic" Rössler/Lorenz systems with a hidden, per-trajectory parameter;
  • as a predictor in the LeWM JEPA world model with a CEM planner on four tasks (PushT, Cube, TwoRoom, Reacher). and reports the best results in all main tables.
Critical strengths
  • Very clear writing and presentation, easy to follow.
  • The idea is simple and easy to plug into existing SF models. A single latent per rollout adds sample diversity at little extra cost.
  • Long-horizon evaluation shows advantages over rollouts of 1,000 and 2,500 steps, with per-timestep error curves (Fig. 13–14).
Critical weaknesses

W1. Limited novelty relative to SF. The method adds two standard components to SF.

  • Latent with encoder and KL. An encoder-inferred latent with a KL term toward N(0,I)\mathcal N(0, I) is the VAE recipe already used in variational flow matching [1].
  • Energy-score reconstruction. Energy-score reconstruction over multiple samples is commonly used to train generative forecasters.
  • One latent per sequence. Holding one latent fixed over a sequence is also not new, e.g.:
    • Latent ODE [2] and Neural ODE Processes [3] keep a per-trajectory latent in a physical-time ODE.
  • The SF paper itself contains a latent "stochastic variant" (Jiang et al., 2025, App. B). The present paper only refers to it indirectly ("SFP + auxiliary latent", Table 4) and does not compare against it.

[1] Guo, Pengsheng, and Alexander G. Schwing. "Variational rectified flow matching." arXiv preprint arXiv:2502.09616 (2025).

[2] Rubanova, Yulia, Ricky TQ Chen, and David K. Duvenaud. "Latent ordinary differential equations for irregularly-sampled time series." Advances in neural information processing systems 32 (2019).

[3] Norcliffe, Alexander, et al. "Neural ode processes." arXiv preprint arXiv:2103.12413 (2021).

W2. Missing stochastic and latent-variable baselines. The only flow-based baselines are SF and an in-house autoregressive rectified flow. The most direct missing baseline is the latent "stochastic variant" in the SF paper itself (Jiang et al., 2025, App. B). It is the existing way of making SF stochastic, so it is the natural comparison for VSF in both the forecasting and world-model experiments. The paper also does not compare against other ways of making flow-based forecasters stochastic:

  • Forecasting: [1,2] and published flow-matching forecasters cited in App. A
  • World model: the only baselines are LeWM's pretrained AR predictor and SF. There is no stochastic predictor, e.g.:
    • the SFP latent variant used as the predictor;
    • a flow-matching predictor in LeWM's latent space (as in FlowWM, Porcher et al., 2026); Therefore It is unclear whether the gains come from VSF's specific variational latent or from any source of stochasticity.

W3. Benchmarks lack real randomness or partial observability. This is the setting where a probabilistic predictor should matter, but the benchmarks largely avoid it.

  • Forecasting systems.
    • All forecasting systems are low-dimensional (d=3d = 3 to 88), noise-free ODEs (App. H).
    • The "stochastic" systems are deterministic given a hidden parameter, with no process noise, even though StocBench (Pfister et al., 2026) is cited.
    • The Lorenz parameter range ρ∈[23.74,25.74]\rho \in [23.74, 25.74] sits at the point where Lorenz's nonzero fixed points lose stability (ρ≈24.74\rho \approx 24.74). So the different futures are mainly "settles to a fixed point" vs. "stays chaotic". This choice is not discussed.
  • Near-duplicate initial conditions.
    • Rössler starts from only 8 centers, −1,13\\{-1, 1\\}^3, and Lorenz from a single center (1,1,1)(1, 1, 1), both with perturbation std 0.01 (App. H.1).
    • The data are split by trajectory, so test trajectories start very close to training ones. The test does not check generalization to new initial conditions.
  • World-model tasks are deterministic. PushT, Cube, TwoRoom and Reacher give the same next state for the same action, so there is no future uncertainty for VSF to capture. No distributional metric is reported for these tasks.
  • The bifurcation motivation is never tested directly. No experiment checks whether VSF assigns the correct probability to each future branch.

W4. World-model evaluation is not fully specified or controlled.

  • Fig. 2 says VSF "samples multiple latents" per action sequence, while Alg. 4 draws one z\boldsymbol z. I did not find how many latents are used per CEM candidate or how their costs are combined. If several are used, the planning compute is not matched to the AR predictor, which contradicts the "fixed planning budget" claim.
Questions for authors
  1. Novelty and positioning.
    • How does VSF differ from the latent "stochastic variant" in the SF paper (Jiang et al., 2025, App. B)?
    • How does it differ from Latent ODE / Neural ODE Processes, which also keep a per-trajectory latent in a physical-time ODE?
  2. Additional stochastic baselines.
    • Could you compare against the latent "stochastic variant" of SF (Jiang et al., 2025, App. B) in both the forecasting experiments (Tables 1–2) and as a world-model predictor (Table 3)? This is the existing way of making SF stochastic, so it seems the most direct baseline.
    • Could you also compare against Variational Rectified FM?
    • For the world model, could you add at least one other stochastic predictor, e.g.:
      • a flow-matching predictor in LeWM's latent space;
      • an RSSM-style predictor;
      • an ensemble of AR predictors?
  3. Benchmarks with real randomness or partial observability. Could you evaluate on settings where a probabilistic predictor is needed? Concretely:
  • Process noise: SDE systems such as those in StocBench (cited), or stochastic Lorenz-63/Lorenz-96 with additive noise.
  • Known branch probabilities: a bistable system such as a double-well Langevin process, to check whether the predicted fraction of samples in each branch matches the truth.
  • Partial observability:
    • observing only the xx-coordinate of Lorenz;
    • observing only positions (not velocities) of the double pendulum;
    • adding observation noise.
  • New initial conditions: test initial conditions drawn away from the training centers.
  • stochastic video benchmarks such as the Bouncing Shapes task used by FlowWM.
  1. VSF performs much better than all baselines, to what extent does the way of the latent is being incorporated matters, e.g. how it is being conditioned in the architecture? Is the posterior meaningful here and gives multimodal distribution under bifurcation? Why does VSF help so much on the deterministic double pendulum, where there is a single future?
10 comments
AuthorsOpenReview ↗

Response to Reviewer KgSz (Part 1)

We thank the reviewer for the constructive feedback.

Novelty and positioning relative to prior latent methods (W1, Q1)

We agree that variational latent variables, energy-score training, and fixed trajectory-level latents have been studied before. Our contribution is to combine these ideas with physical-time flow forecasting, where the learned velocity directly advances the predicted state. We clarify the differences from the most closely related methods below.

(1) Latent Streaming Flow. The stochastic variant of Streaming Flow Policy (Jiang et al., 2025) introduces a latent variable that evolves together with the physical state. Given a training trajectory ξ(t)\xi(t), its constructed paths are

xtξ=ξ(t)+e−κt(x0−ξ(0))+σrtz0,x_t^\xi=\xi(t)+e^{-\kappa t}(x_0-\xi(0))+\sigma_r t z_0, ztξ=[1−(1−σ1)t]z0+tξ(t),z_t^\xi=[1-(1-\sigma_1)t]z_0+t\xi(t),

where z0∼N(0,I)z_0\sim\mathcal N(0,I) is sampled independently of the initial physical state.

The two methods differ in their rollout dynamics:

\dot x_t=v_\theta^x(x_t,z_t,t\mid h), \qquad \dot z_t=v_\theta^z(x_t,z_t,t\mid h)$$ $$\text{VSF:}\qquad \dot x_t=v_\theta(x_t,t,h,z_0), \qquad \dot z_t=0.$$ Latent SF jointly evolves $x_t$ and $z_t$. Since both velocity components depend on the evolving state, errors in $x_t$ may also affect subsequent latent updates. VSF instead samples $z_0$ once and keeps it fixed throughout the rollout. Errors in the predicted physical state therefore do not change the latent code. The training procedures also differ. Latent SF learns a joint flow in the extended state space $(x,z)$, while VSF uses an encoder to infer latent codes from observed local velocities and trains the conditional velocity field with an energy-score reconstruction and KL regularization. **(2) Latent ODE and Neural ODE Processes.** Latent ODE (Rubanova et al., 2019) infers an initial latent state from observations, evolves that state through a learned ODE, and decodes the result into observations. Neural ODE Processes (Norcliffe et al., 2021) also evolves a latent state, but conditions its dynamics on an additional fixed global latent variable. While we agree that the fixed dynamics latent in Neural ODE Processes is structurally similar to VSF's fixed $z$, there is an important difference. VSF directly evolves the forecast state in physical time and trains the velocity field from observed local transitions. In contrast, latent ODE and Neural ODE Processes train through the solution of a learned latent ODE. For VSF, a training pair $(x_s,u_s)$ can be constructed directly from adjacent observations, and the model predicts the corresponding physical-time velocity. Training therefore does not require integrating the learned forecast ODE or backpropagating through its numerical solution. At inference, VSF advances the physical state using one velocity evaluation per transition. We have clarified this positioning in the revised related-work discussion. References: [1] Jiang, Sunshine, Xiaolin Fang, Nicholas Roy, Tomás Lozano-Pérez, Leslie Pack Kaelbling, and Siddharth Ancha. "Streaming Flow Policy: Simplifying diffusion/flow-matching policies by treating action trajectories as flow trajectories." Proceedings of the 9th Conference on Robot Learning (2025). [2] Rubanova, Yulia, Ricky TQ Chen, and David K. Duvenaud. "Latent ordinary differential equations for irregularly-sampled time series." Advances in neural information processing systems 32 (2019). [3] Norcliffe, Alexander, et al. "Neural ODE processes." arXiv preprint arXiv:2103.12413 (2021).
AuthorsOpenReview ↗

Response to Reviewer KgSz (Part 2)

Additional forecasting baselines (W2, Q2)

We added three forecasting baselines: Variational Rectified Flow, Latent ODE, and the stochastic latent variant of Streaming Flow Policy. In the tables below, SF (latent) refers to the jointly evolving state-latent formulation of Jiang et al. (2025), rather than our separate Prior-latent SF + MSE ablation. The following results use the deterministic and stochastic forecasting benchmarks from Tables 1–2.

Deterministic forecasting

DatasetModelCRPS ↓RMSE ↓MAE ↓
Double pendulumVariational rectified flow0.52 ± 0.021.10 ± 0.030.71 ± 0.02
Latent ODE0.70 ± 0.051.30 ± 0.090.83 ± 0.05
SF2.12 ± 0.879.18 ± 5.172.12 ± 0.87
SF (latent)1.56 ± 0.869.53 ± 4.103.04 ± 1.25
VSF0.25 ± 0.070.71 ± 0.110.33 ± 0.08
RösslerVariational rectified flow0.24 ± 0.010.59 ± 0.060.28 ± 0.01
Latent ODE0.90 ± 0.201.85 ± 0.191.14 ± 0.20
SF0.16 ± 0.070.34 ± 0.180.16 ± 0.07
SF (latent)0.15 ± 0.030.45 ± 0.140.21 ± 0.07
VSF0.03 ± 0.000.13 ± 0.010.05 ± 0.00
LorenzVariational rectified flow1.65 ± 0.333.16 ± 0.532.30 ± 0.37
Latent ODE3.25 ± 0.405.13 ± 0.404.18 ± 0.47
SF1.42 ± 0.142.75 ± 0.311.42 ± 0.14
SF (latent)1.14 ± 0.022.77 ± 0.031.60 ± 0.03
VSF0.16 ± 0.020.25 ± 0.020.19 ± 0.01

Stochastic forecasting

DatasetModelCME ↓CSE ↓W1W_1 ↓ED ↓
RösslerVariational rectified flow0.19 ± 0.062.61 ± 1.6012.49 ± 3.2010.31 ± 4.76
Latent ODE0.44 ± 0.182.33 ± 0.3825.21 ± 9.4934.03 ± 17.08
SF0.06 ± 0.021.00 ± 0.004.53 ± 0.675.00 ± 1.34
SF (latent)0.12 ± 0.041.86 ± 0.238.08 ± 1.724.96 ± 1.85
VSF0.04 ± 0.010.36 ± 0.142.62 ± 0.381.28 ± 0.42
LorenzVariational rectified flow0.31 ± 0.020.61 ± 0.0420.06 ± 0.7014.69 ± 1.01
Latent ODE0.44 ± 0.071.39 ± 0.1144.71 ± 5.1114.52 ± 3.85
SF1.04 ± 0.561.00 ± 0.0058.12 ± 24.2389.84 ± 48.45
SF (latent)0.48 ± 0.061.86 ± 0.4355.82 ± 8.7820.02 ± 1.46
VSF0.19 ± 0.030.50 ± 0.2115.95 ± 5.645.19 ± 0.93

VSF achieves the lowest mean error on every reported metric across these five forecasting tasks. In particular, it consistently outperforms the stochastic latent variant of SF, showing that simply adding an evolving latent to SF does not reproduce the gains of VSF.

AuthorsOpenReview ↗

Response to Reviewer KgSz (Part 3)

World-model baselines and matched planning budget (W2, W4, Q2)

(1) Additional world-model predictors. We added a flow-matching predictor and the stochastic latent SF predictor to the world-model comparison. We also clarify the planning budget used in Table 4. As specified in Algorithm 4, VSF samples one latent per CEM candidate action sequence and keeps it fixed for that rollout. Each candidate therefore produces one predicted trajectory, as with the deterministic AR predictor. The JEPA encoder, CEM planner, and number of candidate trajectories are held fixed. The table below reports planning success and predictor RMSE using one rollout per candidate action sequence.

PredictorPushT Success ↑RMSE ↓Cube Success ↑RMSE ↓TwoRoom Success ↑RMSE ↓Reacher Success ↑RMSE ↓
AR predictor (LeWM)84.7 ± 4.10.28 ± 0.0169.3 ± 2.50.80 ± 0.0578.7 ± 9.00.67 ± 0.0361.3 ± 8.21.06 ± 0.01
Flow-matching predictor56.7 ± 8.40.82 ± 0.0164.7 ± 2.51.10 ± 0.0480.7 ± 7.40.95 ± 0.0363.3 ± 9.01.28 ± 0.01
SF predictor88.0 ± 2.80.30 ± 0.0476.7 ± 0.90.64 ± 0.0276.7 ± 8.20.59 ± 0.0549.3 ± 5.01.13 ± 0.03
SF (latent) predictor38.7 ± 8.10.76 ± 0.0370.7 ± 1.91.02 ± 0.0281.3 ± 7.70.65 ± 0.0231.3 ± 7.71.17 ± 0.01
VSF predictor92.7 ± 2.50.27 ± 0.0380.0 ± 1.60.65 ± 0.0882.7 ± 5.00.58 ± 0.0168.0 ± 0.01.03 ± 0.02

VSF achieves the highest planning success on all four tasks and the lowest predictor RMSE on three of them. The SF predictor has slightly lower RMSE on Cube. The latent SF predictor performs worse than VSF on both planning success and predictor RMSE across all four tasks.

(2) Additional rollout-budget control. The original Table 4 comparison used LeWM's publicly available pretrained AR predictor. We therefore also trained an AR predictor from scratch using the same dataset split and training schedule as VSF. We compared the methods using one and three rollouts per CEM candidate action sequence.

  • One rollout per candidate action sequence (Table 4 budget). AR generated one deterministic rollout, and VSF generated one rollout from a single latent draw. VSF achieved 80.9%80.9\% average planning success, compared with 72.5%72.5\% for AR, and had lower predictor RMSE on all four tasks.
  • Three rollouts per candidate action sequence. The AR ensemble generated one rollout from each of three trained predictors, and VSF generated three rollouts from three latent draws. VSF’s advantage persisted, with 81.7%81.7\% average planning success compared with 74.2%74.2\% for the AR ensemble.
Predictor# of samplesPushT Success ↑Cube Success ↑TwoRoom Success ↑Reacher Success ↑
AR predictor (scratch)187.3 ± 4.158.0 ± 4.381.3 ± 6.663.3 ± 5.0
Ensembled AR predictor*390.7 ± 0.959.3 ± 5.282.0 ± 5.964.7 ± 2.5
VSF predictor192.7 ± 2.580.0 ± 1.682.7 ± 5.068.0 ± 0.0
VSF predictor*392.0 ± 1.681.3 ± 1.984.7 ± 7.468.7 ± 0.9

We have added these comparisons to Section L of the revised manuscript.

Reviewer_EFCpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Variational Streaming Flow (VSF), a probabilistic extension of Streaming Flow (SF). SF integrates a learned velocity field directly in physical time, which is efficient but deterministic: a fixed history yields a single future, which fails when stochasticity or partial observation makes multiple futures compatible with the same history. VSF conditions the velocity field on a latent z sampled once per rollout; during training an encoder infers z from the observed local transition (state and velocity) with KL regularization toward a Gaussian prior, and the field is trained with an energy-score loss over multiple latent samples. Experiments on deterministic and partially observed chaotic systems and a JEPA world model show that VSF outperforms SF and diffusion/flow baselines in long-horizon forecasting, distribution matching, and planning success.

Critical strengths
  1. Although VSF is an incremental extension of SF, the problem setting is clearly stated, the connection from motivation (a single deterministic future vs. multiple compatible futures, App. B) to the proposed method is well explained, and the method itself is simple: one latent per rollout, a standard encoder–KL setup, and an energy-score reconstruction. The experiments are designed to directly support the claims made.

  2. The empirical support is thorough. VSF is compared against baselines across deterministic, partially observed stochastic, and world-model settings, with long-horizon rollouts up to 2,500 steps and distribution-level metrics and achieves the best results in nearly all cases while retaining SF’s one-evaluation-per-step cost.

Critical weaknesses
  1. The large improvements on deterministic systems are not explained by the paper’s motivation (bifurcation, App. B), the “anchor” interpretation in App. J is unverified, and the comparison in Fig. 12 confounds the addition of a latent variable with a change in the loss.

  2. The uncertainty in the stochastic benchmarks comes solely from a single hidden constant parameter per trajectory (App. H: ρ or c is sampled once and then held fixed). VSF uses a single fixed z per rollout, so its inductive bias exactly matches the data-generating process. It is therefore natural for VSF to outperform other methods in this setting, but whether these results generalize to process noise (randomness introduced at every step) or time-varying uncertainty has not been tested at all. The finding in Fig. 11 that “resampling z at every step performs worse” is also expected on this benchmark; the opposite could hold under process noise. Since the abstract and introduction motivate VSF with intrinsic randomness and claim results on “stochastic dynamical systems” generally, the scope of applicability is overstated relative to what the benchmark design can support.

Questions for authors
  1. In a fully observed deterministic setting, (x,s,h)(x, s, h) determines the future, so one would expect the model to ignore zz under KL pressure. Yet Fig. 10 and Table 1 show large gains from zz over SF. App. J attributes this to zz acting as an uncontaminated “anchor,” unlike the self-refreshed hh, but this is not tested. Fig. 12 shows that prior-sampled zz without an encoder already beats SF, but that comparison changes both the latent and the loss (MSE → energy score); since the energy score reduces to MSE for a non-latent model, the missing control is the latent variant trained with MSE (the MSE-trained VSF in Fig. 8 still includes the encoder). Could you add Prior-latent SF trained with MSE to separate the two contributions, and (b) test the anchor interpretation by injecting noise into hh at inference and comparing how the error of SF and VSF grows with the noise level?

  2. Please evaluate VSF on a system whose uncertainty is not constant along the trajectory, e.g. Lorenz with fixed ρ and Gaussian process noise added at every integration step (or one of the SDE systems in StocBench, which the paper already cites), using the same conditional-distribution metrics and baselines as Sec. 4.2. Please also repeat the fixed-vs-resampled latent comparison of Fig. 11 in this setting. If VSF’s advantage does not carry over, the claims should be scoped to trajectory-level (constant) latent uncertainty.

4 comments
AuthorsOpenReview ↗

Response to Reviewer EFCp (Part 1)

We thank the reviewer for the constructive feedback.

Deterministic gains and anchor interpretation (W1, Q1)

(1) Effect of latent conditioning versus the training loss. To separate these effects, we added a Prior-latent SF + MSE baseline on the deterministic double pendulum. This model samples z∼p(z)z\sim p(z) during both training and forecasting, without a posterior encoder or KL regularization. We compare it with standard SF, Prior-latent SF trained with the energy score, and full VSF.

VariantCRPS ↓RMSE ↓
SF2.116 ± 0.8689.176 ± 5.167
Prior-latent SF + MSE0.294 ± 0.0361.260 ± 0.484
Prior-latent SF + energy score0.255 ± 0.0601.213 ± 0.734
Full VSF0.254 ± 0.0700.712 ± 0.111

With MSE training in both cases, adding a latent reduces CRPS from 2.116 to 0.294 and RMSE from 9.176 to 1.260. Changing the prior-latent model's loss from MSE to the energy score gives a much smaller improvement. Full VSF further reduces RMSE to 0.712. These results show that most of the improvement over SF comes from latent conditioning rather than the change in training loss. We added this comparison to Appendix J, Full ablation results.

(2) Testing the anchor interpretation. To test whether latent conditioning makes the predictor less sensitive to errors in the observed history, we added Gaussian noise to the history at inference, h~=h+σhϵ\widetilde{h}=h+\sigma_h\epsilon, with ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0,I) and σh∈{0,0.05,…,0.25}\sigma_h\in\{0,0.05,\ldots,0.25\}. We measured the increase in forecasting error relative to the unperturbed history, ΔE(σh)=E(σh)−E(0)\Delta E(\sigma_h)=E(\sigma_h)-E(0). The table below reports the results at σh=0.25\sigma_h=0.25.

ModelΔ\Delta CRPS ↓Δ\Delta RMSE ↓Δ\Delta MAE ↓
SF3.444 ± 3.4276.834 ± 7.6113.444 ± 3.427
Prior-latent SF + MSE1.238 ± 0.2275.643 ± 1.8671.354 ± 0.194
Full VSF0.863 ± 0.4584.922 ± 3.2341.116 ± 0.547

Both latent-conditioned models have smaller error increases than SF, with full VSF performing best across all three metrics. This supports our explanation that latent conditioning makes the predictor less sensitive to errors in the observed history. We added the full noise-sensitivity curves and discussion to Appendix J, Why does VSF improve upon SF on deterministic systems?

AuthorsOpenReview ↗

Response to Reviewer EFCp (Part 2)

Fixed-latent inductive bias and process noise (W2, Q2)

In our original stochastic benchmarks, a hidden parameter is sampled once per trajectory. To test systems where new noise is added at every integration step, we ran experiments on two additional benchmarks:

  1. Stochastic Kolmogorov flow from StocBench: An image-valued spatiotemporal PDE benchmark on a 64×6464\times64 grid, with 1,500 trajectories and 200 frames per trajectory. Training takes approximately 36 hours on 16 A100 GPUs.
  2. Lorenz with additive process noise: We fix ρ\rho and add independent Gaussian noise at every integration step. We evaluate forecasts over 1,000 steps.

We have included both benchmarks in the revised manuscript.

Stochastic Kolmogorov flow

ModelCME ↓CSE ↓ED ↓ESE ↓
AR diffusion0.048 ± 0.0010.177 ± 0.0040.081 ± 0.0012279.280 ± 1954.170
Rectified flow0.040 ± 0.0010.332 ± 0.0030.136 ± 0.00144.644 ± 0.654
SF0.070 ± 0.0011.000 ± 0.0000.468 ± 0.00076.942 ± 0.008
SF (perturb.)0.068 ± 0.0000.961 ± 0.0000.445 ± 0.00077.107 ± 0.030
SF (latent)0.189 ± 0.0000.981 ± 0.0000.494 ± 0.00075.577 ± 0.003
VSF0.068 ± 0.0010.079 ± 0.0060.063 ± 0.00216.577 ± 0.251

Lorenz with additive Gaussian process noise (1,000 steps)

ModelCME ↓CSE ↓W1W_1 ↓ED ↓
AR diffusion0.287 ± 0.0080.342 ± 0.02534.674 ± 0.9595.899 ± 0.374
Rectified flow0.517 ± 0.0430.876 ± 0.10636.253 ± 0.39318.718 ± 1.777
SF0.631 ± 0.0431.000 ± 0.00040.474 ± 1.20428.991 ± 2.413
SF (perturb.)0.592 ± 0.0310.834 ± 0.03938.521 ± 1.02219.179 ± 1.485
SF (latent)0.310 ± 0.0560.307 ± 0.06434.349 ± 1.1325.615 ± 2.045
VSF0.277 ± 0.0300.248 ± 0.01933.316 ± 1.2881.987 ± 0.520

On stochastic Kolmogorov flow, rectified flow achieves the lowest CME, while VSF achieves the lowest CSE, ED, and ESE. On Lorenz, VSF achieves the lowest mean values across all four metrics. These results show that VSF also performs well when the system receives new process noise at every step.

Fixed versus resampled latent on Lorenz (process noise). We also repeated the fixed-versus-resampled latent experiment on the noisy Lorenz benchmark, as suggested by the reviewer.

VSF variantCME ↓CSE ↓W1W_1 ↓ED ↓
Fixed latent0.277 ± 0.0300.248 ± 0.01933.316 ± 1.2881.987 ± 0.520
Resampled latent0.352 ± 0.0500.250 ± 0.01734.714 ± 2.1723.585 ± 1.250

Keeping zz fixed gives lower CME, W1W_1, and ED, while CSE remains nearly unchanged. Thus, resampling zz at every step does not improve forecasting in this process-noise setting.

Our additional analysis helps explain this result. Latent codes inferred from different steps of the same trajectory are closer to one another than codes from different trajectories (Section K, Figure 14). Resampling zz can therefore cause the predicted trajectory to switch between different learned behaviors (Section J, Figure 10; Section K, Figure 15). We also find that resampling can favor the majority branch when the training data contain imbalanced trajectory branches (Section K, Figure 13).

VSF does not sample new noise at every forecast step. Instead, different fixed latent samples produce different complete trajectories. The new experiments show that this approach can accurately forecast trajectory distributions on the two tested process-noise benchmarks, even though the underlying systems receive new noise throughout the rollout.

Thank you for the additional experiments. The Prior-latent SF + MSE control and the noise experiments make the paper’s claims more robust. I will maintain my score.

AuthorsOpenReview ↗

Response to Reviewer EFCp

Thank you for taking the time to review our additional experiments. We are glad that the Prior latent SF + MSE ablations and the noise experiments helped address your concerns. We appreciate your constructive suggestions, which have helped us improve the paper.

Reviewer_veB8OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Streaming Flow forecasts dynamical systems by learning a velocity field directly in physical time, but its deterministic field yields a single future from a fixed history — problematic near bifurcations and under partial observability. VSF conditions the physical-time velocity field on a latent variable z sampled once per rollout and held fixed. Training uses an encoder over local transition velocities, with (a) a conditional-velocity ELBO motivating the latent model and (b) a generalized energy-score reconstruction over M latent samples (first term pulls predictions to the observed velocity, second penalizes collapse of the sample spread), plus KL-to-prior regularization. Under aggregate posterior–prior alignment, the energy reconstruction upper-bounds (half) the energy distance between prior-predictive and true conditional velocity distributions. Experiments cover three settings: long-horizon rollouts on classic chaotic systems, partially observed stochastic variants, and action-conditioned prediction within a JEPA world model — VSF achieves the best accuracy and calibration against a broad baseline set at a fraction of diffusion methods' runtime, and its forecasts translate into improved downstream planning success.

Critical strengths
  1. A well-targeted fix that preserves the very property that made the parent method attractive. VSF adds distributional forecasting to physical-time flows while keeping one velocity evaluation per transition — the property that made SF attractive. The results show this is not a trade-off: VSF is simultaneously the most accurate and best calibrated, at a fraction of diffusion methods' runtime.
  2. Unusually complete evaluation arc for a methods paper. Deterministic → partially observed stochastic → bifurcating regimes → downstream planning in a JEPA world model, with proper distributional metrics (trajectory-level W1/ED, not just pointwise error). The +7.34% planning-success gain at the same planning budget shows the forecast quality is decision-relevant, not just metric-deep.
Critical weaknesses
  1. All test systems are low-dimensional toys. Double pendulum, Rössler, and Lorenz are 3–6 dimensional; the motivation cites weather and complex real systems, but nothing tests high-dimensional state, real sensor noise, or real-world time series. The stochastic "partial observability" is synthetic (a hidden scalar parameter resampled), which is a friendlier setting than genuine process noise. Generalization of the claims remains open.
  2. Single global latent limits the noise model. One z per rollout can select among trajectory modes but cannot represent within-rollout aleatoric variability; the Gaussian local likelihood is only a training device, so at inference all stochasticity flows through z. For systems with continual stochastic forcing (rather than branch selection), the model class may be mismatched — a per-step or hierarchical latent seems the natural extension and is not explored.
Questions for authors
  1. How does VSF behave on systems with continuous stochastic forcing (e.g., additive-noise SDEs), where a single global latent cannot generate new noise along the rollout?
  2. Why does SF perform so poorly on the deterministic double pendulum (CRPS 2.12 vs 0.25) — is this the bifurcation sensitivity of App. B, or an implementation issue? A brief diagnosis would help readers trust the baseline.
  3. In the world-model experiments, is the VSF predictor trained with the same data/steps as the AR predictor, and does the gain persist with a stochastic AR baseline?
4 comments
AuthorsOpenReview ↗

Response to Reviewer veB8 (Part 1)

We thank the reviewer for the constructive feedback.

Benchmark scope and continuous stochastic forcing (W1, W2, Q1)

To address the concerns about benchmark dimensionality and continuous stochastic forcing, we ran additional experiments on two new benchmarks:

  1. Stochastic Kolmogorov flow from StocBench: An image-valued spatiotemporal PDE benchmark on a 64×6464 \times 64 grid, containing 1,500 trajectories with 200 frames each. Training requires approximately 36 hours on 16 A100 GPUs.
  2. Lorenz with additive process noise: We fix the parameter ρ\rho and introduce independent Gaussian noise at every integration step, producing stochastic trajectories over 1,000 steps.

In our original stochastic benchmarks, a hidden parameter is sampled once per trajectory. In the new benchmarks, process noise is added at every integration step. We report the results below and have included both benchmarks in the revised manuscript.

Stochastic Kolmogorov flow

ModelCME ↓CSE ↓ED ↓ESE ↓
AR diffusion0.048 ± 0.0010.177 ± 0.0040.081 ± 0.0012279.280 ± 1954.170
Rectified flow0.040 ± 0.0010.332 ± 0.0030.136 ± 0.00144.644 ± 0.654
SF0.070 ± 0.0011.000 ± 0.0000.468 ± 0.00076.942 ± 0.008
SF (perturb.)0.068 ± 0.0000.961 ± 0.0000.445 ± 0.00077.107 ± 0.030
SF (latent)0.189 ± 0.0000.981 ± 0.0000.494 ± 0.00075.577 ± 0.003
VSF0.068 ± 0.0010.079 ± 0.0060.063 ± 0.00216.577 ± 0.251

Lorenz with additive Gaussian process noise (1,000 steps)

ModelCME ↓CSE ↓W1W_1 ↓ED ↓
AR diffusion0.287 ± 0.0080.342 ± 0.02534.674 ± 0.9595.899 ± 0.374
Rectified flow0.517 ± 0.0430.876 ± 0.10636.253 ± 0.39318.718 ± 1.777
SF0.631 ± 0.0431.000 ± 0.00040.474 ± 1.20428.991 ± 2.413
SF (perturb.)0.592 ± 0.0310.834 ± 0.03938.521 ± 1.02219.179 ± 1.485
SF (latent)0.310 ± 0.0560.307 ± 0.06434.349 ± 1.1325.615 ± 2.045
VSF0.277 ± 0.0300.248 ± 0.01933.316 ± 1.2881.987 ± 0.520

On stochastic Kolmogorov flow, rectified flow achieves the lowest CME, while VSF achieves the lowest CSE, ED, and ESE. On Lorenz, VSF achieves the lowest mean values across all four metrics. These results extend our evaluation to a higher-dimensional PDE and show that VSF also performs well when the underlying system faces noise at every step.

Fixed versus resampled latent on Lorenz (process noise). We also tested whether resampling zz at every step improves forecasting on the noisy Lorenz system.

VSF variantCME ↓CSE ↓W1W_1 ↓ED ↓
Fixed latent0.277 ± 0.0300.248 ± 0.01933.316 ± 1.2881.987 ± 0.520
Resampled latent0.352 ± 0.0500.250 ± 0.01734.714 ± 2.1723.585 ± 1.250

Keeping zz fixed gives lower CME, W1W_1, and ED, with nearly the same CSE. We attribute this difference to two observations from our additional analysis.

  1. Latent codes from different steps of the same trajectory are closer to one another than codes from different trajectories (Section K, Figure 14). Resampling zz can therefore cause the predicted trajectory to switch between different learned behaviors (Section J, Figure 10; Section K, Figure 15).
  2. When the training data contain imbalanced trajectory branches, resampling zz can cause predictions to favor the majority branch. Keeping zz fixed reduces this effect (Section K, Figure 13).

VSF does not explicitly inject new noise at every step. Instead, different latent samples produce different complete trajectories. The new experiments show that this is sufficient to obtain accurate forecast distributions on the two tested process-noise benchmarks, and that resampling zz does not improve performance on stochastic Lorenz.

AuthorsOpenReview ↗

Response to Reviewer veB8 (Part 2)

2. Poor SF performance on the deterministic double pendulum (Q2)

We investigated the large forecasting error of SF on the deterministic double pendulum through additional ablations. The results suggest that the main problem is error accumulation during long rollouts, and that latent conditioning helps reduce it.

Deterministic double-pendulum ablation (Figure 11). Adding a prior-sampled latent to SF, while retaining MSE training, reduces CRPS from 2.116 to 0.294 and RMSE from 9.176 to 1.260. This shows that much of the improvement comes from latent conditioning, even without the full VSF training objective. Full VSF further reduces RMSE to 0.712. One possible explanation is that the latent acts as an additional input to the velocity predictor. During training, different latent samples change the features at the network bottleneck. This may make the predictor less sensitive to errors in the observed history, which can accumulate during a long rollout.

VariantCRPS ↓RMSE ↓
SF2.116 ± 0.8689.176 ± 5.167
Prior-latent SF + MSE0.294 ± 0.0361.260 ± 0.484
Full VSF0.254 ± 0.0700.712 ± 0.111

History-noise ablation: Δ\Delta-error at σh=0.25\sigma_h=0.25. To test this, we added Gaussian noise to the initial observation history at inference. We measured how much each model's error increased relative to its performance with an unperturbed history, using ΔE(σh)=E(σh)−E(0)\Delta E(\sigma_h)=E(\sigma_h)-E(0).

ModelΔ\DeltaCRPS ↓Δ\DeltaRMSE ↓Δ\DeltaMAE ↓
SF3.444 ± 3.4276.834 ± 7.6113.444 ± 3.427
Prior-latent SF + MSE1.238 ± 0.2275.643 ± 1.8671.354 ± 0.194
Full VSF0.863 ± 0.4584.922 ± 3.2341.116 ± 0.547

Both latent-conditioned models show smaller increases in error than SF, and full VSF performs best across all three metrics. These results support our explanation that SF is more sensitive to errors in the observed history, which become increasingly important over a 1,000-step rollout. We have added these experiments and the corresponding discussion to Appendix J of the revised manuscript.

Question 3 — World-model training parity and stochastic AR baseline

The original comparison in Table 4 used LeWM's publicly available pretrained AR predictor. To ensure a fair training comparison, we also trained an AR predictor from scratch using the same dataset split and training schedule as VSF. We kept the JEPA encoder and CEM planner fixed.

  • One rollout per candidate action sequence (Table 4 budget). AR generated one deterministic rollout, and VSF generated one rollout from a single latent draw. VSF achieved 80.9%80.9\% average planning success, compared with 72.5%72.5\% for AR, and had lower predictor RMSE on all four tasks.
  • Three rollouts per candidate action sequence. The AR ensemble generated one rollout from each of three trained predictors, and VSF generated three rollouts from three latent draws. VSF’s advantage persisted, with 81.7%81.7\% average planning success compared with 74.2%74.2\% for the AR ensemble.
Predictor# of samplesPushT Success ↑Cube Success ↑TwoRoom Success ↑Reacher Success ↑
AR predictor (scratch)187.3 ± 4.158.0 ± 4.381.3 ± 6.663.3 ± 5.0
Ensembled AR predictor*390.7 ± 0.959.3 ± 5.282.0 ± 5.964.7 ± 2.5
VSF predictor192.7 ± 2.580.0 ± 1.682.7 ± 5.068.0 ± 0.0
VSF predictor*392.0 ± 1.681.3 ± 1.984.7 ± 7.468.7 ± 0.9

We report the updated comparison in Section L of the revised manuscript.

Thanks for the additional experiments. The Kolmogorov-flow and noisy-Lorenz results strengthen the evaluation, and the double-pendulum ablations plausibly explain SF's poor performance. The from-scratch AR comparison and three-sample control also resolve my concerns about training parity. I still lean slightly positive.

AuthorsOpenReview ↗

Response to Reviewer veB8

Thank you for taking the time to review our additional experiments. We are glad that the new results helped address your concerns. We appreciate your constructive suggestions, which have helped us improve the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewers for their constructive feedback. We have uploaded a revised manuscript with additional experiments and ablations. Below we summarize the main updates:

Highlights

  • Forecasting baselines (Table 1). We added Variational Rectified Flow, Latent ODE, and latent Streaming Flow. VSF achieved the lowest error by comparing with these new baselines.
  • Forecasting with process noise (Table 3). We added 2 new datasets with process noise: stochastic Kolmogorov flow and Lorenz with additive noise. VSF achieves strong long-horizon forecasting performance under process noise.
  • World-model planning (Tables 4 and 8). We added flow-matching and latent SF baselines. With an increased planning budget, VSF achieved higher average planning success than the AR ensemble.
  • Why VSF works well on the deterministic double pendulum (Appendix J, Figures 11–12). Varying decoder features through latent codes during training may make VSF more robust to errors. When we perturbed the input history, SF showed a larger increase in forecast error than VSF.
  • Modeling branching futures with latent codes (Appendix K, Figures 13–15). We show that VSF captures the likelihood of future branches at a bifurcation and that its posterior latent codes distinguish which branch a trajectory follows.
  • Additional experiments. We tested partial observations, noisy initial observations, out-of-distribution initial conditions, and a new stochastic PushT benchmark.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.