- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper builds on Streaming Flow (SF; Jiang et al., 2025), which forecasts a dynamical system by integrating a learned velocity field directly in physical forecast time . Because the SF velocity field is deterministic, it gives one future for a fixed history. The proposed Variational Streaming Flow (VSF) adds a latent variable, sampled once per forecast and kept fixed through the rollout. Training follows from a VAE recipe, with the reconstruction term a generalized energy score over latent samples. VSF is evaluated on
- three deterministic chaotic systems (double pendulum, Rössler, Lorenz);
- two "stochastic" Rössler/Lorenz systems with a hidden, per-trajectory parameter;
- as a predictor in the LeWM JEPA world model with a CEM planner on four tasks (PushT, Cube, TwoRoom, Reacher). and reports the best results in all main tables.
Critical strengths
- Very clear writing and presentation, easy to follow.
- The idea is simple and easy to plug into existing SF models. A single latent per rollout adds sample diversity at little extra cost.
- Long-horizon evaluation shows advantages over rollouts of 1,000 and 2,500 steps, with per-timestep error curves (Fig. 13–14).
Critical weaknesses
W1. Limited novelty relative to SF. The method adds two standard components to SF.
- Latent with encoder and KL. An encoder-inferred latent with a KL term toward is the VAE recipe already used in variational flow matching [1].
- Energy-score reconstruction. Energy-score reconstruction over multiple samples is commonly used to train generative forecasters.
- One latent per sequence. Holding one latent fixed over a sequence is also not new, e.g.:
- Latent ODE [2] and Neural ODE Processes [3] keep a per-trajectory latent in a physical-time ODE.
- The SF paper itself contains a latent "stochastic variant" (Jiang et al., 2025, App. B). The present paper only refers to it indirectly ("SFP + auxiliary latent", Table 4) and does not compare against it.
[1] Guo, Pengsheng, and Alexander G. Schwing. "Variational rectified flow matching." arXiv preprint arXiv:2502.09616 (2025).
[2] Rubanova, Yulia, Ricky TQ Chen, and David K. Duvenaud. "Latent ordinary differential equations for irregularly-sampled time series." Advances in neural information processing systems 32 (2019).
[3] Norcliffe, Alexander, et al. "Neural ode processes." arXiv preprint arXiv:2103.12413 (2021).
W2. Missing stochastic and latent-variable baselines. The only flow-based baselines are SF and an in-house autoregressive rectified flow. The most direct missing baseline is the latent "stochastic variant" in the SF paper itself (Jiang et al., 2025, App. B). It is the existing way of making SF stochastic, so it is the natural comparison for VSF in both the forecasting and world-model experiments. The paper also does not compare against other ways of making flow-based forecasters stochastic:
- Forecasting: [1,2] and published flow-matching forecasters cited in App. A
- World model: the only baselines are LeWM's pretrained AR predictor and SF. There is no stochastic predictor, e.g.:
- the SFP latent variant used as the predictor;
- a flow-matching predictor in LeWM's latent space (as in FlowWM, Porcher et al., 2026); Therefore It is unclear whether the gains come from VSF's specific variational latent or from any source of stochasticity.
W3. Benchmarks lack real randomness or partial observability. This is the setting where a probabilistic predictor should matter, but the benchmarks largely avoid it.
- Forecasting systems.
- All forecasting systems are low-dimensional ( to ), noise-free ODEs (App. H).
- The "stochastic" systems are deterministic given a hidden parameter, with no process noise, even though StocBench (Pfister et al., 2026) is cited.
- The Lorenz parameter range sits at the point where Lorenz's nonzero fixed points lose stability (). So the different futures are mainly "settles to a fixed point" vs. "stays chaotic". This choice is not discussed.
- Near-duplicate initial conditions.
- Rössler starts from only 8 centers, , and Lorenz from a single center , both with perturbation std 0.01 (App. H.1).
- The data are split by trajectory, so test trajectories start very close to training ones. The test does not check generalization to new initial conditions.
- World-model tasks are deterministic. PushT, Cube, TwoRoom and Reacher give the same next state for the same action, so there is no future uncertainty for VSF to capture. No distributional metric is reported for these tasks.
- The bifurcation motivation is never tested directly. No experiment checks whether VSF assigns the correct probability to each future branch.
W4. World-model evaluation is not fully specified or controlled.
- Fig. 2 says VSF "samples multiple latents" per action sequence, while Alg. 4 draws one . I did not find how many latents are used per CEM candidate or how their costs are combined. If several are used, the planning compute is not matched to the AR predictor, which contradicts the "fixed planning budget" claim.
Questions for authors
- Novelty and positioning.
- How does VSF differ from the latent "stochastic variant" in the SF paper (Jiang et al., 2025, App. B)?
- How does it differ from Latent ODE / Neural ODE Processes, which also keep a per-trajectory latent in a physical-time ODE?
- Additional stochastic baselines.
- Could you compare against the latent "stochastic variant" of SF (Jiang et al., 2025, App. B) in both the forecasting experiments (Tables 1–2) and as a world-model predictor (Table 3)? This is the existing way of making SF stochastic, so it seems the most direct baseline.
- Could you also compare against Variational Rectified FM?
- For the world model, could you add at least one other stochastic predictor, e.g.:
- a flow-matching predictor in LeWM's latent space;
- an RSSM-style predictor;
- an ensemble of AR predictors?
- Benchmarks with real randomness or partial observability. Could you evaluate on settings where a probabilistic predictor is needed? Concretely:
- Process noise: SDE systems such as those in StocBench (cited), or stochastic Lorenz-63/Lorenz-96 with additive noise.
- Known branch probabilities: a bistable system such as a double-well Langevin process, to check whether the predicted fraction of samples in each branch matches the truth.
- Partial observability:
- observing only the -coordinate of Lorenz;
- observing only positions (not velocities) of the double pendulum;
- adding observation noise.
- New initial conditions: test initial conditions drawn away from the training centers.
- stochastic video benchmarks such as the Bouncing Shapes task used by FlowWM.
- VSF performs much better than all baselines, to what extent does the way of the latent is being incorporated matters, e.g. how it is being conditioned in the architecture? Is the posterior meaningful here and gives multimodal distribution under bifurcation? Why does VSF help so much on the deterministic double pendulum, where there is a single future?
10 comments
Response to Reviewer KgSz (Part 2)
Additional forecasting baselines (W2, Q2)
We added three forecasting baselines: Variational Rectified Flow, Latent ODE, and the stochastic latent variant of Streaming Flow Policy. In the tables below, SF (latent) refers to the jointly evolving state-latent formulation of Jiang et al. (2025), rather than our separate Prior-latent SF + MSE ablation. The following results use the deterministic and stochastic forecasting benchmarks from Tables 1–2.
Deterministic forecasting
| Dataset | Model | CRPS ↓ | RMSE ↓ | MAE ↓ |
|---|---|---|---|---|
| Double pendulum | Variational rectified flow | 0.52 ± 0.02 | 1.10 ± 0.03 | 0.71 ± 0.02 |
| Latent ODE | 0.70 ± 0.05 | 1.30 ± 0.09 | 0.83 ± 0.05 | |
| SF | 2.12 ± 0.87 | 9.18 ± 5.17 | 2.12 ± 0.87 | |
| SF (latent) | 1.56 ± 0.86 | 9.53 ± 4.10 | 3.04 ± 1.25 | |
| VSF | 0.25 ± 0.07 | 0.71 ± 0.11 | 0.33 ± 0.08 | |
| Rössler | Variational rectified flow | 0.24 ± 0.01 | 0.59 ± 0.06 | 0.28 ± 0.01 |
| Latent ODE | 0.90 ± 0.20 | 1.85 ± 0.19 | 1.14 ± 0.20 | |
| SF | 0.16 ± 0.07 | 0.34 ± 0.18 | 0.16 ± 0.07 | |
| SF (latent) | 0.15 ± 0.03 | 0.45 ± 0.14 | 0.21 ± 0.07 | |
| VSF | 0.03 ± 0.00 | 0.13 ± 0.01 | 0.05 ± 0.00 | |
| Lorenz | Variational rectified flow | 1.65 ± 0.33 | 3.16 ± 0.53 | 2.30 ± 0.37 |
| Latent ODE | 3.25 ± 0.40 | 5.13 ± 0.40 | 4.18 ± 0.47 | |
| SF | 1.42 ± 0.14 | 2.75 ± 0.31 | 1.42 ± 0.14 | |
| SF (latent) | 1.14 ± 0.02 | 2.77 ± 0.03 | 1.60 ± 0.03 | |
| VSF | 0.16 ± 0.02 | 0.25 ± 0.02 | 0.19 ± 0.01 |
Stochastic forecasting
| Dataset | Model | CME ↓ | CSE ↓ | ↓ | ED ↓ |
|---|---|---|---|---|---|
| Rössler | Variational rectified flow | 0.19 ± 0.06 | 2.61 ± 1.60 | 12.49 ± 3.20 | 10.31 ± 4.76 |
| Latent ODE | 0.44 ± 0.18 | 2.33 ± 0.38 | 25.21 ± 9.49 | 34.03 ± 17.08 | |
| SF | 0.06 ± 0.02 | 1.00 ± 0.00 | 4.53 ± 0.67 | 5.00 ± 1.34 | |
| SF (latent) | 0.12 ± 0.04 | 1.86 ± 0.23 | 8.08 ± 1.72 | 4.96 ± 1.85 | |
| VSF | 0.04 ± 0.01 | 0.36 ± 0.14 | 2.62 ± 0.38 | 1.28 ± 0.42 | |
| Lorenz | Variational rectified flow | 0.31 ± 0.02 | 0.61 ± 0.04 | 20.06 ± 0.70 | 14.69 ± 1.01 |
| Latent ODE | 0.44 ± 0.07 | 1.39 ± 0.11 | 44.71 ± 5.11 | 14.52 ± 3.85 | |
| SF | 1.04 ± 0.56 | 1.00 ± 0.00 | 58.12 ± 24.23 | 89.84 ± 48.45 | |
| SF (latent) | 0.48 ± 0.06 | 1.86 ± 0.43 | 55.82 ± 8.78 | 20.02 ± 1.46 | |
| VSF | 0.19 ± 0.03 | 0.50 ± 0.21 | 15.95 ± 5.64 | 5.19 ± 0.93 |
VSF achieves the lowest mean error on every reported metric across these five forecasting tasks. In particular, it consistently outperforms the stochastic latent variant of SF, showing that simply adding an evolving latent to SF does not reproduce the gains of VSF.
Response to Reviewer KgSz (Part 3)
World-model baselines and matched planning budget (W2, W4, Q2)
(1) Additional world-model predictors. We added a flow-matching predictor and the stochastic latent SF predictor to the world-model comparison. We also clarify the planning budget used in Table 4. As specified in Algorithm 4, VSF samples one latent per CEM candidate action sequence and keeps it fixed for that rollout. Each candidate therefore produces one predicted trajectory, as with the deterministic AR predictor. The JEPA encoder, CEM planner, and number of candidate trajectories are held fixed. The table below reports planning success and predictor RMSE using one rollout per candidate action sequence.
| Predictor | PushT Success ↑ | RMSE ↓ | Cube Success ↑ | RMSE ↓ | TwoRoom Success ↑ | RMSE ↓ | Reacher Success ↑ | RMSE ↓ |
|---|---|---|---|---|---|---|---|---|
| AR predictor (LeWM) | 84.7 ± 4.1 | 0.28 ± 0.01 | 69.3 ± 2.5 | 0.80 ± 0.05 | 78.7 ± 9.0 | 0.67 ± 0.03 | 61.3 ± 8.2 | 1.06 ± 0.01 |
| Flow-matching predictor | 56.7 ± 8.4 | 0.82 ± 0.01 | 64.7 ± 2.5 | 1.10 ± 0.04 | 80.7 ± 7.4 | 0.95 ± 0.03 | 63.3 ± 9.0 | 1.28 ± 0.01 |
| SF predictor | 88.0 ± 2.8 | 0.30 ± 0.04 | 76.7 ± 0.9 | 0.64 ± 0.02 | 76.7 ± 8.2 | 0.59 ± 0.05 | 49.3 ± 5.0 | 1.13 ± 0.03 |
| SF (latent) predictor | 38.7 ± 8.1 | 0.76 ± 0.03 | 70.7 ± 1.9 | 1.02 ± 0.02 | 81.3 ± 7.7 | 0.65 ± 0.02 | 31.3 ± 7.7 | 1.17 ± 0.01 |
| VSF predictor | 92.7 ± 2.5 | 0.27 ± 0.03 | 80.0 ± 1.6 | 0.65 ± 0.08 | 82.7 ± 5.0 | 0.58 ± 0.01 | 68.0 ± 0.0 | 1.03 ± 0.02 |
VSF achieves the highest planning success on all four tasks and the lowest predictor RMSE on three of them. The SF predictor has slightly lower RMSE on Cube. The latent SF predictor performs worse than VSF on both planning success and predictor RMSE across all four tasks.
(2) Additional rollout-budget control. The original Table 4 comparison used LeWM's publicly available pretrained AR predictor. We therefore also trained an AR predictor from scratch using the same dataset split and training schedule as VSF. We compared the methods using one and three rollouts per CEM candidate action sequence.
- One rollout per candidate action sequence (Table 4 budget). AR generated one deterministic rollout, and VSF generated one rollout from a single latent draw. VSF achieved average planning success, compared with for AR, and had lower predictor RMSE on all four tasks.
- Three rollouts per candidate action sequence. The AR ensemble generated one rollout from each of three trained predictors, and VSF generated three rollouts from three latent draws. VSF’s advantage persisted, with average planning success compared with for the AR ensemble.
| Predictor | # of samples | PushT Success ↑ | Cube Success ↑ | TwoRoom Success ↑ | Reacher Success ↑ |
|---|---|---|---|---|---|
| AR predictor (scratch) | 1 | 87.3 ± 4.1 | 58.0 ± 4.3 | 81.3 ± 6.6 | 63.3 ± 5.0 |
| Ensembled AR predictor* | 3 | 90.7 ± 0.9 | 59.3 ± 5.2 | 82.0 ± 5.9 | 64.7 ± 2.5 |
| VSF predictor | 1 | 92.7 ± 2.5 | 80.0 ± 1.6 | 82.7 ± 5.0 | 68.0 ± 0.0 |
| VSF predictor* | 3 | 92.0 ± 1.6 | 81.3 ± 1.9 | 84.7 ± 7.4 | 68.7 ± 0.9 |
We have added these comparisons to Section L of the revised manuscript.
Response to Reviewer KgSz (Part 1)
We thank the reviewer for the constructive feedback.
Novelty and positioning relative to prior latent methods (W1, Q1)
We agree that variational latent variables, energy-score training, and fixed trajectory-level latents have been studied before. Our contribution is to combine these ideas with physical-time flow forecasting, where the learned velocity directly advances the predicted state. We clarify the differences from the most closely related methods below.
(1) Latent Streaming Flow. The stochastic variant of Streaming Flow Policy (Jiang et al., 2025) introduces a latent variable that evolves together with the physical state. Given a training trajectory ξ(t), its constructed paths are
xtξ=ξ(t)+e−κt(x0−ξ(0))+σrtz0, ztξ=[1−(1−σ1)t]z0+tξ(t),
where z0∼N(0,I) is sampled independently of the initial physical state.
The two methods differ in their rollout dynamics:
\dot x_t=v_\theta^x(x_t,z_t,t\mid h), \qquad \dot z_t=v_\theta^z(x_t,z_t,t\mid h)$$ $$\text{VSF:}\qquad \dot x_t=v_\theta(x_t,t,h,z_0), \qquad \dot z_t=0.$$ Latent SF jointly evolves $x_t$ and $z_t$. Since both velocity components depend on the evolving state, errors in $x_t$ may also affect subsequent latent updates. VSF instead samples $z_0$ once and keeps it fixed throughout the rollout. Errors in the predicted physical state therefore do not change the latent code. The training procedures also differ. Latent SF learns a joint flow in the extended state space $(x,z)$, while VSF uses an encoder to infer latent codes from observed local velocities and trains the conditional velocity field with an energy-score reconstruction and KL regularization. **(2) Latent ODE and Neural ODE Processes.** Latent ODE (Rubanova et al., 2019) infers an initial latent state from observations, evolves that state through a learned ODE, and decodes the result into observations. Neural ODE Processes (Norcliffe et al., 2021) also evolves a latent state, but conditions its dynamics on an additional fixed global latent variable. While we agree that the fixed dynamics latent in Neural ODE Processes is structurally similar to VSF's fixed $z$, there is an important difference. VSF directly evolves the forecast state in physical time and trains the velocity field from observed local transitions. In contrast, latent ODE and Neural ODE Processes train through the solution of a learned latent ODE. For VSF, a training pair $(x_s,u_s)$ can be constructed directly from adjacent observations, and the model predicts the corresponding physical-time velocity. Training therefore does not require integrating the learned forecast ODE or backpropagating through its numerical solution. At inference, VSF advances the physical state using one velocity evaluation per transition. We have clarified this positioning in the revised related-work discussion. References: [1] Jiang, Sunshine, Xiaolin Fang, Nicholas Roy, Tomás Lozano-Pérez, Leslie Pack Kaelbling, and Siddharth Ancha. "Streaming Flow Policy: Simplifying diffusion/flow-matching policies by treating action trajectories as flow trajectories." Proceedings of the 9th Conference on Robot Learning (2025). [2] Rubanova, Yulia, Ricky TQ Chen, and David K. Duvenaud. "Latent ordinary differential equations for irregularly-sampled time series." Advances in neural information processing systems 32 (2019). [3] Norcliffe, Alexander, et al. "Neural ODE processes." arXiv preprint arXiv:2103.12413 (2021).