- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SPT-Flow, a stochastic continuous-time world model that extends ODEWorld to generate multiple future trajectories through trajectory-level latent conditioning. The authors identify a potential supervision shortcut in teacher-forced training, where ground-truth intermediate states may reduce the need for the model to utilize stochastic latent variables. To address this issue, SPT-Flow introduces free-running rollout matching, supervising integrated trajectories at multiple temporal readouts without intermediate ground-truth conditioning. The authors further introduce posterior specificity and latent-swapping diagnostics to investigate whether the learned stochastic variables contain trajectory-associated information.
Critical strengths
-
Well-motivated research problem. The paper identifies an important issue in stochastic world modeling: introducing latent variables does not necessarily guarantee that they are meaningfully utilized. The analysis of supervision shortcuts provides a useful perspective on the interaction between training objectives and stochastic trajectory representations.
-
Simple and technically coherent approach. The proposed rollout-matching objective is conceptually straightforward and integrates naturally with continuous-time world models. The method preserves the single-integration structure of ODEWorld for each sampled trajectory, with comparable per-sample inference latency.
-
Useful diagnostic analysis. The posterior-specificity and latent-swapping experiments go beyond conventional reconstruction metrics and provide evidence that rollout matching encourages trajectory-associated latent utilization.
-
Extensive ablation studies. The paper evaluates different training budgets, KL configurations, sampling temperatures, and ground-truth re-anchoring strategies. These experiments provide useful insights into the effectiveness and limitations of the proposed training objective.
Critical weaknesses
-
Limited evidence of meaningful multimodal future prediction. The latent-swapping results demonstrate trajectory-associated information, but cannot distinguish meaningful future modes from variations in task identity, timing, or visual details. Therefore, the central claim of modeling multiple possible futures remains only partially supported.
-
Most performance improvements come from rollout matching rather than learned stochastic conditioning. As shown in Table 1, rollout matching without learned latent conditioning already accounts for most of the improvement over ODEWorld. The additional latent-conditioning component provides relatively modest best-of-K gains while reducing mean sample accuracy compared with the rollout control. This raises questions about the incremental contribution of the complete SPT-Flow framework.
-
Insufficiently controlled comparison between teacher forcing and rollout matching. According to Appendix A, the teacher-forced and rollout-trained variants differ not only in their supervision objectives but also in posterior construction, latent sampling, and KL aggregation. Although the re-anchoring experiments partially address this concern, they do not fully isolate the effect of the proposed training objective. Furthermore, Table 4 shows that more frequent re-anchoring can achieve higher prediction accuracy and comparable latent-swapping improvements than fully free-running training. The necessity of the final design therefore remains unclear.
-
Limited evaluation of generalization and downstream utility. Experiments are restricted to held-out trajectories from LIBERO tasks represented during training. The paper does not evaluate unseen-task generalization or whether the generated futures improve downstream robotic control. Additionally, comparisons are matched by optimizer steps rather than training compute, and the rollout control retains random conditioning at inference despite being trained with a zero latent. These factors limit conclusions about computational efficiency and the isolated benefits of learned stochastic conditioning.
Questions for authors
-
Meaningful multimodality: Could the authors clarify whether any existing LIBERO results demonstrate that different latent samples correspond to distinct manipulation strategies under the same initial observation and instruction, rather than variations in timing or visual appearance?
-
Comparison fairness: Appendix A indicates that teacher-forced and rollout-trained models use different posterior constructions and KL aggregation procedures. How much of the observed improvement could be attributed to these differences rather than rollout matching itself?
-
Necessity of fully free-running training: In Table 4, re-anchoring at every readout achieves better PSNR and a comparable latent-swap gap relative to SPT-Flow. How do the authors interpret this result, and what is the specific advantage of removing intermediate ground-truth conditioning entirely?
-
Contribution of learned latent conditioning: Since the rollout control accounts for most of the accuracy improvement and still receives random latent inputs during evaluation, could the authors clarify how the conclusions would change under a strictly deterministic rollout control evaluated with a fixed zero latent?
5.SR of manipulation tasks: Could the authors clarify why similar downstream evaluations are absent from SPT-Flow?
4 comments
Response to Reviewer CQbc (Part 2/3)
W3 & Q2: "How much of the observed improvement could be attributed to these differences rather than rollout matching itself?" A small part. Of SPT-Flow's best-of-10 gain over teacher forcing (1.60 / 2.22 dB, mid / final), the posterior construction and KL aggregation account for 0.24 / 0.26 dB, and rollout supervision for the remaining 1.36 [1.14, 1.59] / 1.96 [1.76, 2.16] dB. Only the supervision makes the latent used.
We trained the controlled arm your question implies. It is teacher-forced, but it uses exactly the rollout arm's latent machinery:
- the pooled posterior over the same four readout latents;
- one ε shared by the four sampled times;
- per-trajectory KL with the same free-bits floor.
Only the supervision differs. Three seeds, same hyperparameters (Table 4, second row):
| Arm | Specificity | Swap gap (dB) | Best-of-10, mid / final | Mean, mid / final | Diversity |
|---|---|---|---|---|---|
| Teacher forcing (Table 1) | 0.009 | 0.00 | 18.22 / 16.20 | 17.86 / 15.87 | 1.44 |
| Teacher forcing, rollout arm's posterior and KL | 0.018 | 0.00 | 18.46 / 16.46 | 18.17 / 16.11 | 1.67 |
| Rollout matching (SPT-Flow) | 0.33 | 1.05 | 19.82 / 18.42 | 18.85 / 17.29 | 2.03 |
With posterior and KL held fixed, teacher forcing again yields a decorative latent: specificity stays far below the 0.25 threshold in all three seeds, and swapping latents between trajectories changes nothing. Changing only the supervision makes the latent specific and useful. The effect is due to the supervision, not to the posterior construction or the KL aggregation (App. A).
The same holds within one network. From this arm's 30k-step checkpoint, 10k further steps of rollout matching make the latent specific (0.35–0.39) and used (swap gap 0.93–1.26 dB) in every seed, and best-of-10 is +1.18 [0.98, 1.38] / +1.88 [1.67, 2.08] dB higher than after 10k further teacher-forced steps, which leave it unused (App. A, Table 5).
W3 & Q3: "what is the specific advantage of removing intermediate ground-truth conditioning entirely?" Your observation sharpens our claim, and the revision adopts it (Sec. 3.3, 4.5).
- What matters. In the re-anchoring variants, the rollout is reset to the ground-truth state at a readout. The loss is still applied to states reached by free Euler integration between readouts. Under teacher forcing, every supervised quantity is a velocity at a ground-truth state. The shortcut is removed by supervising integrated states instead of velocities at ground-truth states, and every rollout variant does this. Accordingly, all of them use the latent: swap gaps are 0.97–1.07 dB (seed 0 for the re-anchored variants, three-seed mean for SPT-Flow), against 0.00 under teacher forcing.
- How the variants differ. Re-anchoring at every readout is the most accurate (+0.12 / +0.16 dB). Fully free running gives the most specific posterior and is the only variant above the pre-set threshold in every seed.
- Revision. The paper now presents rollout matching with optional re-anchoring as one family. Free running is our default because it matches inference, where no ground truth is available. Re-anchoring is a valid choice when accuracy matters most.
Response to Reviewer CQbc (Part 3/3)
W4: "The paper does not evaluate unseen-task generalization." We evaluated all arms on 600 demonstrations from LIBERO-Spatial, LIBERO-Object and LIBERO-Goal (App. M), whose scenes and layouts are not used in training. Three findings, three seeds:
- The latent's contribution transfers—including single-sample mean PSNR. Against the rollout control, SPT-Flow leads in every unseen suite: +0.49 [0.47, 0.52] / +0.58 [0.56, 0.61] dB best-of-10 and +0.36 / +0.35 dB mean PSNR. Its best-of-10 is higher on 96% / 99% of the 600 unseen trajectories, and its energy score is lower on all 600.
- Initializing from ODEWorld improves transfer. Two-stage SPT-Flow (ε input initialized at zero, then 10k steps of Eq. 4) gains +1.14 [0.97, 1.31] dB on the unseen suites over SPT-Flow from scratch, matches it on the test split (20.00 vs 19.82 dB), and still uses the latent (specificity 0.29–0.39, swap gap 0.80–1.47 dB).
- A matched control characterizes the supervision trade-off. From the same ODEWorld checkpoint, 10k further steps of rollout supervision gain 0.56 dB more than teacher forcing on the test split, while teacher forcing is stronger on unseen scenes (17.34 vs 13.02 dB; from scratch, 16.69 for ODEWorld vs 12.39 for SPT-Flow): matching full trajectories from (z₀, z_g) fits training-scene dynamics more tightly (+2.0 dB in-domain), whereas local velocity supervision degrades more conservatively in novel layouts (Sec. 4.5, 5; App. M).
W4: "comparisons are matched by optimizer steps rather than training compute" (App. N). We measured wall-clock time per step on the same 4×A100 node:
| arm | s / step | GPU-h | mid best-of-10 | mid mean | final best-of-10 |
|---|---|---|---|---|---|
| SPT-Flow, 30k | 0.51 | 17 | 19.82 | 18.85 | 18.42 |
| rollout control, 30k | 0.50–0.52 | 17 | 19.03 | 19.03 | 17.79 |
| teacher-forced + latent, 30k | 0.61 | 20 | 18.22 | 17.86 | 16.20 |
| ODEWorld, 30k | 0.62 | 21 | 17.80 | 17.80 | 15.98 |
| teacher-forced + latent, 60k | 0.59 | 39 | 18.74 | 18.45 | 16.69 |
| ODEWorld, 60k | 0.59 | 39 | 18.64 | 18.64 | 16.43 |
A rollout step is cheaper. The teacher-forced target encodes a window of frames around each sampled time and differentiates the decoupling module along it. Rollout matching encodes one frame per readout. Step matching therefore gives teacher forcing more compute, not less. With 2.3× the GPU hours, teacher forcing and ODEWorld still trail 30k-step SPT-Flow in best-of-10 and in mean PSNR.
The remaining W4 point, random conditioning of the rollout control at inference, is answered under W2 & Q4.
W4 & Q5: "Could the authors clarify why similar downstream evaluations are absent from SPT-Flow?" We added a test of the first link from prediction to control (App. O). The same inverse-dynamics model, trained on demonstration frames, reads actions off each world model's predicted frames. A single SPT-Flow sample gives 16% lower action error than ODEWorld (0.139 vs 0.164; −0.026 [−0.030, −0.021]), also beating the teacher-forced latent's best-of-10 (0.153), and SPT-Flow's best-of-10 plan (0.119) is 9% below the deterministic control (0.131; −0.012 [−0.014, −0.010]). The paper's claims concern prediction; closed-loop success additionally requires a policy trained on each world model and is the next step (Sec. 5).
We hope these experiments resolve your concerns; if so, we would be grateful if you would consider raising your rating. We are glad to clarify anything further during the discussion.
Brief Summary
Dear Reviewer, With the discussion period closing soon, we wanted to briefly check if our new experiments and revisions leave any questions open. In case a quick pointer helps:
- Causal supervision controls & re-anchoring family (W3, Q2–Q3; Tables 4–5, App. A): Giving teacher forcing the rollout arm's posterior, shared ε, and per-trajectory KL still leaves the latent unused (swap gap 0.00 dB), whereas switching that checkpoint to rollout matching for 10k steps activates it in all seeds (0.93–1.26 dB). Sec. 3.3 & 4.5 also adopt your framing of optional re-anchoring as one rollout-matching family.
- Strictly deterministic ε = 0 control (W2, Q4; Table 1, App. J): SPT-Flow's best-of-10 lead increases to +0.79 / +0.63 dB, with a 27% lower energy score (lower on all 250 test trajectories).
- Multimodal, compute & 600-demo unseen-suite evaluations (W1, Q1, W4; Sec. 4.4–4.5, App. K–N): Controlled valid-outcome coverage is 29% vs. 22% / 11% (same-instruction swap +0.24 vs. 0.00 dB); rollout steps run faster (0.51 vs. 0.59–0.62 s); and App. M reports both the +0.49 / +0.58 dB latent transfer gain on unseen suites and the teacher-forced vs. rollout scene-transfer trade-off. Please let us know if anything else would be helpful to clarify before the deadline. Thank you again for your thorough review!
Response to Reviewer CQbc (Part 1/3)
Thank you for the detailed review. Q1–Q4 map directly onto experiments, and we ran each of them. All intervals are paired 95% bootstrap intervals over trajectories, with seeds averaged per trajectory.
W1 & Q1: "whether any existing LIBERO results demonstrate that different latent samples correspond to distinct manipulation strategies under the same initial observation and instruction" We answer with a new analysis of the data and with a controlled experiment that tests strategies directly.
To test strategies where they exist, we built them in (Sec. 4.4, App. L).
W2 & Q4: the latent's contribution, against "a strictly deterministic rollout control evaluated with a fixed zero latent" The two components do different jobs. Rollout matching improves single-sample accuracy; the latent adds what a deterministic model cannot: coverage of the realized future and a better predictive distribution. Under the strictly deterministic control you asked for, its contribution is larger than in the submission. We re-evaluated the control at ε = 0 (Table 1, Fig. 3, App. J). Test set, 250 trajectories, three seeds: