- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces LaMo, a probabilistic latent dynamics world model that forecasts continuous motion residuals. Based on the observation that existing world models predict dense latent states that carries lots of static information which is highly redundant, authors propose modeling the differences between latent motion tokens. LaMo uses a frozen Midway Network to obtain latent states, hierarchical motion tokens and a forward dynamics model, and on top of it trains a transformer with a GMM or flow-matching head that samples the next residual; the residual is added to the previous motion token and the frozen forward model advances the latent state, with context-noise augmentation used against exposure bias. Compared to an architecture-matched baseline that predicts dense latent states, LaMo improves FID, PSNR and SSIM on BDD100K, BridgeData V2 and RoboCasa, and segmentation mIoU on BDD100K, using roughly 3.8× fewer FLOPs per frame.
Critical strengths
-
The intuition that dynamics are lower-dimensional than appearance is compelling, and the residual formulation follows naturally from this idea.
-
Better rollouts at substantially lower cost: the 3.8× reduction of per-frame FLOPs together with better rollouts makes a concrete case that forecasting compact motion tokens is preferable than regenerating the full dense state, since the gain is not bought with extra compute, which is practically relevant for long rollouts and planning where cost grows with every step.
-
Controlled comparison across domains: main components like backbone, context length, decoder and training recipe are kept fixed and only the context signal and prediction target changes, and forecasting in compact latent-motion space instead of dense states gives large long-horizon gains on BDD100K (227 to 167 FID, 8.4 to 17.3 mIoU) and on both manipulation datasets (around 28% lower FID at 1.8s), making it a credible and reusable finding for latent world models
Critical weaknesses
- The experiments do not fully support the claimed contributions, and no experimental result positions the paper against prior work. In Table 1, predicting absolute motion tokens already gives about two thirds of the long-horizon FID gain (227 to 188), and the 3.8× FLOPs saving comes entirely from the motion-token space, since m and Δm have identical shapes. The same ablation is missing from Table 2, there is no state-residual (Δz) baseline, and the only baselines are the authors own variants, with no existing latent-motion or feature-forecasting method evaluated. As result, the residual-specific novelty rests on a single BDD100K ablation and it is not easy to assess whether LaMo is better than published latentt world models.
- The evaluation does not directly test the dynamics claim. Although the central claim is about motion, results rely mostly on image metrics (FID, PSNR, SSIM), which measure dynamics only indirectly, while semantic evaluation is limited to segmentation mIoU on BDD100K only, with no depth, no semantics on the manipulation datasets and no decoder-free latent evaluation. No seeds or std are reported for a stochastic model. Absolute quality is also low (long-horizon FID of 167 with the L2 decoder and still 137 with the CFM one, and mIoU dropping from 30.4 at 0.2s to 17.3 at 1.6s), the visualizations in Figure 2 are blurry, and on BDD100K only LaMo is decoded with the CFM decoder, not the state baseline, so it is hard to see whether the residual advantage holds in a higher-quality regime.
Questions for authors
- Could at least one existing latent-motion method or feature-forecasting method (e.g.DeltaTok) be evaluated under the same protocol, and why the authors do not report results on standard future-prediction benchmarks such as Cityscapes or KITTI, which would allow direct comparison with published numbers?
- Could the target-m variant be added to Table 2 and a state-residual (Δz) baseline on at least one dataset, to separate the effect of residuals from the motion-token space?
- Could latent-space error vs. encoded ground-truth states, an oracle rollout with ground-truth motion tokens, FVD and additional semantic metrics (e.g. depth, or segmentation on the manipulation datasets) be reported, to test more directly the motion claim?
- How are samples scored (single draw, average or best-of-N), and could std over seeds be reported?
As a future work, could the approach be adapted to frozen VFM features, for example by training the inverse and forward dynamics on top of a frozen encoder instead of Midway’s own, to remove the need of per-domain retraining?