acceptodds
Under review as a conference paper at ICLR 2027

LaMo: A Latent Motion World Model for Long-Horizon Prediction

Abstract

Visual world models learned from video are often trained to predict future observations from past frames, but pixel-space forecasting is expensive and forces the model to allocate capacity to high-frequency details. Latent-space video models reduce this cost, yet they typically predict a dense latent state at every step, repeatedly carrying largely static information and making long-horizon rollouts harder to sustain. We propose , a probabilistic latent dynamics world model that predicts motion residuals—continuous latent variables encoding differences between consecutive motion tokens in a learned latent dynamics space. Given the current latent state and a short history of past motion tokens, our model autoregressively samples the next motion residual, integrates it with the previous motion token to obtain the next motion token, and recurrently advances the latent state using a frozen forward dynamics model. We instantiate the same contextual transformer backbone with two probabilistic parameterizations for residual prediction: Gaussian mixtures and conditional flow matching. We evaluate long-horizon rollouts on BDD100K using image-space metrics and semantic segmentation mIoU, showing improved long-horizon modeling in complex driving scenes.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
World Models, Video Prediction, Latent Dynamics, Latent Action Models, Long-Horizon Prediction, Flow Matching, Generative Modeling, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_s5uaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LaMo, a probabilistic latent dynamics world model that forecasts continuous motion residuals. Based on the observation that existing world models predict dense latent states that carries lots of static information which is highly redundant, authors propose modeling the differences between latent motion tokens. LaMo uses a frozen Midway Network to obtain latent states, hierarchical motion tokens and a forward dynamics model, and on top of it trains a transformer with a GMM or flow-matching head that samples the next residual; the residual is added to the previous motion token and the frozen forward model advances the latent state, with context-noise augmentation used against exposure bias. Compared to an architecture-matched baseline that predicts dense latent states, LaMo improves FID, PSNR and SSIM on BDD100K, BridgeData V2 and RoboCasa, and segmentation mIoU on BDD100K, using roughly 3.8× fewer FLOPs per frame.

Critical strengths
  • The intuition that dynamics are lower-dimensional than appearance is compelling, and the residual formulation follows naturally from this idea.

  • Better rollouts at substantially lower cost: the 3.8× reduction of per-frame FLOPs together with better rollouts makes a concrete case that forecasting compact motion tokens is preferable than regenerating the full dense state, since the gain is not bought with extra compute, which is practically relevant for long rollouts and planning where cost grows with every step.

  • Controlled comparison across domains: main components like backbone, context length, decoder and training recipe are kept fixed and only the context signal and prediction target changes, and forecasting in compact latent-motion space instead of dense states gives large long-horizon gains on BDD100K (227 to 167 FID, 8.4 to 17.3 mIoU) and on both manipulation datasets (around 28% lower FID at 1.8s), making it a credible and reusable finding for latent world models

Critical weaknesses
  • The experiments do not fully support the claimed contributions, and no experimental result positions the paper against prior work. In Table 1, predicting absolute motion tokens already gives about two thirds of the long-horizon FID gain (227 to 188), and the 3.8× FLOPs saving comes entirely from the motion-token space, since m and Δm have identical shapes. The same ablation is missing from Table 2, there is no state-residual (Δz) baseline, and the only baselines are the authors own variants, with no existing latent-motion or feature-forecasting method evaluated. As result, the residual-specific novelty rests on a single BDD100K ablation and it is not easy to assess whether LaMo is better than published latentt world models.
  • The evaluation does not directly test the dynamics claim. Although the central claim is about motion, results rely mostly on image metrics (FID, PSNR, SSIM), which measure dynamics only indirectly, while semantic evaluation is limited to segmentation mIoU on BDD100K only, with no depth, no semantics on the manipulation datasets and no decoder-free latent evaluation. No seeds or std are reported for a stochastic model. Absolute quality is also low (long-horizon FID of 167 with the L2 decoder and still 137 with the CFM one, and mIoU dropping from 30.4 at 0.2s to 17.3 at 1.6s), the visualizations in Figure 2 are blurry, and on BDD100K only LaMo is decoded with the CFM decoder, not the state baseline, so it is hard to see whether the residual advantage holds in a higher-quality regime.
Questions for authors
  • Could at least one existing latent-motion method or feature-forecasting method (e.g.DeltaTok) be evaluated under the same protocol, and why the authors do not report results on standard future-prediction benchmarks such as Cityscapes or KITTI, which would allow direct comparison with published numbers?
  • Could the target-m variant be added to Table 2 and a state-residual (Δz) baseline on at least one dataset, to separate the effect of residuals from the motion-token space?
  • Could latent-space error vs. encoded ground-truth states, an oracle rollout with ground-truth motion tokens, FVD and additional semantic metrics (e.g. depth, or segmentation on the manipulation datasets) be reported, to test more directly the motion claim?
  • How are samples scored (single draw, average or best-of-N), and could std over seeds be reported?

As a future work, could the approach be adapted to frozen VFM features, for example by training the inverse and forward dynamics on top of a frozen encoder instead of Midway’s own, to remove the need of per-domain retraining?

Reviewer_A1ugOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LaMo, a latent motion world model for future prediction. Instead of directly predicting the next dense state, it predicts a residual in the motion space learned by Midway. The predicted residual updates the previous motion. A frozen forward dynamics network then produces the next state. The paper evaluates the method on BDD100K, BridgeData V2, and RoboCasa. It reports better predictions than a matched state-prediction baseline and lower FLOPs per predicted frame

Critical strengths
  1. Predicting compact motion residuals instead of full scene states is an interesting idea. It offers a different way to model future changes without repeatedly generating the full state representation.
  2. The paper includes useful controlled ablations. It compares state prediction, direct motion prediction, and motion residual prediction. It also studies noise augmentation of the history. These experiments provide evidence for the main design choices.
Critical weaknesses
  1. Comparisons with existing methods are missing. The experiments mainly compare variants within the proposed framework. Related methods such as DeltaTok, DINO-Foresight, and DINO-WM are discussed but not evaluated. I understand that the paper focuses on a controlled study. Still, comparisons with the closest applicable methods under matched conditions are needed to show where LaMo stands among existing approaches.
  2. Generalization beyond the training domain is not tested. Midway and LaMo are trained separately for each data domain. Thus, the results across three datasets do not demonstrate cross-domain generalization. It remains unclear whether the model learns transferable dynamics or relies on motion patterns specific to each training distribution. Tests on unseen scenes, objects, or motion combinations would help. The authors should also discuss the limits of the method's generalization and their implications for practical use.
  3. The evaluated prediction horizons are short for the main claim. The experiments extend to 1.6 seconds on BDD100K and 1.8 seconds on the robot datasets. Since the paper focuses on long-horizon prediction, longer rollouts are needed. In particular, it is unclear whether the gains over direct state and motion prediction persist as errors accumulate.
  4. The scope of the FLOPs comparison is unclear. The paper reports 199 GFLOPs for LaMo and 749 GFLOPs for the state-prediction baseline, excluding the shared RGB decoder. Excluding a shared decoder is reasonable. However, LaMo must first predict a motion residual, recover the motion, and then run the Midway forward dynamics network to obtain the next state. It is unclear whether the reported cost includes this additional forward dynamics network. The comparison should cover all non-shared operations needed to produce the next state. A breakdown of these costs would help verify the claimed efficiency gain.
Questions for authors

Please address the concerns raised in Critical Weaknesses. I also have two additional questions:

  1. How much does the benefit of motion residual prediction depend on the Midway representation? Is there evidence that the same design works with another motion representation? If not, please discuss this limitation.
  2. Could you provide the actual training data sizes and train/validation/test splits for each dataset? Please also clarify the Midway pretraining setup and report parameter counts for LaMo, Midway, and the full system. These details would help readers reproduce the results and assess the required resources.
Reviewer_zFJwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

LaMo forecasts in the motion-token space of a frozen Midway model. It predicts a change from the previous motion token, adds it back, and lets the frozen forward model advance the scene. The forecasting head models possible future changes rather than predicting a full latent scene from scratch.

Critical strengths

Carrying persistent scene information forward instead of repeatedly predicting it is a sensible choice. The direct motion-token comparison also supports the residual target itself, not just the compact bottleneck. That's worth separating from the broader world-model story.

Critical weaknesses
  1. I buy the local result more than the general story. Given the previous motion code, predicting the next code and predicting its difference carry the same information. What changes is the bias: a zero residual means reuse the previous motion code, while the frozen forward model keeps advancing the scene. That's a reasonable persistence prior. But the paper doesn't make clear what we've learned beyond the usefulness of that prior in this particular pre-trained modle
  2. The choice of variable is the deeper question. A token inferred between two frames isn't automatically a velocity with meaningful time differences. Whether its changes are simple depends on how the pretrained model arranged its motion space. Subtraction isn't invalid here, and the code doesn't need to be a physical quantity. But calling it motion doesn't explain why additive updates should make the dynamics easier to learn.
  3. I'd find the paper much more compelling if it identified the property that makes a motion representation good for forecasting, rather than showing that residual prediction helps after a particular representation has already been learned. As it stands, the takeaway feels like a useful modeling tweak, not a new prediction principle or a result that settles a substantial forecasting problem.
Questions for authors
  1. What property of Midway's motion space makes changes between consecutive codes a good prediction target? Does pretraining encourage that property?
  2. What does the forecaster need to learn beyond motion persistence and the frozen transition model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.