acceptodds
Withdrawn from ICLR 2027

Multi-Scale Future Modeling for Latent World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Latent world models efficiently predict future world states in representation space, but most existing approaches model the future only at the timestep level, as a dense sequence of latent states. While this preserves fine-grained dynamics, it provides no explicit representation of future evolution over longer temporal intervals, potentially biasing forecasting toward local dynamics. We present MS-JEPA, a multi-scale latent forecasting framework that models future evolution at complementary temporal scales. The key idea is to represent the same future through both dense latent states and Future Abstractions: the former preserve fine-grained dynamics, while the latter summarize evolution over temporal intervals. A high-level predictor anticipates these abstractions from observed latents to guide dense forecasting. By explicitly modeling temporal structure across scales, our framework combines broader future context with fine-grained dynamics. Experiments across egocentric hand motion, whole-body motion, and physical-scene forecasting show consistent improvements over latent world modeling baselines, demonstrating the effectiveness and generality of our approach.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
World Model, Latent World Model, JEPA

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_HpHyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MS-JEPA supplements dense future-latent prediction with interval-level summaries formed by temporal averaging. A high-level predictor estimates these summaries from observed video and feeds them to the dense predictor through cross-attention. Training first uses oracle summaries, then freezes the low-level pathway while learning summary prediction. Evaluations cover hand motion, body motion, and physical-scene forecasting.

Critical strengths
  1. The construction is simple and the oracle-to-predicted comparison makes the role of guidance easy to inspect. On EgoDex, predicted summaries improve ADE/FDE/accuracy from 0.049/0.067/0.656 to 0.044/0.061/0.706.

  2. Improvements extend across four datasets, several forecast horizons, and both V-JEPA2 and DINOv2 features. The appendix reports full-pipeline latency as well as accuracy, giving some context for the added computation.

Critical weaknesses
  1. The central claim is not sufficiently isolated by the experiments. MS-JEPA adds a 12-layer predictor, cross-attention adapters, and staged supervision, but lacks a dense baseline matched for capacity and training compute. The sparse-state comparison is helpful, though its small advantage lacks uncertainty estimates. Attribution of the gains to interval-level modeling is the main reason for weak rejection.

  2. The paper defines the abstraction loss but does not fully state the low-level and downstream loss combination, gradient paths, or Stage II duration. Results are also reported without seed variation. These omissions make the gains difficult to reproduce and assess, particularly where improvements are modest.

  3. The dense predictor is trained with oracle abstractions and then frozen when predicted guidance is introduced. The oracle-versus-predicted gap does not distinguish abstraction-prediction error from sensitivity to this change in conditioning. A fine-tuning or guidance-noise control is needed to support the claim that improving the high-level predictor alone is the main remaining opportunity.

Questions for authors
  1. Do the gains remain against a dense baseline matched for trainable parameters and training compute, or a dense model with an auxiliary interval-prediction loss but no guidance branch?

  2. What are the exact losses, updated modules, and training durations in both stages, including the task head and any dense-latent loss? Please also specify the dataset splits used for the reported comparisons.

  3. Across independent seeds, how reliable are the gains over dense prediction and sparse future states, especially the small MOVi-C and EgoExo4D differences?

Reviewer_cwwfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MS-JEPA predicts interval averages of future latents alongside a dense future sequence. A high-level Transformer supplies these summaries to the dense predictor through cross-attention. Training first uses ground-truth summaries, then freezes the dense pathway while learning to predict the summaries from past observations. Experiments cover hand motion, whole-body motion, and physical-scene forecasting.

Critical strengths
  • Consistent forecasting gains On EgoDex, predicted summaries improve ADE/FDE/Acc from 0.049/0.067/0.656 to 0.044/0.061/0.706 (Table 2, PDF p.6). Results also improve on EgoExo4D, MOVi-C/E, and with DINOv2 features.

  • Useful ablations Table 6 compares means and endpoint states while keeping the remaining architecture fixed (p.8). The appendix also compares staged and joint training and reports full-pipeline latency of 114.06 versus 101.26 ms (Tables 9–10).

Critical weaknesses
  • The added hierarchy The comparison with the dense predictor adds a 12-layer, 512-dimensional Transformer, cross-attention, and two-stage training together (pp.6–7). I cannot tell how much of the gain needs the proposed hierarchy. Table 6 shows that means work better than endpoints within that hierarchy, but does not compare against a stronger dense predictor or a simple pooled-future auxiliary loss. The joint-training and timing ablations do not answer that question.

  • The temporal-scale evidence is still limited The target ablations are mainly on EgoDex: means reduce ADE from 0.046 to 0.044, and four rather than one interval raises Acc from 0.696 to 0.706 (Tables 6–7). There are no repeated-run uncertainty estimates. Some cross-task gains are small too: MOVi-C FG-ARI is 73.60 versus 73.30, and EgoExo4D MPJPE is 16.00 versus 16.49. I would like to know whether these differences are stable. The oracle results show that future information helps, but do not settle how well the learned summaries explain the gains.

  • Limited experimental coverage Although the paper reports four benchmarks, their coverage is fairly concentrated. EgoDex and EgoExo4D both use egocentric video to forecast human motion, while MOVi-C and MOVi-E come from the same synthetic scene family. These are useful tests, but provide limited evidence for applicability beyond those settings. And better select a few mainstream task and draw a radar chart.

Questions for authors
  1. How does MS-JEPA compare with a stronger dense predictor and a pooled-future auxiliary loss without the high-level pathway, under matched capacity and total training updates?

  2. Does Table 3 use one checkpoint or separately trained models for each horizon, and what are the Stage II duration and total updates for each comparison?

  3. Are the mean-versus-endpoint and interval-count differences stable across runs, including in one non-EgoDex setting?

Reviewer_2oi1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MS-JEPA, a multi-scale latent forecasting framework. The future is represented at two levels: dense timestep-level states and interval-level Future Abstractions. The latent predictor first predicts the Future Abstractions from the past states, and then predicts the future states from both the past states and these Future Abstractions. Experiments and ablations show improvements over latent forecasting baselines, as well as robustness across representation spaces.

Critical strengths
  1. The motivation and the method are both concise.
  2. The gains are consistent across three task types (hand motion, whole-body motion, physical scenes), two encoders, and multiple prediction horizons.
Critical weaknesses
  1. The overall framework does not resemble a world model or a JEPA.
    • (a) The latent dynamics form an autonomous system that does not involve actions.
    • (b) The predictor appears to predict all future states at once from the past states, rather than predicting the next state from past information and actions; the former cannot be used for planning.
  2. The theoretical and empirical support for the method is insufficient. There is no theoretical explanation of why Future Abstractions work, and the ablation results are not convincing. At least the following additional experiments are needed:
    • (a) simply removing ℒ_abs;
    • (b) removing 𝒢_high, predicting F̂^future directly from F^past, and then computing ℒ_abs from F̂^future;
    • (c) randomizing the temporal order of the Â_i of different chunks.
Questions for authors
  1. The core tasks of a world model include not only prediction but also planning and counterfactual prediction. Can your method handle these tasks?
  2. The Future Abstractions in your model are computed by simple averaging. Would a learnable attention mechanism be better? That said, your cross-attention may already serve this role to some extent.
  3. The 𝒢_low + 𝒢_high architecture has more parameters than 𝒢_low alone. Can you demonstrate that the improvement in your experiments is not due to the difference in the number of parameters?
  4. In general, a world model is learned by training the encoder and the predictor jointly. Since only the V-JEPA 2 encoder is retained here, you should also report the performance of the predictor that was pretrained jointly with it.
Reviewer_qUBbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MS-JEPA. Idea is to split the future horizon into N chunks, average the V-JEPA2 latents in each chunk (they call them “Future Abstractions”), train a second predictor to predict these from the past frames, and then let the normal dense predictor cross-attend to them. Training is done in two stages. They show results on EgoDex, EgoExo4D and MOVi-C/E, with improvement over V-JEPA predictor and ThinkJEPA.

Critical strengths

Idea is simple and easy to plug into an existing JEPA predictor, and the improvement is consistent on all the datasets and also when using DINOv2 instead of V-JEPA2. The oracle vs predicted analysis is also nice to have.

Critical weaknesses

Main issue for me is model capacity. The high level predictor is 12 layers with dim 512, so it’s actually bigger than the dense predictor, but there is no dense baseline with similar size. So I can’t really tell if the gain comes from the multi scale idea or just from more parameters. The ablations also don’t support the multi scale story that much. With N=1 (just one vector for whole future) the acc is already 0.696, vs 0.706 for N=4 and 0.656 for dense. Sparse states give similar numbers too. So most of the gain is there even without any temporal structure.

Other thing, gains on EgoExo4D and MOVi-C are quite small (16.50 -> 16.00, 72.93 -> 73.60) and there’s no std or multiple seeds, so hard to say if it’s significant.

Questions for authors

Can you compare with a dense predictor of the same size or FLOPs? Did you try just averaging the dense predictor’s own outputs and feeding that back in, without a separate high level predictor? The dense predictor only sees GT abstractions in training and then it’s frozen, so it never sees predicted ones. Did you try fine tuning it on the predicted abstractions? Please give std over a few seeds, at least for EgoExo4D and MOVi.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.