acceptodds
Under review as a conference paper at ICLR 2027

Matryoshka World Models: Test-Time Scaling Beyond Search

Abstract

Planning with learned world models requires trading off predictive fidelity against computational cost. Existing approaches typically vary search effort while keeping the latent representation fixed, or train separate models for different representation sizes. We introduce Matryoshka World Models (MWM), a framework for training world models to operate at multiple nested predictive fidelities. Unlike independently trained fixed-dimensional models, MWM learns a shared hierarchy of predictive representations that can be truncated or expanded at test time without retraining. This variable-fidelity world model enables a new form of planning-time flexibility: predictive fidelity can change across episode-level replanning steps, CEM optimization iterations, and rollout depth. The planner can therefore allocate high-fidelity computation selectively while using cheaper representations where fine-grained prediction is unnecessary. We evaluate variable-fidelity planning against standard fixed-fidelity planning baselines across navigation and manipulation tasks. MWM matches the success rate of the full-fidelity baseline at up to 108.8x less compute and exceeds the best baseline by up to 59% at the same compute budget. By making predictive fidelity a resource that can be dynamically allocated across different stages of planning, MWM establishes a new axis of test-time scaling for world models. Project website: https://study-9f2c7a.pages.dev.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, model-based planning, Matryoshka representation learning, test-time scaling, latent dynamics, efficient inference, model predictive control, visual planning, multi-resolution representations, robotics

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_psj3OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper proposes building a hierarchical world model adapted from the Matryoshka representation learning (MRL) approach of Kusupari et al., to be used in planning tasks where planning is done using trajectory optimization with CEM in an MPC setting. In a nutshell, a single encoder of the observations is learned, but several levels of prediction are obtained by associating various models of the dynamics to increasing sizes of the prefix of the codes used for prediction, allowing the allocation of different levels of resources to CEM iterations and rollout steps. At a very high level, this appears to be a reasonable approach and to give good results.

But many crucial parts of the proposed approach (planning algorithm, evaluation protocol, schedule construction, ...) are supposed to be found in an appendix, which is not part of the submitted pdf. Without this appendix, it is impossible to fully evaluate the paper. Thus is cannot be accepted in its current form.

Critical strengths

The representation learning part of the paper is a direct adaptation of MRL to a hierarchical JEPA-style approach to planning, but I have not seen this applied to this domain before. The idea of allocating different resources to different parts of the planning task is reasonable as well. At a high level, the proposed approach thus appears to be reasonable, and to give good results.

Critical weaknesses

Although the basic ideas of the paper are simple, the details are quite hard to follow, with many things never defined (e.g., the key concept of fidelity, or the loss and regularizer used during learning), even excluding those that are supposed to be in appendix.

Most papers can be read without referring to their appendix, but not this one. For example, in Section 3.2, the scheduling scheme proposed by the authors is supposed to increase fidelity across CEM iterations and decrease it along rollouts, which is a key idea of the paper. Putting aside the fact that fidelity is never defined, the scheduling scheme itself is not explained, and relegated to the appendix, making it impossible to judge its merits.

The same can be said with other key parts of the approach (planning algorithm, experimental protocol, etc.).

Questions for authors

No questions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer psj3

Comment: “Many crucial parts … are supposed to be found in an appendix, which is not part of the submitted PDF. Without this appendix, it is impossible to fully evaluate the paper.”

Response: Thank you for your feedback. The appendix is included in the submitted supplementary material, rather than appended to the main PDF, as ICLR requires. It contains the experimental protocol in Appendix A.1, schedule construction and a worked example in Appendix A.2, and the planning procedure in Algorithm 1. We respectfully ask that these materials be considered when reassessing the submission, since their perceived absence appears central to the recommendation.

We recognize that the nested scheduling mechanism is technically dense and aim to produce a concise and readable summary within the main text. We were gratified to see that reviewers XgLw and yXv7 rated the presentation “clear and easy to follow”. But we understand that the current explanation could still be improved. We appreciate your suggestions in this regard and will take them seriously when we revise. In particular, we will use the additional page available in the camera-ready version to bring more explanatory content from the supplement into the main paper and make the presentation more self-contained.

THank you for your response. that addresses some of my main concerns.

Reviewer_yXv7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work applies the concept of Matryoshka Representation Learning (MRL) to world models. In MRL, multiple levels of representation fidelity are encoded in a single vector, with each lower-dimension representation serving as a prefix of every larger representation. In Matryoshka World Models (MWM), these nested representations must also serve as action-conditioned predictive states, to support planning. The authors present a variable-fidelity world model for visual goal-reaching, based on LeWorldModel, that uses a shared encoder but jointly trained yet independently parameterized dynamics modules for each supported latent size. This framework enables prediction fidelity to vary during planning across several axes (MPC replans, CEM optimization iterations, and [fixed or decrease only] prediction steps in imagined rollouts). Using pre-defined heuristic schedules, they evaluate their framework on several navigation and manipulation benchmark tasks and compare it with independently trained “single-d” world models and with “fixed-d” variants of the jointly trained MWM. They show that variable fidelity improves the planning performance–compute Pareto frontier and that jointly trained fixed levels can outperform independently trained world models at the corresponding latent dimensionalities.

Critical strengths
  • Novelty and potential impact: The paper’s main novelty is the transfer of Matryoshka representations into action-conditioned world models so that predictive fidelity can be varied during planning. While its individual ingredients are largely established, their combination is conceptually clean and enables fidelity to vary across several stages of planning. The experiments report strong performance-compute trade-offs and gains over single-d models at matched compute budgets.
  • Reproducibility: code, data, evaluation episodes, checkpoints, seeds, etc. are publicly available, and implementation and planning schedules are described in substantial detail.
Critical weaknesses
  • Limited statistical analysis: Each checkpoint is evaluated on one fixed set of held-out episodes using a single evaluation seed, and no uncertainty estimates are reported. Repeated evaluation sets or bootstrap confidence intervals would make it easier to judge the robustness of small differences and Pareto-frontier membership. The paper also provides little error analysis explaining when particular schedules succeed or fail. Also, Pareto plots emphasize the largest success-rate difference between fixed and scheduled MWM, but this point is not generally located in the most densely evaluated region of the frontier and thus it’s somewhat unclear how consistently scheduling helps across the full configuration sweep.
  • Limited baselines: The experiments primarily compare MWM against its own internally matched controls. They do not empirically compare against any of the adaptive-computation or multi-resolution planning methods discussed in the related work. Such comparisons would help establish whether the gains come specifically from nested predictive fidelity rather than adaptive compute allocation more generally.
Questions for authors
  • How consistently does scheduling help across the full configuration sweep?
1 comment
AuthorsOpenReview ↗

Response to Reviewer yXv7

Comment 1: “Limited statistical analysis.… Repeated evaluation sets or bootstrap confidence intervals would make it easier to judge the robustness of small differences and Pareto-frontier membership.”

Response: Thank you for this suggestion. As detailed in our global response on independent validation, we evaluated frozen configurations on two fresh sets of 100 paired episodes per task–offset setting and computed paired-bootstrap 95% confidence intervals. The average scheduled-versus-fixed advantage is positive at all three evaluated budgets, with each confidence interval excluding zero. This provides independent evidence for the scheduling gains beyond the original point estimates. We will include these uncertainty estimates and distinguish validation of selected operating points from the stability of the entire empirical frontier.

Comment 2: “The paper also provides little error analysis explaining when particular schedules succeed or fail.… It’s somewhat unclear how consistently scheduling helps across the full configuration sweep.”

Response: Thank you for this question. To examine outcomes beyond the frontier winners, we analyzed all 3,382 matched schedule–baseline comparisons across the eight task–offset settings, including configurations that never appear on the frontier. Each scheduled run is compared with the same MWM checkpoint held at full dimension, using identical CEM settings and evaluation episodes.

Across this sweep, scheduling matches or exceeds full-dimensional observed success in 51% of comparisons, while scheduled runs use approximately half the dynamics computation overall. CEM-only scheduling improves success in 63% of comparisons, with a median gain of 4 percentage points. Benefits are widespread on OGBench-Cube at both offsets, whereas rollout coarsening hurts TwoRoom and episode-level scheduling often hurts PushT and Reacher at Δ=25\Delta=25. These patterns show that the planning stage and task matter; scheduling does not uniformly help every configuration.

This sweep analysis uses full-dimensional MWM as its reference. Separately, Figure 6 compares scheduling against the best evaluated fixed-level MWM configuration within each compute budget. Our global response on independent validation and statistical robustness further tests frozen selections from that comparison on fresh episodes. Together, these analyses characterize the breadth of scheduling outcomes and their additional benefit over choosing a suitable fixed level. We will include the broader sweep analysis and highlight the observed failure regimes, while distinguishing these aggregate patterns from episode-level causal explanations.

Comment 3: “The experiments primarily compare MWM against its own internally matched controls.… Such comparisons would help establish whether the gains come specifically from nested predictive fidelity rather than adaptive compute allocation more generally.”

Response: Thank you for this suggestion. We cite these works to situate MWM within the broader study of adaptive computation and multi-resolution planning. Some address related planning problems but require different representations and model architectures. For example, Wang et al. vary particle resolution in graph-based dynamics for object-pile manipulation, while M3W uses mixture-of-experts world models for multi-task, multi-agent planning. Adapting these approaches to our visual-goal latent-planning setting would require substantial model development and training, rather than replacing a scheduling rule.

Our experiments isolate the benefits of the proposed construction through matched comparisons: Single-d versus fixed MWM assesses joint training, and fixed versus scheduled MWM assesses fidelity allocation within the learned hierarchy. We do not claim that MWM is the only route to multi-fidelity learned world models. Rather, our results demonstrate the value of this capability and motivate investigating other constructions and allocation mechanisms in future work. We will clarify the applicability of the cited approaches and this broader perspective in the final version.

Reviewer_rKhXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Matryoshka World Models, a concept based on the combination of architecture from LeWorldModel and nested learning technique introduced in Matryoshka Representational Learning. The headline takeaway is matching success at 108.8x less compute or exceeding baseline by 59% at same compute. The evaluation is entangling whether this is a result of studying different levels of representations over presented tasks or a contribution of the method of learning. Either way, this paper's contribution seems an interesting and useful one.

The base literature is adequately referenced and the approach setup is reasonable. The experiments are well-crafted. They range from additive support to the general idea of joint-training working for smaller bottlenecks to more interesting study of how scheduling at different levels of episode step, CEM and MCP loops impacts performance. However, I also found this to be the hardest (and this was one of the sections that needs to be clear) to read and compare against chosen schedules which are not just fine-to-coarse or coarse-to-fine everytime.

Critical strengths
  • Studies impact of varying levels of world-model architecture prowess and compute to performance. Has interesting takeaways
  • Setup, learning as well as planning, is reasonably based on SotA arch and oft-used paradigms lending more credence to generalizability and usefulness of study
  • The future work of adaptive planning is interesting based on these takeaways, however tempered a bit against trade-off versus a fixed but slightly sub-par model
Critical weaknesses
  • Paper can be better written. The nested hierarchy adds another level of recursive complexity to algorithms and the stand-alone paper does not do a good job of explaining these. The supplementary materials add what is missing, but I do think the paper by itself can better explain schedules, how they approach going from fine-to-coarse and coarse-to-fine etc which are central methodologies.
  • As stated above, the reason for paper's contribution is not clear. Is the takeaway a result of their training technique or is this a study finding that a constant level single-d repr does ~80-85% as well as the nested paradigm thus a mid-level repr is generally appropriate for studied tasks.
Questions for authors
  • In appendix A.4 you state that improvement over constant best-level is smaller in certain tasks. I feel this is the entanglement which confuses me. Is there a fixed best-level one can find over a class of tasks thus gaining 80-90% of the performance of your best model? In this case, the contribution still remains however it is less about adaptive scheduling rather than using your framework as a way to find this lever. This would require you to also reframe your own positioning for this paper.
1 comment
AuthorsOpenReview ↗

Response to Reviewer rKhX

Comment 1: “The nested hierarchy adds another level of recursive complexity.… The paper by itself can better explain schedules, how they approach going from fine-to-coarse and coarse-to-fine.”

Response: Thank you for this suggestion. We will bring the worked example and essential schedule-construction details from Appendix A.2 into the main text, making the scheduling mechanism easier to follow without consulting the supplement.

Comment 2: “In Appendix A.4 you state that improvement over constant best-level is smaller in certain tasks.… Is there a fixed best-level one can find over a class of tasks thus gaining 80–90% of the performance of your best model?”

Response: Thank you for this observation. Our results demonstrate benefits from both joint training and scheduling. A well-chosen fixed level is a strong option in some settings, but Figure 6 already accounts for this by comparing scheduled MWM against the best evaluated fixed-level MWM configuration within each compute budget.

Scheduling further improves this frontier. As detailed in our global response on independent validation and statistical robustness, evaluating frozen configurations on fresh paired episodes yields average success improvements over fixed MWM of 3.0, 6.9, and 6.4 percentage points at low, intermediate, and high budgets, respectively, with all three 95% confidence intervals excluding zero.

A well-chosen fixed MWM level is nevertheless a strong option, and this is itself a benefit of MWM’s jointly trained hierarchy. Our contribution includes both effective smaller predictors and the flexibility to achieve further gains through scheduling; it does not require scheduling to be substantially better in every setting. The comparisons allow fixed levels to differ across tasks and budgets, so they do not establish whether one fixed dimension consistently captures 80–90% of the benefit across an entire class of tasks. We will make these distinct contributions clearer in the final version.

Reviewer_XgLwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MWM trains nested latent representations with a shared encoder and separate dynamics predictors. Predefined schedules vary prediction dimension across MPC replans, CEM iterations, and rollout depth. On four simulated tasks at two goal offsets, joint training and scheduling improve the observed trade-off between success and dynamics FLOPs.

Critical strengths
  1. The method extends nested representations to action-conditioned prediction. Shared coordinates let the planner truncate a predicted state and continue the same rollout with a smaller dynamics module.
Critical weaknesses
  1. The main scheduling gains need independent validation. Appendix A.1 uses one training seed and 100 episodes under one evaluation seed per configuration. The frontiers and d_min select settings from evaluation success. Appendix A.4 also keeps the best outcomes of equivalent schedules. Selecting maxima can inflate gains, especially with more scheduled configurations. No separate final test of frozen choices is reported; Appendix A.5 gives confidence intervals only for baseline reproduction. The reliability of the MWM gains therefore remains unclear.
  2. The benefit over other ways of varying prediction cost is not established. Looped World Models varies recurrent prediction depth at inference [1]. Sivasankar studies early-exit latent predictors and fixed-depth versus routed CEM planning [2]. Older work allocates model complexity along MPC trajectories [3] or selects fast, sufficiently accurate models during planning [4]. These uncited works limit the broad claim of a new test-time scaling axis. MWM still differs through shared predictive prefixes, separate dimension-specific dynamics, and three-scale scheduling. A variable-depth comparator would help establish the value of this specific design. Practical savings also need end-to-end timing. Table 6 shows 22.9 times fewer dynamics FLOPs at dimension 48 than 192, but batch-one latency changes only from 0.859 to 0.815 ms; actual CEM batches may behave differently. The main metric excludes encoding and planner overhead. The 108.8-fold result compares with full-dimensional Single-d, combining training and scheduling gains. Matched-success timing against the best fixed MWM level and a variable-depth predictor is needed.

References:
[1] Looped World Models. June 2026 preprint. https://arxiv.org/abs/2606.18208v1
[2] When Does Depth Survive Composition? Compute–Quality Regimes in Latent World Models. July 2026 preprint, v1. https://arxiv.org/abs/2607.10203v1
[3] Adaptive Complexity Model Predictive Control. IEEE T-RO 2024. https://arxiv.org/abs/2209.02849
[4] Learning Model Preconditions for Planning with Multiple Models. CoRL 2021. https://proceedings.mlr.press/v164/lagrassa22a.html

Questions for authors
  1. Were d_min and the frontier configurations selected on episodes separate from final testing? After fixing these choices, do gains over Single-d and the best fixed MWM level persist on new goal pairs and multiple training/evaluation seeds, with confidence intervals?
  2. With comparable encoders, data, and planners, how does MWM compare with an early-exit or variable-depth dynamics predictor? At matched success, please report end-to-end planning time for both methods and the best fixed MWM level, including encoding, CEM updates, and switching overhead.
  3. Changing rollout dimension also changes the goal features used in the terminal cost. If the predictor stays full-dimensional but the cost uses the same scheduled goal prefixes, how much of the performance change remains? This would separate cheaper prediction from a different goal metric.
5 comments
AuthorsOpenReview ↗

Response to Reviewer XgLw

Comment 1: “The main scheduling gains need independent validation.… Selecting maxima can inflate gains, especially with more scheduled configurations.… No separate final test of frozen choices is reported.”

Response: Thank you for this suggestion. We clarify that d_min was selected on episodes separate from those used to construct the reported frontiers. Our original analysis characterized the available success–compute trade-offs rather than prescribing universally optimal schedules. As detailed in our global response on independent validation, we have now selected configurations using only the original results, frozen all settings, and evaluated them on fresh paired episodes without reselection. The results support the main scheduling trends. This evaluation tests variability across new episodes and planning randomness while holding training checkpoints fixed.

Comment 2: “These uncited works limit the broad claim of a new test-time scaling axis.… A variable-depth comparator would help establish the value of this specific design.”

Response: Thank you for pointing out these relevant works. They share our interest in efficient prediction and planning, but their mechanisms differ from MWM’s central contribution:

  • Looped World Models (arXiv, June 2026): Thank you for pointing out this concurrent industry lab tech report. It introduces a recurrent-depth world-model backbone. Our Matryoshka construction could be applied to this backbone too: loop depth controls computational refinement, while MWM controls predictive-state dimensionality, making the two mechanisms complementary. This tech report evaluates exclusively on interactive text-based environment simulation datasets. As such, implementing both approaches on a common platform for proper evaluation of their individual merits and complementarities will require significant follow-up work. 1
  • When Does Depth Survive Composition? (arXiv, July 2026): while it is motivated by adaptive computation in world models, it neither implements such a scheme itself, nor cites prior work that does. It instead studies whether deeper predictors improve multi-step rollouts. MWM learns nested predictive representations that allow the state dimensionality to vary within a predictive rollout, during replanning over an episode, or during optimization iterations. 2
  • Adaptive Complexity MPC applies to classical manually designed reduced-order models (ROM) where explicit consistency and feasibility conditions hold. We will briefly cite this work and other related work from the ROM literature (it was omitted in the manuscript for space). MWM instead deals with learning nested multi-fidelity world models. 3
  • Learning Model Preconditions learns when to use each available model, whereas MWM learns the hierarchy of predictive models itself; its selection principle could potentially guide which MWM level to use. 4

We therefore view these approaches as complementary to our contribution. MWM introduces jointly trained predictive prefixes that allow a state predicted at a larger dimension to be truncated and passed directly to a smaller dynamics module, enabling dimensionality scheduling across MPC replans, CEM iterations, and rollout steps. We will cite these works and clarify our contributions.

AuthorsOpenReview ↗

Response to Reviewer XgLw

Comment 3: “Practical savings also need end-to-end timing.… The main metric excludes encoding and planner overhead. The 108.8-fold result compares with full-dimensional Single-d, combining training and scheduling gains. Matched-success timing against the best fixed MWM level and a variable-depth predictor is needed.”

Response: Thank you for this suggestion. We distinguish computational savings from their realization as wall-clock speedups. Batch-one latency depends on hardware utilization and implementation overhead, and does not alone characterize the benefit of reduced computation under batched planning workloads. Our aim is to establish improved success–compute trade-offs through nested predictive representations; fully realizing these savings in a control stack may require hardware-specific optimizations.

We nevertheless observe practical timing gains. Comparing the minimum measured planning time required to reach the same success target, with fixed MWM free to choose among all evaluated dimensions and planner settings, scheduled MWM achieves geometric-mean speedups of 1.27×, 1.38×, and 1.34× at 50%, 70%, and 90% of the lower of the methods’ peak success rates across the eight published settings. These are empirical comparisons from the original sweep; encoding and other non-planning operations separately account for a median 1.9% of policy time in the held-out runs.

The 108.8× result measures combined dynamics-FLOP savings over full-dimensional Single-d, whereas the timing comparison measures gains relative to fixed MWM. We will distinguish these metrics and baselines explicitly in the final version.

Comment 4: “Changing rollout dimension also changes the goal features used in the terminal cost.… If the predictor stays full-dimensional but the cost uses the same scheduled goal prefixes, how much of the performance change remains?”

Response: Thank you for suggesting this analysis. This would not undermine our claims about the overall benefits of our approach, but rather would help identify where the gains come from. Based on experiments with an earlier iteration of the method, we believe that changing the scoring function alone would not recover the gains. However, our current implementation does not readily support this comparison within the rebuttal period. We will conduct this analysis afterward. Regardless of its outcome, retaining full-dimensional prediction would retain the full predictor’s computational cost, whereas MWM achieves its results using smaller predictors at selected stages of planning.

Thank you for the additional analyses. The independent evaluation substantially addresses my concern about configuration selection. I have three requests for clarification and revision:

  1. please correct the description of Sivasankar’s work, since Section 9 of the July v1 includes depth-routed CEM planning;
  2. acknowledge that the current results do not separate the contributions of changing prediction dimensionality and changing the goal-scoring features;
  3. clarify whether planning time includes CEM updates and model-switching overhead, while identifying the reported speedups as estimates from the original sweep.

These requests do not require additional experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.