acceptodds
Under review as a conference paper at ICLR 2027

NestedWM: World Models with Nested Latent Representations

Abstract

Planning with world models requires repeated rollouts, yet conventional models are trained with a fixed latent size that determines the trade-off between representation capacity and planning cost. We propose NestedWM, which learns nested latent representations that allow a single world model to predict and plan using leading prefixes of its predictive state at different budgets. Each prefix contains either the leading dimensions of a latent vector or the leading tokens of an ordered latent-token sequence. We jointly train an encoder and an action-conditioned predictor over sampled prefix budgets, using each selected prefix as both the predictor input and the target prediction space. In a dimension-wise study with the architecture held fixed, a single encoder and predictor support planning across budgets, achieving high success with only a fraction of the full latent dimensions in most evaluated visual-control and synthetic tasks. Further analyses show that leading prefixes preferentially retain planning-relevant information, while predictable distractors can consume limited representation budget. We then turn this capability into computational savings with a token-wise instantiation that processes only the selected prefix tokens during rollout. In controlled multi-token settings, a single model supports successful planning across multiple token budgets, with smaller prefixes reducing model-based cost-evaluation time.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Model, Latent World Model, LeWorldModel, JEPA, Nested Representation, Matryoshka Representation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_uZukOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper suggests to add elasticity on encoder representations used for world models; one may have to adjust the model based on compute budget or task complexity, and the authors introduce ways to put planning critical information in the earlier tokens/channel dimensions via nested dropout over token/channel dimension, like how FlexTok (Bachmann et al., 2025) enabled similar idea.

Critical strengths
  • Although depending on the task, planning often requires higher-level layouts of the environment and this might be realized better if encoder representations are constrained to have lower rank. Thus, such approach indeed is a valid direction towards efficient and effective world model.
  • In-depth analysis such as Fig.5 and Table 3 makes sense and has good lessons: task-irrelevant-but-predictable information might confound the representation from the desired direction, and one needs to take a good care to realize NestedWM.
Critical weaknesses
  • The model has no ability to adapt itself to find the best budget available that fits the task complexity or the restricted compute budget. This means in practice you never know if you are doing well under good budget-performance trade-off unless you run the same tasks with different budgets over and over again.
  • Speaking of compute budget, which appears multiple times in the intro (L40, L45), and this can only be realized through reducing token count, not via the channel masking. Most of the experiments (from 4.2-4.5) only deal with channel dropout, which is not beneficial for efficient computation. It overperforms the full-channel baseline of LeWM, but this still requires a probe over multiple d-range to realize its performance.
  • Token-nested results are reported only on two tasks.
Questions for authors
  • Could you amplify the Table1,2-like tables over all 6 tasks? This work’s potential lies in how it performs on the token-reduced condition.
  • The reduced-dimension-but-higher-performance works for tasks that require high-level understandings with simpler layouts (such as the 6 tasks in Fig.2) an may not be valid under the tasks that require high-fidelity details (such as robotic manipulation, or any tasks that involves real-world images, not the synthetic ones like those in the paper). I wonder how NestedWM behaves under these circumstances, and if its idea can extend beyond virtual environments.
4 comments

We appreciate your comments on the potential of smaller representations for planning and on our analysis of predictable but task-irrelevant information. We evaluated two ways to select a prefix budget after training and extended the token-wise evaluation to all six tasks.


W1 · Prefix-budget selection

We selected a prefix on a calibration set and measured its planning success rate (SR) on a 500-query evaluation set. We compared two ways to score candidate prefixes.

Planning SR. We ran the planner on 50 calibration queries per candidate. We selected either the prefix with the highest calibration SR or the smallest prefix within two percentage points of that maximum. Both rules still require planning at each candidate budget.

Offline closure ratio. This method scores candidate prefixes using offline predictions and does not require iterative planning. Using recorded observations and actions, we predict the same next-step latent prefix from either the current prefix or the full current latent. We define the closure ratio as the prediction MSE using the current prefix divided by the prediction MSE using the full current latent. A ratio near one means that access to the suffix adds little to predicting the retained prefix, which may indicate that planning success is near a plateau. We select the candidate with the lowest ratio.

Each cell shows the selected prefix and its SR on the existing 500-query evaluation. The last column shows the highest SR among the evaluated candidates and the SR at full budget.

SettingHighest calibration SRSmallest prefix within 2 ppLowest offline closure ratioBest / full-budget SR
Dimension · PushTd24 → 87.8%d24 → 87.8%d128 → 91.4%91.4% / 90.6%
Dimension · TwoRoomd16 → 89.6%d16 → 89.6%d96 → 89.8%89.8% / 84.4%
Dimension · Reacherd16 → 83.6%d16 → 83.6%d32 → 83.4%85.0% / 83.8%
Dimension · Dot1d24 → 97.4%d12 → 97.8%d128 → 99.4%99.4% / 98.0%
Dimension · Dot2d128 → 94.0%d128 → 94.0%d64 → 89.2%94.0% / 69.2%
Dimension · Dot3d128 → 18.8%d128 → 18.8%d128 → 18.8%18.8% / 12.2%
Token · Dot3k2 → 89.4%k2 → 89.4%k3 → 84.0%89.4% / 84.0%
Token · PushTk16 → 88.2%k4 → 85.2%k16 → 88.2%88.2% / 85.0%

Results. Although the selection rules were imperfect, in most settings they found prefixes that matched or came close to the best observed planning success rate.

Figure 12 in Appendix C of the revised version shows how calibration SR and the offline closure ratio relate to SR on the 500-query evaluation set across candidate prefixes in all eight settings.


W2 · Dimension-wise masking and compute savings

We agree that dimension-wise NestedWM yields no computational savings because masking keeps the predictor input size fixed. We designed this variant as a minimal change to LeWM, keeping the architecture fixed to isolate how nested dropout affects the learned representation.


W3 / Q1 · Token-wise evaluation across all six tasks

We trained and evaluated token-wise models on Reacher and TwoRoom following the PushT protocol and on Dot1 and Dot2 following the Dot3 protocol, extending the evaluation to all six tasks in Fig. 2. Following feedback on the original baseline comparison, we trained the fixed-budget LeWM-based students with the same PTF loss and frozen teacher as NestedWM.

PushT and Dot3 show larger changes in planning success as the token budget varies, which is why we used them to illustrate the budget–performance trade-off in the main text. On Dot1, Dot2, and TwoRoom, NestedWM already achieves high success with a single-token prefix. Reacher benefits from a larger prefix, with eight tokens coming within one percentage point of full-budget success.

Reacher

TokensTeacher-matched fixed-budget SR (%)Shared NestedWM SR (%)
173.677.8
282.480.0
882.683.8
2487.684.8

TwoRoom

TokensTeacher-matched fixed-budget SR (%)Shared NestedWM SR (%)
199.096.4
298.496.2
899.297.6
2498.898.8

Dot1

TokensTeacher-matched fixed-budget SR (%)Shared NestedWM SR (%)
199.299.4
299.899.8
3100.099.6

Dot2

TokensTeacher-matched fixed-budget SR (%)Shared NestedWM SR (%)
199.298.6
2100.097.2
396.898.2

Q2 · High-fidelity and real-world observations

We agree that tasks requiring precise manipulation are an important test of NestedWM. Our current experiments leave open whether short prefixes retain the visual details needed for precise alignment and contact. In future work, we plan to compare planning success and latency across prefix budgets on visually realistic simulated manipulation tasks, using matched LeWM-based baselines. We also see evaluation with real camera observations as an important step in assessing whether these results extend beyond simulation.

Reviewer_uZukOpenReview ↗

Post-rebuttal comments on NestedWM

I appreciate the author's effort to resolve weaknesses and questions.

Finding an optimal token / dimension budget for different tasks is indeed a valid research direction, as it is implied in experiments from the rebuttals show; in simple tasks like Dot{1,2} a single tokens seems enough. Yet prefix budget selection still remains questionable, as one has to run the inference on every possible budgets and select which setting is the optimal for each task. My concern was that it is not adaptively chosen by the model itself, but at this stage it sounds more like a future direction that paper cannot provide. Still, the different budget study does deliver some lessons.

The paper still has lot for improvement, as there already have been studies that proposes flexible tokenizers (e.g., FlexTok (ICML 2025) or ElasticTok (ICLR 2025)); it should at least be able to provide a valuable property that is not merely constrained to just applying flexible tokenizer for world modeling. For example, does NestWM's latent representation collect planning-relevant information than others? Or, can it autonomously choos the right token budget when tested on an unseen environment? Something more than showing "the optimal budgets vary by different tasks" is needed.

Therefore, I maintain my rating to 2.

Thank you for the follow-up. We additionally tested automatic, case-dependent budget selection in token-wise PushT and clarify the existing evidence for planning-relevant prefixes.

Planning-relevant information

Section 4.3 / Figure 3 compares frozen NestedWM and LeWM encoders with budget-specific predictors trained under identical settings. NestedWM prefixes achieve higher planning success at all four evaluated budgets on PushT and Dot2. Section 4.4 / Figure 4 complements this dimension-wise comparison with task-state probing, showing that relevant state information is concentrated in the leading coordinates.

Automatic budget selection

Our goal here is to automatically select a budget for each individual start–goal pair within a task, rather than assign one budget to the entire task. Before committing to actions in the environment, we use model predictions to assess whether a small prefix is sufficient or a larger prefix is needed for accurate planning.

We first plan with a small prefix (k=1) and score only the final proposed action sequence using the same frozen model's full prefix (k=24). The score measures predicted endpoint-to-goal latent MSE. A threshold or shallow decision tree accepts the plan or triggers replanning at a larger budget (k=8). We fitted the rules on 500 PushT queries and froze them before evaluation on 500 additional queries with no episode overlap. Decisions use predictions available before execution, without oracle states or observed success.

Selection ruleAverage selected kSelected k=1 / k=8 (%)Adaptive NestedWM SR (%)Matched random SR (%)
Threshold4.6048.6 / 51.488.883.0
Shallow decision tree6.3224.0 / 76.089.286.6

The random reference uses a random budget assignment with matching budget proportions, scored using the corresponding recorded fixed-budget outcomes. The single threshold gains 5.8pp over this reference, demonstrating useful input-dependent allocation, while remaining 2.8pp below always using eight tokens. These results demonstrate the feasibility of automatic, case-dependent budget selection using a single frozen NestedWM.

Case characteristics. We compared cases solved with one token with cases where one token failed but eight succeeded. The one-token group required less rotation to the goal (median 9.2° vs. 11.4°) and started with the agent closer to the block (median surface gap 8.3 vs. 12.8 native px). This descriptive pattern is consistent with a small prefix being sufficient for easier cases in terms of rotation and initial proximity.

Following LeWM, our experiments use world models trained separately for each environment. The additional PushT results demonstrate automatic budget selection for individual start–goal pairs within this setting. Extending selection to unseen environments, where both dynamics predictions and selection criteria must remain reliable, is an important direction for future work.

Reviewer_nE3vOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to make the size of the latent state of a world model adjustable. The authors propose having a representation that does not have a fixed dimension, so that this dimension can be chosen at planning time. They propose NestedWM, where they train the encoder and predictor with nested dropout. At each training step, they sample a prefix budget and restrict both the predictor input and the target to that prefix. At planning time, the goal is encoded at the same prefix budget used for planning. The authors study two versions:

  • A dimension-wise version built on LeWM, where the prefix is the first d out of D dimensions of the latent vector.
  • A token-wise version, where the prefix is the first k out of K tokens (and the predictor processes only these k tokens). They report high success with only a fraction of the full latent dimensions with the dimension-wise version on PushT, TwoRoom, Reacher and synthetic Dot environments.
Critical strengths
  • NestedWM is a single trained model that can then be used for planning with representations of different sizes. This gives flexibility at test time, since one can choose the trade-off between computational cost and planning performance without retraining.
  • The paper is overall well written, and on top of success rate comparisons, the experiments also include analyses of what information the prefixes retain (kNN probing of task states, matched budget-specific predictors, distractors).
Critical weaknesses
  • The dimension-wise version, which is used in most of the experiments, gives no computational savings.
  • Many factors change in the token-wise comparison with LeWM-based models (Tables 1 and 2). NestedWM is initialized from a pretrained full-budget model and trained with a teacher, while the LeWM-based models are trained separately for each token count. Is it thus not clear where the improvements at small token counts come from.
  • The token-wise and dimension-wise versions have different architectures, initializations and training objectives. It is therefore unclear which explains the different results.

Minor

  • Imprecise: “using each selected prefix as both the predictor input and the target prediction space”. A prefix cannot be both an input and a space.
Questions for authors
  • How does the planning time of NestedWM compare to LeWM at full budget?
  • Why is the success rate of the LeWM-based model so low with 1 and 2 tokens in Dot3 (4.0% and 1.2%, Table 1), and why is it lower with 2 tokens than with 1?
  • What happens if you reduce both the number of tokens and the dimension of each token?
  • How does a token-wise NestedWM trained from scratch, without the pretrained initialization and the teacher, compare to the LeWM-based models in Tables 1 and 2?
4 comments

We appreciate your recognition that one trained model can trade off planning cost and success at test time, as well as your comments on our analyses of what the prefixes retain. In response to your questions, we added teacher-matched fixed-budget controls and trained token-wise NestedWM from scratch. We also measured planning latency at matched token budgets and examined per-dot errors on Dot3.


W1 · Dimension-wise compute savings

We agree that dimension-wise NestedWM yields no computational savings because masking keeps the predictor input size fixed. We designed this variant as a minimal change to LeWM, keeping the architecture fixed to isolate how nested dropout affects the learned representation.


W2 / W3 / Q4 · Token-wise performance attribution

Teacher-matched fixed-budget comparison

We trained a separate fixed-budget student for each token budget using the same pretrained initialization, frozen teacher, losses, and number of training epochs as NestedWM, making it a version of NestedWM specialized to a single operating point.

Dot3

TokensTeacher-matched fixed-budget SR (%)Shared NestedWM SR (%)
152.869.0
284.689.4
390.084.0

PushT

TokensTeacher-matched fixed-budget SR (%)Shared NestedWM SR (%)
176.869.0
280.475.6
888.087.2
2485.285.0

On Dot3, matching the training recipe narrows the large gaps reported in Table 1. NestedWM still outperforms the matched students at one and two tokens, suggesting that training across nested budgets can make small prefixes more effective for planning. The fixed-budget student performs better at the full three-token budget.

On PushT, the fixed-budget students achieve higher success rates at every evaluated budget, with small differences at the two larger budgets. Each fixed-budget student commits to one operating point during training, when the best trade-off for a given planning-time constraint may be unknown. NestedWM lets users choose a token budget after training and adjust the trade-off between planning latency and success using the same checkpoint.

NestedWM trained from scratch

To construct the frozen teacher for PTF, we first trained a full-budget model and then reused its weights to initialize the student. We also trained token-wise NestedWM from scratch, removing both the PTF loss and the teacher-derived initialization.

Dot3

TokensFrom-scratch NestedWM SR (%)
118.6
264.6
371.0

PushT

TokensFrom-scratch NestedWM SR (%)
149.0
264.4
879.2
2476.2

Token-wise NestedWM trained from scratch supports planning across token budgets, with success generally improving as more tokens are retained. However, at full budget, its success rate falls below that of the corresponding LeWM-based model on both Dot3 and PushT. These results support our two-stage training recipe. We first learn a predictive representation at full budget, then combine PTF with prefix-to-prefix training so that shorter prefixes learn to preserve the teacher’s predictive information and can support planning on their own.


Q1 · Full-budget planning latency

The per-call savings in Tables 1–2 translate into similar reductions in full planning latency. We measured the wall-clock time of one planning step, covering the entire CEM optimization and its cost-evaluation calls, on an RTX A6000 across 50 planning cases per condition, for both baseline models and NestedWM. Reducing the prefix from 3 to 2 tokens on Dot3 cut full planning latency by 23.3%, and reducing it from 24 to 8 tokens on PushT cut full planning latency by 66.2%.

Dot3

Active tokensLeWM-based (ms/query)NestedWM (ms/query)
1929.0927.2
21,358.11,355.6
3 (full)1,770.81,766.3

PushT

Active tokensLeWM-based (ms/query)NestedWM (ms/query)
1169.4169.8
2238.0238.2
8644.4644.3
24 (full)1,908.41,906.5

Q2 · Dot3 baseline performance

To identify where the baselines failed, we revisited the planning rollouts used to report joint success. For each rollout, we measured the Euclidean distance in pixels from each dot's final position to its target, then averaged these errors separately for each dot across rollouts.

BaselineDot 1 mean error (px) ↓Dot 2 mean error (px) ↓Dot 3 mean error (px) ↓Joint success ↑
Dimension-wise1.601.327.9719.6%
Token-wise, K=10.528.558.704.0%
Token-wise, K=20.419.558.211.2%
Token-wise, K=31.771.601.6686.8%

Although Token-wise baselines with K=1 and K=2 learned more accurate dynamics for first dot compared to Dimension-wise baseline, it lost it's capability on remaining dots, leading to lower success rate. The Token-wise baseline with K=3 achieved low errors on all three dots, suggesting that it learned dynamics sufficient for this task. The dimension-wise model encodes the scene into a single vector, and the token-wise resampler lets us increase the number of latent tokens. A sufficiently large token budget may help the model represent and predict each dot's dynamics with less interference between dots. The three-token result motivated using a larger token budget during training and then learning to plan with smaller prefixes.


Q3 · Reducing token count and token dimension jointly

We appreciate the suggestion to vary both the number of retained tokens and the number of coordinates retained within each token at planning time. Although zero-masking coordinates would provide no additional computational savings, jointly varying these two budgets would give finer control over the representation, making this a natural extension of NestedWM. Evaluating this extension would require training a model across both budgets. This involves deciding how to combine token truncation with coordinate masking, how to define the PTP target for each combination, and how to apply PTF supervision from the full-budget teacher. We look forward to developing and evaluating this extension in future work.


Minor · Wording of the prediction target

We will revise the sentence in the abstract as follows.

We jointly train an encoder and an action-conditioned predictor over sampled prefix budgets, using the selected prefix budget for both the predictor’s latent input and its prediction target.

Thank you for the response and the additional experiments!

The experiments in your response to W2/W3/Q4 separate the contributions of initialization and training objective, but they do not isolate the effect of the architecture. This and the fact that most of the experiments use the dimension-wise version, which gives no compute savings, remain my only concerns.

Thank you again for constructive feedback and further clarifying remaining concern.


Follow-up on W3 · Architecture comparison

We introduced a single-layer cross-attention resampler as a lightweight extension of LeWM’s single-vector encoder. It allows us to choose the token count and token dimension separately and extends planning with multiple latent tokens. Our previous controls kept the token-wise architecture fixed to examine the effects of initialization and training objectives.

Our full-budget LeWM-based baseline for PushT uses a learned-query resampler to produce 24 latent tokens of dimension 8. To examine this encoder design, we constructed a variant using the original LeWM encoder producing single vector without resampler. Its 192-dimensional output was reshaped into 24 tokens of dimension 8. We kept the token-wise predictor and the training and evaluation protocols unchanged. Both models were trained without nested dropout and evaluated using all 24 tokens.

PushT

TokensOriginal LeWM encoder + reshape SR (%)Resampler-based LeWM SR (%)
2487.485.0

Both encoder designs achieve similar observed planning success on PushT at the full 24-token budget.


Follow-up on W1 · Token-wise evaluation across all six tasks

We trained and evaluated token-wise models on Reacher and TwoRoom following the PushT protocol and on Dot1 and Dot2 following the Dot3 protocol, extending the evaluation to all six tasks in Fig. 2. Following feedback on the original baseline comparison, we trained the fixed-budget LeWM-based students with the same PTF loss and frozen teacher as NestedWM.

PushT and Dot3 show larger changes in planning success as the token budget varies, which is why we used them to illustrate the budget–performance trade-off in the main text. On Dot1, Dot2, and TwoRoom, NestedWM already achieves high success with a single-token prefix. Reacher benefits from a larger prefix, with eight tokens coming within one percentage point of full-budget success.

Reacher

TokensTeacher-matched fixed-budget SR (%)Shared NestedWM SR (%)
173.677.8
282.480.0
882.683.8
2487.684.8

TwoRoom

TokensTeacher-matched fixed-budget SR (%)Shared NestedWM SR (%)
199.096.4
298.496.2
899.297.6
2498.898.8

Dot1

TokensTeacher-matched fixed-budget SR (%)Shared NestedWM SR (%)
199.299.4
299.899.8
3100.099.6

Dot2

TokensTeacher-matched fixed-budget SR (%)Shared NestedWM SR (%)
199.298.6
2100.097.2
396.898.2
Reviewer_oGudOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes NestedWM, a latent world model trained so that leading prefixes of its latent state can each be used for prediction and planning. This lets a single encoder-predictor pair operate at several representation budgets chosen after training. Building on LeWorldModel (LeWM), the authors apply nested dropout during joint encoder–predictor training. At each step they sample one prefix budget and train a “prefix-to-prefix” (PTP) predictor to map the current prefix and actions to the next-state prefix at the same budget. SIGReg is applied to the sampled prefix rather than to the full vector.

There are two instantiations:

  • Dimension-wise. Suffix coordinates of a 192-d vector are zero-masked, keeping LeWM’s architecture fixed. This version is used mainly as an analysis tool and saves no compute.
  • Token-wise. A cross-attention resampler produces K ordered tokens and the predictor processes only the retained k tokens. Training is initialized from a pretrained multi-token model, with an extra “prefix-to-full” distillation loss toward a frozen teacher.
Critical strengths
  1. Careful, well designed analysis for the dimension-wise experiments. The matched-predictor experiment (Fig 3) and probing experiment (Fig 4) cleanly attribute the gains of planning performance of NestedWM to improved representation in the prefixes.
  2. It correctly identifies a major weakness of current latent world models of using a fixed latent size, and convincingly shows that in most environments the default latent size from LeWorldModel does not match the inherent dimensionality of the data.
Critical weaknesses
  1. The token-wise comparison, which carries the paper's main empirical demonstration of NestedWM's improvement over baseline, is confounded. Not only does the token-wise NestedWM models initialize from a pretrained full-budget teacher, it is trained using an additional PTF supervision which is essentially distillation from the full-budget teacher. On the other hand, the LeWM-based baselines appear to be trained from scratch without the additional PTF supervision. The large performance improvements from tables 1 and 2 could therefore come largely from teacher distillation and extra training rather than from nesting. If that's the case, that completely shifts the central claim of the paper. In addition, no explanation is given why PTF is used for the token-wise prefix training, but not for the dimension-wise prefix training.
  2. The experimental scope is inconsistent throughout sections. While the dimension-wise prefix planning results (Fig 2) cover all 6 environments, subsequent experiments sand analyses only focus on Push-T and Dot2 / 3, with no explanation given for the selectivity.
Questions for authors
  1. Token-wise attribution: Please report budget-specific LeWM baselines given the same advantages as NestedWM, i.e., initialized from the same pretrained model, distilled from the same teacher, and trained for the same number of epochs. Please also report NestedWM withohut the PTF loss. Do the token-wise gains survive these controls?
  2. Weak Dot3 baselines: Why do the 1- and 2- token LeWM-based baselines in Table 1 reach only 4% and 1.2%, which is significantly lower than the dimension-wise baseline (20%).
  3. Training-seed variance: Please report results over multiple training seeds, with confidence intervals, at least for Figure 2, Table 1, and Table 2
  4. What's the rationale for using PTF loss for token-wise prefix training but not for dimension-wise prefix training?
4 comments

Thank you for the detailed feedback. We added teacher-matched fixed-budget controls and trained token-wise NestedWM from scratch to clarify the sources of its performance. We also analyzed per-dot planning errors on Dot3, extended the token-wise evaluation to all six tasks, and evaluated NestedWM with two additional training seeds.


W1 / Q1 / Q4 · Token-wise performance attribution

Teacher-matched fixed-budget comparison

We trained a separate fixed-budget student for each token budget using the same pretrained initialization, frozen teacher, losses, and number of training epochs as NestedWM, making it a version of NestedWM specialized to a single operating point.

Dot3

TokensTeacher-matched fixed-budget SR (%)Shared NestedWM SR (%)
152.869.0
284.689.4
390.084.0

PushT

TokensTeacher-matched fixed-budget SR (%)Shared NestedWM SR (%)
176.869.0
280.475.6
888.087.2
2485.285.0

On Dot3, matching the training recipe narrows the large gaps reported in Table 1. NestedWM still outperforms the matched students at one and two tokens, suggesting that training across nested budgets can make small prefixes more effective for planning. The fixed-budget student performs better at the full three-token budget.

On PushT, the fixed-budget students achieve higher success rates at every evaluated budget, with small differences at the two larger budgets. Each fixed-budget student commits to one operating point during training, when the best trade-off for a given planning-time constraint may be unknown. NestedWM lets users choose a token budget after training and adjust the trade-off between planning latency and success using the same checkpoint.

NestedWM trained from scratch

To construct the frozen teacher for PTF, we first trained a full-budget model and then reused its weights to initialize the student. We also trained token-wise NestedWM from scratch, removing both the PTF loss and the teacher-derived initialization.

Dot3

TokensFrom-scratch NestedWM SR (%)
118.6
264.6
371.0

PushT

TokensFrom-scratch NestedWM SR (%)
149.0
264.4
879.2
2476.2

Token-wise NestedWM trained from scratch supports planning across token budgets, with success generally improving as more tokens are retained. However, at full budget, its success rate falls below that of the corresponding LeWM-based model on both Dot3 and PushT. These results support our two-stage training recipe. We first learn a predictive representation at full budget, then combine PTF with prefix-to-prefix training so that shorter prefixes learn to preserve the teacher’s predictive information and can support planning on their own.


W2 · Experimental scope

We trained and evaluated token-wise models on Reacher and TwoRoom following the PushT protocol and on Dot1 and Dot2 following the Dot3 protocol, extending the evaluation to all six tasks in Fig. 2. Following feedback on the original baseline comparison, we trained the fixed-budget LeWM-based students with the same PTF loss and frozen teacher as NestedWM.

PushT and Dot3 show larger changes in planning success as the token budget varies, which is why we used them to illustrate the budget–performance trade-off in the main text. On Dot1, Dot2, and TwoRoom, NestedWM already achieves high success with a single-token prefix. Reacher benefits from a larger prefix, with eight tokens coming within one percentage point of full-budget success.

Reacher

TokensTeacher-matched fixed-budget SR (%)Shared NestedWM SR (%)
173.677.8
282.480.0
882.683.8
2487.684.8

TwoRoom

TokensTeacher-matched fixed-budget SR (%)Shared NestedWM SR (%)
199.096.4
298.496.2
899.297.6
2498.898.8

Dot1

TokensTeacher-matched fixed-budget SR (%)Shared NestedWM SR (%)
199.299.4
299.899.8
3100.099.6

Dot2

TokensTeacher-matched fixed-budget SR (%)Shared NestedWM SR (%)
199.298.6
2100.097.2
396.898.2

Q2 · Dot3 baseline performance

To identify where the baselines failed, we revisited the planning rollouts used to report joint success. For each rollout, we measured the Euclidean distance in pixels from each dot's final position to its target, then averaged these errors separately for each dot across rollouts.

BaselineDot 1 mean error (px) ↓Dot 2 mean error (px) ↓Dot 3 mean error (px) ↓Joint success ↑
Dimension-wise1.601.327.9719.6%
Token-wise, K=10.528.558.704.0%
Token-wise, K=20.419.558.211.2%
Token-wise, K=31.771.601.6686.8%

Although Token-wise baselines with K=1 and K=2 learned more accurate dynamics for first dot compared to Dimension-wise baseline, it lost it's capability on remaining dots, leading to lower success rate. The Token-wise baseline with K=3 achieved low errors on all three dots, suggesting that it learned dynamics sufficient for this task. The dimension-wise model encodes the scene into a single vector, and the token-wise resampler lets us increase the number of latent tokens. A sufficiently large token budget may help the model represent and predict each dot's dynamics with less interference between dots. The three-token result motivated using a larger token budget during training and then learning to plan with smaller prefixes.


Q3 · Training-seed variance

We evaluated NestedWM with two additional training seeds (3073 and 3074) and report means and pointwise 95% confidence intervals across the three seeds for both dimension-wise and token-wise experiments. Dimension-wise Reacher runs are in progress.

Dimension-wise (Fig. 2)

Figure 7 in Appendix A.2 of the revised version shows the dimension-wise success curves across training seeds, with pointwise 95% confidence intervals. The Reacher panel shows the original seed only, and will be updated.

Dot3 (Table 1)

Token countSeed 3072 SR (%)Seed 3073 SR (%)Seed 3074 SR (%)Mean SR (%)95% CI (%)
169.065.864.066.3[60.0, 72.6]
289.483.483.485.4[76.8, 94.0]
384.085.284.284.5[82.9, 86.1]

PushT (Table 2)

Token countSeed 3072 SR (%)Seed 3073 SR (%)Seed 3074 SR (%)Mean SR (%)95% CI (%)
169.068.667.268.3[65.9, 70.6]
275.670.473.873.3[66.7, 79.8]
887.287.685.886.9[84.5, 89.2]
2485.084.885.685.1[84.1, 86.2]
Reviewer_oGudOpenReview ↗

Response to rebuttal

I appreciate the authors' extensive additional experiments and efforts to address my concerns.

However, I remain unconvinced by the empirical evidence and would maintain my original rating.

The original submission's main empirical advantage over the LeWM-based baselines came from the token-wise experiments. The new teacher-matched controls show that these advantages largely disappear, or even reverse, when pretrained initialization, PTF distillation, and training duration are matched. This suggests that much of the originally reported improvement may be attributable to differences in the training recipe rather than nesting itself.

Furthermore, the from-scratch NestedWM results are mixed. Although they improve over the original small-budget LeWM baselines on Dot3, they do not consistently outperform LeWM on PushT and fall below the full-budget LeWM baselines on both tasks.

Overall, while the rebuttal strengthens the claim that NestedWM enables flexible planning across multiple token budgets using a single checkpoint, it provides limited evidence that the proposed nesting mechanism offers a substantial performance advantage over comparably trained fixed-budget models. I therefore remain unconvinced that the empirical contribution is sufficiently strong to warrant acceptance.

Thank you for acknowledging that the additional experiments strengthen the evidence for flexible planning across multiple token budgets with a single checkpoint. This flexibility allows users to select a budget suited to their planning-time constraints without committing to one operating point during training. To further address your concern about performance relative to dedicated fixed-budget models, we examined how the prefix-sampling distribution and training duration affect planning performance at each prefix budget.


Follow-up on W1 / Q1 · Token-wise performance attribution

1. Adjusting the prefix-sampling distribution

Our original experiments used uniform sampling over all prefix lengths as a simple default choice. The prefix-sampling distribution is an established design choice within nested dropout. Rippel et al., who introduced nested dropout, use a geometric distribution that assigns higher sampling probabilities to shorter prefixes. Bachmann et al. (FlexTok) compares uniform sampling with power-of-two variants that allocate training differently across token budgets.

Since the largest PushT deficits relative to the teacher-matched fixed-budget students occurred at small prefixes, we examined whether sampling these budgets more frequently improves performance. We assigned 50% of the sampling probability to k∈{1,2,3,4}k\in\{1,2,3,4\} and 50% to k∈{5,…,24}k\in\{5,\ldots,24\}, sampling uniformly within each group. We kept all other training settings unchanged.

PushT

TokensTeacher-matched fixed-budget SR (%)Shared NestedWM, uniform sampling SR (%)Shared NestedWM, adjusted sampling SR (%)
176.869.081.8
280.475.686.2
888.087.287.4
2485.285.085.2

Adjusting the sampling distribution substantially improves small-prefix performance over uniform sampling. NestedWM now outperforms the teacher-matched fixed-budget models by 5.0pp and 5.8pp at one and two tokens, respectively, while remaining competitive at the larger evaluated budgets. This extends the small-prefix advantage of NestedWM previously observed on Dot3 to PushT. Distributing training over only 3 budgets rather than 24 may partly explain why uniform sampling already yielded this advantage on Dot3.

2. Extending training duration

Separately from the sampling-distribution experiment, we extended NestedWM training from 10 to 20 student epochs while retaining the original uniform sampling. Each prefix length is selected on only 1/K1/K of training steps on average (K=24K=24 for PushT), so learning across all budgets may benefit from additional training. We also extended the teacher-matched fixed-budget models to 20 student epochs for comparison. We kept all other training settings unchanged.

PushT

TokensTeacher-matched fixed-budget E20 SR (%)Shared NestedWM E10 SR (%)Shared NestedWM E20 SR (%)
176.869.075.0
280.875.678.2
888.287.290.0
2487.685.091.2

Success improves at all four evaluated budgets after longer training. In this case, performance gain over teacher-matched fixed-budget models are more significant at higher budgets. Here, the student’s planning performance is not limited to that of its initialization teacher and can improve further through our training procedure.

Together, these results indicate that the allocation of training across prefix budgets and training duration affect the performance achievable with a shared model. These findings also motivate further study of how to allocate training across budgets.

Reviewer_67o4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

NestedWM trains a world model on sampled latent prefixes so that one encoder and predictor can plan at several representation budgets. The dimension-wise study examines information allocation while keeping network width fixed. A teacher-guided token-wise variant processes fewer tokens during rollout and demonstrates reduced cost-evaluation time on Dot3 and PushT.

Critical strengths
  • A single model supports several planning budgets without training a separate model for each. Training the predictor recursively at each prefix is a useful extension of nested representations to world models, and the matched-predictor experiments help separate prefix quality from unfamiliarity with truncated inputs.
  • The token-wise results demonstrate a concrete within-model computation trade-off. On PushT, eight tokens achieve 87.2% success at 86.2 ms per cost-evaluation call, compared with 85.0% at 234.9 ms using 24 tokens. The predictable-distractor analysis also usefully distinguishes predictive information from control-relevant information.
Critical weaknesses
  • The token-wise comparison does not isolate the source of the advantage over separately trained low-budget models. NestedWM uses a pretrained full-budget initialization, an additional teacher-target loss, and extra training. A fixed-budget student given the same teacher and training resources would clarify how much of the large Dot3 gap comes from nested training rather than teacher-guided compression. This limits the comparative claim, although it does not negate the demonstrated benefit of supporting several budgets with one model.
  • The robustness of the token-wise operating points is not established across independent training runs. Appendix A.1 explicitly distinguishes the dimension-wise checkpoint evaluations from training-seed variability, while the token-wise tables provide no multi-seed uncertainty. The timings establish within-model per-call savings, but do not yet establish the full planning-time trade-off against matched fixed-budget students.
Questions for authors
  1. How do fixed-budget students trained with the same teacher, a matched initialization scheme, the same losses, and equal training resources compare with NestedWM at representative token budgets? Comparable performance would still support the value of a shared multi-budget model, but would change the interpretation of the reported low-budget gains.
  2. Do the token-wise success/time trade-offs reproduce across independent training seeds? Please clarify the number of training runs behind Tables 1–2 and whether the measured per-call savings translate into similar reductions in full planning latency.
1 comment

Thank you for the constructive feedback and for highlighting the value of using a single world model across planning budgets. We added teacher-matched fixed-budget controls, evaluated two additional training seeds on token-wise tasks, and measured the latency of a complete planning solve beyong per-call latency.


W1 / Q1. Comparison with teacher-matched fixed-budget students

We trained a separate fixed-budget student for each token budget using the same pretrained initialization, frozen teacher, losses, and number of training epochs as NestedWM, making it a version of NestedWM specialized to a single operating point.

Dot3

TokensTeacher-matched fixed-budget SR (%)Shared NestedWM SR (%)
152.869.0
284.689.4
390.084.0

PushT

TokensTeacher-matched fixed-budget SR (%)Shared NestedWM SR (%)
176.869.0
280.475.6
888.087.2
2485.285.0

On Dot3, matching the training recipe narrows the large gaps reported in Table 1. NestedWM still outperforms the matched students at one and two tokens, suggesting that training across nested budgets can make small prefixes more effective for planning. The fixed-budget student performs better at the full three-token budget.

On PushT, the fixed-budget students achieve higher success rates at every evaluated budget, with small differences at the two larger budgets. Each fixed-budget student commits to one operating point during training, when the best trade-off for a given planning-time constraint may be unknown. NestedWM lets users choose a token budget after training and adjust the trade-off between planning latency and success using the same checkpoint.


W2 / Q2. Training seeds and full planning latency

We evaluated NestedWM with two additional seeds (3073 and 3074) alongside the original seed (3072). Each cell shows success rate · mean cost-evaluation time per call.

Dot3

TokensSeed 3072Seed 3073Seed 3074
169.0% · 141.25 ms65.8% · 135.98 ms64.0% · 138.70 ms
289.4% · 187.61 ms83.4% · 180.73 ms83.4% · 182.57 ms
384.0% · 235.33 ms85.2% · 225.01 ms84.2% · 226.31 ms

PushT

TokensSeed 3072Seed 3073Seed 3074
169.0% · 33.40 ms68.6% · 33.20 ms67.2% · 33.07 ms
275.6% · 41.16 ms70.4% · 40.03 ms73.8% · 40.37 ms
887.2% · 87.73 ms87.6% · 81.76 ms85.8% · 82.29 ms
2485.0% · 235.20 ms84.8% · 220.07 ms85.6% · 219.03 ms

Across all three seeds, the eight-token PushT and two-token Dot3 prefixes retained near-full-budget success with lower per-call latency.

The per-call savings in Tables 1–2 translate into similar reductions in full planning latency. We measured the wall-clock time of one planning step, covering the entire CEM optimization and its cost-evaluation calls, on an RTX A6000 across 50 planning cases per condition. Reducing the prefix from 3 to 2 tokens on Dot3 cut full planning latency by 23.3%, and reducing it from 24 to 8 tokens on PushT cut full planning latency by 66.2%.

Dot3

Tokensplanning latency per step (ms)
1927.2
21,355.6
3 (full)1,766.3

PushT

Tokensplanning latency per step (ms)
1169.8
2238.2
8644.3
24 (full)1,906.5

Comments on the paper

1 comment

Dear AC,

Thank you for overseeing the review of our submission. We summarize the strengths recognized across the reviews and how we addressed the main concerns during discussion.


Shared Recognition of Strengths

  • Flexible planning with one trained model. NestedWM allows the latent budget to be chosen after training, without training a separate model for each operating point. Reviewers recognized this flexibility, and the discussion further strengthened the evidence for it. (67o4, nE3v; also acknowledged by oGud during rebuttal)

  • Controlled analysis of why the prefixes work. The matched-predictor experiments separate prefix quality from the predictor’s ability to handle truncated inputs, while task-state probing examines what information the prefixes retain. These analyses were highlighted as a strength of the paper. (67o4, oGud, nE3v)

  • Insight into the limits of predictive representations. The distractor experiments show that predictable information can still be irrelevant to control and interfere with the desired representation. Reviewers found this distinction informative for understanding how to build useful world-model representations. (67o4, uZuk)


How We Addressed the Main Concerns

  • Are the token-wise gains attributable to nesting or to the training recipe?
    We added fixed-budget controls matching initialization, teacher supervision, losses, and training duration, alongside from-scratch NestedWM experiments. These controls clarified the importance of teacher-based training and narrowed the original gaps. Small-prefix advantages nevertheless remained on Dot3 and extended to PushT after adjusting only the prefix-sampling distribution. A separate comparison with equally extended training also showed advantages at larger PushT budgets.

  • Are the token-wise findings sufficiently broad, reproducible, and computationally useful?
    We extended token-wise evaluation to all six tasks and evaluated the original Dot3 and PushT results across three training seeds. We also measured the entire CEM planning step, confirming that the per-call savings translate into reductions in planning latency.

  • Can the model’s flexibility support automatic budget selection?
    Reviewer uZuk asked for additional planning-specific benefits beyond adjustable budgets. We demonstrated case-dependent selection using a frozen PushT model: its predictions assess a small-prefix plan before action execution and trigger larger-budget planning when needed. On queries separate from those used to fit the selection rule, this allocation achieved higher success than random assignment with matching budget proportions. This provides evidence for selecting budgets for individual cases, beyond choosing one budget for an entire task.


These results concern models trained within each environment. Generalization to unseen environments and evaluation on visually realistic manipulation tasks remain open directions. Within this scope, we believe the evidence supports the value of learning one predictive model that remains effective across planning budgets and can use that flexibility to select budgets for individual cases.

Sincerely,
Submission 58181 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.