- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper suggests to add elasticity on encoder representations used for world models; one may have to adjust the model based on compute budget or task complexity, and the authors introduce ways to put planning critical information in the earlier tokens/channel dimensions via nested dropout over token/channel dimension, like how FlexTok (Bachmann et al., 2025) enabled similar idea.
Critical strengths
- Although depending on the task, planning often requires higher-level layouts of the environment and this might be realized better if encoder representations are constrained to have lower rank. Thus, such approach indeed is a valid direction towards efficient and effective world model.
- In-depth analysis such as Fig.5 and Table 3 makes sense and has good lessons: task-irrelevant-but-predictable information might confound the representation from the desired direction, and one needs to take a good care to realize NestedWM.
Critical weaknesses
- The model has no ability to adapt itself to find the best budget available that fits the task complexity or the restricted compute budget. This means in practice you never know if you are doing well under good budget-performance trade-off unless you run the same tasks with different budgets over and over again.
- Speaking of compute budget, which appears multiple times in the intro (L40, L45), and this can only be realized through reducing token count, not via the channel masking. Most of the experiments (from 4.2-4.5) only deal with channel dropout, which is not beneficial for efficient computation. It overperforms the full-channel baseline of LeWM, but this still requires a probe over multiple d-range to realize its performance.
- Token-nested results are reported only on two tasks.
Questions for authors
- Could you amplify the Table1,2-like tables over all 6 tasks? This work’s potential lies in how it performs on the token-reduced condition.
- The reduced-dimension-but-higher-performance works for tasks that require high-level understandings with simpler layouts (such as the 6 tasks in Fig.2) an may not be valid under the tasks that require high-fidelity details (such as robotic manipulation, or any tasks that involves real-world images, not the synthetic ones like those in the paper). I wonder how NestedWM behaves under these circumstances, and if its idea can extend beyond virtual environments.
4 comments
Q2 · High-fidelity and real-world observations
We agree that tasks requiring precise manipulation are an important test of NestedWM. Our current experiments leave open whether short prefixes retain the visual details needed for precise alignment and contact. In future work, we plan to compare planning success and latency across prefix budgets on visually realistic simulated manipulation tasks, using matched LeWM-based baselines. We also see evaluation with real camera observations as an important step in assessing whether these results extend beyond simulation.
Post-rebuttal comments on NestedWM
I appreciate the author's effort to resolve weaknesses and questions.
Finding an optimal token / dimension budget for different tasks is indeed a valid research direction, as it is implied in experiments from the rebuttals show; in simple tasks like Dot{1,2} a single tokens seems enough. Yet prefix budget selection still remains questionable, as one has to run the inference on every possible budgets and select which setting is the optimal for each task. My concern was that it is not adaptively chosen by the model itself, but at this stage it sounds more like a future direction that paper cannot provide. Still, the different budget study does deliver some lessons.
The paper still has lot for improvement, as there already have been studies that proposes flexible tokenizers (e.g., FlexTok (ICML 2025) or ElasticTok (ICLR 2025)); it should at least be able to provide a valuable property that is not merely constrained to just applying flexible tokenizer for world modeling. For example, does NestWM's latent representation collect planning-relevant information than others? Or, can it autonomously choos the right token budget when tested on an unseen environment? Something more than showing "the optimal budgets vary by different tasks" is needed.
Therefore, I maintain my rating to 2.
Thank you for the follow-up. We additionally tested automatic, case-dependent budget selection in token-wise PushT and clarify the existing evidence for planning-relevant prefixes.
Planning-relevant information
Section 4.3 / Figure 3 compares frozen NestedWM and LeWM encoders with budget-specific predictors trained under identical settings. NestedWM prefixes achieve higher planning success at all four evaluated budgets on PushT and Dot2. Section 4.4 / Figure 4 complements this dimension-wise comparison with task-state probing, showing that relevant state information is concentrated in the leading coordinates.
Automatic budget selection
Our goal here is to automatically select a budget for each individual start–goal pair within a task, rather than assign one budget to the entire task. Before committing to actions in the environment, we use model predictions to assess whether a small prefix is sufficient or a larger prefix is needed for accurate planning.
We first plan with a small prefix (k=1) and score only the final proposed action sequence using the same frozen model's full prefix (k=24). The score measures predicted endpoint-to-goal latent MSE. A threshold or shallow decision tree accepts the plan or triggers replanning at a larger budget (k=8). We fitted the rules on 500 PushT queries and froze them before evaluation on 500 additional queries with no episode overlap. Decisions use predictions available before execution, without oracle states or observed success.
| Selection rule | Average selected k | Selected k=1 / k=8 (%) | Adaptive NestedWM SR (%) | Matched random SR (%) |
|---|---|---|---|---|
| Threshold | 4.60 | 48.6 / 51.4 | 88.8 | 83.0 |
| Shallow decision tree | 6.32 | 24.0 / 76.0 | 89.2 | 86.6 |
The random reference uses a random budget assignment with matching budget proportions, scored using the corresponding recorded fixed-budget outcomes. The single threshold gains 5.8pp over this reference, demonstrating useful input-dependent allocation, while remaining 2.8pp below always using eight tokens. These results demonstrate the feasibility of automatic, case-dependent budget selection using a single frozen NestedWM.
Case characteristics. We compared cases solved with one token with cases where one token failed but eight succeeded. The one-token group required less rotation to the goal (median 9.2° vs. 11.4°) and started with the agent closer to the block (median surface gap 8.3 vs. 12.8 native px). This descriptive pattern is consistent with a small prefix being sufficient for easier cases in terms of rotation and initial proximity.
Following LeWM, our experiments use world models trained separately for each environment. The additional PushT results demonstrate automatic budget selection for individual start–goal pairs within this setting. Extending selection to unseen environments, where both dynamics predictions and selection criteria must remain reliable, is an important direction for future work.
We appreciate your comments on the potential of smaller representations for planning and on our analysis of predictable but task-irrelevant information. We evaluated two ways to select a prefix budget after training and extended the token-wise evaluation to all six tasks.
W1 · Prefix-budget selection
We selected a prefix on a calibration set and measured its planning success rate (SR) on a 500-query evaluation set. We compared two ways to score candidate prefixes.
Planning SR. We ran the planner on 50 calibration queries per candidate. We selected either the prefix with the highest calibration SR or the smallest prefix within two percentage points of that maximum. Both rules still require planning at each candidate budget.
Offline closure ratio. This method scores candidate prefixes using offline predictions and does not require iterative planning. Using recorded observations and actions, we predict the same next-step latent prefix from either the current prefix or the full current latent. We define the closure ratio as the prediction MSE using the current prefix divided by the prediction MSE using the full current latent. A ratio near one means that access to the suffix adds little to predicting the retained prefix, which may indicate that planning success is near a plateau. We select the candidate with the lowest ratio.
Each cell shows the selected prefix and its SR on the existing 500-query evaluation. The last column shows the highest SR among the evaluated candidates and the SR at full budget.
Results. Although the selection rules were imperfect, in most settings they found prefixes that matched or came close to the best observed planning success rate.
Figure 12 in Appendix C of the revised version shows how calibration SR and the offline closure ratio relate to SR on the 500-query evaluation set across candidate prefixes in all eight settings.
W2 · Dimension-wise masking and compute savings
We agree that dimension-wise NestedWM yields no computational savings because masking keeps the predictor input size fixed. We designed this variant as a minimal change to LeWM, keeping the architecture fixed to isolate how nested dropout affects the learned representation.
W3 / Q1 · Token-wise evaluation across all six tasks
We trained and evaluated token-wise models on Reacher and TwoRoom following the PushT protocol and on Dot1 and Dot2 following the Dot3 protocol, extending the evaluation to all six tasks in Fig. 2. Following feedback on the original baseline comparison, we trained the fixed-budget LeWM-based students with the same PTF loss and frozen teacher as NestedWM.
PushT and Dot3 show larger changes in planning success as the token budget varies, which is why we used them to illustrate the budget–performance trade-off in the main text. On Dot1, Dot2, and TwoRoom, NestedWM already achieves high success with a single-token prefix. Reacher benefits from a larger prefix, with eight tokens coming within one percentage point of full-budget success.
Reacher
TwoRoom
Dot1
Dot2