acceptodds
Under review as a conference paper at ICLR 2027

World Models with Configurable Perception

Abstract

Real-world observations are noisy and often contain excessive task-irrelevant information. Effective world modeling and planning need to abstract observations into world state representations while filtering out irrelevant information. However, encoders in existing world models are typically non-adaptive, leading to state representations that are easily distracted by task-irrelevant information. As a result, the model may not selectively filter observations to cater to different task demands. To address this problem, we introduce ConfigWM, which realizes configurable perception through task-conditioned state extraction. Given a language description of the task at hand, ConfigWM uses it as a configuration signal to steer how visual embeddings are mapped into a task-specific state space, selectively preserving information relevant to the current task while suppressing task-irrelevant variations. Action-conditioned dynamics are then learned in this task-specific state space for planning. We evaluate ConfigWM in environments with increasing levels of distraction. While existing methods exhibit substantial declines in planning success rate as the environments become increasingly noisy, ConfigWM remains comparatively stable and outperforms the strongest competing method by 22.3% on average across diverse environments. Consistent gains in physical-quantity probing (+0.365 Pearson r) and distracted-to-clean retrieval (+75.5%) further indicate that ConfigWM learns states that preserve task-relevant information while remaining robust to distractors, supporting reliable world modeling and planning in noisy worlds.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Models, Configurable Perception, Noisy Worlds, JEPA

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hLiMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors contribute a method to condition a perception layer of a model-based planning system with task information. Their goal is to use knowledge from a task to enable the perception layer to learn representations that ignore task-irrelevant information.

One of the core assumptions of the method is that it is possible to obtain a distractor-free version of any observation without training an additional component. (The methodology uses a pipeline with off-the-shelf components.)

The validation centers on experiments conducted in three simulated environments. These experiments have the distractors causally isolated from the actual task. E.g., in Push-T, the distractor objects do not interact (collide) with the task's object.

Critical strengths
  • The manuscript presents a synthetic data-driven methodology to address an important problem. This could allow training of the perception layer without relying on costly on-policy rollouts.

  • The experiments measure and validate some of the properties one would want out of "configurable perception". The authors do probing on physical quantities and a causal intervention to establish (to a meaningful, but not overwhelming extent) that the query does carry information and it is used.

Critical weaknesses

1.Experiments are limited by distractors being causally/physically inert. The model is never tested against objects that could interact with the task object, which is one of the central difficulties of multi-object scenes that are the focus of your evaluation; in this situation distractors would become non-distractors depending on the dynamics (e.g., if an object interacts with the task object). This makes the evaluation of limited value for testing the core claim because the "distractors" in the evaluation are of a very specific kind. It seems that the method could incorporate more realistic distractors, but a significant extension to the core experiments would be necessary to show so.

2. Availability of "distractor-free" synthetic versions of any given observation is a strong assumption. A core part of the contribution is the idea of using distractor-free versions of observations, which are synthetically augmented with distractors, but the paper does not show the methodology could be applied (in principle) to a realistic setting. My main concern here is that "augmenting" data in a way that generates realistic/in-distribution distractors is a core challenge faced by the method, but it is not explored and instead is worked around. For instance, distractors in the real world exhibit things like object permanence and can interact with "non-distractors". The fact that you use completely different techniques to generate this data (overlaying, tiling and an image-edit model) I believe supports my claim that this is non-trivial and a solution to a core challenge has not been identified.

3. The dynamics model is not evaluated in isolation. Nothing in the paper measures the dynamics model's prediction accuracy directly; every result either probes the encoder alone or measures the full pipeline (encoder+dynamics+planning) at once. We cannot tell, whether planning gains come from better dynamics modeling (the "world model") versus a representation space that's merely more CEM-friendly.

Questions for authors
  • Would you consider running an experiment where distractors are real objects and the augmentation pipeline used for the real world data is used?
    • E.g., a variation of any of your simulated environments where distractors are other objects and the image-edit model is used.
    • This would be direct evidence for your core claim, whereas your current experiments are only indirect (the core experiments test on highly specific artificial distractors and with varying generation pipelines, as described above.)
  • Do you have any evidence that the learned representation encodes the correct information also when the distractors are more complex? E.g., in situations where distractors become non-distractors, and vice versa?
  • How do you think this approach compares to "emergent" configurable perception which might arise during, e.g., RL? I would imagine it is (in principle) possible to learn configurable perception without generating the synthetic data you propose generating, but perhaps much more inefficiently. Having data that quantifies the efficiency bought by your method would be valuable.
4 comments

We sincerely thank the reviewer for the thorough assessment and insightful feedback. Following your suggestions, we have substantially extended our evaluation with physically interacting objects, image-edit-generated visual distractors, planner-free physical-state prediction, and a Stage-A data-efficiency study. We hope these additional evaluations and analyses help alleviate your concerns and strengthen the experimental evidence supporting our approach.

W1 & Q2: Task-Relevant Representations under Physical Interactions

Thank you for your insightful comments. Our original evaluation setting deliberately introduces non-interacting visual distractors to clearly distinguish task-relevant from task-irrelevant information. Since these distractors are introduced through observation-space augmentation without affecting the queried target or the underlying task dynamics, they are task-irrelevant by construction. The original observations therefore preserve the task-relevant information, naturally serving as clean targets for Stage-A and encouraging the encoder to retain task-relevant information while filtering out irrelevant visual variations.

Meanwhile, we agree that evaluating physically interacting objects is an important extension of our controlled setting. Following your suggestion, we further introduce a physically collidable blue ball in Two-Room to investigate whether ConfigWM can capture interaction-relevant information while the configurable encoder remains frozen with the query "red ball". Contact-window prediction error measures the mean Euclidean distance between predicted and ground-truth red-ball positions around collision events. As shown below, training with physical collisions reduces the contact-window prediction error by 64.5% at one step and 43.4% at five steps compared with training without collisions.

MethodContact 1-step Error ↓Contact 5-step Error ↓
ConfigWM (Collision Training)7.526.24
ConfigWM (No-Collision Training)21.1711.03
DINOv2-WM (Collision Training)24.7917.41

Furthermore, linear probing of the learned state embeddings yields a lower position MSE for the non-queried blue ball, decreasing from 0.859 to 0.695 (19.0%) compared with training without collisions. These results suggest that ConfigWM can faithfully follow language-guided configurable perception while further adapting its latent representations to preserve physically relevant information through downstream dynamics learning. We have incorporated these results into Appx C.8 of the revised paper.

W3: Evaluating World Model Prediction Accuracy

Planning success alone does not directly measure the world model's prediction accuracy. To directly evaluate world model prediction accuracy independently of CEM planning, we conducted additional open-loop physical-state prediction experiments across all three environments. Specifically, we evaluate the predicted states against ground-truth simulator states using frozen linear probes. We report Euclidean position RMSE across different distraction levels and prediction horizons (h1h_1/h5h_5 corresponding to 5/25 simulator steps), as shown below.

EnvironmentLevelConfigWM h1h_1DINOv2-WM h1h_1ConfigWM h5h_5DINOv2-WM h5h_5
Two-Roomd1d_16.4321.7616.1639.10
d4d_48.0033.5217.0657.11
d8d_88.9138.5910.9962.12
Push-Td1d_157.2178.9691.56107.13
d4d_452.47115.2787.47138.79
d8d_851.32129.5785.01151.29
OGBench-Cubed1d_16.5818.699.8717.35
d4d_47.2820.799.4319.88
d8d_86.7719.9410.8120.12

As shown, ConfigWM consistently achieves lower physical prediction errors across evaluated distraction levels and both prediction horizons. These results demonstrate that ConfigWM's improvements extend beyond CEM-friendly representations to more accurate physical-state prediction. We have incorporated these additional results into Section 5.8 of the revised paper.

W2 & Q1: Clean Target Construction and Realistic Distractor Evaluation

We agree that constructing reliable noisy–clean training pairs in realistic scenes is challenging. This is precisely why we adopt a clean-to-noisy augmentation strategy: we start from existing observations and construct noisy inputs by adding visual distractors. Similar to the denoising objective in diffusion models, our strategy does not require perfectly distractor-free original observations: they serve as clean targets relative to the added distractors, allowing the configurable encoder to learn to suppress the injected task-irrelevant information while preserving task-relevant content.

We have already validated the scalability of this approach on real-world data (Sec. 6 and Appx. D), using existing egocentric (e.g., Ego4D[1], EPIC-KITCHENS-100[2]), robot manipulation (e.g., DROID[3], Open X-Embodiment[4]), internet activity and instructional videos (e.g., ActivityNet[5], COIN[6]). Following the same pipeline, we treat original observations as clean targets and generate noisy inputs by adding visual distractors through image editing. Across four visual backbones, our configurable encoder reduces distractor AP by 17.9 percentage points on average while largely preserving task-core AP, with qualitative results shown in Figs. 10 and 22.

Furthermore, following your suggestion, we use the same image-editing model as our real-world data pipeline to generate real objects (e.g., dinosaurs, ducks, and bottles), which are then extracted and inserted into Two-Room, Push-T, and OGBench-Cube as static or moving distractors. We directly evaluate the existing checkpoints through closed-loop planning. Representative results with two inserted objects (d2d_2) are shown below.

EnvironmentMethodStaticMoving
Two-RoomConfigWM86.7 ± 4.187.3 ± 5.0
DINOv2-WM76.7 ± 4.180.7 ± 3.4
Push-TConfigWM27.3 ± 0.928.7 ± 3.4
DINOv2-WM6.7 ± 3.47.3 ± 5.0
OGBench-CubeConfigWM65.3 ± 9.662.7 ± 9.0
DINOv2-WM52.0 ± 5.950.7 ± 4.1

As shown, ConfigWM achieves higher planning success than DINOv2-WM under both static and moving distractors across all three environments at d2{d_2}. These results further support the applicability of our image-edit-based augmentation approach to simulated environments and the robustness of ConfigWM to more diverse visual distractors. We have included the corresponding visualizations and complete experimental results in Appx C.10 of the revised paper.

[1] Ego4D: Around the World in 3,000 Hours of Egocentric Video. CVPR 2022

[2] Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100. IJCV 2022

[3] DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset. RSS 2024

[4] Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration. ICRA 2024

[5] ActivityNet: A Large-Scale Video Benchmark for Human Activity Understanding. CVPR 2015

[6] COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis. CVPR 2019

Q3: Data Efficiency of Configurable Perception

As you suggest, configurable perception could potentially emerge through RL, where task-relevant representations are learned implicitly from reward signals and interactions. In contrast, our approach explicitly learns task-conditioned representations from synthetic paired supervision, without requiring task rewards or on-policy interactions during perceptual pretraining.

To further investigate its data efficiency, we conducted an additional experiment on Two-Room d8d_8, varying the proportion of original trajectories available for Stage-A training while keeping the training steps and full Stage-B data unchanged. All data fractions were independently retrained under the same training and evaluation protocol, including the full dataset. The planning success rates (%) are shown below.

Stage-A DataAvailable TrajectoriesPlanning Success (%)
0.1%1074.7 ± 7.4
0.5%5085.3 ± 8.1
1%10094.7 ± 0.9
10%1,00095.3 ± 3.4
100%10,00095.3 ± 4.1

Notably, using only 1% of the original trajectories for Stage-A training achieves 94.7% planning success, comparable to 95.3% with the full dataset. These results suggest that effective configurable perception can be learned with relatively limited paired supervision. We have incorporated the complete results into Appx C.9 of the revised paper.

Dear Reviewer hLiM,

As the discussion period draws to a close, we would greatly appreciate any feedback you may have on our rebuttal and revised manuscript. We would be happy to address any remaining questions or concerns.

Thank you again for your time and consideration!

Best regards,
Authors

Dear Reviewer hLiM,

As today is the final day of the discussion period, we would like to sincerely thank you for the many helpful suggestions you provided on our paper, especially your perspective on the connection between configurable perception and RL, which helped us recognize another potential advantage and application of our method that we had not fully considered before. Following your suggestions, we have conducted additional experiments and provided further clarifications. We hope our responses have helped address your concerns. Regardless, we truly appreciate your valuable feedback on our paper.

Thank you once again for your careful review and constructive comments.

Best regards,
The Authors

Reviewer_zuL4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ConfigWM uses task queries to construct task-relevant visual states for world modeling and planning. A configurable encoder is trained on noisy observations, queries, and corresponding clean targets, then frozen while a state projector and action-conditioned dynamics model are learned. Planning uses CEM-based MPC in the resulting latent space. Experiments on three environments with synthetic dynamic distractors show improved planning robustness, supported by representation analyses and query-conditioning experiments. An additional study explores representation learning on edited real-world images.

Critical strengths
  1. Important research direction and coherent design. Adapting world representations to task requirements is an important direction for world models. ConfigWM introduces task conditioning at the perception stage, allowing subsequent dynamics prediction and planning to operate on task-relevant states. The overall design is intuitive and well motivated.

  2. Controlled evaluation and complementary mechanism analyses. Increasing visual distraction while preserving underlying trajectories helps isolate its effect on planning. Physical-quantity probing, clean–distracted retrieval, query-dependent reconstructions, and query-mismatch experiments provide complementary evidence that the query changes the representation and affects planning.

Critical weaknesses
  1. Limited validation of task-relevant state abstraction. Added distractors do not affect the underlying simulator dynamics, and OGBench distractors occupy separate tiled scenes. The experiments therefore do not test whether the representation retains non-target objects whose collisions or other physical interactions make them relevant to the task. This limits the evidence for task-relevant abstraction in realistic interactive environments, although the visual-robustness results remain valuable.

  2. Insufficient isolation of the contribution of configurable perception. ConfigWM receives substantial paired noisy–clean supervision during encoder training, whereas DINOv2-WM does not receive comparable adaptation. Matching the downstream dynamics and planner does not isolate the benefit of the representation design from this additional supervision. A simpler query-conditioned masking or target-extraction baseline trained with the same supervision would help clarify the contribution.

Questions for authors
  1. Task relevance. How would clean training targets be defined when non-target objects physically affect the task? Please clarify how the method would retain such objects. As further validation, a test with physically interacting objects would help assess this capability.

  2. Contribution isolation. Could the authors compare against a simpler query-conditioned masking or target-extraction module trained with the same paired supervision and connected to the same dynamics model and planner?

3 comments

We sincerely appreciate your positive assessment and constructive feedback. To further strengthen our evaluation, we have introduced physically interacting objects in Two-Room and conducted additional controlled ablations to examine the contributions of task conditioning and paired perceptual pretraining. We present these new results and address your questions below. We hope these new results help resolve your concerns regarding task-relevant representations and the contributions of configurable perception.

W1 & Q1: Task-Relevant State Abstraction under Physical Interactions

Thank you for your insightful comments. Our original evaluation setting deliberately introduces non-interacting visual distractors to clearly distinguish task-relevant from task-irrelevant information. Since these distractors are introduced through observation-space augmentation without affecting the queried target or the underlying task dynamics, they are task-irrelevant by construction. The original observations therefore preserve the task-relevant information, naturally serving as clean targets for Stage-A and encouraging the encoder to retain task-relevant information while filtering out irrelevant visual variations.

Meanwhile, we agree that evaluating physically interacting objects is an important extension of our controlled setting. Following your suggestion, we further introduce a physically collidable blue ball in Two-Room to investigate whether ConfigWM can capture interaction-relevant information while the configurable encoder remains frozen with the query "red ball". Contact-window prediction error measures the mean Euclidean distance between predicted and ground-truth red-ball positions around collision events. As shown below, training with physical collisions reduces the contact-window prediction error by 64.5% at one step and 43.4% at five steps compared with training without collisions.

MethodContact 1-step Error ↓Contact 5-step Error ↓
ConfigWM (Collision Training)7.526.24
ConfigWM (No-Collision Training)21.1711.03
DINOv2-WM (Collision Training)24.7917.41

Furthermore, linear probing of the learned state embeddings yields a lower position MSE for the non-queried blue ball, decreasing from 0.859 to 0.695 (19.0%) compared with training without collisions. These results suggest that ConfigWM can faithfully follow language-guided configurable perception while further adapting its latent representations to preserve physically relevant information through downstream dynamics learning. We have incorporated these results into Appx C.8 of the revised paper.

W2 & Q2: Isolating the Contribution of Configurable Perception

We agree that distinguishing the contributions of language conditioning and paired perceptual pretraining is important for a fair comparison. To examine this more directly, we conducted two additional ablation experiments. Specifically, (1) DINOv2-WM + Query introduces task-query conditioning into Stage-B without Stage-A pretraining, while (2) DINOv2-WM + Stage-A (w/o Query) receives the same paired noisy–clean pretraining as ConfigWM, but without task-dependent language conditioning. As shown in the table below, neither variant achieves comparable performance to the full ConfigWM, particularly under severe distraction (d8). Importantly, DINOv2-WM + Stage-A (w/o Query) receives the same paired supervision as ConfigWM, yet performs substantially worse. These results indicate that ConfigWM's advantages cannot be attributed solely to additional paired supervision, highlighting the importance of combining task-dependent query conditioning with paired perceptual pretraining. We have incorporated these results into Section 5.9 of the revised paper.

EnvironmentMethodd0d_0d4d_4d8d_8
Two-RoomDINOv2-WM + Query97.3 ± 0.967.3 ± 8.265.3 ± 3.8
DINOv2-WM + Stage-A (w/o Query)96.0 ± 3.372.7 ± 4.164.0 ± 5.9
ConfigWM (Ours)97.3 ± 0.997.3 ± 2.594.0 ± 1.6
Push-TDINOv2-WM + Query54.0 ± 4.34.7 ± 0.94.7 ± 0.9
DINOv2-WM + Stage-A (w/o Query)46.7 ± 5.723.3 ± 3.46.0 ± 2.8
ConfigWM (Ours)49.3 ± 4.135.3 ± 6.636.0 ± 4.9
OGBench-CubeDINOv2-WM + Query82.7 ± 2.558.0 ± 7.152.7 ± 5.7
DINOv2-WM + Stage-A (w/o Query)79.3 ± 9.845.3 ± 9.646.7 ± 5.0
ConfigWM (Ours)83.3 ± 8.468.0 ± 9.968.0 ± 8.5

Thank you for the additional experiments. The ablations address my concern about paired supervision, and the collision results strengthen the evidence for retaining physically relevant information. While the physical-interaction evaluation remains limited in scope, I maintain my rating of 3.

Thank you for your thoughtful follow-up and for acknowledging our additional experimental evidence. We are actively exploring more complex interaction settings to further strengthen our evaluation. We sincerely appreciate your constructive feedback. If you have any further questions or concerns, please feel free to let us know. We would be happy to address them promptly.

Reviewer_uVXNOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes ConfigWM where a language query conditions the predictions of a JEPA world model. There is an initial training stage where they predict clean targets from distracting observations and a language cue about the target. Afterwards, they learn a world model on top of the frozen stage-1 representations with SIGReg and JEPA loss. The context and target have distractors but the action only corresponds to target in the WM training stage. The authors show that baseline WM methods eg LeWM and Dino-WM fail when trained on these distracting rollouts while the ConfigWM maintains performance thanks to its distractor-to-clean pretraining and the language cue conditioning.

Critical strengths
  1. The problem addressed is an important one in world modeling and robotics especially in the age of real-world embodied AI where task-irrelevant distractors can exist in the data. The use of language cues to point the model towards task-relevant objects is a natural and scalable approach to address this problem. Also, the benchmark created can be useful for WM researchers to evaluate model robustness under distractors.

  2. The experiments show a gradual failure in planning performance and physics probing with increasing distractor level for the end-to-end baselines that do not have cue conditioning even when they are trained on distracting rollouts. On the other hand ConfigWM is able to maintain planning and probing performance when distractors are introduced thanks to it stage-1 pretraining and language conditioning.

Critical weaknesses
  1. My main concern is that the benchmark is designed in a way that gives ConfigWM an unfair advantage, and this advantage is never given to baselines. The baselines are trained end-to-end on distracting rollouts (but only the action corresponding to the target rollout is provided for prediction. During planning, given a pair of noisy observation and action, the model has no way of knowing the action corresponds to which object (what is the target), and planning performance degradation is expected. In order to remove any confounders and see if the ConfigWM advantage comes from only language conditioning or the distractor-to-clean predictions in stage-1 or both, would you be able to show experiments where you empower the baselines with language cue embeddings? And vice versa: could you empower the baselines by a pre-training phase where you train on distractor-to-clean rollouts without any language conditioning?

  2. Another fairness concern is that some of the baselines including LeWM have backbones initialised from scratch while ConfigWM and other baselines use pre-trained backbones. Is using a fine-tuned backbone is necessary (maybe due to joint vision-language training) in ConfigWM or can you train it from scratch in stage 1? Also, compare training and planning/inference flops with baselines (i imagine the model is much more expensive due to the Llama 3 backbone).

  3. d0 to dk performance in Figs 12 and 13 point to a flaw of the approach that is unstated by the authors. On Push-T the performance is much worse than LeWM which shoes that the stage-1 training and/or language conditioning can hurt in distractor-free environments. Can you justify why and provide a remedy to maintain the performance in these conditions?

Questions for authors

See weaknesses.

5 comments

We sincerely thank the reviewer for the careful assessment and constructive suggestions. To address your concerns regarding comparison fairness, pretrained initialization, computational cost, and clean-setting performance, we have conducted additional controlled ablations, evaluated scratch initialization, measured training and planning FLOPs, and clarified the clean Push-T performance trade-off. We present our detailed responses and new results below. We hope these additional experiments and clarifications adequately address your concerns and provide stronger support for our proposed approach.

W1: Analyzing the Sources of ConfigWM's Advantages

We agree that distinguishing the contributions of language conditioning and paired perceptual pretraining is important for a fair comparison. To examine this more directly, we conducted two additional ablation experiments. Specifically, (1) DINOv2-WM + Query introduces task-query conditioning into Stage-B without Stage-A pretraining, while (2) DINOv2-WM + Stage-A (w/o Query) receives the same paired noisy–clean pretraining as ConfigWM in Stage-A, but without task-dependent language conditioning. The pretrained encoder is then used for the same Stage-B world model training. Both variants follow the same downstream training and planning protocols as ConfigWM.

EnvironmentMethodd0d_0d4d_4d8d_8
Two-RoomDINOv2-WM + Query97.3 ± 0.967.3 ± 8.265.3 ± 3.8
DINOv2-WM + Stage-A (w/o Query)96.0 ± 3.372.7 ± 4.164.0 ± 5.9
ConfigWM (Ours)97.3 ± 0.997.3 ± 2.594.0 ± 1.6
Push-TDINOv2-WM + Query54.0 ± 4.34.7 ± 0.94.7 ± 0.9
DINOv2-WM + Stage-A (w/o Query)46.7 ± 5.723.3 ± 3.46.0 ± 2.8
ConfigWM (Ours)49.3 ± 4.135.3 ± 6.636.0 ± 4.9
OGBench-CubeDINOv2-WM + Query82.7 ± 2.558.0 ± 7.152.7 ± 5.7
DINOv2-WM + Stage-A (w/o Query)79.3 ± 9.845.3 ± 9.646.7 ± 5.0
ConfigWM (Ours)83.3 ± 8.468.0 ± 9.968.0 ± 8.5

As shown in the table, neither DINOv2-WM + Query nor DINOv2-WM + Stage-A (w/o Query) achieves satisfactory performance, particularly under the highest distraction level (d8). These results indicate that the advantages of the full ConfigWM arise from both Stage-A pretraining and the help of task queries, demonstrating the necessity of our proposed design. We have incorporated these results into Section 5.9 of the revised paper.

W2: Exploring the Necessity of Initializing Weights

Thank you for highlighting the importance of a fair comparison regarding pretrained initialization. To examine whether ConfigWM relies on pretrained visual features, we conducted an additional experiment in which the Stage-A visual encoder is initialized from scratch, followed by the same Stage-B world model training. As shown in the table below, ConfigWM with a randomly initialized visual encoder still achieves strong performance and outperforms the original DINOv2-WM baseline across all evaluated distraction levels. These results suggest that pretrained visual encoder initialization is not strictly necessary for ConfigWM to achieve substantial improvements over DINOv2-WM. We have incorporated these results into Appx C.6 of the revised paper.

EnvironmentMethodd0d_0d4d_4d8d_8
Two-RoomDINOv2-WM98.0 ± 1.662.7 ± 0.952.7 ± 1.9
ConfigWM (Scratch Init.)98.7 ± 1.994.0 ± 2.895.3 ± 3.8
Push-TDINOv2-WM58.7 ± 0.94.0 ± 0.04.0 ± 1.6
ConfigWM (Scratch Init.)61.3 ± 6.652.0 ± 1.621.3 ± 0.9
OGBench-CubeDINOv2-WM81.3 ± 5.750.0 ± 3.353.3 ± 5.7
ConfigWM (Scratch Init.)82.7 ± 9.362.7 ± 6.258.7 ± 8.2

W2: Computational Cost

Thank you for raising the question of computational overhead. To make this trade-off more explicit, we measured the total training and planning FLOPs, as summarized below. As shown in Appendix B.1, ConfigWM uses only the first three Transformer layers of Llama-3.2-1B rather than the full model. Although this increases the training cost, planning FLOPs increase by only 14.2% compared with DINOv2-WM and 34.5% compared with LeWM, since CEM performs candidate action rollouts directly in the latent space without repeatedly reusing the configurable encoder. We have incorporated this comparison into Appx C.7 of the revised paper.

MethodTraining FLOPs (EFLOPs)Planning FLOPs (GFLOPs)
DINOv2-WM1.0712997.2
LeWM0.7272543.8
ConfigWM (Ours)6.1553422.4

W3: Clean-Setting Performance Trade-off

Thank you for highlighting the performance gap on clean Push-T. We would like to clarify that DINOv2-WM provides the most directly comparable baseline to ConfigWM, as both share the same downstream world model architecture and planning protocol, with the main difference lying in how visual representations are constructed. On clean Push-T (d0d_0), DINOv2-WM itself achieves a lower success rate than LeWM (58.7% vs. 81.3%), while the gap between ConfigWM and DINOv2-WM is considerably smaller. Furthermore, our additional experiment with ConfigWM (Scratch Init.) achieves 61.3% success on clean Push-T, outperforming both DINOv2-WM and the original ConfigWM. This suggests that clean-setting performance can be improved through scratch initialization. We clarify this performance trade-off in Section 5.2 of the revised paper.

Dear Reviewer uVXN,

As the discussion period draws to a close, we would greatly appreciate any feedback you may have on our rebuttal and revised manuscript. We would be happy to address any remaining questions or concerns.

Thank you again for your time and consideration!

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.