acceptodds
Under review as a conference paper at ICLR 2027

Learning Modular World Models by Composing Reusable Dynamics

Abstract

Massively multitask world models promise to turn broad interaction data into reusable dynamics for control, but such reuse is not guaranteed by pooling tasks into a single dense conditional model. In novel tasks, a held-out environment may combine familiar objects, goals, initial states, or interaction patterns in a configuration never observed during training. We formulate this challenge as concept accumulation and recombination: each environment constrains only part of a latent concept library, while evaluation requires composing the learned parts under new task contexts. As a guiding analysis, we study this problem in a sparse latent-dynamics setting, where sparse predictive supports and a shared interaction rule are sufficient for recombination under explicit structural assumptions. Motivated by this principle, we introduce CWM, a compositional task-routing and dynamics-masking framework that separates task-expert routing from entity and interaction masking, identifying which factors are predictively relevant before modeling how they evolve under actions. This simple structural bias encourages the world model to share reusable transition rules while suppressing task-irrelevant factors and environment-specific co-occurrence shortcuts. On MM-Bench, which spans more than 200 tasks across 10 domains, CWM preserves the scalability of massively multitask training, yielding 32% overall gain over the state-of-the-art. It also provides stronger adaptation to held-out compositional tasks, showing that a sparse task-routing and dynamics-masking inductive bias can improve both large-scale multitask learning and generalization to new task recombinations.

Primary area:
reinforcement learning
Keywords:
World models, compositionality, generalization, multi-task learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KrYnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CWM, which augments a NewT/TD-MPC2 latent dynamics model with language-conditioned task-expert routing, entity gating, and interaction-message gating. The aim is to improve cross-task sharing and compositional generalization. Experiments cover more than 200 tasks across 10 domains in MM-Bench. The paper evaluates both in-distribution multitask learning and adaptation to held-out task compositions.

Critical strengths
  1. The empirical coverage is broad, and CWM consistently improves over NewT across many domains, demonstrating practical value for large-scale multitask control.

  2. The ablations are reasonably thorough: Dense-OC and Dense-OC + Task-MoE control for entity representations and expert capacity, supporting the effectiveness of the gating mechanisms.

  3. In addition to target-task fine-tuning, the paper provides a no-fine-tuning synthetic recombination experiment and clearly states the limits of its theoretical analysis.

Critical weaknesses
  1. The paper does not compare with prior sparse-interaction or task-conditioned expert-routing world models such as SPARTAN, MoW, and PRISM-WM, so it remains unclear whether the gains come from structured dynamics or from the specific two-level gating design.

  2. The held-out compositional tasks are not specified in enough detail: §5.1 and Appendix E do not list the factors, their source tasks, or the excluded joint configurations, making it hard to distinguish compositional generalization from ordinary parameter transfer.

  3. The main results do not sufficiently separate dynamics reuse from better initialization: Fig. 4 allows 100K fine-tuning steps but does not state which modules are frozen or updated. Table A1(b) helps, but it lacks module definitions, splits, repetitions, prediction-error statistics, and mask-binarization details.

  4. Several experimental descriptions and numbers are inconsistent: Fig. 3 is described as held-out recombination in its caption but as in-distribution multitask performance in §5.2/Table A2, and the ablation numbers for Meta-World, ManiSkill, and MiniArcade differ between Appendix C.3 and Fig. A1.

Questions for authors
  1. Under the same NewT backbone, entity inputs, and training/planning budget, how does CWM compare with a representative sparse-attention or sparse-interaction mechanism when task MoE is retained, and what unique conclusions do the current ablations support relative to MoW, PRISM-WM, and SPARTAN?

  2. Can the authors provide a table for each Fig. 4 target task listing the composed factors, source tasks covering those factors, and joint configurations excluded during training, especially for walker-walk-incline and spinner-jump-four?

  3. During Fig. 4 fine-tuning, which modules are frozen, and can the authors provide a frozen-dynamics diagnostic plus full details of Table A1(b), including transition functions, splits, repetitions, prediction-error metrics, and mask binarization?

  4. Is Fig. 3 based on in-distribution or held-out compositional tasks, and are the ablation-number discrepancies due to different settings, versions, or textual errors?

7 comments
AuthorsOpenReview ↗

Response to Reviewer KrYn (Part 1/4)

(Part 1/4)

Thank you for the detailed comments and for pointing us to strong comparisons. We have added a PRISM-WM dynamics baseline within NewT, related CWM to SPARTAN and MoW through our existing controls, specified the source–target relationships and fine-tuning protocol, expanded the controlled recombination experiment, and corrected the reporting inconsistencies. MoW runs are in progress. We address each concern below.

W1: The paper does not compare with prior sparse-interaction or task-conditioned expert-routing world models such as SPARTAN, MoW, and PRISM-WM, so it remains unclear whether the gains come from structured dynamics or from the specific two-level gating design.

Q1: Under the same NewT backbone, entity inputs, and training/planning budget, how does CWM compare with a representative sparse-attention or sparse-interaction mechanism when task MoE is retained, and what unique conclusions do the current ablations support relative to MoW, PRISM-WM, and SPARTAN?

A1: Thank you for these valuable references; we have added all three to the Related Work section (Appendix A). We have also added a PRISM-WM baseline inside NewT to compare transition designs within the same control pipeline.

Comparison with PRISM-WM. We replace NewT's dynamics module with PRISM-WM's context-conditioned mixture of experts: its router uses the latent state, action, and task context to compute softmax mixture weights, and expert outputs are Gram–Schmidt orthogonalized and combined into a residual latent-state update. NewT's observation interface, reward/value heads, policy learning, and planner are kept unchanged, so this baseline isolates PRISM-WM's dynamics design within our pipeline. Both methods follow the same source-training protocol and receive 100K adaptation steps per target task (2 seeds); the results are also reported in Appendix C.1 and Table A3.

Target taskCWMPRISM-WM dynamics in NewT
walker-walk-incline0.940.86
cartpole-balance-two-poles0.920.92
spinner-jump-four0.810.75
ms-pick-cup0.830.69
ms-push-pear0.860.88
og-point-var10.950.94
pygame-point-maze-var40.790.73
pygame-reacher-hard0.870.70
Mean0.870.81

CWM is evaluated on the same two seeds as the PRISM-WM baseline.

CWM scores higher on six of the eight tasks, ties on cartpole-balance-two-poles, and is within 0.02 on ms-push-pear, with the largest gains on pygame-reacher-hard (+0.17) and ms-pick-cup (+0.14). PRISM-WM selects experts per transition, with each expert updating the whole latent state; CWM additionally selects which entities and interactions each update uses. Together with Fig. 4 and Fig. A1, this result supports the value of combining task-level routing with entity–interaction masking.

Relation to MoW. MoW selects dynamics experts from a learned task embedding and shares a backbone across tasks, which corresponds to CWM's task-level routing. Our closest existing control is Dense-OC + Task-MoE (Table A1(a)), which uses the same task router and experts as CWM with dense entity and interaction computation. CWM improves held-out adaptation from 0.73 to 0.88 over this control. We are also running MoW itself within the same pipeline and will post the results as soon as the runs finish.

Relation to SPARTAN. SPARTAN learns sparse, state-dependent interactions between object tokens through hard attention, which corresponds to CWM's interaction-level masking. Our closest variant is CWM without task routing, which keeps the entity and interaction masks. Removing task routing causes the largest drop in every domain (e.g., Meta-World 0.71 → 0.46, Fig. A1), showing that in the 200-task setting, interaction sparsity works best together with task-level expert selection.

What the ablations show. MoW structures which experts a task uses, while SPARTAN and PRISM-WM structure the transition itself. Each single-level counterpart in our experiments (Dense-OC + Task-MoE, CWM without task routing, and PRISM-WM dynamics) scores below the full model, so combining both levels, as CWM does, yields the strongest adaptation.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer KrYn (Part 2/4)

(Part 2/4)

W2: The held-out compositional tasks are not specified in enough detail: §5.1 and Appendix E do not list the factors, their source tasks, or the excluded joint configurations, making it hard to distinguish compositional generalization from ordinary parameter transfer.

Q2: Can the authors provide a table for each Fig. 4 target task listing the composed factors, source tasks covering those factors, and joint configurations excluded during training, especially for walker-walk-incline and spinner-jump-four?

A2: Thank you for pointing this out. We have updated Sec. 5.1 and Appendix D to specify the source tasks, shared elements, and excluded target configurations for each of the eight tasks in Fig. 4:

Target taskRelevant source tasksGeneralization tested
DMControl: walker-walk-inclinewalker-stand, walker-walk, walker-run, and their backward variantsTerrain transfer (Dynamics). The walker embodiment, controls, and walking objective are retained, while the ground changes from a horizontal surface to an incline.
DMControl: cartpole-balance-two-polesSingle-pole cartpole balance and swing-up tasksStructural extension (Objects & Dynamics). A second pole and hinge are attached to the first pole, extending the familiar system into a serial chain with additional coupling.
DMControl Extended: spinner-jump-fourspinner-jump, spinner-spin, spinner-spin-backwardComponent-level reconfiguration (Objects & Dynamics). One arm and its actuator are removed from the five-arm source morphology; the remaining components and the jumping objective are retained.
ManiSkill3: ms-pick-cupms-pick-mug, ms-pick-apple, ms-pick-can, and other picking tasksObject-instance generalization (Objects & Dynamics). The robot, control interface, and picking setup are shared, while the target uses a cup asset distinct from all source objects, including the mug.
ManiSkill3: ms-push-pearms-push-cube and the source object-picking tasksSkill transfer to a new object (Objects & Attributes). The familiar pushing objective and robot controls are applied to a pear that is absent from the source ManiSkill object set.
OGBench: og-point-var1og-point-arena, og-point-maze, og-point-bottleneck, og-point-circle, og-point-spiralSpatial recombination (Attributes). The point-agent dynamics and wall-construction primitives are shared, while the target introduces a new wall layout.
MiniArcade: pygame-point-maze-var4pygame-point-maze-var1, pygame-point-maze-var2, pygame-point-maze-var3Spatial recombination (Attributes & Dynamics). A new obstacle grid uses the same movement, collision, boundary, and local directional-force rules as the source environments.
MiniArcade: pygame-reacher-hardDMControl reacher-easy and reacher-hard, and mujoco-reacherCross-implementation transfer (Dynamics). The articulated reaching structure and objective are familiar, while the target uses a separate PyGame dynamics implementation and rendering. Both PyGame reacher variants are held out from source training.

The two tasks you highlighted. walker-walk-incline keeps the walker embodiment, joint controls, and walking objective from walker-stand, walker-walk, walker-run, and their backward variants, and changes only the ground, from horizontal to inclined. spinner-jump-four is assembled from a subset of the five-arm source spinner's components: one arm and its actuator are removed, and the jumping objective is retained.

The targets cover three forms of generalization: spatial recombination (the two maze tasks), structural reconfiguration (spinner and two-pole cartpole), and transfer to new objects, terrain, or dynamics implementations (the remaining four). Sec. 5.1 now names the form each target tests, and Appendix D gives the task-by-task source mapping and construction details. All target configurations are excluded from source demonstration pretraining and source-stage online training, so the adaptation results measure how well CWM applies source-task knowledge to these new settings.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer KrYn (Part 3/4)

(Part 3/4)

W3: The main results do not sufficiently separate dynamics reuse from better initialization: Fig. 4 allows 100K fine-tuning steps but does not state which modules are frozen or updated. Table A1(b) helps, but it lacks module definitions, splits, repetitions, prediction-error statistics, and mask-binarization details.

Q3: During Fig. 4 fine-tuning, which modules are frozen, and can the authors provide a frozen-dynamics diagnostic plus full details of Table A1(b), including transition functions, splits, repetitions, prediction-error metrics, and mask binarization?

A3: Thank you for asking for these details; we have added them to Sec. 5.1 and Appendix C.1.

Fig. 4: full adaptation. All trainable components are updated during the 100K-step adaptation stage: (i) the task router and task experts, (ii) the entity-local dynamics and entity-mask networks, and (iii) the pairwise interaction, interaction-mask, and state-update networks. The trainable encoder and reward, value, and policy heads are also updated, and pretrained feature extractors stay frozen as in source training. Fig. 4 therefore measures adaptation from a pretrained model, and Table A1(b) below measures reuse with all parameters frozen. The Fig. 4 caption now states this.

Table A1(b): frozen-dynamics prediction on unseen module combinations. This controlled experiment tests whether learned dynamics predict new combinations without any target-task parameter updates. It uses six ball partitions, each represented by an eight-dimensional state through a fixed VAE encoder, and a two-dimensional action. Four interaction modules have the following candidate directed supports:

ModuleDirected interactions
A1 → 4, 2 → 5
B3 → 4, 4 → 6
C2 → 6, 5 → 4
D1 → 5, 3 → 6

Transition functions. A target partition's next state is generated by its local dynamics plus nonlinear messages from active source partitions, where each interaction uses a fixed two-layer nonlinear network shared across all tasks in which that interaction occurs. Candidate edges are activated by a transition-dependent relative-distance condition, so the active support can vary within a task. Task context is the sum of fixed, randomly initialized embeddings of the active modules.

Splits. Training uses the four singleton tasks A, B, C, and D, with 20,000 transitions per task. Testing uses the six pairwise combinations A+B, A+C, A+D, B+C, B+D, and C+D, with 5,000 transitions per combination. The interaction functions are shared between training and testing, and only their joint activation changes. Transitions from these six combinations are held out from training and fine-tuning, and all learned parameters stay fixed during evaluation.

Metrics and mask binarization. We report one-step prediction MSE on the held-out combinations. For support recovery, AUPRC is computed from continuous mask scores against transition-level ground-truth edges, and F1 uses binarized masks M_hat[k,i] = 1 when m[k,i] >= 0.5, and 0 otherwise. We compare learned masks with oracle transition masks, dense masks, and shuffled masks that preserve row-wise sparsity while permuting source identities.

MaskOne-step MSE ↓
Oracle0.12
Learned (CWM)0.19
Shuffled0.39
Dense0.45

This serves as the frozen-dynamics diagnostic: trained only on singleton modules, the learned masks predict all six unseen combinations with less than half the error of dense masks, close to the oracle, and recover the interaction structure with AUPRC 0.84 and F1 0.81.

W4: Several experimental descriptions and numbers are inconsistent: Fig. 3 is described as held-out recombination in its caption but as in-distribution multitask performance in §5.2/Table A2, and the ablation numbers for Meta-World, ManiSkill, and MiniArcade differ between Appendix C.3 and Fig. A1.

Q4: Is Fig. 3 based on in-distribution or held-out compositional tasks, and are the ablation-number discrepancies due to different settings, versions, or textual errors?

A4: Thank you for pointing these out. Fig. 3 reports in-distribution multitask performance, and its caption now matches Sec. 5.2 and Table A2. The examples in Appendix C.3 had been written before we added a seed to the final results; we have updated all of them to match Fig. A1 (e.g., removing task routing: Meta-World 0.71 → 0.46, ManiSkill 0.77 → 0.50, MiniArcade 0.48 → 0.31). The qualitative conclusions remain the same.

(continued in the next comment)

Reviewer_AfeKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies large-scale multitask world models and argues that simply training a dense conditional model over many tasks does not guarantee compositional generalization. It frames the problem as concept accumulation and recombination: training environments expose only partial subsets of reusable latent concepts and dynamics, while test tasks may require recombining familiar factors in unseen configurations.

Critical strengths
  1. Clear and polished presentation. The paper is well organized, visually clean, and easy to follow. The motivation, method, and experimental setup are presented in a way that makes the overall story accessible.
  2. Reasonable method with convincing analysis. The proposed structure is well motivated, and the paper provides a coherent rationale for why task routing and dynamics masking should help multitask world models. The analysis is logically persuasive and supports the design choices effectively.
Critical weaknesses
  1. The evidence for compositional generalization is still limited. The paper lists held-out tasks and describes them as recombinations of seen factors, but it does not clearly specify which factors were seen during training, how they are recombined in each held-out task, and why success on these tasks should be attributed to compositional reuse rather than better transfer or regularization.
  2. The core components—task routing, object-centric representations, interaction masking, and sparse regularization—are all familiar ideas, and the broader goal of reusable mechanisms for compositional generalization is also not new.
  3. I also noticed a concrete inconsistency in the experimental reporting: for example, Figure A1 seems to show Meta-World changing from 0.71 to 0.46 when removing task routing, while Appendix C.3 describes this as 0.69 to 0.44.
Questions for authors

Please refer to weakness. Overall, I find the method reasonable and the paper well organized, but I currently view the core components as somewhat regular. I would be willing to raise my score if the author shows it revealed a previously under-characterized and reproducible failure mode of existing multitask world models, or demonstrated a genuinely new capability that current architectures struggle to achieve. More importantly, I would like to see evidence that better adaptation is not merely a result of improved sharing or regularization.

6 comments
AuthorsOpenReview ↗

Response to Reviewer AfeK (Part 1/3)

(Part 1/3)

Thank you for recognizing that the method is reasonable and the paper is well organized, and for stating so concretely what would strengthen it. In response, we have (i) specified the source–target relationship for every held-out task, (ii) added a generic-routing control that keeps CWM's module capacity, sharing, and routing while removing its entity–interaction structure, and (iii) updated the numbers in Appendix C.3. We address each point below.

W1: The evidence for compositional generalization is still limited. The paper lists held-out tasks and describes them as recombinations of seen factors, but it does not clearly specify which factors were seen during training, how they are recombined in each held-out task, and why success on these tasks should be attributed to compositional reuse rather than better transfer or regularization.

A1: Thank you for this question. We have updated Sec. 5.1 and Appendix D to specify, for each target, the source tasks, the shared elements, and what changes:

Target taskRelevant source tasksGeneralization tested
DMControl: walker-walk-inclinewalker-stand, walker-walk, walker-run, and their backward variantsTerrain transfer (Dynamics). The walker embodiment, controls, and walking objective are retained, while the ground changes from a horizontal surface to an incline.
DMControl: cartpole-balance-two-polesSingle-pole cartpole balance and swing-up tasksStructural extension (Objects & Dynamics). A second pole and hinge are attached to the first pole, extending the familiar system into a serial chain with additional coupling.
DMControl Extended: spinner-jump-fourspinner-jump, spinner-spin, spinner-spin-backwardComponent-level reconfiguration (Objects & Dynamics). One arm and its actuator are removed from the five-arm source morphology; the remaining components and the jumping objective are retained.
ManiSkill3: ms-pick-cupms-pick-mug, ms-pick-apple, ms-pick-can, and other picking tasksObject-instance generalization (Objects & Dynamics). The robot, control interface, and picking setup are shared, while the target uses a cup asset distinct from all source objects, including the mug.
ManiSkill3: ms-push-pearms-push-cube and the source object-picking tasksSkill transfer to a new object (Objects & Attributes). The familiar pushing objective and robot controls are applied to a pear that is absent from the source ManiSkill object set.
OGBench: og-point-var1og-point-arena, og-point-maze, og-point-bottleneck, og-point-circle, og-point-spiralSpatial recombination (Attributes). The point-agent dynamics and wall-construction primitives are shared, while the target introduces a new wall layout.
MiniArcade: pygame-point-maze-var4pygame-point-maze-var1, pygame-point-maze-var2, pygame-point-maze-var3Spatial recombination (Attributes & Dynamics). A new obstacle grid uses the same movement, collision, boundary, and local directional-force rules as the source environments.
MiniArcade: pygame-reacher-hardDMControl reacher-easy and reacher-hard, and mujoco-reacherCross-implementation transfer (Dynamics). The articulated reaching structure and objective are familiar, while the target uses a separate PyGame dynamics implementation and rendering. Both PyGame reacher variants are held out from source training.

Which factors are seen, and how they are recombined. The targets fall into three groups:

  • Spatial recombination (og-point-var1, pygame-point-maze-var4): familiar movement and obstacle-interaction rules in a new spatial layout.
  • Structural reconfiguration (spinner-jump-four, cartpole-balance-two-poles): familiar body components reassembled into a new morphology (one spinner arm removed; a second pole attached).
  • Transfer to new objects, terrain, or dynamics (ms-pick-cup, ms-push-pear, walker-walk-incline, pygame-reacher-hard): familiar embodiments and skills applied to new object instances, an incline, or a separate dynamics implementation.

Sec. 5.1 now names the form of generalization each target tests. All target configurations are excluded from source demonstration pretraining and source-stage online training.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer AfeK (Part 2/3)

(Part 2/3)

Why the gains reflect CWM's structure. Under the same 100K-step budget, CWM achieves higher final scores than NewT on all eight tasks, and removing any component lowers its final performance (Fig. 4). To isolate the role of CWM's structure from sharing and regularization, we added a generic-routing control that matches CWM's module capacity, sharing, and routing; CWM outperforms it on all eight tasks (see A3).

W2: The core components—task routing, object-centric representations, interaction masking, and sparse regularization—are all familiar ideas, and the broader goal of reusable mechanisms for compositional generalization is also not new.

A2: Thank you for this perspective. Task routing, object-centric representations, interaction masking, and sparsity are indeed well-established building blocks, and we see this as a strength of the design: each is well understood and scales well. CWM's contribution lies in how these building blocks are organized within a massively multitask world model.

Specifically, CWM separates three decisions that a dense conditional model entangles: which task expert to use (task routing), which entities to update (entity masks), and which directed interactions to pass (interaction masks), all driving shared dynamics modules. This organization makes both task selection and the structure of each state update explicit within one predictive model, which is what lets CWM scale to 200+ tasks while improving adaptation to held-out tasks.

To verify that this organization itself drives the gains, we added a generic-routing control (see A3) that keeps CWM's module capacity, parameter sharing, and routing capacity and removes only the entity-local and pairwise-interaction structure. CWM achieves higher final scores on all eight adaptation tasks, and the component ablations in Fig. 4 and Fig. A1 show how each part contributes.

Q3: I would be willing to raise my score if the author shows it revealed a previously under-characterized and reproducible failure mode of existing multitask world models, or demonstrated a genuinely new capability that current architectures struggle to achieve. More importantly, I would like to see evidence that better adaptation is not merely a result of improved sharing or regularization.

A3: Thank you for setting out these criteria so clearly. They gave us a concrete target, and we are glad to report evidence on all three points.

(1) Evidence beyond improved sharing or regularization. We evaluated a generic-routing control that matches CWM in the number and size of modules, parameter sharing, and routing capacity, and replaces only the entity-local and pairwise-interaction structure with generic routing. Both models follow the same source-training protocol and are adapted with the same 100K-step budget.

Target taskCWMControl with generic routing
walker-walk-incline0.940.73
cartpole-balance-two-poles0.920.80
spinner-jump-four0.810.65
ms-pick-cup0.830.69
ms-push-pear0.860.71
og-point-var10.950.76
pygame-point-maze-var40.790.71
pygame-reacher-hard0.870.62
Mean0.870.71

These results use two seeds per task, one fewer than the main experiments, to fit the rebuttal timeline; CWM is evaluated on the same two seeds as the control. We will add the remaining seed and uncertainty estimates in the revision.

CWM wins on all eight tasks, with a mean gain of 0.16, and the margin holds on the spatial and structural reconfiguration tasks: 0.95 vs. 0.76 on og-point-var1, 0.79 vs. 0.71 on pygame-point-maze-var4, and 0.81 vs. 0.65 on spinner-jump-four. Because the control has the same capacity, sharing, and routing, this gain comes from organizing computation around entities and their interactions.

The controlled recombination experiment (Table A1(b), Appendix C.1) isolates this effect further. There, shuffled masks keep exactly the same sparsity as the learned masks and only permute which sources are selected, so any regularization benefit of sparsity is matched. Learned masks reach a one-step MSE of 0.19, compared with 0.39 for shuffled masks and 0.45 for dense masks (oracle: 0.12). The improvement therefore comes from selecting the right interactions, which is precisely the structure CWM is designed to learn.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer AfeK (Part 3/3)

(Part 3/3)

(2) A reproducible failure mode of existing multitask world models. Our controls reveal a consistent pattern: adding modular capacity, entity representations, or task routing to a multitask world model, without transition-level support selection, lowers held-out adaptation. In Table A1(a), Dense-OC (0.69) and Dense-OC + Task-MoE (0.73) both score below dense NewT (0.80), and the generic-routing control above (0.71) follows the same pattern. In Fig. 4, removing any single CWM component brings it below NewT on four of the eight tasks. CWM, which combines routing with entity and interaction masking, reaches 0.88. This failure mode becomes visible only under held-out adaptation: richer modular architectures can transfer worse than a dense model unless they also learn which entities and interactions each transition depends on.

For more details, please kindly refer to Appendix C.1 and Table A3. We hope this evidence meets your criteria, and we would be very grateful if you would consider raising your score. We would be happy to provide any further analysis.

W3: I also noticed a concrete inconsistency in the experimental reporting: for example, Figure A1 seems to show Meta-World changing from 0.71 to 0.46 when removing task routing, while Appendix C.3 describes this as 0.69 to 0.44.

A4: Thank you so much for spotting this! Fig. A1 shows the final results: Meta-World drops from 0.71 to 0.46 when task routing is removed. The examples in Appendix C.3 had been written before we added a seed to the final results, and we have now updated every number in C.3 to match Fig. A1 (e.g., ManiSkill 0.77 → 0.50 and MiniArcade 0.48 → 0.31). The conclusions are the same: the full model performs best in all ten domains, and removing task routing causes the largest drop in each.

Thank you again for the constructive feedback. Your criteria led directly to the generic-routing control and to a clearer account of the failure mode and capability that CWM addresses. We hope these results address your concerns, and we would welcome any further questions.

Reviewer_uPqfOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CWM, which combines task expert routing with entity and interaction masks in a shared latent world model. An idealized analysis characterizes predictive source recovery and illustrates additional conditions for recombination. Experiments on over 200 tasks across 10 domains report an aggregate score of 0.51 versus 0.38 for NewT, together with improved adaptation to eight unseen task configurations using 100K interaction steps.

Critical strengths
  1. This paper has a clear motivation.. Sections 1 and 2 highlight that strong performance across training tasks does not guarantee adaptation to new combinations of familiar factors. Viewing each environment as revealing part of a shared concept library provides a clear rationale for learning reusable dynamics across environments.

  2. This paper features a clearly organized methodological framework. Section 4.2 separates task expert selection, local entity updates, and directed interaction messages through explicit routing and gating operations (Equations 11 to 14). Shared local and pairwise modules provide reusable computation across tasks, and integration with the NewT planning pipeline makes the design directly applicable to control.

  3. This paper verifies the effectiveness of CWM via various experiments. Table A2 reports the highest multitask score in all 10 domains, with the aggregate average increasing from 0.38 to 0.51. Improvements span manipulation, locomotion, and games, providing meaningful evidence that the proposed structure benefits a heterogeneous task collection.

Critical weaknesses
  1. The compositional task split cannot be verified. Section 5.1 and Appendix E assert that target factors occur during pretraining, but omit the source task list, factor definitions, and construction rules for the eight targets in Figure 4. Without a mapping from target factors to source tasks, the claimed coverage of familiar mechanisms and exclusion of their joint configurations remain unverified.

  2. The NewT adaptation comparison is ambiguous. Figure 4 presents NewT as a horizontal dashed reference, although Section 5.1 states that each pretrained model receives 100K adaptation steps. Its meaning as an initial or final adapted score is unspecified.

Questions for authors
  1. For each target in Figure 4, which source tasks cover its constituent factors, and what construction rule excludes the joint target configuration from demonstrations and pretraining data?

  2. What does the dashed NewT reference in Figure 4 represent, and how does NewT perform throughout the same 100K adaptation budget and update protocol?

4 comments
AuthorsOpenReview ↗

Response to Reviewer uPqf (Part 1/2)

(Part 1/2)

Thank you for recognizing the clear motivation, structured design, and broad empirical benefits of CWM! Below, we clarify the source–target relationships for the eight adaptation tasks and the NewT reference in Fig. 4. We have updated Sec. 5.1 and Appendix D with the task details, and the Fig. 4 caption now states that the dashed line shows NewT's performance after 100K adaptation steps.

W1: The compositional task split cannot be verified. Section 5.1 and Appendix E assert that target factors occur during pretraining, but omit the source task list, factor definitions, and construction rules for the eight targets in Figure 4. Without a mapping from target factors to source tasks, the claimed coverage of familiar mechanisms and exclusion of their joint configurations remain unverified.

Q1: For each target in Figure 4, which source tasks cover its constituent factors, and what construction rule excludes the joint target configuration from demonstrations and pretraining data?

A1: Thank you for pointing this out. We have updated Sec. 5.1 and Appendix D to list, for each target, the relevant source tasks, the shared elements, and the target-specific changes. The eight targets test different forms of generalization:

Target taskRelevant source tasksGeneralization tested
DMControl: walker-walk-inclinewalker-stand, walker-walk, walker-run, and their backward variantsTerrain transfer (Dynamics). The walker embodiment, controls, and walking objective are retained, while the ground changes from a horizontal surface to an incline.
DMControl: cartpole-balance-two-polesSingle-pole cartpole balance and swing-up tasksStructural extension (Objects & Dynamics). A second pole and hinge are attached to the first pole, extending the familiar system into a serial chain with additional coupling.
DMControl Extended: spinner-jump-fourspinner-jump, spinner-spin, spinner-spin-backwardComponent-level reconfiguration (Objects & Dynamics). One arm and its actuator are removed from the five-arm source morphology; the remaining components and the jumping objective are retained.
ManiSkill3: ms-pick-cupms-pick-mug, ms-pick-apple, ms-pick-can, and other picking tasksObject-instance generalization (Objects & Dynamics). The robot, control interface, and picking setup are shared, while the target uses a cup asset distinct from all source objects, including the mug.
ManiSkill3: ms-push-pearms-push-cube and the source object-picking tasksSkill transfer to a new object (Objects & Attributes). The familiar pushing objective and robot controls are applied to a pear that is absent from the source ManiSkill object set.
OGBench: og-point-var1og-point-arena, og-point-maze, og-point-bottleneck, og-point-circle, og-point-spiralSpatial recombination (Attributes). The point-agent dynamics and wall-construction primitives are shared, while the target introduces a new wall layout.
MiniArcade: pygame-point-maze-var4pygame-point-maze-var1, pygame-point-maze-var2, pygame-point-maze-var3Spatial recombination (Attributes & Dynamics). A new obstacle grid uses the same movement, collision, boundary, and local directional-force rules as the source environments.
MiniArcade: pygame-reacher-hardDMControl reacher-easy and reacher-hard, and mujoco-reacherCross-implementation transfer (Dynamics). The articulated reaching structure and objective are familiar, while the target uses a separate PyGame dynamics implementation and rendering. Both PyGame reacher variants are held out from source training.

Construction rule. All eight target tasks are excluded from both source demonstration pretraining and source-stage online training. The evaluation therefore covers unseen spatial configurations, body structures, object instances, terrain, and dynamics implementations, and Fig. 4 shows that CWM transfers knowledge from the source tasks and adapts effectively across all of these settings.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer uPqf (Part 2/2)

(Part 2/2)

W2: The NewT adaptation comparison is ambiguous. Figure 4 presents NewT as a horizontal dashed reference, although Section 5.1 states that each pretrained model receives 100K adaptation steps. Its meaning as an initial or final adapted score is unspecified.

Q2: What does the dashed NewT reference in Figure 4 represent, and how does NewT perform throughout the same 100K adaptation budget and update protocol?

A2: Thank you for noting this. The dashed line shows NewT's performance after 100K adaptation steps, under the same budget and update protocol as CWM and its ablations. We plot this final score as a horizontal reference to keep the learning curves readable, and the caption now states this explicitly.

This comparison also shows how CWM's components work together. The full model achieves higher final mean scores than NewT on all eight tasks, and removing any component lowers performance. On walker-walk-incline, cartpole-balance-two-poles, og-point-var1, and pygame-reacher-hard, all three ablations fall below the adapted NewT baseline while the full model exceeds it, so the gains come from the components acting together.

Thank you again for the constructive feedback, which helped us make the evaluation setup and comparisons clearer!

Thank you for the author's rebuttal, which mostly resolved my concern. Since I've already given a positive score, I'll keep the score.

Thank you so much for acknowledging that our rebuttal has addressed your concerns and for upholding your positive support!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers and ACs for their careful feedback. We have addressed their questions in the individual responses and revised the paper and appendix (with changes marked in blue). Our work introduces CWM, a multitask world model framework that integrates task-level expert routing, entity-local updates, and directed interaction masking to organize shared dynamics. Across 200 tasks in 10 domains, CWM improves multitask performance and subsequent adaptation to held-out settings. The added controls and ablations support the value of this organization beyond generic sharing and routing, while the controlled experiment tests reuse of learned dynamics on unseen module combinations without parameter updates. Below is a summary of the main updates.

Additional comparisons and analysis

  • Comparison with prior modular world models. We added PRISM-WM dynamics within NewT and a comparison with MoW. Under the same 100K-step adaptation budget, CWM achieves higher mean scores than both baselines (0.871 vs. 0.809 and 0.773). [KrYn A1 and MoW follow-up, Table A3]

  • Contribution beyond generic sharing. We added a generic-routing control that retains module capacity, sharing, and routing capacity. CWM scores higher on all 8 tasks (mean 0.871 vs. 0.709), supporting the value of entity–interaction organization. [AfeK A2–A3; Appendix C.1, Table A3]

  • Controlled dynamics reuse. We expanded the module definitions, splits, sample counts, and mask-binarization details for Table A1(b). Without target-task updates, learned masks predict unseen module combinations with MSE 0.19, versus 0.39 for shuffled masks and 0.45 for dense masks. [KrYn A3; Appendix C.1, Table A1(b)]

Evaluation clarifications and corrections

  • Source–target relationships. We now list the source tasks, shared elements, and excluded configurations for all eight targets, distinguishing spatial and structural reconfiguration from broader transfer settings. [uPqf A1, AfeK A1, KrYn A2; Sec. 5.1, Appendix D]

  • Adaptation protocol. We clarify that Fig. 4 updates all trainable components and that the NewT dashed line shows its score after 100K adaptation steps. This distinguishes benchmark adaptation from the controlled experiment without parameter updates. [uPqf A2, KrYn A3; Sec. 5.1, Fig. 4 caption]

  • Reporting consistency. We corrected Fig. 3’s caption to identify in-distribution evaluation and updated the stale numerical examples in Appendix C.3 to match Fig. A1. The qualitative conclusions are unchanged. [AfeK A4, KrYn A4; Fig. 3 caption, Appendix C.3]

Thank you again for your time and constructive suggestions. We remain available for any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.