- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CWM, which augments a NewT/TD-MPC2 latent dynamics model with language-conditioned task-expert routing, entity gating, and interaction-message gating. The aim is to improve cross-task sharing and compositional generalization. Experiments cover more than 200 tasks across 10 domains in MM-Bench. The paper evaluates both in-distribution multitask learning and adaptation to held-out task compositions.
Critical strengths
-
The empirical coverage is broad, and CWM consistently improves over NewT across many domains, demonstrating practical value for large-scale multitask control.
-
The ablations are reasonably thorough: Dense-OC and Dense-OC + Task-MoE control for entity representations and expert capacity, supporting the effectiveness of the gating mechanisms.
-
In addition to target-task fine-tuning, the paper provides a no-fine-tuning synthetic recombination experiment and clearly states the limits of its theoretical analysis.
Critical weaknesses
-
The paper does not compare with prior sparse-interaction or task-conditioned expert-routing world models such as SPARTAN, MoW, and PRISM-WM, so it remains unclear whether the gains come from structured dynamics or from the specific two-level gating design.
-
The held-out compositional tasks are not specified in enough detail: §5.1 and Appendix E do not list the factors, their source tasks, or the excluded joint configurations, making it hard to distinguish compositional generalization from ordinary parameter transfer.
-
The main results do not sufficiently separate dynamics reuse from better initialization: Fig. 4 allows 100K fine-tuning steps but does not state which modules are frozen or updated. Table A1(b) helps, but it lacks module definitions, splits, repetitions, prediction-error statistics, and mask-binarization details.
-
Several experimental descriptions and numbers are inconsistent: Fig. 3 is described as held-out recombination in its caption but as in-distribution multitask performance in §5.2/Table A2, and the ablation numbers for Meta-World, ManiSkill, and MiniArcade differ between Appendix C.3 and Fig. A1.
Questions for authors
-
Under the same NewT backbone, entity inputs, and training/planning budget, how does CWM compare with a representative sparse-attention or sparse-interaction mechanism when task MoE is retained, and what unique conclusions do the current ablations support relative to MoW, PRISM-WM, and SPARTAN?
-
Can the authors provide a table for each Fig. 4 target task listing the composed factors, source tasks covering those factors, and joint configurations excluded during training, especially for walker-walk-incline and spinner-jump-four?
-
During Fig. 4 fine-tuning, which modules are frozen, and can the authors provide a frozen-dynamics diagnostic plus full details of Table A1(b), including transition functions, splits, repetitions, prediction-error metrics, and mask binarization?
-
Is Fig. 3 based on in-distribution or held-out compositional tasks, and are the ablation-number discrepancies due to different settings, versions, or textual errors?
7 comments
Response to Reviewer KrYn (Part 2/4)
(Part 2/4)
W2: The held-out compositional tasks are not specified in enough detail: §5.1 and Appendix E do not list the factors, their source tasks, or the excluded joint configurations, making it hard to distinguish compositional generalization from ordinary parameter transfer.
Q2: Can the authors provide a table for each Fig. 4 target task listing the composed factors, source tasks covering those factors, and joint configurations excluded during training, especially for walker-walk-incline and spinner-jump-four?
A2: Thank you for pointing this out. We have updated Sec. 5.1 and Appendix D to specify the source tasks, shared elements, and excluded target configurations for each of the eight tasks in Fig. 4:
| Target task | Relevant source tasks | Generalization tested |
|---|---|---|
DMControl: walker-walk-incline | walker-stand, walker-walk, walker-run, and their backward variants | Terrain transfer (Dynamics). The walker embodiment, controls, and walking objective are retained, while the ground changes from a horizontal surface to an incline. |
DMControl: cartpole-balance-two-poles | Single-pole cartpole balance and swing-up tasks | Structural extension (Objects & Dynamics). A second pole and hinge are attached to the first pole, extending the familiar system into a serial chain with additional coupling. |
DMControl Extended: spinner-jump-four | spinner-jump, spinner-spin, spinner-spin-backward | Component-level reconfiguration (Objects & Dynamics). One arm and its actuator are removed from the five-arm source morphology; the remaining components and the jumping objective are retained. |
ManiSkill3: ms-pick-cup | ms-pick-mug, ms-pick-apple, ms-pick-can, and other picking tasks | Object-instance generalization (Objects & Dynamics). The robot, control interface, and picking setup are shared, while the target uses a cup asset distinct from all source objects, including the mug. |
ManiSkill3: ms-push-pear | ms-push-cube and the source object-picking tasks | Skill transfer to a new object (Objects & Attributes). The familiar pushing objective and robot controls are applied to a pear that is absent from the source ManiSkill object set. |
OGBench: og-point-var1 | og-point-arena, og-point-maze, og-point-bottleneck, og-point-circle, og-point-spiral | Spatial recombination (Attributes). The point-agent dynamics and wall-construction primitives are shared, while the target introduces a new wall layout. |
MiniArcade: pygame-point-maze-var4 | pygame-point-maze-var1, pygame-point-maze-var2, pygame-point-maze-var3 | Spatial recombination (Attributes & Dynamics). A new obstacle grid uses the same movement, collision, boundary, and local directional-force rules as the source environments. |
MiniArcade: pygame-reacher-hard | DMControl reacher-easy and reacher-hard, and mujoco-reacher | Cross-implementation transfer (Dynamics). The articulated reaching structure and objective are familiar, while the target uses a separate PyGame dynamics implementation and rendering. Both PyGame reacher variants are held out from source training. |
The two tasks you highlighted. walker-walk-incline keeps the walker embodiment, joint controls, and walking objective from walker-stand, walker-walk, walker-run, and their backward variants, and changes only the ground, from horizontal to inclined. spinner-jump-four is assembled from a subset of the five-arm source spinner's components: one arm and its actuator are removed, and the jumping objective is retained.
The targets cover three forms of generalization: spatial recombination (the two maze tasks), structural reconfiguration (spinner and two-pole cartpole), and transfer to new objects, terrain, or dynamics implementations (the remaining four). Sec. 5.1 now names the form each target tests, and Appendix D gives the task-by-task source mapping and construction details. All target configurations are excluded from source demonstration pretraining and source-stage online training, so the adaptation results measure how well CWM applies source-task knowledge to these new settings.
(continued in the next comment)
Response to Reviewer KrYn (Part 3/4)
(Part 3/4)
W3: The main results do not sufficiently separate dynamics reuse from better initialization: Fig. 4 allows 100K fine-tuning steps but does not state which modules are frozen or updated. Table A1(b) helps, but it lacks module definitions, splits, repetitions, prediction-error statistics, and mask-binarization details.
Q3: During Fig. 4 fine-tuning, which modules are frozen, and can the authors provide a frozen-dynamics diagnostic plus full details of Table A1(b), including transition functions, splits, repetitions, prediction-error metrics, and mask binarization?
A3: Thank you for asking for these details; we have added them to Sec. 5.1 and Appendix C.1.
Fig. 4: full adaptation. All trainable components are updated during the 100K-step adaptation stage: (i) the task router and task experts, (ii) the entity-local dynamics and entity-mask networks, and (iii) the pairwise interaction, interaction-mask, and state-update networks. The trainable encoder and reward, value, and policy heads are also updated, and pretrained feature extractors stay frozen as in source training. Fig. 4 therefore measures adaptation from a pretrained model, and Table A1(b) below measures reuse with all parameters frozen. The Fig. 4 caption now states this.
Table A1(b): frozen-dynamics prediction on unseen module combinations. This controlled experiment tests whether learned dynamics predict new combinations without any target-task parameter updates. It uses six ball partitions, each represented by an eight-dimensional state through a fixed VAE encoder, and a two-dimensional action. Four interaction modules have the following candidate directed supports:
| Module | Directed interactions |
|---|---|
| A | 1 → 4, 2 → 5 |
| B | 3 → 4, 4 → 6 |
| C | 2 → 6, 5 → 4 |
| D | 1 → 5, 3 → 6 |
Transition functions. A target partition's next state is generated by its local dynamics plus nonlinear messages from active source partitions, where each interaction uses a fixed two-layer nonlinear network shared across all tasks in which that interaction occurs. Candidate edges are activated by a transition-dependent relative-distance condition, so the active support can vary within a task. Task context is the sum of fixed, randomly initialized embeddings of the active modules.
Splits. Training uses the four singleton tasks A, B, C, and D, with 20,000 transitions per task. Testing uses the six pairwise combinations A+B, A+C, A+D, B+C, B+D, and C+D, with 5,000 transitions per combination. The interaction functions are shared between training and testing, and only their joint activation changes. Transitions from these six combinations are held out from training and fine-tuning, and all learned parameters stay fixed during evaluation.
Metrics and mask binarization. We report one-step prediction MSE on the held-out combinations. For support recovery, AUPRC is computed from continuous mask scores against transition-level ground-truth edges, and F1 uses binarized masks M_hat[k,i] = 1 when m[k,i] >= 0.5, and 0 otherwise. We compare learned masks with oracle transition masks, dense masks, and shuffled masks that preserve row-wise sparsity while permuting source identities.
| Mask | One-step MSE ↓ |
|---|---|
| Oracle | 0.12 |
| Learned (CWM) | 0.19 |
| Shuffled | 0.39 |
| Dense | 0.45 |
This serves as the frozen-dynamics diagnostic: trained only on singleton modules, the learned masks predict all six unseen combinations with less than half the error of dense masks, close to the oracle, and recover the interaction structure with AUPRC 0.84 and F1 0.81.
W4: Several experimental descriptions and numbers are inconsistent: Fig. 3 is described as held-out recombination in its caption but as in-distribution multitask performance in §5.2/Table A2, and the ablation numbers for Meta-World, ManiSkill, and MiniArcade differ between Appendix C.3 and Fig. A1.
Q4: Is Fig. 3 based on in-distribution or held-out compositional tasks, and are the ablation-number discrepancies due to different settings, versions, or textual errors?
A4: Thank you for pointing these out. Fig. 3 reports in-distribution multitask performance, and its caption now matches Sec. 5.2 and Table A2. The examples in Appendix C.3 had been written before we added a seed to the final results; we have updated all of them to match Fig. A1 (e.g., removing task routing: Meta-World 0.71 → 0.46, ManiSkill 0.77 → 0.50, MiniArcade 0.48 → 0.31). The qualitative conclusions remain the same.
(continued in the next comment)
Response to Reviewer KrYn (Part 1/4)
(Part 1/4)
Thank you for the detailed comments and for pointing us to strong comparisons. We have added a PRISM-WM dynamics baseline within NewT, related CWM to SPARTAN and MoW through our existing controls, specified the source–target relationships and fine-tuning protocol, expanded the controlled recombination experiment, and corrected the reporting inconsistencies. MoW runs are in progress. We address each concern below.
A1: Thank you for these valuable references; we have added all three to the Related Work section (Appendix A). We have also added a PRISM-WM baseline inside NewT to compare transition designs within the same control pipeline.
Comparison with PRISM-WM. We replace NewT's dynamics module with PRISM-WM's context-conditioned mixture of experts: its router uses the latent state, action, and task context to compute softmax mixture weights, and expert outputs are Gram–Schmidt orthogonalized and combined into a residual latent-state update. NewT's observation interface, reward/value heads, policy learning, and planner are kept unchanged, so this baseline isolates PRISM-WM's dynamics design within our pipeline. Both methods follow the same source-training protocol and receive 100K adaptation steps per target task (2 seeds); the results are also reported in Appendix C.1 and Table A3.
walker-walk-inclinecartpole-balance-two-polesspinner-jump-fourms-pick-cupms-push-pearog-point-var1pygame-point-maze-var4pygame-reacher-hardCWM is evaluated on the same two seeds as the PRISM-WM baseline.
CWM scores higher on six of the eight tasks, ties on
cartpole-balance-two-poles, and is within 0.02 onms-push-pear, with the largest gains onpygame-reacher-hard(+0.17) andms-pick-cup(+0.14). PRISM-WM selects experts per transition, with each expert updating the whole latent state; CWM additionally selects which entities and interactions each update uses. Together with Fig. 4 and Fig. A1, this result supports the value of combining task-level routing with entity–interaction masking.Relation to MoW. MoW selects dynamics experts from a learned task embedding and shares a backbone across tasks, which corresponds to CWM's task-level routing. Our closest existing control is Dense-OC + Task-MoE (Table A1(a)), which uses the same task router and experts as CWM with dense entity and interaction computation. CWM improves held-out adaptation from 0.73 to 0.88 over this control. We are also running MoW itself within the same pipeline and will post the results as soon as the runs finish.
Relation to SPARTAN. SPARTAN learns sparse, state-dependent interactions between object tokens through hard attention, which corresponds to CWM's interaction-level masking. Our closest variant is CWM without task routing, which keeps the entity and interaction masks. Removing task routing causes the largest drop in every domain (e.g., Meta-World 0.71 → 0.46, Fig. A1), showing that in the 200-task setting, interaction sparsity works best together with task-level expert selection.
What the ablations show. MoW structures which experts a task uses, while SPARTAN and PRISM-WM structure the transition itself. Each single-level counterpart in our experiments (Dense-OC + Task-MoE, CWM without task routing, and PRISM-WM dynamics) scores below the full model, so combining both levels, as CWM does, yields the strongest adaptation.
(continued in the next comment)