- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a method for consolidating independently trained specialist world models into one shared, task-conditioned world model. The method first freezes each specialist encoder, constructs a task-specific codebook, and caches its latent features and code assignments. It then maps all specialist features and codebooks into the latent space of a reference specialist using a similarity transformation. Finally, a shared student encoder and predictor are trained using the aligned specialist supervision. The paper studies continuous, hybrid, and discrete supervision. During inference, the shared model receives the observation, action, goal image, and task ID, and uses latent predictions with MPC for closed-loop control. Experiments are conducted on two-, three-, and six-task collections and compare the consolidated models with separately trained specialists, single-task students, and a native joint-training baseline.
Critical strengths
The paper studies an important problem: jointly training one world model on multiple tasks can cause substantial task interference. Consolidating existing specialists is a reasonable alternative to training a multi-task model completely from scratch. The extract-align-consolidate pipeline is conceptually straightforward.
Critical weaknesses
Figure 1 shows that the aligned representation has greater task separability than both unaligned concatenation and native joint training. This suggests that alignment preserves or increases task-specific clustering rather than producing a common task-invariant representation. This raises two possibilities:
- If latent trajectories remain within separate task clusters, the extent of cross-task representation sharing is unclear. In this case, the relative rotation and placement of the clusters may have limited effect, particularly because the predictor already receives the task ID.
- If latent trajectories or code assignments enter another task’s cluster, the model may treat states from different tasks as neighbors. Such cross-task matching is meaningful only if those states have compatible physical meaning and dynamics.
Could the authors visualize the ground-truth and predicted latent trajectories before and after alignment, report the task origin of selected codebook entries, and clarify what latent-space mechanism is responsible for the observed performance improvement?
Questions for authors
-
Since the model already knows the task ID during inference, why is consolidation necessary instead of selecting the corresponding specialist? Why not train a task-ID-free model?
-
Why does feature alignment improve performance when the aligned latent space still clearly separates the tasks?
-
Please provide latent trajectories before and after alignment. Do trajectories remain within their own task clusters or interact with other task clusters?
-
Since the loss does not explicitly align cross-task transitions, what mechanism makes the aligned representations useful to the shared predictor?
-
How often does a latent select a code originating from another task? If this occurs, how do the authors verify that it is not selecting an incorrect task representation?
-
Why is a codebook needed instead of directly aligning the continuous features, given that M4 performs as well as M2?
-
Why is the codebook-size ablation only performed on single-task models? Please evaluate codebook size in the multi-task consolidation setting.
-
Is there a controlled ablation that independently measures the contribution of specialist-encoder supervision?
-
Please clarify the definitions of M0–M5, why M1 was not run, and why several variants change multiple factors simultaneously.