acceptodds
Under review as a conference paper at ICLR 2027

MOTO-WM: Learning Multiple Tasks Online with One World Model

30 views

Abstract

World models support model-based reinforcement learning through planning and imagined policy optimization. Extending them to multiple tasks requires sharing predictive structure while preserving differences between task objectives. Through empirical and representation analyses of independently trained DreamerV3 agents, we find shared physical structure in recurrent states but objective-dependent reward and policy readouts. Motivated by this observation, we propose Multi-task Online Training with One World Model (MOTO-WM), which shares a world model and actor–critic across tasks while conditioning feedback and control on task identity. A unified action interface supports different action dimensions. MOTO-WM learns online from scratch without demonstrations, pre-collected data, or pretrained models. On 20 visual DeepMind Control tasks, a single 10.55M-parameter model outperforms the evaluated model-based baselines at 500K steps per task with only 1.48 GPU-days of training. Meta-World results, ablations, and controlled task interventions further validate the method and its task-conditioned behavior. Essential scripts and checkpoint are provided in the supplementary material.

Primary area:
reinforcement learning
Keywords:
Model-based reinforcement learning, World models, Multi-task reinforcement learning, Task conditioning, Visual control

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Ch3HOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the problem of multitask online RL using a model-based method as backbone learning algorithm. This is a largely empirical study that shows that, on two task sets (20 DMControl tasks and 10 Meta-World tasks), a DreamerV3 model trained on all tasks from that set is more data-efficient than single-task DreamerV3 agents when measured on environment steps per task. The authors ablate various common multi-task design choices such as task conditioning, and backpropping reward info back into the world model representation. The paper also includes analysis that shows that tasks that are highly similar (e.g. stand, walk, run with the same embodiment) also lead to similar representations. The paper concludes that sharing a representation across tasks can be beneficial.

Critical strengths

Overall, the paper was easy to read and the figures and visualizations were helpful in understanding the eperimental setup and results. It's nice to see that the authors are largely able to reproduce results and findings from other related work in their own multi-task setting.

Critical weaknesses

That said, I do not believe that the paper is ready for publication. There's several major issues:

  • First of all, the central finding in the paper is that training on multiple related tasks improves per-task data efficiency. This is not particularly surprising, and I do not believe that it is a new result either. Plenty of exisiting work has demonstrated this.
  • The specific design choices considered are all fairly standard multi-task design choices that have been used extensively in prior work. For example, task conditioning dates back to the original Meta-World paper (although they used one-hot encoding) and in the years since there's been many papers adopting similar task conditioning mechanisms e.g. TD-MPC2 none of which are cited in the relevant section (5.3); this makes each design choice sound more novel than it actually is. It seems to me that, besides the choice of backbone wm, the one design choice thst differs between tdmpc2 and the proposed method is conditioning only the prediction heads on task ID but the relevant ablation seems to indicate that this actually does not matter much (tab 3).
  • Using a shared world model / representation across tasks and then task-specific reward heads on top is not a new idea. Other papers such as Plan2Explore and URLB have explored this as well although not in a online RL setting. I believe it's true that the specific combination of design choices have not been tested in a from scratch online rl setting with wms but that in itself has pretty marginal value over existing work.
Questions for authors

I would appreciate if the authors can please respond to my comments in the "weaknesses" section above.

Reviewer_8MFWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MOTO-WM, which applies Dreamer to multi-task setting where different tasks share the same dynamics backbone while separate feedback readouts. The deterministic latent state similarity is investigated, demonstrating recurrent state with similar structure but leads to different behavior. The empirical results show improvement against other model-based and model-free baselines on DMC20 tasks, while stay on par with baselines on MT-10 tasks.

Critical strengths
  1. The descriptions in the paper are clear and easy to follow. The investigation on Dreamer and analysis on task dynamics similarity is interesting.
Critical weaknesses
  1. Novelty is overall limited. It is a common hypothesis in multi-task RL setting that different tasks share similar dynamics while differs in reward function. All this paper do is sharing Dreamer backbone between tasks, which preserve original design with no modifications, and add a task identification at reward/value feedbacks, which seems incremental to me. Moreover, action is padded to the largest length among tasks, which raises my concern on scalability. Action dimensions among different tasks have different meanings, the padding strategy would lead to unwilling architecture bias.

  2. Missing baselines. More multitask RL baselines should be taken into account. The DreamerV3 and HarmonyDream are not multi-task settings. Previous works like Newt[1] and MoW[2] should be included as baselines.

[1] Learning Massively Multitask World Models for Continuous Control

[2] MIXTURE-OF-WORLD MODELS: SCALING MULTI-TASK REINFORCEMENT LEARNING WITH MODULAR LATENT DYNAMICS

Questions for authors
  1. This paper provides an analysis on the similarity between feature embeded by Dreamer RSSM, is this because Dreamer design already captures similarity between different tasks? I mean, does the similarity come from architecture bias?

  2. Related to W1, why not learn a latent action space across different tasks through VAEs or other encoders? The padding+linear encoder seems vulnerable.

Reviewer_qhzqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes MOTO-WM, a compact framework for multi-task model-based RL in which a single shared world model and actor–critic learn multiple tasks online from scratch, requiring no demonstrations, pre-collected data, or pretrained models. On 20 visual DMC tasks, a single model with 10.55M parameters outperforms the evaluated model-based baselines at 500K steps per task, requiring only 1.48 GPU-days for training. Ablation designs (SharedDreamer, SharedEmbedding, Feedback StopGrad, Full Path) systematically examine the conditioning scope, with a relatively complete experimental design.

Critical strengths

The paper transforms the contribution from "another multi-task RL variant" into a validated representational claim (share the predictive backbone, condition only the readouts). This is what makes the architecture choices interpretable and the approach potentially transferable to other model-based RL settings, rather than a result that works only by tuning.

Critical weaknesses

The capability advantage over multi-task baselines is thin and inconsistent, the real gain is efficiency only.

Questions for authors

Can this method be extended to task families with heterogeneous action spaces (hybrid discrete/continuous)? In what scenarios will the unified action interface and shared backbone fail? On MT10, MOTO-WM underperforms single-task DreamerV3. Under what conditions does multi-task sharing degrade performance, and how is this cost related to the degree of task divergence?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.