- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies the problem of multitask online RL using a model-based method as backbone learning algorithm. This is a largely empirical study that shows that, on two task sets (20 DMControl tasks and 10 Meta-World tasks), a DreamerV3 model trained on all tasks from that set is more data-efficient than single-task DreamerV3 agents when measured on environment steps per task. The authors ablate various common multi-task design choices such as task conditioning, and backpropping reward info back into the world model representation. The paper also includes analysis that shows that tasks that are highly similar (e.g. stand, walk, run with the same embodiment) also lead to similar representations. The paper concludes that sharing a representation across tasks can be beneficial.
Critical strengths
Overall, the paper was easy to read and the figures and visualizations were helpful in understanding the eperimental setup and results. It's nice to see that the authors are largely able to reproduce results and findings from other related work in their own multi-task setting.
Critical weaknesses
That said, I do not believe that the paper is ready for publication. There's several major issues:
- First of all, the central finding in the paper is that training on multiple related tasks improves per-task data efficiency. This is not particularly surprising, and I do not believe that it is a new result either. Plenty of exisiting work has demonstrated this.
- The specific design choices considered are all fairly standard multi-task design choices that have been used extensively in prior work. For example, task conditioning dates back to the original Meta-World paper (although they used one-hot encoding) and in the years since there's been many papers adopting similar task conditioning mechanisms e.g. TD-MPC2 none of which are cited in the relevant section (5.3); this makes each design choice sound more novel than it actually is. It seems to me that, besides the choice of backbone wm, the one design choice thst differs between tdmpc2 and the proposed method is conditioning only the prediction heads on task ID but the relevant ablation seems to indicate that this actually does not matter much (tab 3).
- Using a shared world model / representation across tasks and then task-specific reward heads on top is not a new idea. Other papers such as Plan2Explore and URLB have explored this as well although not in a online RL setting. I believe it's true that the specific combination of design choices have not been tested in a from scratch online rl setting with wms but that in itself has pretty marginal value over existing work.
Questions for authors
I would appreciate if the authors can please respond to my comments in the "weaknesses" section above.