- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes BRICKS-WM, a Dreamer-style world model that is split into an agent module and a background module with slot attention, where the background receives information from the agent only through a small latent interface. For reuse, the background module, encoder, and decoder are taken from a source task, and a new agent module, interface, and adapter are trained on the target task. The authors report that BRICKS-WM trained from scratch reaches scores comparable to DreamerV3 and TD-MPC2, and that reusing the background dynamics improves early learning in related tasks. The experiments use four DMC locomotion tasks and two CarRacing variants with 500K environment steps.
Critical strengths
The split of a DreamerV3-style world model into an agent module and a background module, where the agent affects the background only through a small latent interface, is simple and clear, and the paper gives a concrete algorithm for reusing the background module. I think whether parts of a learned world model can be reused for a new agent is an interesting question, and it is good that the appendix reports all 14 transfer pairs including the ones where reuse fails (Tables 4, 5, and 7) and says openly what is not tested, e.g., that the condition of Proposition 4.3 is not assessed (lines 1296-1297).
Critical weaknesses
My main problem is that the results do not seem to support the claim of modular dynamics reuse. Compared to DreamerV3 trained from scratch (Table 6), reuse reaches a higher mean final return in only 2 of the 14 pairs in Table 4, and in some pairs only about half (e.g., Cheetah Run -> Walker Run 318 vs. 636). The clearest gains are in pairs where source and target are closely related (Walker Walk <-> Walker Run, and Sport Car <-> Heavy Car compared to BRICKS-WM from scratch). For these pairs I would expect that simply fine-tuning DreamerV3 from the source checkpoint also works well, but this baseline is missing. So I cannot tell whether the transfer works because of the separate background module or simply because source and target are so similar, and this is the central claim of the paper.
The second problem is that Tables 6 and 8 only include the baselines and not BRICKS-WM trained from scratch, although performance "on par with" DreamerV3 and TD-MPC2 is one of the three contributions. The final DMC returns in Figure 2 seem close, but the few numbers I found point to slower learning, e.g., a 200K AUC on CarRacing of 164 and 141 vs. 406 and 345 for DreamerV3 (Table 8 and lines 1112-1113).
Questions for authors
- How does pretraining DreamerV3 on the source task and fine-tuning on the target task compare to reuse, for example on Walker Walk -> Walker Run and Sport Car / Heavy Car? If reuse is not better, I think the benefit of the modular split is not shown.
- Could you add BRICKS-WM from scratch to Tables 6 and 8, state the number of seeds for the DMC transfer and baseline results, and clarify which parameters (e.g., actor, critic, agent decoder) are re-initialized, frozen or fine-tuned in reuse?