- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Accumulative World Modeling (AWM), a DreamerV3-based framework for continual reinforcement learning. It combines a trainable shared world model with task-specific dynamics adapters and private actor–critic networks. Adapters can reuse components from previous tasks. At task boundaries, pruning and recovery reduce adapter capacity, with candidates selected using an empirical return constraint. During evaluation, cumulative reconstruction error selects an expert without supplied task identities.
Experiments cover six sequential Atari games and six CoinRun variants, comparing against DreamerV3/FIFO and ARROW. The paper reports improved normalized performance and retention across five seeds. Ablations examine adapter reuse, distillation, gradient projection, compression, and oracle routing.
Critical strengths
Promising retention results with repeated runs. The Atari results show a substantial improvement over ARROW: median normalized ACC increases from 0.6187 to 1.9321, with improved retention metrics. Reporting medians and interquartile ranges over five seeds provides useful evidence beyond a single run. The CoinRun evaluation also examines a complementary setting involving progressive visual changes.
Useful controls for compression and routing. Table 3 includes both a recovery-update-matched compression control and an oracle-routing comparison using identical trained checkpoints. These controls help assess whether compression benefits merely reflect additional recovery updates and whether task inference substantially degrades the trained experts’ performance.
Critical weaknesses
The paper does not sufficiently establish its incremental novelty relative to existing approaches. Reconstruction-based expert routing appears in Expert Gate, reusable neural composition in lifelong RL is explored by Mendez et al., and acquisition followed by consolidation is central to Progress & Compress. Although AWM introduces specific adaptations to world-model dynamics, the discussion and experiments do not clearly establish which changes provide a substantive advance beyond combining these ideas. Several claims also exceed the demonstrated mechanisms: frozen experts still depend on a changing shared backbone, while RCC’s finite-cohort return check does not establish a general performance guarantee or prevent total capacity from growing.
The evaluation leaves important questions about attribution, efficiency, and generality unresolved. The external baselines are limited to DreamerV3/FIFO and ARROW, whereas AWM adds private actor–critic networks and substantial world-model capacity; matching interaction and replay budgets therefore does not establish comparable computational resources. The existing ablations are useful, but a consequential component—shared-gradient projection—is not specified in the supplied method section. Evaluation covers only six-task curricula with known training boundaries, limiting support for broad continual-learning claims. Finally, the CoinRun improvement concerns normalized performance and retention, while raw-average returns remain comparable. These results are encouraging, but stronger architectural controls, complete methodological details, and measured resource costs are needed to substantiate the claimed contribution.
Questions for authors
Please specify the complete shared-model update procedure, including current/past-task sampling, loss weights, gradient projection, and which parameters receive each update. Where are these details provided in the supplementary material?
Does the “Shared world model” row in Table 3 retain private actor–critic networks, teacher distillation, and gradient projection under the same update budget? Please clarify this control and provide total parameter counts, including policies, together with training and inference costs relative to the baselines.
How many episodes and candidate widths are used for RCC selection? Is the selection cohort independent of the episodes used for reported results, and how are candidate recovery and selection costs accounted for?