acceptodds
Under review as a conference paper at ICLR 2027

AWM: Accumulative World Modeling for Continual Reinforcement Learning

Abstract

Continual model-based reinforcement learning requires agents to adapt to non-stationary environments while preserving predictive dynamics from earlier tasks. A fundamental challenge lies in balancing the plasticity required to capture diverse, non-stationary dynamics against the stability needed to shield historical representations from backward interference. To resolve this fundamental problem, we introduce Accumulative World Modeling (AWM), a continual world-modeling framework that separates the capacity required to acquire new tasks from that needed to preserve past dynamics, while maintaining an evolving, trainable shared world model. Specifically, Composable Refinement Mechanisms (CRM) seamlessly adapt the shared world model by composing newly acquired adapters with those from prior tasks. Return-Constrained Compression (RCC) then compresses the adapters into a compact set of atoms, preventing parameter explosion. For task-agnostic inference, Confidence-Guided Expert Routing (CGER) uses cumulative observation-reconstruction error to identify the appropriate expert without task identities. Across two public benchmarks, AWM outperforms the strongest baselines, improving normalized final average performance (ACC) by 212.3% on Atari and 15.5% on CoinRun. Code will be publicly available upon acceptance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual reinforcement learning, model-based reinforcement learning, world models, catastrophic forgetting, continual learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mtDoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Accumulative World Modeling (AWM), a DreamerV3-based framework for continual reinforcement learning. It combines a trainable shared world model with task-specific dynamics adapters and private actor–critic networks. Adapters can reuse components from previous tasks. At task boundaries, pruning and recovery reduce adapter capacity, with candidates selected using an empirical return constraint. During evaluation, cumulative reconstruction error selects an expert without supplied task identities.

Experiments cover six sequential Atari games and six CoinRun variants, comparing against DreamerV3/FIFO and ARROW. The paper reports improved normalized performance and retention across five seeds. Ablations examine adapter reuse, distillation, gradient projection, compression, and oracle routing.

Critical strengths

Promising retention results with repeated runs. The Atari results show a substantial improvement over ARROW: median normalized ACC increases from 0.6187 to 1.9321, with improved retention metrics. Reporting medians and interquartile ranges over five seeds provides useful evidence beyond a single run. The CoinRun evaluation also examines a complementary setting involving progressive visual changes.

Useful controls for compression and routing. Table 3 includes both a recovery-update-matched compression control and an oracle-routing comparison using identical trained checkpoints. These controls help assess whether compression benefits merely reflect additional recovery updates and whether task inference substantially degrades the trained experts’ performance.

Critical weaknesses

The paper does not sufficiently establish its incremental novelty relative to existing approaches. Reconstruction-based expert routing appears in Expert Gate, reusable neural composition in lifelong RL is explored by Mendez et al., and acquisition followed by consolidation is central to Progress & Compress. Although AWM introduces specific adaptations to world-model dynamics, the discussion and experiments do not clearly establish which changes provide a substantive advance beyond combining these ideas. Several claims also exceed the demonstrated mechanisms: frozen experts still depend on a changing shared backbone, while RCC’s finite-cohort return check does not establish a general performance guarantee or prevent total capacity from growing.

The evaluation leaves important questions about attribution, efficiency, and generality unresolved. The external baselines are limited to DreamerV3/FIFO and ARROW, whereas AWM adds private actor–critic networks and substantial world-model capacity; matching interaction and replay budgets therefore does not establish comparable computational resources. The existing ablations are useful, but a consequential component—shared-gradient projection—is not specified in the supplied method section. Evaluation covers only six-task curricula with known training boundaries, limiting support for broad continual-learning claims. Finally, the CoinRun improvement concerns normalized performance and retention, while raw-average returns remain comparable. These results are encouraging, but stronger architectural controls, complete methodological details, and measured resource costs are needed to substantiate the claimed contribution.

Questions for authors

Please specify the complete shared-model update procedure, including current/past-task sampling, loss weights, gradient projection, and which parameters receive each update. Where are these details provided in the supplementary material?

Does the “Shared world model” row in Table 3 retain private actor–critic networks, teacher distillation, and gradient projection under the same update budget? Please clarify this control and provide total parameter counts, including policies, together with training and inference costs relative to the baselines.

How many episodes and candidate widths are used for RCC selection? Is the selection cohort independent of the episodes used for reported results, and how are candidate recovery and selection costs accounted for?

Reviewer_BoxiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a continual world modelling method for continual reinforcement learning. It uses a new adapter build from prior adapters for each new task. The old adapters are compressed to prevent parameter explosion. Cumulative observation-reconstruction error is used to identify the expert to use without task identity. The proposed method outperformed the baselines in experiments.

Critical strengths
  • Architecture is designed to be able to learn new task, reduce forgetting, and reduce the increase in parameters as new tasks are added.
  • Reasonable performance in experiments.
Critical weaknesses
  • While the method is sensible, it is not clear what the new insights in the work are.
  • Runtime per task increases as tasks increase as all experts need to be executed. Experiments are on a small number of tasks, so do not consider scaling.
  • Writing is poor. Symbols are not defined, e.g. the paper starts describing the architecture around eqn 2, but the symbols are not defined in eqn 2, so I am not able to understand how the architecture works.
Questions for authors
  • How would the runtime grow with the number of tasks?
Reviewer_fgGGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

AWM combines a trainable shared world model with reusable task-specific adapters, return-constrained adapter compression, and reconstruction-based expert routing at deployment.

Critical strengths

Atari results show a large improvement over the tested replay baselines: median normalized ACC rises from 0.6187 for ARROW to 1.9321, with low forgetting (p. 7). Five-seed reporting and component ablations are useful. Particularly valuable controls include matching compression-recovery updates and comparing learned routing with oracle routing (pp. 8–9). The paper also acknowledges that CoinRun raw-average returns are comparable to ARROW.

Critical weaknesses
  • Baselines are limited to DreamerV3/FIFO and ARROW. Comparisons with other expandable or modular methods, including controls matched for total parameter count and compute, would better isolate the proposed mechanisms.
  • Compression reduces adapter growth but does not bound total capacity. Atari world-model parameters reach 37.85M, against a 19.50M shared backbone, while each expert also has a private actor–critic. Maintaining recurrent states for every routing candidate adds a scaling cost (pp. 5–6, 9).
  • Training assumes known task boundaries, and the experiments use a fixed six-task sequence per benchmark. Longer sequences, alternative orders, and complete accounting of compression-selection interactions would test the continual-learning claims more thoroughly.
Questions for authors
  1. How much of the gain remains against controls with comparable total parameters, including private actor–critics, and comparable training and inference compute?
  2. Are compression-selection rollouts included in the interaction budget, and how does routing latency grow with expert count? Please clarify these resource totals.
  3. Can you go into more detail In which does the proposed method differ from established work (Mendez et al., 2022)?

Mendez, J. A., van Seijen, H., & Eaton, E. (2022). Modular Lifelong Reinforcement Learning via Neural Composition. International Conference on Learning Representations (ICLR). Paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.