acceptodds
Under review as a conference paper at ICLR 2027

How to Find and Reuse Policies for Continuous Adaptation in Lifelong Reinforcement Learning

Abstract

In lifelong reinforcement learning, retaining previously learned policies is not sufficient for effective transfer to a new task. Useful knowledge may be distributed across several prior policies, and its relevance may change as the learner acquires experience. One hypothesis is that task similarity can be effectively used in a continual learning setting to find and combine previously learned policies. To test it, Adaptive Mask Selection and Composition (AMSC) is designed to estimate similarity from online experience via non-parametric Wasserstein task embeddings from state-action-reward samples. The z-score-normalized sparsemax of the similarity scores are used to derive a variable-size support to periodically choose and weight policies to form a prior when learning a new task. On CT-graph and MiniGrid, AMSC achieves higher mean performance and forward transfer than the evaluated modular composition baselines while exhibiting no forgetting. Results on Continual World suggest that identifying relevant prior knowledge and determining its layer-specific composition may require additional layer-specific tuning. Ablations show that selecting relevant sources and determining how strongly to reuse them are central to these gains. Independently measured pairwise transfer is also positively associated with task-embedding similarity. These results indicate that task similarity can be an effective criterion to select and weight specific knowledge for reuse in lifelong reinforcement learning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
lifelong reinforcement learning, continual reinforcement learning, forward transfer, selective policy reuse, policy composition, task similarity, Wasserstein task embeddings, sparse retrieval, modular policies

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_16BNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Adaptive Mask Selection and Composition (AMSC) for selective policy reuse in lifelong reinforcement learning. AMSC builds online task embeddings from state-action-reward samples, retrieves a sparse subset of previously learned policy masks using similarity-based sparsemax, and composes them with the current task mask. The method is evaluated on CT28, MG16, and CW10.

Critical strengths
  • The paper addresses an important problem beyond catastrophic forgetting: which prior policies should be reused for forward transfer.
  • The method is simple and interpretable, and the ablations are generally well designed.
  • AMSC achieves strong AUC/FWT with zero forgetting on CT28 and MG16, outperforming the evaluated modular baselines.
Critical weaknesses
  • The methodological novelty is somewhat limited. The method mainly combines task embeddings, similarity-based retrieval, sparsemax, and existing mask-composition machinery.
  • The composition rule is heuristic. In particular, the total prior mass is m/(m+1), meaning that retrieving more policies automatically increases reliance on prior knowledge regardless of similarity confidence.
  • The CW10 result is concerning: AMSC obtains negative FWT and underperforms Mask-BLC. This suggests that task similarity may be effective for retrieval, but insufficient for deciding how knowledge should be composed, especially when layer-specific reuse matters.
  • The strongest results are on structured hierarchical curricula where task similarity is likely to be particularly informative. More heterogeneous or adversarial curricula would strengthen the empirical claim.
  • AMSC always selects among available priors once retrieval is activated; it lacks an explicit mechanism for deciding that none of the previous policies should be reused.
  • The policy library grows with the number of tasks. Quantization reduces storage constants but does not address long-term library growth or redundancy caused by recursively storing composed masks.
Questions for authors
  1. Why should the current-task mass be exactly 1/(m+1)? Did the authors try using retrieval confidence to determine the prior/current trade-off?
  2. Given the CW10 results, what happens if AMSC retrieval is combined with learned layer-specific composition weights?
  3. How does AMSC behave when all previous tasks are unrelated or negatively transferring?
  4. Does recursively storing composed masks lead to increasing redundancy in the policy library?
Reviewer_4THnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how an lifelong reinforcement learning agent can selectively reuse previously learned policies with the AMSC method: each task has a binary mask over a shared, frozen backbone network. When a new task arrives, AMSC uses online-collected data to estimate, via a non-parametric Wasserstein embedding, the similarity between the new task and each earlier task. In the CT-graph and MiniGrid experiments, AMSC achieves higher AUC and forward transfer than the modular baselines and claims zero forgetting. On Continual World, AMSC does not outperform methods that learn the composition weights, but the authors acknowledge this limitation: identifying relevant prior knowledge and determining its layer-specific composition may require additional layer-wise tuning. The paper also provides ablation studies and an independently measured pairwise-transfer analysis, and uses these results to support the claim that "task similarity can serve as an effective criterion for selecting task-specific knowledge and weighting it for reuse in lifelong reinforcement learning."

Critical strengths

1.The motivation is well grounded in practice and is reasonably explained. Distinguishing retention from selective reuse is a useful motivation, and the paper's later ablation experiments also demonstrate why unsuitable priors are especially harmful in RL. 2.The method is simple and efficient. AMSC requires no task labels, no task descriptions, and no trial runs of earlier policies, and it does not learn the composition weights through optimization.

Critical weaknesses
  1. The contribution is incremental and may not be sufficient. The whole pipeline, task embedding → similarity → retrieval → mask composition, follows MOSAIC (Nath et al., 2026) and uses the same embedding. The differences are z-score + sparsemax, periodic refreshing, ℓ2 normalization, and replacing learned weights with fixed weights. Similarity-driven weighting also already exists (SDW, Huang et al., 2025).
  2. The author should summarize the differences from and new contributions to other recent related studies such as (Savadikar et al., 2026) and (Liu et al., 2025) in the "Related Work" section.
Questions for authors
  1. When the policy library contains no relevant prior, for example a sequence of mutually unrelated tasks, what weights does AMSC assign, and how does its performance compare with learning from scratch? Could adding an abstention option fix this problem?
  2. On CW10, why is the average AUC higher than that of learning from scratch, while the average FWT is −1.21? Please add Learned-α within the retrieved support set and a layer-wise variant of AMSC to test the shared-weight hypothesis, and provide per-task FWT and the missing baselines.
Reviewer_ae61OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to identify and reuse previously acquired policies for effective forward transfer in lifelong reinforcement learning. The authors propose Adaptive Mask Selection and Composition (AMSC), a framework that estimates task similarity using non-parametric Wasserstein embeddings constructed from online state-action-reward samples. The resulting similarity scores are normalized and transformed through sparsemax to dynamically select and weight a variable number of previously learned policies, which are subsequently composed to provide a prior for learning new tasks. The authors evaluate AMSC on CT-graph, MiniGrid, and Continual World. According to the reported results, AMSC improves average performance and forward transfer over the evaluated modular composition baselines on CT-graph and MiniGrid while avoiding forgetting. Ablation studies suggest that both policy selection and reuse weighting contribute to the observed improvements. The paper also reports a positive association between task-embedding similarity and independently measured pairwise transfer. However, the results on Continual World indicate limitations in more complex settings, particularly regarding layer-specific policy composition.

Critical strengths

Strong Empirical Performance in Forward Transfer Without Forgetting As shown in Table 1 (Section 5.2), AMSC achieves strong performance and forward transfer on CT28 and MG16, outperforming the evaluated modular policy composition baselines while maintaining zero measured forgetting. The results demonstrate that similarity-guided policy selection and composition can effectively facilitate knowledge reuse in lifelong reinforcement learning.

Critical weaknesses

W1. Limited Generalizability to Complex Continuous-Control Tasks As shown in Table 2 (Section 5.3), AMSC fails to achieve positive forward transfer on the more complex continuous-control benchmark CW10 and underperforms the existing Mask-BLC method. The authors attribute this limitation to the use of shared composition weights across all network layers, whereas Mask-BLC learns layer-specific weights. Since AMSC aims to improve forward transfer by automatically selecting and composing previously learned policies based on task similarity, these results reveal a limitation in its composition mechanism. If the effectiveness of policy reuse depends strongly on the environment or policy network architecture, the generalizability of the proposed method may be limited.

W2. Limited Methodological Novelty Beyond Existing Policy Reuse Frameworks As discussed in Section 3.2 and Section 5.2, AMSC builds upon existing policy reuse frameworks, particularly Modulating Masks and MOSAIC. The paper explicitly distinguishes AMSC from MOSAIC by introducing library-relative similarity-based sparse selection and periodically updating both the selected policies and their composition weights. The experimental results in Table 1 also demonstrate improvements over these baselines. However, since the underlying task representations, similarity-based policy retrieval, and mask-based composition have already been explored in prior work, the methodological contribution of AMSC appears largely incremental. Although the reported performance improvements are encouraging, it remains unclear whether these modifications constitute a sufficiently substantial methodological advance beyond existing policy reuse frameworks.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.