- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Adaptive Mask Selection and Composition (AMSC) for selective policy reuse in lifelong reinforcement learning. AMSC builds online task embeddings from state-action-reward samples, retrieves a sparse subset of previously learned policy masks using similarity-based sparsemax, and composes them with the current task mask. The method is evaluated on CT28, MG16, and CW10.
Critical strengths
- The paper addresses an important problem beyond catastrophic forgetting: which prior policies should be reused for forward transfer.
- The method is simple and interpretable, and the ablations are generally well designed.
- AMSC achieves strong AUC/FWT with zero forgetting on CT28 and MG16, outperforming the evaluated modular baselines.
Critical weaknesses
- The methodological novelty is somewhat limited. The method mainly combines task embeddings, similarity-based retrieval, sparsemax, and existing mask-composition machinery.
- The composition rule is heuristic. In particular, the total prior mass is
m/(m+1), meaning that retrieving more policies automatically increases reliance on prior knowledge regardless of similarity confidence. - The CW10 result is concerning: AMSC obtains negative FWT and underperforms Mask-BLC. This suggests that task similarity may be effective for retrieval, but insufficient for deciding how knowledge should be composed, especially when layer-specific reuse matters.
- The strongest results are on structured hierarchical curricula where task similarity is likely to be particularly informative. More heterogeneous or adversarial curricula would strengthen the empirical claim.
- AMSC always selects among available priors once retrieval is activated; it lacks an explicit mechanism for deciding that none of the previous policies should be reused.
- The policy library grows with the number of tasks. Quantization reduces storage constants but does not address long-term library growth or redundancy caused by recursively storing composed masks.
Questions for authors
- Why should the current-task mass be exactly
1/(m+1)? Did the authors try using retrieval confidence to determine the prior/current trade-off? - Given the CW10 results, what happens if AMSC retrieval is combined with learned layer-specific composition weights?
- How does AMSC behave when all previous tasks are unrelated or negatively transferring?
- Does recursively storing composed masks lead to increasing redundancy in the policy library?