- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies rehearsal-free continual multi-agent reinforcement learning, focusing on catastrophic forgetting and negative policy transfer across sequential tasks. It proposes GASPOL, a dual-gating architecture that combines global experts for preserving task-shared knowledge with local experts for learning task-specific policies. The method further introduces learned task representations, a soft-routing mechanism for expert selection, and advantage-guided regularization to balance stability and plasticity.
Critical strengths
- The paper addresses catastrophic forgetting and negative transfer in continual multi-agent reinforcement learning under a rehearsal-free setting. This is practically relevant, especially when storage is limited or past trajectories cannot be retained.
- GASPOL uses global experts to preserve shared knowledge across tasks and local experts to learn task-specific policies, providing an intuitive approach to balancing stability and plasticity.
- The gating mechanism directly selects local experts at inference time without requiring environmental rewards, task IDs, or additional probing trajectories, making the method more practical than existing approaches such as OWL and MACPRO.
Critical weaknesses
- Figure 1(b) appears to contain a factual inconsistency: the randomly initialized policy reportedly starts with a success rate above 90%, while the transferred policy starts near zero. This contradicts the discussion of negative transfer in the Introduction and should be clarified.
- The claim of “highlighting catastrophic forgetting and negative transfer” is overstated, as both are well-known problems in continual learning rather than novel findings of this paper.
- The methodological novelty is limited. GASPOL appears largely to apply a standard MoE-style structure to the policy network and label its components as global and local experts. The paper does not clearly distinguish the proposed framework from existing MoE-based or expandable-network approaches.
- Figures 4 and 5 do not show consistent improvements over the baselines; GASPOL performs similarly or even worse in some stages. Therefore, the claimed advantages in stability and plasticity are not fully supported by the learning curves.
Questions for authors
Please refer to weakness.
4 comments
I thank the authors for their response and revisions. However, the rebuttal has not changed my core assessment. First, Figure 1(b) appears to have been redrawn without explaining the previously reported results or the reason for the change. Second, I agree that a transferred policy can initially perform worse than a randomly initialized policy on a new task. However, related observations have been discussed in Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data and work on neural-network plasticity, offline-to-online RL, and continual learning. Demonstrating catastrophic forgetting and negative transfer in continual MARL therefore appears to extend established phenomena rather than introduce a fundamentally new problem setting. Finally, while the architectural clarification is helpful, the global/local expert design still appears to be an incremental adaptation of existing MoE and expandable-network approaches.
Authors’ 2nd Response to Reviewer KU2g
Dear Reviewer KU2g, Thank you for your timely response and feedback.
Figure 1(b): We revised Figures 1(a) and 1(b) because an error in the normalization values in the previous plot caused the normalized success rate to exceed 1.0. We have corrected the figures accordingly. The raw numerical results and normalization values are presented in Appendix E. We added the reasons for the changes to our summary of changes.
Transferred vs. randomly initialized policy, and related to Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data (WSRL): Our paper differs from the referenced work in several fundamental ways.
1. Setting: While WSRL focuses on a single online reinforcement learning task, our study focuses on a sequence of multi-agent reinforcement learning tasks.
2. Highlight: While WSRL highlights the impact of offline data unavailability (no-retention) during online fine-tuning, our study highlights the impact of policy negative transfer from the previous task to the currently learned task. While WSRL highlights distribution shift between offline and online data, our work highlights policy misalignments between two similar tasks (games).
3. Objective and Evaluation: While WSRL attempts to achieve effective fine-tuning on a single task without offline data, our work focuses on handling catastrophic forgetting and achieving stability-plasticity in a sequence of tasks. As an implication, rather than being evaluated in a single task as in WSRL, our proposed method is evaluated on all learned tasks, from the first task to the current task. In addition, we explicitly quantify forgetting and measure it, while WSRL does not.
4. Neural Plasticity and Continual Learning: While WSRL focuses on plasticity, our work focuses on both stability and plasticity. To our understanding, WSRL discusses “continually training on offline data during fine-tuning”, not “continual reinforcement learning”, let alone “continual multi-agent reinforcement learning”. Please kindly refer to the previous 3 points.
Demonstrating catastrophic forgetting and negative transfer: Indeed, both terms have been mentioned in numerous continual learning studies. However, to our knowledge, in the area of “continual multi-agent reinforcement learning”, our work is the first study to demonstrate it with empirical data and visualization (plot), highlighting how a policy model from a previous task holds back its learning process on the current task. Particularly for continual MARL with continuous actions, our work is the first study.
Architecture: We respect your view on our architectural design. We believe that our bi-level (dual) gating system, which tailors the mixture and switch of experts, customized task representation, our original routing mechanism, and learning objectives, is distinguished from the existing MoE and expandable-network approaches. We believe that how the architecture works and how it is trained are counted as important elements. For instance, MoE and sparse MoE have the same architecture, but each has unique mechanisms, which distinguish one from the other.
Finally: We kindly request that you take the update and our responses into account and reconsider your score.
Kind Regards,
Authors of Submission 42421
Follow up on Reviewer KU2g Feedback
Dear Reviewer KU2g,
We would like to kindly follow up on our latest responses to your comments. We would appreciate it if you could let us know whether you have any further questions, concerns, or clarifications regarding our responses and the revisions made to the manuscript.
We would be happy to address any remaining concerns. Thank you very much for your time and consideration.
Kind regards,
Authors of Submission 42421
Authors' Response to Reviewer KU2g
W1. Figure 1(b) appears to contain a factual inconsistency: the randomly initialized policy reportedly starts with a success rate above 90%, while the transferred policy starts near zero. This contradicts the discussion of negative transfer in the Introduction and should be clarified.
Our response: Figure 1(b) shows the comparison of the transferred policy versus the trained-from-scratch policy for task-2 in MAMuJoCo games. The policy negative transfer from task-1 to task-2 causes the task-2 performance to start from a lower magnitude than the performance of randomly initialized (trained from scratch) policy. Rather than boosting the policy learning on task-2, the transferred model resists achieving plasticity. This is aligned with our narration in the paragraph.
W2. The claim of “highlighting catastrophic forgetting and negative transfer” is overstated, as both are well-known problems in continual learning rather than novel findings of this paper.
Our response: We believe that showing catastrophic forgetting and negative transfer for continual multi-agent reinforcement learning (MARL), with empirical data and visualization is something new to existing studies. To our knowledge, we have not found the detailed analysis on negative transfer impact to currently learned policy, in the previous studies.
W3. The methodological novelty is limited. GASPOL appears largely to apply a standard MoE-style structure to the policy network and label its components as global and local experts. The paper does not clearly distinguish the proposed framework from existing MoE-based or expandable-network approaches.
Our response: As we mentioned in introduction section, there are at least 2 components that distinguish the proposed framework from existing MoE-based or expandable-network approaches. First, we build a customized dual-expert structure tailoring MoE for global experts, a switch mechanism for local experts, which is connected with customized task representation design (K^o, K^f, K^p). Second, our proposed architecture is designed to learn a sequence of tasks in a rehearsal-free manner. While all global experts are trained on all tasks, a local expert is trained on the current task and has a low probability of being trained on upcoming tasks. To our knowledge, these 2 aspects are distinguished from the existing MoE/evolving methods, including the existing continual MARL methods, i.e., OWL and MACPRO.
W4. Figures 4 and 5 do not show consistent improvements over the baselines; GASPOL performs similarly or even worse in some stages. Therefore, the claimed advantages in stability and plasticity are not fully supported by the learning curves.
Our response: As we mentioned in our narration, Figures 4 and 5 show the adaptability to the current task; thus, the Figures show the performance for the currently learned task only, not the average of all learned tasks. The performance of all learned tasks is shown in Table 2. Indeed, in some stages, our proposed method achieves slightly lower performance than some methods for the currently learned task. However, looking at the performance of all learned tasks (on average), our methods achieve the highest performance among the consolidated methods. This means that our proposed method achieves the highest stability and plasticity.