- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Continual Graph Multi-Agent Reinforcement Learning (CGMARL), a framework for cooperative MARL where agents sequentially encounter tasks characterized by different graph structures. The authors formalize three types of structural changes—Topology Switch, Connectivity Switch, and Graph Features Switch—and introduce Graph-based Formation, a benchmark for evaluating continual adaptation and catastrophic forgetting under these changes.
To address structural forgetting, the authors propose Frozen Graph Encoder (FROG), which freezes the GNN-based representation module after learning the first task while allowing the policy head to adapt using existing continual-learning techniques. Experiments with InforMARL and several regularization-based methods demonstrate that FROG generally improves average performance and mitigates forgetting across the proposed task sequences.
Critical strengths
-
The paper identifies an interesting limitation of existing continual MARL formulations: task changes can involve not only reward functions or agent populations but also underlying interaction structures that influence environment dynamics. The proposed CGMARL formulation and its three structural-switch settings provide a systematic way to investigate this problem. I particularly appreciate the introduction of GRAFO, which enables controlled experiments with changing graph structures and agent populations. This is arguably the strongest contribution because it provides a foundation for future research beyond the specific algorithm proposed in this paper.
-
The manuscript is well organized, and the motivation, framework, benchmark, and method are generally easy to follow. The experiments cover three CGMARL scenarios, multiple continual-learning approaches, and ten independent seeds. The authors also examine task-order sensitivity and compare trained versus randomly initialized frozen graph encoders. These analyses provide encouraging evidence that separating structural representation learning from continual policy adaptation can be beneficial.
Critical weaknesses
1. Insufficient motivation and differentiation from adaptive graph-based MARL.
The paper argues that existing CMARL methods fail to leverage task-specific structural information. However, recent graph-based MARL approaches, such as Group-Aware Coordination Graph (IJCAI 2024) and Bayesian Ego-graph Inference (NeurIPS 2025) (already cited in the paper), already learn dynamic, context-dependent interaction structures. Although graph inference and continual learning address different challenges, the manuscript does not clearly explain why existing adaptive graph-learning approaches are insufficient for handling structural task changes or how CGMARL complements them. This distinction is important for establishing the motivation and novelty of the proposed framework.
2. Limited algorithmic novelty and insufficient justification for freezing graph representations.
FROG essentially freezes a GNN encoder after the first task and applies existing continual-learning methods to the action-prediction module. While reasonable, freezing pretrained representations is already a well-established strategy for mitigating catastrophic forgetting. More importantly, preserving encoder parameters does not guarantee that the resulting representations remain informative for subsequent tasks with potentially different graph structures. The experiments also do not clearly isolate whether the improvements come from preserving structural information specifically or simply from reducing trainable parameters. Comparisons with alternative freezing strategies or adaptive graph encoders under the same continual-learning protocol would strengthen the contribution.
3. Theoretical justification and empirical generality need clarification.
The forgetting decomposition in Eq. (5) and Appendix A relies on a Lipschitz-continuity assumption, but the transition from Eq. (7) to Eq. (8) appears to omit the constant . The norm and assumptions needed to bound performance differences through policy-output distances also require clarification. Moreover, freezing the encoder trivially eliminates its parameter-induced changes but does not guarantee reduced overall forgetting.
Experimentally, the evaluation is limited to GRAFO with a single MARL backbone (InforMARL) and primarily assumes known task identities with separate task heads. Although the results are encouraging, broader validation would help establish whether the findings generalize beyond this particular benchmark and learning protocol.
Questions for authors
-
Existing graph-based MARL methods can dynamically infer interaction structures. Why are such approaches insufficient for continual learning across structurally changing tasks? What specific advantage does FROG provide by freezing graph representations rather than adapting them with continual-learning regularization?
-
Could the authors clarify the assumptions behind the forgetting bound in Eq. (5) and Appendix A? In particular, why is the Lipschitz constant $ K_\pi$ omitted in the final inequality, and under what conditions does the policy-output distance bound performance forgetting?
-
How much of FROG's improvement comes specifically from preserving structural representations rather than simply freezing part of the network? Are comparisons with alternative freezing strategies or capacity-matched partially frozen policies available?
-
How strongly does FROG depend on known task identities and separate task heads? Could the authors clarify the results of the task-identity ablation and discuss whether the method remains effective when structural changes occur without explicit task labels?
3 comments
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns regarding the motivation, theoretical analysis, methodological novelty, and task-identity assumptions.
The response provides a clearer distinction between adaptive graph-based MARL and continual structural knowledge retention. I also appreciate the clarification that freezing the graph encoder prevents encoder-induced representation drift rather than guaranteeing the elimination of overall catastrophic forgetting.
The additional freezing-strategy experiment is particularly useful. It provides stronger empirical evidence that preserving the graph encoder after the first task can significantly improve knowledge retention compared with freezing it after subsequent tasks.
I still have some reservations about the algorithmic novelty of FROG, since freezing pretrained representations is an established continual-learning strategy. Furthermore, the current experiments do not fully isolate the advantages of freezing graph representations specifically, and the framework continues to rely on separate task-specific heads.
I have two remaining questions:
-
Task identities and separate task heads: The response indicates that removing task-identity information does not significantly affect performance. However, Appendix D.4 appears to remove only the one-hot task-identity features while retaining separate task-specific heads. Could the authors clarify how the appropriate head is selected at inference time when task identity is unavailable?
-
Methodological differentiation and related work: The additional freezing experiment demonstrates that the freezing stage affects knowledge retention, but does not fully isolate the advantage of freezing graph representations over other network components. Could the authors clarify whether any existing results support this distinction? I also encourage incorporating the discussion of adaptive graph-based MARL from the rebuttal into the Related Work, particularly Group-Aware Coordination Graph for Multi-Agent Reinforcement Learning (IJCAI 2024) [1], to better distinguish dynamic coordination-graph inference from continual structural knowledge preservation.
Nevertheless, I recognize that the proposed CGMARL formulation and GRAFO benchmark represent meaningful contributions beyond the specific FROG algorithm. The manuscript is well organized, the experimental evaluation is relatively comprehensive, and the rebuttal has improved my confidence in the methodological motivation and empirical findings.
I am therefore raising my rating from 2 to 3 (Weak Accept), primarily in recognition of the problem formulation, benchmark contribution, and additional experimental evidence, while retaining some reservations about the novelty and generality of FROG.
Reference
[1] W. Duan, J. Lu, and J. Xuan, "Group-Aware Coordination Graph for Multi-Agent Reinforcement Learning," Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI), pp. 3926–3934, 2024. DOI: 10.24963/ijcai.2024/434.
Thank you for the positive feedback and for the additional questions. We are glad that our rebuttal addressed most of your concerns and that you find our contributions interesting. Please find the answers to the follow-up questions below.
Question 1 (task identities and separate task heads):
Could the authors clarify how the appropriate head is selected at inference time when task identity is unavailable?
We do not consider the setting where task IDs are completely unknown at evaluation time. In the ablation in App. D.4, we simply removed them as features. Both the baselines and FROG use a separate task head (lines 350-353). This is known as the task incremental setting [1], and several works on continual RL/MARL follow this paradigm [2,3,4]. We agree that, in future work, an analysis in other CL settings would be interesting, but this is beyond the scope of this paper. Thank you for the comment, we will clarify this aspect in the revised version.
We hope this clarifies your doubt; please let us know if this is not the case.
Question 2 (methodological differentiation and related work):
The additional freezing experiment demonstrates that the freezing stage affects knowledge retention, but does not fully isolate the advantage of freezing graph representations over other network components. Could the authors clarify whether any existing results support this distinction? I also encourage incorporating the discussion of adaptive graph-based MARL from the rebuttal into the Related Work
Thank you for the comment. The main idea of freezing the graph encoder only (rather than also components in the action-prediction module) is to allow flexibility in learning new tasks. To assess this, we ran an additional experiment to validate our approach against an alternative strategy that freezes a different component of the policy. Specifically, we compare EWC (M) with/without FROG against a variant of FROG in which we freeze the action-prediction module (refer to Eq. 4) while we allow for updates in the graph encoder. Results at https://anonymous.4open.science/r/cgmarl-code/rebuttal-imgs/analysis_frozen_APM.pdf show that this variant (EWC (M) + Frozen APM) loses plasticity (lower AP), failing to learn as effectively as FROG on new tasks (AF is lower because there is “less” to forget, see also lines 405-408). This indicates that alternative freezing strategies can hinder learning, further supporting our proposed design. Note that the EWC (M) + Frozen APM variant performs similarly to EWC (M), suggesting that freezing the action-prediction module does not provide advantages over the base approach.
As for the graph-based MARL methods, we will add the discussion to the Related Work. Thank you for the suggestion.
We are available for any further clarification, if needed.
References
[1] Wang, Liyuan, et al. "A comprehensive survey of continual learning: Theory, method and application." IEEE transactions on pattern analysis and machine intelligence 46.8 (2024): 5362-5383.
[2] Wołczyk, Maciej, et al. "Continual world: A robotic benchmark for continual reinforcement learning." Advances in Neural Information Processing Systems 34 (2021): 28496-28510.
[3] Tomilin, Tristan, et al. "Coom: A game benchmark for continual reinforcement learning." Advances in Neural Information Processing Systems 36 (2023): 67794-67832.
[4] Tomilin, Tristan, et al. "MEAL: a benchmark for continual multi-agent reinforcement learning." arXiv preprint arXiv:2506.14990 (2025).
Thank you for the review; we are pleased that you found our paper interesting and sound. Please find our answers below.
Question 1 (limitations of existing methods)
We agree that approaches designed to infer the interaction structure, such as GACG (IJCAI 2024) or BayesG (NeurIPS 2025), can effectively adapt to graph changes in a CGMARL sequence. However, as stated in lines 279-285, those methods are not designed to retain past structural knowledge, making the corresponding policies prone to forgetting past tasks when learning new ones. Indeed, GNNs learn local (node-level) rules specific to the current graph (task). In a sequential setting, these rules are overwritten when learning new tasks, making them ineffective on past ones. We empirically show this behavior in a simple 2-task sequence in Table 1.
In contrast, by leveraging a frozen graph encoder, FROG mitigates forgetting by preserving the previously learned graph representations: when re-evaluating past topologies, the representations extracted by the graph encoder do not change, preventing representation drift that may be caused by sequentially training on different tasks.
Question 2 (theoretical analysis)
Eq. 5 holds under the assumption of a Lipschitz continuous objective, and removing the Lipschitz constant is a typo; thank you for the comment. We will fix it.
Eq. 5 is not meant to bound the performance forgetting, but rather to show that forgetting caused by the graph representation learning module (first term on the right-hand side of Eq. 5) can be eliminated by freezing the graph encoder. This analysis motivates adopting FROG to prevent performance loss on past tasks (graphs) caused by the graph encoder (see also our answer to Question 1). However, this does not fully eliminate forgetting: as stated in lines 330-334, forgetting can still emerge due to changes in the action-prediction module (second term on the right-hand side of Eq. 5). Future work may build on this analysis to account for such changes and obtain a performance bound, but this is not the purpose of Eq. 5 and is beyond the scope of this paper.
Question 3 (FROG's improvement and comparisons)
Preserving structural information using graph representations is necessary to prevent performance degradation on past tasks. This claim is supported by the comparison in Sec. 6.2 (lines 414-423), where we compared FROG against a completely untrained graph encoder. This variant performs poorly compared to FROG.
To further investigate this aspect, we conducted an additional experiment on the TS sequence using a variant of FROG in which the graph encoder is frozen after learning the second task, instead of after the first one.
The results reported at https://anonymous.4open.science/r/cgmarl-code/rebuttal-imgs/analysis_freeze_from_second_task.pdf show that freezing the graph encoder after learning the second task deteriorates performance on the first task, as it does not preserve the previously learned graph representations. To make this explicit, we report here the performance achieved by both variants on the first task after learning the last one:
These results indicate FROG's improvements depend not merely on reducing the number of trainable parameters, but also on when the graph encoder is frozen. Thank you for the comment; we will include this new experiment in the main text.
Question 4 (task identities)
The experiment in App. D.4 shows that both FROG and the standard CL methods do not depend on known task identities. In particular, removing them does not compromise performance, with average results falling within the uncertainty of the corresponding variant with task identities (Figure 10). On the contrary, preliminary runs showed that both FROG and standard CL methods require separate task heads to prevent catastrophic forgetting on past tasks. These results are consistent with CMARL literature (please refer to the “Ablation Study” in Figure 5 of [1]).
[1] Tomilin, Tristan, et al. "MEAL: a benchmark for continual multi-agent reinforcement learning." arXiv preprint arXiv:2506.14990 (2025).