- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose DCEC, an extension to the Cross-Environment Cooperation training framework which decouples the actor and critic models for training agents to collaborate with new partners zero-shot. Specifically, they ask whether this decoupling leads to better generalization to new partners. Their results show that DCEC encodes less layout-specific information and is more consistent in its actions across states. In the Dual Destination and Overcooked AI environments, DCEC outperforms CEC and other baselines for cross-seed, cross-algorithm, and human-AI collaboration.
Critical strengths
- The evaluation for the probe suggesting DCEC is learning less task-specific information than CEC and is more consistent was very interesting. Moreover, the gradient analysis gave a plausible explanation for why this decoupling of the actor and critic networks leads to benefits that scale well with batch size compared to CEC
- The gains of the method compared to CEC were consistent and appear significant across multiple environments and evaluation conditions, from cross-seed to cross-algorithm to human-AI
Critical weaknesses
- The paper has limited novelty for the broader ZSC or MARL community. Separate policy and value models have been used in many multi-agent implementations and shown benefits, so it's not clear what the broader takeaway should be for the design of multi-agent systems. The paper could be better if it showed there was something specific to coordination that this decoupling is giving you. i.e. Does it help partner generalization independently of environment generalization? Does it help other partner diversity algorithms?
- There are a few confounds in the results. By separating networks, DCEC effectively has double the capacity than CEC which is not clearly controlled for. From my understanding, the baselines also keep the same architecture as CEC, so it's unclear if the benefit of DCEC is coming from architecture only rather than a way the architecture interacts with the multi-agent training protocol.
Overall, I appreciate the authors’ thorough evaluation and analysis. However, I am not convinced that the contribution is significant enough for this venue, given that it mainly applies an existing method to an existing framework. I think the work would be a good fit for a more specialized venue such as AAMAS.
Questions for authors
See weaknesses
2 comments
Rebuttal 2
W2. There are a few confounds in the results. By separating networks, DCEC effectively has double the capacity than CEC which is not clearly controlled for. From my understanding, the baselines also keep the same architecture as CEC, so it's unclear if the benefit of DCEC is coming from architecture only rather than a way the architecture interacts with the multi-agent training protocol.
TL;DR: Decoupled policy–value provides the largest improvement across all batch sizes and both fixed and procedural settings, while matching DCEC to CEC’s parameter count does not reduce performance.
(a) 5 Fixed Tasks
| Method | 8K | 16K | 32K | 65K |
|---|---|---|---|---|
| CEC | 105.9 ± 7.1 | 127.5 ± 2.4 | 120.9 ± 1.1 | 94.4 ± 2.6 |
| + Decoupled | 181.4 ± 4.3 | 243.3 ± 3.3 | 299.1 ± 6.2 | 327.3 ± 2.4 |
| + Adv. | 234.4 ± 8.4 | 289.6 ± 7.0 | 328.1 ± 0.5 | 330.6 ± 0.6 |
| + Invariance (DCEC) | 270.9 ± 7.0 | 285.0 ± 4.3 | 329.0 ± 0.8 | 331.1 ± 0.9 |
Mean ± standard error across six training seeds, averaged over layouts.
(b) 100 Procedurally Generated Tasks
| Method | 8K | 16K | 32K | 65K |
|---|---|---|---|---|
| CEC | 164.7 ± 10.1 | 195.5 ± 1.8 | 178.9 ± 0.9 | 138.9 ± 1.7 |
| + Decoupled | 248.0 ± 5.8 | 283.2 ± 2.6 | 298.8 ± 2.7 | 318.0 ± 1.1 |
| + Adv. | 280.8 ± 1.5 | 305.3 ± 1.3 | 317.3 ± 1.1 | 319.7 ± 0.6 |
| + Invariance (DCEC) | 296.4 ± 1.0 | 307.1 ± 0.7 | 321.3 ± 0.4 | 322.5 ± 0.7 |
Mean ± standard error across six training seeds, averaged over layouts.
Across all batch sizes and both fixed and procedural settings, decoupled policy and value networks produce the largest improvement. Auxiliary objectives provide additional gains compared to decoupled policy and value networks.
Parameter-Matched Comparison at 65K batch sizes
| Model | Parameters | (a) 5 Fixed Layouts | (b) 100 PCG Layouts |
|---|---|---|---|
| CEC | 2.88M | 94.4 ± 6.4 | 138.9 ± 4.2 |
| DCEC-matched | 2.87M | 324.1 ± 3.7 | 315.8 ± 1.7 |
| DCEC | 5.27M | 331.1 ± 2.2 | 322.5 ± 1.6 |
Mean ± standard error across six training seeds, averaged over layouts.
To control for model capacity, we match DCEC to the parameter count of CEC, removing the additional capacity of the full DCEC model. DCEC-Matched performs similarly to full DCEC despite having essentially the same number of parameters as CEC. Together, these results show that policy–value decoupling is the main contributor to the performance gain, while the auxiliary objectives provide smaller improvements and additional model capacity is not required.
Rebuttal 1
We sincerely thank the reviewer for this thoughtful comment. We would like to clarify the main takeaway of our work and share some additional results that we hope will address the reviewer's concern.
First, we would like to highlight what we believe is the core finding of this work. Through our study, we found that overfitting is a major bottleneck in environment-diversity-based ZSC methods such as CEC. Although agents are trained on a large number of procedurally generated environments, they still overfit to environment-specific features, and this limits their ability to generalize to unseen partners.
Regarding the reviewer's question on partner diversity algorithms, we were unfortunately unable to run additional experiments that apply our approach to purely partner-diversity-based methods within the rebuttal period due to limited computational resources, and we deeply apologize for this. However, we would like to share results from combining CEC and DCEC with FCP, a representative partner diversity method.
These results show that adding a partner-diversity-based method to an environment-diversity-based method does not improve performance. This holds for both CEC and DCEC. In addition, prior work has shown that environment-diversity-based methods already outperform partner-diversity-based methods in this setting. Taken together, this suggests that simply combining partner diversity with environment diversity is not sufficient to further improve the performance of ZSC. In contrast, our proposed approach substantially improves performance of ZSC by further improving SOTA environment diversity-based method, CEC, with policy-value decoupling.
For this reason, we view our contribution not as a simple combination of existing techniques, but as breaking through the performance ceiling of environment-diversity-based methods, which cannot be overcome by simply combining them with strong existing approaches such as partner diversity methods. We believe this introduces a new research direction to the MARL community: the performance of environment-diversity-based methods can be further improved by focusing on preventing overfitting, rather than by increasing partner diversity.
We also agree with the reviewer that separate policy and value networks have been widely used in prior multi-agent implementations. However, in those works, the decoupled architecture was mainly adopted because of its strong empirical performance, rather than as a means to address overfitting. In our work, we aim to go beyond reporting empirical gains and show why decoupling helps in ZSC. Through linear probing, we show that decoupling reduces environment-specific information in the policy representation. Through Jensen-Shannon divergence analysis, we show that it leads to more consistent action selection across environment variations. Through value-gradient GSNR analysis, we show that it enables more consistent value learning at larger batch sizes. These analyses together support that the performance gain of the decoupled network comes from mitigating overfitting.
We will revise the paper to state this takeaway more clearly, and we thank the reviewer again for helping us improve the presentation of our contribution.