acceptodds
Under review as a conference paper at ICLR 2027

Cross-Environment Cooperation with Decoupled Policy and Value Learning

Abstract

Zero-shot coordination (ZSC) enables ego agents to collaborate with unseen partners and is essential for human–AI coordination with diverse human behaviors. Recent work has improved coordination generalization by training agents across diverse environments. However, cross-environment training can still be limited by overfitting. Prior work in single-agent reinforcement learning shows that shared policy-value representations can hinder generalization by introducing environment-specific information into the policy representation. We hypothesize that shared policy-value representations also limit generalization in cross-environment ZSC. To investigate this hypothesis, we incorporate IDAAC, which decouples policy and value learning, into Cross-Environment Cooperation and introduce DCEC. Our analysis shows that DCEC selects actions more consistently across similar states from different environments, while encoding less environment-specific information in its policy representation. Moreover, DCEC exhibits higher value-gradient GSNR than CEC at larger training batch sizes, indicating more consistent value-learning gradients under larger-batch training. Across Dual Destination and Overcooked-AI, DCEC improves coordination with unseen partners and generalization to unseen environments, while also achieving strong coordination performance with real humans.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning, Zero-Shot Coordination, Cross-Environment Training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yP1hOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose DCEC, an extension to the Cross-Environment Cooperation training framework which decouples the actor and critic models for training agents to collaborate with new partners zero-shot. Specifically, they ask whether this decoupling leads to better generalization to new partners. Their results show that DCEC encodes less layout-specific information and is more consistent in its actions across states. In the Dual Destination and Overcooked AI environments, DCEC outperforms CEC and other baselines for cross-seed, cross-algorithm, and human-AI collaboration.

Critical strengths
  • The evaluation for the probe suggesting DCEC is learning less task-specific information than CEC and is more consistent was very interesting. Moreover, the gradient analysis gave a plausible explanation for why this decoupling of the actor and critic networks leads to benefits that scale well with batch size compared to CEC
  • The gains of the method compared to CEC were consistent and appear significant across multiple environments and evaluation conditions, from cross-seed to cross-algorithm to human-AI
Critical weaknesses
  • The paper has limited novelty for the broader ZSC or MARL community. Separate policy and value models have been used in many multi-agent implementations and shown benefits, so it's not clear what the broader takeaway should be for the design of multi-agent systems. The paper could be better if it showed there was something specific to coordination that this decoupling is giving you. i.e. Does it help partner generalization independently of environment generalization? Does it help other partner diversity algorithms?
  • There are a few confounds in the results. By separating networks, DCEC effectively has double the capacity than CEC which is not clearly controlled for. From my understanding, the baselines also keep the same architecture as CEC, so it's unclear if the benefit of DCEC is coming from architecture only rather than a way the architecture interacts with the multi-agent training protocol.

Overall, I appreciate the authors’ thorough evaluation and analysis. However, I am not convinced that the contribution is significant enough for this venue, given that it mainly applies an existing method to an existing framework. I think the work would be a good fit for a more specialized venue such as AAMAS.

Questions for authors

See weaknesses

2 comments
AuthorsOpenReview ↗

Rebuttal 1

W1. The paper has limited novelty for the broader ZSC or MARL community. [...] partner diversity algorithms?

We sincerely thank the reviewer for this thoughtful comment. We would like to clarify the main takeaway of our work and share some additional results that we hope will address the reviewer's concern.

First, we would like to highlight what we believe is the core finding of this work. Through our study, we found that overfitting is a major bottleneck in environment-diversity-based ZSC methods such as CEC. Although agents are trained on a large number of procedurally generated environments, they still overfit to environment-specific features, and this limits their ability to generalize to unseen partners.

Regarding the reviewer's question on partner diversity algorithms, we were unfortunately unable to run additional experiments that apply our approach to purely partner-diversity-based methods within the rebuttal period due to limited computational resources, and we deeply apologize for this. However, we would like to share results from combining CEC and DCEC with FCP, a representative partner diversity method.

MethodSPXPHuman Proxy Partner (BC)
CEC96.8 ± 15.895.1 ± 6.353.2 ± 6.1
CEC + FCP89.6 ± 7.192.5 ± 5.149.3 ± 1.8
DCEC332.8 ± 2.4333.5 ± 1.890.8 ± 5.8
DCEC + FCP325.2 ± 1.7327.1 ± 1.989.7 ± 5.0

These results show that adding a partner-diversity-based method to an environment-diversity-based method does not improve performance. This holds for both CEC and DCEC. In addition, prior work has shown that environment-diversity-based methods already outperform partner-diversity-based methods in this setting. Taken together, this suggests that simply combining partner diversity with environment diversity is not sufficient to further improve the performance of ZSC. In contrast, our proposed approach substantially improves performance of ZSC by further improving SOTA environment diversity-based method, CEC, with policy-value decoupling.

For this reason, we view our contribution not as a simple combination of existing techniques, but as breaking through the performance ceiling of environment-diversity-based methods, which cannot be overcome by simply combining them with strong existing approaches such as partner diversity methods. We believe this introduces a new research direction to the MARL community: the performance of environment-diversity-based methods can be further improved by focusing on preventing overfitting, rather than by increasing partner diversity.

We also agree with the reviewer that separate policy and value networks have been widely used in prior multi-agent implementations. However, in those works, the decoupled architecture was mainly adopted because of its strong empirical performance, rather than as a means to address overfitting. In our work, we aim to go beyond reporting empirical gains and show why decoupling helps in ZSC. Through linear probing, we show that decoupling reduces environment-specific information in the policy representation. Through Jensen-Shannon divergence analysis, we show that it leads to more consistent action selection across environment variations. Through value-gradient GSNR analysis, we show that it enables more consistent value learning at larger batch sizes. These analyses together support that the performance gain of the decoupled network comes from mitigating overfitting.

We will revise the paper to state this takeaway more clearly, and we thank the reviewer again for helping us improve the presentation of our contribution.

AuthorsOpenReview ↗

Rebuttal 2

W2. There are a few confounds in the results. By separating networks, DCEC effectively has double the capacity than CEC which is not clearly controlled for. From my understanding, the baselines also keep the same architecture as CEC, so it's unclear if the benefit of DCEC is coming from architecture only rather than a way the architecture interacts with the multi-agent training protocol.

TL;DR: Decoupled policy–value provides the largest improvement across all batch sizes and both fixed and procedural settings, while matching DCEC to CEC’s parameter count does not reduce performance.

(a) 5 Fixed Tasks

Method8K16K32K65K
CEC105.9 ± 7.1127.5 ± 2.4120.9 ± 1.194.4 ± 2.6
+ Decoupled181.4 ± 4.3243.3 ± 3.3299.1 ± 6.2327.3 ± 2.4
+ Adv.234.4 ± 8.4289.6 ± 7.0328.1 ± 0.5330.6 ± 0.6
+ Invariance (DCEC)270.9 ± 7.0285.0 ± 4.3329.0 ± 0.8331.1 ± 0.9

Mean ± standard error across six training seeds, averaged over layouts.

(b) 100 Procedurally Generated Tasks

Method8K16K32K65K
CEC164.7 ± 10.1195.5 ± 1.8178.9 ± 0.9138.9 ± 1.7
+ Decoupled248.0 ± 5.8283.2 ± 2.6298.8 ± 2.7318.0 ± 1.1
+ Adv.280.8 ± 1.5305.3 ± 1.3317.3 ± 1.1319.7 ± 0.6
+ Invariance (DCEC)296.4 ± 1.0307.1 ± 0.7321.3 ± 0.4322.5 ± 0.7

Mean ± standard error across six training seeds, averaged over layouts.

Across all batch sizes and both fixed and procedural settings, decoupled policy and value networks produce the largest improvement. Auxiliary objectives provide additional gains compared to decoupled policy and value networks.

Parameter-Matched Comparison at 65K batch sizes

ModelParameters(a) 5 Fixed Layouts(b) 100 PCG Layouts
CEC2.88M94.4 ± 6.4138.9 ± 4.2
DCEC-matched2.87M324.1 ± 3.7315.8 ± 1.7
DCEC5.27M331.1 ± 2.2322.5 ± 1.6

Mean ± standard error across six training seeds, averaged over layouts.

To control for model capacity, we match DCEC to the parameter count of CEC, removing the additional capacity of the full DCEC model. DCEC-Matched performs similarly to full DCEC despite having essentially the same number of parameters as CEC. Together, these results show that policy–value decoupling is the main contributor to the performance gain, while the auxiliary objectives provide smaller improvements and additional model capacity is not required.

Reviewer_FzWcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether changing policy–value learning can improve coordination with unfamiliar partners and held-out configurations, even when training already spans diverse environments. It applies IDAAC's separate policy and value networks, advantage prediction, and temporal-order regularization to Cross-Environment Cooperation. The experiments cover Dual Destination and Overcooked, including batch-size comparisons, cross-algorithm partners, and human participants. I read the contribution as an empirical study of learning design for cooperative generalization.

Critical strengths

None

Critical weaknesses
  1. The coordination problem is clear and the performance gains matter. My reservation concerns how much the experiments establish about shared-representation overfitting. Section 7 appropriately acknowledges the combined effects of decoupling and the auxiliary objectives. Still, there is no intermediate control separating their roles. The representation probe also uses each model's own rollout trajectories (Appendix E.1), so differences in visited states may contribute to the lower layout-prediction accuracy. I agree that the results are consistent with the authors' hypothesis, but I would be cautious about how much they tell us specifically about reduced overfitting rather than improved learning more generally. This leaves the mechanism less settled than the benefit of the complete recipe.
Questions for authors
  1. Do you have a matched-batch comparison of the decoupled networks with and without IDAAC's auxiliary objectives? Even one intermediate control would help me judge whether the new insight is mainly about policy–value separation or the full regularization recipe.
  2. Does the layout-probe difference persist when the frozen CEC and DCEC models process the same observation sequences, with the same episode-reset protocol? This would help distinguish a representation difference from the different state distributions induced by their policies.
1 comment
AuthorsOpenReview ↗

Rebuttal 1

Q1. Do you have a matched-batch comparison of the decoupled networks with and without IDAAC's auxiliary objectives? Even one intermediate control would help me judge whether the new insight is mainly about policy–value separation or the full regularization recipe.

TL;DR: Decoupled policy–value provides the largest improvement across all batch sizes and both fixed and procedural settings.

We conduct isolation experiments by cumulatively adding decoupled policy and value networks (+ Decoupled), the advantage-prediction objective (+ Adv.), and the instance-invariance objective to form DCEC. We evaluate all components across batch sizes from 8K to 65K.

(a) 5 Fixed Tasks

Method8K16K32K65K
CEC105.9 ± 7.1127.5 ± 2.4120.9 ± 1.194.4 ± 2.6
+ Decoupled181.4 ± 4.3243.3 ± 3.3299.1 ± 6.2327.3 ± 2.4
+ Adv.234.4 ± 8.4289.6 ± 7.0328.1 ± 0.5330.6 ± 0.6
+ Invariance (DCEC)270.9 ± 7.0285.0 ± 4.3329.0 ± 0.8331.1 ± 0.9

Mean ± standard error across six training seeds, averaged over layouts.

(b) 100 Procedurally Generated Tasks

Method8K16K32K65K
CEC164.7 ± 10.1195.5 ± 1.8178.9 ± 0.9138.9 ± 1.7
+ Decoupled248.0 ± 5.8283.2 ± 2.6298.8 ± 2.7318.0 ± 1.1
+ Adv.280.8 ± 1.5305.3 ± 1.3317.3 ± 1.1319.7 ± 0.6
+ Invariance (DCEC)296.4 ± 1.0307.1 ± 0.7321.3 ± 0.4322.5 ± 0.7

Mean ± standard error across six training seeds, averaged over layouts.

Across all batch sizes and both fixed and procedural settings, decoupled policy and value networks produce the largest improvement. Auxiliary objectives provide additional gains compared to decoupled policy and value networks.

Q2. Does the layout-probe difference persist when the frozen CEC and DCEC models process the same observation sequences, with the same episode-reset protocol?

TL;DR The policy-representation difference persists when CEC and DCEC process identical random-action rollouts.

To control observation sequences, we generated identical random-action rollouts for CEC and DCEC and conducted the linear-probe analysis. A similar pattern is observed with random rollouts. The value representations of CEC and DCEC contain similar amounts of base-layout information, while the CEC policy representation contains more base-layout information than the DCEC policy representation.

RepresentationRolloutCECDCECp-value
ValueOwn rollout0.561 ± 0.0140.557 ± 0.0140.843
Random rollout0.558 ± 0.0160.588 ± 0.0160.236
PolicyOwn rollout0.423 ± 0.0090.267 ± 0.009< 0.001
Random rollout0.407 ± 0.0110.285 ± 0.016< 0.001

p-values are computed using paired t-tests across six training seeds.

These results suggest that the policy-representation difference persists even when CEC and DCEC process the same observation sequences. We update the manuscript with the random-rollout probe results and clarify that the policy-representation difference persists under same observation sequences.

Reviewer_14cgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether shared policy-value representations limit zero-shot coordination under cross-environment training. It introduces DCEC by integrating IDAAC into Cross-Environment Cooperation: separate recurrent policy and value networks are trained with PPO, an advantage-prediction objective, and a temporal-order invariance objective. The authors also investigate increasing the number of parallel environments, thereby increasing PPO minibatch size while holding total environment steps fixed. Evaluation covers Dual Destination and Overcooked-AI, including cross-seed partners, held-out procedural task configurations, cross-algorithm partners, and a 20-participant human study.

Critical strengths
  • 1. DCEC improves on CEC across both environments, on fixed and procedurally generated held-out tasks, and with seed, algorithm and human partners. The batch-scaling result (Fig. 8) is practically useful. This matters because it shows that a simple architectural change substantially improves ZSC generalization, a factor that prior ZSC work, focused on data diversity, has largely overlooked.

  • 2. Value representations are similar across methods while policy representations differ (Fig. 3a), and DCEC shows lower divergence on matched cross-environment states (Fig. 3b). These analyses go beyond reward curves and meaningfully support the hypothesis that shared representations leak environment-specific information into the policy.

Critical weaknesses
  • 1. The proposed mechanism is not isolated. DCEC changes three things at once: separate networks, an advantage loss and an invariance loss. It also roughly doubles the parameter count. In IDAAC’s own results, plain decoupling fails, and most of the gain comes from adding advantage prediction. The paper’s central claim is about removing shared representations, so it needs a DAAC-style variant (no invariance loss), an AAC-style variant (shared network plus advantage head) and a parameter-matched CEC. In addition, IDAAC’s coefficients are reused without re-tuning. The invariance loss was designed for partially observable, variable-length settings, which Overcooked is not. Overfitting itself is never measured directly. As it stands, the paper shows that IDAAC transfers to CEC, not that shared representations cause overfitting.

  • 2. The analyses overreach, and baseline fairness is uncertain. The GSNR argument relies on supervised-learning theory, yet value gradients never reach the DCEC policy, so the link to ZSC is unsupported. CEC underperforms IPPO and FCP on some fixed tasks, which suggests it may be under-tuned. The human study’s claim that all pairwise comparisons are significant (App. F.1) conflicts with the Fig. 9 caption and seems implausible given the overlap between IPPO and E3T.

  • 3. The human testing is severely limited in scale. A 20-participant study is remarkably small for evaluating human-AI zero-shot coordination, especially in temporally extended, high-variance domains like Overcooked where human skill levels and play styles vary drastically. Such a limited sample size makes it difficult to draw robust, generalizable conclusions about the algorithm's actual performance with diverse human partners, compounding the statistical significance issues noted above.

Questions for authors
  • 1. Can you report a DAAC-style variant (no invariance loss), an AAC-style variant (shared network with an advantage head), and a CEC baseline with a matched parameter count? If removing shared representations accounts for most of the gain, I would raise my score.

  • 2. How do your CEC results compare with Jha et al. (2025)? Was CEC tuned for the modified environments and for each batch size, including the learning rate?

  • 3. Please reconcile the significance claims in Fig. 9 and Appendix F.1. Which comparisons were run, what are the effect sizes, and is there a test on recipes made?

  • 4. Does CEC’s GSNR include the shared-trunk parameters? Could you report policy-gradient GSNR, or relate GSNR to cross-play reward across seeds and batch sizes?

  • 5. Given the high variance in human play styles and proficiency, how did you ensure the 20-participant human study was adequately powered? Could you provide more details on the participant demographics and skill distribution, or are there plans to expand the study to validate these claims on a broader population?

3 comments
AuthorsOpenReview ↗

Rebuttal 1

W1-1. The proposed mechanism is not isolated. It also roughly doubles the parameter count.

TL;DR: Decoupled policy–value provides the largest improvement across all batch sizes and both fixed and procedural settings. The parameter-count concern is addressed separately in Q1.

We conduct isolation experiments by cumulatively adding decoupled policy and value networks (+ Decoupled), the advantage-prediction objective (+ Adv.), and the instance-invariance objective to form DCEC. We evaluate all components across batch sizes from 8K to 65K.

(a) 5 Fixed Tasks

Method8K16K32K65K
CEC105.9 ± 7.1127.5 ± 2.4120.9 ± 1.194.4 ± 2.6
+ Decoupled181.4 ± 4.3243.3 ± 3.3299.1 ± 6.2327.3 ± 2.4
+ Adv.234.4 ± 8.4289.6 ± 7.0328.1 ± 0.5330.6 ± 0.6
+ Invariance (DCEC)270.9 ± 7.0285.0 ± 4.3329.0 ± 0.8331.1 ± 0.9

Mean ± standard error across six training seeds, averaged over layouts.

(b) 100 Procedurally Generated Tasks

Method8K16K32K65K
CEC164.7 ± 10.1195.5 ± 1.8178.9 ± 0.9138.9 ± 1.7
+ Decoupled248.0 ± 5.8283.2 ± 2.6298.8 ± 2.7318.0 ± 1.1
+ Adv.280.8 ± 1.5305.3 ± 1.3317.3 ± 1.1319.7 ± 0.6
+ Invariance (DCEC)296.4 ± 1.0307.1 ± 0.7321.3 ± 0.4322.5 ± 0.7

Mean ± standard error across six training seeds, averaged over layouts.

Across all batch sizes and both fixed and procedural settings, decoupled policy and value networks produce the largest improvement. Auxiliary objectives provide a additional gain compared to decoupled policy and value networks. The parameter-count concern will be addressed in answer to Q1 below.

W1-2. In addition, IDAAC’s coefficients are reused without re-tuning. The invariance loss was designed for partially observable, variable-length settings, which Overcooked is not.

TL;DR: We use the invariance loss to reduce temporal information in the recurrent actor, while our ablations show that substantial gains come from decoupling and advantage prediction, without the invariance loss.

We used the auxiliary-loss coefficients from the original IDAAC paper without additional tuning. Although Overcooked has a fully observable grid and a fixed episode length, the recurrent actor can learn layout-specific temporal patterns from changes in object positions and states. We use the invariance loss to prevent the actor representation from encoding temporal-order information.

As shown in W1-1, CEC + Decoupling already outperforms CEC across all batch sizes, while CEC + Decoupling + Adv. provides further improvements without the invariance loss. We did not evaluate other invariance-loss coefficients, so we cannot determine whether additional tuning would improve full DCEC. However, our ablations show that the main performance gains do not depend on the invariance loss or its coefficient.

W2-1. The GSNR argument relies on supervised-learning theory, yet value gradients never reach the DCEC policy, so the link to ZSC is unsupported.

TL;DR: Recent RL studies adopt optimization principles from supervised learning, motivating our use of GSNR. Critic learning also influences policy updates through advantage estimation.

Recent RL studies have applied optimization principles from supervised learning to improve training stability[1] and mitigate loss of plasticity[2] in RL. This motivates using GSNR as a diagnostic of gradient consistency during training.

Although value gradients do not directly update the DCEC policy network, the critic affects policy learning through advantage estimation. The value estimates are used to compute GAE advantages, which are used in the PPO policy objective. Prior work [3] has studied the relationship between critic overfitting and agent performance in actor-critic methods. Thus, decoupled value and policy networks do not make value learning independent of policy learning.


[1] Lyle, Clare, et al. "Normalization and effective learning rates in reinforcement learning." Advances in Neural Information Processing Systems 37 (2024): 106440-106473.

[2] Lee, Hojoon, et al. "Hyperspherical normalization for scalable deep reinforcement learning." arXiv preprint arXiv:2502.15280 (2025).

[3] Nauman, Michal, et al. "Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning." arXiv preprint arXiv:2403.00514 (2024).

AuthorsOpenReview ↗

Rebuttal 2

W2-2. CEC underperforms IPPO and FCP on some fixed tasks, which suggests it may be under-tuned.

TL;DR: We implemented CEC using the authors' official code and report the settings used in our experiments.

Some implementation details were not specified in the original CEC paper. We explicity report the our setting:32 parallel environments and shuffle_inv_and_pot=True.

We initially trained CEC with 32 parallel environments, but this setting did not reproduce the reported performance. Therefore, we additionally evaluated CEC with 64 parallel environments, which is the configuration reported in Appendix C.3

CEC underperforms IPPO and FCP on some fixed tasks in our results. However, the original CEC paper mainly reports average performance across the five fixed tasks, rather than performance on each individual task.

W2-3. The human study’s claim that all pairwise comparisons are significant (App. F.1) conflicts with the Fig. 9 caption and seems implausible given the overlap between IPPO and E3T.

TL;DR: “All pairwise comparisons” in Appendix F.1 was a writing error.

Thank you for pointing out this inconsistency. The statistical significance reported in Figure 9 refers specifically to comparisons between DCEC(65K) and each baseline, not to comparisons among all pairs of agents. We will correct the corresponding statement in Appendix F.1 accordingly.

W3. The human testing is severely limited in scale. A 20-participant study is remarkably small for evaluating human-AI zero-shot coordination.

TL;DR: The human study now includes 40 participants.

We recruited 20 additional participants and conducted the same human-AI coordination evaluation. This increased the total sample size from 20 to 40. We updated the human study results in the manuscript using the combined data from all 40 participants.

Q1. Can you report a DAAC-style variant (no invariance loss), an AAC-style variant (shared network with an advantage head), and a CEC baseline with a matched parameter count?

TL;DR: We evaluated the AAC-style, DAAC-style, and parameter-matched variants. The results show that policy–value decoupling provides the main performance gain, while matching DCEC to CEC’s parameter count does not reduce performance.

Component Ablation at 65K batch sizes

Method(a) 5 Fixed Tasks(b) 100 PCG Tasks
CEC94.4 ± 2.6138.9 ± 1.7
CEC + Adv. (AAC-style)85.9 ± 2.5127.1 ± 3.1
CEC + Decoupled327.3 ± 2.4318.0 ± 1.1
CEC + Decoupled + Adv. (DAAC-style)330.6 ± 0.6319.7 ± 0.6
DCEC331.1 ± 0.9322.5 ± 0.7

Mean ± standard error across six training seeds, averaged over layouts.

Decoupled policy–value produces the largest performance improvement. Advantage prediction does not improve performance when applied to the shared network, but provides additional gains after the policy and value networks are decoupled.

Parameter-Matched Comparison at 65K batch sizes

ModelParameters(a) 5 Fixed Layouts(b) 100 PCG Layouts
CEC2.88M94.4 ± 6.4138.9 ± 4.2
DCEC-matched2.87M324.1 ± 5.4315.8 ± 3.1
DCEC5.27M331.1 ± 2.2322.5 ± 1.6

Mean ± standard error across six training seeds, averaged over layouts.

To control for model capacity, we match DCEC to the parameter count of CEC, removing the additional capacity of the full DCEC model. DCEC-Matched performs similarly to full DCEC despite having essentially the same number of parameters as CEC. Together, these results show that policy–value decoupling is the main contributor to the performance gain, while the auxiliary objectives provide smaller improvements and additional model capacity is not required.

Q2. How do your CEC results compare with Jha et al. (2025)?

We address this question in detail in W2-2 above.

AuthorsOpenReview ↗

Rebuttal 3

Q3. Please reconcile the significance claims in Fig. 9 and Appendix F.1. Which comparisons were run, what are the effect sizes, and is there a test on recipes made?

TL;DR: Recipes made and all survey comparisons were significant, except Consistent versus DCEC(8K).

we compared DCEC(65K) with IPPO, E3T, FCP, CEC, and DCEC(8K). We first averaged each participant’s results across the five maps and then performed paired, two-sided t-tests on the 40 participant-level observations. We applied the Holm correction to the five comparisons within each outcome. The tables below report Cohen’s dzd_z and Holm-adjusted pp-values for every comparison.

Recipes made

Compared with DCEC(65K)dzd_z (p)
IPPO4.665 (< .001)
E3T4.272 (< .001)
FCP3.242 (< .001)
CEC3.498 (< .001)
DCEC(8K)0.935 (< .001)

Survey responses

Survey itemIPPOE3TFCPCECDCEC(8K)
Adaptive2.326 (< .001)2.144 (< .001)1.480 (< .001)1.591 (< .001)0.357 (.0296)
Consistent0.672 (< .001)1.156 (< .001)0.773 (< .001)0.938 (< .001)0.158 (.3235)
Human-like2.334 (< .001)2.004 (< .001)1.614 < .001)1.501 (< .001)0.324 (.0475)
In My Way1.598 (< .001)1.528 (< .001)0.885 (< .001)1.049 (< .001)0.371 (.0243)
Frustrating2.205 (< .001)2.318 (< .001)1.568 (< .001)1.713 (< .001)0.473 (.0048)
Enjoyed2.089 (< .001)2.108 (< .001)1.560 (< .001)1.709 (< .001)0.386 (.0193)
Coordination2.504 (< .001)2.750 (< .001)1.847 (< .001)1.769 (< .001)0.505 (.0028)

We tested the number of recipes made. DCEC(65K) differed significantly from all five compared agents. For the survey responses, all comparisons were significant after Holm correction except DCEC(65K) versus DCEC(8K) on Consistent.

Q4. Does CEC’s GSNR include the shared-trunk parameters? Could you report policy-gradient GSNR, or relate GSNR to cross-play reward across seeds and batch sizes?

We apologize for an error in the batch-size labels reported for the GSNR analysis in the original manuscript. This correction changes the absolute batch-size labels but does not affect the controlled comparison or the conclusions of the analysis. All results below are reported using the corrected batch sizes.

MethodBatch SizeValue GSNR (×10⁻³)XP 5 FixedXP 100 PCG
CEC12K1.09 ± 0.0539.5 ± 2.548.7 ± 3.0
25K0.79 ± 0.0234.7 ± 2.549.1 ± 3.8
51K0.77 ± 0.0634.4 ± 0.955.8 ± 0.5
102K1.17 ± 0.1719.7 ± 0.528.6 ± 0.4
DCEC12K1.25 ± 0.03133.0 ± 9.3179.5 ± 9.4
25K1.00 ± 0.04200.1 ± 2.3256.6 ± 1.9
51K1.27 ± 0.07236.5 ± 2.3266.0 ± 1.5
102K2.25 ± 0.08200.6 ± 2.8252.5 ± 0.7

CEC value’s GSNR includes the shared trunk and value-head parameters. In contrast, DCEC value-gradient GSNR includes the decoupled value-trunk and value-head parameters.

DCEC generally achieves both higher value-gradient GSNR and better cross-play performance than CEC. As batch size increases, DCEC's value-gradient GSNR increases, while CEC's value-gradient does not show the same trend. At 102K, DCEC achieves its highest GSNR, but its cross-play reward decreases. Since these experiments are limited to 300M environment steps, the 102K setting receives fewer parameter updates and may not have fully converged.

We also computed policy-gradient GSNR. The full results across batch sizes and seeds are provided in Appendix E.3.

Q5. Given the high variance in human play styles and proficiency, how did you ensure the 20-participant human study was adequately powered? Could you provide more details on the participant demographics and skill distribution, or are there plans to expand the study to validate these claims on a broader population?

TL;DR We expanded the study to 40 participants and added demographic, experience, and skill information to better characterize participant diversity.

As described in W3, we recruited 20 additional participants, increasing the total sample size from 20 to 40, and added more detailed participant demographic information to the manuscript.

We conducted an additional background survey with both the original and newly recruited participants. Using 5-point Likert scales, we collected information on participants' prior experience with Overcooked and self-reported skill in similar cooperative games, and added these distributions to Appendix F.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.