acceptodds
Under review as a conference paper at ICLR 2027

SDM: Synergistic Dependency Maximization for Cooperative Multi-Agent Reinforcement Learning

Abstract

Cooperative multi-agent reinforcement learning requires agents to develop complementary behaviors whose task value emerges through interaction with their teammates. A fundamental challenge is to quantify task-relevant synergy arising from these interactions and translate it into an effective signal for policy learning. To address this challenge, we introduce Synergistic Dependency Maximization (SDM), which promotes task-relevant interaction coupling by encouraging an agent's action to become more informative about the multi-step team return when its teammates' actions are taken into account. Using Partial Information Decomposition (PID), we formalize synergistic dependency through synergy beyond redundancy, yielding a principled task-relevant coordination criterion for multi-agent interactions. To estimate this dependency reliably, SDM adopts a conservative estimation strategy and uses the resulting signal to guide policy optimization while retaining the original task objective for value estimation. Experiments across MPE, MAMuJoCo, SMAC, and SMACv2 demonstrate consistent improvements over strong multi-agent policy optimization baselines and existing explicit coordination objectives. These results establish synergistic dependency as a principled objective for promoting cooperative behavior.

Primary area:
reinforcement learning
Keywords:
Cooperative Multi-Agent Reinforcement Learning, Multi-Agent Coordination, Synergistic Dependency, Partial Information Decomposition, Information-Theoretic Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_x2A2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Synergistic Dependency Maximization (SDM), a framework to encourage cooperation between agents by exploiting task-relevant information. The paper draws from PID and defines SD as the difference between return-action information conditioned on the state and teammates’ actions, and weighted return-action information conditioned on the state. The paper formalizes an estimation of the synergy intended to be conservative using InfoNCE and CLUB estimators, and an optimization scheme for on-policy MARL. Empirical evaluations show the effectiveness of the SDM framework on a set of MPE, MAMuJoCo, SMAC, and SMACv2 problems.

Critical strengths
  1. The derivation of informational synergy between agents from partial information decomposition is clear; subtracting individual dependence from conditional MI is conceptually sound, yielding synergy minus redundancy when β=1\beta = 1.
  2. The authors integrate SDM into HAPPO and MAPPO, demonstrating empirical gains over the baselines. These gains are substantial in some cases, such as on the SMAC 3s_vs_4z environment.
Critical weaknesses
  1. One main concern is that defining synergistic dependency as a way to capture genuine behavioral cooperation between agents is not always correct. The SD could be non-zero without the agents needing to strategically cooperate to get higher returns. Consider a one-step game with independent uniform binary actions and an entirely separable reward: Y=A1+A2Y=A^1+A^2; we obtain I(Y;A1∣A2)=1 bit,I(Y;A1)=0.5 bitI(Y;A^1\mid A^2)=1\text{ bit}, I(Y;A^1)=0.5\text{ bit}. It follows that D1=0.5 bitD^1=0.5\text{ bit} with β=1\beta=1. This means SD is positive, even though each agent could maximize their reward separately by choosing action 11. In other words, knowing teammates’ actions can make the team return more predictable but does not necessarily imply synergistic cooperation. This is a main claim of the behavioral interpretation of SD.
  2. The conservativeness of the estimation of SD is central to avoiding spurious synergy. However, the reliability of the conservative estimation strategy and whether it carries over to policy updates remain unclear outside of the scope of theoretical discussions and the motivating example. More on this in questions 1 and 2.

Although the gains in the paper are promising, the claimed interpretation and reliability of the synergy need further justification.

Questions for authors
  1. The paper estimates the individual dependency’s upper bound using variational CLUB, and the conditional return density model is Gaussian (Appendix B.1). What evidence supports this choice in the experiments? The vCLUB could underestimate the individual dependency, which could shrink the individual dependence term and consequently inflate the value of SD.
  2. In the algorithm, the individual agents’ signals D^ti\hat{D}^i_t are rectified to [D^ti]+[\hat{D}^i_t]_+ before aggregation and policy optimization. Even if raw estimates are conservative on average, this does not establish that their positive parts remain conservative. Could the authors elaborate on whether this could introduce spurious synergy bonuses?
  3. The authors suggest that tuning β\beta below 1 could improve performance on some tasks. Could the authors report the final β\beta values used for the main experiments? β\beta values closer to 0 no longer isolate synergy beyond redundancy. How do the authors justify having better performance at lower values of β\beta?
4 comments
AuthorsOpenReview ↗

Thank you for your positive assessment and constructive feedback

We appreciate the time and effort dedicated to evaluating our work. The comments have helped us clarify our contributions and improve the presentation. Below, we address each concern in detail.

[Weakness 1] Relationship between SD and real behavioral cooperation. The ADD toy example (Y=A1+A2Y = A^1 + A^2)

  1. Behavioral cooperation, as humans understand it, means agents working together to complete a task, i.e., obtaining high rewards. However, this is an abstract concept: the degree to which agents work together is reflected in the degree of task completion. Therefore, the best measure of such real cooperation is actually the reward function. Therefore, researchers do not simply define another variable equivalent to the task reward to encourage cooperative exploration. Like many influence-based coordination studies, our goal is to develop an effective learning signal for cooperation rather than a universal definition of multi-agent cooperation. For example, the measures designed by existing methods in Table 1 of the related work, I(At+1−i;Ati∣St,At−i)I(A_{t+1}^{-i}; A_t^i \mid S_t, A_t^{-i}) and I(St+1;Ati∣St,At−i)I(S_{t+1}; A_t^i \mid S_t, A_t^{-i}), are intended to capture how agents work together, or their dependencies, without inherently distinguishing whether these lead to higher or lower rewards. This follows the same principle as many RL exploration algorithms: an auxiliary reward encourages exploration toward novel states or stronger inter-agent dependencies, while task-driven optimization uses the resulting experience to reinforce behaviors that improve team performance.

  2. The synergistic dependency defined in this work follows the same rationale as previous methods. We do not regard SD as a sufficient criterion for genuine behavioral cooperation. Specifically, SD=I(Yt;Ati∣St,At−i)−I(Yt;Ati∣St)SD=I(Y_t;A_t^i\mid S_t,A_t^{-i})-I(Y_t;A_t^i\mid S_t) measures the extent to which the influence of an agent’s action on task-relevant outcomes depends on its teammates’ action context. It thus captures action dependencies revealed by task outcomes, without inherently distinguishing whether these dependencies lead to higher or lower task rewards. We believe that these dependencies can help the algorithm discover action combinations with stronger inter-agent dependencies, thereby potentially promoting cooperation.

  3. In the ADD toy, YY is not determined solely by a single agent’s action. From this perspective, ADD is not entirely devoid of cooperation, unlike the Individual case in Figure 2(a), where SD is zero.Therefore, a positive SD for ADD is reasonable. Under independent uniform binary actions, without noise and with β=1\beta=1, ADD yields D=0.5D=0.5 bit. For the Continuous Synergy Transition example introduced in this work (Figure 2(b)), Figure 3(c) shows the true DD in nats across different synergy levels (ρ\rho). Under independent uniform binary actions and without noise, as ρ\rho increases from 0 to 1, DD also increases from 0 to 1 bit. Thus, the ADD toy exhibits an intermediate level of synergistic dependency, below the 1 bit achieved by the purely synergistic XOR case. This example therefore does not invalidate our design rationale of using synergistic dependency to help the algorithm discover action combinations with stronger inter-agent dependencies. Our experimental results also demonstrate the performance benefits of this approach.

    In Sec. 4.1 (Motivating Example, lines 365–373), we added a paragraph titled Interpretation of SD. We use the ADD example to clarify the relationship between SD and behavioral cooperation and explain how SD serves as an auxiliary learning signal for discovering task-relevant action dependencies.

[Q3] Final β values and the interpretation of improved performance with β < 1.

The final β values for HAPPO+SDM are listed below. We use β=1\beta=1 on 11 of the 19 tasks.

βTasks (HAPPO + SDM)
0.25simple_reference (continuous), HalfCheetah_6x1
0.5Ant_4x2
0.75terran_5_vs_5, zerg_5_vs_5
1simple_reference (discrete), simple_speaker_listener (continuous/discrete), simple_spread (continuous/discrete), Walker2d_6x1, Walker2d_3x2, Walker2d_2x3, Ant_2x4, manyagent_swimmer_10x2, 3s_vs_4z, 3s_vs_5z, 5m_vs_6m, protoss_5_vs_5

As discussed in Sec. 3.1, synergy beyond redundancy captures net complementary information and motivates our SD formulation (Eq. 3). Eq. 4 introduces a tunable relaxation:

$ SD = I(Y; A^i \mid S, A^{-i}) - \beta I(Y; A^i \mid S). $

With β=1\beta=1, SD measures the additional action–return dependency revealed by teammate context beyond individual dependency. However, tasks may benefit from both individual contributions and synergistic interactions, so fully subtracting individual dependency may not provide the most useful learning signal. A smaller β\beta retains more individual task-relevant information and may therefore improve performance in tasks requiring both effective individual actions and coordination with teammates.

AuthorsOpenReview ↗

Thank you for your positive assessment and constructive feedback

[Weakness 2 & Q1-Q2] Reliability of conservative estimation and its use in policy updates.

Q1. We adopt a Gaussian variational conditional density for the scalar return, following the parameterization used in the original CLUB paper. We agree that an inaccurate variational approximation can underestimate individual dependency and consequently inflate SD. The theoretical upper-bound property of vCLUB holds under the corresponding variational conditions. We train the conditional density by maximum likelihood to reduce approximation error, although this alone does not guarantee that these conditions are always satisfied.

First, Table 2 and Figure 3(b) compare the estimates with analytical quantities. The estimates closely match the analytical CLUB values, and Figure 3(b) shows that vCLUB remains above the true individual dependency in the evaluated setting, avoiding SD inflation caused by underestimating this term.

Second, Figure 7(e) compares alternative estimators during MARL policy learning, showing that InfoNCE–CLUB achieves strong and stable performance. Figures 7(f–h) further examine sensitivity to estimator capacity, the number of negative samples, and the number of estimator updates, supporting robustness across the tested configurations.

Third, we directly examine the Gaussian modeling choice by replacing the single-Gaussian conditional density with a five-component Gaussian mixture while keeping all other settings identical. The final-five returns on Walker2d 6×1 are:

MethodWalker2d 6×1
HAPPO+SDM-Gaussian4568.77 ± 613.47
HAPPO+SDM-Mixture (K=5)4519.48 ± 588.01

The more expressive mixture model does not improve the mean final-five return in this setting, supporting the single-Gaussian model as a practical choice. This experiment is reported in Appendix E.7 (Effect of the Conditional Density Model, lines 1326–1342).

Moreover, gaussian conditional parameterization for vCLUB has also been adopted in MARL policy learning, including the official implementation of PMIC [1], which estimates the mutual information between global states and joint actions. Our use of vCLUB to guide policy updates therefore follows existing MARL practice. Our comparisons with analytical values, estimator sensitivity analyses, and policy-performance gains further provide complementary empirical support for this choice at both the estimation and policy-learning levels.

[1] PMIC: Improving Multi-Agent Reinforcement Learning with Progressive Mutual Information Collaboration , ICML 2022

Q2. When actions are sampled independently conditional on the state and 0≤β≤10\le\beta\le1, the true SD is non-negative. Specifically, using the mutual information identity and I(Ati;At−i∣St)=0I(A_t^i;A_t^{-i}\mid S_t)=0, we obtain $ D_t^i=I(A_t^i;A_t^{-i}\mid S_t,Y_t)+(1-\beta)I(Y_t;A_t^i\mid S_t)\ge0. $ This is also reflected in the non-negative analytical SD values across synergy levels in Figure 2(c-d). We revised the manuscript (lines 247–251) to clarify this non-negativity property of the true SD and explain why we use the positive part of its estimate. Taking the positive part is therefore consistent with the non-negativity of true SD. If a raw estimate lower-bounds the true non-negative SD, taking its positive part preserves this lower-bound property: a negative estimate is raised only to zero, which remains no greater than the true SD.

Thanks to the authors for addressing my concerns and the added experiments. I will accordingly raise my rating to 3.

Regarding Q2, one subtlety remains: even if the raw SD estimates are conservative in expectation, clipping their negative values can still produce inflated bonuses. The non-negativity of the true SD does not rule this out.

Thank you for your continued consideration and for raising your rating. We sincerely appreciate your positive assessment of our work and your recognition of the additional experiments and clarifications.

Regarding Q2, we agree that clipping negative estimates may introduce an upward bias. Sample-level information-theoretic scores have been widely used as auxiliary learning signals in MARL, as in EITI [1], CDS [2], FoX [3], and PMIC [4]. Following this practice, SDM clips the agent-wise sample-level SD estimates before aggregating them into a team-level signal. This choice is motivated by the theoretical non-negativity of the SD and also serves a practical purpose: preventing negative scores from canceling positive SD signals from other agents, thereby preserving the intended encouragement of high-SD behaviors while potentially mitigating the impact of estimation error.

[1] Influence-Based Multi-Agent Exploration. ICLR 2020.

[2] Celebrating Diversity in Shared Multi-Agent Reinforcement Learning.NeurIPS 2021.

[3] FoX: Formation-Aware Exploration in Multi-Agent Reinforcement Learning. AAAI 2024.

[4] PMIC: Improving Multi-Agent Reinforcement Learning with Progressive Mutual Information Collaboration. ICML 2022.

Once again, we sincerely thank you for your time, consideration, and recognition of our work.

Reviewer_FYBQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes SDM, an actor-side bonus for on-policy CTDE methods (MAPPO/HAPPO) built on “synergistic dependency”, defined as I(Y; Aⁱ | S, A⁻ⁱ) − β·I(Y; Aⁱ | S) for an H-step team return Y and motivated via partial information decomposition. The terms are estimated conservatively (InfoNCE lower bound minus CLUB upper bound), then rectified, averaged across agents, gated by A_env > 0, and added to the advantage. It is validated on a binary XOR toy and on MPE, MAMuJoCo, SMAC and SMACv2, with gains over the backbones and over two influence-style baselines.

Critical strengths

SDM is a plug-in actor-side signal that leaves the critic and decentralized execution untouched. Its gains on MAMuJoCo are positive in all 12 backbone comparisons (Table 3).

Critical weaknesses

While the paper offers a clean information-theoretic framing, its weakness is twofold: the criterion does not isolate coordination, and the empirical evidence is too noisy to support the claimed gains.

First, with β = 1 and actions sampled independently given S, SD equals I(Aⁱ; A⁻ⁱ | S, Y), which is non-negative. It is therefore positive for any return depending on both actions, including purely additive, non-coordinating ones. For Y = A¹ + A² with independent binary actions, I(Y;A¹|A²) = 1 bit and I(Y;A¹) = 0.5 bit, so SD ≈ 0.35 nats. It also scores harmful interaction (XOR miscoordination) the same as beneficial interaction, so the A_env > 0 gate does the real work. The toy in Sec. 4.1 never tests an additive structure, which is the typical case under a shared reward. The paper’s own diagnostics point the same way: I_ind/I_int is about 3–4% (Fig. 8c), so SD ≈ I(Y;Aⁱ|S,A⁻ⁱ), and the β ablation (Fig. 7c–d) looks flat. The “conservative” property holds only in expectation at an optimal critic or density, and even an ideal CLUB in the toy is 2.2× the true value (0.416 vs 0.193). No result connects maximizing SD to higher team return.

Second, there are three seeds (App. D.1). On MPE (Table 6), four of the ten “improvements” are ≤0.04 against stds of 0.8–3.3, so “10/12” overstates the result. On MAMuJoCo, stds overlap (e.g., Walker2d 3x2: 4597 ± 2015), and the ablation error bars in Fig. 7 appear to overlap. SMAC and SMACv2 compare only HAPPO vs. HAPPO+SDM on three maps, AA/AS appear only on MAMuJoCo, and the closest method (MAGIC) is excluded. SDM’s λ and β are tuned per task, and it is unclear whether the baselines get the same budget. There is also no control for the simple explanation that a non-negative, A>0-gated bonus just inflates positive advantages. The mechanism analysis is partly circular, since the policy is trained to raise the estimator’s own output, and the independent post-hoc check covers one task.

Questions for authors

Q1. Under independently sampled actions, SD = I(Aⁱ; A⁻ⁱ | S, Y) ≥ 0. What do [·]₊ and “conservative” then do beyond handling estimator noise? What is SD on an additive-return toy (Y = A¹ + A² + noise), where no coordination is required?

Q2. With I_ind/I_int at about 3–4%, and β = 0 vs. β = 1 apparently indistinguishable, does SDM reduce to an I(Y;Aⁱ|S,A⁻ⁱ) bonus? What does the “synergy beyond redundancy” interpretation add empirically?

4 comments
AuthorsOpenReview ↗

We thank the reviewers for their constructive feedback and thoughtful suggestions.

We appreciate the time and effort dedicated to evaluating our work. The comments have helped us clarify our contributions and improve the presentation. Below, we address each concern in detail.

[Weakness 1] First, with β = 1 and actions sampled independently given S, SD equals I(Aⁱ; A⁻ⁱ | S, Y), which is non-negative. It is therefore positive for any return depending on both actions, including purely additive, non-coordinating ones. For Y = A¹ + A² with independent binary actions, I(Y;A¹|A²) = 1 bit and I(Y;A¹) = 0.5 bit, so SD ≈ 0.35 nats.

We agree that SD is non-negative under these assumptions, as illustrated in Figure 3(c-d). The figure reports noisy SD in nats; without noise, SD increases from 0 to 1 bit from Individual to XOR. Although ADD requires no strategic coordination, both agents contribute to the shared return, and knowing the teammate’s action makes the individual action more informative about that return. From this perspective, ADD is not entirely devoid of cooperation. Moreover, under independent uniform binary actions, SD is lower for ADD (0.5 bit) than for noiseless XOR (1 bit).

We do not regard SD as a sufficient criterion for behavioral cooperation. Like novelty-based exploration and influence-based coordination methods (Table 1), SD provides an auxiliary signal favoring potentially useful behaviors. Specifically, it encourages stronger information-theoretic synergistic dependency to help agents discover cooperative action combinations, while task-driven optimization reinforces those that improve team performance.

[Weakness 2] It also scores harmful interaction (XOR miscoordination) the same as beneficial interaction, so the A_env > 0 gate does the real work.

SD measures statistical dependency, without a preference for high over low returns. In our symmetric XOR setting, samples have the same ideal SD score because they exhibit the same dependency structure. The Continuous Synergy Transition experiment (Figures 2b and 3c) examines whether SD captures different synergy strengths ρ. SD and the advantage gate are complementary: the former measures dependency strength, while the latter aligns its reinforcement with the task objective.

More generally, auxiliary exploration signals need not directly measure task benefit. Novelty-based exploration and influence-based coordination methods likewise introduce an exploration bias, while task-driven optimization reinforces behaviors that improve performance. SD follows this rationale by favoring information-theoretic synergistic dependency.

[Weakness 3] The toy in Sec. 4.1 never tests an additive structure, which is the typical case under a shared reward.

In Sec. 4.1 (Motivating Example, lines 365–373), we added a paragraph titled Interpretation of SD. We use the ADD example to clarify the relationship between SD and behavioral cooperation and explain how SD serves as an auxiliary learning signal for discovering task-relevant action dependencies.

In cooperative MARL, a shared reward typically reflects team performance and generally cannot be expressed as a sum of independent contributions from individual actions.

[Weakness 4] The paper’s own diagnostics point the same way: I_ind/I_int is about 3–4% (Fig. 8c), so SD ≈ I(Y;Aⁱ|S,A⁻ⁱ), and the β ablation (Fig. 7c–d) looks flat.

The 3–4% ratio in Fig. 8(c) is a post-hoc estimate of the expected Iind/IintI_{\rm ind}/I_{\rm int} ratio for the final policies on 3s_vs_4z.Final policies typically exhibit stronger synergistic dependency, so a smaller Iind/IintI_{\rm ind}/I_{\rm int} ratio in is expected. Figure 8(b) shows that the individual correction exceeds 20% of IintI_{\rm int} for a larger fraction of samples early in training. Although performance varies modestly with β\beta in Fig. 7(c–d), omitting IindI_{\rm ind} lowers mean return on both backbones in Fig. 7(a). These results support its role as a selective correction, especially early in training.

[Weakness 5] The “conservative” property holds only in expectation at an optimal critic or density, and even an ideal CLUB in the toy is 2.2× the true value (0.416 vs 0.193).

In Appendix D.3, we derive the ideal CLUB upper bound of 0.416 nats for the Individual case, where the true MI is 0.193 nats. This gap persists even with the true conditional distribution and thus reflects the looseness of the bound. We use CLUB because underestimating individual dependency would inflate the estimated SD.

[Weakness 6] No result connects maximizing SD to higher team return.

Our comparative studies, ablations, and mechanistic experiments support SDM’s contribution to improved performance.

AuthorsOpenReview ↗

We thank the reviewers for their constructive feedback and thoughtful suggestions.

[Weakness 7] On MPE (Table 6), four of the ten “improvements” are ≤0.04 against stds of 0.8–3.3, so “10/12” overstates the result.

We have revised the wording in lines 401–404 to clarify that “10/12” refers to higher mean returns. The revised sentence reads: “SDM yields higher mean returns in 10 of 12 backbone comparisons, with more pronounced gains on discrete Reference and on both the continuous and discrete variants of Spread.”

[Weakness 8] On MAMuJoCo, stds overlap (e.g., Walker2d 3x2: 4597 ± 2015), and the ablation error bars in Fig. 7 appear to overlap.

Overlapping standard-deviation bars alone do not rule out a difference between methods. SDM yields higher mean returns for both HAPPO and MAPPO on all six tasks in Table 3. Figure 5(d) also shows a higher normalized IQM for both backbones with SDM across the seven MAMuJoCo tasks.

The ablations show a consistent pattern across backbones. On Walker2d 6×1, removing either the individual-dependency correction or the advantage gate lowers mean return for both HAPPO and MAPPO (Fig. 7(a)).

In future work, we will consider using more random seeds to improve the precision of these estimates.

[Weakness 9] SMAC and SMACv2 compare only HAPPO vs. HAPPO+SDM on three maps, AA/AS appear only on MAMuJoCo, and the closest method (MAGIC) is excluded.

We evaluated SDM as a plug-in for both MAPPO and HAPPO on MPE and MAMuJoCo. Given HAPPO’s strong performance in these experiments, we selected it as the backbone for further validation on three SMAC maps and three SMACv2 scenarios. Figure 6(a–c) also reports additional on-policy baselines on SMAC.

On six MAMuJoCo tasks, we replaced SD with representative action–action (AA) and action–state (AS) influence signals while keeping the HAPPO backbone and advantage gate fixed. This controlled comparison isolates the choice of coordination signal (Table 1). MAGIC measures how an agent’s action influences its teammates’ future states and thus belongs to the AS family. At the time of our experiments, MAGIC was available only as a preprint and had no public implementation. We therefore compared SD with a representative AS signal rather than with MAGIC’s full algorithm.

[Weakness 10] SDM’s λ and β are tuned per task, and it is unclear whether the baselines get the same budget.

We used the standard HARL configurations for MAPPO and HAPPO and kept their backbone hyperparameters fixed when adding SDM. SDM received additional per-task tuning only for λ and β. We believe this controlled comparison provides evidence that adding SDM improves performance over the standard backbones under the same interaction budget per run.

[Weakness 11] There is also no control for the simple explanation that a non-negative, A>0-gated bonus just inflates positive advantages.

We tested this explanation directly in the w/o SD ablation by replacing SD with a constant bonus while retaining the Aenv>0A_{\rm env}>0 gate. On Walker2d 6×1, uniformly increasing positive advantages lowered overall mean return relative to the baseline for both backbones.

BackboneBaselineConstant bonus with Aenv>0A_{\rm env}>0 gate
HAPPO2970.01 ± 824.962513.54 ± 604.59
MAPPO1685.23 ± 203.441446.06 ± 284.64

The gated constant bonus alone reduced mean performance on this task. Together with the gate ablation in Fig. 7(a), these results support SD as the source of SDM’s gains and show that the gate further improves mean return by directing the SD bonus toward samples with positive environment advantages.

We revised the discussion (lines 459–464) and Fig. 7(a), and report the constant-bonus control in Appendix E.5 (Constant-Bonus Control for Advantage Gating).

[Weakness 12] The mechanism analysis is partly circular, since the policy is trained to raise the estimator’s own output, and the independent post-hoc check covers one task.

Figures 8(a–b) track the SD signal during training. To address the circularity concern, we froze both policies and trained a new evaluator from scratch on a balanced mixture of fresh trajectories from both policies (App. E.3). On held-out episodes from 3s_vs_4z, this shared evaluator measured higher mean SD for HAPPO+SDM than for HAPPO (1.50 vs. 1.19).

We extended the same post-hoc check to Walker2d 6×1. The independent evaluator again measured higher mean SD and a lower Iind/IintI_{\mathrm{ind}}/I_{\mathrm{int}} ratio for HAPPO+SDM.

TaskMethodPost-hoc SDIind/IintI_{\mathrm{ind}}/I_{\mathrm{int}}Mean return
Walker2d 6×1HAPPO0.958 ± 0.19910.77% ± 1.90%4292.75 ± 671.24
Walker2d 6×1HAPPO+SDM1.153 ± 0.1546.03% ± 0.61%4568.77 ± 613.47

We added this experiment to Fig. 8(d) and revised the corresponding text in lines 481–485.

AuthorsOpenReview ↗

We thank the reviewers for their constructive feedback and thoughtful suggestions.

[Q1] Under independently sampled actions, SD = I(Ai;A−i∣S,Y)≥0I(A^i; A^{-i}\mid S,Y)\geq 0. What do [⋅]+[\cdot]_+ and “conservative” then do beyond handling estimator noise? What is SD on an additive-return toy (Y=A1+A2+noiseY=A^1+A^2+\text{noise}), where no coordination is required?

We estimate SD conservatively by subtracting an upper bound on IindI_{\mathrm{ind}} from a lower bound on IintI_{\mathrm{int}}. Even with ideal estimators, bound looseness can make this difference negative. In the Individual case in Table 2, true SD is zero, but the ideal bounds give 0.193−0.416=−0.2230.193-0.416=-0.223 nats. Applying [⋅]+[\cdot]_+ therefore assigns zero bonus instead of penalizing the sample.

For independent uniform binary actions, we use the same noise distribution as in Sec. 4.1, ϵ∼Bernoulli(0.2)\epsilon\sim\mathrm{Bernoulli}(0.2), and add it to both outputs. The additive toy Y=A1+A2+ϵY=A^1+A^2+\epsilon yields SD = 0.280 bit (0.194 nats), compared with 0.639 bit (0.443 nats) for Y=(A1⊕A2)+ϵY=(A^1\oplus A^2)+\epsilon. Under matched additive noise, ADD exhibits weaker informational dependency than XOR.

[Q2] With I_ind/I_int at about 3–4%, and β = 0 vs. β = 1 apparently indistinguishable, does SDM reduce to an I(Y;Aⁱ|S,A⁻ⁱ) bonus? What does the “synergy beyond redundancy” interpretation add empirically?

The 3–4% ratio in Fig. 8(c) is a post-hoc estimate based on the expected values of IindI_{\rm ind} and IintI_{\rm int} under the final policies on 3s_vs_4z. Figure 8(b) shows that, among samples with positive IintI_{\mathrm{int}}, the fraction for which IindI_{\mathrm{ind}} exceeds 20% of IintI_{\mathrm{int}} is higher early in training and generally declines thereafter. Thus, SDM does not generally reduce to an I(Y;Ai∣S,A−i)I(Y;A^i\mid S,A^{-i}) bonus in training.

At β=1\beta=1, Iint−IindI_{\mathrm{int}}-I_{\mathrm{ind}} measures synergy minus redundancy, where redundancy is information about the return available from either the agent’s action alone or its teammates’ actions alone. Table 2 illustrates the empirical distinction. IintI_{\mathrm{int}} assigns 0.190 nats to both Individual and XOR. After subtracting IindI_{\mathrm{ind}} and taking the positive part, the Individual score falls to zero, while XOR retains 0.190 nats.

All main experiments use β>0\beta>0, with β=1\beta=1 in most settings. Although performance varies modestly with β\beta in Fig. 7(c–d), removing IindI_{\mathrm{ind}} lowers mean return on both backbones in Fig. 7(a). This supports the practical value of subtracting individual dependency from interaction-conditioned dependency to measure synergy beyond redundancy.

[Weakness] Second, there are three seeds (App. D.1).

Regarding the concern about the number of random seeds, three-seed evaluations have also been adopted in prior MARL studies, including [1], [2], [3], and [4]. We comprehensively evaluate SDM across 19 tasks in four environments. To ensure a fair comparison, all algorithms are evaluated on every task using the same three random seeds (1, 2, and 3). SDM achieves higher mean performance on the majority of these tasks, providing broad empirical evidence of its effectiveness and performance advantages.

[1] SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning. NeurIPS 2023.

[2] RPM: Generalizable Multi-Agent Policies for Multi-Agent Reinforcement Learning. ICLR 2023.

[3] Language Grounded Multi-agent Reinforcement Learning with Human-interpretable Communication. NeurIPS 2024.

[4] Investigating Relational State Abstraction in Collaborative MARL. AAAI 2025.

Dear Reviewer,

As the discussion period is nearing its end, we would like to follow up to see whether any of your concerns remain unaddressed. We would greatly appreciate any further questions or feedback and would be happy to provide additional clarifications to improve our work.

We sincerely hope that our responses and revisions have addressed your concerns and that you will reconsider your assessment of our submission.

Thank you again for your time and thoughtful review.

Reviewer_BEaCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SDM, an additional training signal that encourages cooperation among agents. The idea is to measure whether an agent’s action becomes more informative about the team return when its teammates’ actions are also considered. This signal is then used to guide policy updates, while the critic continues to learn from the original task rewards.

Critical strengths

I like that the coordination signal is tied to the team’s return. Simply encouraging agents to influence one another does not necessarily help the task; SDM gives a clearer reason for encouraging a particular interaction.

The method can be added to MAPPO and HAPPO without changing how agents act during execution. This makes it reasonably easy to use with existing systems.

The experiments cover four benchmarks, both discrete and continuous actions, and two underlying algorithms. The component and sensitivity studies also help explain the design choices.

Critical weaknesses

The comparisons mainly cover methods based on policy optimization. Value-decomposition methods such as QMIX and QPLEX are missing, even though they are relevant alternatives on SMAC. Comparing against them would help show whether adding SDM makes these agents competitive with other established approaches to cooperation.

SDM requires extra computation to train the dependency estimators. The paper reports this overhead, but most performance comparisons use the same number of environment interactions. It would be useful to know whether the gains remain when methods receive the same amount of training time.

Questions for authors

Can you include QMIX or QPLEX as performance baselines on SMAC, using the same environment settings and comparable interaction budgets? This would not require adding SDM to those methods.

How do the results compare when training time is held fixed, accounting for the additional estimator updates?

4 comments
AuthorsOpenReview ↗

Thank you for your positive assessment and constructive feedback

We appreciate the time and effort dedicated to evaluating our work. The comments have helped us clarify our contributions and improve the presentation. Below, we address each concern in detail.

We added the PyMARL2 implementations of QMIX and QPLEX and evaluated all four methods under the same SMAC settings and a budget of 4 million environment steps per method. The table reports mean test win rates over three seeds.

MapQMIXQPLEXHAPPOHAPPO+SDM
3s_vs_4z81.55%44.58%58.00%91.70%
3s_vs_5z42.35%53.01%4.48%37.15%
5m_vs_6m71.67%75.83%25.33%37.33%

We also compared performance when training time was held fixed. For each map, we used the wall-clock time HAPPO needed to complete 4 million steps as the reference budget. The table reports mean test win rates over three seeds under approximately matched time budgets, with completed environment steps in parentheses (millions).

MapTimeQMIXQPLEXHAPPOHAPPO+SDM
3s_vs_4z78 min19.93% (2.20)2.08% (2.00)58.00% (4.00)67.46% (2.46)
3s_vs_5z108 min22.41% (2.68)13.98% (2.35)4.48% (4.00)23.33% (2.62)
5m_vs_6m146 min64.69% (2.75)72.19% (2.36)25.33% (4.00)28.33% (2.50)

We added these experiments to Appendix E.5 (Comparison with Value-Decomposition Methods on SMAC) and Appendix E.6 (Performance under Matched Training-Time Budgets), lines 1276–1324.

[Weakness 1 & Q1] Can you include QMIX or QPLEX as performance baselines on SMAC, using the same environment settings and comparable interaction budgets?

Under the same budget of 4 million environment interactions, QMIX and QPLEX generally outperform HAPPO. QMIX and QPLEX are off-policy value-decomposition methods designed for discrete action spaces and can repeatedly reuse past experience to improve performance. HAPPO is an on-policy method. For this reason, these two classes of methods are typically not compared directly. Even so, HAPPO+SDM achieves the highest mean win rate on 3s_vs_4z.

Under matched training-time budgets, the performance advantage of QMIX and QPLEX diminishes further. HAPPO+SDM achieves the highest mean win rate on both 3s_vs_4z and 3s_vs_5z.

[Weakness 2 & Q2] How do the results compare when training time is held fixed, accounting for the additional estimator updates?

When training time is fixed at the time HAPPO needs to complete 4 million environment steps, HAPPO+SDM achieves higher mean test win rates than HAPPO on all three maps, even after accounting for the additional estimator updates. These results support the advantage of SDM in accelerating convergence.

Dear Reviewer,

We sincerely appreciate the time and effort you have devoted to reviewing our work. As the discussion period is nearing its end, we hope that our responses and revisions have addressed your concerns and that you will take them into consideration in your final assessment.

Thank you again for your time and thoughtful review.

I think the experiments is more comprehensive. The authors addressed my concerns, so I maintain my score.

Thank you for your positive feedback and for acknowledging our additional experiments and clarifications. We sincerely appreciate your time and constructive comments, which have helped us improve the quality of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.