- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Synergistic Dependency Maximization (SDM), a framework to encourage cooperation between agents by exploiting task-relevant information. The paper draws from PID and defines SD as the difference between return-action information conditioned on the state and teammates’ actions, and weighted return-action information conditioned on the state. The paper formalizes an estimation of the synergy intended to be conservative using InfoNCE and CLUB estimators, and an optimization scheme for on-policy MARL. Empirical evaluations show the effectiveness of the SDM framework on a set of MPE, MAMuJoCo, SMAC, and SMACv2 problems.
Critical strengths
- The derivation of informational synergy between agents from partial information decomposition is clear; subtracting individual dependence from conditional MI is conceptually sound, yielding synergy minus redundancy when .
- The authors integrate SDM into HAPPO and MAPPO, demonstrating empirical gains over the baselines. These gains are substantial in some cases, such as on the SMAC 3s_vs_4z environment.
Critical weaknesses
- One main concern is that defining synergistic dependency as a way to capture genuine behavioral cooperation between agents is not always correct. The SD could be non-zero without the agents needing to strategically cooperate to get higher returns. Consider a one-step game with independent uniform binary actions and an entirely separable reward: ; we obtain . It follows that with . This means SD is positive, even though each agent could maximize their reward separately by choosing action . In other words, knowing teammates’ actions can make the team return more predictable but does not necessarily imply synergistic cooperation. This is a main claim of the behavioral interpretation of SD.
- The conservativeness of the estimation of SD is central to avoiding spurious synergy. However, the reliability of the conservative estimation strategy and whether it carries over to policy updates remain unclear outside of the scope of theoretical discussions and the motivating example. More on this in questions 1 and 2.
Although the gains in the paper are promising, the claimed interpretation and reliability of the synergy need further justification.
Questions for authors
- The paper estimates the individual dependency’s upper bound using variational CLUB, and the conditional return density model is Gaussian (Appendix B.1). What evidence supports this choice in the experiments? The vCLUB could underestimate the individual dependency, which could shrink the individual dependence term and consequently inflate the value of SD.
- In the algorithm, the individual agents’ signals are rectified to before aggregation and policy optimization. Even if raw estimates are conservative on average, this does not establish that their positive parts remain conservative. Could the authors elaborate on whether this could introduce spurious synergy bonuses?
- The authors suggest that tuning below 1 could improve performance on some tasks. Could the authors report the final values used for the main experiments? values closer to 0 no longer isolate synergy beyond redundancy. How do the authors justify having better performance at lower values of ?
4 comments
Thank you for your positive assessment and constructive feedback
[Weakness 2 & Q1-Q2] Reliability of conservative estimation and its use in policy updates.
Q1. We adopt a Gaussian variational conditional density for the scalar return, following the parameterization used in the original CLUB paper. We agree that an inaccurate variational approximation can underestimate individual dependency and consequently inflate SD. The theoretical upper-bound property of vCLUB holds under the corresponding variational conditions. We train the conditional density by maximum likelihood to reduce approximation error, although this alone does not guarantee that these conditions are always satisfied.
First, Table 2 and Figure 3(b) compare the estimates with analytical quantities. The estimates closely match the analytical CLUB values, and Figure 3(b) shows that vCLUB remains above the true individual dependency in the evaluated setting, avoiding SD inflation caused by underestimating this term.
Second, Figure 7(e) compares alternative estimators during MARL policy learning, showing that InfoNCE–CLUB achieves strong and stable performance. Figures 7(f–h) further examine sensitivity to estimator capacity, the number of negative samples, and the number of estimator updates, supporting robustness across the tested configurations.
Third, we directly examine the Gaussian modeling choice by replacing the single-Gaussian conditional density with a five-component Gaussian mixture while keeping all other settings identical. The final-five returns on Walker2d 6×1 are:
| Method | Walker2d 6×1 |
|---|---|
| HAPPO+SDM-Gaussian | 4568.77 ± 613.47 |
| HAPPO+SDM-Mixture (K=5) | 4519.48 ± 588.01 |
The more expressive mixture model does not improve the mean final-five return in this setting, supporting the single-Gaussian model as a practical choice. This experiment is reported in Appendix E.7 (Effect of the Conditional Density Model, lines 1326–1342).
Moreover, gaussian conditional parameterization for vCLUB has also been adopted in MARL policy learning, including the official implementation of PMIC [1], which estimates the mutual information between global states and joint actions. Our use of vCLUB to guide policy updates therefore follows existing MARL practice. Our comparisons with analytical values, estimator sensitivity analyses, and policy-performance gains further provide complementary empirical support for this choice at both the estimation and policy-learning levels.
[1] PMIC: Improving Multi-Agent Reinforcement Learning with Progressive Mutual Information Collaboration , ICML 2022
Q2. When actions are sampled independently conditional on the state and , the true SD is non-negative. Specifically, using the mutual information identity and , we obtain $ D_t^i=I(A_t^i;A_t^{-i}\mid S_t,Y_t)+(1-\beta)I(Y_t;A_t^i\mid S_t)\ge0. $ This is also reflected in the non-negative analytical SD values across synergy levels in Figure 2(c-d). We revised the manuscript (lines 247–251) to clarify this non-negativity property of the true SD and explain why we use the positive part of its estimate. Taking the positive part is therefore consistent with the non-negativity of true SD. If a raw estimate lower-bounds the true non-negative SD, taking its positive part preserves this lower-bound property: a negative estimate is raised only to zero, which remains no greater than the true SD.
Thanks to the authors for addressing my concerns and the added experiments. I will accordingly raise my rating to 3.
Regarding Q2, one subtlety remains: even if the raw SD estimates are conservative in expectation, clipping their negative values can still produce inflated bonuses. The non-negativity of the true SD does not rule this out.
Thank you for your continued consideration and for raising your rating. We sincerely appreciate your positive assessment of our work and your recognition of the additional experiments and clarifications.
Regarding Q2, we agree that clipping negative estimates may introduce an upward bias. Sample-level information-theoretic scores have been widely used as auxiliary learning signals in MARL, as in EITI [1], CDS [2], FoX [3], and PMIC [4]. Following this practice, SDM clips the agent-wise sample-level SD estimates before aggregating them into a team-level signal. This choice is motivated by the theoretical non-negativity of the SD and also serves a practical purpose: preventing negative scores from canceling positive SD signals from other agents, thereby preserving the intended encouragement of high-SD behaviors while potentially mitigating the impact of estimation error.
[1] Influence-Based Multi-Agent Exploration. ICLR 2020.
[2] Celebrating Diversity in Shared Multi-Agent Reinforcement Learning.NeurIPS 2021.
[3] FoX: Formation-Aware Exploration in Multi-Agent Reinforcement Learning. AAAI 2024.
[4] PMIC: Improving Multi-Agent Reinforcement Learning with Progressive Mutual Information Collaboration. ICML 2022.
Once again, we sincerely thank you for your time, consideration, and recognition of our work.
Thank you for your positive assessment and constructive feedback
We appreciate the time and effort dedicated to evaluating our work. The comments have helped us clarify our contributions and improve the presentation. Below, we address each concern in detail.
Behavioral cooperation, as humans understand it, means agents working together to complete a task, i.e., obtaining high rewards. However, this is an abstract concept: the degree to which agents work together is reflected in the degree of task completion. Therefore, the best measure of such real cooperation is actually the reward function. Therefore, researchers do not simply define another variable equivalent to the task reward to encourage cooperative exploration. Like many influence-based coordination studies, our goal is to develop an effective learning signal for cooperation rather than a universal definition of multi-agent cooperation. For example, the measures designed by existing methods in Table 1 of the related work, I(At+1−i;Ati∣St,At−i) and I(St+1;Ati∣St,At−i), are intended to capture how agents work together, or their dependencies, without inherently distinguishing whether these lead to higher or lower rewards. This follows the same principle as many RL exploration algorithms: an auxiliary reward encourages exploration toward novel states or stronger inter-agent dependencies, while task-driven optimization uses the resulting experience to reinforce behaviors that improve team performance.
The synergistic dependency defined in this work follows the same rationale as previous methods. We do not regard SD as a sufficient criterion for genuine behavioral cooperation. Specifically, SD=I(Yt;Ati∣St,At−i)−I(Yt;Ati∣St) measures the extent to which the influence of an agent’s action on task-relevant outcomes depends on its teammates’ action context. It thus captures action dependencies revealed by task outcomes, without inherently distinguishing whether these dependencies lead to higher or lower task rewards. We believe that these dependencies can help the algorithm discover action combinations with stronger inter-agent dependencies, thereby potentially promoting cooperation.
In the ADD toy, Y is not determined solely by a single agent’s action. From this perspective, ADD is not entirely devoid of cooperation, unlike the Individual case in Figure 2(a), where SD is zero.Therefore, a positive SD for ADD is reasonable. Under independent uniform binary actions, without noise and with β=1, ADD yields D=0.5 bit. For the Continuous Synergy Transition example introduced in this work (Figure 2(b)), Figure 3(c) shows the true D in nats across different synergy levels (ρ). Under independent uniform binary actions and without noise, as ρ increases from 0 to 1, D also increases from 0 to 1 bit. Thus, the ADD toy exhibits an intermediate level of synergistic dependency, below the 1 bit achieved by the purely synergistic XOR case. This example therefore does not invalidate our design rationale of using synergistic dependency to help the algorithm discover action combinations with stronger inter-agent dependencies. Our experimental results also demonstrate the performance benefits of this approach.
In Sec. 4.1 (Motivating Example, lines 365–373), we added a paragraph titled Interpretation of SD. We use the ADD example to clarify the relationship between SD and behavioral cooperation and explain how SD serves as an auxiliary learning signal for discovering task-relevant action dependencies.
The final β values for HAPPO+SDM are listed below. We use β=1 on 11 of the 19 tasks.
As discussed in Sec. 3.1, synergy beyond redundancy captures net complementary information and motivates our SD formulation (Eq. 3). Eq. 4 introduces a tunable relaxation:
$ SD = I(Y; A^i \mid S, A^{-i}) - \beta I(Y; A^i \mid S). $
With β=1, SD measures the additional action–return dependency revealed by teammate context beyond individual dependency. However, tasks may benefit from both individual contributions and synergistic interactions, so fully subtracting individual dependency may not provide the most useful learning signal. A smaller β retains more individual task-relevant information and may therefore improve performance in tasks requiring both effective individual actions and coordination with teammates.