- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper analyzes two ways of aggregating information in cooperative multi-agent policy optimization: combining other agents’ rewards in the advantage, and multiplying their policy ratios in the importance weight. The authors use two support matrices to describe these choices and argue that neighboring agents’ information should be aggregated in the advantage while keeping importance ratios per-agent. They evaluate this idea through four synthetic games and a large-scale traffic signal control simulation.
Critical strengths
-
Discussing advantage aggregation and ratio aggregation separately helps clarify the connections between different multi-agent policy optimization methods and where information from other agents should enter the update.
-
When the current and old policies are identical, the paper shows that the product of the support matrices determines the expected gradient. This result, together with the variance formula for a product of independent importance ratios, provides useful insight into the two aggregation choices.
-
The experiments cover several settings, including four cooperative games and a traffic control simulation with 196 intersections.
Critical weaknesses
-
The paper shows that the two aggregation choices have the same expected gradient under specific conditions, but this does not mean they remain equivalent throughout training. The broader claim that cross-agent importance ratios offer no additional value therefore needs stronger support.
-
Greater variation in the importance ratios does not necessarily mean noisier policy updates, since the updates also depend on the rewards and their correlations. The distinction between adding rewards and multiplying ratios alone is not enough to establish that one approach always has lower variance.
-
Whether an agent’s information should be included also depends on how strongly it affects other agents, how noisy that information is, and how the rewards are correlated. The conclusion that the aggregation range should exactly match the coupling neighborhood is therefore too strong.
-
The approach relies on separate rewards for individual agents and known coupling relationships. Real-world tasks may provide only a team reward, and interactions between agents may change across states and over time, so the paper should explain how its recommendations would change in these settings.
Questions for authors
Please refer to the weaknesses.
4 comments
Response Part 2
W3: Coupling range, correlations, and the MSE-optimal radius
Thank you for your insightful comment. We replace the exact coupling-radius claim with a finite-batch MSE criterion that explicitly includes influence strength, noise, and correlation:
\frac{1}{B}\operatorname{tr}\operatorname{Cov}(G).$$ Here $G$ is the single-sample gradient estimator for the retained support, and MSE measures the error of its mean over $B$ independent samples relative to the full team gradient. Adding a component changes variance by its own variance plus twice its covariance with the retained estimator. The value of additional mean influence must therefore be considered alongside this covariance cost. The graph experiment illustrates this tradeoff: a wider neighborhood can recover omitted influence while adding noisy credit. Its value depends on whether the recovered signal outweighs the added variance and covariance. The MSE-optimal radius therefore depends on influence strength and noise, as well as the reward-dependency graph. Proposition 2 identifies when further enlargement adds only variance. Conditional on the pre-action context and the actor's action, each added processed credit has zero mean, and its residual is uncorrelated with retained residuals and the other added residuals. With credit processing, coefficients, and overall scale fixed, these additions preserve the gradient mean and add nonnegative covariance. Interactions and correlated residuals within the retained support remain allowed. The general MSE expression retains covariance so that it also captures shared future states and bootstrap estimates. The radius and MSE analysis are marked in blue in Section 3.3, with proofs in Appendices A.6–A.7 and the graph diagnostic in Appendix E.5. ## W4: Team-only rewards and changing interactions Thank you for your insightful comment. Our results target decomposable individual rewards; with an irreducible team reward, the ratio-side result still shows that multiplying ratios adds no first-order credit when all agents already use the same global advantage. When interactions vary across states or time, the useful credit support becomes context-dependent, so radius selection is a separate problem. Traffic signal control can sum intersection-level queueing costs, and ride-hailing can sum completed-trip revenues attributed to individual vehicles. These objectives provide individual reward components in systems with strongly coupled decisions and dynamics. For tasks with only a team reward, counterfactual [1] or difference-reward methods [2] offer ways to develop agent-specific credit. At matched loss scale, joint and per-agent ratios give identical sample gradients at the behavior policy when all agents use the same detached global advantage. The global advantage already supplies the first-order cooperative credit, so multiplying ratios adds no further credit at that point. For separate actor blocks, the near-policy bound quantifies the gradient discrepancy through accumulated ratio divergences within the trust region. Our theory takes the credit support as an input; it does not provide a practical radius-selection algorithm. Choosing a radius under unknown and changing interactions remains a separate open research question. Adaptive neighborhood selection is a natural approach to investigate. Relevant work studies spatial reward discounting in MA2C [3], learned attention within geographic or minimum-hop candidate neighborhoods in CoLight [4], and localization error versus neighborhood complexity in Qu et al. [5]. These approaches provide ingredients for constructing and adapting supports. Pre-action, state-dependent supports fit our formulation when held fixed during differentiation; the central comparison then determines how the selected credit enters the surrogate. The reward and interaction setting is clarified in blue in Section 5. Sections 3.1–3.2 give the ratio-side results, Appendix B covers history-dependent supports, and Appendix F discusses neighborhood selection and its open questions. [1] Jakob Foerster et al. *Counterfactual Multi-Agent Policy Gradients.* AAAI, 2018. [2] David H. Wolpert and Kagan Tumer. *Optimal Payoff Functions for Members of Collectives.* Advances in Complex Systems, 4(2–3):265–279, 2001. [3] Tianshu Chu et al. *Multi-Agent Deep Reinforcement Learning for Large-Scale Traffic Signal Control.* IEEE Transactions on Intelligent Transportation Systems, 21(3):1086–1095, 2020. [4] Hua Wei et al. *CoLight: Learning Network-level Cooperation for Traffic Signal Control.* CIKM, 2019. [5] Guannan Qu, Adam Wierman, and Na Li. *Scalable Reinforcement Learning for Multiagent Networked Systems.* Operations Research, 70(6):3601–3628, 2022.Thank you for the detailed response. The revised paper clarifies the local scope of gradient equivalence, accounts for correlations in the full-gradient variance analysis, and revises the overly strong claim that the optimal aggregation range must match the coupling neighborhood. The limitations regarding individual rewards and unknown or changing interactions are also clearer. These revisions largely address my main concerns.
The results still do not guarantee lower variance in a particular task or equivalence throughout PPO training, but the paper’s rigor has substantially improved. I am therefore increasing my rating from 2 to 3.
Thank you for your careful review and constructive feedback. We are glad that the revisions have addressed your main concerns. Your comments helped us clarify the scope of our claims and strengthen the theoretical analysis. The revised variance comparison provides a worst-case criterion for choosing between the aggregation schemes, which we hope will inform future research on stable multi-agent policy optimization.
Response Part 1
We thank the reviewer for the detailed examination of the equivalence, variance, and radius arguments. We have revised the results and proofs to address these points.
W1: Gradient equivalence during repeated policy updates
Thank you for your insightful comment. We now separate the exact behavior-policy identity from repeated updates and add a quantitative near-policy discrepancy bound; repeated updates are no longer treated as exactly equivalent. The revised argument analyzes candidate policies reached on a behavior-policy batch by combining the exact sample-gradient identity at the behavior policy with the discrepancy bound at subsequent policies.
Under factorized policies and joint differentiation, the effective credit is S=(SR)⊤SA. Transferring this credit into the advantage with per-agent ratios preserves the behavior-policy sample gradient when detached credits, baselines, and scale match, and therefore preserves its covariance.
For the objectives L=∑iEold[wiAi] defined in Section 2, where wi is the importance weight and Ai is detached credit, bounded credits and separate actor parameter blocks give
|V_Q-V_P|\le2\sqrt{V_P^*}K+K^2,$$ where $g_X$ is the mean gradient, $V_X$ its covariance trace, and $V_P^*$ the largest attainable trace variance over the common bounded-credit class. Let $\varrho_j$ be agent $j$'s candidate-to-behavior likelihood ratio. For ratio supports of at most $k$ actors with $\chi_j^2=\mathbb E_{\mathrm{old}}[\varrho_j^2]-1\le c$, $$K^2\le V_P^*\big[(1+c)^{k-1}-1\big].$$ During repeated updates, the behavior policy remains the reference for each candidate's ratios. Keeping the accumulated neighborhood divergences small under a bounded credit–gradient envelope therefore keeps the gradient discrepancy small at these update points. At these candidate policies, the bound quantifies the accuracy of the local redundancy in the proximal-update regime targeted by PPO [1], including its dependence on support size. The sample identity and near-policy analysis are marked in blue in Sections 3.1–3.2, with proofs in Appendices A.1 and A.5. [1] John Schulman et al. *Proximal Policy Optimization Algorithms.* arXiv:1707.06347, 2017. ## W2: Ratio-weight variance and policy-update variance Thank you for your insightful comment. We replace the weight-only variance argument with an exact worst-case comparison for the complete gradient over the same bounded-credit class. This comparison accounts jointly for the credit, policy score, importance weight, and their dependence. The original ratio-product moments enter the expression for the excess worst-case variance. Theorem 2 fixes the context, factorized policies, supports, and loss scale. It compares both paths over the same class of component credits satisfying $|A_i|\le a$ after baseline subtraction and preprocessing, held fixed during differentiation. The class allows arbitrary credit–action dependence and correlations among credits, and ratio supports may differ and overlap. For separate actor parameter blocks, the exact worst total gradient variances satisfy $$V_{Q}^{\ast}=V_{P}^{\ast}+a^2\sum_{m}d_{m}^2\mathbb{E}_{\mathrm{old}}[\Vert h_{m}\Vert^2]\mathrm{Var}(\bar{v}_{m})\ge V_{P}^{\ast}.$$ Here $h_m=\nabla_{\theta^m}\varrho_m$ differentiates the actor's own ratio, $d_m$ counts the surrogate terms containing that actor, and $\bar v_m$ averages their other-agent ratio products. Its variance is determined by support intersections and individual ratio second moments. The same admissible credit law, $A_i=a\xi$ with a zero-mean random sign independent of actions, attains both maxima. Since each maximum ranges over all admissible credit–action dependence and credit correlations, the comparison incorporates the dependence raised in your comment. It gives a sharp worst-case criterion for choosing advantage aggregation when that dependence is unknown. Appendix C develops the covariance terms for shared actors. The complete-gradient result is marked in blue in Section 3.2 and Appendix A.4; Appendix E.9 checks the attainable extrema.