acceptodds
Under review as a conference paper at ICLR 2027

Aggregate in the Advantage, Not the Ratio: A Canonical-Form Analysis of Cooperative Multi-Agent Policy Optimization

Abstract

Multi-agent policy optimization, exemplified by PPO-based methods, is a key branch of cooperative Multi-Agent Reinforcement Learning (MARL). A central design question is how many neighboring agents to aggregate so as to effectively utilize global information for cooperation. This decision must be made along two dimensions: in the advantage (which agents' rewards contribute to the credit signal) and in the ratio (which agents' likelihood ratios form the clipped importance weight). We formalize these two design choices as a pair of support patterns, one for the advantage and one for the ratio, and prove a canonical structure: the expected multi-agent policy optimization objective depends on the two patterns only through a single combined quantity, the product of the ratio pattern and the advantage pattern. This yields two consequences: (i) Redundancy: the two patterns are interchangeable with respect to the resulting learning signal, so neither way of aggregating neighbors is inherently superior to the other. (ii) Variance Ordering: the advantage combines neighbors' rewards additively, so its variance grows only mildly and it admits an interior sweet spot where the aggregation range matches the coupling neighborhood, whereas the ratio combines neighbors' likelihood ratios multiplicatively, so its variance grows exponentially with the number of aggregated agents and brings no compensating reduction in bias. Together these results point to an unambiguous design principle: aggregate neighbors in the advantage, sized to the coupling neighborhood, and keep the ratio per-agent. This also explains why neighbor-based advantages are far more common than neighbor-based ratios across prior heuristic and empirical designs. We prove these results under specified assumptions and validate them across four carefully designed synthetic cooperative games and a real-world large-scale traffic-signal control task. The code for our experiments is available in an anonymous repository at https://anonymous.4open.science/r/MAPO-803E .

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning, PPO

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cNRrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes two ways of aggregating information in cooperative multi-agent policy optimization: combining other agents’ rewards in the advantage, and multiplying their policy ratios in the importance weight. The authors use two support matrices to describe these choices and argue that neighboring agents’ information should be aggregated in the advantage while keeping importance ratios per-agent. They evaluate this idea through four synthetic games and a large-scale traffic signal control simulation.

Critical strengths
  1. Discussing advantage aggregation and ratio aggregation separately helps clarify the connections between different multi-agent policy optimization methods and where information from other agents should enter the update.

  2. When the current and old policies are identical, the paper shows that the product of the support matrices determines the expected gradient. This result, together with the variance formula for a product of independent importance ratios, provides useful insight into the two aggregation choices.

  3. The experiments cover several settings, including four cooperative games and a traffic control simulation with 196 intersections.

Critical weaknesses
  1. The paper shows that the two aggregation choices have the same expected gradient under specific conditions, but this does not mean they remain equivalent throughout training. The broader claim that cross-agent importance ratios offer no additional value therefore needs stronger support.

  2. Greater variation in the importance ratios does not necessarily mean noisier policy updates, since the updates also depend on the rewards and their correlations. The distinction between adding rewards and multiplying ratios alone is not enough to establish that one approach always has lower variance.

  3. Whether an agent’s information should be included also depends on how strongly it affects other agents, how noisy that information is, and how the rewards are correlated. The conclusion that the aggregation range should exactly match the coupling neighborhood is therefore too strong.

  4. The approach relies on separate rewards for individual agents and known coupling relationships. Real-world tasks may provide only a team reward, and interactions between agents may change across states and over time, so the paper should explain how its recommendations would change in these settings.

Questions for authors

Please refer to the weaknesses.

4 comments
AuthorsOpenReview ↗

Response Part 1

We thank the reviewer for the detailed examination of the equivalence, variance, and radius arguments. We have revised the results and proofs to address these points.

W1: Gradient equivalence during repeated policy updates

Thank you for your insightful comment. We now separate the exact behavior-policy identity from repeated updates and add a quantitative near-policy discrepancy bound; repeated updates are no longer treated as exactly equivalent. The revised argument analyzes candidate policies reached on a behavior-policy batch by combining the exact sample-gradient identity at the behavior policy with the discrepancy bound at subsequent policies.

Under factorized policies and joint differentiation, the effective credit is S~=(SR)⊤SA\widetilde S=(S^R)^\top S^A. Transferring this credit into the advantage with per-agent ratios preserves the behavior-policy sample gradient when detached credits, baselines, and scale match, and therefore preserves its covariance.

For the objectives L=∑iEold[wiAi]L=\sum_i\mathbb E_{\mathrm{old}}[w_i\mathcal A_i] defined in Section 2, where wiw_i is the importance weight and Ai\mathcal A_i is detached credit, bounded credits and separate actor parameter blocks give

|V_Q-V_P|\le2\sqrt{V_P^*}K+K^2,$$ where $g_X$ is the mean gradient, $V_X$ its covariance trace, and $V_P^*$ the largest attainable trace variance over the common bounded-credit class. Let $\varrho_j$ be agent $j$'s candidate-to-behavior likelihood ratio. For ratio supports of at most $k$ actors with $\chi_j^2=\mathbb E_{\mathrm{old}}[\varrho_j^2]-1\le c$, $$K^2\le V_P^*\big[(1+c)^{k-1}-1\big].$$ During repeated updates, the behavior policy remains the reference for each candidate's ratios. Keeping the accumulated neighborhood divergences small under a bounded credit–gradient envelope therefore keeps the gradient discrepancy small at these update points. At these candidate policies, the bound quantifies the accuracy of the local redundancy in the proximal-update regime targeted by PPO [1], including its dependence on support size. The sample identity and near-policy analysis are marked in blue in Sections 3.1–3.2, with proofs in Appendices A.1 and A.5. [1] John Schulman et al. *Proximal Policy Optimization Algorithms.* arXiv:1707.06347, 2017. ## W2: Ratio-weight variance and policy-update variance Thank you for your insightful comment. We replace the weight-only variance argument with an exact worst-case comparison for the complete gradient over the same bounded-credit class. This comparison accounts jointly for the credit, policy score, importance weight, and their dependence. The original ratio-product moments enter the expression for the excess worst-case variance. Theorem 2 fixes the context, factorized policies, supports, and loss scale. It compares both paths over the same class of component credits satisfying $|A_i|\le a$ after baseline subtraction and preprocessing, held fixed during differentiation. The class allows arbitrary credit–action dependence and correlations among credits, and ratio supports may differ and overlap. For separate actor parameter blocks, the exact worst total gradient variances satisfy $$V_{Q}^{\ast}=V_{P}^{\ast}+a^2\sum_{m}d_{m}^2\mathbb{E}_{\mathrm{old}}[\Vert h_{m}\Vert^2]\mathrm{Var}(\bar{v}_{m})\ge V_{P}^{\ast}.$$ Here $h_m=\nabla_{\theta^m}\varrho_m$ differentiates the actor's own ratio, $d_m$ counts the surrogate terms containing that actor, and $\bar v_m$ averages their other-agent ratio products. Its variance is determined by support intersections and individual ratio second moments. The same admissible credit law, $A_i=a\xi$ with a zero-mean random sign independent of actions, attains both maxima. Since each maximum ranges over all admissible credit–action dependence and credit correlations, the comparison incorporates the dependence raised in your comment. It gives a sharp worst-case criterion for choosing advantage aggregation when that dependence is unknown. Appendix C develops the covariance terms for shared actors. The complete-gradient result is marked in blue in Section 3.2 and Appendix A.4; Appendix E.9 checks the attainable extrema.
AuthorsOpenReview ↗

Response Part 2

W3: Coupling range, correlations, and the MSE-optimal radius

Thank you for your insightful comment. We replace the exact coupling-radius claim with a finite-batch MSE criterion that explicitly includes influence strength, noise, and correlation:

\frac{1}{B}\operatorname{tr}\operatorname{Cov}(G).$$ Here $G$ is the single-sample gradient estimator for the retained support, and MSE measures the error of its mean over $B$ independent samples relative to the full team gradient. Adding a component changes variance by its own variance plus twice its covariance with the retained estimator. The value of additional mean influence must therefore be considered alongside this covariance cost. The graph experiment illustrates this tradeoff: a wider neighborhood can recover omitted influence while adding noisy credit. Its value depends on whether the recovered signal outweighs the added variance and covariance. The MSE-optimal radius therefore depends on influence strength and noise, as well as the reward-dependency graph. Proposition 2 identifies when further enlargement adds only variance. Conditional on the pre-action context and the actor's action, each added processed credit has zero mean, and its residual is uncorrelated with retained residuals and the other added residuals. With credit processing, coefficients, and overall scale fixed, these additions preserve the gradient mean and add nonnegative covariance. Interactions and correlated residuals within the retained support remain allowed. The general MSE expression retains covariance so that it also captures shared future states and bootstrap estimates. The radius and MSE analysis are marked in blue in Section 3.3, with proofs in Appendices A.6–A.7 and the graph diagnostic in Appendix E.5. ## W4: Team-only rewards and changing interactions Thank you for your insightful comment. Our results target decomposable individual rewards; with an irreducible team reward, the ratio-side result still shows that multiplying ratios adds no first-order credit when all agents already use the same global advantage. When interactions vary across states or time, the useful credit support becomes context-dependent, so radius selection is a separate problem. Traffic signal control can sum intersection-level queueing costs, and ride-hailing can sum completed-trip revenues attributed to individual vehicles. These objectives provide individual reward components in systems with strongly coupled decisions and dynamics. For tasks with only a team reward, counterfactual [1] or difference-reward methods [2] offer ways to develop agent-specific credit. At matched loss scale, joint and per-agent ratios give identical sample gradients at the behavior policy when all agents use the same detached global advantage. The global advantage already supplies the first-order cooperative credit, so multiplying ratios adds no further credit at that point. For separate actor blocks, the near-policy bound quantifies the gradient discrepancy through accumulated ratio divergences within the trust region. Our theory takes the credit support as an input; it does not provide a practical radius-selection algorithm. Choosing a radius under unknown and changing interactions remains a separate open research question. Adaptive neighborhood selection is a natural approach to investigate. Relevant work studies spatial reward discounting in MA2C [3], learned attention within geographic or minimum-hop candidate neighborhoods in CoLight [4], and localization error versus neighborhood complexity in Qu et al. [5]. These approaches provide ingredients for constructing and adapting supports. Pre-action, state-dependent supports fit our formulation when held fixed during differentiation; the central comparison then determines how the selected credit enters the surrogate. The reward and interaction setting is clarified in blue in Section 5. Sections 3.1–3.2 give the ratio-side results, Appendix B covers history-dependent supports, and Appendix F discusses neighborhood selection and its open questions. [1] Jakob Foerster et al. *Counterfactual Multi-Agent Policy Gradients.* AAAI, 2018. [2] David H. Wolpert and Kagan Tumer. *Optimal Payoff Functions for Members of Collectives.* Advances in Complex Systems, 4(2–3):265–279, 2001. [3] Tianshu Chu et al. *Multi-Agent Deep Reinforcement Learning for Large-Scale Traffic Signal Control.* IEEE Transactions on Intelligent Transportation Systems, 21(3):1086–1095, 2020. [4] Hua Wei et al. *CoLight: Learning Network-level Cooperation for Traffic Signal Control.* CIKM, 2019. [5] Guannan Qu, Adam Wierman, and Na Li. *Scalable Reinforcement Learning for Multiagent Networked Systems.* Operations Research, 70(6):3601–3628, 2022.

Thank you for the detailed response. The revised paper clarifies the local scope of gradient equivalence, accounts for correlations in the full-gradient variance analysis, and revises the overly strong claim that the optimal aggregation range must match the coupling neighborhood. The limitations regarding individual rewards and unknown or changing interactions are also clearer. These revisions largely address my main concerns.

The results still do not guarantee lower variance in a particular task or equivalence throughout PPO training, but the paper’s rigor has substantially improved. I am therefore increasing my rating from 2 to 3.

Thank you for your careful review and constructive feedback. We are glad that the revisions have addressed your main concerns. Your comments helped us clarify the scope of our claims and strengthen the theoretical analysis. The revised variance comparison provides a worst-case criterion for choosing between the aggregation schemes, which we hope will inform future research on stable multi-agent policy optimization.

Reviewer_9iCxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyses the design choices of cooperative multi-agent RL algorithms that are specifically based on PPO. The central question addressed in the paper is how many neighbouring agents information to use for effective collaboration. They first introduce advantage support SAS^A denoting which agents' rewards are summed to agent i's advantage, and the second is likelihood ratio support SRS^R denoting which agents’ likelihood ratios are multiplied into agent i’s importance weight. The supports are formalized as binary matrices. The paper introduces main Theorem 1 which shows that for the on-policy case, the expected multi-agent policy gradient depends on SAS^A and SRS^R only through the product S~=SRSA\tilde{S} = S^R S^A. The theoretical results are validated on four synthetic cooperative games and a large-scale traffic-signal control task.

Critical strengths
  • The theoretical development presented in the paper is both well-grounded and rigorous, offering valuable insights into PPO, the most widely used policy optimization method in MARL. Theorem 1 establishes that for the on-policy case, the expected multi-agent policy gradient depends only on the product of the two supports, denoted as S~=SRSA\tilde{S} = S^R S^A. This canonical form indicates that the two supports are mutually redundant in relation to the gradient signal. The paper further shows that advantage aggregation is a sum of bounded rewards with additive variance, whereas ratio aggregation is a product of importance weights whose variance grows multiplicatively and hence exponentially with support size. Proposition 2 complements this finding by showing that the advantage support achieves a mean squared error (MSE) optimum precisely at the coupling neighborhood. This leads to the paper’s design rule (Corollary 2): aggregate neighbors in the advantage while maintaining the per-agent ratio.

  • The experiments are well-controlled and align with the predictions outlined in the theory. Figure 1 displays the identical bias curves for both advantage and ratio support sweeps, confirming the redundancy of Theorem 1. Figure 2 illustrates the clear training-level validation of the advantage radius.

Critical weaknesses
  • A significant limitation of the paper is its reward structure. The team return is assumed to be a linear sum of per-agent rewards, R(a)=∑iri(a)R(a) = \sum_i r_i(a). This additive decomposability is a strong structural assumption that substantially simplifies credit assignment. It is unclear how the results extend for general cooperative objectives with irreducible team-level rewards. The domains used in the experiments also have the decomposable joint reward R=∑iriR = \sum_i r_i. The method is not tested on the non-decomposable case, which is precisely where credit assignment is hardest and where the proposed design principle could fail.

  • Remark 6 extends the analysis to finite horizons, but it is presented without proof. This represents a significant gap, as the entire theoretical framework—comprising Lemmas 1 and 2, Theorem 1, and Propositions 1 and 2—has been established solely for a single decision step. The finite-horizon setting is, in fact, the more relevant scenario in practice. In MARL, the PPO algorithm operates on rollouts of length H, utilizing GAE advantages. Thus, the analysis centered on a single decision step is a simplification rather than the general case.

  • The empirical scope of this study is limited. All validation relies on four custom-designed synthetic games in tabular settings and one large-scale traffic scenario. The paper does not evaluate any standard MARL benchmarks, such as SMAC or MPE, so it is unclear how its conclusions apply to the environments where multi-agent PPO is typically used. Additionally, the evaluation results for the real-world large-scale traffic scenario are included in the appendix.

Questions for authors
  • How sensitive is the MSE-optimal advantage radius to misspecification of coupling graph CC?

  • In the traffic experiment, what is the coupling graph? How was the neighborhood radius chosen?

  • Why are no standard MARL benchmarks SMAC, MPE, etc included in the experiments?

4 comments
AuthorsOpenReview ↗

Response Part 1

We thank the reviewer for the detailed comments on reward decomposition, finite-horizon analysis, and experimental coverage.

W1: Additive individual rewards and team objectives

Thank you for your insightful comment. We restrict the paper to decomposable rewards and state why this is a common, practical setting in cooperative control and service systems. In traffic signal control, the system objective sums intersection-level queueing costs; in ride-hailing, it can sum completed-trip revenues attributed to the serving vehicle, counting each trip once. These applications provide identifiable individual rewards and motivate the aggregation problem studied here.

These reward components can be strongly coupled. Each ri(a)r_i(\mathbf a) may depend on several agents' actions, and an action can change other agents' future states and returns. Including the affected components in the advantage brings these cross-agent effects into the acting agent's update. The finite-horizon analysis makes this return-influence interpretation explicit.

For an irreducible team reward, our ratio analysis addresses the intuition that multiplying agents' ratios should promote cooperation when every agent already uses the global advantage. For factorized policies, joint and per-agent ratios give identical sample gradients at the behavior policy when the detached global advantage and loss scale match. Thus sharing the global advantage already conveys the first-order cooperative credit; multiplying ratios adds no further credit at that point. For separate actor blocks, the near-policy bound quantifies their gradient discrepancy through accumulated ratio divergences within the trust region. This gives subsequent work a theoretical reason to use per-agent ratios with shared team credit.

The reward setting and cross-agent effects are clarified in blue in Sections 2 and 5. Sections 3.1–3.2 give the identity and near-policy bound, and Appendix E.8 examines a scalar team credit.

W2: Finite-horizon proof and the use of GAE

Thank you for your insightful comment. We replace the unsupported finite-horizon remark with a proposition and proof in Appendix B. Differentiating the trajectory probability gives a sum of actor scores. The zero-mean score identity removes past rewards, leaving the future component returns affected by the current action. Collecting these terms at each behavior-policy time step yields

S~t=(StR)⊤StA.\widetilde S_t=(S_t^R)^\top S_t^A.

Supports can depend on the pre-action history and are held fixed during differentiation. The relevant influence is therefore on future component returns, which can extend beyond immediate reward neighbors through the dynamics.

The proof covers the detached advantages used in rollout updates. Their realized values give the sample-gradient rearrangement; their conditional means determine the corresponding return-gradient contributions. Exact returns satisfy the required conditional-mean identity. Finite-horizon GAE also satisfies it for every λ∈[0,1]\lambda\in[0,1] with exact component values, complete behavior-policy rollouts, and zero terminal value. Appendix B.1 then expresses the gradient error from approximate critics and truncated-rollout bootstraps explicitly.

The rollout variance analysis retains covariance across time: the per-decision product-moment formula is applied at each fixed context, and cross-time terms enter the variance of the summed update. The finite-horizon result and these corrections are marked in blue in Section 3.4 and Appendix B. Appendix E.7 illustrates why delayed effects require temporal credit support: including the affected successor brings the measured mean closer to the behavior-policy temporal-support reference while increasing variability.

W3: Empirical coverage and placement of the traffic results

Thank you for your insightful comment. We moved the traffic findings into Section 4.5 and use them as the large-scale empirical validation of the aggregation choices. They test aggregation in a large system with decomposable rewards and coupled dynamics. Local credit outperforms credit from all 196 intersections, and, with the credit fixed, per-agent ratios outperform a joint ratio. These two comparisons identify the effects of credit range and ratio range in PPO learning with downstream flow and delayed congestion.

The controlled games answer the complementary mechanistic questions. Their known reward dependencies allow direct measurement of omitted gradient contributions, full-gradient variance, and support-selection error. We have added exact gradient and graph-error diagnostics and clarified the traffic architecture and metric aggregation in Appendix D.3, with the substantive clarifications marked in blue. The rationale for the benchmark selection is given in Q3 below.

AuthorsOpenReview ↗

Response Part 2

Q1: Sensitivity of the MSE-optimal radius to graph error

Thank you for your insightful comment. In our perturbation study, the MSE-optimal radius usually remains unchanged, while some graph errors favor a slightly wider neighborhood. This occurs because a misspecified graph can omit influential credit or include additional noisy credit: missing connections can make a wider radius preferable, whereas spurious connections add noise.

Widening can recover omitted credit, but can still leave more error than the correct graph because the altered neighborhood also includes different noise. Thus graph error affects both the preferred radius and the accuracy attainable at that radius.

Section 3.3 explains this sensitivity through the tradeoff between omitted mean influence and gradient variance. The analysis and experiment are marked in blue in Section 3.3 and Appendix E.5.

Q2: The traffic graph and how the radius was selected

Thank you for your insightful comment. We use road adjacency on the 28×728\times7 traffic grid and select the radius through a predefined sweep of individual, one-hop, two-hop, and global supports. A radius-kk neighborhood contains the intersection itself and the intersections reachable within kk graph hops; one- and two-hop supports have nearly equal returns and both outperform global aggregation.

Section 4.5 presents the comparison, and Appendix D.3 specifies the graph, support construction, networks, and training protocol; the substantive setup clarifications are marked in blue.

Our theory compares aggregation for a given credit support; it does not prescribe a practical radius-selection rule. Selecting a radius from finite data under unknown or changing return influence remains an open research question. Related work studies how to construct and weight local neighborhoods: MA2C [1] uses graph-distance reward discounting, CoLight [2] learns attention within geographic or minimum-hop candidate neighborhoods, and Qu et al. [3] analyze localization error and neighborhood complexity. These approaches provide useful starting points for heuristic or adaptive support selection. Our focus is how the selected credit enters the policy update, as explained in Appendix F.

[1] Tianshu Chu et al. Multi-Agent Deep Reinforcement Learning for Large-Scale Traffic Signal Control. IEEE Transactions on Intelligent Transportation Systems, 21(3):1086–1095, 2020.

[2] Hua Wei et al. CoLight: Learning Network-level Cooperation for Traffic Signal Control. CIKM, 2019.

[3] Guannan Qu, Adam Wierman, and Na Li. Scalable Reinforcement Learning for Multiagent Networked Systems. Operations Research, 70(6):3601–3628, 2022.

Q3: Why SMAC and MPE were not used for the experimental analysis

Thank you for your insightful comment. We use analytically specified games and the traffic application instead of SMAC or native MPE as the main validation environments because the controlled part of this paper requires decomposable credits whose gradient means, variances, and omitted influence can be computed directly. Standard SMAC supplies a team reward and therefore lies outside this decomposable-reward setting. The implication for ratio aggregation with a global advantage is explained in W1.

MPE's simple particle-world scenarios are commonly used for controlled analyses of multi-agent interaction. For studying aggregation mechanisms, these scenarios serve much the same purpose as our synthetic games. We use analytically specified games so that gradient means, variances, and omitted influence can be calculated directly. Traffic control complements this analysis by testing the aggregation choices in a dynamic system of 196 interacting intersections.

Section 5 clarifies the reward setting in blue, and Appendix D.1 explains the experimental design.

I thank the authors for their response. However, my main concerns regarding the limited scope of the paper to decomposible rewards and the lack of standard cooperative MARL benchmarks remain. Therefore, I intend to keep my score unchanged.

Thank you for your careful review and follow-up comment. Since the scope of our paper focuses on scenarios with individual rewards, it is difficult to directly apply our method to most standard benchmarks (e.g. Google Research Football, SMAC, and MaMuJoCo) where only group rewards are provided and typically used for research like credit assignment. We would like to explore these group-reward scenarios as a future direction. Thank you again for your insightful suggestions.

Reviewer_WHNFOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates - both theoretically and empirically - the different effects of aggregating agents in the advantage vs in the ratio for multiagency ppo based methods. It is a good analysis that helps the community understand the motivations behind aggregation and possible effects. Theoretical analysis is supported with empirical data, experiments of multiagency rl setups, which are stylized and a bit far from providing a difficult multiagent game.

Critical strengths
  • Extensive theoretical analysis is provided for the question of aggregation.
  • Although the experiment environments are toy, games are selected such that they show different multiagency dynamics; also the experiments enable investigation with different radiuses and how the performance scales with very high number of agents.
Critical weaknesses
  • The writing is difficult to follow: it doesn't flow through reading and doesn't feel coherent, it feels like someone just verbally explaining to you
  • Environments used for empirical study are toy (even though they scale in number).
Questions for authors
  • Although the work is informative, it feels like supporting the same (or very similar) design choices that have been used up to now. What do you think is the exact significance of this work such that it builds upon whatever researchers of prior work experienced?
  • How would the results from this paper be used for more realistic MARL problems with many agents? Would the technical constraints dominate or the theoretical implications of training? If it's not about these many agent realistic MARL problem, would the differences in aggregation affect MARL learning such as SMAC?
  • Could you explain the results in Figure 2, why the effect is different for directed dilemma?
4 comments
AuthorsOpenReview ↗

Response Part 1

We thank the reviewer for the careful reading and suggestions on the exposition, contribution, and applications.

W1: Clarity and coherence of the presentation

Thank you for your insightful comment. We rewrote the presentation around a single question, how should neighboring credit enter a policy update, and made the answer, supporting theory, and experimental roles explicit. Section 2 introduces advantage and ratio supports. Section 3 then explains their common gradient at the behavior policy, their statistical cost after policy movement, and the tradeoff in choosing the aggregation range. This sequence carries through the abstract, introduction, and conclusion.

Throughout the paper, we have rewritten conversational passages, introduced notation before using it, and added explanations of what each experiment tests and what its results mean. Section 4 distinguishes fixed-policy gradient measurements from PPO learning curves. The substantive corrections to the analysis are marked in blue in Section 3, with detailed proofs and experimental protocols in the appendices.

W2: Whether the empirical tasks are sufficiently realistic

Thank you for your insightful comment. The empirical validation includes a SUMO traffic-control experiment with 196 interacting intersections; the synthetic games provide the controlled measurements needed to test the theory directly. Their known reward dependencies allow us to measure gradient bias and variance without confounding them with an application simulator.

Traffic signal control provides a practical, large-scale application of these results. It is widely studied in multi-agent reinforcement learning [1, 2]. Our SUMO experiment coordinates 196 interacting intersections, where signal decisions change downstream traffic flow and future congestion. The traffic results further support the design principle developed in the theory. With per-agent ratios fixed, local advantages outperform global advantages; with one-hop advantages fixed, per-agent ratios outperform a joint ratio. These findings support aggregating useful neighboring credit in the advantage while retaining per-agent ratios in a large, dynamically coupled control system.

The traffic findings are presented in Section 4.5, with the full setup and results in Appendix D.3; substantive setup clarifications are marked in blue there.

[1] Tianshu Chu et al. Multi-Agent Deep Reinforcement Learning for Large-Scale Traffic Signal Control. IEEE Transactions on Intelligent Transportation Systems, 21(3):1086–1095, 2020.

[2] Hua Wei et al. CoLight: Learning Network-level Cooperation for Traffic Signal Control. CIKM, 2019.

Q1: Contribution beyond established aggregation choices

Thank you for your insightful comment. Our contribution is a formal characterization of when moving cooperative credit from ratios to advantages preserves the behavior-policy gradient, how the two forms compare in attainable worst-case variance, and how omitted credit trades off against ratio noise near the behavior policy. This gives a theoretical basis for the advantage aggregation used in earlier multi-agent methods [1, 2].

The canonical form

S~=(SR)⊤SA\widetilde S=(S^R)^\top S^A

identifies the effective credit represented by a jointly differentiated ratio construction. Transferring this credit into the advantage preserves the behavior-policy sample gradient when credits, baselines, and scale match. The matrix product gives the required weights explicitly, including repeated contributions that a nominal neighborhood radius alone does not reveal. For separate actor blocks, the near-policy bound quantifies the gradient discrepancy between these ratio-weighted surrogates during small policy updates.

Theorem 2 supplies a statistical reason to prefer the advantage-side representation: for separate actor parameter blocks over a common bounded-credit class, it has no larger attainable worst total gradient variance. The excess from ratio aggregation is determined by the overlap and second moments of the extra ratio factors. The radius analysis explains the accompanying tradeoff between omitted influence and added credit noise.

These results turn an established empirical choice into an explicit design principle: include useful cooperative credit in the advantage and retain per-agent ratios. This gives subsequent work a basis for designing cooperative credit and communication mechanisms. The identity, variance criterion, and radius analysis are marked in blue in Sections 3.1–3.3, with proofs in Appendix A.

[1] Chao Yu et al. The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games. NeurIPS, 2022.

[2] Tianshu Chu et al. Multi-Agent Deep Reinforcement Learning for Large-Scale Traffic Signal Control. IEEE Transactions on Intelligent Transportation Systems, 21(3):1086–1095, 2020.

AuthorsOpenReview ↗

Response Part 2

Q2: Use in large systems, computational cost, and SMAC

Thank you for your insightful comment. With bounded local support degree dd, the construction costs O(nd)O(nd) scalar operations per batch element, so its computation remains linear in system size; the main scalability issue is statistical, because enlarging ratio supports adds variance factors. The construction changes only the aggregation within the PPO loss and reuses the actors, critics, and rollouts. Identical global supports share an O(n)O(n) reduction, and the traffic learner implements the construction with a shared actor and local-observation critic.

The statistical cost is governed by the ratios inside each support. With local supports and their policy moments fixed, adding agents outside an actor's support leaves its ratio-induced worst-case variance multiplier unchanged. Enlarging the ratio support instead introduces additional factors whose accumulated moments enter Theorem 2 and the near-policy bound. This makes local aggregation particularly relevant as the overall system grows.

Our local-credit construction assumes meaningful individual reward components. Standard SMAC instead supplies a shared team reward. For this setting, the ratio analysis addresses the intuition that multiplying agents' ratios promotes cooperation: with the same detached global advantage and matched loss scale, joint and per-agent ratios give identical sample gradients at the behavior policy. Sharing the global advantage already supplies this first-order cooperative credit. For separate actor blocks, Section 3.2 quantifies their gradient discrepancy through accumulated ratio divergences within the trust region. Counterfactual [1] or difference-reward methods [2] offer ways to develop agent-specific credit for further study of local aggregation.

The identity and near-policy analysis are marked in blue in Sections 3.1–3.2. Section 5 explains the reward setting and computational cost; Appendix C treats shared actors, and Appendix E.8 examines a fixed scalar team credit.

[1] Jakob Foerster et al. Counterfactual Multi-Agent Policy Gradients. AAAI, 2018.

[2] David H. Wolpert and Kagan Tumer. Optimal Payoff Functions for Members of Collectives. Advances in Complex Systems, 4(2–3):265–279, 2001.

Q3: Why the directed dilemma responds differently

Thank you for your insightful comment. The directed dilemma differs because an action creates a directed externality: it benefits the acting agent but imposes costs on successor agents. An individual advantage rewards the acting agent for its own benefit while leaving those costs out of its update. Including successor rewards changes which action the update favors: the actor now accounts for the harm it creates. One graph hop covers these affected successors in the training task. This explains the sharp transition in Figure 2 from a harmful joint outcome with individual advantages to a cooperative outcome with neighborhood advantages.

The other games already place part of the interaction in the actor's own reward. A symmetric pairwise payoff appears in both endpoint rewards, and an agent choosing a congested resource bears some of the congestion it creates. Adding neighboring rewards consequently changes the incentive less dramatically.

Section 4.3 explains this mechanism, with the graph-hop clarification marked in blue. Appendix E.6 isolates it in an exact directed-ring calculation: adding affected successors restores the omitted mean contributions while adding variance, explaining why recovering the full gradient and minimizing finite-batch MSE can favor different supports.

Thank you for improving and clarifying the writing and presentation. I liked that toy experiments led the paper investigate everything in a controlled and detailed way. While I was asking for more complex environments, I was interested more in games like SMAC the problem is not just learning to coordinate but also learning good strategies that are harder to find as the game gets more complex. Nevertheless, for the current work, traffic signal control is convincing for me, though more impressive additions could be nice. Although you responded but I'm not sure if that answers my question, what I'd like to better understand is the following: what do the canonical form representation and the theoretical contributions teach us that prior empirical work did not, and is it possible to directly gain insight from those so that we can decide on how to aggregate without any hyper parameter tuning? I am currently keeping my score as I'd already like to see it accepted but still have some follow up concerns about the impact and significance of the paper.

Thank you for the careful review, positive score and the follow-up question. The canonical form separates two design choices that are often conflated and identifies the effective credit induced by their product. At the behavior policy, this shows exactly when moving cooperative credit from a joint ratio into the advantage preserves the sample gradient; with a shared global advantage, additional ratio multiplication supplies no first-order credit. The near-policy bound then quantifies how this redundancy changes as the policy moves within the trust region. Our complete-gradient theorem adds a conservative statistical criterion: over the same bounded-credit class, advantage aggregation has no larger attainable worst-case trace variance than ratio aggregation, including credit–score–ratio dependence. Together, these results turn prior empirical practice into an explicit design principle and explain its mechanism.

In conclusion, under the conservative worst-case-variance criterion studied here, particularly relevant when unstable updates are costly or an application has little tolerance for adverse cases, the theory gives a direct rule once the credit support has been chosen: aggregate cooperative credit in the advantage and retain per-agent ratios. We hope this clarification address your concerns sufficiently.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank the AC, PC, and all reviewers for their careful reading and constructive feedback. As the rebuttal period will close tomorrow, we summarize our contributions and the substantive changes made in response to the reviews.

Contributions

  • A central design question in cooperative multi-agent policy optimization. To our knowledge, we are the first to explicitly formulate and analyze the relationship between aggregating cooperative information in the advantage and aggregating it in the likelihood ratio, including the distinct roles and consequences of these two choices.

  • A unified theoretical analysis. We establish the canonical-form equivalence of the two aggregation schemes at the behavior policy, derive a quantitative near-policy discrepancy bound, and characterize their variance difference through an exact worst-case comparison of the complete policy-gradient estimator under a common bounded-credit class.

  • Controlled and large-scale empirical validation. Controlled cooperative games isolate the mechanisms predicted by the theory, while a 196-intersection SUMO traffic-control experiment evaluates the resulting design principle in a large, dynamically coupled system with decomposable rewards.

Substantive changes made in response to the reviews

  • Correction of the variance analysis. We corrected the original weight-only variance argument and replaced it with a complete-gradient worst-case analysis that accounts for the credit signal, policy score, importance ratios, and their dependence.

  • Finite-horizon proposition and proof. We added a formal finite-horizon extension and proof, including the conditions under which the result applies to detached GAE advantages, and clarified temporal credit and cross-time covariance.

  • Reorganization of the presentation. We reorganized the exposition and refined the language throughout the manuscript. This includes clearer notation, stronger logical transitions, an explicit explanation of each experiment’s purpose, a clearer statement of the reward and interaction scope, and the placement of the traffic results in the main experimental section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.