acceptodds
Under review as a conference paper at ICLR 2027

Grouped Heterogeneous-Agent Proximal Policy Optimization

Abstract

Proximal Policy Optimization (PPO) has been successfully extended to cooperative multi-agent reinforcement learning (MARL), providing an effective framework for learning coordinated policies. Multi-Agent Proximal Policy Optimization (MAPPO) achieves strong sample and parameter efficiency through parameter sharing, but a fully shared policy can induce homogeneous behaviors and fail to represent asymmetric optimal solutions. Heterogeneous-Agent Proximal Policy Optimization (HAPPO) supports heterogeneous behaviors by maintaining separately parameterized policies and coordinating agent-by-agent sequential updates, but the number of dependent update stages grows with the number of agents, limiting training efficiency. In this paper, we propose Grouped Heterogeneous-Agent Proximal Policy Optimization (GHAPPO). By extending multi-agent advantage decomposition to groups, GHAPPO shares a policy backbone within each group and updates groups sequentially, thereby reducing the number of dependent update stages. In addition, we introduce a lightweight agent-specific action-replacement module to promote policy heterogeneity within each group. Under an explicit bounded-ratio assumption, our analysis characterizes the theoretical relationship between grouping granularity and sequential-update overhead, while an example shows that action replacement can raise the attainable performance upper bound relative to full parameter sharing. Experiments on SMAC and SMACv2 demonstrate that GHAPPO improves training efficiency while maintaining strong performance across different scenarios.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning, Heterogeneous-Agent Reinforcement Learning, Proximal Policy Optimization, Parameter Sharing, Grouped Policy Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_heZuOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I do not have sufficient expertise to assess this submission reliably and have alerted the AC. None of my current scores should be taken into account when deciding whether to accept this paper.

Critical strengths

I do not have sufficient expertise to assess this submission reliably and have alerted the AC. None of my current scores should be taken into account when deciding whether to accept this paper.

Critical weaknesses

I do not have sufficient expertise to assess this submission reliably and have alerted the AC. None of my current scores should be taken into account when deciding whether to accept this paper.

Questions for authors

I do not have sufficient expertise to assess this submission reliably and have alerted the AC. None of my current scores should be taken into account when deciding whether to accept this paper.

1 comment
AuthorsOpenReview ↗

Author Response

We appreciate the reviewer’s candor in helping to ensure a fair review process. We would like to share a brief summary of our work in the hope that it offers the reviewer some inspiration for their future research.

We focus on a practical question in cooperative multi-agent reinforcement learning: How to enhance training efficiency with parameter sharing while retaining the ability to behave differently with sequential update pattern? The core contributions are as follows:

  1. To the best of our knowledge, our work is the first to extend the agent-wise advantage decomposition to groups by expressing the joint advantage as a sum of conditional group advantages. This decomposition provides the basis for group-wise PPO update.
  2. Based on the grouped multi-agent advantage decomposition theorem, we propose GHAPPO, in which each group shares one actor backbone and each agent has a lightweight private action-replacement module that allows it to specialize within the group.
  3. We incorporate a lightweight action replacement module to let each agent replace a proposed action using only local information. This mechanism uses a small set of private parameters to encode agent-specific behavior while allowing agents within a group to reuse most of their policy parameters through a shared backbone.
  4. Our experiments demonstrate that GHAPPO uses 58.95% fewer total parameters than HAPPO and achieves the highest mean win rate among GHAPPO, MAPPO, and HAPPO on all five tested SMACv2 scenarios.
Reviewer_CwEcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper formulates cooperative MARL as a Dec-POMDP and proposes GHAPPO to balance parameter sharing and agent specialization. It partitions agents into predefined, ordered, disjoint groups with shared policy backbones within groups and sequential updates across groups. To preserve within-group behavioral diversity, each agent has a lightweight private action-replacement module. During training, a centralized critic estimates the joint advantage, which is reweighted using accumulated likelihood ratios from previously updated groups. A group-level clipped PPO objective then jointly optimizes the current group’s shared backbone and replacement modules, while execution requires only local information. To support this design, the paper establishes a grouped advantage decomposition and analyzes the trade-off F(k)=U(k)−λP(k)F(k)=U(k)-\lambda P(k) under an additional bounded-likelihood-ratio assumption. Here, U(k)U(k) represents modeled policy improvement and P(k)=(k−1)CP(k)=(k-1)C represents sequential-update overhead. A second-order approximation yields a closed-form continuous optimal group count for the proxy. Complementing this analysis, a binary coordination example demonstrates that action replacement can achieve a strictly higher optimal return than identity-blind full parameter sharing. Experiments compare GHAPPO with MAPPO and HAPPO on six SMAC maps and five SMACv2 scenarios. GHAPPO performs competitively overall and achieves the highest mean win rate on all five SMACv2 scenarios. Ablations further support action replacement and role-based grouping, while showing how group count affects performance, parameter count, and network-update cost. These findings support parameter savings, although they do not establish reduced overall training time.

Critical strengths
  1. To address HAPPO’s large parameter count and long sequential update chain, as well as the potential limitations of full parameter sharing on agent specialization, the paper combines within-group sharing, group-wise sequential updates, and private action replacement to reduce parameters and update stages while preserving behavioral diversity.
  2. Choosing the number of groups involves a trade-off between policy flexibility and training cost. The paper makes this trade-off explicit through a group-granularity proxy analysis and supporting ablations, providing insight into how group count affects performance, parameter size, and update overhead.
Critical weaknesses
  1. Overall training efficiency remains unresolved. Although grouping reduces parameter count and sequential update stages, GHAPPO achieves only a small reduction in network-update time relative to HAPPO on 3s5z and takes longer overall. The reported results therefore do not demonstrate an end-to-end training speedup.
  2. The specific benefit of action replacement remains unclear. The paper lacks comparisons with agent-ID-conditioned sharing and parameter-matched private action heads, leaving open whether the gains come from replacement itself or simply from adding agent-specific parameters.
Questions for authors
  1. Section 3.2 assumes a positive expected advantage and bounded individual likelihood ratios. Since advantages can be negative, how do these assumptions establish the stated upper bound on each group’s contribution? Does this require additional assumptions, or is U(k) simply a defined analytical proxy?
  2. Example 1 compares against an identity-blind shared policy, whereas an agent-ID-conditioned policy could also represent the required behaviors. Do MAPPO and GHAPPO use agent IDs in the experiments? What additional benefit does action replacement provide over agent-ID conditioning or parameter-matched private action heads?
  3. The main text provides GHAPPO’s optimization objective and framework diagram, but the complete training procedure remains insufficiently specified. Could the authors provide pseudocode for the complete training procedure?
  4. The experiments compare GHAPPO only with MAPPO and HAPPO. Could the authors include more recent baselines, which also studies sequential batch updates to balance performance and efficiency? This would help assess GHAPPO’s added value over more recent approaches.
6 comments
AuthorsOpenReview ↗

Author Response (Part 1/4)

We sincerely thank Reviewer CwEc for the constructive feedback. We have addressed the critical weaknesses and questions by providing additional explanations and experimental results, all of which have been incorporated into the revised manuscript.

Overall training efficiency remains unresolved. Although grouping reduces parameter count and sequential update stages, GHAPPO achieves only a small reduction in network-update time relative to HAPPO on 3s5z and takes longer overall. The reported results therefore do not demonstrate an end-to-end training speedup.

We thank the reviewer for the constructive feedback. In our SMAC experiments, CPU-side trajectory collection accounts for most of the time per training iteration, with routine execution overhead also contributing. These costs are outside the direct control of GHAPPO’s group-wise update mechanism. GHAPPO mainly improves efficiency during the network-update phase. To better demonstrate how grouping affects the cost of sequential updates and realize training speedup, we measured the time spent on the forward pass, backward pass, and optimizer step separately.

The table reports the mean time per training iteration over 100,000 environment steps on 3s5z. All times are in milliseconds (ms).

AlgorithmTotal parametersTotal network updateForwardActor forwardCritic forwardBackwardActor backwardCritic backwardOptimizer stepActor optimizer stepCritic optimizer stepOther update operations
MAPPO107,86998.22118.34010.7217.61914.5599.0265.5335.2792.7922.48760.043
HAPPO469,895243.70576.75368.4688.28552.53246.9765.55522.17219.7532.41992.248
GHAPPO192,875194.72477.30669.4557.85135.06829.4455.62211.6749.1982.47670.676

Compared with HAPPO, GHAPPO uses 58.95% fewer parameters and reduces total network-update time by 20.10%.

  • Forward pass: Total time increases by 0.72%. Actor time increases by 1.44%.
  • Backward pass: Total time decreases by 33.24%. Actor time decreases by 37.32%.
  • Optimizer step: Total time decreases by 47.35%. Actor time decreases by 53.43%.

GHAPPO and HAPPO use the same underlying training codebase, and both implementations process each agent’s inputs during the forward pass. This is consistent with the similar forward-pass times we observe. During network updates, GHAPPO reduces actor backward-pass time by 37.32% and actor optimizer-step time by 53.43% relative to HAPPO. These reductions demonstrate the efficiency benefits of group-wise training during network updates.

We have updated Section 4.2 with the new profiling results to show how GHAPPO contributes to training efficiency.

AuthorsOpenReview ↗

Author Response (Part 2/4)

The specific benefit of action replacement remains unclear. The paper lacks comparisons with agent-ID-conditioned sharing and parameter-matched private action heads, leaving open whether the gains come from replacement itself or simply from adding agent-specific parameters.

We thank the reviewer for suggesting these comparisons. To assess the contribution of action replacement, we compare three GHAPPO variants using action replacement, ID-conditioned sharing, and parameter-matched private action heads, respectively. The table reports test win rates after 10710^7 environment steps.

Algorithm3s5z5m_vs_6m
GHAPPO with action replacement95.00% ± 3.16%50.00% ± 6.32%
GHAPPO with ID-conditioned sharing80.09% ± 6.14%40.83% ± 4.96%
GHAPPO with parameter-matched private action heads84.52% ± 3.81%27.67% ± 5.07%

In the ID-conditioned sharing variant, agent identity are provided as an input to the shared actor. In the private-head variant, we matched the number of parameters used by each agent’s policy with the action-replacement variant. The corresponding parameters are stored separately for each agent. In action replacement, agents within a group reuse the same backbone and retain only a lightweight private replacement module. Thus, the two variants have the same per-agent policy size, while action replacement uses fewer unique trainable parameters across the team.

The action-replacement variant achieved the highest mean test win rate on both maps. These results further support the effectiveness of combining within-group parameter sharing with agent-specific action replacement.

We have revised Section 4.4 with these new ablation results.

Section 3.2 assumes a positive expected advantage and bounded individual likelihood ratios. Since advantages can be negative, how do these assumptions establish the stated upper bound on each group’s contribution? Does this require additional assumptions, or is U(k) simply a defined analytical proxy?

The modeling assumption in Section 3.2 is motivated by HAPPO’s discussion of improvement directions: If each action selected in sequence has a positive conditional advantage given the preceding actions, then the resulting joint action also has positive advantage. This is a sufficient condition for identifying a direction of improvement. The upper bound in Section 3.2 refers to an idealized setting. The upper-bound argument can also extend to signed advantages by considering their absolute magnitude with the same derivation. In this formulation, the proxy captures the magnitude of the contributions.

We do not introduce any additional assumptions beyond those mentioned above. U(k)U(k) sums the upper bounds within the idealized model and serves as an analytical proxy for the improvement signal. The objective F(k)F(k) uses this proxy to study how the preferred group count changes with the modeled cost of sequential updates.

We have revised Section 3.2 to clarify the assumptions underlying Theorem 2.

AuthorsOpenReview ↗

Author Response (Part 3/4)

Example 1 compares against an identity-blind shared policy, whereas an agent-ID-conditioned policy could also represent the required behaviors. Do MAPPO and GHAPPO use agent IDs in the experiments? What additional benefit does action replacement provide over agent-ID conditioning or parameter-matched private action heads?

In our experiments, both MAPPO and GHAPPO receive a one-hot agent ID as part of their local observations. HAPPO uses a separately parameterized actor for each agent.

Compared with ID conditioning, action replacement uses a small set of private parameters to encode agent-specific behavior. Each agent’s module uses its hidden state to determine the probability of replacing the action proposed by the shared policy, giving agents a direct way to specialize within a group. At the same per-agent policy size as the private action heads, action replacement allows agents within a group to reuse most of their policy parameters through a shared backbone. This reduces the total number of trainable parameters and allows agents to pool their experience when updating the backbone, which can improve training efficiency and stability.

We also added direct comparisons with ID-conditioned sharing and parameter-matched private action heads. The table reports test win rates after 10710^7 environment steps.

Algorithm3s5z5m_vs_6m
GHAPPO action replacement95.00% ± 3.16%50.00% ± 6.32%
GHAPPO ID-conditioned sharing80.09% ± 6.14%40.83% ± 4.96%
GHAPPO parameter-matched private action heads84.52% ± 3.81%27.67% ± 5.07%

The action-replacement variant achieved the highest mean test win rate on both maps. The comparison supports the effectiveness of combining within-group parameter sharing with agent-specific action replacement at the same per-agent policy size.

We have revised Section 4.4 with these new ablations on the action-replacement module.

The main text provides GHAPPO’s optimization objective and framework diagram, but the complete training procedure remains insufficiently specified. Could the authors provide pseudocode for the complete training procedure?

We thank the reviewer for this suggestion. We have added pseudocode for the complete GHAPPO training procedure to the revised manuscript. We hope this addition makes the algorithm easier to understand and reproduce.

Inputs. Agent set N\mathcal{N}, number of episodes KK, steps per episode TT, and batch size BB.

  1. Choose a group count gg guided by F(g)=U(g)−λP(g)F(g)=U(g)-\lambda P(g) in the group granularity analysis.

  2. Construct an ordered, disjoint partition {G1,…,Gg}\lbrace G_1,\ldots,G_g\rbrace.

  3. Initialize the group-shared backbones {θ0Gm}m=1g\lbrace\theta_0^{G_m}\rbrace_{m=1}^{g}, replacement parameters {ϕi,0}i∈N\lbrace\phi_{i,0}\rbrace_{i\in\mathcal{N}}, centralized critic Vψ0V_{\psi_0}, and buffer D\mathcal{D}.

  4. For k=0,1,…,K−1k=0,1,\ldots,K-1:

    1. Collect trajectories using the executed joint policy π^θk,ϕk\hat{\boldsymbol{\pi}}_{\boldsymbol{\theta}_k,\boldsymbol{\phi}_k} and store the transitions in D\mathcal{D}.

    2. Sample BB transitions and compute the advantage estimates A^k(st,at)\hat{A}_k(s_t,\mathbf{a}_t) and return targets \hat{R\}_t using VψkV_{\psi_k} and GAE.

    3. Set MkG1(st,at)←A^k(st,at)M_k^{G_1}(s_t,\mathbf{a}_t)\gets\hat{A}_k(s_t,\mathbf{a}_t).

    4. For m=1,2,…,gm=1,2,\ldots,g:

      1. Aggregate the trajectories of all agents i∈Gmi\in G_m.

      2. Jointly update θGm\theta^{G_m} and ϕGm={ϕi:i∈Gm}\boldsymbol{\phi}^{G_m}=\lbrace\phi_i:i\in G_m\rbrace by maximizing the GHAPPO clipped objective.

      3. If m<gm<g, update the factor for the next group.

        \gets r_k\left(\theta_{k+1}^{G_m}, \boldsymbol{\phi}_{k+1}^{G_m}\right) M_k^{G_{1:m}}.$$
    5. Update the centralized critic.

      \gets \arg\min_{\psi} \frac{1}{BT} \sum_{b=1}^{B}\sum_{t=0}^{T-1} \left(V_{\psi}(s_t)-\hat{R}_t\right)^2.$$

We have added pseudocode for the complete training procedure in Appendix E.

Reviewer_sP8FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Authors propose GHAPPO, which extends sequential heterogeneous MARL. They derive a grouped multi‑agent advantage decomposition theorem to enable group‑level sequential updates. They aim to reduce the number of dependent update stages. To recover intra‑group heterogeneity brought by within‑group parameter sharing, they propose a lightweight agent‑specific action‑replacement module. They conducted evaluations on SMAC and SMACv2 benchmarks, with comparison to MAPPO and HAPPO.

Critical strengths

The motivation is clear and worth further investigation.

The proposed approach to balancing the extra-group and intra-group sequential updates seams sound.

The experimental results are encouraging on MARL benchmarks.

Critical weaknesses

Theorem 2’s granularity‑vs‑overhead trade‑off result is derived under a very strong assumption and further requires second‑order Taylor approximations. The analysis uses a stylised proxy objective F(k)=U(k)-\lambda P(k), which is not the actual MARL surrogate objective optimised in implementation. The analysis gives a continuous optimal grouping number \tilde{k}^, but the authors explicitly note this does not guarantee the discrete empirical optimal group count. There is no empirical validation showing that the theoretical formula predicts good grouping hyper‑parameters in practice. In Table 3 ablation on 8m vs 9m, best performance is with k=2. However, I can’t see there is any evidence showing that it aligns with the theoretical \tilde{k}^, which is given from task properties.

The main concerns about the paper is about the evaluation. As mentioned by authors in the related work section, GoMARL performs dynamic group formation within value-factorization frameworks and B2MAPO gives batch-wise sequential updates to reduce sequential dependency depth for HAPPO-family algorithms. They seam to be suitable baselines to the proposed approach. Unfortunately I can’t find any reasons from the paper for not comparing the proposed approach to them. It would be helpful if authors can give some comments on this.

Questions for authors

Please refer to the above-mentioned critical weaknesses.

3 comments
AuthorsOpenReview ↗

Author Response (Part 1/3)

We sincerely thank Reviewer sP8F for the constructive feedback. We have addressed the critical weaknesses by providing additional explanations and experimental results, all of which have been incorporated into the revised manuscript.

Theorem 2’s granularity‑vs‑overhead trade‑off result is derived under a very strong assumption and further requires second‑order Taylor approximations.

Theorem 2 describes a trade-off between a modeled policy-improvement signal and the cost of sequential updates. The model assumes a common positive contribution scale AA, bounded individual likelihood ratios, and equal-size groups. Our choice of A>0A>0 is motivated by HAPPO’s discussion of improvement directions. If each selected action has positive conditional advantage given the preceding actions, then the resulting joint action has positive advantage. The decomposition itself holds for signed advantages, and the bounding argument can be extended to the magnitude of their contributions by taking absolute values. We assume that each individual likelihood ratio satisfies 0≤rℓ≤1+ε0\le r_\ell\le 1+\varepsilon, using the PPO clipping ratio as a reference for ε\varepsilon. The Equal-size groups is the default partition when role information is unavailable.

We use a second-order Taylor expansion to obtain an interpretable closed-form solution for the continuous proxy. The approximation is accurate when x=nlog⁡(1+ε)/kx=n\log(1+\varepsilon)/k is sufficiently small. For a given set of proxy parameters, the original F(k)F(k) can also be evaluated over the feasible equal-size group counts.

We have revised Section 3.2 of the manuscript to clarify the assumptions underlying Theorem 2 and the scope of the analysis.

The analysis uses a stylised proxy objective F(k)=U(k)-\lambda P(k), which is not the actual MARL surrogate objective optimised in implementation.

F(k)F(k) is an analytical proxy that can guide the choice of group count before training. Once the partition is specified, GHAPPO keeps it fixed and optimizes the policy using the group-wise PPO surrogate in Eq. (12) to improve the expected team return. We have also added pseudocode that shows how the partition is constructed and how the policy is updated during training.

Inputs. Agent set N\mathcal{N}, number of episodes KK, steps per episode TT, and batch size BB.

  1. Choose a group count gg guided by F(g)=U(g)−λP(g)F(g)=U(g)-\lambda P(g) in the group granularity analysis.

  2. Construct an ordered, disjoint partition {G1,…,Gg}\lbrace G_1,\ldots,G_g \rbrace.

  3. Initialize the group-shared backbones {θ0Gm}m=1g\lbrace \theta_0^{G_m}\rbrace_{m=1}^{g}, replacement parameters {ϕi,0}i∈N\lbrace\phi_{i,0}\rbrace_{i\in\mathcal{N}}, centralized critic Vψ0V_{\psi_0}, and buffer D\mathcal{D}.

  4. For k=0,1,…,K−1k=0,1,\ldots,K-1:

    1. Collect trajectories using the executed joint policy π^θk,ϕk\hat{\boldsymbol{\pi}}_{\boldsymbol{\theta}_k,\boldsymbol{\phi}_k} and store the transitions in D\mathcal{D}.

    2. Sample BB transitions and compute the advantage estimates A^k(st,at)\hat{A}_k(s_t,\mathbf{a}_t) and return targets R^t\hat{R}_t using VψkV_{\psi_k} and GAE.

    3. Set MkG1(st,at)←A^k(st,at)M_k^{G_1}(s_t,\mathbf{a}_t)\gets\hat{A}_k(s_t,\mathbf{a}_t).

    4. For m=1,2,…,gm=1,2,\ldots,g:

      1. Aggregate the trajectories of all agents i∈Gmi\in G_m.

      2. Jointly update θGm\theta^{G_m} and ϕGm={ϕi:i∈Gm}\boldsymbol{\phi}^{G_m}=\lbrace\phi_i:i\in G_m\rbrace by maximizing the GHAPPO clipped objective.

      3. If m<gm<g, update the factor for the next group.

        \gets r_k\left(\theta_{k+1}^{G_m}, \boldsymbol{\phi}_{k+1}^{G_m}\right) M_k^{G_{1:m}}.$$
    5. Update the centralized critic.

      \gets \arg\min_{\psi} \frac{1}{BT} \sum_{b=1}^{B}\sum_{t=0}^{T-1} \left(V_{\psi}(s_t)-\hat{R}_t\right)^2.$$

We have revised Section 3.2 to clarify how Theorem 2 guides the selection of candidate group counts. We have also added pseudocode for the complete training procedure in Appendix E.

AuthorsOpenReview ↗

Author Response (Part 2/3)

The analysis gives a continuous optimal grouping number \tilde{k}^, but the authors explicitly note this does not guarantee the discrete empirical optimal group count.

Applying Theorem 2 to a specific task requires estimating the cost of each additional sequential update stage and the advantage scale, then solving the continuous approximation to the proxy. Determine the optimal group count directly from the proxy can be difficult across tasks. In practice, we used the continuous estimate as a guide and ran short pilot experiments to compare nearby feasible group counts. We provide a detailed example later in the response.

We have revised Section 3.2 to clarify how the proxy guided our choice of candidate group counts and how we selected the final count empirically.

There is no empirical validation showing that the theoretical formula predicts good grouping hyper‑parameters in practice. In Table 3 ablation on 8m vs 9m, best performance is with k=2. However, I can’t see there is any evidence showing that it aligns with the theoretical \tilde{k}^, which is given from task properties.

We thank the reviewer for the constructive feedback. Theorem 2 provides a reference for balancing a modeled policy-improvement signal against the cost of sequential updates. To provide an empirical validation showing that the theoretical formula predicts good grouping hyper‑parameters in practice, we illustrate how it can guide the choice of candidate group counts on 8m_vs_9m with the following detailed parameters.

ParameterDescription
n=8The number of controllable agents in 8m_vs_9m
ε=0.2\varepsilon=0.2Default PPO clipping ratio
C=0.02C=0.02Estimated by fitting a linear model to the network-update times from pilot runs (0.0216s per additional group in Table 3)
λ=1\lambda=1Equal-Weight sequential update overhead and policy improvement proxy
A=0.02A=0.02The advantage scale estimated from pilot runs

Using these values, we obtain

= nA\big[(1+\varepsilon)^n-1\big] \\ = 8\times0.02\times \big[(1.2)^8-1\big] \\ \approx 0.53$$ $$\widetilde{k}^{*} = \sqrt{\frac{B}{2\lambda C}} -\frac{n\ln(1+\varepsilon)}{2} \\ = \sqrt{\frac{0.53}{2\times 1 \times 0.02}}-4\ln(1.2) \\ \approx 2.91$$ In `8m_vs_9m`, the feasible group counts are $k\in\{1,2,4,8\}$, so $k=2$ and $k=4$ are the two feasible counts that bracket the continuous estimate. Table 3 reports the full sweep. In Table 3, $k=2$ achieves the highest mean test win rate among the configurations evaluated. Network-update time increases with the number of groups, whereas mean win rate does not increase monotonically. This pattern is consistent with the trade-off in Theorem 2. We also calculate the optimal group count for other SMAC maps: | Map | Optimal group count ($\tilde{k}^*$) | | ---------- | ----------------------------------- | | 3s5z | 2.90 | | 5m_vs_6m | 1.47 | | 10m_vs_11m | 4.18 | | MMM2 | 1.89 | | corridor | 1.89 | For consistency across experiments, we partition agents into two groups by index on all maps without predefined roles. On maps with predefined roles, we group agents by unit type. In Appendix C, we have added this calculation and explained how Theorem 2 can guide the selection of candidate group counts for empirical evaluation.
AuthorsOpenReview ↗

Author Response (Part 3/3)

The main concerns about the paper is about the evaluation. As mentioned by authors in the related work section, GoMARL performs dynamic group formation within value-factorization frameworks and B2MAPO gives batch-wise sequential updates to reduce sequential dependency depth for HAPPO-family algorithms. They seam to be suitable baselines to the proposed approach. Unfortunately I can’t find any reasons from the paper for not comparing the proposed approach to them. It would be helpful if authors can give some comments on this.

We thank the reviewer for the feedback. Our initial evaluation focused on MAPPO and HAPPO because they provide two reference points for GHAPPO’s design. MAPPO uses a shared actor with simultaneous updates, while HAPPO uses separate actors with agent-wise sequential updates. HAPPO also provides the multi-agent advantage decomposition that motivates our group-wise formulation.

Following the reviewer's suggestion, we added comparisons with GoMARL and B2MAPO. The results are summarized below:

Algorithm3s5z5m_vs_6m10m_vs_11mMMM28m_vs_9mcorridor
GHAPPO95.94% ± 3.16%55.83% ± 4.96%82.75% ± 4.57%68.00% ± 5.02%87.00% ± 5.83%96.67% ± 2.24%
GoMARL95.00% ± 2.44%62.25% ± 8.36%75.00% ± 3.75%77.19% ± 3.55%82.63% ± 4.68%93.44% ± 3.26%
B2MAPO93.75% ± 3.13%42.23% ± 6.39%62.96% ± 2.32%63.47% ± 2.81%76.88% ± 4.84%92.95% ± 3.14%

GHAPPO achieved the highest mean test win rate on four of the six maps and exceeded B2MAPO on all six. The advantages of GHAPPO’s grouping scheme are more pronounced in scenarios with more agents. While GoMARL achieved the highest mean win rate on 5m_vs_6m and MMM2, it requires 17.08% more trainable parameters than GHAPPO, including those in its grouping and value-mixing modules. Overall, GHAPPO achieved the highest test win rate averaged across the six SMAC maps.

GoMARL learns group membership for value factorization, and B2MAPO uses a DAG to determine the order of policy updates. These choices are orthogonal to how GHAPPO how GHAPPO parameterizes group policies and updates them sequentially. GoMARL’s grouping approach and B2MAPO’s scheduling approach could therefore inform GHAPPO’s group membership and update order, respectively, with the potential to further improve performance.

To the best of our knowledge, our work is the first to extend the agent-wise advantage decomposition to groups by expressing the joint advantage as a sum of conditional group advantages. This decomposition provides the basis for group-wise PPO update. Based on this theorem, we propose GHAPPO, in which each group shares one actor backbone and each agent has a lightweight private action-replacement module that allows it to specialize within the group.

In Appendix G, we have included these results and clarified the relationship among the methods.

Comments on the paper

2 comments

Dear reviewers,

Thanks for serving as a reviewer. As the discussion period comes to a close and the authors have submitted their rebuttals, I kindly ask you to take a moment to review them and provide any final comments.

If you have already updated your comments, please disregard this message.

Thank you once again for your dedication to the OpenReview process.

Best,

Area Chair

AuthorsOpenReview ↗

Summary of Responses and Revisions

Dear Area Chair and Reviewers,

We would like to thank the Area Chair for coordinating the discussion and the reviewers for their constructive feedback. We appreciate your recognition of the motivation behind GHAPPO and the value of combining within-group parameter sharing with sequential group updates. Your feedback helped us strengthen the rigor and readability of the paper.

In response to the feedback from the reviewers, we have made the following major revisions:

  1. Assumptions and approximation in Theorem 2 (Section 3.2). We clarified the assumptions underlying the analytical proxy, the conditions for the second-order Taylor approximation, and how the resulting continuous estimate can guide the selection of candidate group counts.
  2. Group-count selection in practice (Appendix C). We provided a detailed example on 8m_vs_9m. The continuous estimate k~∗≈2.91\tilde{k}^{*}\approx 2.91 lies between the feasible group counts k=2k=2 and k=4k=4. In the sweep reported in Table 3, k=2k=2 achieved the highest mean test win rate among the configurations evaluated.
  3. Additional baselines (Appendix G). We added comparisons with GoMARL and B2MAPO on six SMAC maps and clarified how the three methods use grouping. GHAPPO achieved the highest mean test win rate on four maps and outperformed B2MAPO on all six.
  4. Network-update efficiency (Section 4.2). Our profiling on 3s5z shows that GHAPPO reduced network-update time by 20.10% relative to HAPPO. The actor backward pass and optimizer step accounted for most of the savings.
  5. Action replacement ablation (Section 4.4). We added direct comparisons with ID-conditioned sharing and private action heads matched for per-agent policy size. The action-replacement variant achieved the highest mean test win rate on both tested maps.
  6. Training procedure (Appendix E). We added pseudocode covering group partitioning, trajectory collection, advantage estimation, sequential group updates, and critic optimization.

We hope these substantial revisions adequately address the concerns raised by the reviewers and further strengthen the quality of our work. We would be deeply grateful if you could consider these clarifications and new results in your final assessment.

Best regards,

The authors of Submission 39562

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.