acceptodds
Under review as a conference paper at ICLR 2027

Capacity Scaling Fragility in Multi-Agent Policy Optimization

Abstract

Increasing network capacity is generally expected to improve the performance of reinforcement-learning policies, or at least not to harm it. However, we find that this expectation can fail in cooperative multi-agent reinforcement learning (MARL), because network width interacts with how each policy network is up- dated. On SMACv2 protoss 5 vs 5, under standard tuned hyperparameters, in- creasing HAPPO’s hidden size from 64 to 256 reduces the win rate from 0.474 to 0.368 over five seeds, whereas MAPPO does not show the same degradation. A four-way identification experiment crossing sequential versus synchronous up- dates with independent versus shared parameters shows that the capacity penalty appears in every standard configuration except shared, synchronous MAPPO. Further interventions transfer this protection to independent actors, showing that parameter sharing itself cannot explain the difference. Across these configura- tions and interventions, the key quantity that consistently separates protected and fragile regimes is the realized policy-update scale. At the same nominal learn- ing rate, increasing network width amplifies the net per-iteration policy KL by 2.2–3.6×, while fragile topologies exhibit 2.4–12× larger absolute policy dis- placement than protected ones. When the larger network’s realized KL is brought back to the small network’s default level, its performance is restored. Reduc- ing the learning rate shrinks the capacity gap in every fragile configuration we test, while directly bounding per-update KL provides a partial recovery. Together, these results suggest that network capacity, update topology, and learning rate jointly determine the realized optimization regime: the same nominal learning rate can produce substantially different policy updates across network widths and training configurations. At hidden size 256, the default-rate comparison gives MAPPO a 0.094 performance advantage over HAPPO; applying the same lower learning rate to both algorithms reduces this margin roughly fourfold to 0.022. Thus, fixed-architecture, fixed-learning-rate MARL comparisons can confound algorithmic differences with update-scale calibration.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, policy optimization, network scaling, policy KL, optimization stability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_J9McOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates why wider networks hurt HAPPO in several SMACv2 settings but not shared MAPPO. It varies update order, parameter sharing, and data per update, then measures policy KL and tests learning-rate and batch interventions. Additional environments and reverse interventions examine the limits of the update-scale explanation.

Critical strengths
  1. The topology controls answer a concrete question. The four-way comparison and no-prefix control rule out sequential updating, parameter sharing, and the cumulative ratio as sufficient explanations on their own. The actor/critic width split further localizes the capacity effect (Sections 3 and 4).

  2. The accumulated-batch experiment is a useful control. Collecting five segments with a frozen policy and critic reproduces the larger-batch result without adding rollout threads. Together with the phase-count control, this strengthens the evidence for the per-network update regime (Appendix D and Table 7).

  3. Several independent checks support a qualified mechanism. Exact KL, learning-rate changes, KL stopping, and the HATRPO check provide more than an observational correlation. The paper also reports the failed de-pooling threshold and MPE performance-rescue prediction, rather than claiming a universal KL rule (Appendices D, E, and H).

Critical weaknesses
  1. The learning-rate intervention needs a precise specification. The default actor and critic rates are equal, but it is unclear whether both change in the rescue experiments. The actor/critic width control does not resolve this. KL stopping provides direct but partial support, so the role of critic optimization in the learning-rate rescue remains uncertain (Section 5 and Appendix A).

  2. Cross-condition scores use different training windows. Last-ten means span five times more environment steps in dose-matched runs than in standard runs. This does not confound h64 versus h256 within a condition, but it complicates comparisons of gap reduction across conditions. Table 9 does not cover the dose and accumulated-batch controls (Appendices A and F).

  3. Table 8 mixes reference cohorts. Its caption attributes the reference rows to the two-seed Table 6 runs, but the MAPPO, synchronous-independent, and HAPPO gaps are copied from Table 2 instead. For example, the instrumented MAPPO gap is -0.144, not -0.017. The distinction matters when comparing KL and performance; uncertainty is also substantial in several rescue conditions (Tables 6, 8, and 10).

Questions for authors
  1. Were the learning-rate interventions actor-only or joint actor/critic changes? If joint, is an actor-only result available?

  2. Do the dose and accumulated-batch conclusions hold when scores use the same final environment-step interval?

  3. Please reconcile the reference cohorts in Table 8. Using matched-run KL and performance, which quantitative rescue claims remain supported by the reported uncertainty?

Reviewer_J13BOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates fragility when scaling network width in multi-agent policy optimization. On SMACv2, increasing the hidden size from 64 to 256 reduces HAPPO’s win rate from 0.474 to 0.368, whereas MAPPO does not exhibit the same degradation. The authors investigate this difference through a factorial comparison of sequential versus synchronous updates and independent versus shared parameters, supplemented by per-update experience matching, exact policy-KL measurements, and learning-rate interventions. The proposed explanation is that update topology determines each policy network’s baseline optimization regime, while increasing width amplifies realized policy changes at a fixed learning rate. Reducing the learning rate, increasing the experience used per update, and introducing target-KL early stopping mitigate the degradation to varying degrees. The experimental design is systematic, and several controls support this explanation.

Critical strengths
  1. Systematic controlled experiments. The paper goes beyond attributing the performance difference to parameter sharing or update order. It disentangles these factors and examines the cumulative importance ratio, providing a more detailed analysis of the optimization process experienced by each policy network.
  2. Intervention-based support for the proposed mechanism. The authors complement KL measurements with interventions on learning rate, per-update experience, and KL constraints. The phase-count and accumulated-batch controls are particularly useful for understanding why experience matching improves performance.
Critical weaknesses
  1. The additional insight specific to MARL remains unclear. The paper demonstrates that multi-agent update topologies induce different per-network optimization regimes. However, its central explanation—that width, batch size, and learning rate jointly affect realized policy-update scale—also applies to general PPO optimization. Although the authors examine role heterogeneity and agent count, the paper should clarify what new understanding it provides beyond established optimization sensitivity. A single-agent or fixed-teammate control with matched optimization conditions would help distinguish these effects.
  2. The motivation for focusing on width and the distinction from related work are insufficiently developed. Section 4.2 and Appendix D connect the observed width effect to width–learning-rate coupling, but do not sufficiently explain why width is the principal scaling dimension for the research question. The related-work discussion should more clearly distinguish the paper’s question and findings from prior RL studies on depth, width, and model-size scaling.
Questions for authors
  1. What distinguishes the findings from general PPO optimization sensitivity? After matching per-network sample counts, update counts, and realized policy KL, does multi-agent interaction introduce additional width-scaling fragility? Could the authors provide a single-agent or fixed-teammate control, or explain how the existing experiments distinguish these effects?
  2. Why is width the primary scaling dimension studied? Beyond width–learning-rate coupling, how does this choice follow from the research question? What additional insight does the paper provide relative to existing RL research on depth, width, and model-size scaling?
  3. Under what conditions does the update-scale explanation predict performance degradation and recovery? MPE exhibits KL amplification without performance recovery from learning-rate reduction, while larger networks perform better on SMACv1. What conditions explain these differences, and how could one determine whether a new task would benefit from update-scale calibration?
Reviewer_zkakOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper starts with an observation that widening policy networks hurts HAPPO but not MAPPO. To explain this difference, the authors first vary policy sharing and update order. Since neither alone explains the results, they examine the data used in each update and find that larger batches largely remove the penalty for independent policies. To understand why, they measure policy KL divergence and find that wider networks make larger policy changes at the same learning rate. They therefore test smaller updates and find that lowering HAPPO’s learning rate brings the wider network’s win rate back to the smaller network’s level, while imposing a KL limit recovers less of the lost performance. Together, these results suggest that widening hurts when it makes training updates too large. Because lowering the learning rate also substantially shrinks MAPPO’s lead over HAPPO, the authors argue that fixed-learning-rate comparisons can mistake unsuitable training settings for an algorithmic weakness.

Critical strengths

The paper examines its findings in depth through a thorough hyperparameter tuning analysis. The authors use complementary controls to make the effects of network width and learning rate easier to interpret. For example, distinguishing larger batches from fewer updates helps separate plausible explanations for the improvement. Measuring actual policy changes strengthens the analysis by explaining why the same learning rate can produce different outcomes.

Critical weaknesses
  • The paper is poorly written and the contribution is unclear. The abstract and the introduction are especially representitive, which compress many configurations, numerical comparisons, and interventions into a dense experimental narrative. The whole paper is more close to a technical note that records the authors’ parameter tuning and analysis process than a research paper that provides sufficient insights to the community.

  • Increasing network size often requires adjusting the learning rate and other training settings, so showing that such adjustments recover performance is not, by itself, a new finding. The paper should explain more clearly on what its analysis of multi-agent training adds to this established understanding, which findings go beyond known interactions between network width, learning rate, and batch size, and why they provide a distinct insight into MARL.

  • The empirical scope is very narrow. Although the appendices include HATRPO, SMACv1, and MPE, the main evidence is concentrated on related policy-optimization algorithms in SMACv2, especially Protoss 5 vs. 5. Crucially, MPE reproduces KL amplification but not the performance recovery from reducing the learning rate. The paper therefore provide limited generalizable conclusions to promote MARL research.

  • Figure readability and layout issues. Figure 1 contains very small labels despite substantial unused horizontal space. There are also lots of conspicuous blank regions between figure captions and the following paragraphs, such as lines 73-78.

Questions for authors
  • Beyond the closely related MAPPO/HAPPO/HATRPO family, how broadly should the mechanism apply? Is there evidence that it persists across a wider capacity range, increased depth, or policies based on Transformers or Mamba?

  • Would a width-aware parameterization or learning-rate scaling rule remove the topology-dependent penalty? Current findings are too weak to justify a new discovery.

  • Given that reducing the learning rate does not restore performance on MPE, what properties of an environment determine when excessive policy displacement explains the scaling penalty? Can the authors predict these conditions before observing the performance results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.