acceptodds
Under review as a conference paper at ICLR 2027

From MARL Policies to Swarm Laws: The Compressibility Spectrum of Learned Coordination

Abstract

Swarm intelligence and multi-agent reinforcement learning (MARL) are two com- mon paradigms for multi-agent coordination, yet they construct collective behavior from opposite directions. Swarm rules such as Boids and mean-shift consensus are local, interpretable, and independent of population size in their formulation. However, they must be designed by humans. MARL policies instead learn co- ordination from environmental rewards and can adapt to different tasks through training. However, the learned policies remain black boxes and may degrade when deployed beyond their training population. These complementary strengths mo- tivate a more specific question. When can a trained collective policy be distilled into a compact local controller that preserves useful organization and transfers across population sizes? Our results show that the answer depends jointly on the policy, its observables, and the rule language. A causal task makes the information boundary explicit: when a hidden bit determines the correct target among locally indistinguishable options, blind mining falls to coin-flip performance, while expos- ing that bit restores the teacher’s performance curve. In SMACv2, three unrelated compressors reveal a sharp Protoss–Terran contrast. On the compressible side, mined local laws preserve learned organization and transfer beyond the training population. In continuous pursuit–encirclement, a sixteen-number law exceeds its own teacher and maintains capture performance up to 32× the training population at constant density. We further remove the hand-designed vocabulary and recover compact executable programs from a frozen grammar of 208 generic local terms. The recovered structure changes systematically across pursuit, coverage, and public MPE Predator–Prey tasks. Finally, direct return optimization and mining from dif- ferent teachers separate rule-class capacity from teacher-specific content. Together, these results characterize when learned collective behavior can be compressed into a local swarm law, connecting MARL’s automatic discovery of coordination with the interpretability and population scalability of swarm intelligence.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, swarm intelligence, policy distillation, interpretable reinforcement learning, collective behavior, population scalability, policy compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eDL7OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper asks when a trained MARL policy can be distilled into a compact local controller that preserves collective performance and transfers across population sizes. It introduces a local policy projection objective and a notion of local compressibility, followed by experiments involving decision trees, hand-designed local controllers, and sparse programs. The paper attributes successful compression to sufficient local information, adequate rule expressivity, and behavioral stability under population transfer.

Critical strengths

The paper raises a worthwhile question: whether coordination learned by MARL admits a compact local representation that remains useful beyond the training population. The distinction between being executable at a new population size and preserving collective performance at that size is also relevant. These provide a promising starting point for a research contribution.

Critical weaknesses
  • The paper does not develop its research question into a model that yields substantive conclusions. Section 3.1 gives a generic complexity-regularized imitation objective, and Section 3.2 defines conditions under which a controller would count as successfully compressed. However, the paper does not analyze this formulation to establish when such a controller exists, what determines its required complexity, or which properties of the teacher make compression possible. Proposition 1 establishes an elementary indistinguishability obstruction, but this is not developed into a broader characterization. The central gap is therefore between posing the question and deriving an informative answer.
  • The proposed frontier does not adequately characterize teacher-specific compressibility. The expression K↦max⁡Ω(f)≤KJN(f)K\mapsto\max_{\Omega(f)\le K}J_N(f) measures the best return achievable within a complexity-constrained controller class. Once the environment, observables, and rule language are fixed, this expression does not depend on the teacher or the probing distribution. Two teachers can therefore have the same stated frontier despite differing substantially in how their behavior can be represented or extracted. The paper needs to distinguish controller-class capacity from teacher-policy compressibility within its formulation, not only discuss that distinction after reporting results.
  • The reported observations are not organized around a sufficiently developed central argument. Numerous extraction procedures, controller classes, and task settings are considered, but the paper does not connect their outcomes through a model that produces a specific explanatory claim. It remains unclear what the reader should conclude beyond the fact that the tested procedures perform differently in different settings. Even if all reported numbers are reproducible, this would not resolve the missing conceptual contribution. In addition, incomplete definitions of the motion primitives, gates, and generic grammar make the underlying technical objects difficult to assess.
Questions for authors
  • What is the principal new insight of the paper, stated independently of the performance numbers on the selected tasks? Which part of the formulation or analysis establishes that insight?
  • What properties of a teacher policy determine whether it admits a compact local representation? Can the proposed framework distinguish compressible and resistant policies without simply defining the distinction through the outcome of extraction?
  • How is Ω(f)\Omega(f) computed for each controller class, including its gates and primitives? What relationship does this complexity have to the policy structure being represented?
  • How should the performance frontier in Section 3.2 be interpreted as teacher-specific compressibility when the optimization itself does not depend on the teacher?
Reviewer_LwamOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a interesting question that when a learned MARL policy can be compressed into a interpretable local swarm rule, while preserving task performance and generalizing to larger populations. To answer this, the authors propose their compressibility spectrum based on local information, rule expressivity and population transfer. They also provide a proposition showing how missing local information can limit compression.

The authors evaluate their framework across several multi-agent coordination tasks. The experiments examine the effects of local information and rule complexity, demonstrate population transfer in some settings, and investigate how extracted rules reflect teacher behavior and collective organization.

Critical strengths

In this paper, the author conducted a lot of experiments, and most claims in the paper are supported by the experimental evidence. I particularly appreciate the detailed analyzes in Sections 4.5 and 4.6. Section 4.5 investigates rule expressivity by examining the effects of different rule primitives, automatically discovered gates, and generic rule grammars. Section 4.6 analyzes how the extracted rules reflect teacher-specific behavior and how action fidelity relates to collective organization and task performance. These analyzes provide useful insights beyond the performance comparisons.

Critical weaknesses
  1. Limited characterization of necessary information. Section 4.2 shows that the hidden bit gg is necessary for successful compression, while Section 4.3 shows that explicit role labels can be replaced by other local observations. These results raise a broader question: how can we identify which information is necessary for compressing a new MARL policy? I would like to see a more general, formal characterization of the information required to preserve the learned collective behavior.

  2. Limited evidence for the diagnostic capability. The paper suggests that compression resistance can help identify whether the bottleneck lies in information, rule expressivity, or population transfer. However, Section 4.2 demonstrates this in a controlled task where the missing information is already known. In contrast, Section 4.1 observes different compression outcomes between Protoss and Terran without identifying their causes. Therefore, the current experiments do not fully establish how the proposed framework can diagnose compression failure when the underlying bottleneck is unknown.

  3. Minor: The results are hard to read. Across the Section 4, a lot of numerical results scattered throughout the text, making them difficult to read and compare. I recommend author use some tables to re-organize their results.

Questions for authors
  1. Characterization of necessary information. (Corresponding to Weakness 1) Can the authors provide a more general, formal characterization of what information is necessary for preserving a MARL policy's collective behavior? In Sections 4.2 and 4.3, the relevant information is known from the task design. How can we identify the necessary information for a new MARL policy where this is not known in advance?

  2. Diagnosis of compression failure.(Corresponding to Weakness 2) In Section 4.1, the compression outcomes differ significantly between Protoss and Terran. Can the authors further analyze what causes this difference? In particular, how can the proposed framework distinguish limitations in local information, rule expressivity, and population transfer?

  3. Neural baselines under constant-density scaling. In Section 4.4, DeepSets and Attention-GNN are evaluated at smaller population sizes, but not under the full constant-density scaling protocol. Could the authors evaluate these same trained models across the full scaling range for a more direct comparison?

  4. Teacher–student collective organization. Section 4.6 analyzes how group geometry relates to task performance. Could the authors also report the same geometry metrics for the original MARL teacher, alongside the mined law and other controllers? This would help clarify whether the mined law preserves the teacher's coordination pattern or develops a different, more effective one.

Reviewer_GkEMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies a middle ground between swarm rules and multi-agent RL, which are complementary tools for studying multi-agent policies. Swarm rules are local, independent, and independent of population size, whereas historically, MARL policies learn coordination from environmental rewards, and their policies can be complex and hard to interpret. This paper suggests using a trained collective policy and distilling it into a decentralized control policy to preserve organization and transfer across population sizes. This paper empirically (in SMACv2, continuous pursuit-encirclement, and MPE Predator-Prey tasks) characterizes when learned collective behavior can be compressed into a local swarm law.

Critical strengths

It is a very neat idea to try to "filter" multi-agent RL policies into swarm laws, which have seen a lot of success in practice. In this sense, the paper is very cleanly motivated, and I think the idea here is novel.

Critical weaknesses
  1. I disagree with the premise of the statement in the abstract that the learned policies from mean-field MARL may "remain black boxes and may degrade when deployed beyond their training population," since it disregards an entire line of work on mean-field multi-agent RL that was created to solve this exact problem. In mean-field MARL [1,2,3], a policy is learned by allowing an agent to interact with a "representative agent" (the mean-field of its neighbors) of the population, getting an error that decays on the order of O(1/sqrt(N)) where N is the number of agents. While this continues to display a degradation beyond the training population, works on mean-field sampling [4,5,6] show that you can sample from the population and continue to generalize well, incurring a training error that is completely independent of the number of agents. I think all these works should be discussed in the related work, and the positioning of this work should be clarified a lot more with respect to them.

  2. It's hard to understand the main point of the result here: ideally, a theoretical characterization of when we can transfer the multi-agent policy to a swarm law would be really cool, but this is currently missing. Similarly, the empirical characterization also appears somewhat limited (only 3 environments), reading as somewhat preliminary work. Can the authors expand on both of these characterizations?

References:

[1] Mean field multi-agent reinforcement learning. Yang et. al. (2018)

[2] Mean-field controls with Q-learning for cooperative MARL: convergence and complexity analysis. Gu et. al. (2021)

[3] Scalable multi-agent reinforcement learning for networked systems with average reward. Qu et. al. (2020)

[4] Mean-field sampling for cooperative multi-agent reinforcement learning. Anand et. al. (2025)

[5] Learning Approximate Nash Equilibria in Cooperative Multi-Agent Reinforcement Learning via Mean-Field Subsampling. Anand et. al. (2026)

[6] Graphon Mean-Field Subsampling for Cooperative Heterogeneous Multi-Agent Reinforcement Learning. Anand et. al. (2026)

Questions for authors

I'd certainly appreciate it if the authors could respond to the questions above in the weaknesses section, and am certainly open to increasing my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.