acceptodds
Under review as a conference paper at ICLR 2027

HiComm: Hierarchical Communication for Cooperative Multi-Agent RL

Abstract

Communication is essential for coordination in cooperative multi-agent reinforcement learning, yet existing protocols often overlook how the structure of the observation space can guide information exchange. We identify a shared hierarchical structure in many cooperative tasks: entities are organized into publicly indexed groups, while each agent observes only a subset of them. We formalize this setting as a hierarchically indexed Markov decision process (HI-MDP) and introduce HiComm, a Hierarchical Communication protocol that exploits the shared index for targeted information retrieval. Each receiver sends a request to a single teammate, chosen by the teammate's position in the hierarchy, and the teammate computes its reply from its local observation of one entry. Across cybersecurity, unit-control, and football benchmarks, HiComm achieves comparable or better performance than recent learned communication baselines while substantially reducing communication cost. Our theoretical analysis further shows that, compared with the vanilla approach (flat communication), which does not use the hierarchical index, HiComm has the same learning error bound, while its reward guarantee differs from that of flat communication only by the gap between the two protocols' optimal returns. In contrast, HiComm's communication complexity is only that of flat communication, where is the number of agents. These theoretical results support our empirical findings.

Primary area:
reinforcement learning
Keywords:
cooperative marl, communication

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_v8jnOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes HiComm, a hierarchical communication framework for cooperative MARL. It introduces the HI-MDP formulation and a two-stage protocol in which each agent queries one teammate for information about a single entity, reducing communication overhead.

Experiments on CC4, SMACv2, and GRF show that HiComm outperforms communication baselines in 9 of 10 configurations while consistently reducing communication costs. Ablations validate key design choices, and theoretical analysis establishes communication complexity advantages and learning guarantees under an idealized setting, rather than the actual PPO implementation.

Critical strengths
  1. The use of a shared group–entity index for targeted communication is a reasonable design choice. It enables agents to retrieve specific observations rather than exchanging generic learned messages.

  2. The ablation on the number of senders provides some evidence for the single-sender design. In CC4, increasing the number of senders substantially raises communication costs without consistently improving rewards.

Critical weaknesses
  1. The communication comparisons in Table 2 do not use matched bandwidth budgets. Since HiComm restricts each receiver to one sender and one entity, it is unclear whether the improvements come from hierarchical selection or simply reduced communication.

  2. In SMACv2 and GRF, sender-relative positions and distances are directly inserted into the receiver's observation without coordinate transformation, potentially introducing inconsistent spatial information.

  3. The auxiliary loss in Eq. (B.4) favors positive feature values. As acknowledged in Appendix B.1, this systematically biases entity selection against negative coordinates in SMACv2 and GRF.

  4. Tables 3–5 evaluate key design choices only on CC4. Moreover, the experiments use just three random seeds, and several performance improvements fall within the reported standard deviations.

  5. Theorem 6.1 analyzes an idealized primal-dual algorithm rather than the implemented PPO learner. The communication constraints are trivially satisfied, and the optimal-return gap remains unbounded, limiting the practical significance of the guarantees.

Questions for authors
  1. In Table 2, can the authors report the rewards of CACOM and T2MAC when their communication costs are constrained to match HiComm's measured KiB per receiver per episode?

  2. In SMACv2 and GRF, sender-relative coordinates are copied into the receiver's observation without transformation. How does the policy distinguish these values from observations expressed in the receiver's coordinate frame?

  3. In Eq. (B.4), negative feature values contribute zero to the auxiliary targets. How do the results on SMACv2 and GRF change when using the unbiased mask-count alternative mentioned in Appendix B.1?

  4. Tables 3–5 only report ablations on CC4. Does hierarchical group–entity selection outperform joint selection on SMACv2 and GRF? Can the authors provide variance estimates and statistical significance tests for the improvements in Table 2?

  5. Theorem 6.1 applies to a primal-dual learner rather than PPO, and its communication constraints are trivially satisfied. What specific guarantee does this theorem provide for the implemented HiComm? Can the authors establish a non-trivial bound on the optimal-return gap Δ?

Reviewer_KYrqOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies communication in cooperative multi-agent reinforcement learning when agents share a group–entity index but observe different subsets of the entities. This setting is formulated as a hierarchically indexed MDP and proposes HiComm, a two-stage protocol in which each receiver selects one teammate through the hierarchy, and that teammate selects one observed entity and returns its index and features. The receiver inserts the reply into an unobserved entry before choosing its action. Communication is trained jointly with the action policy using task gradients and an auxiliary loss encouraging the retrieval of missing information. Experiments on CC4, SMACv2, and Google Research Football report competitive or improved mean returns with lower communication costs than the evaluated baselines.

Critical strengths
  1. The shared entity index provides a useful interface for communication. HiComm uses the index to connect teammate selection, message content, and the location where the reply enters the receiver’s observation. Returning one named entity makes the exchanged information explicit and can avoid the burden of learning a separate message representation for low-dimensional features.
  2. The method shows a promising empirical return–communication trade-off across different tasks and learning backbones. In Table 2, HiComm has the highest mean return among the communication methods in nine of ten configurations and the lowest communication cost among them in all ten. The inclusion of no-communication and full-observation references, seed variation, and design ablations helps contextualize the results.
Critical weaknesses

The experiments do not sufficiently isolate the contribution of the hierarchy from the other changes in the protocol. HiComm combines single-teammate routing, hierarchical selection, raw entity-feature replies, direct observation completion, and auxiliary supervision based on all agents’ training observations. Table 3 usefully replaces hierarchical entity selection with a flat selection, but all such design ablations are on CC4, where each teammate group contains exactly one agent and the second stage of teammate selection is therefore degenerate. The flat entity variant reduces the mean return from 1572 to 1536 under IPPO and from 1917 to 1604 under MAPPO, but the ablation reports no variability and does not test the receiver’s hierarchical routing. Meanwhile, replacing raw replies with encoded replies also changes performance substantially. These results support the complete system, but leave unclear how much of its benefit comes specifically from hierarchical selection.

Questions for authors
  1. Could you provide further evidence isolating the contribution of hierarchical selection from the protocol’s other design choices? Controlled comparisons with non-hierarchical selection under comparable communication and training conditions, particularly on a semantic-grouping task such as SMACv2 or GRF, would help address this concern.
  2. Could you clarify the added value of learned selection over simple non-learned selection rules under the same execution-time information and communication constraints?
  3. Could you give a more detailed breakdown of the actual request and reply sizes for each method, including feature precision or quantization, entity and sender indices, null replies, and the counting of broadcast copies? Moreover, does the CACOM port retain its message quantization, and what precision is used? Reporting request rates and average bytes per step alongside episode totals would help separate protocol savings from silence and episode-length differences.
  4. Appendix B.2 states that Table 2 does not use the theoretical message widths. Could you clarify the message widths used for Table 2?
  5. On SMACv2 and GRF, some reply features are observer-relative and retain the sender’s values when inserted without coordinate transformation (Appendix B.1). What sender identity, provenance, or visibility indicators does the action policy receive to distinguish these values from locally observed features in the same slots? Does transforming the observer-relative geometric components to the receiver’s frame, or explicitly indicating their source, change the results? This question concerns components such as relative distance or position, rather than observer-independent attributes such as health. A relevant thing is, do you have a sensitivity result for the count-based auxiliary target already suggested in Appendix B.1, which avoids weighting missing information by the positive parts of signed feature coordinates?
Reviewer_KfUNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formalizes publicly shared group-entity indices as a hierarchically indexed MDP (HI-MDP) and proposes a learned two-stage request-reply protocol for cooperative MARL. Each receiver selects a group and then a teammate to query, and the sender selects a group and then one observed entity, returning its index and raw features to fill a missing entry in the receiver's observation. Selections are trained with straight-through Gumbel-Softmax via the actor's task gradient and a centralized auxiliary loss that favors information missing from the receiver, and the protocol reduces per-step communication capacity from O(N^2) to O(N), i.e., to 1/(N-1) of flat communication.

Experiments cover CC4, SMACv2, and GRF under IPPO and MAPPO, with additional SMACv2 results under QMIX. HiComm attains the highest mean reward among the communication baselines in nine of ten configurations and the lowest communication volume in all ten. A separate finite-model primal-dual analysis compares the protocol-specific optima of hierarchical and flat communication.

Critical strengths

Implementation: Using a shared observation index as the address of communication is a useful and interpretable design. A reply names one entity and supplies its observed features, so each exchange has a well-defined role in the receiver's action input, unlike an unconstrained latent message.

Experiment: The measured reward-communication trade-off is consistent across tasks and learning backbones. HiComm matches or exceeds the compared communication baselines in mean reward in most configurations while consistently using the least communication, and the QMIX results extend this pattern beyond the two PPO backbones.

Critical weaknesses

The evidence does not isolate a hierarchy-specific benefit. The 1/(N-1) capacity ratio follows directly from querying one sender for one entity rather than all teammates at equal message widths; it does not by itself establish the value of grouping. The key missing baseline is a flat one-sender, one-entity, raw-feature protocol with matched supervision and byte budget. The joint versus hierarchical control is limited to CC4, and the detach controls retain the auxiliary loss rather than ablating it, so the hierarchy is not separated from the raw-feature interface and privileged auxiliary supervision. Finally, team sizes are three to six agents, which limits the support for scalability claims.

The theoretical results provide little support for performance preservation. Theorem 6.1 leaves the optimal-return gap Delta unbounded, relies on common worst-case size bounds, and analyzes a finite-model primal-dual learner rather than the implemented neural PPO method; the communication constraints hold for every policy by construction. The theorem therefore does not establish that the restriction costs little reward.

Questions for authors
  1. Under the same message precision, one-sender/one-entity budget, raw-feature interface, and training budget, how does flat sender/entity selection compare with HiComm, and how do both change when the auxiliary loss is removed or replaced by a simpler target?
  2. Beyond a shared group-entity index, what structural condition makes Delta small, and which conclusion of Theorem 6.1 supports performance preservation beyond convergence to each protocol's own optimum and its built-in capacity limit?
  3. Can you report per-step message counts, actual query/reply/index widths, and uncertainty for the key ablation differences?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.