acceptodds
Under review as a conference paper at ICLR 2027

On the Role of Communication in Offline Multi-Agent Reinforcement Learning

Abstract

This paper presents CODA, a framework for learning inter-agent communication from offline multi-agent datasets collected without communication. CODA jointly learns action and communication policies from fixed offline data, without message supervision or additional environment interaction, and deploys the learned communication after training. We analyze the role of communication in offline multi-agent RL, where communication can provide additional information about dataset-supported actions but can also increase the difficulty of learning a message-conditioned actor. Our analysis formalizes this trade-off and provides a condition under which communication tightens an overestimation bound. Experiments across several multi-agent tasks show that CODA can substantially improve offline policy learning, with gains that depend on the underlying offline MARL algorithm and dataset. Further analysis shows that communication often reduces policy-data action deviation and, in several settings, empirical critic overestimation, consistent with the stabilization perspective suggested by the theory. CODA also remains effective under delayed communication in several tasks.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning, Communication, Offline Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YR3YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CODA, a framework for learning inter-agent communication in offline MARL from datasets without recorded communication messages. CODA augments existing offline MARL algorithms with differentiable communication modules, jointly optimizing action and communication policies through the underlying offline RL objectives. The paper also provides a theoretical analysis of when communication can reduce critic overestimation by improving information about dataset-supported actions. Experiments on multi-agent MuJoCo and Predator–Prey demonstrate performance improvements, particularly when CODA is integrated with B3C.

Critical strengths
  • Relevant problem setting: Learning communication from offline datasets without message supervision is practically relevant and worth investigating.

  • Promising empirical results: CODA improves B3C performance in most partially observable settings, with substantial gains in several tasks.

  • Useful empirical analysis: The paper investigates action deviation, critic overestimation, network capacity, observation sharing, and delayed communication, providing insights beyond return comparisons.

Critical weaknesses
  • Limited methodological novelty. The proposed framework primarily incorporates established differentiable communication mechanisms into existing offline MARL algorithms. Communication is optimized through the original offline actor objectives, without introducing an explicit mechanism or novel optimization objective to address communication-induced distribution shift. The methodological contribution beyond existing communication-learning techniques is therefore unclear.

  • Insufficient comparisons with existing communication methods. The evaluation mainly compares CODA against offline MARL algorithms without learned communication, such as B3C, OMIGA, ICQ, OMAR, and CFCQL. Although these comparisons demonstrate the potential benefits of communication, they do not establish the advantages of CODA over existing communication-learning approaches. Comparisons with relevant methods, such as MASIA, I2C (Learning Individually Inferred Communication for Multi-Agent Cooperation), and many other differentiable communication frameworks adapted to offline training, would help establish the contribution.

  • Inconsistent performance across offline RL algorithms. CODA substantially improves B3C in 19/21 partially observable settings, but improves OMIGA in only 9/21 settings, with a median relative change of −0.1%. This significant dependence on the underlying offline RL algorithm raises questions about the generality of the framework. The paper provides limited explanation for why the same communication mechanism performs so differently across the two algorithms.

Questions for authors
  • Methodological novelty: What is the key technical innovation in CODA compared with directly integrating existing differentiable communication methods into behavior-constrained offline MARL algorithms? Is there any specific component designed to mitigate distribution shift introduced by communication?

  • Communication baselines: Can the authors compare CODA against existing communication-learning methods, particularly those that can learn communication representations from offline datasets without recorded messages?

  • Backbone dependence: Why does CODA consistently improve B3C but provide limited benefits for OMIGA? Can the authors offer additional analysis explaining how the underlying offline RL objective affects communication learning?

  • Understanding the performance mechanism: Table 2 shows that CODA sometimes improves returns even when both policy–data action deviation and empirical critic overestimation increase (e.g., HC-k0-m2). What explains these improvements? Could communication provide benefits beyond the proposed support-stabilization mechanism, such as improved coordination or action representations?

2 comments

We appreciate the reviewer's valuable comments and questions.

Below, we provide our responses to the reviewer's concerns and questions.

[3-1] Regarding "Limited methodological novelty"

We would first like to clarify that our main contribution is to understand the benefits and limitations of communication in offline MARL. The theory identifies when the reduction in action-center uncertainty outweighs any increase in actor error, giving a tighter overestimation bound.

CODA jointly trains communication and action policies through the backbone's existing offline objective. It uses the backbone's behavior-related constraints without adding a separate communication regularizer. This design lets us study how the same communication mechanism interacts with different offline objectives. The strong gains with B3C and the mixed results with OMIGA are both important findings of this study.


[3-2] Regarding "Insufficient comparisons with existing communication methods"

Appendix I, Table 14, already compares RIAL and IC3Net as alternative communication architectures on twelve partially observable HalfCheetah settings. Our main experiments hold the offline backbone fixed and compare learning with and without communication.

MASIA also differs in its data-collection setting. Its behavior policies use communication, whereas our data are collected by non-communicating agents. We have not conducted matched offline comparisons with MASIA or I2C. Our empirical claims concern the effect of introducing communication into the tested offline methods, rather than superiority over these methods.


[3-3] Regarding "Inconsistent performance across offline RL algorithms"

To study this difference, we repeated HC-k1-e-m1 with five seeds per method and the existing hyperparameters at α = 1, after correcting episode-boundary handling in the mixture data. The following table shows mean ± SD of returns averaged from 1.20M to 1.45M updates. Each checkpoint is evaluated over 32 episodes with the same initial states across methods.

BackboneWithout communicationWith CODA
B3C2472 ± 1063742 ± 23
OMIGA3442 ± 1022678 ± 194

Communication improves B3C and reduces OMIGA's return in all five seed pairs. However, final-checkpoint MSE on medium-quality actions decreases for both, from 0.00402 to 0.00289 for B3C and from 0.02021 to 0.00209 for OMIGA.

OMIGA learns communication through the weighted log-likelihood of logged actions, using critic-derived weights. B3C directly optimizes critic values at its chosen actions together with a behavior-cloning loss. A plausible explanation is that communication improves OMIGA's ability to reproduce medium-quality behavior without improving its return. This shows why better behavior fitting alone is insufficient.

We also repeated two B3C comparisons over ten seeds with fixed hyperparameters. The table below reports mean ± SD of paired return differences at 1.95M updates, evaluated with the same initial states across methods.

SettingCODA − B3CSeeds with higher return
HC-k1-e-m2 (α = 4)+1471 ± 11369/10
Sw-k0-e-m2 (α = 1)+167 ± 6510/10

These results support substantial gains with B3C while confirming that communication does not benefit every offline objective. This dependence is a central finding of our study.


[3-4] Regarding "benefits beyond the proposed support-stabilization mechanism"

Communication can also improve action selection through additional information about other agents. Support stabilization is one possible benefit, not a necessary condition for every return improvement.

For HC-k0-m2, we reevaluated the two available paired checkpoint seeds using the same initial states. The following table shows CODA-minus-B3C differences.

SeedReturnAction deviationPositive critic–return gap
3+186.2+0.00222+12.16
4+118.7+0.00443+12.16

Returns improve in both cases despite increases in the two diagnostics. In a separate test with five CODA actors, replacing received messages with those from another episode reduces mean return from 1343 ± 37 to 748 ± 66, with decreases in all five cases.

This shows that the learned policies use the messages and that improved returns need not coincide with lower action deviation or critic gaps. The test does not identify a specific coordination mechanism. These diagnostics also differ from the quantities used in the theoretical bound.

Thank you again for your comments. Our response clarifies the contribution and existing communication comparisons, and provides additional experiments on backbone dependence and message use. In particular, communication improves behavior fitting for both B3C and OMIGA but has opposite effects on return, highlighting why the offline objective matters.

As the discussion deadline approaches, we would appreciate it if you could review our response and let us know whether any concerns remain.

Reviewer_UF7UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose CODA, an framework for offline MARL to learn communication policies together with policies from datasets without messages. The authors tried to provide a theoretical analysis to identify the conditions under which communication can reduce overestimation of the joint Q-functions. They also conduct experiments with B3C and OMIGA across multiple environments and datasets.

Critical strengths
  1. The paper is well motivated and easy to follow.
  2. The experimental results, although not perfectly consistent with their theories, are informative to the community.
Critical weaknesses
  1. The theoretical analysis (Section 3.3 and Appendix A) is problematic. My primal concern is that Assumption 1 seems way too strong that it almost contains all the theoretical results. The necessity and validity requires further explanation. Additionally, the presentation of the related sections is inconsistent. For example, ϵπ\epsilon_\pi is not defined, while ϵQ\epsilon_Q can be arbitrarily large.
  2. The Inconsistency in experimental results are not well explained. The communication gains for OMIGA is very limited, but the authors do not provide any explanations or hypothesis to help understand this phenomenon. Besides, the authors report performance improvements on fully observable settings, but this comparison do not exclude the gains by a larger actor.
Questions for authors

See Weaknesses.

2 comments

We appreciate the reviewer's valuable comments and questions.

Below, we provide our responses to the reviewer's concerns and questions.

[2-1] Regarding "Assumption 1 seems way too strong"

Assumption 1 provides the link between deviation from the dataset action center and positive critic error. It is motivated by the offline-RL problem that actions far from the behavior data can have unreliable value estimates.

For example, Eq. (9) follows if positive critic error at the action center is bounded by ϵQ\epsilon_Q and the error is Lipschitz continuous in actions with constant κ\kappa. These conditions must hold for both actors under the common reference distribution. They provide a sufficient condition for the assumption, rather than a guarantee from behavior regularization alone.

Assumption 1 does not assume that communication reduces critic error. Proposition 1 establishes the reduction in action-center uncertainty, and Corollary 1 compares it with the change in actor error. This comparison is what identifies when communication tightens the bound.


[2-2] Regarding "ϵπ\epsilon_\pi is not defined" and "ϵQ\epsilon_Q can be arbitrarily large"

ϵπ(Z)\epsilon_\pi(Z) is specified in Appendix A, Eq. (10), as a bound on the learned actor's excess squared error over the best predictor of the dataset action center. Using Eq. (17), this means

\leq\epsilon_\pi(Z).$$ All expectations use the same reference distribution. The term includes approximation error, optimization error, and other deviations of the learned actor from this predictor. It is not a tuning parameter. $\epsilon_Q$ is the same additive term in both bounds, so it does not affect the condition for communication to tighten the bound. A large value does make the absolute bounds loose. The result therefore gives a conditional comparison of upper bounds, not a guarantee that actual critic errors or returns improve. --- ## [2-3] Regarding "communication gains for OMIGA" We conducted additional HC-k1-e-m1 experiments with five seeds per method and the existing hyperparameters at α = 1, after correcting episode-boundary handling in the mixture data. The following table shows mean ± SD of returns averaged from 1.20M to 1.45M updates. Each checkpoint is evaluated over 32 episodes with the same initial states across methods. | Backbone | Without communication | With CODA | |---|---:|---:| | B3C | 2472 ± 106 | 3742 ± 23 | | OMIGA | 3442 ± 102 | 2678 ± 194 | Communication improves B3C and reduces OMIGA's return in all five seed pairs. Yet both fit medium-quality actions more closely. Final-checkpoint MSE decreases from 0.00402 to 0.00289 for B3C and from 0.02021 to 0.00209 for OMIGA. OMIGA learns communication through the weighted log-likelihood of logged actions, using critic-derived weights. B3C directly optimizes critic values at its chosen actions together with a behavior-cloning loss. This difference provides a plausible explanation for the results. Better fitting of medium-quality behavior need not improve return. The contrast is an important example of why the benefit of offline communication depends on the learning objective. --- ## [2-4] Regarding "gains by a larger actor" The FO experiments do not separate communication gains from increased actor capacity. Our capacity controls are in the PO experiments in Table 3 and Appendix C.4. The parameter-matched baseline has the same number of actor parameters as CODA but blocks information from other agents. For example, on Sw-k0-e-m2, this baseline obtains 177.5 ± 22.1, compared with 286.3 ± 105.0 for CODA over ten seeds. Thus, our evidence beyond increased capacity comes from the controlled PO comparisons. The FO results are supplementary.

Thank you again for your comments. Our response explains sufficient conditions for Assumption 1 and clarifies ϵπ\epsilon_\pi, ϵQ\epsilon_Q, and the conditional scope of the bound. We also provide additional B3C/OMIGA experiments and identify the parameter-matched controls in the partially observable settings.

As the discussion deadline approaches, we would appreciate it if you could review our response and let us know whether any concerns remain.

Reviewer_C9DAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether agents can learn useful communication from offline datasets collected by non-communicating policies. CODA adds learned message encoders and attention-based aggregation to existing offline MARL actors, training communication and action policies jointly without message supervision. The paper provides a conditional analysis of communication's effect on critic overestimation and evaluates CODA with B3C and OMIGA on multi-agent MuJoCo and predator–prey tasks. Results show broad gains with B3C but mixed outcomes with OMIGA.

Critical strengths
  1. Learning deployment-time communication from existing non-communicating datasets is a useful and practically motivated setting.

  2. CODA achieves substantial gains with B3C on several partially observable tasks, with additional fixed-hyperparameter, ten-seed experiments supporting some of these improvements.

  3. Parameter-matched and raw-observation-sharing baselines, together with diagnostic analyses, help assess the gains beyond simply increasing network capacity.

Critical weaknesses
  1. The methodological novelty is limited, as CODA combines established differentiable communication with existing offline MARL objectives. Its distinction from prior offline communication work, particularly MASIA, is insufficiently discussed, leaving the contribution beyond the non-communicating data-collection setting unclear.

  2. The theoretical bound relies on assumptions linking critic error to the dataset action center and controlling actor error, without establishing that CODA satisfies them with informative bounds. It therefore offers a conditional explanation rather than a guarantee of reduced overestimation or improved returns.

  3. The benefits are strongly backbone-dependent: CODA improves B3C broadly but provides little overall gain with OMIGA. The paper does not sufficiently explain this disparity or establish when communication is likely to help.

Questions for authors
  1. Could sharing raw observations with a matched receiver-side attention network achieve similar gains? This would help isolate the value of sender-side message encoding, especially where the 8-dimensional messages exceed the raw observation size.

  2. With a single memoryless non-communicating behavior policy, deterministic local observations, and the global state as latent context, the action center is already determined by the local observation. What produces a nonzero information gain in the evaluated datasets, particularly the expert datasets?

  3. How do the different actor objectives in B3C and OMIGA affect communication learning? In particular, why can improved fitting of logged actions fail to improve returns with OMIGA?

2 comments

We appreciate the reviewer's valuable comments and questions.

Below, we provide our responses to the reviewer's concerns and questions.

[1-1] Regarding "The methodological novelty is limited"

We would first like to clarify that our main contribution is to understand the benefits and limitations of communication in offline MARL. Our theory separates the additional information provided by messages from the error of learning an actor that uses those messages. Our experiments show how the benefits depend on the dataset and the offline learning objective.

MASIA uses data collected by communicating behavior policies. In our setting, the data have already been collected by non-communicating agents. We study whether useful communication can be learned from these fixed data and when it helps or fails. CODA provides a framework for studying these questions.


[1-2] Regarding "The theoretical bound relies on assumptions" and "a nonzero information gain"

The purpose of the analysis is to identify when communication can tighten an overestimation bound. Corollary 1 gives this condition explicitly. The reduction in action-center uncertainty must outweigh any increase in actor error. Thus, the analysis describes a trade-off and does not assume that communication always helps.

For a single memoryless behavior policy with deterministic local observations, the reviewer is correct that the information gain is zero when the latent context is the global state. Our data-collection checks confirm this case for HC-k0-e and HC-k0-m2. Proposition 1 allows zero gain, so we do not attribute every return improvement to this term.

In mixture datasets, messages can help identify which behavior policy generated an action. In our held-out action-prediction experiments, adding CODA messages reduces MSE from 0.00164 to 0.00041 on HC-k1-e-m1 and from 0.00215 to 0.00049 on HC-k1-e-m2. These results show improved prediction of logged actions, rather than directly measuring the information-gain term in the bound.


[1-3] Regarding "why can improved fitting of logged actions fail to improve returns with OMIGA?"

To examine this question, we repeated B3C and OMIGA with and without CODA on HC-k1-e-m1, using five seeds per method and their existing hyperparameters at α = 1. We also corrected the handling of episode boundaries in the mixture data before these runs.

The following table shows mean ± SD across seeds. For each seed, returns are averaged over checkpoints from 1.20M to 1.45M updates, with 32 evaluation episodes using the same initial states across methods.

BackboneWithout communicationWith CODA
B3C2472 ± 1063742 ± 23
OMIGA3442 ± 1022678 ± 194

Communication improves B3C and reduces OMIGA's return in all five seed pairs. However, both methods fit medium-quality actions more closely. At the final checkpoint, medium-action MSE decreases from 0.00402 to 0.00289 for B3C and from 0.02021 to 0.00209 for OMIGA.

OMIGA learns communication through the weighted log-likelihood of logged actions, with weights determined by the critic. B3C directly optimizes the critic's value of its chosen actions together with a behavior-cloning loss. A plausible explanation is that communication helps OMIGA reproduce medium-quality behavior more accurately, while B3C also receives a direct value gradient through its actions. The results show that better behavior fitting alone does not ensure higher returns.


[1-4] Regarding "a matched receiver-side attention network"

We conducted this comparison by keeping CODA's receiver attention and action head unchanged and replacing the sender encoder with raw observations, zero-padded to eight dimensions. The following table shows mean ± SD over ten seeds at 1.95M updates.

SettingB3CRawObs-AttnB3C+CODA
HC-k1-e-m12446.8 ± 95.53669.2 ± 62.43727.9 ± 79.5
HC-k1-e-m21154.1 ± 370.01878.1 ± 1203.32433.0 ± 1290.0
Sw-k0-e-m2174.8 ± 23.7203.8 ± 51.4286.3 ± 105.0
HC-k0-e1394.4 ± 10.01393.4 ± 15.21392.0 ± 15.0

The two HalfCheetah mixture rows were run before the episode-boundary correction described in [1-3].

Raw observation sharing recovers most of CODA's gain on HC-k1-e-m1. This is consistent with the observation-sharing results in Table 3. Our central finding is that access to other agents' information can substantially affect offline learning, with benefits that depend on the setting and objective. These comparisons do not support a general advantage of learned encoding over raw observations, and the eight-dimensional setting does not test message compression.

Thank you again for your comments. Our response provides the requested matched-attention comparison, additional B3C/OMIGA experiments, and clarification of the information-gain condition and the distinction from MASIA. These results support our focus on the benefits and limitations of communication in offline MARL.

As the discussion deadline approaches, we would appreciate it if you could review our response and let us know whether any concerns remain.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.