- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CODA, a framework for learning inter-agent communication in offline MARL from datasets without recorded communication messages. CODA augments existing offline MARL algorithms with differentiable communication modules, jointly optimizing action and communication policies through the underlying offline RL objectives. The paper also provides a theoretical analysis of when communication can reduce critic overestimation by improving information about dataset-supported actions. Experiments on multi-agent MuJoCo and Predator–Prey demonstrate performance improvements, particularly when CODA is integrated with B3C.
Critical strengths
-
Relevant problem setting: Learning communication from offline datasets without message supervision is practically relevant and worth investigating.
-
Promising empirical results: CODA improves B3C performance in most partially observable settings, with substantial gains in several tasks.
-
Useful empirical analysis: The paper investigates action deviation, critic overestimation, network capacity, observation sharing, and delayed communication, providing insights beyond return comparisons.
Critical weaknesses
-
Limited methodological novelty. The proposed framework primarily incorporates established differentiable communication mechanisms into existing offline MARL algorithms. Communication is optimized through the original offline actor objectives, without introducing an explicit mechanism or novel optimization objective to address communication-induced distribution shift. The methodological contribution beyond existing communication-learning techniques is therefore unclear.
-
Insufficient comparisons with existing communication methods. The evaluation mainly compares CODA against offline MARL algorithms without learned communication, such as B3C, OMIGA, ICQ, OMAR, and CFCQL. Although these comparisons demonstrate the potential benefits of communication, they do not establish the advantages of CODA over existing communication-learning approaches. Comparisons with relevant methods, such as MASIA, I2C (Learning Individually Inferred Communication for Multi-Agent Cooperation), and many other differentiable communication frameworks adapted to offline training, would help establish the contribution.
-
Inconsistent performance across offline RL algorithms. CODA substantially improves B3C in 19/21 partially observable settings, but improves OMIGA in only 9/21 settings, with a median relative change of −0.1%. This significant dependence on the underlying offline RL algorithm raises questions about the generality of the framework. The paper provides limited explanation for why the same communication mechanism performs so differently across the two algorithms.
Questions for authors
-
Methodological novelty: What is the key technical innovation in CODA compared with directly integrating existing differentiable communication methods into behavior-constrained offline MARL algorithms? Is there any specific component designed to mitigate distribution shift introduced by communication?
-
Communication baselines: Can the authors compare CODA against existing communication-learning methods, particularly those that can learn communication representations from offline datasets without recorded messages?
-
Backbone dependence: Why does CODA consistently improve B3C but provide limited benefits for OMIGA? Can the authors offer additional analysis explaining how the underlying offline RL objective affects communication learning?
-
Understanding the performance mechanism: Table 2 shows that CODA sometimes improves returns even when both policy–data action deviation and empirical critic overestimation increase (e.g., HC-k0-m2). What explains these improvements? Could communication provide benefits beyond the proposed support-stabilization mechanism, such as improved coordination or action representations?
2 comments
Thank you again for your comments. Our response clarifies the contribution and existing communication comparisons, and provides additional experiments on backbone dependence and message use. In particular, communication improves behavior fitting for both B3C and OMIGA but has opposite effects on return, highlighting why the offline objective matters.
As the discussion deadline approaches, we would appreciate it if you could review our response and let us know whether any concerns remain.
We appreciate the reviewer's valuable comments and questions.
Below, we provide our responses to the reviewer's concerns and questions.
[3-1] Regarding "Limited methodological novelty"
We would first like to clarify that our main contribution is to understand the benefits and limitations of communication in offline MARL. The theory identifies when the reduction in action-center uncertainty outweighs any increase in actor error, giving a tighter overestimation bound.
CODA jointly trains communication and action policies through the backbone's existing offline objective. It uses the backbone's behavior-related constraints without adding a separate communication regularizer. This design lets us study how the same communication mechanism interacts with different offline objectives. The strong gains with B3C and the mixed results with OMIGA are both important findings of this study.
[3-2] Regarding "Insufficient comparisons with existing communication methods"
Appendix I, Table 14, already compares RIAL and IC3Net as alternative communication architectures on twelve partially observable HalfCheetah settings. Our main experiments hold the offline backbone fixed and compare learning with and without communication.
MASIA also differs in its data-collection setting. Its behavior policies use communication, whereas our data are collected by non-communicating agents. We have not conducted matched offline comparisons with MASIA or I2C. Our empirical claims concern the effect of introducing communication into the tested offline methods, rather than superiority over these methods.
[3-3] Regarding "Inconsistent performance across offline RL algorithms"
To study this difference, we repeated HC-k1-e-m1 with five seeds per method and the existing hyperparameters at α = 1, after correcting episode-boundary handling in the mixture data. The following table shows mean ± SD of returns averaged from 1.20M to 1.45M updates. Each checkpoint is evaluated over 32 episodes with the same initial states across methods.
Communication improves B3C and reduces OMIGA's return in all five seed pairs. However, final-checkpoint MSE on medium-quality actions decreases for both, from 0.00402 to 0.00289 for B3C and from 0.02021 to 0.00209 for OMIGA.
OMIGA learns communication through the weighted log-likelihood of logged actions, using critic-derived weights. B3C directly optimizes critic values at its chosen actions together with a behavior-cloning loss. A plausible explanation is that communication improves OMIGA's ability to reproduce medium-quality behavior without improving its return. This shows why better behavior fitting alone is insufficient.
We also repeated two B3C comparisons over ten seeds with fixed hyperparameters. The table below reports mean ± SD of paired return differences at 1.95M updates, evaluated with the same initial states across methods.
These results support substantial gains with B3C while confirming that communication does not benefit every offline objective. This dependence is a central finding of our study.
[3-4] Regarding "benefits beyond the proposed support-stabilization mechanism"
Communication can also improve action selection through additional information about other agents. Support stabilization is one possible benefit, not a necessary condition for every return improvement.
For HC-k0-m2, we reevaluated the two available paired checkpoint seeds using the same initial states. The following table shows CODA-minus-B3C differences.
Returns improve in both cases despite increases in the two diagnostics. In a separate test with five CODA actors, replacing received messages with those from another episode reduces mean return from 1343 ± 37 to 748 ± 66, with decreases in all five cases.
This shows that the learned policies use the messages and that improved returns need not coincide with lower action deviation or critic gaps. The test does not identify a specific coordination mechanism. These diagnostics also differ from the quantities used in the theoretical bound.