acceptodds
Under review as a conference paper at ICLR 2027

Communication Gain and Delay Cost under Cross-Timestep Delays in Cooperative Multi-Agent Reinforcement Learning

Abstract

Communication is crucial for cooperative multi-agent reinforcement learning (MARL) under partial observability, but bounded cross-timestep communication delays make received messages temporally misaligned and potentially stale. We formalize this setting as a delayed-communication partially observable Markov game (DeComm-POMG) and introduce Communication Gain and Delay Cost (CGDC), a gain–cost metric that evaluates delayed-message utility through no-message and timely-reference counterfactuals. We further derive a delay-induced return-loss bound that relates the return gap between policies using timely references and those using delayed messages to the accumulated information gap. Guided by CGDC, we propose CDCMA, an actor–critic framework for selecting communication partners, constructing future-aware outgoing messages, and aggregating received delayed messages. Experiments on MPE tasks without teammate vision and on SMAC maps show that CDCMA achieves higher mean performance than the communication baselines across the evaluated bounded stochastic delay regimes; on two representative tasks, it also yields smaller decision-distribution gaps. CDCMA remains competitive in the evaluated zero-delay settings and exhibits favorable zero-shot cross-difficulty generalization on Cooperative Navigation.

Primary area:
reinforcement learning
Keywords:
cooperative multi-agent reinforcement learning, delayed communication, learned communication, selective communication, partial observability, multi-agent coordination

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SZ9nOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies cooperative MARL under cross-timestep communication delays. The authors propose CGDC, a metric that evaluates delayed-message utility by balancing communication gain against delay cost, and derive a theoretical bound on delay-induced return loss.

Based on CGDC, they develop CDCMA, which combines communication partner selection, future-aware message generation, and attention-based message aggregation.

Experiments on MPE and SMAC under stochastic communication delays show that CDCMA outperforms the selected baselines. Ablation studies, information-gap analyses, and zero-shot cross-difficulty experiments provide additional validation.

Critical strengths
  1. The proposed CGDC metric evaluates delayed messages against both no-message and timely-message references. This provides a more fine-grained criterion for communication selection by explicitly distinguishing communication utility from temporal misalignment.

  2. The Observation Trajectory Generator (OTG) incorporates predicted future observations into outgoing messages to mitigate communication delays. This is a reasonable design choice, and the ablation results suggest that it contributes substantially to the overall performance improvements.

Critical weaknesses
  1. The theoretical justification is not fully convincing. The return-loss bound relies on actor-policy divergence, while the implemented CGDC uses a critic-based surrogate. The approximation gap between them is not adequately validated.

  2. The communication gain metric ignores potential redundancy among messages by evaluating each sender independently against a no-message baseline.

  3. DCOS estimates communication utility at request time, but CAMA reuses these scores after delayed delivery. Their reliability is questionable when the receiver's state changes significantly.

  4. The baseline comparison is not entirely fair, as DACOM is evaluated without TimeNet and CoDe is reimplemented.

  5. The paper lacks sufficient analysis of communication efficiency and scalability. CDCMA introduces additional prediction and scoring modules, but their computational overhead and practical benefits are not convincingly evaluated.

Questions for authors
  1. Can the authors quantify the actor–probe approximation errors in Corollary 3.4 to justify the practical relevance of the theoretical bound? 

  2. How does CGDC handle redundant or correlated messages from different agents? 

  3. How reliable are request-time CGDC scores when messages arrive several timesteps later? 

  4. Why is DACOM evaluated without TimeNet, and how was the CoDe reimplementation validated? 

  5. Can the authors report communication frequency, inference latency, and computational overhead relative to the baselines?

5 comments

Thank you for suggesting that we measure both discrepancies in Corollary 3.4. We compared the actor and the critic-induced probe before and after replacing each delayed message with its timely reference.

W1 / Q1. The actor–probe gap in Corollary 3.4

Across the four task–delay settings in Table 8 (Appendix E.10), the mean actor–probe error was 0.0214–0.0364 in total variation with a timely reference and 0.0212–0.0319 with a delayed message. Together, the two errors accounted for 12.9–16.8% of the mean local upper bound on the sampled decisions.

We took decisions at which receiver ii used a message from sender jj. For each decision we ran the actor once with the delivered message and once with jj’s current representation ej,te_{j,t} in its place. We kept the receiver’s observation and hidden state, the other senders’ messages, and the saved request scores fixed, and recomputed attention. For the same two inputs we computed the corresponding probe by taking a softmax over the gain critic’s action values (η=1\eta=1, as in training). This gives the actor and the probe under each condition. For x∈{tar,del}x\in\{\mathrm{tar},\mathrm{del}\}, their discrepancy is δx=12∑a∣πx(a)−π~x(a)∣\delta^x=\tfrac12\sum_a|\pi^x(a)-\widetilde\pi^x(a)|. Figure 4(a,b) in the original submission replaced the entire set of messages. Replacing one sender at a time lets us measure the two terms for the particular message considered in the corollary.

The local inequality is G≤E+CG\le E+C. Here G=∥πtar−πdel∥TVG=\Vert\pi^{\mathrm{tar}}-\pi^{\mathrm{del}}\Vert_{\mathrm{TV}} is the observed change in the actor, E=δtar+δdelE=\delta^{\mathrm{tar}}+\delta^{\mathrm{del}} is the contribution from approximating the actor, and C=DKL(π~tar∥π~del)/2C=\sqrt{D_{\mathrm{KL}}(\tilde{\pi}^{\mathrm{tar}}\Vert\tilde{\pi}^{\mathrm{del}})/2} is computed entirely from the critic. The inequality follows from the TV triangle inequality and Pinsker’s inequality. We evaluated these quantities per message, averaged within each trained seed, and report the mean ± sample SD of the four seed averages.

TaskDelayTimely errorDelayed errorEECCGG
PPeasy0.0249 ± 0.00520.0254 ± 0.00590.0503 ± 0.01110.2499 ± 0.05110.1989 ± 0.0449
PPsuper_hard0.0322 ± 0.00630.0319 ± 0.00630.0641 ± 0.01250.3690 ± 0.10790.2723 ± 0.0870
1o_10b_vs_1reasy0.0214 ± 0.00350.0212 ± 0.00310.0426 ± 0.00670.2111 ± 0.04820.1784 ± 0.0441
1o_10b_vs_1rsuper_hard0.0364 ± 0.00590.0318 ± 0.00450.0682 ± 0.01020.4599 ± 0.08020.3585 ± 0.0643

For example, under super-hard delay on SMAC, substituting one timely message changed the actor distribution by 0.3585 TV on average. Of the mean upper bound of 0.5281, the critic-derived cost contributed 0.4599 and the two errors together contributed 0.0682. The share Eˉ/(Eˉ+Cˉ)\bar E/(\bar E+\bar C) was 16.8%, 14.8%, 16.8% and 12.9% in table order. In each of the 16 trained models, Eˉ<Cˉ/4\bar E<\bar C/4. This establishes the scale of the approximation on decisions where the receiver used the message. The full return-loss bound in Eq. (8) also takes a discounted expectation along the timely-reference policy’s trajectories.

W2 / Q2. Redundancy between senders

Two teammates reporting the same useful information can both receive positive scores and both be queried. The gain in Eq. (1) compares access to sender jj alone with receiving no messages. In Eq. (9), we estimate that gain by giving the same gain critic jj’s message or a null message with the other inputs held fixed. DCOS has access to the receiver’s available messages when it predicts a score, and CAMA weights the messages jointly once they arrive. The pairwise target itself does not discount a sender solely because another candidate could report the same information. Appendix D.3 describes how we construct the target for each sender.

We used a leave-one-out test to examine the consequence of this choice. For each arrival, we retained all the other messages and compared the full critic’s value with and without jj’s message. We held the observations and joint action fixed and recomputed the aggregate after masking jj. Call the difference ΔQ=Qifull(all)−Qifull(without j)\Delta Q=Q_i^{\rm full}(\mathrm{all})-Q_i^{\rm full}(\mathrm{without}\ j). If a high pairwise gain mostly picked out information already supplied by teammates, retaining that sender would add little to the full critic’s estimate. On PP, mean ΔQ\Delta Q was 0.088 for the top 30% of messages ranked by pairwise gain and −0.019 for the bottom 30%. On 1o_10b_vs_1r the corresponding means were 0.121 and −0.012. We formed the groups within each seed and delay setting and then weighted the four seeds and four settings equally (Appendix E.7, Table 6). The ranking thus identified messages with larger average contributions in the presence of other arrivals. It can still assign high scores to an individual pair of duplicates.

W3 / Q3. Using a request-time score after delivery

The saved score remains useful as an attention prior in our experiments. We have not directly measured how accurately it predicts arrival-time utility after a large change in the receiver’s state. To test its use in the policy, we compared CDCMA with PlainAttn, which keeps sender selection, future prediction and content attention, removing just the saved score from the aggregation weights. Figure 11(a,b) (Appendix E.9) gives the learning curves on 1o_10b_vs_1r under easy and super-hard delays, extending the CN experiment in Fig. 3(a). With the same checkpoint-selection rule, the four-seed mean win rates ± sample SD were as follows.

DelayPlainAttnCDCMAGain from the prior
Easy79.69 ± 7.86%92.19 ± 4.03%12.50 percentage points
Super-hard58.59 ± 3.93%73.44 ± 5.41%14.84 percentage points

The two methods differ at the moment of consumption. Suppose the receiver requested a message at kk and uses it at t=k+dt=k+d. CDCMA saves its positive request score cˉij,t\bar c_{ij,t} at kk. At tt the receiver computes a fresh query qi,tq_{i,t} from its current state representation and a key kij,tk_{ij,t} from the delivered content. Equation (14) assigns each arrival a weight proportional to cˉij,texp⁡(βqi,t⊤kij,t)\bar c_{ij,t}\exp(\beta q_{i,t}^{\top}k_{ij,t}), normalized over the available senders. With several arrivals, a message that scored well at kk can receive less weight than one that better matches the current query. With just one arrival its normalized weight is one. The 12.50- and 14.84-point gains quantify what retaining the prior adds to this content comparison. The 79.1% and 82.1% sign agreements in Table 6 compare the current DCOS score with ΔQ\Delta Q on PP and SMAC, respectively. These statistics use scores recomputed at arrival.

W4 / Q4. DACOM’s TimeNet and validation of CoDe

DACOM. TimeNet's within-step waiting mechanism does not apply to our cross-step delivery protocol. The original TimeNet learns a waiting threshold within a decision step: a longer wait allows more messages to arrive before the agent acts in that step. In our channel, a message sent at step kk with delay dd becomes available only when the environment reaches step k+dk+d. Extending the wait within step kk therefore leaves the messages available for that decision unchanged, so the original TimeNet waiting variable has no effect here.

For waiting to change the available information in our setting, the agent would have to defer its decision to a later environment step. This requires defining which actions it executes in the intervening steps and how to train the waiting policy from the resulting returns. The result would be a new method for waiting across steps, beyond reproducing the original TimeNet. We therefore reimplemented the remaining DACOM modules from the paper and used the messages available at each decision under the common cross-step delivery protocol. Section 5.1 identifies all DACOM results as the variant without TimeNet; revised Appendix E.1 states that we reimplemented this variant from the paper and explains the adaptation rationale above.

CoDe. We checked our paper-based implementation against the released source and found differences that change the messages the receiver uses. For example, the official aggregator sums values with weights wj=αjγTdjw_j=\alpha_j\gamma_T^{d_j}, where djd_j is message age. Our implementation additionally normalized these weights by ∑ℓwℓ\sum_\ell w_\ell. If all messages have aged by the same number of steps, this extra normalization cancels the common discount. Our future-action predictor also used replayed actions as targets, whereas the official code uses the online Q-network’s greedy actions. At test time our intent encoder sampled from its distribution and the official encoder used its mean. Tables 1, 2 and 5 and the original baseline curves refer to our implementation.

We connected the released code to our cross-step delivery interface and MPE environments, checked its deterministic forward output against upstream at zero delay, and tested delayed delivery, replay and gradient propagation. We also ran a short MPE training run and checked checkpoint save/load. Both versions are included with the paper as CoDe-original-reimplementation.zip and CoDe-official-adapted.zip. Official CoDe implementation, revision 255383b

We trained the adapted official CoDe implementation on four tasks under four delay settings with four seeds per setting (64 runs). The table pairs our reimplementation from Table 1 with the adapted official code from Table 9 in Appendix E.11. Each entry is the mean ± sample SD across four seeds. For the adapted official code, Table 9 uses the final evaluation for each seed, with 100 evaluation episodes on CN and PP. Figure 12(a–d) shows its learning curves across 400 aligned evaluation positions.

TaskCoDe implementation and metricEasyMediumHardSuper-hard
1o_10b_vs_1rOur reimplementation, win rate (%)32.81 ± 1.8035.16 ± 4.6924.22 ± 22.8825.00 ± 15.31
1o_10b_vs_1rAdapted official code, win rate (%)41.41 ± 9.6738.28 ± 13.1035.16 ± 5.9225.78 ± 5.34
1o_2r_vs_4rOur reimplementation, win rate (%)58.59 ± 14.2940.63 ± 7.6528.91 ± 9.3324.22 ± 6.93
1o_2r_vs_4rAdapted official code, win rate (%)39.06 ± 26.3343.75 ± 9.8835.94 ± 15.4230.47 ± 8.98
CNOur reimplementation, mean episode reward−3.11 ± 0.14−3.09 ± 0.05−4.36 ± 2.55−4.42 ± 2.75
CNAdapted official code, mean episode reward−3.76 ± 0.81−3.20 ± 0.26−3.07 ± 0.33−2.95 ± 0.24
PPOur reimplementation, mean episode reward−1.82 ± 0.03−1.83 ± 0.03−1.86 ± 0.08−1.85 ± 0.07
PPAdapted official code, mean episode reward−2.02 ± 0.36−2.01 ± 0.09−2.13 ± 0.20−2.32 ± 0.41

On CN, the adapted official code had lower mean episode rewards than our reimplementation under easy and medium delay (by 0.65 and 0.11) and higher rewards under hard and super-hard delay (by 1.29 and 1.47). On PP, our reimplementation had higher mean episode rewards in all four settings, by 0.18–0.47.

Reviewer_Zk9jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CDCMA, a framework for cooperative MARL under partial observability and cross-timestep communication delays. The authors propose a gain-cost metric to quantify delayed-message utility using no-message and timely-reference counterfactuals. The metric is used to guide agents' attention over delayed messages delivered by other agents, and to select proper communication partners. Outgoing messages are also constructed with a trajectory generator to make them future-aware.On CN, PP, and SMAC, CDCMA reports higher mean performance than seven baselines including CoDe and DACOM.

Critical strengths

1.The sensitivity analysis in section 5.3 shows that the framework performs worse whenλ= 0, indicating that the delay-cost penalty based on the timely-reference counterfactual is beneficial. 2.Table 1 shows that on CN and PP,CDCMA's performance degrades less than most baselines as the delay difficulty increases, and on the four tasks it remains the best among all methods even under the super_hard setting.The strong performance is impressive.

Critical weaknesses

1.The ablation analyses in section 5.3 are narrow: each is conducted on a single task and only under the super_hard setting, so the generality of the conclusions is unclear. 2.The contributions of DCOS and CAMA, which directly use CGDC, are not sufficiently explained. In Fig. 3a, removing OTG causes the largest performance drop, so it is unclear how much of the overall improvement comes from CGDC itself rather than from the future-aware messages.

Questions for authors

Could the authors extend the module ablations in Fig. 3a to at least one SMAC map and another difficulty setting, and report how the frequency of communication changes when DCOS is removed? These results would clarify how much of the improvement comes from the CGDC-based components and whether DCOS can reduce unnecessary communication.

2 comments

Thank you for your careful reading and for suggesting that we extend the CN ablations to SMAC. To examine the respective contributions of future prediction, sender selection and message aggregation, we ran the component ablations under two delay settings and counted communication requests with and without DCOS.

W1. Extending the component ablations to SMAC and a second delay setting

We evaluated CDCMA and its four variants on the SMAC map 1o_10b_vs_1r. We kept the super_hard delay setting from the original CN ablation and added the easy setting. The table gives mean win rate and sample standard deviation across four seeds, using the same checkpoint-selection and reporting rules for all five methods. Figures 11(a) and 11(b) in Appendix E.9 show the corresponding learning curves.

MethodEasy win rate (%)Super-hard win rate (%)
CDCMA92.19 ± 4.0373.44 ± 5.41
−OTG66.41 ± 15.3942.97 ± 9.67
−DCOS71.88 ± 14.6649.22 ± 8.61
−CAMA78.13 ± 9.8856.25 ± 4.42
PlainAttn79.69 ± 7.8658.59 ± 3.93

In −OTG, the sender still communicates its history, and the message dimension, DCOS and CAMA are unchanged. Removing the future prediction from the message reduced mean win rate by 25.78 points on easy and 30.47 points on super-hard. The benefit of prediction that you observed on CN at the highest delay difficulty also appears on this SMAC map at the lower difficulty.

W2. What does CGDC contribute to sender selection and aggregation when OTG is retained?

Both comparisons retain OTG. Comparing CDCMA with −DCOS measures the effect of selecting senders by CGDC. Comparing CDCMA with PlainAttn measures the effect of using the saved request score during aggregation.

Sender selection. −DCOS retains OTG and CAMA and requests messages from all ten teammates at every step. Restoring CGDC-based selection raised mean win rate by 20.31 points on easy and 24.22 points on super-hard. Table 6 also tests the score at the level of an individual message. On this map, the sign of the current DCOS score agreed 82.1% of the time with the full critic's estimate of the value of keeping that message, holding other arrivals fixed.

Message aggregation. Both PlainAttn and CDCMA generate messages with OTG, select teammates with DCOS and use the receiver's current representation to compute content attention over incoming messages. CDCMA also uses the CGDC score saved when the receiver requested each message. In Eq. (14), that score multiplies the exponentiated content score before normalization over arrivals. Including the saved score raised mean win rate by 12.50 percentage points on easy and 14.84 points on super-hard relative to PlainAttn. On super-hard, for example, mean win rate rose from 58.59% for PlainAttn to 73.44% for CDCMA. Removing the entire CAMA module reduced mean win rate by 14.06 points on easy and 17.19 points on super-hard. That broader intervention also removes content weighting.

DCOS determines which messages can arrive, and CAMA determines how much weight each arrival receives. Because selection changes the set of messages available to aggregation, the two ablations estimate the effect of changing each mechanism with OTG in place. Their win-rate differences cannot be added to assign a percentage of the overall improvement to CGDC. Appendix E.9 reports both comparisons alongside the SMAC learning curves.

We also tested the delay penalty on the map used in Fig. 3(c), 1o_2r_vs_4r, this time holding the absolute coefficient fixed. Under super-hard delay, mean win rate was 60.94% with a fixed λ=0.1\lambda=0.1 and 46.88% with gain alone (λ=0\lambda=0) (Appendix E.9, Fig. 11(c)).

Q1. How does removing DCOS affect communication frequency in the extended ablations?

The SMAC ablations above cover the two delay settings you requested. On the same map, removing DCOS made each agent query all ten teammates at every step, an average of 5.09 additional requests per agent per step on easy and 5.60 on super-hard. We counted a directed request when an agent asked one teammate for a message. For each seed, we divided the total count over 1,000 environment steps by 11×1,00011\times1{,}000 to obtain requests per agent per step. Appendix E.9 reports the four-seed results under “Communication requests.”

DelayCDCMA requests/agent/step−DCOS requests/agent/stepReduction with DCOS
Easy4.9085 ± 0.59371050.92%
Super-hard4.4022 ± 0.50141055.98%

On super-hard, for example, asking 4.40 teammates on average yielded a team win rate of 73.44%. Asking all ten at every step yielded 49.22%. In this setting, allowing the receiver to choose whom to query reduced actual request counts and improved team performance.

Dear reviewer, the discussion period is about to close. Please read the authors' response and let us know briefly which of your concerns remain. If you have any further questions for the authors, please post them soon so the authors have time to respond.

Reviewer_UYMHOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies cooperative MARL with partial observability and bounded cross-timestep communication delays. They first introduce CGDC, which is a metric for the utility of one teammate's message for one agent. The authors provide an empirical surrogate of CGDC, and propose a framework CDCMA based on the empirical surrogate of CGDC. CDCMA has three modules: DCOS predicts the per-pair CGDC to decide whom to query, OTG predicts the future observation trajectory so that the future information can be approximated and included in the message, and CAMA aggregates the message by assigning different weights based on CGDC. Experiments show that CDCMA significantly outperforms previous baselines on classical MARL benchmarks like MPE and SMAC. Some ablation studies also show that each module has a positive effect on the final performance.

Critical strengths
  1. The experimental results are solid, and the improvement is significant. Comprehensive ablation studies are provided and they clearly show that each module is useful.

  2. The approach that predicts the per-pair CGDC and the future observation trajectory is novel and interesting.

Critical weaknesses
  1. The presentation is unclear and sometimes hard to follow. A large amount of important information, including how to train the Q function and what the loss is for each module, is moved to Appendix D.3. Then, it is hard to understand the algorithm procedure in the main paper.

  2. An important baseline is missing. Previous work CAIC [1] also studied predicting future information to solve the delayed communication setting. It would be important to compare with this work.

[1]. Li, Pei, et al. "CAIC: Congestion-Aware Intent Communication for Multi-Agent Reinforcement Learning." Forty-Second Annual Conference on Uncertainty in Artificial Intelligence. 2026.

Questions for authors
  1. How is the message encoder trained under delayed communication? Could it implicitly learn under a delayed-communication setting?

  2. The paper selects an adaptive λ\lambda that matches the gain and cost scales within each training minibatch, and compares different ratios kλ0k\lambda_0 with different kk. What is the performance of a fixed λ\lambda? Could the author provide the comparison or an explanation of why you need an adaptive λ\lambda?

  3. When agents use different networks without parameter sharing, is it practical for each agent to maintain a separate OTG for each teammate?

4 comments

Thank you for pointing out the gap between Section 4 and Appendix D.3. The original paper described how an agent scores teammates and chooses whom to query before explaining how it learns those scores. We rewrote the overview and brought the training objectives into the main paper.

W1. Following the algorithm and its training in the main paper

Section 4.1 now connects the centralized CGDC target to the score the receiver can predict from its local information. Figure 2 follows that prediction from choosing a sender to weighting its reply. At step kk, receiver ii uses DCOS to choose whom to query. Only messages already available to ii can influence its action at that step. If it queries jj, sender jj uses OTG to predict its own future observations and encodes them together with its history. At step t=k+dt=k+d, when the reply arrives, CAMA combines the score saved at kk with the message content and ii’s observation at tt. Thus the action at tt can depend on a request made several decisions earlier. Sections 4.2–4.4 give the selection, prediction and aggregation rules in that order.

We added §4.5 at the end of the method section. Section 4.1 follows a single message through a decision. Section 4.5 explains how a batch of replayed trajectories supplies the targets for both critics and the learning signals for the other modules. Both critics minimize squared one-step TD errors. The full critic evaluates the joint action with the receiver’s aggregate of delivered messages. Its TD target uses the reward and the target critic’s value for the next joint action with the next delivered-message aggregate. The gain critic evaluates a sender’s message twice, keeping the other inputs fixed and supplying either the message or a null input. Its two TD targets use the same reward and the same next joint action sampled from the target actors. Each branch bootstraps with its corresponding next-step message input. Section 4.5 then gives the following updates alongside the two critic objectives.

ComponentTraining signal
DCOSWith the replay observations and joint action fixed, the gain critic supplies the CGDC target in Eq. (10) for every candidate sender. DCOS uses the receiver’s local observation and actual arrivals to predict these targets and minimizes squared prediction error.
OTGThe action predictor minimizes cross-entropy on the recorded actions of teammates whose messages are available. The observation predictor minimizes squared error on the sender’s one-step observation change.
Actor, CAMA and encodersAt the replay state, the actor samples its own action from the current policy. The full critic weights the policy gradient, which also updates CAMA through its weighted aggregate and the local observation encoder through the receiver’s representation. The gradient through each delivered message reaches the sending-time encoder, as detailed in Q1.

For example, if only one message is available at a decision, that is the only sender represented in the actor’s aggregate. Centralized training can still construct a DCOS target for every candidate sender at the same transition. Appendix D.3 gives the expanded objectives (Eqs. 29–37), and Algorithm 1 in Appendix F gives the episode-level procedure.

W2. Comparison with CAIC

CAIC is directly relevant. It already uses trajectory prediction at the sender and attention over asynchronous messages at the receiver, and it also reports results on MPE and SMAC. We added the comparison in Appendix B.3. We have not evaluated CAIC and CDCMA head to head under the same channel protocol.

The overlap is closest in the construction of future-aware messages. CAIC uses a temporal masked autoencoder to encode predicted trajectories as intent. It also adjusts how often agents transmit in response to changes in intent and estimated delay, which can reduce queueing on its shared channel. In our setting, the delay on a link is sampled independently of the traffic. CDCMA asks which sender a receiver should query given that delay, and how much to use its reply once it arrives. CGDC compares the delayed message with both no message (gain) and timely information (cost). DCOS learns this criterion for requesting a message, and CAMA uses its saved value as a prior at delivery. Li et al., UAI 2026

Q1. Can the message encoder learn implicitly under delay?

Yes. Agent jj’s message encoder receives a policy gradient from the decision in which a teammate actually uses its message. If jj sends mj,ks=fjmsg(ej,k,τ^j,k)m^s_{j,k}=f_j^{\rm msg}(e_{j,k},\hat\tau_{j,k}) at step kk, the message contains its history representation ej,ke_{j,k} and its forecast τ^j,k\hat\tau_{j,k}. At step t=k+dt=k+d, receiver ii’s CAMA aggregates the delivered message mij,trm^r_{ij,t} with its other arrivals to obtain m~i,t\tilde m_{i,t}, which conditions ii’s action distribution.

In the backward pass, let gi,t=Qifull∇m~i,tlog⁡πi(ai,t∣ei,t,m~i,t)g_{i,t}=Q_i^{\rm full}\nabla_{\tilde m_{i,t}}\log\pi_i(a_{i,t}\mid e_{i,t},\tilde m_{i,t}) be the signal from the receiving actor at the aggregate. The contribution reaching the payload that jj produced at kk is

\underbrace{\left(\frac{\partial m^{r}_{ij,t}}{\partial m^{s}_{j,k}}\right)^{\top}}_{I} \left(\frac{\partial \tilde{m}_{i,t}}{\partial m^{r}_{ij,t}}\right)^{\top} g_{i,t}.$$ The right-hand Jacobian backpropagates through CAMA at the time of use. For this realized delivery, $m^r_{ij,t}=m^s_{j,k}$, so the channel Jacobian is the identity. The gradient therefore reaches $f_j^{\rm msg}$ and, through its history input $e_{j,k}$, the sender’s observation encoder. If several receivers use the same message, their contributions add at the sending-time encoding. The observation encoder also receives the local policy gradient from $j$’s own actor, which uses $e_{j,k}$ for its own action. This is how the *later* decision can teach the sender which features remain useful after a delay. Section 4.5 describes the local and cross-step gradient paths. Appendix D.3 explains the delivery Jacobian and the sum over receivers after Eq. (32). OTG supplies explicit prediction targets in addition to this implicit task signal. We tested the contribution of the resulting future-aware content by removing the forecast from the message while retaining its history, dimensionality, DCOS and CAMA. On `1o_10b_vs_1r`, mean win rate fell from **92.19 ± 4.03% to 66.41 ± 15.39%** under easy delay and from **73.44 ± 5.41% to 42.97 ± 9.67%** under super-hard delay (four seeds, mean ± sample SD). Appendix E.9, Fig. 11(a,b), shows the learning curves. In Predator Prey, Fig. 9(a–c) shows observed and predicted trajectories and per-step prediction loss. The horizon experiment in Fig. 3(b) found that performance improved from zero prediction steps to the maximum delay $d_{\max}$ and declined at $2d_{\max}$. **Q2. Fixed versus adaptive $\lambda$** We held the absolute value of $\lambda$ fixed throughout training on `1o_2r_vs_4r` under super-hard delay. The best of the four constants, $\lambda=0.1$, reached 60.94% mean win rate, within 2.34 percentage points of the adaptive default. Figure 11(c) in Appendix E.9 shows the four fixed-coefficient learning curves. The adaptive reference comes from Table 1 for the same map and delay. | Coefficient | Win rate (%), mean ± sample SD over four seeds | |---|---:| | Fixed 0 | 46.88 ± 11.41 | | Fixed 0.1 | 60.94 ± 6.51 | | Fixed 1 | 55.47 ± 5.34 | | Fixed 10 | 50.00 ± 9.20 | | Adaptive (default) | 63.28 ± 11.23 | A suitable constant comes close to the adaptive mean on this task. Going from $0$ to $0.1$ raised mean win rate by 14.06 points, while increasing the constant further reduced it. We use an adaptive default because the gain is a difference in learned action values and the cost is a KL divergence between action distributions. A fixed numerical weight need not keep them on comparable scales as the critic learns. For pairs with an available message in the current minibatch, we compute the mean positive part of the gain, $\bar g_+$, and the mean cost, $\bar v_c$. The rule $\lambda=\kappa\,\mathrm{stopgrad}(\bar g_+/\bar v_c)$ matches their scales, with $\kappa=1$ by default. We use zero weight when there are no such pairs or $\bar v_c\le 10^{-8}$. The positive part is used only to estimate the scale. Each sender’s training label retains its signed gain. Thus Fig. 3(c) in the original paper varies the multiplier $\kappa$ on the minibatch estimate, while Fig. 11(c) directly answers the fixed-$\lambda$ question (Appendix D.3, Eqs. 36–37). **Q3. Number of OTGs without parameter sharing** An agent needs one OTG for its own outgoing messages. The teammate indices in §4.3 identify the actions its predictor estimates from incoming messages. Those actions help the sender predict **its own** future observations. If three receivers request the same sender at one step, it rolls out that trajectory once and sends the resulting message to each of them. With distinct parameters for $N$ agents, the team would therefore maintain $N$ OTGs. At a step when $n_s$ distinct senders are requested, it would run $n_s$ forecasts. Delivery and aggregation still depend on the number of active links (Appendix D.4). Our reported experiments use parameter sharing across agents, with agent identity as an input where needed.

Thanks for the additional experiments and explanations. I maintain my positive score.

Thank you for your response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.