- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies cooperative MARL under cross-timestep communication delays. The authors propose CGDC, a metric that evaluates delayed-message utility by balancing communication gain against delay cost, and derive a theoretical bound on delay-induced return loss.
Based on CGDC, they develop CDCMA, which combines communication partner selection, future-aware message generation, and attention-based message aggregation.
Experiments on MPE and SMAC under stochastic communication delays show that CDCMA outperforms the selected baselines. Ablation studies, information-gap analyses, and zero-shot cross-difficulty experiments provide additional validation.
Critical strengths
-
The proposed CGDC metric evaluates delayed messages against both no-message and timely-message references. This provides a more fine-grained criterion for communication selection by explicitly distinguishing communication utility from temporal misalignment.
-
The Observation Trajectory Generator (OTG) incorporates predicted future observations into outgoing messages to mitigate communication delays. This is a reasonable design choice, and the ablation results suggest that it contributes substantially to the overall performance improvements.
Critical weaknesses
-
The theoretical justification is not fully convincing. The return-loss bound relies on actor-policy divergence, while the implemented CGDC uses a critic-based surrogate. The approximation gap between them is not adequately validated.
-
The communication gain metric ignores potential redundancy among messages by evaluating each sender independently against a no-message baseline.
-
DCOS estimates communication utility at request time, but CAMA reuses these scores after delayed delivery. Their reliability is questionable when the receiver's state changes significantly.
-
The baseline comparison is not entirely fair, as DACOM is evaluated without TimeNet and CoDe is reimplemented.
-
The paper lacks sufficient analysis of communication efficiency and scalability. CDCMA introduces additional prediction and scoring modules, but their computational overhead and practical benefits are not convincingly evaluated.
Questions for authors
-
Can the authors quantify the actor–probe approximation errors in Corollary 3.4 to justify the practical relevance of the theoretical bound?
-
How does CGDC handle redundant or correlated messages from different agents?
-
How reliable are request-time CGDC scores when messages arrive several timesteps later?
-
Why is DACOM evaluated without TimeNet, and how was the CoDe reimplementation validated?
-
Can the authors report communication frequency, inference latency, and computational overhead relative to the baselines?
5 comments
W3 / Q3. Using a request-time score after delivery
The saved score remains useful as an attention prior in our experiments. We have not directly measured how accurately it predicts arrival-time utility after a large change in the receiver’s state. To test its use in the policy, we compared CDCMA with PlainAttn, which keeps sender selection, future prediction and content attention, removing just the saved score from the aggregation weights. Figure 11(a,b) (Appendix E.9) gives the learning curves on 1o_10b_vs_1r under easy and super-hard delays, extending the CN experiment in Fig. 3(a). With the same checkpoint-selection rule, the four-seed mean win rates ± sample SD were as follows.
| Delay | PlainAttn | CDCMA | Gain from the prior |
|---|---|---|---|
| Easy | 79.69 ± 7.86% | 92.19 ± 4.03% | 12.50 percentage points |
| Super-hard | 58.59 ± 3.93% | 73.44 ± 5.41% | 14.84 percentage points |
The two methods differ at the moment of consumption. Suppose the receiver requested a message at and uses it at . CDCMA saves its positive request score at . At the receiver computes a fresh query from its current state representation and a key from the delivered content. Equation (14) assigns each arrival a weight proportional to , normalized over the available senders. With several arrivals, a message that scored well at can receive less weight than one that better matches the current query. With just one arrival its normalized weight is one. The 12.50- and 14.84-point gains quantify what retaining the prior adds to this content comparison. The 79.1% and 82.1% sign agreements in Table 6 compare the current DCOS score with on PP and SMAC, respectively. These statistics use scores recomputed at arrival.
W4 / Q4. DACOM’s TimeNet and validation of CoDe
DACOM. TimeNet's within-step waiting mechanism does not apply to our cross-step delivery protocol. The original TimeNet learns a waiting threshold within a decision step: a longer wait allows more messages to arrive before the agent acts in that step. In our channel, a message sent at step with delay becomes available only when the environment reaches step . Extending the wait within step therefore leaves the messages available for that decision unchanged, so the original TimeNet waiting variable has no effect here.
For waiting to change the available information in our setting, the agent would have to defer its decision to a later environment step. This requires defining which actions it executes in the intervening steps and how to train the waiting policy from the resulting returns. The result would be a new method for waiting across steps, beyond reproducing the original TimeNet. We therefore reimplemented the remaining DACOM modules from the paper and used the messages available at each decision under the common cross-step delivery protocol. Section 5.1 identifies all DACOM results as the variant without TimeNet; revised Appendix E.1 states that we reimplemented this variant from the paper and explains the adaptation rationale above.
CoDe. We checked our paper-based implementation against the released source and found differences that change the messages the receiver uses. For example, the official aggregator sums values with weights , where is message age. Our implementation additionally normalized these weights by . If all messages have aged by the same number of steps, this extra normalization cancels the common discount. Our future-action predictor also used replayed actions as targets, whereas the official code uses the online Q-network’s greedy actions. At test time our intent encoder sampled from its distribution and the official encoder used its mean. Tables 1, 2 and 5 and the original baseline curves refer to our implementation.
We connected the released code to our cross-step delivery interface and MPE environments, checked its deterministic forward output against upstream at zero delay, and tested delayed delivery, replay and gradient propagation. We also ran a short MPE training run and checked checkpoint save/load. Both versions are included with the paper as CoDe-original-reimplementation.zip and CoDe-official-adapted.zip. Official CoDe implementation, revision 255383b
We trained the adapted official CoDe implementation on four tasks under four delay settings with four seeds per setting (64 runs). The table pairs our reimplementation from Table 1 with the adapted official code from Table 9 in Appendix E.11. Each entry is the mean ± sample SD across four seeds. For the adapted official code, Table 9 uses the final evaluation for each seed, with 100 evaluation episodes on CN and PP. Figure 12(a–d) shows its learning curves across 400 aligned evaluation positions.
| Task | CoDe implementation and metric | Easy | Medium | Hard | Super-hard |
|---|---|---|---|---|---|
1o_10b_vs_1r | Our reimplementation, win rate (%) | 32.81 ± 1.80 | 35.16 ± 4.69 | 24.22 ± 22.88 | 25.00 ± 15.31 |
1o_10b_vs_1r | Adapted official code, win rate (%) | 41.41 ± 9.67 | 38.28 ± 13.10 | 35.16 ± 5.92 | 25.78 ± 5.34 |
1o_2r_vs_4r | Our reimplementation, win rate (%) | 58.59 ± 14.29 | 40.63 ± 7.65 | 28.91 ± 9.33 | 24.22 ± 6.93 |
1o_2r_vs_4r | Adapted official code, win rate (%) | 39.06 ± 26.33 | 43.75 ± 9.88 | 35.94 ± 15.42 | 30.47 ± 8.98 |
| CN | Our reimplementation, mean episode reward | −3.11 ± 0.14 | −3.09 ± 0.05 | −4.36 ± 2.55 | −4.42 ± 2.75 |
| CN | Adapted official code, mean episode reward | −3.76 ± 0.81 | −3.20 ± 0.26 | −3.07 ± 0.33 | −2.95 ± 0.24 |
| PP | Our reimplementation, mean episode reward | −1.82 ± 0.03 | −1.83 ± 0.03 | −1.86 ± 0.08 | −1.85 ± 0.07 |
| PP | Adapted official code, mean episode reward | −2.02 ± 0.36 | −2.01 ± 0.09 | −2.13 ± 0.20 | −2.32 ± 0.41 |
On CN, the adapted official code had lower mean episode rewards than our reimplementation under easy and medium delay (by 0.65 and 0.11) and higher rewards under hard and super-hard delay (by 1.29 and 1.47). On PP, our reimplementation had higher mean episode rewards in all four settings, by 0.18–0.47.
Thank you for suggesting that we measure both discrepancies in Corollary 3.4. We compared the actor and the critic-induced probe before and after replacing each delayed message with its timely reference.
W1 / Q1. The actor–probe gap in Corollary 3.4
Across the four task–delay settings in Table 8 (Appendix E.10), the mean actor–probe error was 0.0214–0.0364 in total variation with a timely reference and 0.0212–0.0319 with a delayed message. Together, the two errors accounted for 12.9–16.8% of the mean local upper bound on the sampled decisions.
We took decisions at which receiver i used a message from sender j. For each decision we ran the actor once with the delivered message and once with j’s current representation ej,t in its place. We kept the receiver’s observation and hidden state, the other senders’ messages, and the saved request scores fixed, and recomputed attention. For the same two inputs we computed the corresponding probe by taking a softmax over the gain critic’s action values (η=1, as in training). This gives the actor and the probe under each condition. For x∈{tar,del}, their discrepancy is δx=21∑a∣πx(a)−πx(a)∣. Figure 4(a,b) in the original submission replaced the entire set of messages. Replacing one sender at a time lets us measure the two terms for the particular message considered in the corollary.
The local inequality is G≤E+C. Here G=∥πtar−πdel∥TV is the observed change in the actor, E=δtar+δdel is the contribution from approximating the actor, and C=DKL(π~tar∥π~del)/2 is computed entirely from the critic. The inequality follows from the TV triangle inequality and Pinsker’s inequality. We evaluated these quantities per message, averaged within each trained seed, and report the mean ± sample SD of the four seed averages.
1o_10b_vs_1r1o_10b_vs_1rFor example, under super-hard delay on SMAC, substituting one timely message changed the actor distribution by 0.3585 TV on average. Of the mean upper bound of 0.5281, the critic-derived cost contributed 0.4599 and the two errors together contributed 0.0682. The share Eˉ/(Eˉ+Cˉ) was 16.8%, 14.8%, 16.8% and 12.9% in table order. In each of the 16 trained models, Eˉ<Cˉ/4. This establishes the scale of the approximation on decisions where the receiver used the message. The full return-loss bound in Eq. (8) also takes a discounted expectation along the timely-reference policy’s trajectories.
W2 / Q2. Redundancy between senders
Two teammates reporting the same useful information can both receive positive scores and both be queried. The gain in Eq. (1) compares access to sender j alone with receiving no messages. In Eq. (9), we estimate that gain by giving the same gain critic j’s message or a null message with the other inputs held fixed. DCOS has access to the receiver’s available messages when it predicts a score, and CAMA weights the messages jointly once they arrive. The pairwise target itself does not discount a sender solely because another candidate could report the same information. Appendix D.3 describes how we construct the target for each sender.
We used a leave-one-out test to examine the consequence of this choice. For each arrival, we retained all the other messages and compared the full critic’s value with and without j’s message. We held the observations and joint action fixed and recomputed the aggregate after masking j. Call the difference ΔQ=Qifull(all)−Qifull(without j). If a high pairwise gain mostly picked out information already supplied by teammates, retaining that sender would add little to the full critic’s estimate. On PP, mean ΔQ was 0.088 for the top 30% of messages ranked by pairwise gain and −0.019 for the bottom 30%. On
1o_10b_vs_1rthe corresponding means were 0.121 and −0.012. We formed the groups within each seed and delay setting and then weighted the four seeds and four settings equally (Appendix E.7, Table 6). The ranking thus identified messages with larger average contributions in the presence of other arrivals. It can still assign high scores to an individual pair of duplicates.