acceptodds
Under review as a conference paper at ICLR 2027

Behaviour-Aware Delay-Aligned Belief for Multi-Agent Reinforcement Learning with Communication Delays

Abstract

Multi-agent reinforcement learning with communication delays is challenging as information transmission among agents is not always instantaneous: each agent acts on local states that are stale by a different lag on each connection. State-of-the-art (SOTA) methods typically employ a belief representation that first predicts the current global state, and the agent then learns and acts on that predicted state. We theoretically demonstrate that the prediction error of such a belief is governed by two quantities: the approximation errors of the dynamics and of the agents' behaviours within the delay window, both amplified and accumulated with the delay by recursive forward propagation. We further analyse how belief error affects the induced policy and value function. Guided by these results, we propose Behaviour-Aware Delay-Aligned Belief (BADA). Specifically, BADA formulates the ragged information state into one token per other agent, aligned to that agent's freshest received state, and directly forecasts the agents' behaviours and the current global state explicitly. The empirical results show that BADA can effectively reduce the belief error compared to SOTA baselines. Furthermore, BADA achieves superior task performance in various benchmarks under different delays.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, timing-delay system, delayed reinforcement learning, communication delays

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5yW1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents BADA, an approach motivated by a belief error analysis under Lipschitz continuity assumptions. Using a two-head Transformer backbone, BADA takes the agent's own state and action history, and the freshest states of other agents received through communication. It predicts the other agents' action distributions over the delay window, then feeds these predictions back to the state head to predict the other agents' current states. The states are then fed into QMIX to compute Q-values.

Critical strengths

1.The experiments cover both constant and random delays on 2 benchmarks,each with 3 tasks.BADA’s performance in most settings shows its effectiveness compared to 7 other baseline methods.The ablation also shows behaviour prediction is beneficial, reducing belief error and improving return.

Critical weaknesses

1.The analysis in Section 4 is not directly aligned with the method. Theorem 1 bounds the error of a belief propagated recursively, but BADA predicts the current state in a single forward pass.The misalignment weakens the connection between theory and method.

Questions for authors

The relationship between Lipschitz continuity assumptions and BADA is still unclear,the misalignment between Theorem 1 and BADA needs further explanation.

1 comment

Thank you for your careful review and constructive comments.

W1 and Q1: Recursive analysis, direct prediction, and assumptions

The analysis in Section 4 is not directly aligned with the method. Theorem 1 bounds the error of a belief propagated recursively, but BADA predicts the current state in a single forward pass.The misalignment weakens the connection between theory and method.

The relationship between Lipschitz continuity assumptions and BADA is still unclear,the misalignment between Theorem 1 and BADA needs further explanation.

Response: Thank you for requesting a clearer explanation. Our contribution combines the belief-error analysis with behaviour-conditioned, delay-aligned reconstruction. Section 5 connects the analysis to two complementary design choices:

  1. Explicit behaviour prediction. Theorem 1 separates transition and behaviour approximation errors. The bound includes a behaviour approximation term even with exact dynamics, reflecting the unobserved actions of other agents. This motivates a behaviour head that predicts their action distributions over the delay window.

  2. Direct state reconstruction. The analysis shows how recursive predictions accumulate approximation errors across the delay window. This motivates predicting the current state directly, without intermediate state rollouts. BADA's state head receives the predicted behaviour distributions and delay-aligned history representations, then reconstructs the current state in a single forward pass. The reconstruction target is the delay-free state.

The common snapshot is used for the recursive analysis. BADA instead retains each agent's freshest received state through delay-aligned histories encoded by the shared transformer. The two heads have distinct, connected roles: the predicted behaviour distributions condition state reconstruction. Table 1 supports this feedback pathway: retaining behaviour supervision while restoring feedback reduces SMAX belief error from 4.75 to 2.92 and improves win rate from 0.53 to 0.65.

The Lipschitz assumptions define the scope of the recursive error analysis; they do not follow from using a transformer, and ε\varepsilon-greedy policies need not satisfy the policy Lipschitz condition. Theorem 1 motivates this architecture but is not a direct bound on BADA's estimator. In the revised version, we will make the link between the error analysis and the two-head design more explicit and clarify the distinction between the recursive analysis and BADA's direct estimator, including the scope of the Lipschitz assumptions.

Extending error guarantees to BADA's direct estimator and relaxing policy regularity assumptions are useful directions for future work.

Reviewer_iwLjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies cooperative MARL where each agent receives teammates' states over links with heterogeneous, time-varying delays. Theoretically, it analyzes the belief error bound between exact and approximated beliefs over the delay window. Then it proposes the BADA framework which embeds the delay-aligned agents' states, predict agent behavior over the delayed window and reconstruct current state, which is further combined as input to its QMIX utility. Experiments on MPE and SMAX under constant and random delays show lower belief error and higher task performance than QMIX-based delay baselines, with larger margins at longer delays and larger teams; ablations attribute most of the gain to the behavior head.

Critical strengths
  1. The experiments provide evidence that behavior feedback improves delay compensation. The ablation retaining behavior supervision while removing feedback is particularly informative: on the reported SMAX setting, feedback improves win rate from 0.53 to 0.65 and reduces reconstruction error from 4.75 to 2.92. This supports the practical usefulness of conditioning reconstruction on predicted behavior.
  2. The method addresses a consequential information constraint. Heterogeneous delays leave agents with observations from different timesteps and unknown intervening actions. BADA provides a mechanism for combining this information into current-state estimates for decentralized decision-making.
  3. The paper is easy to follow and presentation is clear.
Critical weaknesses
  1. The theory section has weak connection to the proposed BADA architecture.
  • First, the error bound is never quantitatively used to design, select, or validate BADA. For example, it does not determine the architecture, feedback representation, or loss weights; it did not establish when explicit behavior prediction improves upon direct state prediction. It is unknown how much BADA reduces the bound. In addition, the theory serves as motivation of widely adopted patterns (see below point 2).
  • Second, the analyzed belief "propagates from the common snapshot without conditioning on newer observations" (page 3), i.e. a recursive rollout from the oldest timestep. However, BADA is a direct predictor conditioned on each teammate's freshest state. So Theorem 1 is not a bound on BADA's estimator.
  1. The central idea of the paper is to predict other agents’ behavior and using behavior-related representations in downstream computation. However, this is a well-established pattern. To name a few, MADDPG (Lowe et al., NeurIPS 2017, Sec. 4.2) learns approximate policies of other agents by maximizing the likelihood of their observed actions and inputs their action probabilities into the critic; DPIQN/DRPIQN (Hong et al., AAMAS 2018) incorporates learned policy features into value estimation, while AMS/AMF (Hernandez-Leal et al., AIIDE 2019) studies auxiliary behavior supervision and explicit policy-feature conditioning. RDC (Fu et al., NeurIPS 2025) already reconstructs observations under heterogeneous delays, including with transformer-based compensators, and DFBT (Wu et al., ICML 2025) directly forecasts beliefs under delays. What is the distinctive contribution over established behavior modeling and delay compensation?

  2. In terms of the integration of agent-aligned histories and behavior-conditioned reconstruction, although the paper compares against relevant delay predictors, it does not sufficiently distinguish this integration from augmenting an existing predictor with a conventional behavior-prediction and conditioning module. The feedback ablation demonstrates usefulness within BADA, but does not establish the novelty of that mechanism.

  3. All baselines run on QMIX; DFBT, DUPO, and DDC are single-agent or plug-in methods adapted by the authors. No native multi-agent delay method is compared, despite cited only: CoDe (Song et al., AAAI 2025), which is also QMIX-based and evaluated under time-varying delays, DACOM (Yuan et al., AAAI 2023), and DAMARL (Chen et al., 2020).

Questions for authors

See "weakness" section. In addition, 1.can the author discuss CDCMA (Gao et al., 2026), which already pairs a teammate-action predictor with an observation-dynamics predictor under delays. 2. How well does the behavior head match teammates' executed behavior? App. B.2 supervises it with ε-greedy targets recomputed from delay-free observations, which differ from the policies actually executed on delayed inputs. Can you quantify this mismatch and compare against targets from logged executed actions?

1 comment

Thank you for your careful review and constructive comments. We summarise each concern below.

W1 — Concern summary: The recursive bound neither directly applies to BADA nor quantitatively determines its architecture or gains.

Response: Section 5 explicitly connects the analysis to two design choices. Theorem 1 separates transition and behaviour approximation errors; the behaviour term appears even with exact dynamics. This motivates predicting other agents' unobserved behaviour. The accumulation of errors through recursive propagation motivates direct current-state reconstruction without intermediate rollouts. BADA preserves each agent's freshest information. The theorem provides design motivation, not a direct estimator bound or a rule for choosing loss weights.

W2 — Concern summary: Behaviour modelling and delay compensation already exist in MADDPG, DPIQN/DRPIQN, AMS/AMF, RDC and DFBT; what is distinctive?

Response: Thank you for suggesting these comparisons. Our contribution is the error analysis and the resulting behaviour-conditioned, delay-aligned reconstruction. MADDPG [r2] uses inferred policies in critic learning; DPIQN/DRPIQN [r3] and AMS/AMF [r4] use behaviour models in value/policy learning. RDC [r1] reconstructs observations, including by direct prediction; DFBT [r5] directly predicts states using known own actions. BADA explicitly predicts unobserved teammate action distributions over the delay window and feeds them into current-state reconstruction from histories aligned to each agent's freshest received state. We will expand the comparison with behaviour-modelling methods in the revised Related Work.

W3 — Concern summary: Feedback ablation shows utility, but does not establish novelty over adding a conventional behaviour module to another predictor.

Response: Table 1 isolates the feedback pathway while retaining behaviour supervision: feedback reduces SMAX belief error from 4.75 to 2.92 and improves win rate from 0.53 to 0.65. On MPE, error falls from 0.32 to 0.26 and return improves from -38.88 to -32.11. These results demonstrate the benefit of using predicted behaviour in reconstruction beyond auxiliary supervision alone. They test this mechanism within BADA, not the same module added to another predictor.

W4 — Concern summary: No native multi-agent delay method is compared, and all baselines use QMIX.

Response: The claim that no native multi-agent delay method is compared is incorrect. RDC [r1] is a SOTA MARL delay-compensation method with public official code, training scripts and configurations supporting reproducibility. Following RDC, Section 6.1 uses QMIX as the common backbone and compares RDC-GRU and RDC-TF under the same delay protocol. This provides a controlled comparison with a strong native MARL baseline. Sections 6.2–6.3 show BADA's gains in belief accuracy and task performance across multiple tested settings.

Q1 — Concern summary: How does BADA differ from CDCMA's action-to-observation prediction?

Response: Thank you for highlighting CDCMA. We will include a comparison with CDCMA (without released official implementation) and report the results in the revised version. CDCMA [r6] recursively predicts future observations for outgoing messages. Our contribution combines the analysis of transition and behaviour errors with behaviour-conditioned, delay-aligned reconstruction (Sections 4–5). BADA predicts teammate action distributions over the delay window and directly reconstructs current states at the receiver from each agent's freshest history, avoiding intermediate state rollouts. The Recursive Belief comparison in Section 6.2 supports this design motivation.

Q2 — Concern summary: Do delay-free supervision targets match executed behaviour? Can the mismatch and logged-action targets be evaluated?

Response: Appendix B.2 defines supervision as recomputed delay-free policy references, which can differ from delayed-execution behaviour. Reconstruction gradients also train the behaviour head through feedback. Executed-action accuracy and logged-target comparisons are not reported; the oracle tests privileged action feedback, not target choice. Thank you for this suggestion. We will discuss behaviour-target comparisons and mismatch diagnosis as future work.

References

[r1] Rainbow Delay Compensation: A Multi-Agent Reinforcement Learning Framework for Mitigating Observation Delays.

[r2] Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments.

[r3] A Deep Policy Inference Q-Network for Multi-Agent Systems.

[r4] Agent Modeling as Auxiliary Task for Deep Reinforcement Learning.

[r5] Directly Forecasting Belief for Reinforcement Learning with Delays.

[r6] Communication Gain and Delay Cost Under Cross-Timestep Delays in Cooperative Multi-Agent Reinforcement Learning.

Reviewer_adAmOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a method to align delays to predict locally unobserved states to aggregate-process each agent’s behavior despite heterogeneous delays. The presented method involves a transformer backbone with two heads, one of which forecasts the current states of the agents (whose reported states are stale), and the other estimates the other agents’ action distributions. This method, BADA, operates on top of well-known baselines such as QMIX, and is shown to be effective on MPE and SMAC environments.

Critical strengths

None

Critical weaknesses

While the paper is an interesting take on communication delays, its weakness is twofold: the treatment of delays as a control knob and the simplistic delay modelling. Communication delays are a consequence of a number of other variables: number of hops, medium contention, channel bandwidth, interference, number of agents, etc. All of these directly govern delay profiles, and yet none of these is studied in this paper. Instead, delays are artificially controlled independent of their aforementioned governing variables. This contrived isolation makes the reported results and experimental setting difficult to credit. The delay profile, if sufficiently sophisticated and reflective of real-world constraints, could have made the evaluation appear more promising, but that is not the case. All delay amounts are uniformly random, which is highly unrealistic, as mentioned by the authors. There seems to be a major disconnect between the theory and the experiment. Please see questions below.

Questions for authors
  1. QMIX is known to have limited representational power in its handling of joint action-value functions, up to monotonicity. How would this inherent limitation of QMIX affect the operation of BADA? What if VDN or QTRAN were used?

  2. The presented bound seems too loose. Isn’t it vacuous under the tested delay values?

  3. Lipschitz does not hold under the epsilon-greedy policies in Q values. How does BADA behave with this violation?

  4. Could you give a measurement of the theory-provided estimates versus the experiment results on L_p and L_n?

2 comments

Thank you for your careful review and constructive comments.

W1: Delay modelling

While the paper is an interesting take on communication delays, its weakness is twofold: the treatment of delays as a control knob and the simplistic delay modelling. Communication delays are a consequence of a number of other variables: number of hops, medium contention, channel bandwidth, interference, number of agents, etc. All of these directly govern delay profiles, and yet none of these is studied in this paper. Instead, delays are artificially controlled independent of their aforementioned governing variables. This contrived isolation makes the reported results and experimental setting difficult to credit. The delay profile, if sufficiently sophisticated and reflective of real-world constraints, could have made the evaluation appear more promising, but that is not the case. All delay amounts are uniformly random, which is highly unrealistic, as mentioned by the authors. There seems to be a major disconnect between the theory and the experiment. Please see questions below.

Response: Thank you for highlighting network-dependent delays. Our contribution concerns belief reconstruction from delayed information. We follow RDC's protocol [r1] and compare all methods under the same constant and random delays. Under random delays, nominal delays are uniform, while received source timestamps are nondecreasing (Appendix B.1). This isolates compensation for stale information under a common protocol. We will clarify this evaluation scope in the revised version and discuss network-dependent and temporally correlated delays as directions for future work.

Q1: QMIX, VDN, and QTRAN

QMIX is known to have limited representational power in its handling of joint action-value functions, up to monotonicity. How would this inherent limitation of QMIX affect the operation of BADA? What if VDN or QTRAN were used?

Response: Thank you for suggesting these backbones. Our main contribution is the belief module: predicting unobserved teammate behaviour and using it to directly reconstruct current states. Following prior work such as RDC [r1], we use QMIX as the common backbone for all methods to compare their belief modules. BADA retains its monotonicity constraint. We will add VDN and QTRAN results in the revised version to examine performance with other backbones. These extend the evaluation of the belief module beyond the common QMIX setting.

Q2: Tightness of the bound

The presented bound seems too loose. Isn’t it vacuous under the tested delay values?

Response: Theorem 1 separates transition and behaviour approximation errors and characterizes their propagation, motivating explicit behaviour prediction and direct reconstruction in Section 5. Its bound is ϵbi=ci∑k=0δti−1αik\epsilon_b^i=c_i\sum_{k=0}^{\delta_t^i-1}\alpha_i^k, where cic_i combines transition and behaviour errors and αi=LP(1+∑j≠iLπj)\alpha_i=L_{\mathcal P}(1+\sum_{j\ne i}L_{\pi^j}). It grows geometrically for αi>1\alpha_i>1, linearly for αi=1\alpha_i=1, and remains bounded for αi<1\alpha_i<1. Global constants are not measured, so non-vacuity at the tested delays is unverified; a bound above diam⁡(S)\operatorname{diam}(\mathcal S) is uninformative. Thank you for raising this point. We will clarify the bound's interpretation and scope in the revised version.

Q3: Lipschitz assumptions and ε\varepsilon-greedy policies

Lipschitz does not hold under the epsilon-greedy policies in Q values. How does BADA behave with this violation?

Response: Thank you for pointing out this distinction. Appendix B.2 notes that ε\varepsilon-greedy policies need not satisfy the policy Lipschitz assumption: greedy-action switches can be discontinuous even with continuous Q-values. We will make this scope explicit in the main text. BADA is trained and evaluated with this policy; Table 1 supports behaviour feedback empirically. The theory motivates the design under its assumptions and does not guarantee performance for the implemented policy.

Q4: Measurements of theoretical constants

Could you give a measurement of the theory-provided estimates versus the experiment results on L_p and L_n?

Response: Thank you for suggesting this diagnostic. Figure 2 and Table 1 measure reconstruction error, not the global Lipschitz constants or uniform approximation errors in Definition 2. These constants have not been estimated. Sampled sensitivities alone cannot certify global bounds across greedy-action switches. We will clarify this distinction when discussing the theory and empirical results. Connecting empirical sensitivity measurements to informative error guarantees is a useful direction for future work.

References

[r1] Rainbow Delay Compensation: A Multi-Agent Reinforcement Learning Framework for Mitigating Observation Delays.

I'd like to thank the authors for actively participating in the discussion.

I have to admit that some of the concerns were addressed well while others remain, leading me to keep my rating.

First, the delay modeling needs a fuller treatment. Yes, taking into account delay itself is an important and significant step towards real-world MARL, but with the current modeling, the realism of the delay is, in my opinion, not quite sufficient even to resemble simple scenarios.

Second, while the trajectory of the related research from VDN to QMIX to QTRAN is an important and representative one, showing that BADA can operate on top of each one of these is rather an anecdotal evidence than a principled demonstration of applicability. I think BADA can be much better highlighted if it were shown with theoretical rigor how it works (or fails) under the different natures of decomposable joint action value functions, rather than instantiating it on a number of specific algorithms.

Third, I am not entirely convinced with the bounds. I believe I'm questioning more of its tightness whereas Reviewer iwLj went on further to criticize its usefulness as well. In this regard, I side with Reviewer iwLj; there should be a more convincing and tighter incorporation of the bounds into the algorithm design.

BADA can earn its place in the MARL literature, if delay modeling can be made realistic enough. At its current state, it's difficult to avoid critique on novelty, and hence, having to compete with broader MARL algorithms such as MADDPG is diluting BADA's contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.