acceptodds
Under review as a conference paper at ICLR 2027

Spiking Actor-Critic with Temporal Dendritic Heterogeneity for Asynchronous Multi-Agent Reinforcement Learning

Abstract

Asynchronous multi-agent reinforcement learning requires agents to coordinate while making decisions at different macro-action completion times. Information exchange can supplement incomplete local observations, and yet under asynchronous execution, a receiver's current observation can be combined with teammate messages generated at the same or earlier times. This temporal mismatch can make it difficult to retain useful teammate information without allowing outdated information to influence current decisions. To integrate this asynchronous information, we propose CoSpAC, a compartmentalized spiking actor-critic framework inspired by temporal dendritic heterogeneity. Each actor processes local observations and aggregated teammate messages through separate dendritic compartments with distinct decay factors. In this way, CoSpAC provides different temporal integration timescales for observations and messages from different decision times, which are subsequently combined through somatic dynamics for action selection. Additionally, CoSpAC exchanges teammate information at agents' individual decision events and optimizes message generation through a receiver-oriented objective based on its influence on receiving agents' policies. Experiments across asynchronous multi-agent tasks show that CoSpAC outperforms state-of-the-art asynchronous MARL approaches, demonstrating the effectiveness of the proposed framework for asynchronous cooperation.

Primary area:
reinforcement learning
Keywords:
Asynchronous Multi-Agent Reinforcement Learning, Spiking Reinforcemet learning, Multi-Compartment Spiking Neural Networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xUHNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CoSpAC, a compartmentalized spiking actor-critic framework for asynchronous multi-agent reinforcement learning with macro-actions. Local observations and teammate messages are processed through separate dendritic compartments with independently learned decay factors, before being combined through somatic dynamics. Messages are generated at individual decision events and trained using a receiver-oriented policy objective. Experiments on BoxPushing and Overcooked variants report improvements over several MARL baselines, together with component ablations and estimated inference compute energy.

Critical strengths
  1. The proposed separation of local observations and teammate messages is a reasonable architectural choice for asynchronous cooperation.
  2. The reported evaluation covers multiple layouts, randomized initial configurations, and heterogeneous macro-action durations. CoSpAC shows substantial improvements on BoxPushing and favorable results under heterogeneous durations in Overcooked, making the overall framework promising.
Critical weaknesses
  1. Implementation and evaluation details are missing. Message aggregation, timestamp encoding, historical-message gradients, and SNN critic training are insufficiently specified. The return evaluation protocol and final-performance statistics are also unclear, limiting reproducibility.
  2. The contribution of SNNs is not isolated. Main comparisons change both communication and network architecture. HC supports independent decay factors, but does not establish the benefit of spiking dynamics. A communication-matched non-spiking counterpart is needed.
  3. The membrane updates (Eqs. 7–8) run a fixed number of internal steps per decision event and never use macro-action duration or message age (A.4), so temporal integration is event-scaled rather than time-scaled. The source of the reported Macro+ advantage (Fig. 4) is therefore unexplained.
  4. Although the experiments include ACAC (2025), they omit ToMacVF (2025), which evaluates on overlapping benchmarks. The paper should also clarify its differences from AsynCoMARL (2025) and provide an appropriately adapted or communication-matched comparison.
  5. Figure 1 doesn't show the core contribution: the separate dendritic compartments, independent decay factors, or somatic integration described in its caption.
Questions for authors
  1. Please specify the message generator, aggregation operator, timestamp encoding, and asynchronous critic implementation.
  2. Please also address the omission of ToMacVF.
  3. Equations (7)–(8) apply a fixed number of internal updates per decision. How does the model use elapsed environment time or message age, including the timestamp in Eq. (11)? Can a controlled message-age or timing experiment distinguish general memory benefits from improved handling of stale information?
Reviewer_VFsfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CoSpAC, a compartmentalized spiking actor-critic framework for asynchronous MARL. It processes local observations and teammate messages through separate dendritic compartments with independently learned decay factors, incorporates event-based communication and receiver-oriented message learning, and demonstrates competitive or improved performance on BoxPushing and Overcooked variants.

Critical strengths

The paper addresses a relevant challenge in asynchronous cooperation through a well-motivated separation of local and communicated information into distinct temporal processing pathways. Evaluations under heterogeneous macro-action durations and comparisons against shared decay factors provide targeted empirical support for this design.

Critical weaknesses
  1. The paper does not report the learned decay factors across environments or examine whether the two compartments develop distinct memory timescales. Although the HC ablation supports independently parameterized decay factors, it does not directly establish the proposed temporal specialization mechanism.
  2. ACAC’s actors rely on their own macro-observation histories, whereas CoSpAC’s actors additionally receive teammate messages during execution. Consequently, the comparison cannot isolate improvements from the proposed architecture from those attributable to additional teammate information.
Questions for authors

Q1. Information-matched ANN comparison. Could the authors provide an ANN baseline with comparable information access, communication settings, model capacity, critic architecture, and training budget? This would help distinguish the contribution of spiking dynamics from that of communication and separate memory pathways.

Q2. Curve completeness and statistical reliability. Why do CoSpAC-HeteroMacro and ACAC-Micro in Figure 4’s Overcooked B panel, along with some curves in Figures 5 and 6, appear to stop before 10M timesteps? Is this due to plotting or curve overlap, or did some runs end prematurely? The use of only three random seeds also limits confidence in the results; could the authors expand the evaluation to five or more seeds to better assess the stability and reliability of the reported improvements?

Reviewer_gHQFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoSpAC, a spiking actor-critic framework for asynchronous MARL. The method addresses coordination among agents that execute macro-actions with different durations and exchange information at different decision times. The main contributions are:

  • Multi-compartment spiking architecture: Separate memory compartments for local observations and teammate messages, each with independently learned decay factors.
  • Asynchronous communication: Agents generate and retrieve messages at their individual decision events.
  • Receiver-oriented communication learning: A communication objective that trains message generation using the receiving agent's advantage. Experiments on BoxPushing and Overcooked demonstrate competitive or improved performance against existing asynchronous MARL methods, along with estimated computational energy savings.
Critical strengths
  • Relevant problem setting: Addresses asynchronous coordination under partial observability, an important challenge in multi-agent systems.
  • Evidence supporting heterogeneous memory: Ablations comparing shared and separate decay factors demonstrate benefits from independently modeling local observations and communicated information.
  • Potential energy efficiency: Spiking computation offers promising theoretical energy savings relative to conventional recurrent architectures.
Critical weaknesses
  • Limited novelty and insufficient isolation of contributions. The framework combines existing techniques from asynchronous MARL, inter-agent communication, and spiking neural networks. While the integration is potentially useful, the paper does not sufficiently establish the advantages of the proposed spiking architecture over conventional recurrent architectures with comparable communication and memory mechanisms.

  • Missing communication-enabled baselines. The evaluated asynchronous MARL baselines (ACAC, Mac-NIACC, and Mac-IAICC) do not incorporate explicit learned inter-agent communication comparable to CoSpAC. Consequently, the observed improvements may be attributable to communication rather than the proposed architecture. Comparisons with communication-enabled approaches, such as AsynCoMARL, or communication-augmented recurrent baselines would strengthen the evaluation.

  • Missing ablation of the receiver-oriented communication objective. The receiver-oriented loss in Equation 13 is presented as a methodological contribution, but its independent effectiveness is not experimentally evaluated. While the paper ablates communication and shared versus separate decay factors, it does not isolate the benefit of the proposed communication-training objective relative to ordinary end-to-end policy optimization.

Questions for authors

Q1. Contribution of receiver-oriented learning: Could the authors provide an ablation removing the receiver-oriented communication objective while retaining the same architecture and communication mechanism? This would clarify its contribution beyond standard policy optimization.

Q2. Fair comparison with communication-enabled methods: How does CoSpAC compare with existing asynchronous MARL approaches equipped with explicit communication, particularly AsynCoMARL? Could a conventional recurrent actor with separate observation and communication memories serve as an additional baseline?

Q3. Sensitivity to internal spiking steps: The paper uses Q=2Q=2 for BoxPushing and Q=4Q=4 for Overcooked. How were these values selected, and how sensitive is performance to different choices of QQ?

Q4. Practical energy efficiency: The reported energy savings rely on theoretical operation-cost assumptions. Could the authors clarify the assumed hardware model, whether memory access and other computational overheads are included, and whether similar performance and energy advantages would be achievable on actual hardware?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.