acceptodds
Under review as a conference paper at ICLR 2027

Spike-BiST: Bidirectional Stabilization of Spike Representation and Surrogate Gradient for Multi-Agent Reinforcement Learning

Abstract

Spiking neural networks provide an energy efficient computing paradigm for decision making by exploiting sparse spike activity and event-driven computation. Training spiking policies under changing conditions remains challenging in multi-agent reinforcement learning. Concurrent policy updates across agents alter observation distributions and learning signals, which can destabilize spike representations and surrogate gradient propagation. To address these issues, we propose Spike-BiST, a bidirectional stabilization framework that coordinates spike representation adaptation and surrogate gradient regulation. A dual-channel adaptive population encoding mechanism is introduced to adapt receptive fields to observations and preserve consistent spike representations across training conditions. Surrogate gradient propagation is regulated by a reliability-driven modulation mechanism that combines reinforcement learning signals with spiking dynamics to reduce gradient fluctuations. Experiments on SMAC, GRF, and MPE demonstrate that Spike-BiST achieves improved task performance and training stability, with lower estimated policy-network inference energy than ANN-based models.

Primary area:
reinforcement learning
Keywords:
Spiking Neural Networks, Multi-Agent Reinforcement Learning, Adaptive Spike Encoding, Surrogate Gradient Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8qb2OpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Spike-BiST, a method that enables spiking neural networks to learn cooperation more stably in multi-agent reinforcement learning. To address input variations and gradient fluctuations arising from simultaneous learning by multiple agents, the method dynamically adjusts and smooths the encoding of observational information while modulating backpropagation based on a combination of team performance, individual learning signals, and neuronal states. Experiments conducted in StarCraft, soccer, and particle environments demonstrate that the method improves cooperative performance and training stability while maintaining low energy consumption for inference.

Critical strengths

The experimental coverage is solid, and the results are competitive.

Critical weaknesses
  1. The paper cites [1] and [2], but the novelty relative to them is limited: PopSAN [1] already applies population coding to deep RL, and [2] already studies adaptive surrogate gradients for spiking RL.

  2. The core claim of "improved stability" lacks direct evidence. The central claim of improved training stability is not directly measured. The paper reports final win rates/returns with 5 seeds and 95% CIs, and Fig. 3 ablates the two components; these show that both components contribute to performance, but they do not show that the gains are due to improved stability.

[1] Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control.

[2] Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks.

Questions for authors
  1. Equations (14)–(17) define the agent-specific surrogate-gradient scale kq,tik_{q,t}^i. This scale does not affect the forward policy outputs, since spikes are generated by the hard threshold. It enters the surrogate derivative in Equation (3). Further explanation is needed regarding how the MAPPO loss updates the modulation networks GdynG_{dyn} and GgateG_{gate}. While the regularization term in Equation (19) provides gradients, it primarily encourages smoothness and proximity to the base scale, without automatically demonstrating that the network has learned a task-beneficial modulation strategy. Could the authors elaborate on this aspect?

  2. Appendix A.1 specifies the hypernetwork hφh_φ as Linear(D+did,64)→ReLU→Linear(64,2DP)Linear(D+d_id, 64) → ReLU → Linear(64, 2DP). For SMAC 3m (D=48,P=8,did=1)(D=48, P=8, d_id=1), this is (48+1)×64+64×(2×48×8)≈52.3K(48+1)×64 + 64×(2×48×8) ≈ 52.3K MACs per forward pass per agent. Table 1, however, reports only 2.3K dense MACs for Spike-BiST. Please clarify whether the 2.3K is per SNN timestep or per environment step (Ts=8), and whether it is per agent or per team; otherwise the comparison is not apples-to-apples. In either case, the table does not appear to include the encoder’s cost, so the reported energy saving should be recomputed with the encoder included.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 8qb2(Q1-Q2)

Q1. MAPPO loss and modulation-network updates

Both modulation networks GdynG_{\mathrm{dyn}} and GgateG_{\mathrm{gate}} are updated using the MAPPO policy loss, with the modulation scale kq,tik_{q,t}^{i} participating in the parameter updates. During each policy update, the current network parameters generate kq,tik_{q,t}^{i}, which modulates the surrogate gradient during backpropagation. Let δk,q,ti=∂Lπ/∂kq,ti\delta_{k,q,t}^{i}=\partial\mathcal{L}_{\pi}/\partial k_{q,t}^{i} denote the scale gradient returned by this operation. The expressions below give the gradient contributions of a single agent-step sample. Specifically, according to Eq. (17), the gradients with respect to ωdyn\omega_{\mathrm{dyn}} and ωgate\omega_{\mathrm{gate}} are given by:

For GdynG_{\mathrm{dyn}}:

\frac{\partial\mathcal{L}_{\pi}}{\partial\omega_{\mathrm{dyn}}}=\frac{\partial\mathcal{L}_{\pi}}{\partial k_{q,t}^{i}} \frac{\partial k_{q,t}^{i}}{\partial\Delta_{\mathrm{dyn},t}^{i}} \frac{\partial\Delta_{\mathrm{dyn},t}^{i}}{\partial G_{\mathrm{dyn}}} \frac{\partial G_{\mathrm{dyn}}}{\partial\omega_{\mathrm{dyn}}}=\delta_{k,q,t}^{i}k_{q,t}^{i}(1-w_t^{i})a_1 \left[1-\tanh^2\left(G_{\mathrm{dyn}}(z_t^{i})\right)\right] \frac{\partial G_{\mathrm{dyn}}(z_t^{i})}{\partial\omega_{\mathrm{dyn}}}. \end{aligned}$$ For $G_{\mathrm{gate}}$: $$\begin{aligned} \frac{\partial\mathcal{L}_{\pi}}{\partial\omega_{\mathrm{gate}}}=\frac{\partial\mathcal{L}_{\pi}}{\partial k_{q,t}^{i}} \frac{\partial k_{q,t}^{i}}{\partial w_t^{i}} \frac{\partial w_t^{i}}{\partial G_{\mathrm{gate}}} \frac{\partial G_{\mathrm{gate}}}{\partial\omega_{\mathrm{gate}}}=\delta_{k,q,t}^{i}k_{q,t}^{i} \left(\Delta_{\mathrm{adv},t}^{i}-\Delta_{\mathrm{dyn},t}^{i}\right)g_{V,t}^{i}\sigma\left(G_{\mathrm{gate}}(z_t^{i})\right) \left[1-\sigma\left(G_{\mathrm{gate}}(z_t^{i})\right)\right] \frac{\partial G_{\mathrm{gate}}(z_t^{i})}{\partial\omega_{\mathrm{gate}}}. \end{aligned}$$ After each update, the updated network parameters generate new modulation scales for subsequent policy updates. As for Eq. (19), we use it to encourage smooth modulation and constrain deviations from the base scale, rather than to learn task-beneficial modulation through this regularization term alone. The modulation networks are optimized using the overall MAPPO loss in Eq. (21), which includes the policy loss, the centralized value loss, the policy entropy term, and regularization terms for receptive-field widths and firing rates, in addition to the regularization term for surrogate-gradient scales. Q2. Inference accounting and encoder cost During deployment, population encoding uses the fixed Gaussian receptive-field centers and widths generated during training. In other words, the hypernetwork is used only during training and is not executed during inference. The action-output layer requires $256\times9=2304\approx2.3\mathrm{K}$ dense MACs per agent per environment step. The ANN baseline is counted on the same per-agent, per-environment-step basis, providing a consistent inference unit for a fair comparison. The encoding cost was already included in our original energy estimates. Gaussian population-response computation is accounted for separately, while the spike-driven input projection is included in the SOP counts. These encoding costs are included in the table’s energy estimates, rather than in the 2.3K MACs, which account only for the action-output layer. Moreover, following Reviewer Yhkj’s suggestion, we have extended the original MAC/SOP-based energy estimate to component-wise accounting that also includes Gaussian population-response computation and LIF state updates. The updated estimates are consistent with the original ones, with only negligible differences, and show (Table 2) energy reductions of 50.08%, 62.23%, and 74.16% on SMAC 3m, GRF counterattack (easy), and MPE spread, respectively. This result continues to support the energy-efficiency advantage of Spike-BiST.
AuthorsOpenReview ↗

Response to Reviewer 8qb2(W1-W2)

W1. Novelty relative to PopSAN and adaptive surrogate gradients

PopSAN [1] uses learnable Gaussian receptive fields to encode observations into spike representations for deep RL. However, it does not consider cross-dimensional dependencies or constrain receptive-field ordering, which can compromise representation consistency. The adaptive surrogate gradients in [2] do not account for heterogeneous learning signals and spiking dynamics across agents, which limits individualized gradient modulation under shared team rewards. We address these limitations through a bidirectional stabilization framework that jointly adapts population encoding and surrogate-gradient modulation to stabilize spike representations and gradient propagation in cooperative MARL.

W2. Meaning and evidence of improved stability

When we refer to “improved stability,” we mean ordered and temporally consistent spike representations and smooth changes in surrogate-gradient scales. Bounded center shifts and EMA–STE preserve receptive-field order and smooth updates to encoding parameters. The regularizer in Eq. (19) penalizes abrupt scale changes and excessive deviations from the shared base scale. In the ablations, we observe disrupted receptive-field ordering and greater scale fluctuations when the corresponding mechanisms are removed, alongside the performance decline shown in Fig. 3(b,c). These observations support the role of these mechanisms in maintaining representation and gradient stability.

Thank you again for your time and constructive feedback. As the discussion period approaches its end, we would like to kindly follow up on our response. We would be very grateful for any feedback on whether our clarifications address your concerns and whether any questions remain. We are happy to provide further clarification and continue the discussion.

Reviewer_YhkjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Spike-BiST to improve representation and training stability in spiking policies for multi-agent reinforcement learning. Built on MAPPO, Spike-BiST contains two main components. First, a hypernetwork dynamically adjusts the centers and widths of population receptive fields according to local observations and agent identities, while ordering constraints and temporal smoothing stabilize the resulting representations. Second, the surrogate-gradient scale is regulated using team returns, agent-specific advantages, spiking dynamics, and critic reliability. Experiments on SMAC, GRF, and MPE show competitive performance across most evaluated tasks. Component ablations further examine the proposed designs, and inference energy is estimated through operation counting.

Critical strengths

1. The method is well motivated and makes targeted adaptations for the multi-agent setting. The paper extends learnable population encoding and adaptive surrogate gradients to spiking multi-agent reinforcement learning. It further introduces observation- and identity-conditioned encoding, team- and agent-level modulation, and reliability-aware fusion. These designs address the discrepancy between shared policy optimization and agent-specific experience, constituting a reasonable methodological contribution. The paper is also clearly organized and easy to follow.

2. The empirical evaluation is broad and demonstrates practical effectiveness. The paper evaluates Spike-BiST on 15 SMAC, 7 GRF, and 3 MPE tasks using multiple random seeds and comparisons with both spiking and conventional MARL methods. Spike-BiST is competitive on most evaluated tasks and substantially improves over SNN-MAPPO in several cases. The component ablations also provide initial evidence for the contributions of the main designs.

Critical weaknesses

1. The inference-energy analysis appears to omit the adaptive encoder and other required computations. The operation counts in Table 1 appear to cover only part of the complete Spike-BiST inference path. Appendix A.1 defines the encoder hypernetwork as Linear(D+1,64) → ReLU → Linear(64,2DP). For SMAC 3m, where D=48 and P=8, these two linear layers alone require 49 × 64 + 64 × 768 = 52,288 dense MACs per forward pass, whereas Table 1 reports only 2.3K dense MACs for the entire Spike-BiST policy. Because the hypernetwork generates receptive-field parameters conditioned on the current observation, it remains part of the inference path after training. If this computation is excluded, the reported energy savings characterize only part of the network and do not support the claimed inference efficiency of the complete method.

The released code further supports this concern. HyperPopulationEncoder is executed during every actor forward pass, and its continuous Gaussian population responses are projected to the first LIF layer through Linear(DP,64). However, the built-in energy counter uses the firing rate of the first LIF layer to count only the spike-driven SOPs associated with the hidden projection, without completely accounting for the hypernetwork, continuous input projection, or action-output layer. Under the released implementation, these three dense components alone require at least 77,440 MACs per agent on SMAC 3m. At 4.6 pJ/MAC, this corresponds to 0.3562 μJ, already exceeding the total Spike-BiST energy of 0.3546 μJ reported in Table 1. Complete inference additionally requires the population-response computation and spike-driven synaptic operations within the LIF policy network. Therefore, the operation counts in Table 1 are inconsistent with the end-to-end inference path in the released code.

2. Experimental design

2.1 Limited ablation scope. The full component ablations are reported only on SMAC 6h_vs_8z, while the supplementary MuJoCo study evaluates only the encoder. It therefore remains unclear whether the component-level gains generalize to GRF or MPE.

2.2 Missing ANN-MAPPO performance baseline. EMU and ICES do not replace a direct comparison with the ANN version of the underlying MAPPO framework. Since Table 1 reports ANN energy without its task performance, the performance–efficiency trade-off cannot be assessed directly.

2.3 Insufficient comparison with recent SNN-RL work. The manuscript does not sufficiently compare with Proxy Target [1], SATR [2], and SpikeVLA [3]. These methods respectively address SNN-specific continuous-control training, RSNN optimization stability, and population-coded spiking policies. The authors should clarify how Spike-BiST relates to and differs from these works.

3. Code issue: the SMAC entry point initializes the proposed modules with a single agent. train_smac.py computes the map-specific num_agents but does not propagate it to all_args.num_agents. The actor, adaptive encoder, and GH-SGM therefore default to num_agents=1, while PPO supplies multiple agent IDs. This mismatch can cause dimension or indexing errors, so the released entry point does not directly reproduce the reported SMAC experiments.

Questions for authors

1. Missing components in the energy accounting. Please clarify exactly which computation components are included in Table 1 and reconcile the reported operation counts with the complete inference path. Please provide a module-by-module breakdown of dense MACs and spike-driven SOPs, explicitly covering the hypernetwork, population-response computation, continuous input projection, LIF policy network, and action-output layer. Please also specify the unit of one inference, the architecture of the ANN comparison, and the code or procedure used to generate Table 1. In particular, how can the reported total of 2.3K dense MACs be reconciled with the lower bound of 77,440 dense MACs in the released implementation? If complete accounting supports the reported savings, please provide it; otherwise, please correct the estimates or restrict the efficiency claim to the portion of the network that was actually counted.

2. Additional experiments.

2.1 Could the authors provide the main component ablations on another cooperative task, preferably from GRF or MPE, to determine whether the component-level gains extend beyond the current single SMAC task?

2.2 Could the authors include a conventional ANN-MAPPO baseline under comparable training and evaluation budgets and report the task performance of the ANN used in Table 1, so that the complete performance–efficiency trade-off can be assessed?

2.3 Please discuss and clarify how Spike-BiST relates to Proxy Target [1], SATR [2], and SpikeVLA [3], particularly regarding population encoding, training or gradient stabilization, and energy-efficient inference.

3. SMAC implementation and reproducibility. Were the reported SMAC experiments conducted using the currently released code? The standard train_smac.py entry point does not pass the actual num_agents value to the actor, dynamic encoder, or GH-SGM. How were these modules initialized with the correct number of agents in the reported experiments? Please provide the exact configurations and commands used to generate the results, together with the corresponding corrected code.

If the authors satisfactorily clarify these issues and provide convincing supporting evidence in the rebuttal, I would be willing to raise my rating accordingly.

References

[1] Zijie Xu, Tong Bu, Zecheng Hao, Jianhao Ding, and Zhaofei Yu. “Proxy Target: Bridging the Gap Between Discrete Spiking Neural Networks and Continuous Control.” Advances in Neural Information Processing Systems 38, 2025.

[2] Jinhao Li, Yuhao Sun, Zhiyuan Ma, Hao He, Xinche Zhang, Xing Chen, Jin Li, and Sen Song. “Signal-Adaptive Trust Regions for Gradient-Free Optimization of Recurrent Spiking Neural Networks.” arXiv:2601.21572, 2026. Accepted at NeurIPS 2026.

[3] Ruiqi Song, Dujun Nie, Siyu Teng, Baiyong Ding, Xiaotong Zhang, Dong Li, Chenming Zhang, Yuchen Li, Hangbin Wu, and Long Chen. “SpikeVLA: Vision-Language-Action Models with Spiking Neural Networks.” Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:114844–114859, 2026.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Yhkj(Q1/W1-Q2.1/W2.1)

We sincerely thank you for your careful review and constructive feedback. We organize our responses into two sections, addressing your three questions (Q1–Q3) and three concerns (W1–W3). Revisions are marked in blue in the revised PDF.

Q1/W1. Complete inference energy accounting

Following your suggestion, we have added a component-wise energy breakdown of Spike-BiST during inference to the revised manuscript. The results are presented in the Table below.

ComponentSMAC 3mGRF counterattack (easy)MPE spread
Hypernetwork (bypassed)0 μJ0 μJ0 μJ
Gaussian encoding (D→8D)(D\to 8D)0.004620 μJ0.002887 μJ0.001732 μJ
Input projection (8D→256,SOP)(8D\to 256,\mathrm{SOP})0.192428 μJ0.108753 μJ0.058763 μJ
Hidden projection (256→256,SOP)(256\to 256,\mathrm{SOP})0.087427 μJ0.082400 μJ0.067572 μJ
LIF updates (DP+256+256)(DP+256+256)0.039424 μJ0.033088 μJ0.028864 μJ
Action head (256→A,MAC)(256\to A,\mathrm{MAC})0.010598 μJ0.022374 μJ0.005888 μJ
Spike-BiST subtotal0.334497 μJ0.249503 μJ0.162820 μJ
ANN (D→256→256→256→A,MAC)(D\to 256\to 256\to 256\to A,\mathrm{MAC})0.670054 μJ0.660634 μJ0.630016 μJ
Estimated energy reduction50.08%62.23%74.16%

For these estimates, one inference corresponds to one action decision by one agent per environment step. Each agent’s local observation is first processed by the Gaussian population encoder, followed by input projection and spike propagation through the hidden layers. LIF neurons update their states over eight internal SNN timesteps, after which the action-output layer is computed once. The hypernetwork is executed only during training to generate adaptive Gaussian receptive-field centers and widths. During deployment, these parameters are fixed and directly loaded for Gaussian population encoding, without executing the hypernetwork. Therefore, its 52.3K MACs are excluded from inference-time energy accounting. The input projection uses binary spike inputs and is counted as spike-driven SOPs, together with the hidden-layer projections, while the action-output layer is counted as dense MACs. The costs of Gaussian population-response computation and LIF neuron state updates are included separately in this table.

Regarding the discrepancy between the reported 2.3K dense MACs and the 77,440 dense MACs identified in the released implementation, the originally reported 2.3K MACs (256×9256\times9) account only for the action-output layer in SMAC 3m. As explained above, the hypernetwork is bypassed during deployment, and the input projection is accounted for as spike-driven SOPs rather than dense MACs.

We would also like to clarify the differences between the revised estimates and those in our original submission. The original estimates followed a commonly used MAC/SOP-based accounting approach [Tang et al. (2021), Qin et al. (2023), Rathi and Roy (2023) ], which did not include the costs of Gaussian population encoding and LIF neuron state updates. In the revised manuscript, following your suggestion, we have included these two components in the energy accounting and updated the estimates accordingly. The updated results show that Spike-BiST achieves estimated energy reductions of 50.08%, 62.23%, and 74.16% on SMAC 3m, GRF counterattack (easy), and MPE spread, respectively, retaining its energy-efficiency advantage under the revised accounting.

Q2.1/W2.1. Component ablations on GRF and MPE

Following your suggestion, we have extended the component ablations to GRF(3vs1 wth keeper) and MPE(spread reference). We have compared the full model with variants that disable adaptive encoding, surrogate-gradient modulation, or both. The results provide additional evidence for the contributions of individual components beyond the original SMAC scenario. These results are also presented in the main text of the revised manuscript (Table 13-15).

VariantGRF 3vs1with keeperMPE reference
Spike-BiST72.7 ± 11.5-21.8 ± 2.1
w/o dual-channel encoding45.6 ± 17.6-24.3 ± 3.0
w/o reliability-driven SGM35.6 ± 33.5-25.8 ± 2.1
w/o both components7.5 ± 12.2-27.2 ± 0.7
VariantGRF 3vs1with keeperMPE reference
Spike-BiST72.7 ± 11.5-21.8 ± 2.1
w/o neuron ordering constraint60.0 ± 36.6-23.2 ± 1.7
w/o EMA-STE mechanism44.4 ± 27.0-25.0 ± 2.3
Fixed encoding45.6 ± 17.6-24.3 ± 3.0
VariantGRF 3vs1with keeperMPE reference
Spike-BiST72.7 ± 11.5-21.8 ± 2.1
w/o advantage modulation52.3 ± 30.1-24.0 ± 2.3
w/o spiking modulation56.3 ± 22.8-23.2 ± 1.0
w/o both modulations48.3 ± 25.4-24.8 ± 3.1
Fixed surrogate gradient35.6 ± 33.5-25.8 ± 2.1
AuthorsOpenReview ↗

Response to Reviewer Yhkj(Q2.2/W2.2-Q3/W3)

Q2.2/W2.2. ANN-MAPPO performance and efficiency

Following your suggestion, we have added an ANN-MAPPO baseline under matched training and evaluation budgets. We have also reported the task performance of the ANN policy used in Table 1, enabling a joint assessment of task performance and energy efficiency. Spike-BiST outperforms ANN-MAPPO in 18 out of 25 scenarios, ties for first place in 4, and achieves the second-best performance in the remaining 3. These results are also presented in the main text of the revised manuscript (Tables 11-13).

TaskSpike-BiSTANN-MAPPO
3m100.0 ± 0.0100.0 ± 0.0
8m100.0 ± 0.0100.0 ± 0.0
2s3z100.0 ± 0.0100.0 ± 0.0
1c3s5z98.5 ± 2.0100.0 ± 0.0
25m100.0 ± 0.085.7 ± 7.1
3s5z98.1 ± 3.896.7 ± 0.7
5m_vs_6m85.9 ± 4.074.3 ± 5.9
8m_vs_9m98.1 ± 1.784.6 ± 1.5
10m_vs_11m100.0 ± 0.096.9 ± 6.6
2c_vs_64zg93.0 ± 5.991.3 ± 10.4
3s_vs_5z100.0 ± 0.0100.0 ± 0.0
MMM290.0 ± 4.285.3 ± 6.0
3s5z_vs_3s6z81.3 ± 12.184.4 ± 21.4
27m_vs_30m99.6 ± 1.291.4 ± 15.2
6h_vs_8z69.3 ± 7.878.5 ± 8.6
3vs1 (keeper)72.7 ± 11.568.8 ± 13.4
Counterattack (easy)64.0 ± 16.559.4 ± 13.8
Counterattack (hard)49.8 ± 10.738.8 ± 11.4
Pass and shoot93.5 ± 2.782.4 ± 6.3
Run, pass and shoot69.1 ± 2.558.7 ± 8.2
Run to score92.2 ± 6.388.8 ± 7.4
Run to score (keeper)96.2 ± 4.492.4 ± 3.2
Reference-21.2 ± 1.2-23.2 ± 0.8
Speaker-listener-13.0 ± 6.8-15.2 ± 1.3
Spread-112.8 ± 18.1-122.4 ± 9.3

Q2.3/W2.3. Relation to Proxy Target, SATR, and SpikeVLA

We have added a discussion of Proxy Target [1], SATR [2], and SpikeVLA [3] to the related-work section of the revised manuscript. For single-agent settings, Proxy Target [1] stabilizes spiking RL training through a differentiable proxy network, SATR [2] improves gradient-free optimization of recurrent SNNs through signal-adaptive trust regions, and SpikeVLA [3] applies population coding to vision-language-action models for energy-efficient embodied tasks. In contrast, our work focuses on multi-agent settings, jointly adapting population encoding and surrogate gradient modulation to address representation and gradient instability under concurrent policy updates across agents.

Q3/W3. SMAC implementation and reproducibility

We have checked the code and confirmed that the map-specific agent count is correctly propagated and used to initialize the relevant modules. In the inspected code snapshot, train_smac.py assigns the map-specific agent count to all_args.n_agents before constructing the runner. The actor’s SNNBase prioritizes this field and passes the count to the dynamic encoder and agent-level SGM. Separately, the runner explicitly passes num_agents to the trainer to initialize the SGM scheduler. Thus, although all_args.num_agents is not separately assigned, these modules receive the correct agent count through the above paths and do not default to a single-agent configuration.

I thank the authors for the additional experiments and analysis. These have addressed my concerns well. I’m happy to raise my rating by a point.

Reviewer_bTToOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes Spike-BiST, a bidirectional stabilization framework for spiking multi-agent reinforcement learning (MARL). The method consists of two components: a dual-channel adaptive population encoder that dynamically adjusts receptive fields while maintaining structural and temporal consistency, and a reliability-driven surrogate gradient modulation mechanism that adjusts surrogate gradient scales using team-level learning progress, agent-specific advantages, spiking dynamics, and critic confidence. The proposed method is implemented within MAPPO and evaluated on SMAC, GRF, and MPE.

Critical strengths
  1. The experiments cover multiple cooperative MARL benchmarks, including SMAC, GRF, and MPE. Spike-BiST achieves competitive performance on many challenging tasks, and the ablation studies show that both proposed components contribute to the improvements.

  2. The method maintains the potential energy-efficiency advantages of SNNs, with lower estimated inference energy than ANN-based policies in the reported settings.

Critical weaknesses
  1. The related work section omits several representative and recent SNN-RL studies, such as [1–3].

  2. The two main components appear closely related to existing techniques. For adaptive encoding, [4] has already introduced learnable spike encoding, and [5] employs population coding with trainable Gaussian receptive-field parameters. For surrogate gradient modulation, [6] has already investigated adaptive surrogate gradient slopes in sequential RL.

  3. The motivation behind both components is not fully convincing. First, the paper emphasizes the limitations of fixed population encoding. However, learnable encoding mechanisms have already been adopted in existing SNN-RL methods without requiring the proposed hypernetwork-based adaptation [2],[5],[7] . Second, the paper attributes training instability to surrogate gradient propagation, but it remains unclear whether a properly tuned fixed surrogate gradient [1],[2],[4],[5],[7] or a simpler predefined slope schedule [6] could achieve comparable performance.

  4. The paper motivates the proposed framework through the non-stationarity induced by concurrent policy updates across multiple agents. However, both observation distribution shifts and surrogate gradient instability can also arise in conventional single-agent reinforcement learning, or even in supervised SNN training.

  5. Appendix A.3.2 evaluates adaptive encoding on Hopper and Walker2d. However, the reported rewards appear surprisingly low compared with commonly reported results from established SNN-RL methods [2],[5],[7].

References

[1] Spiking World Model with Multicompartment Neurons for Model-Based Reinforcement Learning.

[2] CaRe-BN: Precise Moving Statistics for Stabilizing Spiking Neural Networks in Reinforcement Learning.

[3] Signal-Adaptive Trust Regions for Gradient-Free Optimization of Recurrent Spiking Neural Networks.

[4] A Low Latency Adaptive Coding Spike Framework for Deep Reinforcement Learning.

[5] Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control.

[6] Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks.

[7] Fully Spiking Actor Network with Intralayer Connections for Reinforcement Learning.

Questions for authors
  1. Can the authors include stronger SNN-RL baselines under the same MARL framework? The current experiments include an adaptation of ILC-SAN to MAPPO. However, several recent SNN-RL techniques may also be incorporated into MAPPO, such as [2],[4],[6].

  2. Could the authors provide performance comparisons with ANN-MAPPO, ANN-to-SNN conversion, and ANN-to-SNN policy distillation? (Spike-BiST does not necessarily need to outperform these methods.)

  3. Can the authors provide more comprehensive experiments beyond MARL, especially in single-agent reinforcement learning? I also suggest to put them into the main text instead of the Appendix.

4 comments
AuthorsOpenReview ↗

Response to Reviewer bTTo (Q1–Q3)

We sincerely thank you for your careful review and constructive feedback. We organize our responses into two sections, addressing your three questions (Q1–Q3) and five concerns (W1–W5). Revisions are marked in blue in the revised PDF.

Q1. Stronger SNN-RL baselines

Following your suggestion, due to time constraints, we selected two methods, CaRe-BN [2] and ASG [6], for comparison. The experimental results show that Spike-BiST outperforms both methods, and are also reported in Tables 11-13 of the revised manuscript.

TaskSpike-BiSTCaRe-BN-MAPPO [2]ASG-MAPPO [6]
3m100.0 ± 0.098.2 ± 2.2100.0 ± 0.0
8m100.0 ± 0.0100.0 ± 0.096.8 ± 0.0
2s3z100.0 ± 0.096.8 ± 3.197.9 ± 3.6
1c3s5z98.5 ± 2.080.2 ± 4.675.0 ± 6.2
25m100.0 ± 0.072.8 ± 8.488.3 ± 7.8
3s5z98.1 ± 3.893.4 ± 3.295.4 ± 4.6
5m_vs_6m85.9 ± 4.065.3 ± 10.272.4 ± 6.7
8m_vs_9m98.1 ± 1.777.2 ± 7.882.5 ± 4.4
10m_vs_11m100.0 ± 0.078.1 ± 12.483.6 ± 8.1
2c_vs_64zg93.0 ± 5.984.8 ± 7.662.5 ± 13.5
3s_vs_5z100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0
MMM290.0 ± 4.256.3 ± 14.667.1 ± 10.2
3s5z_vs_3s6z81.3 ± 12.140.6 ± 22.150.6 ± 17.7
27m_vs_30m99.6 ± 1.297.0 ± 3.895.4 ± 2.9
6h_vs_8z69.3 ± 7.840.6 ± 15.353.3 ± 7.6
3vs1 (keeper)72.7 ± 11.568.1 ± 7.459.5 ± 11.2
Counterattack (easy)64.0 ± 16.557.3 ± 9.846.9 ± 14.2
Counterattack (hard)49.8 ± 10.75.8 ± 5.432.4 ± 8.6
Pass and shoot93.5 ± 2.786.4 ± 5.872.4 ± 10.2
Run, pass and shoot69.1 ± 2.555.4 ± 4.857.8 ± 7.6
Run to score92.2 ± 6.384.3 ± 9.489.2 ± 7.2
Run to score (keeper)96.2 ± 4.487.2 ± 6.875.9 ± 2.2
Reference-21.2 ± 1.2-28.0 ± 2.3-27.3 ± 1.6
Speaker-listener-13.0 ± 6.8-16.9 ± 0.5-17.2 ± 0.7
Spread-112.8 ± 18.1-132.4 ± 12.9-125.4 ± 10.9

Q2. ANN-MAPPO, ANN-to-SNN conversion, and policy distillation

Following your suggestion, we have compared Spike-BiST with ANN-MAPPO, ANN-to-SNN conversion, and ANN-to-SNN policy distillation. Across 25 scenarios, Spike-BiST outperforms all comparison methods in 18, ties for first in 4, and achieves the second-best performance in the remaining 3. The results are presented in the revised manuscript’s main text (Tables 11–13).

TaskSpike-BiSTANN-MAPPOANN-to-SNNANN-to-SNN(distillation)
3m100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0
8m100.0 ± 0.0100.0 ± 0.0100.0 ± 0.096.8 ± 5.4
2s3z100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0
1c3s5z98.5 ± 2.0100.0 ± 0.095.2 ± 2.787.8 ± 2.8
25m100.0 ± 0.085.7 ± 7.182.4 ± 9.278.9 ± 12.2
3s5z98.1 ± 3.896.7 ± 0.773.6 ± 5.884.7 ± 10.6
5m_vs_6m85.9 ± 4.074.3 ± 5.971.2 ± 3.562.5 ± 10.4
8m_vs_9m98.1 ± 1.784.6 ± 1.584.3 ± 2.172.5 ± 5.6
10m_vs_11m100.0 ± 0.096.9 ± 6.661.4 ± 7.270.8 ± 9.0
2c_vs_64zg93.0 ± 5.991.3 ± 10.488.2 ± 8.477.8 ± 12.7
3s_vs_5z100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0
MMM290.0 ± 4.285.3 ± 6.076.5 ± 4.978.6 ± 8.5
3s5z_vs_3s6z81.3 ± 12.184.4 ± 21.463.4 ± 14.272.5 ± 9.4
27m_vs_30m99.6 ± 1.291.4 ± 15.277.4 ± 14.882.3 ± 12.7
6h_vs_8z69.3 ± 7.878.5 ± 8.653.1 ± 10.865.0 ± 12.4
3vs1 (keeper)72.7 ± 11.568.8 ± 13.452.4 ± 7.844.7 ± 15.1
Counterattack (easy)64.0 ± 16.559.4 ± 13.855.0 ± 14.941.2 ± 18.9
Counterattack (hard)49.8 ± 10.738.8 ± 11.432.5 ± 6.626.8 ± 10.1
Pass and shoot93.5 ± 2.782.4 ± 6.381.2 ± 8.762.5 ± 13.2
Run, pass and shoot69.1 ± 2.558.7 ± 8.228.8 ± 12.843.2 ± 7.6
Run to score92.2 ± 6.388.8 ± 7.484.7 ± 1.882.6 ± 5.8
Run to score (keeper)96.2 ± 4.492.4 ± 3.275.0 ± 12.188.5 ± 7.2
Reference-21.2 ± 1.2-23.2 ± 0.8-24.4 ± 1.2-29.2 ± 1.4
Speaker-listener-13.0 ± 6.8-15.2 ± 1.3-23.1 ± 6.9-18.7 ± 4.2
Spread-112.8 ± 18.1-122.4 ± 9.3-125.6 ± 7.2-142.1 ± 12.2

Q3. Experiments beyond MARL

We have conducted experiments on 4 single-agent tasks: Hopper, Walker2d, HalfCheetah, and Ant. Our method outperforms the comparison methods, indicating that its benefits extend beyond MARL to single-agent continuous control. These results are presented in the main text of the revised manuscript (Table 1).

TasksHalfCheetahAntHopperWalker2d
Dual-channel adaptive population encoding10832.5 ± 216.35564.5 ± 574.73524.0 ± 240.44690.6 ± 168.7
Adaptive encoding10671.8 ± 194.95022.5 ± 1156.13432.6 ± 514.83908.9 ± 448.1
Fixed encoding9824.61 ± 438.43478.1 ± 1521.33362.7 ± 504.44039.4 ± 143.7
AuthorsOpenReview ↗

Response to Reviewer bTTo (W1–W5)

W1. Coverage of recent SNN-RL studies

Thank you for introducing these relevant studies. We have added references [1–3] and the corresponding discussion to the related work section on page 2 of the revised manuscript.

W2. Novelty of adaptive encoding and surrogate gradient modulation

For encoding, [4] and [5] perform dimension-wise encoding but do not model dependencies among observation dimensions. Each dimension is optimized independently, which overlooks correlations between different dimensions and limits the ability to capture cross-dimensional contextual information. Moreover, [5] lacks an explicit ordering constraint, which causes receptive-field centers to cross during optimization and disrupts the predefined population coding order. To address these limitations, our encoder jointly generates receptive-field adjustments for all dimensions based on each agent's full local observation and identity, so that each dimension's encoding incorporates information from other dimensions. We also introduce bounded, order-preserving center shifts to maintain the ordering of receptive-field centers. The ablation results in Fig. 3(b) further demonstrate the advantages of our encoding design.

For surrogate gradients, [6] introduces an adaptive slope schedule for single-agent RL, which adjusts the slope according to rewards and reward changes to track learning progress. However, cooperative MARL provides only team rewards rather than individual rewards, which makes it fail to capture the heterogeneous learning signals and spiking dynamics of individual agents. To address this issue, we combine a team-level learning progress indicator with agent-specific advantages and spiking dynamics to derive individualized surrogate gradient modulation.

W3. Motivation and comparison with simpler alternatives

For encoding, [2], [5], and [7] learn encoding parameters to improve spike representation. However, they do not address cross-dimensional context, receptive-field ordering, and temporal consistency. We use each agent’s full local observation and identity to generate receptive-field adjustments, with bounded center shifts and EMA-STE to preserve ordering and smooth parameter changes. The ablation results in Fig. 3(b) further demonstrate the advantages of our encoding design.

For surrogate gradient, we have conducted ablation experiments comparing our method with fixed surrogate gradients [1], [2], [4], [5], and [7]. Fixed surrogate-gradient scales cannot adapt to changes in training progress or agent-specific learning signals. As shown in Fig. 3(c), our gradient modulation achieves better performance. Due to time constraints, we have not completed the comparison with the predefined slope schedule in [6]. Nevertheless, [6] is designed for single-agent RL, whereas cooperative MARL involves heterogeneous learning signals and spiking dynamics across agents. To address this issue, our method combines team-level learning progress with agent-specific advantages and spiking dynamics to achieve individualized surrogate gradient modulation.

W4. Non-stationarity in multi-agent learning

In our initial submission (Para 2, Sec 1), we had mentioned the non-stationarity issue in both conventional single-agent RL and supervised SNN training. We attempted to highlight that concurrent policy updates across agents further exacerbate non-stationarity in MARL, leading to coupled changes in local observation distributions and learning signals, rather than denying the presence of these challenges in single-agent RL or supervised SNN training.

W5. Return reporting in the single-agent experiments

The “Returns” originally reported in Appendix A.3.2 represent the mean training return per timestep, whereas [2], [5], and [7] report episodic evaluation returns. Therefore, it is natural to observe differences between the reported values and those of [2], [5], [7]. In the revised manuscript, we have used the mean episodic evaluation return and updated Fig. 5 accordingly. The updated results show that our method achieves higher episodic evaluation returns than those reported in [2], [5], and [7].

Thank you again for your time and constructive feedback. As the discussion period approaches its end, we would like to kindly follow up on our response. We would be very grateful for any feedback on whether our clarifications address your concerns and whether any questions remain. We are happy to provide further clarification and continue the discussion.

Thank you for the detailed response and additional experiments. Most of my concerns have been addressed. Although the authors argue that concurrent policy updates among agents exacerbate non-stationarity in SNN-based MARL, I remain unconvinced that the proposed problem formulation and methods adequately address the distinctive challenges of MARL. I also think this work is similar to and incremental over several existing works such as the adaptive surrogate gradient [1] and trainable Gaussian encodings [2,3]. I will therefore keep my initial rating.

[1] Korneel Van den Berghe, Stein Stroobants, Vijay Janapa Reddi, and G. C. H. E. de Croon. Adaptive surrogate gradients for sequential reinforcement learning in spiking neural networks. In 39th Conference on Neural Information Processing Systems, pp. 147904–147926, San Diego, USA, December 2025.

[2] Guangzhi Tang, Neelesh Kumar, Raymond Yoo, and Konstantinos P. Michmizos. Deep reinforcement learning with population-coded spiking neural network for continuous control. In 4th Conference on Robot Learning, pp. 2016–2029, London, UK, November 2021.

[3] Duzhen Zhang, Tielin Zhang, Shuncheng Jia, and Bo Xu. Multi-scale dynamic coding improved spiking actor network for reinforcement learning. In 36th AAAI Conference on Artificial Intelligence, pp. 59–67, Washington, USA, February 2022.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.