- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Spike-BiST, a method that enables spiking neural networks to learn cooperation more stably in multi-agent reinforcement learning. To address input variations and gradient fluctuations arising from simultaneous learning by multiple agents, the method dynamically adjusts and smooths the encoding of observational information while modulating backpropagation based on a combination of team performance, individual learning signals, and neuronal states. Experiments conducted in StarCraft, soccer, and particle environments demonstrate that the method improves cooperative performance and training stability while maintaining low energy consumption for inference.
Critical strengths
The experimental coverage is solid, and the results are competitive.
Critical weaknesses
-
The paper cites [1] and [2], but the novelty relative to them is limited: PopSAN [1] already applies population coding to deep RL, and [2] already studies adaptive surrogate gradients for spiking RL.
-
The core claim of "improved stability" lacks direct evidence. The central claim of improved training stability is not directly measured. The paper reports final win rates/returns with 5 seeds and 95% CIs, and Fig. 3 ablates the two components; these show that both components contribute to performance, but they do not show that the gains are due to improved stability.
[1] Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control.
[2] Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks.
Questions for authors
-
Equations (14)–(17) define the agent-specific surrogate-gradient scale . This scale does not affect the forward policy outputs, since spikes are generated by the hard threshold. It enters the surrogate derivative in Equation (3). Further explanation is needed regarding how the MAPPO loss updates the modulation networks and . While the regularization term in Equation (19) provides gradients, it primarily encourages smoothness and proximity to the base scale, without automatically demonstrating that the network has learned a task-beneficial modulation strategy. Could the authors elaborate on this aspect?
-
Appendix A.1 specifies the hypernetwork as . For SMAC 3m , this is MACs per forward pass per agent. Table 1, however, reports only 2.3K dense MACs for Spike-BiST. Please clarify whether the 2.3K is per SNN timestep or per environment step (Ts=8), and whether it is per agent or per team; otherwise the comparison is not apples-to-apples. In either case, the table does not appear to include the encoder’s cost, so the reported energy saving should be recomputed with the encoder included.
6 comments
Response to Reviewer 8qb2(W1-W2)
W1. Novelty relative to PopSAN and adaptive surrogate gradients
PopSAN [1] uses learnable Gaussian receptive fields to encode observations into spike representations for deep RL. However, it does not consider cross-dimensional dependencies or constrain receptive-field ordering, which can compromise representation consistency. The adaptive surrogate gradients in [2] do not account for heterogeneous learning signals and spiking dynamics across agents, which limits individualized gradient modulation under shared team rewards. We address these limitations through a bidirectional stabilization framework that jointly adapts population encoding and surrogate-gradient modulation to stabilize spike representations and gradient propagation in cooperative MARL.
W2. Meaning and evidence of improved stability
When we refer to “improved stability,” we mean ordered and temporally consistent spike representations and smooth changes in surrogate-gradient scales. Bounded center shifts and EMA–STE preserve receptive-field order and smooth updates to encoding parameters. The regularizer in Eq. (19) penalizes abrupt scale changes and excessive deviations from the shared base scale. In the ablations, we observe disrupted receptive-field ordering and greater scale fluctuations when the corresponding mechanisms are removed, alongside the performance decline shown in Fig. 3(b,c). These observations support the role of these mechanisms in maintaining representation and gradient stability.
Thank you again for your time and constructive feedback. As the discussion period approaches its end, we would like to kindly follow up on our response. We would be very grateful for any feedback on whether our clarifications address your concerns and whether any questions remain. We are happy to provide further clarification and continue the discussion.
Response to Reviewer 8qb2(Q1-Q2)
Q1. MAPPO loss and modulation-network updates
Both modulation networks Gdyn and Ggate are updated using the MAPPO policy loss, with the modulation scale kq,ti participating in the parameter updates. During each policy update, the current network parameters generate kq,ti, which modulates the surrogate gradient during backpropagation. Let δk,q,ti=∂Lπ/∂kq,ti denote the scale gradient returned by this operation. The expressions below give the gradient contributions of a single agent-step sample. Specifically, according to Eq. (17), the gradients with respect to ωdyn and ωgate are given by:
For Gdyn:
\frac{\partial\mathcal{L}_{\pi}}{\partial\omega_{\mathrm{dyn}}}=\frac{\partial\mathcal{L}_{\pi}}{\partial k_{q,t}^{i}} \frac{\partial k_{q,t}^{i}}{\partial\Delta_{\mathrm{dyn},t}^{i}} \frac{\partial\Delta_{\mathrm{dyn},t}^{i}}{\partial G_{\mathrm{dyn}}} \frac{\partial G_{\mathrm{dyn}}}{\partial\omega_{\mathrm{dyn}}}=\delta_{k,q,t}^{i}k_{q,t}^{i}(1-w_t^{i})a_1 \left[1-\tanh^2\left(G_{\mathrm{dyn}}(z_t^{i})\right)\right] \frac{\partial G_{\mathrm{dyn}}(z_t^{i})}{\partial\omega_{\mathrm{dyn}}}. \end{aligned}$$ For $G_{\mathrm{gate}}$: $$\begin{aligned} \frac{\partial\mathcal{L}_{\pi}}{\partial\omega_{\mathrm{gate}}}=\frac{\partial\mathcal{L}_{\pi}}{\partial k_{q,t}^{i}} \frac{\partial k_{q,t}^{i}}{\partial w_t^{i}} \frac{\partial w_t^{i}}{\partial G_{\mathrm{gate}}} \frac{\partial G_{\mathrm{gate}}}{\partial\omega_{\mathrm{gate}}}=\delta_{k,q,t}^{i}k_{q,t}^{i} \left(\Delta_{\mathrm{adv},t}^{i}-\Delta_{\mathrm{dyn},t}^{i}\right)g_{V,t}^{i}\sigma\left(G_{\mathrm{gate}}(z_t^{i})\right) \left[1-\sigma\left(G_{\mathrm{gate}}(z_t^{i})\right)\right] \frac{\partial G_{\mathrm{gate}}(z_t^{i})}{\partial\omega_{\mathrm{gate}}}. \end{aligned}$$ After each update, the updated network parameters generate new modulation scales for subsequent policy updates. As for Eq. (19), we use it to encourage smooth modulation and constrain deviations from the base scale, rather than to learn task-beneficial modulation through this regularization term alone. The modulation networks are optimized using the overall MAPPO loss in Eq. (21), which includes the policy loss, the centralized value loss, the policy entropy term, and regularization terms for receptive-field widths and firing rates, in addition to the regularization term for surrogate-gradient scales. Q2. Inference accounting and encoder cost During deployment, population encoding uses the fixed Gaussian receptive-field centers and widths generated during training. In other words, the hypernetwork is used only during training and is not executed during inference. The action-output layer requires $256\times9=2304\approx2.3\mathrm{K}$ dense MACs per agent per environment step. The ANN baseline is counted on the same per-agent, per-environment-step basis, providing a consistent inference unit for a fair comparison. The encoding cost was already included in our original energy estimates. Gaussian population-response computation is accounted for separately, while the spike-driven input projection is included in the SOP counts. These encoding costs are included in the table’s energy estimates, rather than in the 2.3K MACs, which account only for the action-output layer. Moreover, following Reviewer Yhkj’s suggestion, we have extended the original MAC/SOP-based energy estimate to component-wise accounting that also includes Gaussian population-response computation and LIF state updates. The updated estimates are consistent with the original ones, with only negligible differences, and show (Table 2) energy reductions of 50.08%, 62.23%, and 74.16% on SMAC 3m, GRF counterattack (easy), and MPE spread, respectively. This result continues to support the energy-efficiency advantage of Spike-BiST.