- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors introduce asymmetric membrane credit (AMC) for “fully spiking” actor-critic reinforcement learning. This aims to cope with the limitations of event-only critics by using both discrete spikes and real-value membrane potential. This mechanism is integrated with a reusable stateful spiking substrate by adapting this across various RL algorithms like PPO, SAC, and TD3. AMC was subject to systematic ablation analysis to isolate the effect of each ingredient, event-only (E), detached (D), full AMC (A). Additionally, the trained actors were deployed on Lynxi HP201 neuromorphic hardware to validate its practical use.
Critical strengths
- Rigorous and systematic evaluation of AMC: The systematic ablation study can be helpful for the algorithm evaluation.
- Deployment on dedicated hardware: The authors validated the viability of AMC by deploying it on Lynxi HP201.
Critical weaknesses
- Missing motivation and justification: I do not read any motivation of this work other than some statements like event-only critic cannot read continuous membrane potential. The authors should have systematically addressed any practical challenges that the event-only critic holds.
- Detached control (D) undermining AMC: I found that the detached control (D) often substantially outperforms AMC like on SAC Walker. These results undermine the needs for AMC.
- Clarity: several equations are written without clear explanation of each term. In the first equation on Page 2, I don’t understand what the episode mask means. I found the same term in Appendix, which is a binary mask determined by “reset”. I don’t understand what this reset means. The related Appendix section (Appendix B) also includes a number of variables that are never explained like \rho, a_{max}, a_{min}, \sigma_p^V, \mu_p^V. Where does the actor use the variable o_t?
Questions for authors
- What specific failure is caused by event-only critics?
- How can the authors explain my comment 2 in Critical weaknesses?
4 comments
Dear Reviewer mGmT,
With the discussion period coming to a close soon, we wanted to briefly follow up to check if our response and manuscript revisions have clarified your questions, or if there are any remaining points you would like us to elaborate on. We remain actively available and will gladly address any further questions promptly.
Thank you again for your time and constructive feedback in reviewing our paper.
Response to authors
I appreciate the authors' effort on the rebuttal. Nevertheless, the motivation of this work (fully SNN-based system) is not fully justified, the authors show some advantages in a specific domain though. I therefore keep my original score.
Dear Reviewer mGmT,
Thank you for reading our rebuttal and recognizing the advantages demonstrated within the full-SNN setting. Since our earlier reply to W1 focused on why event-only readouts suffer from state aliasing rather than why the fully spiking Actor–Critic paradigm itself is worth studying, we briefly clarify that system-level motivation here:
- Toward self-contained neuromorphic learning. Hybrid SNN-Actor/ANN-Critic designs rely on a separate floating-point MLP Critic, which confines training to conventional off-chip hosts. Fully spiking Actor–Critics (e.g., Zanatta et al., 2024; Zhang et al., 2025) are studied as the algorithmic step toward self-contained neuromorphic adaptation, where both policy and value estimation operate on spiking primitives without maintaining a separate conventional DNN.
- Shared-representation efficiency even in off-chip training. Sharing one spiking core avoids maintaining a large separate Critic network while allowing value learning to shape the policy representation directly. With only 1,025 Critic-specific parameters—versus 5,121–179,201 in a width-matched one-layer ANN Critic (H512) and 68,353–155,393 in a two-layer ANN Critic (H256; Table F5)—MCrPPO outperforms the hybrid baseline PopSAN-v5 on all six tasks and exceeds width-matched H512 on three of six tasks (Table 5), while keeping Actor deployment strictly event-mediated with zero overhead.
Prior fully spiking RL defaulted to event-only value readouts without examining the subthreshold information lost at the spike threshold. Establishing how value estimation and encoder credit should be separated inside a shared spiking core (with membrane access improving final return in all 18/18 settings across PPO, SAC, and TD3) addresses a foundational bottleneck for this direction. We hope this clarifies the broader motivation of the paradigm, and we thank you again for your time and feedback.
Response to Reviewer mGmT
Thank you for the comments. Page numbers refer to the revised PDF.
W1 and Q1. The specific failure of an event-only Critic. The failure is state aliasing in value estimation. A hard threshold maps many different membrane states to the same events: in the recorded trace of Figure 1(a) (p. 3), every subthreshold value gives a zero event. An event-only Critic cannot tell these states apart, although they differ in accumulated input and in how close the neuron is to firing. When these differences predict return, the value estimate is wrong, and the error enters the PPO advantages and the bootstrapped targets of SAC and TD3. The second paragraph of the Introduction (p. 1) now states this. Two results confirm it. Giving the Critic the membrane readout raises mean final return in all 18 task–algorithm settings, for example from 507.9 to 1804.5 on SAC Walker (Table D4, p. 39). On frozen representations, a membrane readout predicts held-out return better than an equal-size event readout in 17 of 18 conditions for each Q head (Appendix E.4, p. 43).
W2 and Q2. D outperforming A does not undermine AMC. D is not a baseline without AMC. It reads the same membrane through the same trainable readout as A, and only blocks the extra gradient from that readout into the shared encoder (Table 1, p. 4). D above E is therefore direct evidence for the membrane readout.
What the comparison of A and D shows is that membrane access and direct credit are two different things. Access helps throughout (D–E is positive in all 18 settings), whereas the extra gradient helps on some tasks and hurts on others. In every one of the 18 settings, A or D improves both mean AUC and mean final return over the event-only control. The paper therefore offers both modes and names D as the practical choice where direct credit is harmful (Section 4.2, p. 7; Section 5, p. 10). Separating the two is a contribution of the work, not a weakness of AMC.
W3. Episode mask and undefined symbols. These are now defined in the main text and collected in a notation guide (Appendix A, pp. 15–17). The episode mask is mₜ = 1 − resetₜ, where resetₜ = 1 on the first observation of a new episode; it sets the recurrent state of that environment to zero and is different from the spike reset of a single neuron (Section 2.1, p. 3). The Actor uses oₜ as the mean of its Gaussian policy: the action is aₜ = c + s · tanh(oₜ + exp(ρ) · εₜ), where ρ is the learned log standard deviation and c and s are the midpoint and half-range of the task's action bounds a_min and a_max. Finally, μᵥ and σᵥ are the running mean and standard deviation of the returns, used to decode the value as Vₜ = σᵥ uₜ + μᵥ (Section 2.2, p. 4).