acceptodds
Under review as a conference paper at ICLR 2027

Asymmetric Membrane Credit for Spiking Actor–Critic Learning

Abstract

A stateful spiking core produces both binary events and continuous membrane states, but an event-only Critic does not directly read the latter. We therefore introduce Asymmetric Membrane Credit (AMC), a training-time interface that gives the Critic a zero-initialized readout of the shared pre-threshold membrane without adding an Actor inference branch. A matched detached control retains the trainable readout while blocking only its additional encoder-gradient route, separating membrane-informed Critic learning from direct encoder credit. A reusable stateful spiking Actor–Critic substrate implements AMC across PPO, SAC and TD3, with common policy computation and rule-specific state reconstruction and encoder updates. Across six MuJoCo-v5 tasks and three paired training seeds per setting, the complete AMC pathway achieves higher mean area under the learning curve (AUC) than event-only controls in 17 of 18 settings. Fixed final policies achieve higher mean return than the same controls in 16 settings when evaluated on 20 common reset seeds. Direct credit improves both AUC and final-policy return on SAC HalfCheetah but reduces both on SAC Walker in all three training pairs. In a separate Ant-v4 system comparison at the same interaction budget, Membrane-Credit PPO (MCrPPO) improves mean AUC by 64.8% over a SpikeGym reproduction retaining its native training schedule. Paired GPU and neuromorphic evaluation verifies Actor-only execution of selected MCrPPO policies after Critic removal. Together, AMC and the substrate combine improved training performance with explicit control over state access and encoder updates, while retaining an Actor-only deployment path.

Primary area:
reinforcement learning
Keywords:
spiking neural networks, reinforcement learning, actor-critic, neuromorphic computing, continuous control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_mGmTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce asymmetric membrane credit (AMC) for “fully spiking” actor-critic reinforcement learning. This aims to cope with the limitations of event-only critics by using both discrete spikes and real-value membrane potential. This mechanism is integrated with a reusable stateful spiking substrate by adapting this across various RL algorithms like PPO, SAC, and TD3. AMC was subject to systematic ablation analysis to isolate the effect of each ingredient, event-only (E), detached (D), full AMC (A). Additionally, the trained actors were deployed on Lynxi HP201 neuromorphic hardware to validate its practical use.

Critical strengths
  1. Rigorous and systematic evaluation of AMC: The systematic ablation study can be helpful for the algorithm evaluation.
  2. Deployment on dedicated hardware: The authors validated the viability of AMC by deploying it on Lynxi HP201.
Critical weaknesses
  1. Missing motivation and justification: I do not read any motivation of this work other than some statements like event-only critic cannot read continuous membrane potential. The authors should have systematically addressed any practical challenges that the event-only critic holds.
  2. Detached control (D) undermining AMC: I found that the detached control (D) often substantially outperforms AMC like on SAC Walker. These results undermine the needs for AMC.
  3. Clarity: several equations are written without clear explanation of each term. In the first equation on Page 2, I don’t understand what the episode mask means. I found the same term in Appendix, which is a binary mask determined by “reset”. I don’t understand what this reset means. The related Appendix section (Appendix B) also includes a number of variables that are never explained like \rho, a_{max}, a_{min}, \sigma_p^V, \mu_p^V. Where does the actor use the variable o_t?
Questions for authors
  1. What specific failure is caused by event-only critics?
  2. How can the authors explain my comment 2 in Critical weaknesses?
4 comments
AuthorsOpenReview ↗

Response to Reviewer mGmT

Thank you for the comments. Page numbers refer to the revised PDF.

W1 and Q1. The specific failure of an event-only Critic. The failure is state aliasing in value estimation. A hard threshold maps many different membrane states to the same events: in the recorded trace of Figure 1(a) (p. 3), every subthreshold value gives a zero event. An event-only Critic cannot tell these states apart, although they differ in accumulated input and in how close the neuron is to firing. When these differences predict return, the value estimate is wrong, and the error enters the PPO advantages and the bootstrapped targets of SAC and TD3. The second paragraph of the Introduction (p. 1) now states this. Two results confirm it. Giving the Critic the membrane readout raises mean final return in all 18 task–algorithm settings, for example from 507.9 to 1804.5 on SAC Walker (Table D4, p. 39). On frozen representations, a membrane readout predicts held-out return better than an equal-size event readout in 17 of 18 conditions for each Q head (Appendix E.4, p. 43).

W2 and Q2. D outperforming A does not undermine AMC. D is not a baseline without AMC. It reads the same membrane through the same trainable readout as A, and only blocks the extra gradient from that readout into the shared encoder (Table 1, p. 4). D above E is therefore direct evidence for the membrane readout.

Final return (20 resets)EDAD–EA–D
SAC Walker507.91804.5561.7+1296.6 (3/3 pairs)−1242.8 (0/3)
SAC HalfCheetah1975.42047.53273.7+72.1 (1/3)+1226.3 (3/3)

What the comparison of A and D shows is that membrane access and direct credit are two different things. Access helps throughout (D–E is positive in all 18 settings), whereas the extra gradient helps on some tasks and hurts on others. In every one of the 18 settings, A or D improves both mean AUC and mean final return over the event-only control. The paper therefore offers both modes and names D as the practical choice where direct credit is harmful (Section 4.2, p. 7; Section 5, p. 10). Separating the two is a contribution of the work, not a weakness of AMC.

W3. Episode mask and undefined symbols. These are now defined in the main text and collected in a notation guide (Appendix A, pp. 15–17). The episode mask is mₜ = 1 − resetₜ, where resetₜ = 1 on the first observation of a new episode; it sets the recurrent state of that environment to zero and is different from the spike reset of a single neuron (Section 2.1, p. 3). The Actor uses oₜ as the mean of its Gaussian policy: the action is aₜ = c + s · tanh(oₜ + exp(ρ) · εₜ), where ρ is the learned log standard deviation and c and s are the midpoint and half-range of the task's action bounds a_min and a_max. Finally, μᵥ and σᵥ are the running mean and standard deviation of the returns, used to decode the value as Vₜ = σᵥ uₜ + μᵥ (Section 2.2, p. 4).

Dear Reviewer mGmT,

With the discussion period coming to a close soon, we wanted to briefly follow up to check if our response and manuscript revisions have clarified your questions, or if there are any remaining points you would like us to elaborate on. We remain actively available and will gladly address any further questions promptly.

Thank you again for your time and constructive feedback in reviewing our paper.

Reviewer_mGmTOpenReview ↗

Response to authors

I appreciate the authors' effort on the rebuttal. Nevertheless, the motivation of this work (fully SNN-based system) is not fully justified, the authors show some advantages in a specific domain though. I therefore keep my original score.

Dear Reviewer mGmT,

Thank you for reading our rebuttal and recognizing the advantages demonstrated within the full-SNN setting. Since our earlier reply to W1 focused on why event-only readouts suffer from state aliasing rather than why the fully spiking Actor–Critic paradigm itself is worth studying, we briefly clarify that system-level motivation here:

  1. Toward self-contained neuromorphic learning. Hybrid SNN-Actor/ANN-Critic designs rely on a separate floating-point MLP Critic, which confines training to conventional off-chip hosts. Fully spiking Actor–Critics (e.g., Zanatta et al., 2024; Zhang et al., 2025) are studied as the algorithmic step toward self-contained neuromorphic adaptation, where both policy and value estimation operate on spiking primitives without maintaining a separate conventional DNN.
  2. Shared-representation efficiency even in off-chip training. Sharing one spiking core avoids maintaining a large separate Critic network while allowing value learning to shape the policy representation directly. With only 1,025 Critic-specific parameters—versus 5,121–179,201 in a width-matched one-layer ANN Critic (H512) and 68,353–155,393 in a two-layer ANN Critic (H256; Table F5)—MCrPPO outperforms the hybrid baseline PopSAN-v5 on all six tasks and exceeds width-matched H512 on three of six tasks (Table 5), while keeping Actor deployment strictly event-mediated with zero overhead.

Prior fully spiking RL defaulted to event-only value readouts without examining the subthreshold information lost at the spike threshold. Establishing how value estimation and encoder credit should be separated inside a shared spiking core (with membrane access improving final return in all 18/18 settings across PPO, SAC, and TD3) addresses a foundational bottleneck for this direction. We hope this clarifies the broader motivation of the paradigm, and we thank you again for your time and feedback.

Reviewer_x6cWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Asymmetric Membrane Credit (AMC) for stateful spiking Actor-Critic reinforcement learning. A shared LIF encoder produces binary events and continuous pre-threshold membrane states. The Actor continues to use the events, while a training-only Critic gains a zero-initialized linear membrane readout. A carefully specified three-way comparison distinguishes an event-only Critic (E), a detached membrane readout that trains the Critic without the added encoder-gradient route (D), and the attached readout that enables direct encoder credit (A). The paper gives a local gradient identity for A versus D and implements the design with distinct PPO, SAC, and TD3 state-reconstruction/update protocols. Across six MuJoCo-v5 tasks, three paired training seeds per task/algorithm, and common-reset final-policy evaluation, A exceeds E in mean learning-curve AUC in 17 of 18 settings and in mean final return in 16 of 18.

Critical strengths

A precise, scientifically useful intervention. Sections 2.2-2.3 and Table 1 separate E (event-only), D (membrane readout with its additional gradient detached), and A (attached). D and A have the same forward function class and parameter count. Equation (3) isolates the current-step additional encoder-gradient term; Appendix D.5 reports matched shadow-branch numerical checks. The authors correctly limit this identity to current-step gradients under truncated recurrence. The negative A-D examples are not hidden. This makes the work more informative than a simple "add a membrane feature and report a gain" study.

Broad evaluation and transparent limitations. The main E/D/A tests cover PPO, SAC, and TD3 on six MuJoCo-v5 tasks, with 54 matched training triplets and fixed 20-reset final-model evaluations. The complete A-E pathway improves mean AUC in 17/18 and mean final return in 16/18 settings. Importantly, SAC Walker shows D-E = +1296.6 mean final return but A-D = -1242.8, while SAC HalfCheetah shows a positive A-D effect (Section 4.2). The system comparison (including the Ant-v4 SpikeGym reproduction) is explicitly distinguished from the causal E/D/A comparison, and Lynxi HP201 execution checks show that Critic removal preserves the evaluated Actor control behavior. The authors do not claim that the measured hardware pipeline is faster or more energy efficient than CUDA Graph.

Critical weaknesses

Justification of membrane potential (Vmem) in SNNs: I am not sure why one would use the vmem in SNNs. If we are to use the continuous signal, why won't we just use ANNs (conventional DNNs)?

Comparison: if Vmem is used (which is an additional analog input), comparison should be with analog counterparts, not just Spikegym. Also, is spikegym events only? If so, only AUC is not a fair comparison; we need the efficiency metrics too.

Efficiency metrics: what's the computational and memory overhead for the proposed method? specially, the analog Vmem would need huge storage and memory fetching load.

The specific contribution of membrane information and of direct credit remains incompletely established. E versus D adds both a learnable readout and a new source of information. The inert parameter-count control Z in Appendix D.1 is useful for implementation checks but is not a trainable, capacity-matched event-only Critic. A trainable event-only control with comparable extra Critic capacity would help determine whether D-E gains arise from membrane amplitude rather than simply a more expressive Critic. Moreover, the direct encoder route is not consistently beneficial: mean A-D final-policy gains are positive in only 11/18 settings (Section 4.2), and SAC Walker's benefit from D is almost erased by enabling A. The single-seed surrogate-family tests in Appendix H also reverse some effect signs. Thus, the most compelling result concerns asymmetric Critic access rather than a generally beneficial direct-credit mechanism. The title and main claims should maintain this distinction.

The evidence for reliable practical improvement remains limited by training replication and comparison scope. A mean win in 17/18 settings should not be read as significance in 17 independent controlled experiments; some paired training-run effects differ in sign, and 20 evaluation resets are not additional training replicates (Appendix C). The external PPO comparisons involve differing architectures and learner schedules; for example, the Ant-v4 SpikeGym comparison matches environment steps but uses substantially different optimizer-update structures (Appendix F.5). Its +64.8% AUC is an interesting complete-system outcome, not evidence for the isolated AMC mechanism. Finally, the hardware experiments verify Actor-only execution on selected policies, but HP201 has higher measured host-to-action latency (0.534-0.608 ms versus 0.200-0.215 ms for CUDA Graph), and reported dynamic-energy differences fall below the setup's noise floor (Appendix G). These boundaries constrain the broader efficiency argument, even though the authors acknowledge them.

Questions for authors

See weaknesses please. Additionally,

Membrane information versus extra Critic capacity. Do you have an existing trainable event-only Critic control with the same additional readout parameter budget as D/A (for example, an additional event-only projection)? If not, can you sharpen the claim to make explicit that D-E combines increased Critic capacity and membrane access? This distinction matters to the novelty assessment.

Replication and sensitivity. From the already recorded runs, can you summarize how many A-E and D-E improvements are positive across all three training pairs, separately for AUC and final policy, rather than only their setting-wise means? Which conclusions do you consider replicated at the training-seed level?

Direct-credit conditions. What evidence from the existing A/D gradient checks or outcomes explains why the direct route helps on SAC HalfCheetah but hurts on SAC Walker? Given the reversal seen for the one-seed ATan versus fast-sigmoid tests, how broadly should readers expect A-D gains to transfer between surrogate families?

3 comments
AuthorsOpenReview ↗

Response to Reviewer x6cW

Thank you for the careful and balanced review. Page numbers refer to the revised PDF.

W1. Why read the membrane potential instead of using an ANN. The membrane is not an additional analog input. It is the internal state that the spiking encoder computes anyway, and AMC lets only the Critic read it, only during training. The deployed policy remains a spiking Actor that reads binary events, with no added parameters or operations (Proposition 1, p. 19; Table B3, p. 29). AMC therefore improves training with information that already exists, while keeping an event-driven Actor at deployment, which an ANN does not offer.

W2. Analog counterparts and efficiency. Table 5 (p. 9) now includes them: ANN-PPO and independent ANN Critics (H512, H256) attached to our spiking Actor. MCrPPO exceeds the width-matched H512 on three of six tasks with 1,025 Critic-specific parameters instead of 5,121–179,201. SpikeGym's Critic reads events only, which is why we added AMC directly to it: AUC rises by 18.1% (first half) and 4.9% (full budget) in all three seed pairs (Table 4, p. 8). On efficiency, the deployed Actors cost the same on Lynxi HP201, 0.316–0.436 ms per call for MCrPPO and 0.344–0.435 ms for SpikeGym-v5 (Table G3, p. 59), because AMC adds nothing to the Actor.

W3. Computational and memory overhead. AMC stores no additional data, since the membrane belongs to the recurrent state that every condition already computes. It adds 512 weights in PPO or 1,024 across the twin Q heads. One learner step takes 3.33% longer in PPO (85.721 to 88.579 ms) and 5.03% longer in SAC (8.043 to 8.447 ms), and peak memory rises by 8 KiB and 16 KiB (Appendix E.7, p. 46). The overhead at deployment is zero.

W4 and Q1. Membrane information versus extra capacity. We added exactly this control. C is a trainable event-only readout with the same number of extra parameters as D (one seed, six tasks). D exceeds C in final return in 8 of 12 PPO/SAC settings, for example by +1344.4 on PPO HalfCheetah and +1024.9 on SAC Walker2d, and in AUC in 7 of 12 (Table E2, p. 45). On frozen representations, a membrane readout also predicts held-out return better than an equal-size event readout in 17 of 18 conditions for each Q head (Appendix E.4, p. 43). Extra capacity alone does not explain the gain. We agree on the framing, and the paper follows it: the abstract, Section 4.2 and Section 5 present membrane access as the reliable component (D–E is positive in final return in all 18 settings) and direct credit as task-dependent. The asymmetry in the title refers to this Critic-only access.

W5 and Q2. Replication at the training-seed level. The counts you asked for are in Table D1 (p. 34) and Figure 1(d). Each cell gives the settings with a positive mean, followed by the settings where all three pairs are positive.

OutcomeA–ED–EA–D
Learning-curve AUC17/18, 7/1814/18, 8/1811/18, 6/18
Final return (20 resets)16/18, 6/1818/18, 6/1811/18, 3/18

We consider replicated the effects with the same sign in all three pairs, such as SAC Walker D–E +1296.6 and A–D −1242.8, SAC HalfCheetah A–D +1226.3, TD3 HalfCheetah A–E +2199.9, and the native SpikeGym AUC gains. The 17/18 and 18/18 counts are reported as trends of setting-wise means, and Appendix C.2 states that the analysis is descriptive. The 64.8% figure is labelled a system result and is now complemented by the isolated SpikeGym ±AMC ablation; the hardware section claims Actor-only execution, not a latency or energy advantage (Section 4.4, p. 9).

Q3. When direct credit helps, and transfer across surrogates. Shadow updates at 24 checkpoints confirm the mechanism: A and D give identical predictions and readout gradients, and differ only in the encoder update (Appendix E.2, p. 42). The sign of the long-run effect is not determined by the single-step gradient: the A/D gradient cosine is 0.106–0.548 on SAC HalfCheetah, where direct credit helps, and 0.603–0.959 on SAC Walker2d, where it hurts. Across surrogates, direct credit also helps under fast-sigmoid on HalfCheetah (AUC +177.8) but not under ATan (−577.0; one seed, Table E3, p. 47), so a positive case exists beyond the rectangular surrogate, while the sign does not transfer uniformly. This is precisely why AMC separates the two: membrane access is the reliable part, direct credit is an explicit switch, and in every one of the 18 settings A or D improves both mean AUC and final return over the event-only control.

Reviewer_x6cWOpenReview ↗

Thanks for your response

Thanks for the rebuttal, my initial score already reflects my judgement, hence my score is the same.

Dear Reviewer x6cW,

Thank you very much for reading our response and for maintaining your positive assessment of our work. Your constructive comments have been very helpful in improving the clarity and rigor of our manuscript, and we sincerely appreciate your time and thoughtful engagement.

Reviewer_6qUqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work proposes Asymmetric Membrane Credit (AMC), a critic-side membrane-potential readout mechanism for training spiking neural networks (SNNs) in reinforcement learning. The method employs a shared stateful SNN encoder for the actor and critic, where the actor accesses binary spike outputs while the critic additionally accesses the pre-threshold membrane potentials. This allows the critic to exploit continuous membrane information and provide direct gradients to the shared encoder, without introducing an additional inference branch for the actor. The authors implement AMC in PPO (MCrPPO) and further extend the framework to SAC and TD3. The paper also evaluates actor-only deployment on GPU and neuromorphic hardware.

Critical strengths
  1. The experimental evaluation covers multiple continuous-control tasks and three RL algorithms, including PPO, SAC, and TD3. The authors conduct extensive ablation studies across different task-algorithm settings.

  2. The additional evaluation of actor-only deployment on GPU and neuromorphic hardware provides useful evidence of the practical feasibility of deploying the learned SNN policies after removing the critic.

Critical weaknesses
  1. The motivation for using an SNN critic rather than an ANN critic is insufficiently justified. Hybrid architectures combining an SNN actor with an ANN critic are commonly used in SNN-based reinforcement learning, particularly because the critic is only required during training and can be removed during deployment. The proposed method instead adopts a shared SNN encoder for both the actor and critic, but does not provide a comparison with a hybrid architecture using an SNN actor and an ANN critic. Moreover, the proposed method achieves lower performance than the SNN-actor/ANN-critic baseline reported by Tang et al. (2021).

  2. The experimental comparisons are not sufficiently fair. The reported ANN-PPO baseline exhibits relatively weak performance compared with commonly used PPO implementations. More importantly, the network architectures and training configurations differ between the ANN and SNN methods.

One additional minor weakness: The manuscript is difficult to follow in several places, particularly when explaining the proposed mechanism and distinguishing between different experimental variants. In addition, several tables are difficult to read, and the overall layout and formatting could be improved.

Questions for authors
  1. What is the motivation for using an SNN critic instead of a conventional ANN critic?

  2. Could you provide comparisons with other recent SNN-based reinforcement learning baselines?

  3. Can you also compare AMC with ANN-to-SNN conversion and ANN-to-SNN policy distillation methods?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6qUq

Thank you for the constructive comments. Page numbers refer to the revised PDF. The table below reproduces the new Table 5 (p. 9), which adds the comparisons you asked for: best monitoring return after 20 Mi steps from one training seed, with each task scaled to its best method.

MethodHalfCheetahWalker2dHopperAntSwimmerHumanoid
MCrPPO (ours, spiking Critic)95.176.995.085.264.780.5
SpikeGym-v567.772.590.984.651.875.9
ANN-PPO24.276.610010067.576.2
H512 (our Actor, one-layer ANN Critic)96.349.991.683.392.299.0
H256 (our Actor, two-layer ANN Critic)10010096.884.9100100
PopSAN-v5 (code of Tang et al.)28.337.078.227.159.941.6
ANN-to-SNN conversion23.176.599.698.664.775.8
Policy distillation21.74.221.575.359.647.6

W1 and Q1. Why a spiking Critic; hybrid comparison. Our goal is a fully spiking Actor–Critic (Zanatta et al., 2024; Zhang et al., 2025), a setting distinct from the hybrid one. With a shared spiking encoder, the Critic's gradients shape the representation that the deployed Actor uses, so the Critic is not merely a training component that is discarded afterwards. How it should read and update the shared state had not been studied, and this is the question we address. We nevertheless added the hybrid comparison you requested. H512 and H256 keep our spiking Actor and replace the Critic with an independent ANN. MCrPPO exceeds the width-matched H512 on three tasks with 1,025 Critic-specific parameters instead of 5,121–179,201 (Table F5, p. 52). H256, a two-layer Critic with 67 to 152 times more Critic parameters, is higher on five tasks; Section 5 (p. 10) states this openly, since surpassing ANN Critics is not our claim. Regarding Tang et al. (2021), their reported numbers come from earlier MuJoCo versions and a different protocol and are not directly comparable. We therefore ran their released code on our tasks and budget (PopSAN-v5), and MCrPPO is higher on all six tasks.

W2. ANN-PPO baseline and fairness. ANN-PPO is an external reference in our study, not the baseline against which AMC is evaluated. Every claim about AMC rests on the E/D/A comparison, in which the three conditions share the same architecture, hyperparameters, seeds and training schedule and differ only in the membrane readout (54 matched triplets; Table D1, p. 34). For the system references, we deliberately ran ANN-PPO at MCrPPO's rollout size and number of optimizer updates rather than with separately tuned defaults, so that both use the same update budget; this is why its scores differ from commonly reported PPO results (Section 3, p. 5; Appendix C.4, p. 32). The new Discussion and Limitations section (Section 5, p. 10) states this scope explicitly: we study direct RL within a shared full-SNN architecture and do not aim to match unconstrained ANN baselines. To remove the remaining architecture difference, we added H512 and H256, which use the same Actor, value objective and schedule as MCrPPO and change only the Critic.

Q2. Recent SNN-RL baselines. We compare with two baselines run under our own protocol. Against SpikeGym-v5 (fully spiking PPO) over five seeds, MCrPPO has higher mean AUC on five of six tasks and higher final return on four, including +105.7% AUC on HalfCheetah and +88.4% final return on Swimmer, each in all five pairs (Tables F1–F4, pp. 48–50). Against PopSAN-v5 (spiking Actor with ANN Critic), MCrPPO is higher on all six tasks. Appendix H.2 (p. 64) discusses how AMC differs from Proxy Target, CaRe-BN, SpikingNav and the fully spiking Actor–Critic of Zhang et al.

Q3. Conversion and distillation. Both are added and reuse a 20 Mi ANN-PPO teacher (Appendix F.2, p. 51). MCrPPO needs no teacher, is higher than distillation on all six tasks, and matches or exceeds conversion on four. Conversion inherits the return of its ANN teacher and needs 128 steps per decision, whereas the MCrPPO Actor uses one step and is 9.0–12.4 times faster on HP201 (Table G3, p. 59).

Minor weakness. Readability. Figure 1 (p. 3) now shows the mechanism, the E/D/A variants and the learning rules in one place, Tables 1–3 define the variants and the experimental questions, and the appendix is reorganized with a notation guide and 29 tables instead of 45 (Appendix A, p. 15).

Dear Reviewer 6qUq,

With the discussion period coming to a close soon, we wanted to briefly follow up to check if our response and manuscript revisions have clarified your questions, or if there are any remaining points you would like us to elaborate on. We remain actively available and will gladly address any further questions promptly.

Thank you again for your time and constructive feedback in reviewing our paper.

Thank you for the detailed response and additional experiments. The presentation of this work has been improved, and the additional experiment also shows the effectiveness of the proposed AMC. However, it remains unclear why an SNN critic is used. Thus, I will keep my initial rating.

Dear Reviewer 6qUq,

Thank you very much for confirming that the presentation improvements and additional experiments establish the effectiveness of AMC. Regarding your remaining question—why use an SNN Critic at all when a separate ANN Critic can be discarded after training—there are three concrete reasons why the SNN-Critic regime is studied alongside hybrid designs:

  1. On-device and neuromorphic learning compatibility. A hybrid ANN Critic (such as H256 or PopSAN) requires dense floating-point MLP forward/backward passes, which permanently restricts training and fine-tuning to conventional off-chip hosts (CPUs/GPUs). Studying an SNN Critic is a necessary algorithmic step toward self-contained neuromorphic adaptation and on-chip continual RL, where both Actor and Critic must operate on homogeneous spiking primitives without maintaining a separate conventional DNN.
  2. Shared-representation supervision vs. policy-only learning. With an independent ANN Critic (H256/H512), the spiking Actor encoder receives only high-variance policy gradients, learning no state features from value supervision (Table F5). Sharing a spiking Critic core lets value estimation directly shape the spiking encoder representation while keeping Critic-specific overhead minimal: MCrPPO uses just 1,025 Critic-specific parameters—two orders of magnitude fewer than H256 (68,353–155,393) and H512 (5,121–179,201)—yet outperforms PopSAN-v5 on all six tasks and width-matched H512 on three of six tasks (Table 5).
  3. Closing an open gap in an established research line. Fully spiking Actor–Critics are already an active paradigm (e.g., Zanatta et al., 2024; Zhang et al., 2025), yet prior methods defaulted to symmetric spike-only Critic readouts that discard subthreshold state already computed by the LIF core. Showing how to unlock that subthreshold information (improving final return in 18/18 settings across PPO, SAC, and TD3) without adding any Actor deployment branch resolves a core bottleneck of this paradigm.

In short, hybrid ANN Critics optimize off-chip training return with unconstrained Critic capacity, whereas our work addresses how reinforcement learning can work effectively within a homogeneous spiking architecture. We sincerely appreciate your constructive engagement throughout the discussion.

Reviewer_G1vJOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks which internal state a spiking Critic should read when the Actor and Critic share a stateful spiking encoder. It introduces Asymmetric Membrane Credit (AMC), a zero-initialized linear readout of the shared pre-threshold membrane added only to the Critic; the Actor remains event-mediated and can be deployed without the Critic. A detached control separates membrane access from direct encoder credit. Across PPO, SAC and TD3 on six MuJoCo-v5 tasks, AMC improves mean AUC in 17/18 settings and mean final return in 16/18, with task-dependent direct-credit effects; Actor-only execution is verified on a Lynxi HP201 chip. The question is well posed and the controls are clean, but the gains are small and noisy, and the 64.8% system-level gain is not an isolated AMC effect.

Critical strengths

It separates membrane access (D–E) from the added encoder-gradient route (A–D) at matched parameter count and forward function class, and the substrate spans PPO, SAC and TD3 with real neuromorphic deployment.

Critical weaknesses

1.The figures and tables appear to be directly AI generated.
2.The 64.8% SpikeGym gain is a system comparison, not an AMC effect. Architecture, training schedule and implementation differ, so the gain cannot be attributed to membrane credit without a same-architecture ablation or an AMC-added SpikeGym reproduction. 3.“Full-SNN” is misleading. The Critic reads continuous membrane states and the encoder receives continuous gradients; only the Actor is event-mediated. In addition, HP201 latency is worse than GPU (0.534–0.608 ms vs. 0.200–0.215 ms), and no main-text energy result supports the deployment case.

Questions for authors

Can you isolate AMC under the same architecture and schedule, and report full-loop (not Actor-only) energy and latency on HP201?

2 comments
AuthorsOpenReview ↗

Response to Reviewer G1vJ

Thank you for the specific comments. Page numbers refer to the revised PDF.

W1. AI-generated figures and tables. All numbers come from our own experiments. Quantitative figures and tables are produced by scripts from the recorded results, and the anonymized code for the E/D/A experiments is in the supplementary material. AI assistance in the figures was limited to decorative elements such as task renders and icons, as stated in the original AI Use Disclosure. We have nevertheless redrawn Figure 1, with recorded data in panels (a) and (d), to make it clearer (p. 3).

W2. The 64.8% gain is a system comparison. We agree, and Section 4.3 labels it as such. We ran the requested ablation: native SpikeGym on Ant-v4 with its network, surrogate, optimizer and schedule unchanged, adding only a zero-initialized 512-weight membrane readout to the value head (three paired seeds; Table 4, p. 8).

MetricNativeNative + AMCChangePositive pairs
AUC, 0–16.4M steps1396.01648.3+18.1%3/3
AUC, 0–32.8M steps3118.43271.9+4.9%3/3
Best checkpoint return5555.45600.5+45.22/3

AMC alone speeds up learning in all three pairs. A schedule-only control changes AUC far more (+75.2%, one seed; Table F9, p. 55), so we attribute only the gains above to AMC. Across our 54 matched E/D/A triplets, which share one architecture and schedule, AMC raises mean AUC in 17 of 18 settings (Table D1, p. 34).

W3. "Full-SNN", latency and energy. The reviewer's description of training is correct. We use "full-SNN" as an architectural term meaning no separate ANN Critic (Section 1, p. 2). The deployed Actor reads only binary events and is unchanged when the Critic and its membrane readout are removed (Proposition 1, p. 19). We agree that HP201 is slower than the GPU and claim no latency or energy advantage (Section 4.4, p. 9). As requested, we measured the full control loop including the simulator step. The mean loop time is 1.490–2.173 ms on HP201 against 0.513–2.513 ms with CUDA Graph, which is below each task's 8–50 ms control period, and 99.2–104.5% of the GPU return is preserved. After idle subtraction, net energy stays within measurement noise in 53 of 54 windows, so no energy difference is resolved (Appendix G.4, Tables G5–G6, pp. 60–63).

Dear ReviewerG1vJ,

With the discussion period coming to a close soon, we wanted to briefly follow up to check if our response and manuscript revisions have clarified your questions, or if there are any remaining points you would like us to elaborate on. We remain actively available and will gladly address any further questions promptly.

Thank you again for your time and constructive feedback in reviewing our paper.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading. The revision (changes in blue; 10 pages main text, 64 pages in total) keeps the original claims and adds every control that was requested. All locations refer to the revised PDF.

What the paper claims. AMC gives the Critic of a shared, stateful spiking Actor–Critic a zero-initialized, training-only readout of the pre-threshold membrane. The Actor reads events only and is unchanged when the Critic is removed (Prop. 1, p. 19). We study how a spiking Critic should read and update the shared representation; we do not claim to surpass ANN or hybrid Critics (new Sec. 5, p. 10).

Controlled evidence (original cohort: 54 matched E/D/A triplets, three seeds × 18 task–algorithm settings; Table D1, p. 34). Parentheses give the number of settings in which all three pairs are positive.

ContrastMean AUC > 0Mean final return > 0 (20 resets)
A–E, complete pathway17/18 (7)16/18 (6)
D–E, detached membrane readout14/18 (8)18/18 (6)
A–D, direct credit11/18 (6)11/18 (3)

In every setting, A or D exceeds E in both mean AUC and mean final return. Direct credit is task-dependent: +1226.3 on SAC HalfCheetah (3/3 pairs) and −1242.8 on SAC Walker (0/3).

New experiments.

  1. Native SpikeGym ±AMC on Ant-v4, three pairs: AUC over 0–16.4M steps +18.1%, over 0–32.8M +4.9%, both 3/3 (Table 4, p. 8).
  2. A trainable event-capacity control C and a return-prediction probe on frozen representations (Sec. 4.2, p. 7; Table E2, p. 45).
  3. An eight-method comparison with independent ANN Critics (H512, H256), PopSAN-v5, ANN-to-SNN conversion and policy distillation (Table 5, p. 9).
  4. Full-loop latency and energy on Lynxi HP201 (Tables G5–G6, pp. 62–63).
  5. A Discussion and Limitations section (Sec. 5, p. 10).

Baseline naming. The baseline called "SNN-PPO (adapted)" in the original submission is not a baseline of our own design. It is the published SpikeGym learner (Zanatta et al., 2024) run on the six MuJoCo-v5 tasks. The original name was imprecise, and the revision calls it SpikeGym-v5 throughout (Sec. 3, p. 5; App. C.4, p. 33). "Native SpikeGym" denotes the original code and schedule on Ant-v4.

Evidence scope. E/D/A uses three training seeds, the six-task system comparison five, and native ±AMC three. C, H512/H256, PopSAN-v5, the teacher routes and the surrogate tests use one seed and are reported as references. The appendix is now Appendices A–H; locations cited in the reviews map as: original D.1/D.5 → E.1/E.2, H → E.6, F.5 → F.3.

AuthorsOpenReview ↗

Broader SNN-RL Context, Core Bottleneck, and Motivation of This Work

Dear Area Chair and Reviewers,

As several comments touch on the broader motivation for studying a fully spiking Actor–Critic rather than using conventional ANNs or hybrid ANN Critics, we would like to briefly summarize the field context, the specific bottleneck in spiking RL, and how our work fits in.

1. SNN-RL context. Current spiking continuous control follows three main routes, each with clear trade-offs:

  • ANN-to-SNN conversion requires a pre-trained ANN teacher and many simulation ticks per decision to approximate continuous activations (e.g., 128 ticks, running 9.0–12.4× slower per decision on Lynxi HP201 than a single-step persistent SNN Actor; Table G3).
  • Hybrid SNN-Actor / ANN-Critic methods (e.g., PopSAN) offload value estimation to a large, separate floating-point MLP. While effective for maximizing raw return, this permanently ties learning to a conventional DNN and gives up shared spiking representations, preventing self-contained spiking Actor–Critic architectures.
  • Directly trained fully spiking Actor–Critics (e.g., SpikeGym; Zanatta et al., 2024; Zhang et al., 2025) maintain both policy and value learning on spiking primitives with single-step (T=1T=1) persistent state, forming the algorithmic foundation toward self-contained neuromorphic reinforcement learning.

2. The core bottleneck in shared spiking Actor–Critics. Prior fully spiking Actor–Critics enforce symmetric event-only access: both Actor and Critic read only binary spikes. In continuous control, this creates two coupled challenges:

  • Value aliasing across the spike threshold: Hard thresholds collapse distinct subthreshold membrane trajectories into identical zero-spike outputs (Figure 1(a)). While discrete events suffice to drive bounded actions, value estimation requires fine-grained return prediction; losing subthreshold amplitude degrades PPO advantages and SAC/TD3 bootstrap targets.
  • Representation coupling and deployment constraints: Exposing continuous membrane states symmetrically to the Actor would break event-mediated deployment, whereas routing extra continuous value gradients directly into a shared spiking encoder can either accelerate or destabilize policy features depending on the task and learning rule.

3. Motivation and contribution of AMC. Rather than claiming that compact full-SNN learners already surpass unconstrained multi-layer ANN Critics on raw return (explicitly bounded in Section 5), our goal is to resolve this foundational interface bottleneck within fully spiking RL:

  • Zero-cost internal state reuse: Pre-threshold membrane potential is already computed by every LIF core. AMC gives only the training Critic a lightweight readout (512–1,024 weights) while keeping the deployed Actor strictly single-step (T=1T=1), event-mediated, and unchanged upon Critic removal (Proposition 1).
  • Decoupling state access from gradient routing: Pairing attached (A) and detached (D) readouts separates membrane-informed value estimation—which reliably improves 20-reset final return in all 18/18 PPO, SAC, and TD3 settings (Table D1)—from direct encoder credit, which provides strong task-dependent gains (e.g., +1226.3 on SAC HalfCheetah, and +18.1% early AUC inside unchanged native SpikeGym; Table 4) while remaining cleanly detachable where harmful (e.g., SAC Walker). Even with 1,025 Critic-specific parameters, MCrPPO outperforms the hybrid baseline PopSAN-v5 on all six tasks and matches or exceeds a width-matched one-layer ANN Critic (H512, 5,121–179,201 parameters) on three tasks (Table 5).

We hope this places our problem formulation and empirical findings in clear perspective, and we sincerely thank the Area Chair and Reviewers for their time and engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.