- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This work aims to address the limitations of existing reinforcement learning (RL)-based deep spiking Q-networks (DSQNs), which typically need multiple timesteps to achieve competitive performance. Consequently, RL-based DSQNs incur substantial computational and energy costs. Specifically, this paper proposes Common-Mode Compensation Deep Spiking Q-Network (CMC-DSQN), which employs a conventional artificial neural network (ANN) during training to compensate for common-mode errors in spiking neural network (SNN) outputs. CMC-DSQN first investigates performance degradation due to timestep reduction while decomposing the errors across actions into common-mode and differential-mode components, and observes that low-timestep DSQNs are dominated by common-mode errors. The paper addresses a relevant problem and proposes a simple and potentially useful solution. The method employs a small CliffWalking environment for error analysis, and is evaluated on seven Atari games and five MiniAtar environments. The empirical evidence on performance is promising, but the evaluation still lacks certain design aspects, which will be discussed below.
Critical strengths
• The target research problem is relevant and clearly motivated with a simple solution based on a common-mode error compensation mechanism.
• The idea of error decomposition is interesting and useful for identifying that common-mode errors affect temporal-difference learning more directly than differential-mode errors.
Critical weaknesses
• The case-study experiments using the CliffWalking environment provide evidence for a specific environment. Therefore, the respective analysis is limited in supporting broader claims across more environments and conditions.
• The case-study experiments focus on a particular DSQN formulation and a small CliffWalking environment. Therefore, it remains unclear whether common-mode error dominance still persists across different DSQN models, neuron models, network architectures, timestep ranges, or other RL algorithms.
• The evaluation does not discuss energy efficiency gains offered by the CMC-DSQN solution over the ANN-based solutions. Since the CMC-DSQN solution does not offer significantly higher performance than the ANN-based solutions, the lack of energy efficiency metrics makes the practicality of CMC-DSQN unclear.
Questions for authors
• It is recommended to clearly define the common-mode and differential-mode error components in the first section (i.e., Introduction) to help readers understand them better.
• Can the authors clarify whether common-mode error dominance holds for other DSQN models, network architectures, and RL algorithms?
• It is suggested that the case-study experiments consider multiple environments beyond CliffWalking to provide more generalizable analysis. Therefore, multiple observation results (like in Figure 4) should be obtained for different environments.
• Can the authors clarify why an auxiliary ANN is required, rather than an SNN-based model or other analytical-based techniques?
• What are the overhead costs (i.e., latency/throughput and power/energy) for employing an auxiliary ANN during the training phase?
• What are the efficiency gains obtained by the proposed CMC-DSQN as compared to the ANN-based solution? Because, performance-wise, the CMC-DSQN solution does not offer significantly higher scores than the ANN-based solutions. Hence, efficiency metrics may justify the practical benefits of CMC-DSQN.
• If possible, it is suggested to compare with another recent work on DSQN that employs active dendrites and a dueling network structure to complete the DSQN-based study, such as “Scalable Multi-Task Learning through Spiking Neural Networks with Adaptive Task-Switching Policy for Intelligent Autonomous Agents”, arXiv preprint arXiv:2504.13541.
11 comments
Reply 2/5
W2: The case-study experiments focus on a particular DSQN formulation and a small CliffWalking environment. Therefore, it remains unclear whether common-mode error dominance still persists across different DSQN models, neuron models, network architectures, timestep ranges, or other RL algorithms.
Thank you for this important comment. We agree that the original case study provides limited evidence of the generality of our findings. We have therefore extended the analysis to different neuron models (LIF, IF, and CLIF), network widths (64, 128, and 256), simulation timesteps (T=2,3,4,5), and RL algorithms (DQN, Double DQN, and DQN with prioritized experience replay (PER)).
As reported in Tables R5–R8, common-mode (CM) errors consistently exceed differential-mode (DM) errors across all tested configurations without compensation, confirming that common-mode dominance is not specific to the original DSQN setting. Moreover, CMC consistently reduces both error components and substantially improves learning performance, with the benefits persisting across these configurations.
Table R5: Results with different neuron models.
| Neuron | CMC | Success rate | CM RMSE | DM RMSE |
|---|---|---|---|---|
| LIF | No | 15% | 2.4632 | 1.2190 |
| LIF | Yes | 90% | 0.6616 | 0.4854 |
| IF | No | 25% | 2.1300 | 1.1281 |
| IF | Yes | 100% | 0.4513 | 0.4305 |
| CLIF | No | 15% | 2.2709 | 1.0703 |
| CLIF | Yes | 90% | 0.3901 | 0.4077 |
Table R6: Results with different network widths.
| Hidden width | CMC | Success rate | CM RMSE | DM RMSE |
|---|---|---|---|---|
| 64 | No | 15% | 2.4632 | 1.2190 |
| 64 | Yes | 90% | 0.6616 | 0.4854 |
| 128 | No | 65% | 0.8029 | 0.5635 |
| 128 | Yes | 100% | 0.3154 | 0.3287 |
| 256 | No | 100% | 0.5045 | 0.3751 |
| 256 | Yes | 100% | 0.2771 | 0.3442 |
Table R7: Results with different simulation timesteps.
| Timestep | CMC | Success rate | CM RMSE | DM RMSE |
|---|---|---|---|---|
| T=2 | No | 15% | 2.4632 | 1.2190 |
| T=2 | Yes | 90% | 0.6616 | 0.4854 |
| T=3 | No | 15% | 1.9828 | 0.9714 |
| T=3 | Yes | 100% | 0.3991 | 0.3602 |
| T=4 | No | 60% | 1.2887 | 0.7024 |
| T=4 | Yes | 95% | 0.3328 | 0.3040 |
| T=5 | No | 30% | 1.3093 | 0.7062 |
| T=5 | Yes | 100% | 0.4021 | 0.3312 |
Table R8: Results with different RL algorithms.
| RL algorithm | CMC | Success rate | CM RMSE | DM RMSE |
|---|---|---|---|---|
| DQN | No | 15% | 2.4632 | 1.2190 |
| DQN | Yes | 90% | 0.6616 | 0.4854 |
| Double DQN | No | 25% | 1.2036 | 1.0955 |
| Double DQN | Yes | 95% | 0.5144 | 0.4837 |
| DQN + PER | No | 10% | 3.6699 | 1.2732 |
| DQN + PER | Yes | 75% | 0.6923 | 0.5138 |
We have added a paragraph to Sec. 4 to summarize these findings:
We further examine different SNN timesteps, neuron models, RL algorithms, and network architectures in Appendix C.1, consistently observing common-mode dominance across configurations. We have also visualized these additional results in Figs. 10-18.
Reply 3/5
W3: The evaluation does not discuss energy efficiency gains offered by the CMC-DSQN solution over the ANN-based solutions. Since the CMC-DSQN solution does not offer significantly higher performance than the ANN-based solutions, the lack of energy efficiency metrics makes the practicality of CMC-DSQN unclear.
Thank you for raising this important concern. We have added an analysis of inference energy consumption to evaluate the efficiency of CMC-DSQN relative to ANN-based solutions.
Following the widely adopted estimation framework in [1], we assign 12.5 pJ per floating-point operation (FLOP) and 77 fJ per synaptic operation (SOP) [2-3]. Energy consumption is estimated by multiplying the corresponding operation counts by their per-operation energy costs. For image-based environments (Atari and MinAtar), we exclude the first convolutional layer consistently for both ANN and SNN models, treating it as part of the visual sensing front-end.
Importantly, the auxiliary ANN in CMC-DSQN is used only during training and is discarded at inference. Thus, only the SNN is retained for deployment, introducing no additional ANN inference overhead.
As shown in Tables R9 and R10, CMC-DSQN substantially reduces estimated inference energy compared with ANN baselines. Moreover, compared with vanilla DSQN, CMC-DSQN further reduces energy consumption by approximately 21–22% on Atari and 54–59% on MinAtar, owing to its lower SOP counts. These results demonstrate that CMC-DSQN achieves competitive performance with substantially lower estimated inference energy than ANNs, while also improving the energy efficiency of SNNs.
Table R9: Estimated inference energy consumption on Atari.
| Metric | ANN | DSQN (T=2) | CMC-DSQN (T=2) | DSQN (T=3) | CMC-DSQN (T=3) | DSQN (T=4) | CMC-DSQN (T=4) |
|---|---|---|---|---|---|---|---|
| FLOPs (M) | 12.14 | - | - | - | - | - | - |
| SOPs (M) | - | 1.88 | 1.48 | 2.91 | 2.29 | 3.87 | 3.01 |
| Energy (nJ) | 151,760 | 145 | 114 | 224 | 177 | 298 | 232 |
Table R10: Estimated inference energy consumption on MinAtar.
| Metric | ANN | DSQN (T=2) | CMC-DSQN (T=2) | DSQN (T=4) | CMC-DSQN (T=4) | DSQN (T=6) | CMC-DSQN (T=6) |
|---|---|---|---|---|---|---|---|
| FLOPs (K) | 263 | - | - | - | - | - | - |
| SOPs (K) | - | 36 | 15 | 69 | 30 | 109 | 50 |
| Energy (nJ) | 3290 | 2.76 | 1.14 | 5.32 | 2.31 | 8.40 | 3.87 |
Q1: It is recommended to clearly define the common-mode and differential-mode error components in the first section (i.e., Introduction) to help readers understand them better.
Thank you for this suggestion. We have added definitions of common-mode and differential-mode errors to the Introduction (Lines 69–70): The common-mode error represents the average estimation error shared across all actions at a given state, whereas the differential-mode error captures action-specific deviations from this shared component.
Q2: Can the authors clarify whether common-mode error dominance holds for other DSQN models, network architectures, and RL algorithms?
Please refer to our response to your W2, where we provide additional analyses across different settings to confirm the generality of common-mode error dominance.
Q3: It is suggested that the case-study experiments consider multiple environments beyond CliffWalking to provide more generalizable analysis. Therefore, multiple observation results (like in Figure 4) should be obtained for different environments.
Thank you for this suggestion. As discussed in our response to your W1, Fig. 5 (previously Fig. 4) now includes additional results on FrozenLake and Taxi. We further provide common-mode error analyses on Atari and MinAtar in Fig. 6 and Appendix Figs. 19-23.
Reference
[1] Paul A Merolla, John V Arthur, Rodrigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spikingneuron integrated circuit with a scalable communication network and interface. Science, 345 (6197):668–673, 2014.
[2] Ning Qiao, Hesham Mostafa, Federico Corradi, Marc Osswald, Fabio Stefanini, Dora Sumislawska, and Giacomo Indiveri. A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128K synapses. Frontiers in Neuroscience, 9:141, 2015.
[3] Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions on Neural Networks and Learning Systems, 34(8):5200–5205, 2021.
Reply 1/5
We appreciate the time and effort you invested in reviewing our paper. We would like to address your concerns as follows.
Thank you for raising this important point. We agree that the original CliffWalking case study alone is insufficient to establish the generality of our explanation. We have therefore expanded our analysis to additional experiments on FrozenLake, Taxi, seven Atari games, and all five MinAtar environments.
For FrozenLake and Taxi, we compute exact Q-values via dynamic programming and evaluate estimation errors against both the exact Qπ of each learned greedy policy and the optimal action-value function Q∗. As shown in Tables R1 and R2, CMC-DSQN reduces both common-mode (CM) and differential-mode (DM) errors compared with vanilla SNN. It also improves the success rate from 41.97% to 57.00% on FrozenLake and from 12.93% to 99.47% on Taxi. These results provide additional evidence that the benefits of common-mode compensation are not limited to CliffWalking. We have added these experiments and their learning curves to Sec. 5.2 and Fig. 5.
Table R1: Results on FrozenLake.
Table R2: Results on Taxi.
For Atari and MinAtar, where exact Q-values are unavailable, we estimate reference Qπ using Monte Carlo rollouts under the current greedy policy. Tables R3 and R4 summarize the aggregate errors across different timesteps. CMC-DSQN consistently reduces CM errors on Atari (T=2,3,4) and MinAtar (T=2,4,6). We have added a new
Error Analysissubsection (Sec. 6.2). Fig. 6 presents CM and DM errors across seven Atari games at T=2, showing consistent CM error reduction. Further results are added into Appendix C.2 (Figs. 19–23).Table R3: Average CM and DM RMSEs across seven Atari environments.
Table R4: Average CM and DM RMSEs across five MinAtar environments.
We further extend the analysis to different timesteps, neuron models, RL algorithms, and network architectures. Common-mode dominance and the benefits of its compensation persist across these configurations, as detailed in our response to your W2.