acceptodds
Under review as a conference paper at ICLR 2027

Common-Mode Errors Limit Low-Timestep Deep Spiking Q-Networks

Abstract

Spiking neural networks (SNNs) offer sparse and event-driven computation, making them attractive for energy-constrained reinforcement learning (RL) on edge devices. In value-based RL, deep spiking Q-networks (DSQNs) combine such efficiency with action-value estimation for decision making. However, existing DSQNs often require multiple simulation timesteps for competitive performance, increasing computational and energy costs, whereas reducing the timesteps can cause substantial performance degradation. We investigate this degradation from the perspective of Q-value estimation errors. By decomposing errors across actions into common-mode and differential-mode components, we find that low-timestep DSQNs suffer disproportionately from common-mode errors shared across action values, which are particularly detrimental to temporal-difference learning through bootstrapped targets. Based on this finding, we propose Common-Mode Compensation Deep Spiking Q-Network (CMC-DSQN), which uses an auxiliary ANN to compensate for common-mode errors in the SNN outputs. At inference, greedy action selection can be performed directly from the SNN outputs, allowing the auxiliary ANN to be completely removed and preserving the energy efficiency of SNNs. Extensive experiments on Atari and MiniAtar environments demonstrate substantial performance improvements under low-timestep settings. CMC-DSQN outperforms state-of-the-art DSQN baselines by nearly % at and further surpasses the ANN baseline at .

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spiking Neutral Networks, Brain-inspired Computing, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_X3nqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work aims to address the limitations of existing reinforcement learning (RL)-based deep spiking Q-networks (DSQNs), which typically need multiple timesteps to achieve competitive performance. Consequently, RL-based DSQNs incur substantial computational and energy costs. Specifically, this paper proposes Common-Mode Compensation Deep Spiking Q-Network (CMC-DSQN), which employs a conventional artificial neural network (ANN) during training to compensate for common-mode errors in spiking neural network (SNN) outputs. CMC-DSQN first investigates performance degradation due to timestep reduction while decomposing the errors across actions into common-mode and differential-mode components, and observes that low-timestep DSQNs are dominated by common-mode errors. The paper addresses a relevant problem and proposes a simple and potentially useful solution. The method employs a small CliffWalking environment for error analysis, and is evaluated on seven Atari games and five MiniAtar environments. The empirical evidence on performance is promising, but the evaluation still lacks certain design aspects, which will be discussed below.

Critical strengths

• The target research problem is relevant and clearly motivated with a simple solution based on a common-mode error compensation mechanism.

• The idea of error decomposition is interesting and useful for identifying that common-mode errors affect temporal-difference learning more directly than differential-mode errors.

Critical weaknesses

• The case-study experiments using the CliffWalking environment provide evidence for a specific environment. Therefore, the respective analysis is limited in supporting broader claims across more environments and conditions.

• The case-study experiments focus on a particular DSQN formulation and a small CliffWalking environment. Therefore, it remains unclear whether common-mode error dominance still persists across different DSQN models, neuron models, network architectures, timestep ranges, or other RL algorithms.

• The evaluation does not discuss energy efficiency gains offered by the CMC-DSQN solution over the ANN-based solutions. Since the CMC-DSQN solution does not offer significantly higher performance than the ANN-based solutions, the lack of energy efficiency metrics makes the practicality of CMC-DSQN unclear.

Questions for authors

• It is recommended to clearly define the common-mode and differential-mode error components in the first section (i.e., Introduction) to help readers understand them better.

• Can the authors clarify whether common-mode error dominance holds for other DSQN models, network architectures, and RL algorithms?

• It is suggested that the case-study experiments consider multiple environments beyond CliffWalking to provide more generalizable analysis. Therefore, multiple observation results (like in Figure 4) should be obtained for different environments.

• Can the authors clarify why an auxiliary ANN is required, rather than an SNN-based model or other analytical-based techniques?

• What are the overhead costs (i.e., latency/throughput and power/energy) for employing an auxiliary ANN during the training phase?

• What are the efficiency gains obtained by the proposed CMC-DSQN as compared to the ANN-based solution? Because, performance-wise, the CMC-DSQN solution does not offer significantly higher scores than the ANN-based solutions. Hence, efficiency metrics may justify the practical benefits of CMC-DSQN.

• If possible, it is suggested to compare with another recent work on DSQN that employs active dendrites and a dueling network structure to complete the DSQN-based study, such as “Scalable Multi-Task Learning through Spiking Neural Networks with Adaptive Task-Switching Policy for Intelligent Autonomous Agents”, arXiv preprint arXiv:2504.13541.

11 comments

We appreciate the time and effort you invested in reviewing our paper. We would like to address your concerns as follows.

W1: The case-study experiments using the CliffWalking environment provide evidence for a specific environment. Therefore, the respective analysis is limited in supporting broader claims across more environments and conditions.

Thank you for raising this important point. We agree that the original CliffWalking case study alone is insufficient to establish the generality of our explanation. We have therefore expanded our analysis to additional experiments on FrozenLake, Taxi, seven Atari games, and all five MinAtar environments.

For FrozenLake and Taxi, we compute exact Q-values via dynamic programming and evaluate estimation errors against both the exact QπQ^\pi of each learned greedy policy and the optimal action-value function Q∗Q^*. As shown in Tables R1 and R2, CMC-DSQN reduces both common-mode (CM) and differential-mode (DM) errors compared with vanilla SNN. It also improves the success rate from 41.97% to 57.00% on FrozenLake and from 12.93% to 99.47% on Taxi. These results provide additional evidence that the benefits of common-mode compensation are not limited to CliffWalking. We have added these experiments and their learning curves to Sec. 5.2 and Fig. 5.

Table R1: Results on FrozenLake.

MethodSuccess rate(%)CM RMSE (QπQ^\pi)DM RMSE (QπQ^\pi)RMSE (QπQ^\pi)CM RMSE (Q∗Q^*)DM RMSE (Q∗Q^*)RMSE (Q∗Q^*)
SNN41.970.27860.08750.29290.12370.08410.1502
CMC-DSQN57.000.18630.05980.19660.09170.05950.1096

Table R2: Results on Taxi.

MethodSuccess rate(%)CM RMSE (QπQ^\pi)DM RMSE (QπQ^\pi)RMSE (QπQ^\pi)CM RMSE (Q∗Q^*)DM RMSE (Q∗Q^*)RMSE (Q∗Q^*)
SNN12.9330.359114.194533.902576.10741.218876.1189
CMC-DSQN99.476.68895.11158.44740.56380.35680.6704

For Atari and MinAtar, where exact Q-values are unavailable, we estimate reference QπQ^\pi using Monte Carlo rollouts under the current greedy policy. Tables R3 and R4 summarize the aggregate errors across different timesteps. CMC-DSQN consistently reduces CM errors on Atari (T=2,3,4) and MinAtar (T=2,4,6). We have added a new Error Analysis subsection (Sec. 6.2). Fig. 6 presents CM and DM errors across seven Atari games at T=2, showing consistent CM error reduction. Further results are added into Appendix C.2 (Figs. 19–23).

Table R3: Average CM and DM RMSEs across seven Atari environments.

MethodCMDM
DSQN (T=2)7.1280.781
CMC-DSQN (T=2)3.3670.932
DSQN (T=3)7.6150.808
CMC-DSQN (T=3)3.8660.944
DSQN (T=4)7.8990.807
CMC-DSQN (T=4)4.3170.981

Table R4: Average CM and DM RMSEs across five MinAtar environments.

MethodCMDM
DSQN (T=2)9.6851.682
CMC-DSQN (T=2)7.7091.427
DSQN (T=4)10.5541.934
CMC-DSQN (T=4)6.0891.083
DSQN (T=6)9.0701.622
CMC-DSQN (T=6)5.9331.010

We further extend the analysis to different timesteps, neuron models, RL algorithms, and network architectures. Common-mode dominance and the benefits of its compensation persist across these configurations, as detailed in our response to your W2.

W2: The case-study experiments focus on a particular DSQN formulation and a small CliffWalking environment. Therefore, it remains unclear whether common-mode error dominance still persists across different DSQN models, neuron models, network architectures, timestep ranges, or other RL algorithms.

Thank you for this important comment. We agree that the original case study provides limited evidence of the generality of our findings. We have therefore extended the analysis to different neuron models (LIF, IF, and CLIF), network widths (64, 128, and 256), simulation timesteps (T=2,3,4,5), and RL algorithms (DQN, Double DQN, and DQN with prioritized experience replay (PER)).

As reported in Tables R5–R8, common-mode (CM) errors consistently exceed differential-mode (DM) errors across all tested configurations without compensation, confirming that common-mode dominance is not specific to the original DSQN setting. Moreover, CMC consistently reduces both error components and substantially improves learning performance, with the benefits persisting across these configurations.

Table R5: Results with different neuron models.

NeuronCMCSuccess rateCM RMSEDM RMSE
LIFNo15%2.46321.2190
LIFYes90%0.66160.4854
IFNo25%2.13001.1281
IFYes100%0.45130.4305
CLIFNo15%2.27091.0703
CLIFYes90%0.39010.4077

Table R6: Results with different network widths.

Hidden widthCMCSuccess rateCM RMSEDM RMSE
64No15%2.46321.2190
64Yes90%0.66160.4854
128No65%0.80290.5635
128Yes100%0.31540.3287
256No100%0.50450.3751
256Yes100%0.27710.3442

Table R7: Results with different simulation timesteps.

TimestepCMCSuccess rateCM RMSEDM RMSE
T=2No15%2.46321.2190
T=2Yes90%0.66160.4854
T=3No15%1.98280.9714
T=3Yes100%0.39910.3602
T=4No60%1.28870.7024
T=4Yes95%0.33280.3040
T=5No30%1.30930.7062
T=5Yes100%0.40210.3312

Table R8: Results with different RL algorithms.

RL algorithmCMCSuccess rateCM RMSEDM RMSE
DQNNo15%2.46321.2190
DQNYes90%0.66160.4854
Double DQNNo25%1.20361.0955
Double DQNYes95%0.51440.4837
DQN + PERNo10%3.66991.2732
DQN + PERYes75%0.69230.5138

We have added a paragraph to Sec. 4 to summarize these findings: We further examine different SNN timesteps, neuron models, RL algorithms, and network architectures in Appendix C.1, consistently observing common-mode dominance across configurations. We have also visualized these additional results in Figs. 10-18.

W3: The evaluation does not discuss energy efficiency gains offered by the CMC-DSQN solution over the ANN-based solutions. Since the CMC-DSQN solution does not offer significantly higher performance than the ANN-based solutions, the lack of energy efficiency metrics makes the practicality of CMC-DSQN unclear.

Thank you for raising this important concern. We have added an analysis of inference energy consumption to evaluate the efficiency of CMC-DSQN relative to ANN-based solutions.

Following the widely adopted estimation framework in [1], we assign 12.5 pJ per floating-point operation (FLOP) and 77 fJ per synaptic operation (SOP) [2-3]. Energy consumption is estimated by multiplying the corresponding operation counts by their per-operation energy costs. For image-based environments (Atari and MinAtar), we exclude the first convolutional layer consistently for both ANN and SNN models, treating it as part of the visual sensing front-end.

Importantly, the auxiliary ANN in CMC-DSQN is used only during training and is discarded at inference. Thus, only the SNN is retained for deployment, introducing no additional ANN inference overhead.

As shown in Tables R9 and R10, CMC-DSQN substantially reduces estimated inference energy compared with ANN baselines. Moreover, compared with vanilla DSQN, CMC-DSQN further reduces energy consumption by approximately 21–22% on Atari and 54–59% on MinAtar, owing to its lower SOP counts. These results demonstrate that CMC-DSQN achieves competitive performance with substantially lower estimated inference energy than ANNs, while also improving the energy efficiency of SNNs.

Table R9: Estimated inference energy consumption on Atari.

MetricANNDSQN (T=2)CMC-DSQN (T=2)DSQN (T=3)CMC-DSQN (T=3)DSQN (T=4)CMC-DSQN (T=4)
FLOPs (M)12.14------
SOPs (M)-1.881.482.912.293.873.01
Energy (nJ)151,760145114224177298232

Table R10: Estimated inference energy consumption on MinAtar.

MetricANNDSQN (T=2)CMC-DSQN (T=2)DSQN (T=4)CMC-DSQN (T=4)DSQN (T=6)CMC-DSQN (T=6)
FLOPs (K)263------
SOPs (K)-3615693010950
Energy (nJ)32902.761.145.322.318.403.87

Q1: It is recommended to clearly define the common-mode and differential-mode error components in the first section (i.e., Introduction) to help readers understand them better.

Thank you for this suggestion. We have added definitions of common-mode and differential-mode errors to the Introduction (Lines 69–70): The common-mode error represents the average estimation error shared across all actions at a given state, whereas the differential-mode error captures action-specific deviations from this shared component.

Q2: Can the authors clarify whether common-mode error dominance holds for other DSQN models, network architectures, and RL algorithms?

Please refer to our response to your W2, where we provide additional analyses across different settings to confirm the generality of common-mode error dominance.

Q3: It is suggested that the case-study experiments consider multiple environments beyond CliffWalking to provide more generalizable analysis. Therefore, multiple observation results (like in Figure 4) should be obtained for different environments.

Thank you for this suggestion. As discussed in our response to your W1, Fig. 5 (previously Fig. 4) now includes additional results on FrozenLake and Taxi. We further provide common-mode error analyses on Atari and MinAtar in Fig. 6 and Appendix Figs. 19-23.

Reference

[1] Paul A Merolla, John V Arthur, Rodrigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spikingneuron integrated circuit with a scalable communication network and interface. Science, 345 (6197):668–673, 2014.

[2] Ning Qiao, Hesham Mostafa, Federico Corradi, Marc Osswald, Fabio Stefanini, Dora Sumislawska, and Giacomo Indiveri. A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128K synapses. Frontiers in Neuroscience, 9:141, 2015.

[3] Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions on Neural Networks and Learning Systems, 34(8):5200–5205, 2021.

Reviewer_kUTEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the performance degradation of deep spiking Q-networks (DSQNs) at low simulation timesteps. It decomposes Q-value estimation errors into an action-shared common-mode component and an action-dependent differential-mode component. Through exact value calculations and interventions on bootstrap targets in CliffWalking, the authors identify common-mode errors as a potential contributor to unstable learning. The proposed CMC-DSQN replaces the action-wise mean of the SNN outputs with a scalar estimate from an independent auxiliary ANN, with both networks trained jointly through temporal-difference learning. Because this replacement preserves action rankings, the auxiliary ANN can be removed during inference without changing greedy action selection. Evaluation covers seven Atari games and all five MiniAtar environments, with ablations examining the auxiliary network type, full-Q compensation, and application to ANN-based DQN. The reported results show substantial improvements over low-timestep DSQN baselines and average normalized scores slightly above the ANN baseline at four timesteps. Direct evidence for the proposed error mechanism is primarily established in CliffWalking.

Critical strengths

A useful mechanistic investigation with targeted interventions. The paper studies low-timestep DSQN degradation through common-mode and differential-mode Q-value errors. In CliffWalking, exact reference values allow the authors to selectively remove each error component from bootstrap targets and examine learning across 20 seeds. These interventions provide evidence beyond correlation that correcting common-mode errors can improve learning in this setting. This diagnostic perspective is the paper’s strongest potential scientific contribution. Substantial low-timestep gains with an exactly removable auxiliary network. CMC-DSQN preserves action rankings by construction, allowing the entire auxiliary ANN to be discarded at inference. On MiniAtar at T=2, its reported average performance ratio reaches 90.30%, compared with 67.76% for vanilla DSQN and 72.46% for the strongest competing baseline. Together with improvements in aggregate Atari performance, these results support the practical value of the approach: better performance under a small simulation budget without requiring an additional network during deployment.

Critical weaknesses

The evidence does not establish the central mechanism at the scope claimed. Equations (10)-(12) show how common-mode errors enter bootstrap targets, but do not establish that they are more harmful than differential-mode errors of comparable magnitude. The CliffWalking interventions remove components with substantially different error magnitudes, leaving error size and error type confounded. Moreover, direct error measurements and interventions are limited to CliffWalking, primarily at T=2; Atari and MiniAtar results demonstrate performance gains without verifying the proposed mechanism. This substantially weakens the paper’s main scientific claim that common-mode errors constitute a general bottleneck of low-timestep DSQNs. The contribution beyond dueling-style value decomposition is insufficiently isolated. Equation (16) has the same aggregation form as mean-centered dueling networks, with an ANN supplying the shared value component and an SNN supplying action-dependent outputs. Although the independent ANN/SNN encoders are a meaningful design choice, the paper lacks direct comparisons with a standard dueling-SNN and a shared-encoder variant. S-CMC and QEC-DSQN do not isolate these architectural factors, and QEC-DSQN’s mathematical definition and inference procedure are insufficiently specified. Consequently, the experiments do not establish whether the gains specifically validate the proposed error-compensation explanation or primarily reflect the benefits of value–advantage decomposition and additional ANN representation capacity. This uncertainty limits both the novelty assessment and the attribution of the reported improvements.

Questions for authors

Clarification of the central claim: Are common-mode errors claimed to be more harmful per unit of error, or primarily more consequential because their magnitude is larger? Equations (10)–(12) do not establish the former. Do you have evidence from interventions that match the magnitudes of common-mode and differential-mode errors? Clarification of QEC-DSQN: Please specify its output-combination equation, training objective, gradient flow, and inference procedure. Is the auxiliary ANN retained during evaluation? If it is removed, how is consistency between the training and inference policies ensured? Robustness of the performance conclusions: How is “average maximum return” computed, and are selected checkpoints independently reevaluated? Do the main improvements, particularly the claimed advantage over ANN at T=4, persist using final-checkpoint or final-window scores with uncertainty estimates across seeds?

7 comments

We would like to thank you for your constructive questions and suggestions. We will conduct a detailed clarification and discussion for your concerns.

W1.1: The evidence does not establish the central mechanism at the scope claimed. Equations (10)-(12) show how common-mode errors enter bootstrap targets, but do not establish that they are more harmful than differential-mode errors of comparable magnitude. The CliffWalking interventions remove components with substantially different error magnitudes, leaving error size and error type confounded.

Thank you for this insightful comment. We agree that the original experiments did not fully disentangle error type from error magnitude. To address this, we introduce an additional magnitude-matched control, SNN-CM†, which rescales the common-mode compensation to match the RMSE of the differential-mode errors.

\tilde e_c^\pi(s)=\alpha e_c^\pi(s), \qquad \alpha= \frac{\operatorname{RMSE}(e_d^\pi(s))} {\operatorname{RMSE}(e_c^\pi(s))}.$$ This allows us to compare the effects of the two error modes at comparable magnitudes. Table R1 reports the common-mode (CM) and differential-mode (DM) RMSEs of each variant, computed against the exact $Q^\pi$ of its learned greedy policy. SNN-CM† achieves lower residual CM and DM RMSEs than SNN-DM (43.9 vs. 55.9 and 19.1 vs. 21.0, respectively), indicating that CM compensation more effectively reduces both error components. Furthermore, Figs. 3(a)–(b) show that SNN-CM† achieves better learning performance than SNN-DM, even when the RMSE of CM compensation is matched to that of DM errors. These results suggest that the greater impact of CM errors cannot be attributed solely to their larger magnitude, but also reflects their distinct role in TD learning. Table R1: Common-mode and differential-mode RMSEs in CliffWalking. |Method|CM|DM| |:-:|:-:|:-:| |ANN|0.18|0.21| |SNN|60.1|25.0| |SNN-CM|40.3|17.9| |SNN-CM†|43.9|19.1| |SNN-DM|55.9|21.0| We have incorporated this additional experiment into Fig. 3 in Sec. 4.3. We have also clarified the theoretical discussion following Eqs. (10)–(12) in the previous version (now Eqs. (9)–(11)): `The common-mode error introduces an action-independent shift to the TD target, whereas the differential-mode error affects the target only through the perturbed maximization operation. By the 1-Lipschitz property of the max operator, its contribution is bounded by the maximum absolute differential-mode error. Thus, CM errors have a direct propagation pathway through TD bootstrapping, while the influence of DM errors depends on their effect on the maximization over actions.`

W1.2: Moreover, direct error measurements and interventions are limited to CliffWalking, primarily at T=2; Atari and MiniAtar results demonstrate performance gains without verifying the proposed mechanism. This substantially weakens the paper’s main scientific claim that common-mode errors constitute a general bottleneck of low-timestep DSQNs.

Thank you for raising this important point. We agree that the original CliffWalking case study alone is insufficient to establish the generality of our explanation. We have therefore expanded our analysis to additional tabular environments, Atari and MinAtar benchmarks, and different SNN simulation timesteps.

Additional error analysis beyond CliffWalking.

We conduct additional experiments on FrozenLake, Taxi, seven Atari games, and all five MinAtar environments.

For FrozenLake and Taxi, we compute exact Q-values via dynamic programming and evaluate estimation errors against both the exact QπQ^\pi of each learned greedy policy and the optimal action-value function Q∗Q^*. As shown in Tables R2 and R3, CMC-DSQN reduces both common-mode (CM) and differential-mode (DM) errors compared with vanilla SNN. It also improves the success rate from 41.97% to 57.00% on FrozenLake and from 12.93% to 99.47% on Taxi. These results provide additional evidence that the benefits of common-mode compensation are not limited to CliffWalking. We have added these experiments and their learning curves to Sec. 5.2 and Fig. 5.

Table R2: Results on FrozenLake.

MethodSuccess rate(%)CM RMSE (QπQ^\pi)DM RMSE (QπQ^\pi)RMSE (QπQ^\pi)CM RMSE (Q∗Q^*)DM RMSE (Q∗Q^*)RMSE (Q∗Q^*)
SNN41.970.27860.08750.29290.12370.08410.1502
CMC-DSQN57.000.18630.05980.19660.09170.05950.1096

Table R3: Results on Taxi.

MethodSuccess rate(%)CM RMSE (QπQ^\pi)DM RMSE (QπQ^\pi)RMSE (QπQ^\pi)CM RMSE (Q∗Q^*)DM RMSE (Q∗Q^*)RMSE (Q∗Q^*)
SNN12.9330.359114.194533.902576.10741.218876.1189
CMC-DSQN99.476.68895.11158.44740.56380.35680.6704

For Atari and MinAtar, where exact Q-values are unavailable, we estimate reference QπQ^\pi using Monte Carlo rollouts under the current greedy policy. Tables R4 and R5 summarize the aggregate errors across different timesteps. CMC-DSQN consistently reduces CM errors on Atari (T=2,3,4) and MinAtar (T=2,4,6). We have added a new Error Analysis subsection (Sec. 6.2). Fig. 6 presents CM and DM errors across seven Atari games at T=2, showing consistent CM error reduction. Further results are provided in Appendix C.2 (Figs. 19–23).

Table R4: Average CM and DM RMSEs across seven Atari environments.

MethodCMDM
DSQN (T=2)7.1280.781
CMC-DSQN (T=2)3.3670.932
DSQN (T=3)7.6150.808
CMC-DSQN (T=3)3.8660.944
DSQN (T=4)7.8990.807
CMC-DSQN (T=4)4.3170.981

Table R5: Average CM and DM RMSEs across five MinAtar environments.

MethodCMDM
DSQN (T=2)9.6851.682
CMC-DSQN (T=2)7.7091.427
DSQN (T=4)10.5541.934
CMC-DSQN (T=4)6.0891.083
DSQN (T=6)9.0701.622
CMC-DSQN (T=6)5.9331.010

Common-mode dominance across timesteps.

We further evaluate CM and DM errors of SNNs at T∈2,3,4,5,6T\in{2,3,4,5,6} in CliffWalking, using both exact QπQ^\pi and Q∗Q^* as references. As shown in Tables R6–R7, both error components generally increase as the timestep decreases, with a more pronounced increase in CM errors. We have added these results to Sec. 4 and Fig. 4 to further demonstrate the increasing dominance of CM errors at lower timesteps.

Table R6: CM and DM RMSEs of SNNs across different timesteps in CliffWalking (reference: exact QπQ^\pi). The CM/DM ratio is calculated per seed and then averaged across seeds.

T23456
CM60.1453.4729.7137.3416.73
DM25.0422.4915.9919.7812.72
CM/DM2.392.291.741.771.36

Table R7: CM and DM RMSEs of SNNs across different timesteps in CliffWalking (reference: exact Q∗Q^*). The CM/DM ratio is calculated per seed and then averaged across seeds.

T23456
CM2.461.981.291.310.88
DM1.220.970.700.710.56
CM/DM2.132.001.751.761.57

Additional generality analysis.

Furthermore, Appendix C.1 extends the analysis to different timesteps (T=3,4,5), neuron models (IF and CLIF), RL algorithms (Double DQN and PER), and network widths (128 and 256). Common-mode dominance and the benefits of its compensation persist across these configurations.

W2: The contribution beyond dueling-style value decomposition is insufficiently isolated. Equation (16) has the same aggregation form as mean-centered dueling networks, with an ANN supplying the shared value component and an SNN supplying action-dependent outputs. Although the independent ANN/SNN encoders are a meaningful design choice, the paper lacks direct comparisons with a standard dueling-SNN and a shared-encoder variant. S-CMC and QEC-DSQN do not isolate these architectural factors, and QEC-DSQN’s mathematical definition and inference procedure are insufficiently specified. Consequently, the experiments do not establish whether the gains specifically validate the proposed error-compensation explanation or primarily reflect the benefits of value–advantage decomposition and additional ANN representation capacity. This uncertainty limits both the novelty assessment and the attribution of the reported improvements.

Thank you for this insightful comment. We agree that the formulation in Eq. (16) of the previous version (Eq. (15) in the revised version) shares the mean-centered form of Dueling DQN. Our contribution, however, lies not in introducing a new Q-value factorization, but in identifying and addressing common-mode estimation errors in low-timestep DSQNs.

Distinction from standard Dueling architectures.

Unlike standard Dueling architectures with a shared encoder and separate value and advantage heads, CMC-DSQN uses independent ANN and SNN networks. The SNN estimates the differential component, while the ANN replaces the common-mode component, explicitly addressing the dominant common-mode errors in low-timestep DSQNs.

Clarification of S-CMC and QEC-DSQN.

S-CMC retains the explicit common-mode compensation mechanism but replaces the ANN common-mode estimator with an SNN, isolating the benefit of using an ANN for common-mode estimation.

QEC-DSQN does not combine the outputs of the two networks. The SNN directly produces the action values used for behavior policy selection, while the ANN is used exclusively to construct TD targets:

\max_{a'}Q_{\mathrm{ANN}}(s_{t+1},a'),$$ where $d_t$ indicates episode termination. The same detached TD targets supervise both the SNN and ANN Q-networks, which are updated independently. The SNN selects actions during both training and evaluation, while the ANN is used only for TD-target computation and is not required at inference. Thus, S-CMC examines the choice of common-mode estimator, whereas QEC-DSQN controls for generic ANN-assisted training without explicit common-mode compensation. ### Comparison with Dueling architectures. We additionally evaluate Dueling (SNN), with two SNN heads, and Dueling (mix), with an ANN value head and an SNN advantage head, both with a shared SNN encoder. Table R8 reports the average performance ratio (APR) and 95% confidence intervals across five MinAtar environments, using five random seeds and 5M environment transitions (per-environment results are provided in Table 9 of the revised manuscript). CMC-DSQN consistently outperforms both Dueling variants across all three timesteps. This demonstrates that conventional Dueling factorization and an ANN value head alone cannot fully explain the improvements. Table R8: APR (\%) with 95\% confidence intervals on MinAtar. |T|2|4|6| |:-|:-:|:-:|:-:| |DSQN|67.76 [65.82, 71.22]|77.31 [75.26, 80.82]|81.88 [80.01, 85.84]| |Dueling (SNN)|80.39 [78.80, 84.23]|84.64 [83.50, 87.91]|86.74 [85.59, 90.24]| |Dueling (mix)|82.81 [81.04, 86.67]|89.12 [87.22, 94.12]|90.07 [88.13, 94.25]| |CMC-DSQN|90.30 [87.83, 95.35]|101.9 [100.8, 106.4]|99.51 [98.15, 104.0]| We have added these comparisons in Sec. 6.4 of the revised manuscript (lines 519-523). ### Comparison with ANN-assisted training. QEC-DSQN introduces an auxiliary ANN for TD target computation without explicitly replacing the SNN common-mode component. Although it sometimes improves over vanilla DSQN, it consistently underperforms CMC-DSQN, indicating that ANN-assisted training alone cannot explain the gains. We have clarified this in the revised manuscript: `Are the gains simply due to ANN-assisted training? We consider QEC-DSQN, which uses an SNN for environment interaction and an auxiliary ANN for computing TD targets. The same detached TD targets are used to independently update both networks. As shown in Figures 7(a)-(c), although QEC-DSQN sometimes improves over vanilla DSQN, it consistently underperforms CMC-DSQN, indicating that ANN-assisted training alone cannot fully explain the gains.` Overall, these results show that CMC-DSQN's contribution lies in its heterogeneous ANN–SNN architecture specifically designed to compensate for dominant common-mode errors in low-timestep DSQNs, achieving improvements beyond conventional Dueling architectures and ANN-assisted training.
Reviewer_8FCaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates performance degradation and training instability in low-timestep deep spiking Q-networks. It decomposes Q-value estimation errors into common-mode errors shared across actions and differential errors affecting relative action values. The authors argue that common-mode errors can impair learning through bootstrapped temporal-difference targets, despite leaving greedy action rankings unchanged. Based on this analysis, they propose common-mode compensation (CMC), which combines mean-centered SNN outputs with a state-dependent scalar predicted by an auxiliary ANN. Both branches are trained jointly, and the ANN can be removed at inference without changing greedy action selection.

The paper validates its diagnosis through exact-value analysis and targeted error-correction interventions in deterministic CliffWalking. It further evaluates CMC on seven Atari and five MiniAtar games across multiple timestep settings and random seeds. Ablations examine the auxiliary estimator, the scope of compensation, and application to ANN-based DQN. The reported results show improved learning stability in CliffWalking and higher average normalized peak returns than the evaluated SNN baselines, with particularly substantial gains in the lowest-timestep MiniAtar setting.

Critical strengths
  1. A focused and informative analysis of a relevant failure mode. The paper decomposes Q-value estimation errors into common-mode and differential components and connects this decomposition to bootstrapped TD targets. Exact-value diagnostics and targeted error-correction interventions in CliffWalking provide evidence that common-mode errors contribute to learning instability in this controlled setting. This analysis is a substantive strength because it identifies an interpretable limitation of low-timestep SNNs and provides a concrete motivation for the proposed method.

  2. Meaningful improvements in the targeted low-timestep regime. On MiniAtar at (T=2), CMC achieves an APR of 90.30%, compared with 72.46% for the strongest competing baseline in Table 2—a gain of 17.84 percentage points. Ablations involving an auxiliary SNN, full-vector compensation, and ANN-based DQN help assess the proposed design choices. These results support the method’s practical value, particularly because the auxiliary ANN can be removed at deployment without changing greedy action selection.

Critical weaknesses
  1. The incremental methodological contribution is not sufficiently isolated. ANN-assisted SNN training has been explored previously, and the proposed aggregation follows the same mean-centered value/advantage form used in Dueling DQN. Although the common-mode error perspective is valuable, a direct comparison with a standard Dueling-SNN is missing, and the S-CMC ablation does not fully resolve this distinction. Consequently, it remains unclear how much of the improvement specifically requires the proposed heterogeneous decomposition, rather than established value/advantage factorization or the additional ANN training branch. This uncertainty limits the strength of the methodological novelty claim.

  2. The evaluation does not sufficiently substantiate reliable performance and training stability. The main tables report average maximum returns, which capture peak performance but do not establish stable convergence or final-checkpoint quality. Atari results use three seeds without reporting dispersion, while MiniAtar provides standard deviations but no uncertainty estimates for aggregate APR or performance differences. Fixed final-checkpoint or late-training-window results, together with appropriate uncertainty estimates, are needed to determine whether the gains persist consistently. This matters because improved stability and performance approaching or slightly exceeding the ANN baseline are central empirical arguments for the paper.

Questions for authors
  1. Clarification of the mechanistic evidence and its scope. The common-mode error diagnosis is supported by exact-value analysis and targeted interventions in deterministic CliffWalking, while the Atari and MiniAtar experiments primarily demonstrate performance improvements. What evidence supports attributing these larger-benchmark improvements specifically to common-mode error compensation? Were any error diagnostics performed outside CliffWalking, or across different timestep settings, to examine whether common-mode dominance becomes more pronounced as TT decreases? Please specify the reference used for any such diagnostics and clarify which mechanistic conclusions are directly supported by the existing experiments. This would affect my assessment of how broadly the proposed explanation applies.

  2. Clarification of the incremental contribution and the relevant controls. Given prior ANN-assisted SNN training approaches and the relationship between Eq. (16) and value–advantage decomposition, what existing evidence isolates the benefit of assigning the shared component to an independent ANN? In particular, how does the S-CMC control differ from a standard shared-encoder Dueling-SNN, and is a controlled comparison with the latter available? If so, how much additional improvement does CMC provide under comparable experimental conditions? This comparison would help determine whether the proposed architecture offers a substantive advance beyond established hybrid-training and decomposition strategies.

  3. Additional analysis of existing runs to assess robustness. For the existing Atari and MiniAtar experiments, could the authors report performance at a fixed final checkpoint or averaged over a clearly specified late-training window, together with seed-level results and confidence intervals for the principal aggregate comparisons? Do the improvements over SNN baselines, and especially the relatively small margins above the ANN baseline at T=4T=4, persist under these evaluation choices? Since the main tables report average maximum returns, this analysis would materially affect my assessment of whether the improvements reflect consistently maintained performance and stable learning.

14 comments

Thank you for your detailed review and valuable suggestions. We would like to address your concerns as follows.

Response to Weakness 1 and Question 2

W1: The incremental methodological contribution is not sufficiently isolated. ANN-assisted SNN training has been explored previously, and the proposed aggregation follows the same mean-centered value/advantage form used in Dueling DQN. Although the common-mode error perspective is valuable, a direct comparison with a standard Dueling-SNN is missing, and the S-CMC ablation does not fully resolve this distinction. Consequently, it remains unclear how much of the improvement specifically requires the proposed heterogeneous decomposition, rather than established value/advantage factorization or the additional ANN training branch. This uncertainty limits the strength of the methodological novelty claim.

Q2: Clarification of the incremental contribution and the relevant controls. Given prior ANN-assisted SNN training approaches and the relationship between Eq. (16) and value–advantage decomposition, what existing evidence isolates the benefit of assigning the shared component to an independent ANN? In particular, how does the S-CMC control differ from a standard shared-encoder Dueling-SNN, and is a controlled comparison with the latter available? If so, how much additional improvement does CMC provide under comparable experimental conditions? This comparison would help determine whether the proposed architecture offers a substantive advance beyond established hybrid-training and decomposition strategies.

Thank you for this insightful comment. We agree that the formulation in Eq. (16) of the previous version (Eq. (15) in the revised version) shares the mean-centered form of Dueling DQN. Our contribution, however, lies not in introducing a new Q-value factorization, but in identifying and addressing common-mode estimation errors in low-timestep DSQNs.

Distinction from standard Dueling architectures. Unlike standard Dueling architectures with a shared encoder and separate value and advantage heads, CMC-DSQN uses independent ANN and SNN networks. The SNN estimates the differential component, while the ANN replaces the common-mode component, explicitly addressing the dominant common-mode errors in low-timestep DSQNs.

Comparison with Dueling architectures beyond S-CMC. S-CMC retains the proposed compensation mechanism but replaces the auxiliary ANN with an SNN, making it different from a standard shared-encoder Dueling-SNN. To provide a direct comparison, we additionally evaluate Dueling (SNN), with two SNN heads, and Dueling (mix), with an ANN value head and an SNN advantage head, both with a shared SNN encoder. Table R1 reports the average performance ratio (APR) and 95% confidence intervals across five MinAtar environments, using five random seeds and 5M environment transitions (per-environment results are provided in Table 9 of the revised manuscript). CMC-DSQN consistently outperforms both Dueling variants across all three timesteps. This demonstrates that conventional Dueling factorization and an ANN value head alone cannot fully explain the improvements. We have added these comparisons in Sec. 6.4 of the revised manuscript (lines 519-523).

Table R1: APR (%) with 95% confidence intervals on MinAtar.

T246
DSQN67.76 [65.82, 71.22]77.31 [75.26, 80.82]81.88 [80.01, 85.84]
Dueling (SNN)80.39 [78.80, 84.23]84.64 [83.50, 87.91]86.74 [85.59, 90.24]
Dueling (mix)82.81 [81.04, 86.67]89.12 [87.22, 94.12]90.07 [88.13, 94.25]
CMC-DSQN90.30 [87.83, 95.35]101.9 [100.8, 106.4]99.51 [98.15, 104.0]

Comparison with ANN-assisted training. QEC-DSQN introduces an auxiliary ANN for TD target computation without explicitly replacing the SNN common-mode component. Although it sometimes improves over vanilla DSQN, it consistently underperforms CMC-DSQN, indicating that ANN-assisted training alone cannot explain the gains. We have clarified this in the revised manuscript:

Are the gains simply due to ANN-assisted training? We consider QEC-DSQN, which uses an SNN for environment interaction and an auxiliary ANN for computing TD targets. The same detached TD targets are used to independently update both networks. As shown in Figures 7(a)-(c), although QEC-DSQN sometimes improves over vanilla DSQN, it consistently underperforms CMC-DSQN, indicating that ANN-assisted training alone cannot fully explain the gains.

Overall, these results show that CMC-DSQN's contribution lies in its heterogeneous ANN–SNN architecture specifically designed to compensate for dominant common-mode errors in low-timestep DSQNs, achieving improvements beyond conventional Dueling architectures and ANN-assisted training.

Response to Weakness 2 and Question 3

W2: The evaluation does not sufficiently substantiate reliable performance and training stability. The main tables report average maximum returns, which capture peak performance but do not establish stable convergence or final-checkpoint quality. Atari results use three seeds without reporting dispersion, while MiniAtar provides standard deviations but no uncertainty estimates for aggregate APR or performance differences. Fixed final-checkpoint or late-training-window results, together with appropriate uncertainty estimates, are needed to determine whether the gains persist consistently. This matters because improved stability and performance approaching or slightly exceeding the ANN baseline are central empirical arguments for the paper.

Q3: Additional analysis of existing runs to assess robustness. For the existing Atari and MiniAtar experiments, could the authors report performance at a fixed final checkpoint or averaged over a clearly specified late-training window, together with seed-level results and confidence intervals for the principal aggregate comparisons? Do the improvements over SNN baselines, and especially the relatively small margins above the ANN baseline at (T=4), persist under these evaluation choices? Since the main tables report average maximum returns, this analysis would materially affect my assessment of whether the improvements reflect consistently maintained performance and stable learning.

Thank you for this important comment. We agree that maximum returns alone are insufficient to assess sustained performance. We therefore additionally report the average returns over the final five evaluation points (FINAL-5), with 95% bootstrapped confidence intervals (CIs) following [1].

Tables R2 and R3 compare the MAX and FINAL-5 results with their corresponding CIs. For ANN-to-SNN conversion baselines, MAX uses the best ANN checkpoints, whereas FINAL-5 uses the final ANN checkpoints.

CMC-DSQN consistently outperforms vanilla DSQN under FINAL-5 evaluation across all tested timesteps. Notably, its gains over ANN at T=4 are also maintained, achieving 109.6% APR on Atari (95% CI: [102.6, 117.4]) and 107.4% on MinAtar (95% CI: [105.8, 115.5]), with both CI lower bounds exceeding 100%. These results demonstrate that the improvements persist beyond transient performance peaks.

Table R2: APR (%) on Atari under MAX and FINAL-5 evaluation protocols, with 95% CIs.

TimestepMethodMAX APR [95% CI]FINAL-5 APR [95% CI]
2DSQN70.75 [64.01, 80.45]54.88 [46.41, 62.16]
2CMC-DSQN87.06 [78.33, 97.44]79.16 [70.16, 88.88]
3DSQN83.32 [76.03, 94.00]60.83 [55.19, 68.56]
3CMC-DSQN95.57 [92.04, 99.22]100.9 [94.49, 107.3]
4DSQN88.76 [77.50, 103.0]65.91 [60.09, 73.69]
4CMC-DSQN103.5 [99.95, 107.9]109.6 [102.6, 117.4]

Table R3: APR (%) on MinAtar under MAX and FINAL-5 evaluation protocols, with 95% CIs.

TimestepMethodMAX APR [95% CI]FINAL-5 APR [95% CI]
2DSQN-CVT10.12 [8.158, 13.00]11.48 [10.03, 13.93]
2DCGS57.33 [56.05, 59.81]73.53 [72.24, 79.14]
2CRPI65.72 [64.73, 68.14]86.22 [84.97, 93.30]
2DSQN67.76 [65.82, 71.22]70.27 [69.18, 74.82]
2pbLN65.43 [64.64, 67.84]66.35 [65.70, 70.71]
2DATSQN59.88 [58.89, 62.22]60.81 [59.77, 64.70]
2CaRe-BN72.46 [71.97, 75.11]79.43 [78.79, 84.82]
2TP-DSQN69.01 [67.36, 72.63]74.55 [73.50, 78.94]
2SwitchMT72.58 [71.95, 75.78]72.77 [72.29, 77.45]
2CMC-DSQN90.30 [87.83, 95.35]92.37 [90.48, 99.25]
4DSQN-CVT27.15 [26.29, 28.55]32.06 [30.74, 34.72]
4DCGS67.11 [65.16, 70.66]88.51 [85.93, 95.08]
4CRPI80.72 [79.16, 83.97]105.5 [104.0, 111.9]
4DSQN77.31 [75.26, 80.82]79.37 [78.17, 84.98]
4pbLN70.08 [69.51, 72.53]75.33 [74.22, 80.29]
4DATSQN80.35 [78.45, 84.79]80.96 [80.24, 86.38]
4CaRe-BN81.44 [80.29, 84.48]80.72 [79.74, 86.23]
4TP-DSQN89.64 [88.56, 93.88]101.4 [99.05, 109.1]
4SwitchMT83.55 [82.63, 86.97]90.81 [89.75, 96.68]
4CMC-DSQN101.9 [100.8, 106.4]107.4 [105.8, 115.5]
6DSQN-CVT34.48 [33.27, 36.35]38.71 [37.05, 41.61]
6DCGS73.30 [71.75, 77.11]90.04 [87.60, 96.13]
6CRPI83.66 [82.00, 87.57]107.0 [106.2, 113.7]
6DSQN81.88 [80.01, 85.84]81.44 [79.82, 87.02]
6pbLN78.39 [77.26, 81.90]80.32 [79.64, 86.10]
6DATSQN82.51 [80.42, 86.72]90.10 [88.63, 97.02]
6CaRe-BN81.87 [80.93, 84.81]83.14 [81.88, 88.39]
6TP-DSQN93.80 [92.29, 98.70]104.6 [103.6, 112.3]
6SwitchMT86.87 [86.06, 90.35]90.73 [89.05, 97.50]
6CMC-DSQN99.51 [98.15, 104.0]108.1 [106.7, 114.5]

In the revised manuscript, Tables 1 and 2 additionally report per-environment standard deviations and 95% CIs for APR, while Tables 7 and 8 in Appendix C.3 provide per-environment FINAL-5 results. We have also revised Sec. 6.3 to highlight the sustained improvements and the performance gains over ANN at T=4 in the FINAL-5 evaluation.

Response to Question 1 (part 1)

Q1.1: Clarification of the mechanistic evidence and its scope. The common-mode error diagnosis is supported by exact-value analysis and targeted interventions in deterministic CliffWalking, while the Atari and MiniAtar experiments primarily demonstrate performance improvements. What evidence supports attributing these larger-benchmark improvements specifically to common-mode error compensation? Were any error diagnostics performed outside CliffWalking, or across different timestep settings, to examine whether common-mode dominance becomes more pronounced as T decreases?

Thank you for raising this important point. We agree that the original CliffWalking case study alone is insufficient to establish the generality of our explanation. We have therefore expanded our analysis to additional tabular environments, Atari and MinAtar benchmarks, and different SNN simulation timesteps.

Additional error analysis beyond CliffWalking.

We conduct additional experiments on FrozenLake, Taxi, seven Atari games, and all five MinAtar environments.

For FrozenLake and Taxi, we compute exact Q-values via dynamic programming and evaluate estimation errors against both the exact QπQ^\pi of each learned greedy policy and the optimal action-value function Q∗Q^*. As shown in Tables R4 and R5, CMC-DSQN reduces both common-mode (CM) and differential-mode (DM) errors compared with vanilla SNN. It also improves the success rate from 41.97% to 57.00% on FrozenLake and from 12.93% to 99.47% on Taxi. These results provide additional evidence that the benefits of common-mode compensation are not limited to CliffWalking. We have added these experiments and their learning curves to Sec. 5.2 and Fig. 5.

Table R4: Results on FrozenLake.

MethodSuccess rate(%)CM RMSE (QπQ^\pi)DM RMSE (QπQ^\pi)RMSE (QπQ^\pi)CM RMSE (Q∗Q^*)DM RMSE (Q∗Q^*)RMSE (Q∗Q^*)
SNN41.970.27860.08750.29290.12370.08410.1502
CMC-DSQN57.000.18630.05980.19660.09170.05950.1096

Table R5: Results on Taxi.

MethodSuccess rate(%)CM RMSE (QπQ^\pi)DM RMSE (QπQ^\pi)RMSE (QπQ^\pi)CM RMSE (Q∗Q^*)DM RMSE (Q∗Q^*)RMSE (Q∗Q^*)
SNN12.9330.359114.194533.902576.10741.218876.1189
CMC-DSQN99.476.68895.11158.44740.56380.35680.6704

For Atari and MinAtar, where exact Q-values are unavailable, we estimate reference QπQ^\pi using Monte Carlo rollouts under the current greedy policy. Tables R6 and R7 summarize the aggregate errors across different timesteps. CMC-DSQN consistently reduces CM errors on Atari (T=2,3,4) and MinAtar (T=2,4,6). We have added a new Error Analysis subsection (Sec. 6.2). Fig. 6 presents CM and DM errors across seven Atari games at T=2, showing consistent CM error reduction. Further results are added into Appendix C.2 (Figs. 19–23).

Table R6: Average CM and DM RMSEs across seven Atari environments.

MethodCMDM
DSQN (T=2)7.1280.781
CMC-DSQN (T=2)3.3670.932
DSQN (T=3)7.6150.808
CMC-DSQN (T=3)3.8660.944
DSQN (T=4)7.8990.807
CMC-DSQN (T=4)4.3170.981

Table R7: Average CM and DM RMSEs across five MinAtar environments.

MethodCMDM
DSQN (T=2)9.6851.682
CMC-DSQN (T=2)7.7091.427
DSQN (T=4)10.5541.934
CMC-DSQN (T=4)6.0891.083
DSQN (T=6)9.0701.622
CMC-DSQN (T=6)5.9331.010

Common-mode dominance across timesteps.

We further evaluate CM and DM errors of SNNs at T∈2,3,4,5,6T\in{2,3,4,5,6} in CliffWalking, using both exact QπQ^\pi and Q∗Q^* as references. As shown in Tables R8–R9, both error components generally increase as the timestep decreases, with a more pronounced increase in CM errors. We have added these results to Sec. 4 and Fig. 4 to further demonstrate the increasing dominance of CM errors at lower timesteps.

Table R8: CM and DM RMSEs of SNNs across different timesteps in CliffWalking (reference: exact QπQ^\pi). The CM/DM ratio is calculated per seed and then averaged across seeds.

T23456
CM60.1453.4729.7137.3416.73
DM25.0422.4915.9919.7812.72
CM/DM2.392.291.741.771.36

Table R9: CM and DM RMSEs of SNNs across different timesteps in CliffWalking (reference: exact Q∗Q^*). The CM/DM ratio is calculated per seed and then averaged across seeds.

T23456
CM2.461.981.291.310.88
DM1.220.970.700.710.56
CM/DM2.132.001.751.761.57

Additional generality analysis.

Furthermore, Appendix C.1 extends the analysis to different timesteps (T=3,4,5), neuron models (IF and CLIF), RL algorithms (Double DQN and PER), and network widths (128 and 256). Common-mode dominance and the benefits of its compensation persist across these configurations.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your careful evaluation and constructive feedback, which have helped us substantially strengthen the manuscript. As the discussion period draws to a close, we would greatly appreciate your feedback on whether our clarifications address your concerns and whether any questions remain. For ease of assessment, we summarize the main additions below.

  1. Stronger evidence for common-mode error dominance and compensation. We added a magnitude-matched intervention in CliffWalking, demonstrating that the benefits of CM correction extend beyond differences in error magnitude. We also expanded our experiments to different environments and examined different neuron models, network widths, simulation timesteps, and RL algorithms. For larger benchmarks, we added error analysis on Atari and MinAtar in Section 6.2. These analyses provide additional evidence for CM error dominance and the effectiveness of CM compensation across the evaluated settings.

  2. Direct comparisons with Dueling architectures and ANN-assisted training. We added Dueling (SNN) and Dueling (mix), alongside S-CMC with the same parameter count as CMC-DSQN and the auxiliary ANN baseline QEC-DSQN. CMC-DSQN achieves the highest performance among these alternatives at every tested timestep, supporting the proposed design choices.

  3. Evaluation beyond peak returns. We supplemented maximum-return results with FINAL-5 scores, learning curves, confidence intervals, and individual results of each seed. The aggregate improvements over vanilla DSQN persist under FINAL-5 evaluation at every tested timestep. At T=4, the mean FINAL-5 APRs reach 109.6% on Atari and 107.4% on MinAtar.

  4. Broader baseline coverage. We incorporated a single-task adaptation of SwitchMT, which CMC-DSQN outperforms across all tested timesteps. On MinAtar at T=2, CMC-DSQN achieves a mean maximum-return APR of 90.30%, compared with 72.58% for SwitchMT, the strongest competing baseline, a gain of 17.72 percentage points.

  5. Computational costs and deployment efficiency. We added measurements of training throughput, GPU power consumption, and energy costs, together with an inference-energy analysis. CMC-DSQN achieves lower estimated inference energy than vanilla DSQN. At T=4, its estimated inference energy is approximately 1/650 of the ANN baseline on Atari and 1/1,420 on MinAtar, under our operation-based model excluding the first convolutional layer for both ANN and SNN models.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.