acceptodds
Under review as a conference paper at ICLR 2027

DTSA: Dual-Timescale Stability and Adaptation for Robust Offline Reinforcement Learning with Spiking Neural Networks

Abstract

Combining offline reinforcement learning with spiking neural networks exposes a distributional instability that a state–action mismatch account does not capture: fixed data support interacts with time-varying neuronal dynamics. We formalize this as a dual-timescale shift: slow offline optimization drifts spike-conditioned action relations away from the supported structure, while deployment perturbations move a frozen policy into neural-dynamic regimes not covered by training. Treating the spike response induced by the decision history as an explicit condition variable, we propose DTSA, whose two mechanisms operate on one shared state–spike condition space: Conditional Structure Preservation (CSP) stabilizes spike-conditioned local action structure during training, and Adaptive Episodic Memory (AEM) corrects actions within an episode through confidence-gated, magnitude-limited fusion with all parameters frozen, functionally inspired by the stability–plasticity principles of biological learning. Both mechanisms are analyzed under a unified dual-timescale error decomposition. Experiments show that DTSA consistently improves the average normalized return across multiple benchmark tasks, by points on Adroit over the matched spiking backbone, and exhibits better overall performance retention and distribution-shift robustness under observation noise, sensor dropout, and environment-dynamics shifts.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spiking Neural Networks, Neuromorphic Computing, Event-driven Computing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_M5UtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper frames SNN-based offline reinforcement learning as a dual-timescale problem involving training-time drift from fixed behavioral support and deployment-time changes in internal spike dynamics. It proposes DTSA, which uses a shared state–spike condition across two components: CSP regularizes conditional action structure during training, and AEM performs episode-local action correction with frozen network parameters. The paper provides a unified error analysis and evaluates DTSA on D4RL tasks through matched component ablations, deployment perturbations, and a memory-pairing intervention.

Critical strengths
  1. A coherent SNN-specific training–deployment framework. DTSA explicitly uses the internal spike response as a shared condition for training-time action-structure regularization and deployment-time memory retrieval. This provides a meaningful framework-level connection between offline policy learning and frozen-parameter adaptation.

  2. Empirical performance improvements. DTSA improves over the matched mechanism-free SNN backbone by 4.0 average points on MuJoCo and 10.0 points on Adroit, with both CSP and AEM contributing. AEM operates without parameter updates, and shuffling spike–action pairings removes its benefit, providing direct evidence that the correction uses the correspondence stored in memory.

Critical weaknesses

1. Limited novelty

1.1 The dual-timescale formulation combines established distribution shifts. The paper’s “slow-timescale shift”—the learned policy drifting away from the fixed behavioral support during offline optimization—is the standard policy–dataset distribution shift widely studied in offline RL [1]. Its “fast-timescale shift”—deployment perturbations changing temporally evolving spike states, actions, and subsequent state distributions—has also been studied in SNN dynamics and continuous control [2,3]. The proposed dual-timescale framework combines these established shifts under a shared state–spike condition, without identifying a new shift mechanism or demonstrating a new interaction between them.

1.2 CSP and AEM are incremental adaptations of existing methods. CSP applies conditional Cauchy–Schwarz divergence, previously developed for kernel-based conditional-distribution matching and sequential decision-making [4], to state–spike-conditioned action distributions. AEM builds on established episodic-memory and nearest-neighbor kernel-retrieval methods in RL [5], including their extension to offline RL [6]. The methodological contribution is an incremental adaptation and integration of these existing techniques for SNN-based offline RL.

2. Evidence does not establish the claimed training–deployment mechanism

The paper claims that CSP preserves state–spike-conditioned action relations during training for AEM to recover at deployment; the current evidence establishes only that CSP and AEM are individually effective. Figure 2 shows that CSP reduces held-out DCS, without measuring whether AEM’s local spike–action fitting becomes more accurate. The bank-pairing intervention shows that AEM requires correct spike–action pairings, without showing that these pairings were improved or preserved by CSP. The component ablations attribute performance gains to both modules and provide no test of the proposed CSP-to-AEM pathway. The shared training–deployment mechanism therefore remains unverified.

3. Mathematical and methodological flaws

3.1 The CSP loss is inconsistent with its claimed theoretical objective. The defined embeddings imply ∣μ^p(xi)∣Hk2=∑j,kwijwikκ(aj,ak)\\|\hat{\mu}_p(x_i)\\|_{\mathcal H_k}^{2}=\sum_{j,k}w_{ij}w_{ik}\kappa(a_j,a_k) and ⟨μ^p(xi),μ^q(xi)⟩Hk=∑j,kwijwikκ(aj,a^k)\langle\hat{\mu}_p(x_i),\hat{\mu}_q(x_i)\rangle_{\mathcal H_k}=\sum_{j,k}w_{ij}w_{ik}\kappa(a_j,\hat a_k). Eq. (4) instead uses the single-anchor sums Vp(xi)=∑jwijκ(ai,aj)V_p(x_i)=\sum_j w_{ij}\kappa(a_i,a_j) and Vpq(xi)=∑jwijκ(ai,a^j)V_{pq}(x_i)=\sum_j w_{ij}\kappa(a_i,\hat a_j). These quantities do not equal the stated embedding norm and inner product, so the CS-overlap interpretation and its derived guarantees do not follow. A two-position counterexample with x1=x2x_1=x_2, ω0=0.1\omega_0=0.1, σa=1\sigma_a=1, M11=M21=M22=1M_{11}=M_{21}=M_{22}=1, M12=0M_{12}=0, (a1,a2)=(−0.5,0.5)(a_1,a_2)=(-0.5,0.5), and (a^1,a^2)=(−0.5,1)(\hat a_1,\hat a_2)=(-0.5,1) gives LCSP≈−0.238\mathcal L_{\mathrm{CSP}}\approx-0.238, while perfect prediction gives 00. The CSP term can therefore assign a lower value to an incorrect prediction and violates the claimed non-negativity.

3.2 AEM’s coverage gate does not measure absolute memory coverage. The coverage gate is defined in Section 3.3 and Eq. (14) as gcov=exp⁡(−rt/2)g_{\mathrm{cov}}=\exp(-r_t/2), where rt=dnn,t/τtr_t=d_{\mathrm{nn},t}/\tau_t, to determine whether the current query is sufficiently covered by the episodic memory and should permit action correction. However, it fails to measure absolute coverage because scaling all retrieved distances by the same factor leaves the gate unchanged, and dnn,t≤τtd_{\mathrm{nn},t}\leq\tau_t normally gives e−1/2≤gcov≤1e^{-1/2}\leq g_{\mathrm{cov}}\leq1. For example, distance sets (1,2,3)(1,2,3) and (100,200,300)(100,200,300) both produce gcov=e−1/4≈0.779g_{\mathrm{cov}}=e^{-1/4}\approx0.779. The gate therefore cannot distinguish a well-covered query from one where every memory entry is distant, undermining its stated role as a coverage test.

4. Factual errors

  • 4.1 Maze2D and AntMaze are conflated. Table 5 reports three maze2d tasks, while its title and discussion call them AntMaze. Maze2D and AntMaze have different agents, action spaces, and control problems [7].

  • 4.2 The count and percentage in Figure 2(a) are inconsistent. The figure reports “2001 conflicts / 5000 samples (14.5%),” although 2001/5000=40.022001/5000=40.02\\%.

  • 4.3 The abstract misattributes the Adroit improvement. The reported 18.1-point gain compares DTSA’s 66.7 with DSFormer’s 48.6, while the matched mechanism-free control scores 56.7. The gain attributable to CSP and AEM is 66.7−56.7=10.066.7-56.7=10.0; the remaining 8.1 points arise from backbone or protocol differences.

References

[1] S. Levine, A. Kumar, G. Tucker, and J. Fu. “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems.” 2020.

[2] J. K. Eshraghian, M. Ward, E. O. Neftci, X. Wang, G. Lenz, G. Dwivedi, M. Bennamoun, D. S. Jeong, and W. D. Lu. “Training Spiking Neural Networks Using Lessons From Deep Learning.” Proceedings of the IEEE, 2023.

[3] Z. Xu, Z. Huang, Y. Dong, K. Chen, W. Liu, and Z. Yu. “Error Amplification Limits ANN-to-SNN Conversion in Continuous Control.” ICML, 2026.

[4] S. Yu, H. Li, S. Løkse, R. Jenssen, and J. C. Príncipe. “The Conditional Cauchy–Schwarz Divergence With Applications to Time-Series Data and Sequential Decision Making.” IEEE TPAMI, 2025.

[5] A. Pritzel, B. Uria, S. Srinivasan, A. Puigdomènech Badia, O. Vinyals, D. Hassabis, D. Wierstra, and C. Blundell. “Neural Episodic Control.” ICML, 2017.

[6] X. Ma, Y. Yang, H. Hu, Q. Liu, J. Yang, C. Zhang, Q. Zhao, and B. Liang. “Offline Reinforcement Learning with Value-Based Episodic Memory.” ICLR, 2022.

[7] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. “D4RL: Datasets for Deep Data-Driven Reinforcement Learning.” 2020.

Questions for authors
  1. How does the dual-timescale formulation differ technically from established offline-RL policy–dataset shift [1] and SNN temporal or deployment shift [2,3]? What new algorithmic elements do CSP and AEM introduce beyond the methods in [4–6]?

  2. What mathematical relationship connects the single-anchor quantities in Eq. (4) to the stated embedding norms and inner products? Which formulation is used in the implementation, and how is the negative-loss counterexample addressed?

  3. How does gcovg_{\mathrm{cov}} detect inadequate memory coverage when all retrieved distances increase proportionally? Does any other gate identify this case?

  4. What evidence shows that CSP improves AEM’s deployment-time fitting or corrections, beyond the independent gains of the two components?

  5. Please clarify the environment used in Table 5, the denominator underlying Figure 2(a)’s 14.5%, and why the abstract reports an 18.1-point matched-backbone improvement when the matched mechanism gain is 10.0 points.

2 comments
AuthorsOpenReview ↗

Response to Reviewer M5Ut — Estimator Corrections and Framework Positioning

We thank the reviewer for the precise review. We agree that Eq. (4), the coverage gate, and the novelty positioning require correction. Below we separate the mathematical corrections (3.1–3.2), factual corrections (4.1–4.3), and framework contribution (1.1–1.2, 2). Details are in Appendices B.1, D.1–D.2, H, and I.

1.1 / 1.2 Novelty. We agree that CSP builds on conditional kernel-divergence matching and AEM on episodic retrieval; neither component is new in isolation. The contribution is the integration; §2.4 and Appendix I.1 separate reused from added components. The integration has three layers. (i) Unified condition space: both training and deployment use the joint state–spike condition (s,ξ)(s,\xi), not a state-only representation. (ii) Mechanism complementarity: CSP preserves local behavior structure during training, while AEM performs confidence-gated correction with frozen parameters. (iii) Closed-loop constraints: reliability records, write admission, and bounded correction reduce, but cannot eliminate, the risk that unreliable corrections propagate. The framework-level claim is that one condition space couples training-time structure preservation with deployment-time gated correction. We add the [1]–[6] comparison in Appendix I.1 and remove any new-shift-mechanism claim. Since Appendix E.5 finds MMD marginally better than CS, we do not claim CS-specific superiority.

2. CSP→AEM evidence. The evidence supports two separate findings. First, CSP reduces the held-out single-anchor DCSD_{\mathrm{CS}} proxy. Second, the shuffled-ξ\xi intervention shows that AEM's benefit depends on the correct spike–action correspondence. It does not show that CSP improves AEM's local fit: the findings are independent. We label mediation as a hypothesis in §1, §2.4, and §5 and remove causal wording from the abstract and conclusion. The mechanisms can be used independently; their combination is motivated by the shared condition, not a proven pathway. Conditioning evidence is in Appendix D.1.

3.1 Eq. (4). We accept the counterexample (−0.238-0.238 vs. 00). The issue is a definition–estimator mismatch. In population form, Vp=∥μp∥2V_p=\|\mu_p\|^2, Vq=∥μq∥2V_q=\|\mu_q\|^2, Vpq=⟨μp,μq⟩V_{pq}=\langle\mu_p,\mu_q\rangle, so LCSP=−2log⁡c(x)\mathcal{L}_{\mathrm{CSP}}=-2\log c(x) with c=Vpq/VpVq∈(0,1]c=V_{pq}/\sqrt{V_pV_q}\in(0,1]; the objective is non-negative and vanishes iff μq∝μp\mu_q\propto\mu_p. The exact empirical counterpart is the double-anchor sum (Appendix H); with positive weights and a PSD kernel, Cauchy–Schwarz gives Vpq2≤VpVqV_{pq}^2\le V_pV_q, so this loss is non-negative. The reported code uses a single-anchor numerator with a Ki2K_i^2 denominator, which is not the RKHS norm or inner product; its bracket can be negative, as the counterexample shows. We distinguish the population objective, the exact double-anchor counterpart, and the implemented single-anchor approximation. Lemma 1 and non-negativity are population-only; the counterexample is included; empirical results use the single-anchor implementation without the population guarantee.

3.2 Coverage gate. gcov=e−rt/2g_{\mathrm{cov}}=e^{-r_t/2} with rt=dnn/τtr_t=d_{\mathrm{nn}}/\tau_t is scale-invariant: a uniform inflation of all memory distances leaves it unchanged. We therefore treat gcovg_{\mathrm{cov}} as relative local support, not absolute coverage. We state this in §3.3 and in the Limitations (uniformly distant entries may go undetected). We will not add an unvalidated absolute gate without a calibration protocol. The detailed gate/write specification is in Appendix B.1; detailed limitations in Appendix I.3.

4.1 Maze2D. Table 5 reports the D4RL sparse-reward Maze2D tasks maze2d-umaze-v1, maze2d-medium-v1, and maze2d-large-v1, not AntMaze. We correct the title, caption, discussion, and cross-references. AntMaze references in §4.3, §4.4, and Appendix E concern separate experiments and remain. Per-cell seed protocol and environment versions are in Appendix C.

4.2 Figure 2(a). The figure mixes units: 2001 counts conflicting pairs, while 5000 counts sampled anchors; their ratio is not a valid conflict percentage. Because the pair-level denominator is not verifiable, we remove the 14.5% annotation and report only the verified count (2001 conflicting pairs), without a percentage.

4.3 Abstract attribution. The 18.1-point difference is DTSA 66.7 vs. official DSFormer-PSSA 48.6; the matched mechanism-free backbone scores 56.7, so the matched gain is 10.0 and the remaining 8.1 is baseline/protocol difference. The abstract names both baselines and reports the 10.0 gain separately; §4.2 uses the same labels (matched control: 56.7/80.0).

Summary. We will clarify the implemented CSP estimator and the scope of its guarantee, narrow the AEM coverage claim to relative local support, correct the factual errors, and retain the framework-level contribution. Appendices B.1, D.1–D.2, H, and I contain the detailed specifications, evidence, counterexample, and limitations.

AuthorsOpenReview ↗

Supplementary Response to Reviewer M5Ut

Thank you for the precise review. This is a supplementary response. Our 18 Nov response remains the main answer to your seven weaknesses. Below we only record what has changed since then.

Implemented from the previous response.

  • §2.4 now contains the integration claim: one condition space, complementary mechanisms, and closed-loop constraints. It also states that the problem comes from the pointwise training objective, not a new shift mechanism.
  • §3.2 now contains the sentence "We keep three forms separate because they carry different guarantees." The theorem note now says the experiments "are not assigned the theorem's guarantees" (the earlier response said "without the population guarantee").
  • §3.3 now states the coverage limitation in the text and repeats it in Appendix I.3. It also states that "uniformly distant entries may go undetected."
  • Table 5 is now labeled Maze2D with the three v1 identifiers. The ethics statement now separates the AntMaze mechanism experiments from the Maze2D results.
  • Figure 2(a) now shows "2001 conflicting pairs" in the image itself. The "5000 samples (14.5%)" annotation is gone.
  • The abstract now separates the 18.1-point DSFormer-PSSA gap from the 10.0-point matched-backbone gain.

New since the previous response.

  • §3 now points to the notation table in Appendix A. This pointer did not exist before.
  • The manuscript was rewritten for clarity: definitions precede use, long compound sentences were split, and the running text no longer uses w/o or vs. This addresses the "requires substantial effort" presentation score.
  • The counterexample is accepted in Appendix H.2, and the single-anchor numerator with the K_i^2 denominator is explicitly not the RKHS norm or inner product.

Unchanged. The technical positions on novelty, the CSP-to-AEM evidence, the estimator distinction, the coverage gate, Table 5, Figure 2(a), and the abstract attribution are the same as in the previous response. No result changed.

We hope these supplementary changes are easy to verify. Thank you for the precise review.

Reviewer_Qz78OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors tackle two problems of offline RL with (stateful) spiking neural networks (SNNs); One slow time-scale drift relative to the training data and one based on faster perturbations that bring the controller to conditions unseen in training. The authors propose two related solutions: Conditional Structure Preservation (CSP) and Adaptive Episodic Memory (AEM). The latter mechanism sets a threshold that only accepts actions similar to those in the training dataset for similar states. The mechanisms slightly improve offline RL results with SNNs for the given control tasks.

Critical strengths
  • The approach brings (light) improvements over other SNN approaches (such as DSFormer) in the control domain, which is a very difficult domain.
  • The ablations show that both mechanisms help for this improvement.
  • The work is tested on many tasks in the mujoco and adroit gyms
Critical weaknesses
  • The main weakness of the article is how unclear it is. This leads to many questions, even on the first, long-term problem. The authors say in their introduction: "Offline RL with SNNs therefore exhibits a dual-timescale shift. Slow offline updates accumulate unsupported behavioral drift relative to the fixed data support (Kumar et al. 2019)". I don't know what the behavioral drift is the authors talk about. Going to the original paper by Kumar et al, I read: "Bootstrapping error is due to bootstrapping from actions that lie outside of the training data distribution, and it accumulates via the Bellman backup operator.". In fact, the error then comes from taking into account actions in the update that are actually hardly ever taken in the offline dataset. The authors continue: "while deployment perturbations rapidly move even a frozen controller into neuraldynamic conditions not covered during training (Morimoto & Doya, 2005; Zhang et al., 2021; Xu et al., 2025a)." What is meant with deployment perturbation? Again, going to the original articles gives an idea about noise present on the observations, actions, and rewards logged. Without the problem being clear, also the solutions are difficult to understand. In order to keep this short, I will ask those questions below. In any case, because of unclarity, the method is very hard to evaluate.
  • Limited neuromorphic nature of the approach. The SNN outputs spikes, which seem to be window-averaged and conditioned on the state, and then passed through elements such as the non-neuromorphic AEM. In the end, the solution would then need a (potentially small) companion processor to perform the post-SNN steps (window averaging, memory bank, etc.). This eats into any advantages that may be obtained by using a spiking net.
Questions for authors

Some of the clarification questions - but to be honest, there were many:

  1. To be honest, the whole CSP solution is unclear to me. Here are some of the reasons why:
  • "Behavior cloning anchors pointwise prediction, LBC(θ) = ... over valid positions I, but is blind to relations between positions." What do you mean with postiions?
  • "valid causal pair": what do you mean here? And is the mask 1 when valid?
  • "The loss controls what it targets.": This seems like a circular reasoning... what is meant here?
  • "negative, vanishing exactly when the policy’s conditional action embedding is proportional to the behavior’s". The behavior's is referring to the behavior present in the offline dataset? It would help a lot if the authors intuitively explained the use of Vp, Vq, and Vpq, or the reasoning behind their solution and how it is different from previous solutions.
  1. Also the AEM solution is unclear.
  • "and write-after-execute causality." What is that?
  • "dx Eq. (2) to select the top-Kr neighbors and form weights qtj with adaptive bandwidth τt" What is this adaptive bandwidth? What is that useful for? Can the authors intuitively explain what they are doing here? On first reading I thought the idea was to select similar actions as the ones present in the offline RL dataset, but that is the first problem, not the second. Previous work has used averaging and dropout to tackle wrongly recorded offline data, but this seems to be tackled by the window averaging over spikes. The main thing to explain here is what dx really means and what is compared with what in the memory bank.
  • Lower, at the confidence gates: "Three statistics decide whether local evidence may act": How can local evidence act? Evidence is evidence. The policy outputs window-averaged values for actions, which are then combined with states (as far as I understand).
  • "and at capacity the bank evicts arg minj ρj ." Why? What is the reasoning behind this, again, intuitively?
  1. The performance differences in numbers and percentage on the tasks seems small: what do these improvements mean from a behavior point of view?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Qz78 — Definitional and Implementation Clarifications

We appreciate these questions. We will clarify §3 with a running example and notation table, distinguishing supported claims from open boundaries.

Running example. A robot can visit the same state under two histories yet produce different spike responses and supported actions. CSP regularizes this state–spike relation during training; AEM retrieves from jointly similar conditions.

  1. Behavioral drift vs. Kumar et al. We do not claim a new bootstrapping-error mechanism. Kumar et al. study out-of-support actions in the backup; we study drift between pθ(a∣s,ξ)p_\theta(a\mid s,\xi) and pB(a∣s,ξ)p_B(a\mid s,\xi) at a fixed state–spike condition. CSP regularizes this relation, without changing the backup. We will drop the "new shift mechanism" framing, not the problem.

  2. Deployment perturbation. We mean changed observations or dynamics after training, with parameters frozen. Table 7 covers observation (noise, dropout), dynamics (mass, friction), initial-condition, and composite shifts. Behavior/reward noise are not among the eight settings; we will align the wording.

  3. Neuromorphic nature. Retrieval, gating, and fusion require a companion processor: the SNN is the front-end and AEM is a hybrid deployment module, not an end-to-end neuromorphic component. Appendix G estimates DTSA at ≈298.5 µJ/pass (DT 410.5; FCNet 1022.0); AEM accounts for 248.17 µJ, uses ≈1.1 MB, and requires no gradients. We will state this cost explicitly.

  4. "Positions" II. A position is a time index in the sampled context window. I={i:mi=1}I=\{i:m_i=1\}, mim_i the validity mask. II is all valid anchors, not a neighborhood or terminal mask; WiW_i only restricts which neighbors jj enter the pair sum. The mim_i factor in LBCL_{BC} is redundant and will be removed.

  5. "Valid causal pair" and Mask. Mij=1M_{ij}=1 iff i,ji,j are both valid and jj lies in the causal past window of ii (0≤i−j<W0\le i-j<W, W=8W=8); Mij=0M_{ij}=0 excludes the pair. The pair contributes (si,ξi,ai)(s_i,\xi_i,a_i),(sj,ξj,aj)(s_j,\xi_j,a_j); self-pair down-weighted by ω0=0.1\omega_0=0.1. No future position is used. The earlier ∣i−j∣<W|i-j|<W wording will be corrected.

  6. "The loss controls what it targets." We agree the sentence is circular. We will replace it with "CSP penalizes a kernel-based divergence between the policy's conditional action distribution and the behavior's, weighted by joint state–spike local support." It is a regularizer, not a failure detector.

  7. "Negative, vanishing…" The intended term is non-negative. In the population formulation, LCSP=−2log⁡c(x)L_{\mathrm{CSP}}=-2\log c(x), with c=Vpq/VpVq∈(0,1]c=V_{pq}/\sqrt{V_pV_q}\in(0,1]; it vanishes iff μq∝μp\mu_q\propto\mu_p. The implemented single-anchor plug-in is only a finite-sample surrogate and is not guaranteed non-negative. We will not transfer the population guarantee to it.

  8. Vp,Vq,VpqV_p,V_q,V_{pq}. In the population formulation these are ∥μp∥2\|\mu_p\|^2, ∥μq∥2\|\mu_q\|^2, and ⟨μp,μq⟩\langle\mu_p,\mu_q\rangle: two embedding norms and their overlap. The double-anchor expression is the exact empirical RKHS counterpart, but the reported implementation uses the single-anchor approximation.

  9. Write-after-execute. Retrieve from Ht−1H_{t-1}, form at0a_t^0, correct to ata_t, execute, observe, write (st,ξt,atexec,ρt)(s_t,\xi_t,a_t^{\mathrm{exec}},\rho_t) if gt≥δwriteg_t\ge\delta_{\mathrm{write}} (corrected steps need the magnitude clip inactive). The current transition is never used for its own correction; the bank is episode-local.

  10. Adaptive bandwidth τt\tau_t. τt\tau_t is the median positive distance over the top-KrK_r. Sparse: large τ\tau, smoother weights; dense: small τ\tau, more local. The deployment code uses per-coordinate RMS normalization averaged per channel; we will state this in Appendix C/F.

  11. dxd_x and memory comparison. Eq. (2) defines dxd_x with median bandwidths; deployment uses a per-dimension RMS variant. Query qt=(st,ξt)q_t=(s_t,\xi_t); memory stores (sj,ξj,ajexec,ρj)(s_j,\xi_j,a_j^{\mathrm{exec}},\rho_j). Retrieval compares qtq_t with (sj,ξj)(s_j,\xi_j); the action is returned, not compared, and weighted by ρj\rho_j.

  12. "Local evidence." We will rename it "local memory support": gsupg_{\mathrm{sup}}, gcovg_{\mathrm{cov}}, gactg_{\mathrm{act}}; product gtg_t gates correction; below δread\delta_{\mathrm{read}} the policy action is used unchanged.

  13. Eviction arg⁡min⁡jρj\arg\min_j \rho_j. ρj\rho_j is reliability: 1 for admitted uncorrected entries, gtg_t at admission for corrected ones. At capacity we evict the least reliable entry, ties oldest.

  14. Small performance differences. We agree the gains can be small. Figure 6 and Appendix D report paired CIs and action-variation/jerk metrics; our claim is improved retention under shift, not a large behavioral change.

AuthorsOpenReview ↗

Supplementary Response to Reviewer Qz78

Thank you for the follow-up. This is a supplementary response. Our 18 Nov response remains the main answer to your questions. Below we only record what is new or has changed since then, so you do not have to read the same answers again.

New in this revision.

  • §3 now points to the notation table in Appendix A and the running example in Appendix I.2. The earlier response promised a notation table but did not fix its location.
  • §3.2 and §3.3 were rewritten in reading order: position before pair mask, pair mask before weights, retrieval and gates before their formal statements. This is the change that most directly addresses your "unclear" assessment.
  • The eight deployment settings are now Table 8 (Table 7 in the previous version).
  • The running text no longer uses w/o or vs. Abbreviations are expanded at first use (MDP, BC, RKHS).

Changed answers.

  • Eviction. The rule is unchanged (least reliable entry, ties oldest). The paper now states the reliability definition that motivates it: ρt=1\rho_t=1 for uncorrected steps and ρt=gt\rho_t=g_t for corrected ones. Reliability therefore records how strongly an entry passed admission, so the least reliable entry is the least trustworthy. Appendix B.1 states the tie-break: "with ties broken by age (oldest first)".
  • Coverage gate. The paper now states the consequence explicitly: "uniformly distant entries may go undetected." No other gate measures absolute distance, and Appendix I.3 records this as a limitation rather than introducing an unvalidated absolute gate.
  • Adaptive bandwidth. Appendix C now gives the per-coordinate RMS variant used at deployment, and Appendix F gives the deterministic deployment protocol. The median form is the idealized geometry used in the analysis.

Unchanged. The answers for behavioral drift, deployment perturbation, neuromorphic nature, positions, valid causal pair/mask, the loss description, negative/vanishing, Vp/Vq/Vpq, write-after-execute, d_x and the memory comparison, local memory support, the gate product, and small gains are the same as in the previous response. No technical position changed.

We hope these supplementary changes are easy to verify. Thank you for the detailed questions.

Reviewer_tCBpOpenReview ↗
Rating
1(−1)
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper combines a training regularizer (CSP) with a deployment memory module (AEM) to improve offline reinforcement learning with spiking neural networks. The reported performance gains are promising. However, the experiments do not fully establish the proposed explanation for why the method works, and some experimental details need clarification.

Critical strengths

1.The two components have clear roles: CSP regularizes training, while AEM adjusts actions using recent experience without updating network parameters. 2.Tables 3–4 show improvements over the matched backbone on MuJoCo and Adroit. The component ablations help separate the contribution of each module. 3.The shuffled-memory experiment provides useful evidence that correct spike–action pairings matter for the memory module.

Critical weaknesses

1.The paper argues that training gradually damages supported action relationships. However, Figure 2 mainly shows that CSP lowers a relationship-error metric. This does not directly establish that training causes harmful drift or that this drift reduces task performance. 2.The experiments show that spike features help distinguish behaviors and that spike changes correlate with action changes. However, these findings do not establish whether the benefit comes from spike dynamics or simply from including more history information. Comparisons with other history representations would help clarify this. 3.The robustness results are mixed. In Table 9, DTSA achieves the highest Mean AUC, which measures overall performance across perturbation levels, on only one of four tasks. The authors should qualify the broad robustness claims and explain the weaker cases. 4.Table 5 is labeled “AntMaze” in lines 400–406, but its rows list Maze2D tasks. These are different benchmarks. The authors should correct the label and provide the exact environment versions.

Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer tCBp — Scope and Factual Corrections

We appreciate these concerns. They identify places where our claims need sharper qualification; several concern the scope of the current evidence rather than the validity of the proposed framework. We will correct the factual inconsistencies and state the evidence boundaries explicitly.

W1: Training drift and performance. Figure 2(c,d) establishes that CSP reduces held-out DCS on all four mechanism tasks, and the component ablations show that CSP is associated with higher returns (Tables 3–4: +2.0 MuJoCo, +6.6 Adroit). It does not establish the full causal chain from a training update to relation drift to deployment loss. We will present that chain as a hypothesis, not as proven: we will replace "accumulate unsupported behavioral drift" with "can drift," add a Limitations sentence, and clarify that Figure 2(c,d) plots held-out DCS rather than deployment return. This is a correction of scope, not a retraction: CSP lowers the held-out single-anchor DCS proxy, a result consistent with improved local structure preservation but not by itself proof of the underlying mechanism. The shuffled-memory intervention shows that AEM uses the spike–action correspondence; we will not extend that result into a claim that CSP preserves the correspondence for AEM.

W2: Spike dynamics vs. history. The current experiments support two claims. First, the PCA-denoised spike-conditioned representation (16-dimensional projection; explained variance 76.3% on Walker2d-medium and 65.7% on Walker2d-medium-replay) improves separability over state-only on all four mechanism tasks (+39% to +159%); on the raw 256-dimensional ξ\xi, the Walker2d-medium gain is negative (−10.9%-10.9\%), so the supported claim is about the denoised representation, not the raw spike vector. Second, the shuffled-ξ\xi intervention shows that AEM's benefit depends on the spike–action correspondence. The experiments do not isolate spike dynamics from additional history: the two representations are not history-matched. We will state this as an evidence boundary and avoid claiming that spike dynamics are the unique source of the gain. We will retain the supported claim that the PCA-denoised spike-conditioned representation is useful, and present the history-controlled comparison as the missing step rather than as evidence against the framework.

W3: Robustness. We agree that the broad-robustness wording overstates the deployment table (Table 10 in the revised manuscript). The table shows the highest Mean AUC on one of four tasks: Walker2d-m 37.88 (vs. 35.67/35.66/33.61); Walker2d-m-r 23.23 (vs. 25.70/25.54/25.49); Hopper-m 17.78 (vs. 18.40/18.31/18.37); Hopper-m-r 19.76 (vs. 20.76/17.08/17.76). We will replace "broad robustness" with a task-dependent statement and correct two sentences the table does not support: §4.4's "highest nominal return in all four panels" (it is highest in two: Walker2d-m 84.06 and Hopper-m-r 92.06) and the conclusion's "remains above the baselines at most severities" (worst-bin retention is below the best baseline on all four tasks, e.g., 0.0016 vs. 0.0703). The supported claim is narrower but real: gains are concentrated in nominal and low-to-moderate shifts, and we report per-task AUC rather than a single aggregate.

W4: Table 5. We apologize for the factual error. The tasks are D4RL maze2d-umaze-v1, maze2d-medium-v1, and maze2d-large-v1 (sparse-reward v1), not AntMaze. We will correct the title, caption, discussion, and cross-references, and add the exact identifiers and versions. The AntMaze mentions to fix are §4.1, §4.2, the Table 5 caption, and the ethics statement; the AntMaze experiments in §4.3, §4.4, and Appendix E are separate and will be kept. We will keep the fact that scores are not clipped at 100 but state it for Maze2D ("Maze2D normalized scores are not clipped at 100, D4RL REF_MIN/REF_MAX normalization"); if the corrected cells no longer exceed 100, we will drop it. We will also state the per-cell seed protocol and add an environment-version table to Appendix C (MuJoCo v2, Adroit v0, Maze2D v1 sparse, AntMaze v2).

Summary. We will correct the factual errors and overbroad wording, while retaining the supported evidence: CSP reduces DCS, the spike-conditioned representation improves separability, and AEM's benefit depends on the spike–action correspondence. We thank the reviewer for separating these issues.

Thank you for the response. Maybe the author had a really excellent idea while the writing is really confusing and AI-generated without detailed check. The paper is filled with ';' 'p' ':' which is not curated by human. Hope the author could settle down to revise the paper and express the idea more clearly. :)

AuthorsOpenReview ↗

Response to Reviewer tCBp

Thank you for the reminder. You are right that the paper needed a careful editorial pass: too many logical relations were packed into single sentences, and the reader had to reconstruct definitions that should have been stated first. We rewrote the manuscript from the abstract through the conclusion. No experimental result, table value, or technical claim was changed.

Below we map your comment to concrete passages. The "before" text is from the submitted version. The "after" text is what is in the revised manuscript.

1. "Difficult to follow": definitions now precede use

Before (§3.2):

Behavior cloning anchors pointwise prediction, LBC(θ)=…L_{\mathrm{BC}}(\theta)=\ldots over valid positions II, but is blind to relations between positions.

After (§3.2):

Behavior cloning treats each position independently. A position ii is a time index in the sampled context window. The set I={i:mi=1}\mathcal{I}=\{i:m_i=1\} contains all valid anchors under the binary validity mask mim_i. It is not a neighborhood or a terminal mask. Behavior cloning is blind to relations between positions. Those relations form the policy's local action structure, and CSP is designed to preserve them.

The same change is applied to AEM:

Before (§3.3):

AEM keeps a fixed-capacity bank ... with reliability-weighted replacement, cold-start warmup of h0h_0 steps (retrieval requires at least hmin⁡h_{\min} entries), and write-after-execute causality.

After (§3.3):

AEM keeps a fixed-capacity bank ... with reliability-weighted replacement. It uses a cold-start warmup of h0h_0 steps, and retrieval requires at least hmin⁡h_{\min} entries. Writes happen after execution, so the current transition is never used for its own correction.

2. "Too compressed": one relation per sentence

Long compound sentences were split. The three CSP estimator forms, which were previously mixed in one paragraph with their caveats, are now separated:

We keep three forms separate because they carry different guarantees. The analysis uses the population form. The exact empirical counterpart is the double-anchor form, and the reported experiments use the single-anchor form.

The CSP loss is now described in plain language before its formal form:

CSP penalizes a kernel-based divergence between the policy's conditional action distribution and the behavior's, weighted by joint state–spike local support. It is a regularizer, not a failure detector.

3. "Not curated": notation and shorthand

  • Abbreviations are expanded at first use: Markov decision process (MDP), behavior cloning (BC), reproducing kernel Hilbert space (RKHS).
  • The running text no longer uses the informal forms w/o and w/, or the comparison shorthand vs. The text says "without AEM or CSP", "the AEM-free variant", and "the CSP-free variant". Comparisons are written as "against", "alongside", "compared with", or "relative to".
  • Table headers use the same terms as the prose.

4. "Express the idea more clearly": a reading order in each section

  • §3.1: MDP → spike readout → geometry conventions.
  • §3.2: position → valid anchors → pair mask → weights → kernel → embeddings → double-anchor sums → loss → estimator forms.
  • §3.3: retrieval → bandwidth → transport → confidence gates → write admission → implementation scope.
  • §4.1: benchmark and baseline protocol, then seed and deployment protocol.
  • §4.2–4.4: positive result first, scope limit in the next paragraph.

5. "Without detailed check": the prose now matches the tables and figures

  • Table 5 is labeled Maze2D throughout, with maze2d-umaze-v1, maze2d-medium-v1, and maze2d-large-v1.
  • Figure 2(a) reports "2001 conflicting pairs" in the image itself. The old "5000 samples (14.5%)" annotation is removed.
  • The Adroit result is split into the 18.1-point DSFormer-PSSA gap and the 10.0-point matched-backbone gain.

6. Voice

Where the evidence supports a judgment, the text states it directly:

We cast this interaction as a dual-timescale shift.

An SNN's response is a function of its history, not only its input.

We keep three forms separate because they carry different guarantees.

The deployment gains are real but narrow.

We hope the revision is easier to follow. We thank the reviewer for pressing on this point.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.