- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper frames SNN-based offline reinforcement learning as a dual-timescale problem involving training-time drift from fixed behavioral support and deployment-time changes in internal spike dynamics. It proposes DTSA, which uses a shared state–spike condition across two components: CSP regularizes conditional action structure during training, and AEM performs episode-local action correction with frozen network parameters. The paper provides a unified error analysis and evaluates DTSA on D4RL tasks through matched component ablations, deployment perturbations, and a memory-pairing intervention.
Critical strengths
-
A coherent SNN-specific training–deployment framework. DTSA explicitly uses the internal spike response as a shared condition for training-time action-structure regularization and deployment-time memory retrieval. This provides a meaningful framework-level connection between offline policy learning and frozen-parameter adaptation.
-
Empirical performance improvements. DTSA improves over the matched mechanism-free SNN backbone by 4.0 average points on MuJoCo and 10.0 points on Adroit, with both CSP and AEM contributing. AEM operates without parameter updates, and shuffling spike–action pairings removes its benefit, providing direct evidence that the correction uses the correspondence stored in memory.
Critical weaknesses
1. Limited novelty
1.1 The dual-timescale formulation combines established distribution shifts. The paper’s “slow-timescale shift”—the learned policy drifting away from the fixed behavioral support during offline optimization—is the standard policy–dataset distribution shift widely studied in offline RL [1]. Its “fast-timescale shift”—deployment perturbations changing temporally evolving spike states, actions, and subsequent state distributions—has also been studied in SNN dynamics and continuous control [2,3]. The proposed dual-timescale framework combines these established shifts under a shared state–spike condition, without identifying a new shift mechanism or demonstrating a new interaction between them.
1.2 CSP and AEM are incremental adaptations of existing methods. CSP applies conditional Cauchy–Schwarz divergence, previously developed for kernel-based conditional-distribution matching and sequential decision-making [4], to state–spike-conditioned action distributions. AEM builds on established episodic-memory and nearest-neighbor kernel-retrieval methods in RL [5], including their extension to offline RL [6]. The methodological contribution is an incremental adaptation and integration of these existing techniques for SNN-based offline RL.
2. Evidence does not establish the claimed training–deployment mechanism
The paper claims that CSP preserves state–spike-conditioned action relations during training for AEM to recover at deployment; the current evidence establishes only that CSP and AEM are individually effective. Figure 2 shows that CSP reduces held-out DCS, without measuring whether AEM’s local spike–action fitting becomes more accurate. The bank-pairing intervention shows that AEM requires correct spike–action pairings, without showing that these pairings were improved or preserved by CSP. The component ablations attribute performance gains to both modules and provide no test of the proposed CSP-to-AEM pathway. The shared training–deployment mechanism therefore remains unverified.
3. Mathematical and methodological flaws
3.1 The CSP loss is inconsistent with its claimed theoretical objective. The defined embeddings imply and . Eq. (4) instead uses the single-anchor sums and . These quantities do not equal the stated embedding norm and inner product, so the CS-overlap interpretation and its derived guarantees do not follow. A two-position counterexample with , , , , , , and gives , while perfect prediction gives . The CSP term can therefore assign a lower value to an incorrect prediction and violates the claimed non-negativity.
3.2 AEM’s coverage gate does not measure absolute memory coverage. The coverage gate is defined in Section 3.3 and Eq. (14) as , where , to determine whether the current query is sufficiently covered by the episodic memory and should permit action correction. However, it fails to measure absolute coverage because scaling all retrieved distances by the same factor leaves the gate unchanged, and normally gives . For example, distance sets and both produce . The gate therefore cannot distinguish a well-covered query from one where every memory entry is distant, undermining its stated role as a coverage test.
4. Factual errors
-
4.1 Maze2D and AntMaze are conflated. Table 5 reports three
maze2dtasks, while its title and discussion call them AntMaze. Maze2D and AntMaze have different agents, action spaces, and control problems [7]. -
4.2 The count and percentage in Figure 2(a) are inconsistent. The figure reports “2001 conflicts / 5000 samples (14.5%),” although .
-
4.3 The abstract misattributes the Adroit improvement. The reported 18.1-point gain compares DTSA’s 66.7 with DSFormer’s 48.6, while the matched mechanism-free control scores 56.7. The gain attributable to CSP and AEM is ; the remaining 8.1 points arise from backbone or protocol differences.
References
[1] S. Levine, A. Kumar, G. Tucker, and J. Fu. “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems.” 2020.
[2] J. K. Eshraghian, M. Ward, E. O. Neftci, X. Wang, G. Lenz, G. Dwivedi, M. Bennamoun, D. S. Jeong, and W. D. Lu. “Training Spiking Neural Networks Using Lessons From Deep Learning.” Proceedings of the IEEE, 2023.
[3] Z. Xu, Z. Huang, Y. Dong, K. Chen, W. Liu, and Z. Yu. “Error Amplification Limits ANN-to-SNN Conversion in Continuous Control.” ICML, 2026.
[4] S. Yu, H. Li, S. Løkse, R. Jenssen, and J. C. Príncipe. “The Conditional Cauchy–Schwarz Divergence With Applications to Time-Series Data and Sequential Decision Making.” IEEE TPAMI, 2025.
[5] A. Pritzel, B. Uria, S. Srinivasan, A. Puigdomènech Badia, O. Vinyals, D. Hassabis, D. Wierstra, and C. Blundell. “Neural Episodic Control.” ICML, 2017.
[6] X. Ma, Y. Yang, H. Hu, Q. Liu, J. Yang, C. Zhang, Q. Zhao, and B. Liang. “Offline Reinforcement Learning with Value-Based Episodic Memory.” ICLR, 2022.
[7] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. “D4RL: Datasets for Deep Data-Driven Reinforcement Learning.” 2020.
Questions for authors
-
How does the dual-timescale formulation differ technically from established offline-RL policy–dataset shift [1] and SNN temporal or deployment shift [2,3]? What new algorithmic elements do CSP and AEM introduce beyond the methods in [4–6]?
-
What mathematical relationship connects the single-anchor quantities in Eq. (4) to the stated embedding norms and inner products? Which formulation is used in the implementation, and how is the negative-loss counterexample addressed?
-
How does detect inadequate memory coverage when all retrieved distances increase proportionally? Does any other gate identify this case?
-
What evidence shows that CSP improves AEM’s deployment-time fitting or corrections, beyond the independent gains of the two components?
-
Please clarify the environment used in Table 5, the denominator underlying Figure 2(a)’s 14.5%, and why the abstract reports an 18.1-point matched-backbone improvement when the matched mechanism gain is 10.0 points.
2 comments
Supplementary Response to Reviewer M5Ut
Thank you for the precise review. This is a supplementary response. Our 18 Nov response remains the main answer to your seven weaknesses. Below we only record what has changed since then.
Implemented from the previous response.
- §2.4 now contains the integration claim: one condition space, complementary mechanisms, and closed-loop constraints. It also states that the problem comes from the pointwise training objective, not a new shift mechanism.
- §3.2 now contains the sentence "We keep three forms separate because they carry different guarantees." The theorem note now says the experiments "are not assigned the theorem's guarantees" (the earlier response said "without the population guarantee").
- §3.3 now states the coverage limitation in the text and repeats it in Appendix I.3. It also states that "uniformly distant entries may go undetected."
- Table 5 is now labeled Maze2D with the three v1 identifiers. The ethics statement now separates the AntMaze mechanism experiments from the Maze2D results.
- Figure 2(a) now shows "2001 conflicting pairs" in the image itself. The "5000 samples (14.5%)" annotation is gone.
- The abstract now separates the 18.1-point DSFormer-PSSA gap from the 10.0-point matched-backbone gain.
New since the previous response.
- §3 now points to the notation table in Appendix A. This pointer did not exist before.
- The manuscript was rewritten for clarity: definitions precede use, long compound sentences were split, and the running text no longer uses
w/oorvs.This addresses the "requires substantial effort" presentation score. - The counterexample is accepted in Appendix H.2, and the single-anchor numerator with the K_i^2 denominator is explicitly not the RKHS norm or inner product.
Unchanged. The technical positions on novelty, the CSP-to-AEM evidence, the estimator distinction, the coverage gate, Table 5, Figure 2(a), and the abstract attribution are the same as in the previous response. No result changed.
We hope these supplementary changes are easy to verify. Thank you for the precise review.
Response to Reviewer M5Ut — Estimator Corrections and Framework Positioning
We thank the reviewer for the precise review. We agree that Eq. (4), the coverage gate, and the novelty positioning require correction. Below we separate the mathematical corrections (3.1–3.2), factual corrections (4.1–4.3), and framework contribution (1.1–1.2, 2). Details are in Appendices B.1, D.1–D.2, H, and I.
1.1 / 1.2 Novelty. We agree that CSP builds on conditional kernel-divergence matching and AEM on episodic retrieval; neither component is new in isolation. The contribution is the integration; §2.4 and Appendix I.1 separate reused from added components. The integration has three layers. (i) Unified condition space: both training and deployment use the joint state–spike condition (s,ξ), not a state-only representation. (ii) Mechanism complementarity: CSP preserves local behavior structure during training, while AEM performs confidence-gated correction with frozen parameters. (iii) Closed-loop constraints: reliability records, write admission, and bounded correction reduce, but cannot eliminate, the risk that unreliable corrections propagate. The framework-level claim is that one condition space couples training-time structure preservation with deployment-time gated correction. We add the [1]–[6] comparison in Appendix I.1 and remove any new-shift-mechanism claim. Since Appendix E.5 finds MMD marginally better than CS, we do not claim CS-specific superiority.
2. CSP→AEM evidence. The evidence supports two separate findings. First, CSP reduces the held-out single-anchor DCS proxy. Second, the shuffled-ξ intervention shows that AEM's benefit depends on the correct spike–action correspondence. It does not show that CSP improves AEM's local fit: the findings are independent. We label mediation as a hypothesis in §1, §2.4, and §5 and remove causal wording from the abstract and conclusion. The mechanisms can be used independently; their combination is motivated by the shared condition, not a proven pathway. Conditioning evidence is in Appendix D.1.
3.1 Eq. (4). We accept the counterexample (−0.238 vs. 0). The issue is a definition–estimator mismatch. In population form, Vp=∥μp∥2, Vq=∥μq∥2, Vpq=⟨μp,μq⟩, so LCSP=−2logc(x) with c=Vpq/VpVq∈(0,1]; the objective is non-negative and vanishes iff μq∝μp. The exact empirical counterpart is the double-anchor sum (Appendix H); with positive weights and a PSD kernel, Cauchy–Schwarz gives Vpq2≤VpVq, so this loss is non-negative. The reported code uses a single-anchor numerator with a Ki2 denominator, which is not the RKHS norm or inner product; its bracket can be negative, as the counterexample shows. We distinguish the population objective, the exact double-anchor counterpart, and the implemented single-anchor approximation. Lemma 1 and non-negativity are population-only; the counterexample is included; empirical results use the single-anchor implementation without the population guarantee.
3.2 Coverage gate. gcov=e−rt/2 with rt=dnn/τt is scale-invariant: a uniform inflation of all memory distances leaves it unchanged. We therefore treat gcov as relative local support, not absolute coverage. We state this in §3.3 and in the Limitations (uniformly distant entries may go undetected). We will not add an unvalidated absolute gate without a calibration protocol. The detailed gate/write specification is in Appendix B.1; detailed limitations in Appendix I.3.
4.1 Maze2D. Table 5 reports the D4RL sparse-reward Maze2D tasks
maze2d-umaze-v1,maze2d-medium-v1, andmaze2d-large-v1, not AntMaze. We correct the title, caption, discussion, and cross-references. AntMaze references in §4.3, §4.4, and Appendix E concern separate experiments and remain. Per-cell seed protocol and environment versions are in Appendix C.4.2 Figure 2(a). The figure mixes units: 2001 counts conflicting pairs, while 5000 counts sampled anchors; their ratio is not a valid conflict percentage. Because the pair-level denominator is not verifiable, we remove the 14.5% annotation and report only the verified count (2001 conflicting pairs), without a percentage.
4.3 Abstract attribution. The 18.1-point difference is DTSA 66.7 vs. official DSFormer-PSSA 48.6; the matched mechanism-free backbone scores 56.7, so the matched gain is 10.0 and the remaining 8.1 is baseline/protocol difference. The abstract names both baselines and reports the 10.0 gain separately; §4.2 uses the same labels (matched control: 56.7/80.0).
Summary. We will clarify the implemented CSP estimator and the scope of its guarantee, narrow the AEM coverage claim to relative local support, correct the factual errors, and retain the framework-level contribution. Appendices B.1, D.1–D.2, H, and I contain the detailed specifications, evidence, counterexample, and limitations.