- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this work, the authors tackle the challenge of accelerating direct training for SNNs without compromising update fidelity, resulting in the SpikingPhoenix system. Technically, the approach brings together forward-backward iterative scans, piecewise reconstruction of membrane states, and graph replay backed by persistent on-device states. Crucially, the authors note that execution boundaries play a major role in balancing reconstruction overhead against the benefits of local operator optimizations. The evaluation covers numerical consistency, state persistence, controlled performance studies, and broader training benchmarks across diverse models, datasets, and hardware setups.
Critical strengths
- Careful preservation of stateful training semantics. Rather than just checking outputs and gradients, the paper explicitly tracks BatchNorm running stats, gradient existence, optimizer states, and dynamic loss scaling. Combining overflow detection with a mechanism to safely skip optimizer updates solves a subtle bug that simpler graph-replay validations often overlook.
- Solid empirical evidence across varied workloads. Full Transformer schedule evaluations report 1.41x to 2.73x end-to-end speedups, capturing workloads beyond just the accelerated updates. Comparative analyses across architectures and platforms, alongside ImageNet runs, further show that the recurrent acceleration is not confined to a single model setup.
Critical weaknesses
- The incremental system contributions are not clearly isolated from prior work. While the paper notes that SpikingJelly already supports fused multi-step neuronal execution, Section 4.1 and Appendix B mostly walk through the math of forward and backward scans rather than detailing the low-level implementation choices that actually make SEQUENCE faster than existing fused kernels. The backend profiling shows a clear performance gap, but it fails to explain where this speedup really comes from at the system level.
- The core comparison on rematerialization placement lacks sufficient detail to prove both sides are doing equivalent compute work. By regenerating prefixes at each backward step, SEGMENT-K incurs neuron updates during reactivation. Yet Appendix D.3 describes the external control baseline as doing just one initial forward pass, one checkpoint replay, and one temporal backward pass per segment. It is not obvious how or where that same repeated-prefix schedule happens in the baseline. Because of this, I am not convinced that the large latency gaps in Table 11 purely reflect placement differences. Comparing kernel launch counts shows that the execution structures differ, but it does not prove the underlying rematerialization workload is identical. Since the main thesis of the paper hinges on moving equivalent work across an execution boundary, the exact algorithm for the external baseline needs to be spelled out and matched much more carefully.
Questions for authors
- What concrete code-level changes distinguish SEQUENCE from the fused Triton and CuPy paths already in SpikingJelly? I am trying to understand which specific choices in temporal scheduling, memory representation, or backward kernel design drive the performance gap.
- Looking at Table 3, which state transitions remain outside the raw graph, and are they fully included in the timing measurements? Please also clarify whether this baseline shares the exact same recurrent kernels and optimizer code as GRAPH-V2.
- To help verify workload equivalence, can you share the pseudocode, retained tensor lists, and total rematerialized neuron update counts for both placement options under the same temporal partition? If their recomputation schedules do differ, how much of the placement speedup remains once the external control executes the exact same prefix schedule as SEGMENT-K?
- It seems the final and best accuracies for the complete Transformer schedules are missing. Section 5.5 points to Appendix E, and Appendix E.4 says Table 4 has the accuracy numbers, but Tables 4 and 19 only show timing data. Could you provide these accuracy results?
7 comments
Q1 (W1):SEQUENCE implementation and gains over existing fused kernels
For the evaluated LIF operator, native SJ fused Triton exposes time-indexed membrane states and accepts intermediate membrane gradients through grad_v_seq_ptr. SEQUENCE returns the spike sequence and terminal membrane state v_final, retains the FP32 pre-reset trajectory u_seq for the ATan surrogate derivative, and propagates grad_v_final through its reverse-time kernel. The kernel carries the membrane adjoint across time, including the reset Jacobian, using flattened temporal offsets (t*S+offs) rather than the block-pointer addressing and autotuned temporal loops of native SJ Triton.
To distinguish the effect of the state interface from further kernel optimization, we implemented an endpoint-specialized SJ control that preserves SJ's LIF recurrence and surrogate-gradient arithmetic while using the same terminal-state interface. With Graph replay disabled, the FP32 LIF results at T=32 and 262,144 membrane elements on RTX 6000D are:
| Implementation | Forward (ms) | Backward (ms) | F+B (ms) | Additional allocation (MiB) |
|---|---|---|---|---|
| Native SJ fused Triton | 0.686 | 3.089 | 3.770 | 193 |
| SJ endpoint-specialized | 0.429 | 0.579 | 0.989 | 98 |
| Phoenix SEQUENCE FULL | 0.337 | 0.441 | 0.790 | 98 |
Specializing the SJ endpoint reduces backward latency from 3.089 to 0.579 ms, after which Phoenix reduces it further to 0.441 ms. The independently timed F+B latency is 4.77× faster than native SJ and 1.25× faster than endpoint-specialized SJ, while the additional allocation decreases from 193 to 98 MiB relative to native SJ. Forward, backward, and full F+B medians are measured independently.
The CuPy implementation applies the same terminal-state interface using separate forward and backward RawKernels. Its CuPyLIFWorkspace reuses fixed-shape buffers for spikes, pre-reset membrane states, terminal states, and gradients. Backend-matched FP32 F+B measurements from the submitted Appendix D.4 give:
| Temporal length | SJ CuPy (ms) | Phoenix CuPy (ms) | Speedup |
|---|---|---|---|
| 8 | 4.947 | 4.083 | 1.212× |
| 16 | 4.987 | 4.030 | 1.237× |
| 32 | 5.045 | 4.206 | 1.199× |
| 64 | 5.851 | 4.885 | 1.198× |
| 128 | 7.532 | 6.150 | 1.225× |
Phoenix achieves a 1.214× geometric-mean CuPy F+B speedup across these temporal lengths. We will expand Section 4.1 and Appendix B with the forward–backward state interfaces, reverse-time adjoint organization, Triton addressing, and CuPy buffer reuse, alongside the endpoint-specialized comparison.
Q2:Graph capture, optimizer semantics, and complete-update timing
The raw SJ Graph and GRAPH-V2 use different recurrent kernels and optimizer implementations. SJ captures forward, loss, and backward with its native fused recurrent kernels, while conventional timm AdamW and dynamic AMP execute outside the graph. GRAPH-V2 uses Phoenix recurrent kernels and captures compatible recurrent resets, gradient clearing, forward and BatchNorm computation, loss, scaled backward, device-side unscaling and nonfinite checks, conditional fused AdamW updates, and AMP scale/growth transitions.
PyTorch requires stable replay addresses and does not replay ordinary host-side control flow. Accordingly, GRAPH-V2 preallocates stable gradient buffers for the active parameter set, with each captured signature retaining the distinction between grad=None and a materialized zero gradient that AdamW treats differently. Learning-rate changes from the host scheduler reach the update through persistent device scalars, while the device-side nonfinite condition determines whether AdamW steps or is skipped without omitting the corresponding AMP scale transition.
Both systems are timed over the complete synchronized training update, including minibatch staging, forward, loss, backward, AdamW, AMP, and operations outside the SJ graph. Data loading and augmentation are excluded, and initial graph setup is reported separately. The SEW-ResNet18 comparison in Table 3 shows 27.809 versus 26.068 ms per graph-enabled update, while initial setup decreases from 5.47 to 2.11 s. Under the manuscript's Eq. (8), the corresponding setup-amortization thresholds are 119 versus 45 updates.
We also measured spiking Transformers on RTX 6000D at T=4 and FP16, using three runs of 80 synchronized complete updates for each execution mode:
| Transformer (batch) | SJ raw Graph (ms) | Phoenix GRAPH-V2 (ms) | Speedup | Initial setup SJ / Phoenix (s) |
|---|---|---|---|---|
| Spikformer (128) | 56.000 | 54.438 | 1.029× | 9.329 / 6.467 |
| QKFormer (64) | 41.448 | 40.025 | 1.036× | 12.136 / 7.474 |
| MaxFormer (128) | 64.535 | 61.920 | 1.042× | 13.517 / 9.667 |
Graph replay batches GPU work submissions without fusing the computations inside recurrent kernels. The SEQUENCE and SEGMENT-K comparisons above therefore measure optimizations that also apply without graph capture, while these complete-update measurements show additional system speedups after CUDA Graph replay is enabled in both training stacks and 28.5–38.4% shorter initial setup.
We thank the reviewer for the detailed implementation questions and for recognizing the preservation of stateful training semantics. SEQUENCE specializes the recurrent forward–backward state representation and propagates membrane adjoints through a fused reverse-time scan, while SEGMENT-K reconstructs the required membrane states within backward execution. Both optimizations operate without CUDA Graph replay. We evaluate SEQUENCE against native and endpoint-specialized SJ kernels, and compare SEGMENT-K with an external backward implementation that follows the same per-position prefix-reconstruction schedule. GRAPH-V2 further replays compatible optimizer and mixed-precision state transitions within complete minibatch updates.
Across four complete CIFAR-10 spiking-Transformer training schedules, SpikingPhoenix achieves 1.414–2.730× end-to-end speedups over SJ, reducing aggregate single-GPU training time from 26.395 to 13.473 hours. The recurrent and graph-enabled measurements below identify how these gains arise from the corresponding GPU computations and training-state operations.