acceptodds
Under review as a conference paper at ICLR 2027

SpikingPhoenix: Semantics-Preserving Execution Optimization for Direct SNN Training on GPUs

Abstract

Direct multi-step Spiking Neural Network (SNN) training remains costly on graphics processing units (GPUs) after temporal fusion because backward propagation spans recurrent state, backward-required forward quantities, and persistent minibatch updates. Existing systems optimize these costs separately, leaving their composition under a fixed training update unresolved. We present SpikingPhoenix, which reorganizes nested recurrent and minibatch execution boundaries without changing that update. SEQUENCE pairs forward and reverse scans, SEGMENT-K rematerializes discarded membrane state within recurrent backward, and GRAPH-V2 replays the device-side state transitions required by each minibatch update. The transformed paths match the reference exactly on discrete observables and within data-type-specific tolerances on floating-point observables. Across representative Spiking Transformers, including QKFormer and MaxFormer, SpikingPhoenix achieves 1.41-2.73× end-to-end training speedups over matched SpikingJelly runs on complete CIFAR-10 schedules. On ImageNet-1K, SEQUENCE further achieves 1.37-1.45× geometric-mean speedups across T=2,4,8.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spiking Neural Networks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wxMrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors tackle the challenge of accelerating direct training for SNNs without compromising update fidelity, resulting in the SpikingPhoenix system. Technically, the approach brings together forward-backward iterative scans, piecewise reconstruction of membrane states, and graph replay backed by persistent on-device states. Crucially, the authors note that execution boundaries play a major role in balancing reconstruction overhead against the benefits of local operator optimizations. The evaluation covers numerical consistency, state persistence, controlled performance studies, and broader training benchmarks across diverse models, datasets, and hardware setups.

Critical strengths
  1. Careful preservation of stateful training semantics. Rather than just checking outputs and gradients, the paper explicitly tracks BatchNorm running stats, gradient existence, optimizer states, and dynamic loss scaling. Combining overflow detection with a mechanism to safely skip optimizer updates solves a subtle bug that simpler graph-replay validations often overlook.
  2. Solid empirical evidence across varied workloads. Full Transformer schedule evaluations report 1.41x to 2.73x end-to-end speedups, capturing workloads beyond just the accelerated updates. Comparative analyses across architectures and platforms, alongside ImageNet runs, further show that the recurrent acceleration is not confined to a single model setup.
Critical weaknesses
  1. The incremental system contributions are not clearly isolated from prior work. While the paper notes that SpikingJelly already supports fused multi-step neuronal execution, Section 4.1 and Appendix B mostly walk through the math of forward and backward scans rather than detailing the low-level implementation choices that actually make SEQUENCE faster than existing fused kernels. The backend profiling shows a clear performance gap, but it fails to explain where this speedup really comes from at the system level.
  2. The core comparison on rematerialization placement lacks sufficient detail to prove both sides are doing equivalent compute work. By regenerating prefixes at each backward step, SEGMENT-K incurs ∑jkj(kj+1)/2\sum_j k_j(k_j+1)/2 neuron updates during reactivation. Yet Appendix D.3 describes the external control baseline as doing just one initial forward pass, one checkpoint replay, and one temporal backward pass per segment. It is not obvious how or where that same repeated-prefix schedule happens in the baseline. Because of this, I am not convinced that the large latency gaps in Table 11 purely reflect placement differences. Comparing kernel launch counts shows that the execution structures differ, but it does not prove the underlying rematerialization workload is identical. Since the main thesis of the paper hinges on moving equivalent work across an execution boundary, the exact algorithm for the external baseline needs to be spelled out and matched much more carefully.
Questions for authors
  1. What concrete code-level changes distinguish SEQUENCE from the fused Triton and CuPy paths already in SpikingJelly? I am trying to understand which specific choices in temporal scheduling, memory representation, or backward kernel design drive the performance gap.
  2. Looking at Table 3, which state transitions remain outside the raw graph, and are they fully included in the timing measurements? Please also clarify whether this baseline shares the exact same recurrent kernels and optimizer code as GRAPH-V2.
  3. To help verify workload equivalence, can you share the pseudocode, retained tensor lists, and total rematerialized neuron update counts for both placement options under the same temporal partition? If their recomputation schedules do differ, how much of the placement speedup remains once the external control executes the exact same prefix schedule as SEGMENT-K?
  4. It seems the final and best accuracies for the complete Transformer schedules are missing. Section 5.5 points to Appendix E, and Appendix E.4 says Table 4 has the accuracy numbers, but Tables 4 and 19 only show timing data. Could you provide these accuracy results?
7 comments

We thank the reviewer for the detailed implementation questions and for recognizing the preservation of stateful training semantics. SEQUENCE specializes the recurrent forward–backward state representation and propagates membrane adjoints through a fused reverse-time scan, while SEGMENT-K reconstructs the required membrane states within backward execution. Both optimizations operate without CUDA Graph replay. We evaluate SEQUENCE against native and endpoint-specialized SJ kernels, and compare SEGMENT-K with an external backward implementation that follows the same per-position prefix-reconstruction schedule. GRAPH-V2 further replays compatible optimizer and mixed-precision state transitions within complete minibatch updates.

Across four complete CIFAR-10 spiking-Transformer training schedules, SpikingPhoenix achieves 1.414–2.730× end-to-end speedups over SJ, reducing aggregate single-GPU training time from 26.395 to 13.473 hours. The recurrent and graph-enabled measurements below identify how these gains arise from the corresponding GPU computations and training-state operations.

AuthorsOpenReview ↗

Q1 (W1):SEQUENCE implementation and gains over existing fused kernels

For the evaluated LIF operator, native SJ fused Triton exposes time-indexed membrane states and accepts intermediate membrane gradients through grad_v_seq_ptr. SEQUENCE returns the spike sequence and terminal membrane state v_final, retains the FP32 pre-reset trajectory u_seq for the ATan surrogate derivative, and propagates grad_v_final through its reverse-time kernel. The kernel carries the membrane adjoint across time, including the reset Jacobian, using flattened temporal offsets (t*S+offs) rather than the block-pointer addressing and autotuned temporal loops of native SJ Triton.

To distinguish the effect of the state interface from further kernel optimization, we implemented an endpoint-specialized SJ control that preserves SJ's LIF recurrence and surrogate-gradient arithmetic while using the same terminal-state interface. With Graph replay disabled, the FP32 LIF results at T=32 and 262,144 membrane elements on RTX 6000D are:

ImplementationForward (ms)Backward (ms)F+B (ms)Additional allocation (MiB)
Native SJ fused Triton0.6863.0893.770193
SJ endpoint-specialized0.4290.5790.98998
Phoenix SEQUENCE FULL0.3370.4410.79098

Specializing the SJ endpoint reduces backward latency from 3.089 to 0.579 ms, after which Phoenix reduces it further to 0.441 ms. The independently timed F+B latency is 4.77× faster than native SJ and 1.25× faster than endpoint-specialized SJ, while the additional allocation decreases from 193 to 98 MiB relative to native SJ. Forward, backward, and full F+B medians are measured independently.

The CuPy implementation applies the same terminal-state interface using separate forward and backward RawKernels. Its CuPyLIFWorkspace reuses fixed-shape buffers for spikes, pre-reset membrane states, terminal states, and gradients. Backend-matched FP32 F+B measurements from the submitted Appendix D.4 give:

Temporal length TTSJ CuPy (ms)Phoenix CuPy (ms)Speedup
84.9474.0831.212×
164.9874.0301.237×
325.0454.2061.199×
645.8514.8851.198×
1287.5326.1501.225×

Phoenix achieves a 1.214× geometric-mean CuPy F+B speedup across these temporal lengths. We will expand Section 4.1 and Appendix B with the forward–backward state interfaces, reverse-time adjoint organization, Triton addressing, and CuPy buffer reuse, alongside the endpoint-specialized comparison.

AuthorsOpenReview ↗

Q2:Graph capture, optimizer semantics, and complete-update timing

The raw SJ Graph and GRAPH-V2 use different recurrent kernels and optimizer implementations. SJ captures forward, loss, and backward with its native fused recurrent kernels, while conventional timm AdamW and dynamic AMP execute outside the graph. GRAPH-V2 uses Phoenix recurrent kernels and captures compatible recurrent resets, gradient clearing, forward and BatchNorm computation, loss, scaled backward, device-side unscaling and nonfinite checks, conditional fused AdamW updates, and AMP scale/growth transitions.

PyTorch requires stable replay addresses and does not replay ordinary host-side control flow. Accordingly, GRAPH-V2 preallocates stable gradient buffers for the active parameter set, with each captured signature retaining the distinction between grad=None and a materialized zero gradient that AdamW treats differently. Learning-rate changes from the host scheduler reach the update through persistent device scalars, while the device-side nonfinite condition determines whether AdamW steps or is skipped without omitting the corresponding AMP scale transition.

Both systems are timed over the complete synchronized training update, including minibatch staging, forward, loss, backward, AdamW, AMP, and operations outside the SJ graph. Data loading and augmentation are excluded, and initial graph setup is reported separately. The SEW-ResNet18 comparison in Table 3 shows 27.809 versus 26.068 ms per graph-enabled update, while initial setup decreases from 5.47 to 2.11 s. Under the manuscript's Eq. (8), the corresponding setup-amortization thresholds are 119 versus 45 updates.

We also measured spiking Transformers on RTX 6000D at T=4 and FP16, using three runs of 80 synchronized complete updates for each execution mode:

Transformer (batch)SJ raw Graph (ms)Phoenix GRAPH-V2 (ms)SpeedupInitial setup SJ / Phoenix (s)
Spikformer (128)56.00054.4381.029×9.329 / 6.467
QKFormer (64)41.44840.0251.036×12.136 / 7.474
MaxFormer (128)64.53561.9201.042×13.517 / 9.667

Graph replay batches GPU work submissions without fusing the computations inside recurrent kernels. The SEQUENCE and SEGMENT-K comparisons above therefore measure optimizations that also apply without graph capture, while these complete-update measurements show additional system speedups after CUDA Graph replay is enabled in both training stacks and 28.5–38.4% shorter initial setup.

Reviewer_rxJ1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SpikingPhoenix is a GPU training framework for SNNs that combines fused temporal scans, segmented membrane-state rematerialization, and state-complete minibatch graph replay. The paper studies how recurrent execution and minibatch replay boundaries affect runtime and memory while aiming to preserve the reference training update.

Critical strengths
  1. The paper addresses practical GPU training overheads while explicitly considering recurrent and persistent minibatch state.
  2. Controlled placement studies and paired measurements help distinguish execution effects from changes in computational work.
  3. Numerical validation covers recurrent outputs, gradients, update state, and mixed-precision overflow handling.
Critical weaknesses
  1. The advance over existing temporal fusion, checkpointing, and graph replay is insufficiently isolated; Table 3 attributes most replay acceleration to the raw CUDA Graph control.

  2. Complete Transformer accuracies are missing, and GRAPH-V2 has lower final accuracy for all three SEW-ResNet18 seeds, leaving long-run training-quality preservation insufficiently supported.

  3. In Table 2, segmented Phoenix still uses more process memory than SJ and MemOpt; comparisons at equal memory budgets are missing.

  4. ImageNet evaluates SEQUENCE only, while accelerated kernels cover fixed-parameter IF/LIF configurations and replay requires fixed signatures; full-pipeline transfer and recapture costs remain untested.

  5. RTX 6000D profiling lacks Nsight and driver version details needed to assess compatibility with the reported CUDA 13.2 stack; several references also omit publication venues.

Questions for authors
  1. What gains remain against matched temporal-fusion and compiled, graph-enabled baselines, and which additional state transitions account for GRAPH-V2’s improvement over raw CUDA Graphs?

  2. Can you provide final and best accuracies for all four complete Transformer schedules and explain the SEW-ResNet18 accuracy shifts, including whether replay checks use identical data and RNG trajectories?

  3. How does Phoenix compare with SJ and MemOpt at equal process-memory budgets, and how should K be selected when graph storage and recomputation costs are included?

  4. Can you evaluate GRAPH-V2 on ImageNet and quantify recapture overhead for changing shapes or gradient participation, as well as support for learnable neuron parameters?

8 comments

We thank the reviewer for probing the source of our speedups and their relevance to full training under realistic memory constraints. SpikingPhoenix preserves the neuron dynamics and learning rule while reorganizing their execution on GPUs. SEQUENCE carries the terminal membrane adjoint through a reverse-time scan, SEGMENT-K reconstructs the required states inside backward execution, and GRAPH-V2 replays the conditional optimizer and AMP transitions that connect successive minibatches. With the terminal-state interface matched, SEQUENCE remains 1.25× faster than SJ. When external backward performs the same prefix reconstruction, SEGMENT-K is 1.117–1.828× faster.

The gains also carry through complete training. Across CIFAR-10 spiking-Transformer schedules, SpikingPhoenix delivers 1.414–2.730× end-to-end speedups, reducing total single-GPU time from 26.395 to 13.473 hours. We provide the requested final and best accuracies, supported by paired 200-epoch training, a fixed-checkpoint backend crossover, and consecutive Graph/eager updates with matching model, optimizer, BatchNorm, and AMP states. On ImageNet-1K, GRAPH-V2 SEGMENT-8 also reduces complete update-call latency by 1.144× while fitting the same 33 GiB process-memory ceiling as SJ native Graph. Together, these results show that SpikingPhoenix reduces the GPU time required for direct SNN training while preserving the prescribed recurrent computations and training-state transitions. The resulting savings make repeated training and broader architecture comparisons more affordable within a fixed GPU budget.

AuthorsOpenReview ↗

Q1 (W1): Incremental gains and GRAPH-V2's contribution

We appreciate the request to distinguish recurrent optimizations from fusion, checkpointing, and Graph replay. SEQUENCE remains faster after matching SJ's terminal-state interface. At T=32 with 262,144 FP32 LIF elements and Graph disabled:

ImplementationF+B (ms)Forward (ms)Backward (ms)Extra alloc. (MiB)
SJ fused Triton3.770——193
Endpoint-specialized SJ0.9890.4290.57998
Phoenix SEQUENCE0.7900.3370.44198

SEQUENCE is 4.77× faster than native SJ and 1.25× faster than the matched-interface control, with equal additional allocation. It passes grad_v_final through the reverse-time adjoint scan instead of SJ's time-indexed grad_v_seq_ptr, retaining FP32 pre-reset values for the ATan derivative and using flattened t*S+offs indexing.

SEGMENT-K isolates persistent execution from reconstruction work. Its external control uses the same segmented forward pass, FP32 checkpoints, per-position prefix reconstruction, LIF recurrence, and adjoints. Both reconstruct R(T,K)=∑jkj(kj+1)/2R(T,K)=\sum_j k_j(k_j+1)/2 recurrences per element, without Graph:

TTKKRecurrences/elementExternal (ms)Persistent (ms)Gain
164403.6422.7561.321×
168722.7832.4921.117×
324804.3652.5201.732×
3281443.3452.5531.310×
6441604.8592.6581.828×
6482883.4062.6411.290×

At (T,K)=(64,8)(T,K)=(64,8), each path performs 56,623,104 reconstructed neuron recurrences across three modules and allocates 99.25 MiB. Persistent backward carries the adjoint in-kernel, reducing temporal launches from 9 to 2 per module (including forward). Table 11 (Appendix D.3) separately documents the longer-horizon conventional checkpoint schedule. At Q=⌈T/K⌉=64Q=\lceil T/K\rceil=64, its external latency remains near 680 ms across T=128,256,512T=128,256,512, illustrating the cost of repeated segment orchestration.

GRAPH-V2 expands the captured update beyond forward/loss/backward. SJ raw Graph executes timm AdamW and dynamic AMP outside capture. Phoenix replays recurrent resets, gradient clearing, compatible BatchNorm operations, device-side nonfinite checks, conditional fused AdamW, and AMP scale/growth transitions. A persistent device learning-rate scalar and fixed active-gradient signature preserve scheduled LR and AdamW's grad=None versus zero-gradient behavior. Both graph-enabled stacks were timed over complete updates, including optimizer/AMP work:

Model (batch)Eager SJ / Phoenix (ms)Graph SJ / Phoenix (ms)Graph gainSetup SJ / Phoenix (s)
Spikformer (128)116.405 / 83.86256.000 / 54.4381.029×9.329 / 6.467
QKFormer (64)111.404 / 75.48741.448 / 40.0251.036×12.136 / 7.474
MaxFormer (128)103.290 / 77.03564.535 / 61.9201.042×13.517 / 9.667

Three runs per mode measured 80 synchronized FP16 updates at T=4, including prepared-batch staging, forward, loss, backward, AdamW, and AMP, excluding loading/augmentation. Phoenix gains are 1.341–1.476× in eager and 1.029–1.042× with Graph on both sides, with 28.5–38.4% shorter setup. In Table 3, SEW-ResNet18 Graph latency falls from 27.809 to 26.068 ms and setup from 5.47 to 2.11 s, reducing the eager-relative break-even point from 119 to 45 updates.

GRAPH-V2's value is not limited to its additional steady-state speedup: it makes conditional optimizer and mixed-precision state evolution part of reusable GPU execution. The Graph/eager checks in Q2 show matching parameters, buffers, AdamW, and AMP states at every step for each implementation against its eager counterpart. By combining faithful stateful-update replay, reduced host-side orchestration, and lower capture cost, GRAPH-V2 complements the Graph-independent recurrent and memory optimizations of SEQUENCE and SEGMENT-K, helping their gains translate into efficient full SNN training.

AuthorsOpenReview ↗

Q2 (W2): Full accuracy, FP16 variation, and update fidelity

Thank you for identifying the missing accuracy columns. Table 4 already reports full-training times, and we will add these final/best CIFAR-10 top-1 accuracies (310 epochs for Spikformer, 410 for the others):

ModelSJ finalPhoenix finalSJ bestPhoenix best
Spikformer94.50%94.22%94.70%94.67%
QKFormer96.01%96.00%96.32%96.38%
MaxFormer96.74%96.64%96.85%96.81%
MS-QKFormer96.73%96.19%96.83%96.50%

We will correct Section 5.5 and Appendix E.4 cross-references.

The SEW-ResNet18 FP16 results in Table 15 and Figure 5 (Appendix E.1) include SJ, eager SEQUENCE, and GRAPH-V2 over the same 100-epoch recipe (T=4T=4, batch 128, 3 seeds). Their mean final accuracies are 88.00%, 87.74%, and 87.40%, respectively, with across-seed ranges of 0.80, 1.52, and 1.06 pp. Even without Graph, SEQUENCE obtains 88.59% versus SJ's 88.18% for seed 342. Thus the endpoint variation also occurs in eager recurrent execution, consistent with the sensitivity of FP16 spike-threshold training trajectories to numerical perturbations.

We additionally completed three paired 200-epoch FP32 eager runs per backend. Each pair starts from identical weights, and 78,000 optimizer steps in every run. Mean accuracy and paired differences (Phoenix−SJ) are:

EpochSJ (%)Phoenix (%)Checkpoint Δ (pp)Prior 40 epochs Δ (pp)
4073.8172.92−0.89−0.10
8075.4572.60−2.85−0.35
12079.6378.98−0.65+0.84
16086.0985.94−0.15+0.05
20088.4688.95+0.50+0.31

Phoenix attains 88.95% versus 88.46% mean final accuracy and 89.21% versus 88.82% mean best accuracy. All three paired final differences and final 40-epoch mean differences favor Phoenix. The mean gap changes from −2.85 pp at epoch 80 to +0.50 pp at epoch 200, as both backends continue improving through the later training stages. In an SNN, a small numerical perturbation near the firing threshold can switch a binary spike between 0 and 1, altering the membrane reset and subsequent surrogate-gradient updates. These discrete events can produce different optimization trajectories even with the same initial weights and augmented minibatches. The positive late-window results show that Phoenix's final advantage extends beyond a single checkpoint, providing long-run evidence to the original FP16 seedwise study.

Direct numerical and persistent-state controls provide further evidence:

ControlVerified result
SJ vs. Phoenix, identical FP32 checkpointIdentical logits/loss and 314,572,800/314,572,800 spikes
Same checkpoint/Max input-gradient difference 7.05×10−77.05\times10^{-7}
QKFormer Graph vs. its corresponding eager, 12 updatesMatching loss and 107/107 parameters every step
Same 12 updates93/93 buffers (including BN), 321/321 AdamW states, and AMP scale/growth matched every step
FP32 fixed-input trace35 neuron resets, 26 matching activation records

Switching backend at a fixed checkpoint preserves spikes, whereas changing checkpoints changes them. Graph tests begin from matched post-capture states, use identical cached augmented batches, and restore RNG state around capture. Each Graph implementation matches its own eager path across the 12 updates. Together, these controls establish fixed-state recurrent agreement and faithful replay of the state transitions governing subsequent training.

Reviewer_N5tkOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper speeds up GPU training of spiking neural networks by reorganizing how recurrence, backward-required state, and minibatch updates are executed. It proposes three transformations: fused temporal scans, segment-based rematerialization inside the backward kernel, and state-complete CUDA Graph replay, and shows they preserve training semantics while improving throughput across spiking Transformers, GPUs, and complete CIFAR-10 schedules.

Critical strengths
  1. The authors take semantics seriously. They check spikes, membrane states, gradients, and optimizer state across precisions, including AMP overflow and recovery, so the “same training, faster execution” claim is actually verified.

  2. The evaluation transfers well. Speedups hold across four spiking Transformer architectures, two GPUs, ImageNet-1K batch and horizon sweeps, and complete CIFAR-10 schedules.

Critical weaknesses
  1. The methodological novelty appears limited relative to the strength of the engineering contribution.

  2. The framework builds on established techniques, including fused recurrent execution, rematerialization, and CUDA Graph replay.

  3. While the implementation and semantic validation are careful, the additional benefit over an optimized combination of existing techniques remains insufficiently established.

Questions for authors
  1. Given the K sweeps in Table 2 and Appendix D, could the authors provide a practical guideline for choosing K under a given memory budget?

  2. Table 3 shows that raw CUDA Graph replay accounts for most of the latency reduction on SEW-ResNet18. How much additional speedup does SpikingPhoenix provide over SpikingJelly with CUDA Graph replay on the Transformer workloads?

  3. A paired comparison on a representative Transformer, preserving the same training-update semantics and including any work performed outside the graph in the timing, would help clarify the benefit of the proposed execution optimizations.

7 comments

We thank the reviewer for recognizing the semantic validation and transfer results. SpikingPhoenix improves the computation and memory costs of direct SNN training through recurrent differentiation, temporal-state rematerialization, and replay of persistent training-state updates. SEQUENCE specializes the recurrent forward–backward interface, while SEGMENT-K reconstructs membrane prefixes inside backward execution to reduce temporal state retention. Neither requires CUDA Graph replay, which GRAPH-V2 uses separately to capture compatible optimizer and mixed-precision transitions within the minibatch update.

NVIDIA distinguishes fusion inside GPU kernels from CUDA Graph batching of kernel launches, which reduces host submission overhead without automatically combining the kernels' computations. PyTorch additionally requires stable replay addresses and compatible shapes and control flow. Consequently, SEQUENCE and SEGMENT-K remain applicable when complete-update capture is unavailable, whereas GRAPH-V2 provides a further optimization for training paths that satisfy those requirements.

On SEW-ResNet18 (T=16, batch 64, FP16), the measured throughput and NVML process-memory peaks show how these execution modes compare:

ExecutionGraphThroughputProcess peak
SpikingJelly (SJ) FULLOff770.9 img/s5.686 GiB
SpikingPhoenix SEQUENCE FULLOff1052.5 img/s8.643 GiB
SpikingPhoenix SEQUENCE SEGMENT-8Off1057.0 img/s5.889 GiB
SpikingPhoenix GRAPH-V2 SEGMENT-8On1196.0 img/s6.956 GiB

Relative to SJ FULL, SEQUENCE FULL delivers 1.365× the throughput while using more process memory. With Graph disabled, SEQUENCE SEGMENT-8 lowers the Phoenix FULL process peak by 31.9% and reaches 1057.0 img/s, whereas enabling GRAPH-V2 on SEGMENT-8 increases throughput to 1196.0 img/s, a further 1.132×. The four complete CIFAR-10 Transformer schedules achieve 1.414–2.730× end-to-end speedups, saving 12.922 single-GPU hours in aggregate as wall time decreases from 26.395 to 13.473 hours.

Without Graph replay, SEQUENCE accelerates all eight architecture–GPU combinations evaluated across RTX 6000D and RTX 5090. Its transfer to ImageNet-1K is also measured with Graph and local architecture rewrites disabled, yielding geometric-mean SEQUENCE/SJ speedups of 1.446×, 1.426×, and 1.373× across four Transformer families at T=2,4,8T=2,4,8, respectively. Mechanism-level controls give a further 1.25× F+B speedup over endpoint-specialized SJ for SEQUENCE and 1.117–1.828× over matched-prefix external backward for SEGMENT-K.

AuthorsOpenReview ↗

Q1: Choosing K under a memory budget

The submitted code implements budget-aware planning across stateful modules. StateProfileSession profiles state shapes and temporal dimensions, while StateCostModel combines analytical storage and recomputation estimates with measured execution costs. UniversalStatePlanner then searches a global Pareto frontier for module-specific FULL, qualified RECOMPUTE, and SEGMENT-K policies, accounting for base memory, reusable workspace, persistent state, and Graph-pool storage under a shared process-memory cap. The implementation exposes both CompiledSNN.plan_states() and cli_state_plan.py, allowing different recurrent modules to receive different policies instead of enforcing one global K.

For Q=⌈T/K⌉Q=\lceil T/K\rceil segments and SS membrane elements, the Q−1Q-1 interior FP32 checkpoints require 4S(Q−1)4S(Q-1) bytes, while backward prefix reconstruction performs R(T,K)=∑jkj(kj+1)/2R(T,K)=\sum_j k_j(k_j+1)/2 recurrence evaluations per element. The planner uses this time–memory trade-off together with measured update costs when choosing K and other state policies across the model.

For ImageNet-1K QKFormer-10-384 (T=16, batch 16, FP16), the following eager configurations were measured with Graph replay disabled:

PolicyFP32 boundariesPrefix recurrences / neuronNVML process peakComplete-update latency
SEGMENT-272435.646 GiB564.420 ms
SEGMENT-434032.613 GiB653.347 ms
SEGMENT-817230.496 GiB597.381 ms

Under a 32 GiB process-memory cap, SEGMENT-8 is the fastest measured feasible eager choice, whereas a 36 GiB cap admits the faster SEGMENT-2 configuration. In GRAPH-V2 mode, SEGMENT-8 uses 32.148 GiB and takes 529.457 ms per update, which is why the planner accounts for the storage costs of the selected execution mode.

AuthorsOpenReview ↗

Q2 & Q3: Complete Transformer updates with and without CUDA Graphs

With CUDA Graph replay enabled on both systems, SpikingPhoenix GRAPH-V2 achieves 1.029–1.042× faster complete training updates than SpikingJelly raw Graph on three spiking Transformers, while reducing initial graph setup time by 28.5–38.4%. On the same workloads, Phoenix achieves 1.341–1.476× speedups in eager execution, without CUDA Graph replay. We measured each model on an RTX 6000D at T=4 and FP16, using three runs of 80 synchronized updates per execution mode.

Model (batch)Eager update: SJ / PhoenixEager gainGraph update: SJ / PhoenixGraph gainInitial setup: SJ / Phoenix
Spikformer (128)116.405 / 83.862 ms1.388×56.000 / 54.438 ms1.029×9.329 / 6.467 s
QKFormer (64)111.404 / 75.487 ms1.476×41.448 / 40.025 ms1.036×12.136 / 7.474 s
MaxFormer (128)103.290 / 77.035 ms1.341×64.535 / 61.920 ms1.042×13.517 / 9.667 s

Both systems follow the same training-update specification, with their respective recurrent kernels and optimizer implementations. The measured latency includes minibatch staging, forward computation, loss, backward computation, AdamW, and AMP, including operations executed outside SJ's captured graph. Data loading, augmentation, and initial setup are excluded from steady-state latency, with setup measured separately. SJ raw Graph captures forward, loss, and backward while executing conventional timm AdamW and dynamic AMP outside the graph. GRAPH-V2 uses Phoenix recurrent kernels and captures compatible neuron resets, gradient clearing, forward and BatchNorm computation, loss, scaled backward, device-side nonfinite checks, conditional fused AdamW updates, and AMP scale/growth transitions. The active-gradient set remains fixed within each captured execution signature, preserving AdamW's distinct treatment of absent and zero gradients, while a persistent device scalar receives learning-rate changes from the host scheduler.

NVIDIA documents that CUDA Graph launch overhead depends on graph topology, GPU architecture, and the CUDA software stack. Graph replay reduces work-submission overhead, while recurrent neuron dynamics and the temporal-state computations required for BPTT still need to be executed. As SNN models grow in scale and complexity, efficient direct training has become an important research direction, motivating advances in GPU-accelerated SNN frameworks, temporal fusion, and memory-efficient BPTT. SEQUENCE accelerates recurrent forward–backward computation, while SEGMENT-K reduces temporal state retention through in-kernel reconstruction, both without relying on graph capture. GRAPH-V2 further optimizes compatible complete training updates by capturing optimizer and mixed-precision state transitions. By reducing the GPU cost of direct SNN training without changing the reference learning procedure, SpikingPhoenix enables broader architecture exploration, more comprehensive ablation studies, and repeated evaluations within the same computational budget, supporting more systematic and reproducible SNN research.

Comments on the paper

1 comment

Dear Reviewers,

We sincerely thank you for your thoughtful feedback and valuable suggestions on our work! We have carefully addressed your concerns, completed the latest revision of our manuscript, and incorporated additional experiments and discussions to further strengthen our work.

We believe that SpikingPhoenix makes a meaningful scientific contribution. Beyond accelerating semantics-preserving direct SNN training, our findings reveal how the effectiveness of an optimization depends on its surrounding execution context. By substantially reducing training costs, SpikingPhoenix also makes broader architectural comparisons, longer temporal investigations, and more ambitious SNN research increasingly accessible.

We sincerely hope that our clarifications, additional evidence, and revised manuscript have satisfactorily addressed your concerns and further demonstrated the significance of our work. If you find these additions convincing, we would be truly grateful if you would consider updating your score to reflect the new results and discussion.

Thank you again for your time, patience, and thoughtful consideration. We remain happy to address any further questions or concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.