acceptodds
Under review as a conference paper at ICLR 2027

ReLIF: Exploiting Temporal Parallelism through Finite-Time Coalescence in Spiking Neural Networks

Abstract

Spiking neural networks carry recurrent neuronal states across time, making exact temporal execution serial even when the surrounding network is highly parallel. We introduce ReLIF, which rectifies the recurrent state before integration. Under a shared input sequence, sufficiently negative inputs erase the carried state through rectification, while sufficiently positive inputs trigger a common hard reset, allowing executions from different incoming membrane states to converge to the same zero state. We formalize this event as finite-time state coalescence and develop Coalescent Chunking, an exact execution scheme that precomputes boundary-independent suffixes in the forward pass and composes independent gradient intervals in the backward pass. ReLIF matches LIF accuracy across vision backbones and shows better performance in language modeling. In kernel-level measurements, parallel ReLIF reaches up to forward and backward speedup over SpikingJelly CuPy LIF. Across three SpikeTransformer++ scales, the complete stack reaches – inference and – training throughput over paired SpikingJelly CuPy LIF.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Spiking neural networks, Temporal parallelism, Parallel computing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xDxDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReLIF modifies hard-reset LIF by rectifying the membrane carried across time, so that trajectories entering from different states can meet exactly at zero after a short prefix, either through rectification or through a hard reset. Coalescent Chunking uses that meeting point to precompute the boundary-independent suffix in the forward pass and to split the backward pass into independent gradient segments, while matching stepwise ReLIF. With the membrane time constant fixed at 2, accuracy stays close to LIF on CIFAR-10, CIFAR-100, and CIFAR10-DVS, and validation perplexity improves at three SpikeTransformer++ scales on WikiText-103. In those trained models, about 98% of length-32 chunks meet within the chunk. On a GPU, parallel CUDA ReLIF is substantially faster than SpikingJelly CuPy LIF at the operator level, and full-model inference and training throughput rise by about 2.9–3.2 times and 2.0–2.1 times; against a matched serial CUDA ReLIF, the end-to-end gains are only about 1.1–1.3 times for inference and 1.2–1.5 times for training.

Critical strengths
  1. The parallelism is a property of the recurrence, and the execution is built to stay exact. Rectifying the carried state is a small change, but it gives a bounded entry domain and two transitions that map a whole reachable set to zero. Chunking then follows: suffixes that no longer depend on the incoming membrane can be precomputed, and zero-Jacobian anchors cut the backward scan. This is more convincing than a kernel that happens to be fast. The paper states the coalescence criterion (Theorem 2), separates ReLU coalescence from reset coalescence, shows why unbounded-below LIF does not get the same finite-time guarantee, and treats the coalescing step as still boundary-dependent so that forward outputs and surrogate gradients match stepwise ReLIF. The round-safe hull argument is aimed at the right failure mode (a certificate that is false in FP16/BF16/FP32). That exactness claim is what makes the speed numbers interpretable, and it is the main reason the algorithmic contribution is real rather than a rewrite of associative scan.

  2. The experiments check the mechanism, not only the leaderboard. The same replacement is run in a conv net, a spike Transformer, and a 4096-token language model at three scales, with three seeds and a paired LIF control. ReLIF does not trade away accuracy; on WikiText-103 the perplexity gap widens with scale. They then measure coalescence on currents from those trained models (about 98% within a chunk, suffix about 85%, survival below 10% by step 12), and on synthetic currents the suffix ratio tracks forward speedup at r=0.994. The serial CUDA ReLIF control is also the right ablation: it keeps the neuron and the tensor interface fixed and changes only the schedule. Together, these results support the claim that finite-time coalescence actually occurs in trained networks and that exploiting it changes operator runtime.

Critical weaknesses
  1. Inference time should be measured on asynchronous hardware, or at least in a simulator, to show how much it is reduced.
  2. ImageNet results are needed to show that the accuracy drop remains small.
Questions for authors

See the Weaknesses.

3 comments

Response to Reviewer xDxD

W1: Full-model inference latency on asynchronous hardware.

Thank you for your careful review. Following your suggestion, we evaluated full-model inference on a proposed asynchronous architecture.

To evaluate asynchronous hardware as you suggested, we must provide the full-window inputs required by Coalescent Chunking. Loihi's neuromorphic cores keep local states and process spikes, update states, and route events sequentially at tt, t+1t+1, and t+2t+2. This does not expose complete windows for concurrent chunks. Layer-by-layer GPU execution instead computes each layer over the full window before passing its outputs onward, improving projection and convolution efficiency. Neuron layers receive full-window currents, but conventional LIF remains sequential. ReLIF and Coalescent Chunking exploit within-window temporal parallelism through finite-time coalescence. We add window buffers to the asynchronous design to supply concurrent chunks.

We use a trace-driven hybrid timing model for this Loihi-style design, based on SANA-FE, a neuromorphic architecture simulator. Recorded currents, coalescence points, and replay lengths drive discrete-event scheduling of neuronal execution, including SRAM access, certification, replay, and boundary transfers. Projections, attention, and tensor transfers use analytic timing constrained by compute, bandwidth, and cache resources. Both schedules use the same hardware budget, neuronal dynamics, and complete inference workload. The same configuration is used at all three scales.

ComponentConfiguration
Neuronal compute128 partitions, each with 64 FP32 microengines at 200 MHz
FP32 timing parametersOutput step 17 cycles; hull update 30 cycles
On-chip storage48 MiB neuronal SRAM, 16 MiB matrix SRAM, and 2 MiB working registers in total
Matrix compute256 arrays of 16×16 FP32 MACs at 500 MHz
Vector and special-function units4096 vector lanes and 1024 special-function pipelines at 500 MHz
External memory2 GiB capacity and 512 GB/s bandwidth
Data interconnect8×4 mesh with four parallel 64-byte rails per link at 500 MHz and 512 GB/s bisection bandwidth

We evaluate seed42 57M, 102M, and 191M SpikeTransformer++ models with batch size 1, 4096 tokens, and FP32. Complete outputs are checked for bitwise agreement with an independent stepwise reference before timing the recorded graph, covering attention, feedforward projections, neurons, the vocabulary head, and full logits writeback. The table gives predicted latencies.

ModelStepwise execution (ms)Coalescent Chunking (ms)Speedup
57M23.56013.8411.702×
102M41.89128.0631.493×
191M59.35448.8591.215×

Latency is Ttotal=Tcommon+TneuronT_{\mathrm{total}}=T_{\mathrm{common}}+T_{\mathrm{neuron}}, with common operator and transfer costs TcommonT_{\mathrm{common}} shared by both schedules. The fractions and local speedups below come from this decomposition, so full-model gains follow Amdahl's relation.

ModelNeuronal share of serial full-model timeLocal neuronal speedupEnd-to-end speedup
57M53.72%4.31×1.702×
102M45.32%3.68×1.493×
191M31.99%2.24×1.215×

With fixed resources, wider networks allocate more microengines to spatial parallelism, leaving fewer temporal resources per trajectory. Attention, projections, and vocabulary output also take a larger runtime share. Consequently, under this hardware configuration, full-model inference speedup decreases as model size increases. The model predicts 1.215–1.702×\boldsymbol{1.215\text{–}1.702\times} full-model inference speedups for Coalescent Chunking on the window-capable asynchronous architecture.

W2: Recognition quality on a higher-resolution visual task.

Following your suggestion, we add paired ImageNet-100 evaluation using all training images from the selected classes and 5,000 validation images. Time and GPU constraints make full ImageNet-1K training from scratch currently difficult.

SEW-ResNet18 is trained from scratch with a membrane time constant of 22 and paired initialization, data processing, and training settings. The table reports means and sample standard deviations of each run's best Top-1 across seeds 42, 123, and 2026.

ModelNeuronEpochsTTTop-1 (%)
SEW-ResNet18LIF80472.17 ± 0.34
SEW-ResNet18ReLIF80472.41 ± 0.75

ReLIF improves mean Top-1 by 0.24\boldsymbol{0.24} percentage points. ReLIF maintains LIF-level recognition accuracy on this higher-resolution task.

The architecture model predicts that Coalescent Chunking reduces full-model inference time on a window-capable asynchronous architecture. The three-seed ImageNet-100 results also show that ReLIF preserves recognition accuracy at higher resolution. We welcome any further questions you may have.

You solve my questions. I would raise the score.

Dear Reviewer xDxD,

We are grateful for your helpful feedback and for raising the score after considering our rebuttal. Your suggestions helped us broaden and strengthen the evaluation.

We have uploaded the updated manuscript, with changes highlighted in blue. The revisions addressing your comments are listed below.

  • Section 5.6 and Appendix E describe the window-capable asynchronous architecture, hybrid timing model, and predicted full-model inference latencies.
  • Section 5.2 and Appendix D.9 present the three-seed ImageNet-100 comparison, with training settings in Appendix C.1.

Thank you again for your careful consideration and for helping us improve the paper.

Reviewer_DMCNOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReLIF and Coalescent Chunking. The goal is to exploit the bounded membrane potential after ReLU so that different boundary states coalesce to zero within a finite number of steps, thereby dividing the otherwise serial recurrent-state computation into parallelizable chunks. The paper gives a lemma stating that the first coalescence occurs at zero, necessary and sufficient conditions over fixed parameters and reachable sets, and uses reachable-set certificates, replay of prefixes from the true boundary state, and zero-local-Jacobian anchors to handle forward and backward propagation. The experiments cover vision replacements, WikiText-103, real currents, operator-level measurements, and end-to-end training/inference, reporting approximately 1.07–1.50× speedup relative to serial CUDA ReLIF and larger speedups relative to CuPy LIF.

Critical strengths
  1. ReLIF's finite-time state coalescence and the execution strategy of Coalescent Chunking form a relatively tight dynamics–systems co-design. The theoretical conditions can be converted into practical chunk certificates rather than serving only as an empirical parallelization heuristic.
  2. The validation is relatively comprehensive, covering theory, finite-precision certificates, CUDA operators, real currents, and vision and language tasks. The matched serial ReLIF baseline also helps isolate part of the scheduling gain.
Critical weaknesses

There’s a lack of experiments and measurements on how the ReLIF impact firing rate and performance of (long) time sequence tasks, which might impact the capabilities of SNNs.

Questions for authors
  1. Please add experiments on temporal and long-sequence tasks to examine whether ReLIF changes task quality and whether this design damages the temporal modeling capability of SNNs in exchange for parallelism. Its effect on firing rate is also unclear, which may alter the inference-efficiency properties traditionally emphasized for SNN models.
  2. Could you include a chunked LIF baseline, in which the state is hard-reset to zero at fixed chunk boundaries, and perform an ablation to determine whether the effect comes from this simple way of cutting temporal dependencies or from the complete ReLIF dynamics design?
5 comments

Response to Reviewer DMCN (part 1 of 2)

Q3: Periodic resetting and input-driven coalescence.

Thank you for this insightful suggestion. We agree that fixed-boundary resetting offers direct dependency management. The key distinction is that a reset period determines when the model forgets, whereas the chunk size in Coalescent Chunking determines only how computation is organized. Fixed-boundary Chunked LIF zeros membranes at predefined positions, cutting cross-chunk dependencies and defining new dynamics. ReLIF's dependency endpoints are input-driven. Coalescent Chunking certifies coalescence through reachable sets and replays still-dependent prefixes from true boundary states. Unresolved state dependencies continue across chunks, with no fixed memory limit.

Within the time and computational budget of this response, we have prioritized evaluating ReLIF's modeling capabilities, finite-time coalescence, and the practical gains from exact parallel execution. We have supplemented evaluations of neuronal dynamics, temporal modeling, and computational efficiency around these objectives. We therefore have not pursued further matched ablations of fixed-boundary resetting in this round. We greatly appreciate this constructive comparison direction, which has helped us articulate the design principle of our work more clearly. Our temporal parallelism derives from input-driven, certifiable state coalescence without imposing additional fixed-period state truncation.

Q2 & W1: Temporal modeling, firing activity, and inference energy efficiency.

Thank you for your careful review. We address temporal modeling and the efficiency impact of firing changes through feedforward and recurrent psMNIST, activity statistics, and event-driven energy evaluation. WikiText-103 uses the manuscript's 40964096-token configuration. The table reports validation perplexity means and sample standard deviations across seeds 42, 123, and 2026.

ModelLIF PPLReLIF PPL
57M62.74 ± 0.8960.55 ± 0.54
102M54.02 ± 0.2749.29 ± 0.08
191M45.89 ± 1.5940.81 ± 0.29

Under equal budgets, ReLIF's PPL improvement grows with scale. We further test sequence-order use with fixed-permutation, 784784-step psMNIST. The feedforward network is 1–64–256–256–101\text{–}64\text{–}256\text{–}256\text{–}10, with LayerNorm and spiking neurons in all three hidden layers and temporal-mean readout. The recurrent version adds 256×256256\times256 learnable synapses to the second hidden layer, feeding back previous-step spikes with full 784784-step backpropagation. Paired models share splits, permutations, initialization, and training settings. The psMNIST and activity tables report three-seed means and sample standard deviations; “Shuffled” adds temporal shuffling beyond the fixed permutation.

ArchitectureNeuronTest acc. (%)Shuffled acc. (%)Drop (pp)
FeedforwardLIF39.88 ± 1.7017.96 ± 2.4121.92 ± 0.83
FeedforwardReLIF39.65 ± 0.7021.05 ± 1.0218.60 ± 0.87
RecurrentLIF72.36 ± 0.7517.54 ± 1.1854.82 ± 1.47
RecurrentReLIF73.26 ± 1.3316.40 ± 1.8856.86 ± 2.53

Feedforward ReLIF differs from LIF by 0.23\boldsymbol{0.23} percentage points, while recurrent ReLIF improves accuracy by 0.90\boldsymbol{0.90} percentage points. Temporal shuffling reduces feedforward and recurrent ReLIF accuracy by 18.60\boldsymbol{18.60} and 56.86\boldsymbol{56.86} percentage points, respectively. Both architectures use sequence order and retain the temporal modeling ability needed for the 784\boldsymbol{784}-step task; improved long-context PPL further supports task performance.

Firing rate is the fraction of output elements containing spikes, and silence is the fraction of channels that never spike during a sequence. psMNIST uses the first eight test batches. Units are percent.

SettingLIF firingReLIF firingLIF silenceReLIF silence
psMNIST (feedforward)9.15 ± 1.3311.57 ± 0.3840.31 ± 18.3425.45 ± 2.60
psMNIST (recurrent)19.74 ± 0.5120.38 ± 0.509.12 ± 0.658.86 ± 1.06
SpikeTransformer++ 57M9.26 ± 0.1410.86 ± 0.051.58 ± 0.300.54 ± 0.10
SpikeTransformer++ 102M8.01 ± 0.049.61 ± 0.132.25 ± 0.320.85 ± 0.18
SpikeTransformer++ 191M7.47 ± 0.089.17 ± 0.052.00 ± 0.220.71 ± 0.09

Feedforward and recurrent psMNIST firing increases by 2.42\boldsymbol{2.42} and 0.64\boldsymbol{0.64} percentage points, while silence decreases by 14.86\boldsymbol{14.86} and 0.26\boldsymbol{0.26} percentage points. Language-model firing increases by 1.60–1.71\boldsymbol{1.60\text{–}1.71} percentage points and silence decreases by 1.04–1.39\boldsymbol{1.04\text{–}1.39} percentage points. We believe more active responses and fewer silent channels strengthen inter-layer information transfer in SpikeTransformer++, helping explain PPL improvements at all three scales.

Response to Reviewer DMCN (part 2 of 2)

Q2 & W1: Temporal modeling, firing activity, and inference energy efficiency.

To assess energy cost, we use hybrid inference with Tensor Cores for dense computations and event-driven spike projection operators that run on CUDA Cores, skip zero inputs, and load and accumulate weights only at active positions. We evaluate existing paired seed42 checkpoints with the same event operators at batch=1 on one validation sequence, covering Q/K/V/O and gate/up/down projections in every layer with three repetitions. Prefill processes 40964096 tokens. Decode comprises 20482048 measured single-token forward steps after a 20482048-token prefix, with continuously updated KV caches and membrane states.

L and R denote LIF and ReLIF. Cumulative NVML GPU energy is normalized as mJ/token\mathrm{mJ/token}, SOPs are in millions per token, and Loads R/L is the ratio of logical weight-load requests after sharing within token groups. Total SOPs=∑pnpdout,p\mathrm{SOPs}=\sum_p n_p d_{\mathrm{out},p} is normalized per token, where npn_p counts active input spikes to projection pp and dout,pd_{\mathrm{out},p} is its output width.

ModelPhaseFiring L / R (%)SOPs L / RLoads R/L
57MPrefill9.26 / 10.842.389 / 2.6761.085
57MDecode9.20 / 10.732.387 / 2.6661.117
102MPrefill8.02 / 9.775.677 / 6.5471.114
102MDecode7.96 / 9.695.641 / 6.5071.154
191MPrefill7.45 / 9.2211.678 / 13.6581.128
191MDecode7.40 / 9.1511.641 / 13.5951.168

Higher firing entails more event accumulations, whose cost depends on downstream connection widths. At 57M, 102M, and 191M, prefill SOPs increase by 12.0\boldsymbol{12.0} percent, 15.3\boldsymbol{15.3} percent, and 17.0\boldsymbol{17.0} percent, while weight-load requests increase by 8.5\boldsymbol{8.5} percent, 11.4\boldsymbol{11.4} percent, and 12.8\boldsymbol{12.8} percent. Weight-load requests grow more slowly because our event-driven projection kernel processes 88 tokens together during prefill, allowing active spikes at the same input position within the group to share a single weight load and reduce repeated loads. Decode processes one token at a time, with weight-load requests and SOPs both increasing by 11.7\boldsymbol{11.7} percent, 15.4\boldsymbol{15.4} percent, and 16.8\boldsymbol{16.8} percent. The table gives measured projection energy in mJ/token\mathrm{mJ/token}.

ModelPhaseLIFReLIF
57MPrefill1.0191.100
57MDecode108.510119.036
102MPrefill2.5442.833
102MDecode172.992195.941
191MPrefill5.3026.061
191MDecode311.219365.380

Within-group weight sharing, cache reuse, and amortized launch and scheduling costs give prefill lower energy per token than single-token decode. Across the three scales, projection energy increases over LIF by 7.9\boldsymbol{7.9} percent, 11.4\boldsymbol{11.4} percent, and 14.3\boldsymbol{14.3} percent during prefill and 9.7\boldsymbol{9.7} percent, 13.3\boldsymbol{13.3} percent, and 17.4\boldsymbol{17.4} percent during decode. Wider activations and downstream connections amplify the event-accumulation and weight-access costs of firing differences, while Tensor Core dense computation costs change relatively little, giving larger projection-energy increases at larger scales. Relative to the end-to-end decode energy of same-scale serial CUDA LIF, the increased decode projection energy is equivalent to only 1.3\boldsymbol{1.3} percent, 1.8\boldsymbol{1.8} percent, and 4.0\boldsymbol{4.0} percent of that budget, respectively.

Under the SpikingJelly GPU execution used in the manuscript, projections use dense operators, so additional spikes change neither matrix dimensions nor dense operation counts. The event-driven evaluation shows more active neuronal responses alongside more event accumulations and weight accesses, while the increase in projection energy remains a small share of the full decode budget.

We hope this response addresses your concerns, and we welcome any further questions.

Thank you for this detailed rebuttal. My questions have been adequately addressed, and I now have a much clearer understanding of your method. While preserving the low-energy characteristics and temporal modeling capacity of SNNs, ReLIF achieves noticeable improvements in training efficiency and model quality. I think this is an interesting and valuable work.

Reviewer_uPaLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ReLIF, an LIF variant that applies a ReLU to the carried membrane before integration, so the effective state always stays in [0,θ)[0, \theta). The key observation is that under a shared input, trajectories starting from any initial state can merge exactly at zero (through rectification or hard reset) after finitely many steps, which the authors call "finite-time coalescence". They give a clean one-step condition for this (Thm 2) and an E[T]≤1/p0\mathbb{E}[T] \le 1/p_0 bound under iid inputs.

On top of this they build Coalescent Chunking. In the forward pass, each chunk certifies its coalescence point, precomputes the boundary-independent suffix, and only replays the short prefix once the true boundary state is known. In the backward pass, zero-Jacobian anchors split the gradient recurrence into independent pieces. Everything runs in custom CUDA kernels, including a round-safe certificate for FP16/BF16/FP32. Empirically, swapping LIF for ReLIF in SEW-ResNet and Spikformer keeps accuracy on CIFAR-10/100 and CIFAR10-DVS, and SpikeTransformer++ models (57M–191M) get lower perplexity on WikiText-103. On currents from trained models, about 98% of chunks coalesce within 32 steps (mean τ≈5\tau \approx 5). Kernel speedups reach 38× forward over SpikingJelly CuPy LIF, and end-to-end throughput improves by 1.07–1.50× over a matched serial ReLIF kernel.

Critical strengths
  1. The core idea is really neat: one ReLU, no new parameters, and you get a bounded entry domain that allows exact temporal parallelism. I like that it's exact rather than a finite-iteration approximation like fixed-point/Picard-style methods.
  2. The theory is short and easy to follow (Lemma 1, Thm 2). The LIF contrast makes the motivation concrete: an unbounded negative tail gives TLIF=∞T_{\text{LIF}} = \infty, and even bounded LIF has E[T]≈34\mathbb{E}[T] \approx 34 vs ≈3.3\approx 3.3 for ReLIF.
  3. The engineering and validation are careful. The round-safe hull certificate (Prop. 4) is something many papers would skip, the correctness checks are thorough (zero spike mismatch, error ≤2.4×10−7\le 2.4 \times 10^{-7}), and coalescence is measured on currents from trained networks rather than just Gaussian noise, with the suffix ratio tracking speedup closely (r=0.994r = 0.994).
Critical weaknesses

I have the following concerns:

  • The headline speedups (38× forward, 2.9–3.2× end-to-end) mainly come from comparing against SpikingJelly CuPy LIF, which mixes "new neuron" with "new backend". The matched gain from coalescence itself is much more modest: 1.07–1.30× inference and 1.22–1.50× training over serial ReLIF. The kernel benchmarks are also a bit favorable: they use FP32 with 1024 neurons while training uses BF16, and there's no scaling over batch × hidden size (where a serial kernel already fills the GPU) and no memory overhead. I'd lead with the matched numbers and add a fused serial CUDA LIF baseline so readers can see kernel gain vs algorithm gain.
  • ReLIF throws away any negative membrane at every step. That's exactly why it parallelizes, but it's also a memory/capacity trade-off the paper doesn't really examine. The vision tasks use T∈{4,16}T \in \lbrace 4, 16 \rbrace, and WikiText PPL isn't a test of neuron-level memory. I'd like to see SHD/SSC, psMNIST, or LRA-style tasks.
  • Everything uses τm=2\tau_m = 2. The full-domain coalescence condition (x≤−(τm−1)θx \le -(\tau_m - 1)\theta or x≥τmθx \ge \tau_m \theta) gets much harder for larger τm\tau_m, which is the regime people use for longer memory (and PLIF learns τ\tau). A sensitivity study over τm\tau_m and θ\theta (coalescence rate, speedup, accuracy) feels essential. The same goes for soft (subtractive) reset, where reset-coalescence disappears.
  • The LM gain is interesting but unexplained. Why does ReLIF beat LIF by ~5 PPL at 191M? Is LIF suffering from runaway negative membranes or dead neurons in the RMSNorm→LIF→Linear setup? The LIF 191M std is also quite large (±1.59), which makes me wonder about baseline tuning. Firing rates, membrane histograms, or a "clamped LIF" (v≥−cv \ge -c) ablation would help.

If you address my concerns, I will raise my score.

Questions for authors
  1. How do coalescence rate, speedup, and accuracy change for τm∈{2,4,8}\tau_m \in \lbrace 2, 4, 8 \rbrace, for learnable τ\tau, and with soft reset?
  2. Can you add a fused serial CUDA LIF baseline to separate "backend gain" from "coalescence gain"?
  3. Why does ReLIF improve perplexity? Could you show firing rates and membrane statistics for both models, and try a clamped-LIF baseline?
5 comments

Response to Reviewer uPaL (part 1 of 4)

Q1 & W3: Coalescence, acceleration, and task quality under different time constants and resets.

Thank you for your careful review. You ask how coalescence, acceleration, and task quality change beyond the default time constant and reset rule. We conduct paired WikiText-103 experiments with 57M SpikeTransformer++, using seed42, θ=1\theta=1, and the training settings reported in the manuscript, first comparing τm∈{2,4,8}\tau_m \in \{2,4,8\}. Coalescence is measured for ReLIF within 3232 steps on trained-model currents. The τm=2\tau_m=2 results reuse existing records.

τm\tau_mLIF PPLReLIF PPLCoalescence (%)Firing (%) LIF / ReLIF
262.6160.9397.809.25 / 10.84
41585.381427.4384.053.18 / 3.43
81650.081602.5837.732.57 / 3.00

Increasing τm\tau_m from 22 to 44 or 88 reduces the input weight from 0.5\boldsymbol{0.5} to 0.25\boldsymbol{0.25} or 0.125\boldsymbol{0.125}, with the threshold fixed at 11. PPL rises in both models while firing rates fall to 2.57–3.43\boldsymbol{2.57\text{–}3.43} percent. The smaller input weight weakens the effect of each new input on the membrane potential, consistent with the observed reduction in firing.

The reusable suffix fractions at τm=4\tau_m=4 and 88 are 58.95\boldsymbol{58.95} percent and 20.50\boldsymbol{20.50} percent. The inference speedups over serial CUDA ReLIF and serial CUDA LIF are 1.0370×\boldsymbol{1.0370\times} and 1.0351×\boldsymbol{1.0351\times} at τm=4\tau_m=4, and 0.8886×\boldsymbol{0.8886\times} and 0.8870×\boldsymbol{0.8870\times} at τm=8\tau_m=8, respectively. ReLIF still produces certified coalescence at larger membrane time constants, and PPL is lower than the corresponding LIF model at both settings. At τm=4\boldsymbol{\tau_m=4}, parallel ReLIF also runs faster than both serial CUDA baselines.

We also made τm\tau_m learnable at each spiking site in the same model, initializing it to 22.

NeuronPPLLearned τm\tau_m (min / median / max)
LIF52.041.299 / 1.458 / 2.025
ReLIF51.731.294 / 1.409 / 2.068

Of ReLIF's 32 spiking sites, 30 learn τm<2\tau_m<2. PPL decreases from 60.93\boldsymbol{60.93} with fixed τm=2\tau_m=2 to 51.73\boldsymbol{51.73}. Within-chunk coalescence reaches 99.12\boldsymbol{99.12} percent, with a conditional mean time of 3.80\boldsymbol{3.80} steps and a reusable suffix fraction of 87.35\boldsymbol{87.35} percent. With the learned time constants fixed for inference, end-to-end speedups are 1.2426×\boldsymbol{1.2426\times} over serial CUDA ReLIF and 1.2402×\boldsymbol{1.2402\times} over serial CUDA LIF. Learnable membrane time constants further improve PPL while preserving coalescence and inference acceleration.

You point out that soft reset removes the shared zero state provided by hard reset. To test rectification itself, we use subtractive reset in the same model with τm=2\tau_m=2 and θ=1\theta=1. LIF and ReLIF achieve PPL of 60.83\boldsymbol{60.83} and 58.66\boldsymbol{58.66}, respectively. ReLIF coalesces within 3232 steps in 80.49\boldsymbol{80.49} percent of chunks, with a conditional mean time of 10.24\boldsymbol{10.24} steps and a reusable suffix fraction of 54.73\boldsymbol{54.73} percent.

Because residual states can exceed the threshold, we use a conservative entry domain [0,U][0,U] derived from the recorded currents rather than [0,θ)[0,\theta), with maximum UU of 21.29\boldsymbol{21.29}. Coalescence is certified only when rectification maps the entire reachable hull to zero. ReLIF reduces PPL by 2.17\boldsymbol{2.17} points and achieves inference speedups of 1.1207×\boldsymbol{1.1207\times} over serial CUDA ReLIF and 1.1184×\boldsymbol{1.1184\times} over serial CUDA LIF. Under soft reset, ReLIF still coalesces through rectification, achieves lower PPL, and runs faster at inference.

Response to Reviewer uPaL (part 2 of 4)

Q2 & W1: Separating backend optimization from coalescence-based scheduling.

The manuscript's matched serial and parallel CUDA ReLIF comparison holds neuronal dynamics fixed and changes only temporal scheduling, directly measuring Coalescent Chunking's contribution. Following your suggestion, we added fused serial CUDA LIF to test full-model gains when conventional LIF also receives comparable backend optimization.

This LIF baseline is derived from the manuscript's serial CUDA ReLIF, retaining conventional LIF dynamics and using the same fusion strategy and compiler optimizations. Inference and training-step timing use the manuscript's settings. The model uses BF16 for projections and attention, but activations are explicitly converted to FP32 before entering neurons, so the neuronal recurrence matches the FP32 operator benchmarks.

For standard Gaussian currents at T=4096T=4096, batch=1\mathrm{batch}=1, and hidden=1024\mathrm{hidden}=1024, parallel ReLIF achieves F/B/F+B speedups of 6.33×\boldsymbol{6.33\times}, 16.41×\boldsymbol{16.41\times}, and 11.04×\boldsymbol{11.04\times} over this LIF baseline. We use seed42 and report throughput in thousands of tokens per second. Parallel ReLIF values come from the manuscript, with fused serial CUDA LIF as the speedup baseline.

ModelWorkloadFused serial CUDA LIFParallel ReLIFSpeedup
57MInference276.73289.881.05×
Training step44.4653.981.21×
102MInference158.03198.111.25×
Training step26.5136.161.36×
191MInference129.01165.221.28×
Training step23.0934.371.49×

Inference throughput increases by 4.75\boldsymbol{4.75} percent, 25.36\boldsymbol{25.36} percent, and 28.07\boldsymbol{28.07} percent, and training-step throughput by approximately 21–49\boldsymbol{21\text{–}49} percent. We also tested larger workloads, where the serial kernel has more independent work. We swept batch∈{1,2,4,8,16,32}\mathrm{batch} \in \{1,2,4,8,16,32\}, hidden∈{64,128,256,512,1024}\mathrm{hidden} \in \{64,128,256,512,1024\}, and T∈{64,256,1024,4096}T \in \{64,256,1024,4096\}, totaling 120 configurations, and measured memory and bandwidth. The baseline is fused serial CUDA LIF. F+B follows the manuscript by summing separately measured forward and backward times.

PhaseFaster cases (>1>1)Speedup range
Forward89 / 1200.36×–8.87×
Backward116 / 1200.78×–65.89×
Combined F+B108 / 1200.52×–20.29×

F+B is faster in 90.0\boldsymbol{90.0} percent of the grid configurations. At T=4096\boldsymbol{T=4096}, batch=1\boldsymbol{\mathrm{batch}=1}, all six channel dimensions used by the models (384–3072\boldsymbol{384\text{–}3072}) achieve speedups of 4.23×–16.54×\boldsymbol{4.23\times\text{–}16.54\times}. When N=batch×hiddenN=\mathrm{batch}\times\mathrm{hidden} reaches 8192 or 32768, serial forward already reaches 1.23–1.31 TB/s\boldsymbol{1.23\text{–}1.31\text{ }\mathrm{TB/s}}, while the parallel path adds 40–49\boldsymbol{40\text{–}49} percent DRAM traffic, creating a bandwidth bottleneck. These workloads exceed the manuscript models' per-call sizes.

PyTorch peak memory increases by at most 0.98\boldsymbol{0.98} percent. At T=4096T=4096, batch=32\mathrm{batch}=32, hidden=1024\mathrm{hidden}=1024, extra forward and backward memory is 24 MiB\boldsymbol{24\text{ }\mathrm{MiB}} and 40 MiB\boldsymbol{40\text{ }\mathrm{MiB}}, respectively. For the channel dimensions used by the models, these increments are 0.28–2.25 MiB\boldsymbol{0.28\text{–}2.25\text{ }\mathrm{MiB}} and 0.47–3.75 MiB\boldsymbol{0.47\text{–}3.75\text{ }\mathrm{MiB}}. Full-model training-step peak memory is unchanged, at 3.67\boldsymbol{3.67}, 4.38\boldsymbol{4.38}, and 5.81 GiB\boldsymbol{5.81\text{ }\mathrm{GiB}} for 57M, 102M, and 191M. The existing serial and parallel ReLIF comparison isolates the gains from temporal scheduling. The added fused CUDA LIF control shows that parallel ReLIF still improves full-model throughput when conventional LIF uses a comparably optimized backend.

Response to Reviewer uPaL (part 3 of 4)

Q3 & W4: Explaining PPL gains, neuronal activity, and baseline performance.

To understand the relationship between PPL improvement and neuronal activity, we analyzed 18 existing models across three scales and seeds 42, 123, and 2026, using the first eight 40964096-token validation sequences. Firing and silence use all channels, while membrane statistics sample 16 channels per spiking site. The table reports 191M three-seed means and standard deviations. Silence means no spikes throughout a sequence.

Metric (%)LIFReLIF
Fraction of signed membrane states below zero38.82 ± 0.9828.16 ± 0.43
Fraction of carried states below zero38.82 ± 0.980.00 ± 0.00
Firing rate7.47 ± 0.089.17 ± 0.05
Within-sequence silence2.00 ± 0.220.71 ± 0.09

ReLIF increases firing by approximately 1.7\boldsymbol{1.7} percentage points and reduces silence by 1.3\boldsymbol{1.3} percentage points. Higher firing and lower negative membrane occupancy, silence, and PPL occur across all paired seeds. Rectification changes the carried-state distribution and is accompanied by more responsive channels.

We also trained the c=1c=1 clamped-LIF control you suggested, using the 191M model and seed42. Its update is

ht=γmax⁡(vt−1,−c)+ηxt.h_t=\gamma\max(v_{t-1},-c)+\eta x_t.

Here c=0c=0 gives ReLIF. With c>0c>0, the carried state can be negative but has a finite lower bound.

NeuronSeed42 validation perplexity
LIF45.91
Lower-clamped LIF, c=1c=145.35
ReLIF, c=0c=040.54

In the clamped-LIF comparison, ReLIF reduces PPL by a further 4.81\boldsymbol{4.81} points over c=1\boldsymbol{c=1}. Using zero as the lower bound for the carried state improves language-model quality relative to retaining bounded negative states, while also enabling coalescence for parallel execution.

Regarding your concern about the variability and tuning of the 191M LIF baseline, we examined the paired results for each seed. The reported ±1.59\boldsymbol{\pm 1.59} for 191M LIF is the sample standard deviation across three seeds. Paired PPL improvements for seeds 42, 123, and 2026 are 5.37\boldsymbol{5.37}, 3.18\boldsymbol{3.18}, and 6.69\boldsymbol{6.69}, respectively, including the best LIF seed, 123. PPL improves across every paired seed, demonstrating consistent gains under the same training settings.

The activity statistics show reduced negative-state carry and fewer silent channels in ReLIF. In the clamping comparison, a zero lower bound reduces PPL by 4.81\boldsymbol{4.81} points relative to retaining bounded negative states. PPL improvements over conventional LIF hold across every paired seed. We believe that rectification alleviates persistent suppression from negative membrane states and allows more channels to transmit information through SpikeTransformer++ projections, contributing to the PPL improvement.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank reviewers uPaL, DMCN, and xDxD for their constructive feedback and recognition of ReLIF's core contribution, the link between neuronal dynamics and exact temporal parallelism, and the rigor of our validation.

The manuscript incorporates the additional experiments and clarifications below, with changes highlighted in blue.

Core Contributions

  1. Finite-time coalescence through neuronal design. Rectifying the carried membrane bounds the effective state and allows trajectories under shared inputs to merge exactly at zero. The input determines when their boundary-state differences disappear.

  2. Certified forward and backward temporal parallelism. Coalescent Chunking certifies independent suffixes, replays dependent prefixes, and uses zero-Jacobian anchors for parallel backward segments. Round-safe certification supports finite-precision reuse, and backward execution follows the stepwise surrogate-gradient recurrence.

Additional Evidence

  • Temporal modeling for uPaL and DMCN. Section 5.2 and Table 3 report three-seed 784-step psMNIST results for feedforward and recurrent networks. Recurrent ReLIF reaches 73.26\boldsymbol{73.26} percent accuracy versus 72.36\boldsymbol{72.36} percent for LIF; temporal shuffling confirms sequence-order use in both architectures. Settings, shuffling drops, and activity appear in Appendix D.8 and Table 16.

  • Neuronal dynamics for uPaL. Section 5.3 summarizes the time-constant and reset studies in Appendix D.7 and Tables 13–15. Learnable time constants and subtractive reset retain 32-step coalescence rates of 99.12\boldsymbol{99.12} percent and 80.49\boldsymbol{80.49} percent, respectively, with lower PPL than paired LIF and faster inference than both serial CUDA baselines. Section 5.2 and Table 4 show more responsive channels after rectification. The clamped-LIF comparison in Appendix D.6 and Table 12 shows lower PPL with a zero lower bound.

  • Matched CUDA evaluation and scaling for uPaL. Sections 5.5–5.6 and Table 6 add fused serial CUDA LIF alongside the existing matched ReLIF control. Parallel ReLIF achieves 1.05–1.28×\boldsymbol{1.05\text{–}1.28\times} inference throughput and 1.21–1.49×\boldsymbol{1.21\text{–}1.49\times} training-step throughput over fused LIF. Appendices D.4–D.5 and Tables 10–11 report scaling and memory. Combined forward and backward execution is faster in 90.0\boldsymbol{90.0} percent of 120 configurations, and full-model training-step peak memory remains unchanged. Protocols appear in Appendices C.6–C.7.

  • Event-driven inference costs for DMCN. Appendix D.10 and Tables 18–19 report SOPs, logical weight-load requests, and NVML energy for CUDA Core event-driven projections. The decode projection-energy increase is 1.3–4.0\boldsymbol{1.3\text{–}4.0} percent of the same-scale serial CUDA LIF end-to-end decode energy budget. Section 5.2 connects this finding to neuronal activity.

  • Higher-resolution vision for xDxD. Section 5.2 and Appendix D.9, Table 17, report three-seed ImageNet-100 results for SEW-ResNet18. ReLIF reaches 72.41±0.75\boldsymbol{72.41\pm0.75} percent Top-1 versus 72.17±0.34\boldsymbol{72.17\pm0.34} percent for LIF, preserving higher-resolution recognition quality. Training settings appear in Appendix C.1.

  • Architecture-level evaluation for xDxD. Section 5.6 and Appendix E present a trace-driven hybrid timing model combining neuronal event scheduling with compute and memory costs for a window-capable asynchronous architecture. Under matched hardware budgets, it predicts 1.215–1.702×\boldsymbol{1.215\text{–}1.702\times} full-model inference speedups. Tables 20–22 give the resources, latencies, and cost decomposition.

Further Analysis

  • Input-adaptive dependence and memory for DMCN. Section 4.1 distinguishes fixed boundary resets from Coalescent Chunking. Chunk length organizes computation; unresolved state dependencies continue across boundaries until input-driven coalescence.

  • Rectification and neuronal response for uPaL and DMCN. Section 5.2 explains how rectification removes the negative carried-state contribution, allowing new input to drive neuronal responses more directly. Positive-state integration and reset are preserved. Appendix D.10 explains the associated event-driven costs and their model-scale dependence.

  • Input availability and execution models for xDxD. Appendix E.1 explains the input-window requirement. Layer-by-layer GPU execution provides full-window currents; window buffers supply the same inputs for chunk scheduling on the asynchronous architecture.

These additions connect input-driven coalescence to exact temporal parallelism, preserved task quality, and practical execution gains. We welcome further questions and discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.