- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ReLIF modifies hard-reset LIF by rectifying the membrane carried across time, so that trajectories entering from different states can meet exactly at zero after a short prefix, either through rectification or through a hard reset. Coalescent Chunking uses that meeting point to precompute the boundary-independent suffix in the forward pass and to split the backward pass into independent gradient segments, while matching stepwise ReLIF. With the membrane time constant fixed at 2, accuracy stays close to LIF on CIFAR-10, CIFAR-100, and CIFAR10-DVS, and validation perplexity improves at three SpikeTransformer++ scales on WikiText-103. In those trained models, about 98% of length-32 chunks meet within the chunk. On a GPU, parallel CUDA ReLIF is substantially faster than SpikingJelly CuPy LIF at the operator level, and full-model inference and training throughput rise by about 2.9–3.2 times and 2.0–2.1 times; against a matched serial CUDA ReLIF, the end-to-end gains are only about 1.1–1.3 times for inference and 1.2–1.5 times for training.
Critical strengths
-
The parallelism is a property of the recurrence, and the execution is built to stay exact. Rectifying the carried state is a small change, but it gives a bounded entry domain and two transitions that map a whole reachable set to zero. Chunking then follows: suffixes that no longer depend on the incoming membrane can be precomputed, and zero-Jacobian anchors cut the backward scan. This is more convincing than a kernel that happens to be fast. The paper states the coalescence criterion (Theorem 2), separates ReLU coalescence from reset coalescence, shows why unbounded-below LIF does not get the same finite-time guarantee, and treats the coalescing step as still boundary-dependent so that forward outputs and surrogate gradients match stepwise ReLIF. The round-safe hull argument is aimed at the right failure mode (a certificate that is false in FP16/BF16/FP32). That exactness claim is what makes the speed numbers interpretable, and it is the main reason the algorithmic contribution is real rather than a rewrite of associative scan.
-
The experiments check the mechanism, not only the leaderboard. The same replacement is run in a conv net, a spike Transformer, and a 4096-token language model at three scales, with three seeds and a paired LIF control. ReLIF does not trade away accuracy; on WikiText-103 the perplexity gap widens with scale. They then measure coalescence on currents from those trained models (about 98% within a chunk, suffix about 85%, survival below 10% by step 12), and on synthetic currents the suffix ratio tracks forward speedup at r=0.994. The serial CUDA ReLIF control is also the right ablation: it keeps the neuron and the tensor interface fixed and changes only the schedule. Together, these results support the claim that finite-time coalescence actually occurs in trained networks and that exploiting it changes operator runtime.
Critical weaknesses
- Inference time should be measured on asynchronous hardware, or at least in a simulator, to show how much it is reduced.
- ImageNet results are needed to show that the accuracy drop remains small.
Questions for authors
See the Weaknesses.
3 comments
You solve my questions. I would raise the score.
Dear Reviewer xDxD,
We are grateful for your helpful feedback and for raising the score after considering our rebuttal. Your suggestions helped us broaden and strengthen the evaluation.
We have uploaded the updated manuscript, with changes highlighted in blue. The revisions addressing your comments are listed below.
- Section 5.6 and Appendix E describe the window-capable asynchronous architecture, hybrid timing model, and predicted full-model inference latencies.
- Section 5.2 and Appendix D.9 present the three-seed ImageNet-100 comparison, with training settings in Appendix C.1.
Thank you again for your careful consideration and for helping us improve the paper.
Response to Reviewer xDxD
Thank you for your careful review. Following your suggestion, we evaluated full-model inference on a proposed asynchronous architecture.
To evaluate asynchronous hardware as you suggested, we must provide the full-window inputs required by Coalescent Chunking. Loihi's neuromorphic cores keep local states and process spikes, update states, and route events sequentially at t, t+1, and t+2. This does not expose complete windows for concurrent chunks. Layer-by-layer GPU execution instead computes each layer over the full window before passing its outputs onward, improving projection and convolution efficiency. Neuron layers receive full-window currents, but conventional LIF remains sequential. ReLIF and Coalescent Chunking exploit within-window temporal parallelism through finite-time coalescence. We add window buffers to the asynchronous design to supply concurrent chunks.
We use a trace-driven hybrid timing model for this Loihi-style design, based on SANA-FE, a neuromorphic architecture simulator. Recorded currents, coalescence points, and replay lengths drive discrete-event scheduling of neuronal execution, including SRAM access, certification, replay, and boundary transfers. Projections, attention, and tensor transfers use analytic timing constrained by compute, bandwidth, and cache resources. Both schedules use the same hardware budget, neuronal dynamics, and complete inference workload. The same configuration is used at all three scales.
We evaluate seed42 57M, 102M, and 191M SpikeTransformer++ models with batch size 1, 4096 tokens, and FP32. Complete outputs are checked for bitwise agreement with an independent stepwise reference before timing the recorded graph, covering attention, feedforward projections, neurons, the vocabulary head, and full logits writeback. The table gives predicted latencies.
Latency is Ttotal=Tcommon+Tneuron, with common operator and transfer costs Tcommon shared by both schedules. The fractions and local speedups below come from this decomposition, so full-model gains follow Amdahl's relation.
With fixed resources, wider networks allocate more microengines to spatial parallelism, leaving fewer temporal resources per trajectory. Attention, projections, and vocabulary output also take a larger runtime share. Consequently, under this hardware configuration, full-model inference speedup decreases as model size increases. The model predicts 1.215–1.702× full-model inference speedups for Coalescent Chunking on the window-capable asynchronous architecture.
Following your suggestion, we add paired ImageNet-100 evaluation using all training images from the selected classes and 5,000 validation images. Time and GPU constraints make full ImageNet-1K training from scratch currently difficult.
SEW-ResNet18 is trained from scratch with a membrane time constant of 2 and paired initialization, data processing, and training settings. The table reports means and sample standard deviations of each run's best Top-1 across seeds 42, 123, and 2026.
ReLIF improves mean Top-1 by 0.24 percentage points. ReLIF maintains LIF-level recognition accuracy on this higher-resolution task.
The architecture model predicts that Coalescent Chunking reduces full-model inference time on a window-capable asynchronous architecture. The three-seed ImageNet-100 results also show that ReLIF preserves recognition accuracy at higher resolution. We welcome any further questions you may have.