acceptodds
Withdrawn from ICLR 2027

Toward General-Purpose Spiking Neurons for Vision and Language Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Widely regarded as the third generation of neural networks, Spiking Neural Networks (SNNs) have attracted considerable attention due to their biological plausibility and energy efficiency. Recent advances in large models call for spiking neurons that combine high performance, adaptability, and training efficiency. In this work, we first present a functional perspective that provides general guidance for designing the next generation of spiking neurons. Guided by these principles, we propose the Adaptive Spiking Neuron (ASN), which incorporates trainable parameters to learn membrane-potential dynamics and enable adaptive firing. ASN adopts an integer-training and spike-inference paradigm that facilitates efficient SNN training. To further improve training stability, we introduce a normalized variant, the Normalized Adaptive Spiking Neuron (NASN). We evaluate the ASN family on 20 datasets spanning five task groups across vision and language, demonstrating its effectiveness and versatility. In particular, NASN exhibits promising scalability in spike-based language modeling, with increasingly pronounced advantages as model and data scales grow. These results highlight the potential of the ASN family to serve as a next-generation general-purpose spiking neuron for large-scale SNNs.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Adaptive Spiking Neurons, General-Purpose Spiking Neurons

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GgK2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce an adaptive spiking neuron that learns a layer-wise shift for its activation window, alongside a normalized variant designed to keep training stable. The main goal of this formulation is to marry efficient integer-based training with adaptive firing dynamics. On the empirical side, the paper evaluates the approach across image classification, event-based recognition, and language modeling benchmarks, and further examines how the neuron behaves as both model size and pre-training data scale up.

Critical strengths

The breadth of the experimental section is a clear plus, as it covers static images, event-based datasets, and multiple language benchmarks. It is also helpful that the authors benchmark their approach against recent spiking neuron models using identical backbone architectures, which makes it much easier to isolate and judge the actual impact of the proposed changes across different tasks.

Critical weaknesses
  1. There is a mathematical contradiction in the spike-driven inference claim, alongside limited technical novelty. If α\alpha is not an integer, the quantizer in Eq. (7) can produce non-integer boundary values. That means taking its output and subtracting ⌈α⌉\lceil\alpha\rceil will not necessarily give a non-negative integer, which contradicts the binary-spike decomposition needed for Eq. (10), and NASN runs into the same problem in Eq. (12). Even setting this aside, the proposed mechanism mostly just introduces a learnable layer-wise offset. How is this meaningfully different from simply learning a bias in the integrated input current? Because this distinction is not established, the methodological contribution looks quite thin and does not justify calling it a next-generation ``general-purpose'' neuron.
  2. The experimental results do not convincingly prove stronger scalability. First, the improvements over NILIF on several ImageNet and CIFAR-100 runs are quite small. Second, the scaling comparison conflates model size with pre-training data size, and Table 6 is missing the NILIF results across the different token budgets. Because of these missing controls, I cannot tell if the proposed neurons genuinely scale better or if the gains just come from extra pre-training.
Questions for authors
  1. My main concern lies with the methodological novelty. How does the firing mechanism in Eq. (7) meaningfully differ from simply adding a learnable layer-wise bias to the integrated input current, or just learning a layer-wise firing threshold? Is there a real functional difference here, or does it boil down to a trivial reparameterization? Given how close these formulations are, what justifies pitching this as a next-generation ``general-purpose'' neuron rather than a straightforward tweak to an existing integer-based neuron?
  2. When α\alpha takes a non-integer value, how can NASN still satisfy the binary-spike decomposition in Eq. (10)? Please write out the exact quantizer formulation used in your actual implementation.
  3. Do you have the matching NILIF baseline numbers for all the pre-training token budgets listed in Table 6? Without those side-by-side numbers, how can we separate the actual architectural benefit of the proposed neuron from the obvious gains of simply training on more tokens?
Reviewer_fWy6OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper re-examines the design principles of general-purpose spiking neurons across four dimensions: training efficiency, adaptive firing, architectural compatibility, and spike-driven inference. Building on this, the authors propose the Adaptive Spiking Neuron (ASN) and its normalized variant, NASN. ASN introduces layer-wise learnable offset parameters into the ILIF framework—which supports integer-based training and spike-based inference—enabling the quantization window to adaptively shift based on the membrane potential distribution of different network layers. NASN further enhances training stability in large-scale networks through normalization. The authors validated the models across 20 datasets—covering tasks such as static vision, event-based vision, natural language understanding, question answering, and commonsense reasoning—and scaled the model up to 1 billion parameters.

Critical strengths
  1. The research addresses a fundamental problem with broad applicability. Rather than designing specific modules for individual network architectures, this paper focuses directly on the spiking neuron—the fundamental computational unit. The four design principles proposed by the authors effectively encapsulate the requirements of modern large-scale SNNs regarding neuron models, providing a clear framework for future research on general-purpose spiking neurons.

  2. The method is concise and easy to integrate. ASN/NASN extends fixed quantization windows to adaptive ones by introducing only a single learnable offset parameter per layer. This design avoids complex additional network modules and incurs minimal parameter and training overhead, allowing for easy replacement of ILIF/NILIF neurons in existing models.

  3. The experimental scope is extensive. The paper evaluates the method across ImageNet, CIFAR, DVS128, GLUE, question-answering, and commonsense reasoning tasks, covering static vision, event-based vision, and language domains. These cross-modal experiments provide empirical support for the authors' claims regarding architectural compatibility and generalizability.

Critical weaknesses
  1. The performance gains in some experiments are relatively limited. On visual tasks such as ImageNet, CIFAR, and DVS128, the improvement of NASN over NILIF is minor.

  2. The spiking inference process requires further clarification. Is this learnable offset value truncated to an integer during both training and inference, or is a floating-point offset value used? The authors are requested to clarify this.

Questions for authors

See Weaknesses

Reviewer_oxuVOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a stable, effective, general, and scalable Spiking Neural Network (SNN) architecture, namely the Adaptive Spiking Neuron (ASN), along with a variant, the Normalized Adaptive Spiking Neuron (NASN). The experiments on ASN and NASN validate their effectiveness and great potential on VLM classification tasks.

Critical strengths

The paper is clearly written and logically coherent, and the analysis of the experiments and results is convincing.

Critical weaknesses
  1. The motivation of the paper is not very clear. What problem in existing methods is ASN intended to solve? Is it about better adapting spiking neurons to large models, or a completely new framework paradigm? The ambiguous motivation cannot support the novelty claimed by the paper.
  2. In Table 1, the authors claim that ASN and NASN enable "Efficient Training", yet the paper lacks sufficient experiments to support this conclusion, which weakens the novelty of the paper.
Questions for authors
  1. In Section 3.1, the authors devote a large amount of space to PRELIMINARIES content. This material would be more appropriately placed in RELATED WORK, particularly since these formulas and contents are not referenced later in the paper.

  2. The leading advantage of NASN on the GLUE task is far less pronounced than on other tasks (e.g., QAT). This is an interesting phenomenon: the strong performance on QAT could be attributed to the structural characteristics of (ASN, NASN), which are better at capturing salient information in language. What, then, causes the less significant improvements on the GLUE and CRT tasks? Is it hallucination in complex contexts, or the neglect of contextual information? Or is the current pre-training scale insufficient to fully exploit the reasoning capability of ASN and NASN in complex contexts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.