acceptodds
Under review as a conference paper at ICLR 2027

N-Spikformer: Nested Spiking Transformer

Abstract

Spiking Transformers achieve strong representation capability while retaining the energy-efficient characteristics of spiking neural networks. However, adapting them to different resource constraints typically requires repeated training or fine-tuning, which is computationally expensive. In this paper, we propose a nested spiking Transformer (N-Spikformer) that directly provides multiple sub-models with different accuracy–size trade-offs from a single network to meet different resource constraints. First, we construct a nested spiking Transformer architecture by scoring and reordering attention heads and MLP hidden neurons according to their importance. Within the nested model, smaller sub-models use the higher-ranked heads and hidden neurons. In this way, each sub-model is nested within a larger one, and all sub-models share weights. Second, we develop a nested training strategy to efficiently optimize these shared-weight sub-models. Instead of training all sub-models at each update step, we sample one representative sub-model from four representative width ratios and progressively introduce smaller sub-models during training, reducing training overhead and alleviating optimization interference. Extensive experiments across multiple spiking Transformer architectures, image classification datasets, and dense prediction tasks show that N-Spikformer directly provides customized sub-models under different resource constraints while achieving comparable or better performance than independently trained models. These results demonstrate the practical value of N-Spikformer for deployment scenarios with varying resource constraints.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spiking neural networks, spiking transformer, model compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xEGSOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Nested Spiking Transformer (N-Spikformer). It constructs a nested spiking Transformer architecture by reordering attention heads and MLP hidden neurons according to their magnitude. It also proposes a multi-stage training strategy for N-Spikformer. Experimental results show that nested sub-models achieve comparable performance to the retrained sub-models.

Critical strengths
  • The nested sub-models achieve comparable performance to the retrained sub-models.
  • This paper is well-supported by experiments. The effectiveness of the proposed method is validated across various tasks.
Critical weaknesses
  • This paper offers limited innovation. The N-Spikformer proposed in this paper appears to simply adapt the nested network technique from ANNs to SNNs. The technical workflow of N-Spikformer is similar to that of some existing work. First, the importance scoring and parameter reordering processes in Flextron [1] and N-Spikformer are particularly similar. Flextron starts with an existing pre-trained LLM, uses a small number of samples to calculate the magnitudes of the attention head outputs and the corresponding MLP responses, and reorders the corresponding weights in descending order of importance. It then directly selects the first few heads or hidden neurons in each layer to form nested submodels. Second, the gradual training strategy has been explored in previous work, including OFA [2] and EA-ViT [3]. The progressive shrinking training in OFA first optimizes the large network and then gradually supports smaller structures to minimize the interference caused by smaller models on the optimization of the large model. EA-ViT also starts training with the largest model and then gradually lowers the lower bound of the sampling structure to incorporate smaller models into the training process.
  • This paper does not provide a comprehensive comparison of end-to-end training cost. Therefore, it is unclear exactly how much training cost is actually saved.

[1] Cai, Ruisi, et al. Flextron: Many-in-one flexible large language model. International Conference on Machine Learning. 2024.

[2] Cai, Han, et al. Once-for-all: Train one network and specialize it for efficient deployment.International Conference on Learning Representations. 2020.

[3] Zhu, Chen, et al. EA-ViT: Efficient adaptation for elastic vision transformer. International Conference on Computer Vision. 2025.

Questions for authors
  1. Please explain how the proposed method is innovative compared to existing methods.
  2. Please provide a comprehensive comparison of end-to-end training cost.
2 comments
AuthorsOpenReview ↗

Thank You and a Brief Follow-up

Dear Reviewer xEGS,

We understand that your time is valuable. Please allow us to thank you once again for your time and effort.

As the discussion deadline approaches, we would be grateful for any further feedback on our responses. Please let us know if any concerns or questions remain.

Best regards,

The Authors

Sorry for the late reply. The rebuttal has addressed my concerns regarding the end-to-end training cost. Furthermore, the rebuttal has detailed the differences between the proposed N-Spikformer and existing methods. Therefore, I am willing to raise my rating.

However, I have also noted that other reviewers have raised similar concerns regarding the novelty of the proposed N-Spikformer. While it is true that the proposed N-Spikformer differs from each of the existing methods individually, it is more likely a combination of existing methods in terms of importance scoring, parameter reordering, and training strategy. The specific design related to spiking neural networks appears to be limited to changing the ranking strategy from one based on ANN response magnitudes to one based on SNN input currents aggregated across timesteps. Therefore, my concerns regarding the novelty of the proposed method remain.

Reviewer_FJGbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes N-Spikformer, a shared-weight spiking Transformer intended to accommodate different deployment budgets. The method scores and reorders attention heads and MLP hidden neurons based on input current magnitudes, then constructs subnetworks of different widths by retaining prefixes of the reordered components. During training, one width is sampled at each step, and smaller subnetworks are introduced progressively. Experiments cover three backbone architectures, five classification datasets, semantic segmentation, and object detection.

Critical strengths
  • The problem has practical relevance. The training overhead associated with multiple timesteps in spiking networks makes avoiding separate model training for each resource budget valuable.
  • The experiments have broad coverage. Results across multiple architectures, static and event-based data, and dense prediction tasks support the method's applicability across different settings. Most subnetworks maintain the performance of their independently trained counterparts.
Critical weaknesses
  1. The scoring–reordering–prefix extraction pipeline closely resembles Section 3.1 of Flextron, while randomly sampling a single width at each training step also appears in Section 3.2 of the published version of MatFormer. Although the paper cites these works, it does not sufficiently distinguish its modifications for spiking neural networks from a direct adaptation of existing methods. Applying elastic models to spiking neural networks is valuable. However, the current comparisons with independently trained models mainly demonstrate accuracy retention and do not sufficiently establish the additional value of the proposed designs over directly adapting existing methods. I suggest adding a direct adaptation of at least one of the closest existing methods as a baseline, using the same backbone architecture, initialization, and computational budget, to clarify the benefits of the designs specific to spiking networks.

  2. The method uses a pretrained model, and the three training stages together comprise approximately 1.4 times the number of epochs in the original schedule. However, the source of the pretrained model, the initialization used for subsequent training, and the actual computational cost are not sufficiently specified. Epoch counts do not directly measure computation, and “Train Once” does not quantify the savings. I suggest reporting the total GPU-hours required to obtain the full set of subnetworks, both including and excluding pretraining costs, and comparing this cost with training each model independently and adapting each model separately.

Questions for authors

The breadth of the experiments and the subnetworks' ability to retain accuracy are strengths of the paper. However, the current version does not sufficiently establish its additional contribution or overall efficiency advantages, and the consistency of some experimental results and the evaluation procedures require clarification. I would be willing to reconsider my rating if the authors add direct adaptations of existing elastic methods as baselines, provide a complete cost comparison, and resolve the result consistency issues.

1 comment
AuthorsOpenReview ↗

Thank You and a Brief Follow-up

Dear Reviewer FJGb,

We understand that your time is valuable. Please allow us to thank you once again for your time and effort.

As the discussion deadline approaches, we would be grateful for any further feedback on our responses. Please let us know if any concerns or questions remain.

Best regards,

The Authors

Reviewer_cNipOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes N-Spikformer, a nested spiking Transformer that provides multiple model sizes through weight sharing to meet different deployment requirements. The method ranks hidden neurons by their average absolute input currents, allowing smaller subnetworks to retain higher-ranked components. It also adopts a progressive training strategy that introduces narrower subnetworks in stages. Experiments cover multiple network architectures, datasets, and vision tasks, with the resulting subnetworks generally achieving accuracy comparable to or better than independently trained models.

Critical strengths
  1. Practical support for multiple deployment budgets. N-Spikformer produces subnetworks of different sizes from a single shared-weight network without additional training after extraction. The subnetworks generally match or exceed the accuracy of independently trained counterparts, supporting the practical value of obtaining multiple deployment options through a unified training process.
  2. Validation across architectures and tasks. The evaluation spans three spiking Transformer architectures, five classification datasets, semantic segmentation, and object detection. This breadth provides evidence that the approach is applicable beyond a single backbone or benchmark, strengthening confidence in its generality.
Critical weaknesses
  1. The novelty relative to prior methods is insufficiently established. Flextron already employs importance ranking and weight permutation in non-spiking Transformers, while NESTformer introduces elasticity into spiking Transformers. However, the paper lacks direct comparisons with closely related elastic-network approaches, relying primarily on independently trained models as baselines. These comparisons do not sufficiently establish the additional value of current-based scoring and progressive training over existing approaches.
  2. The training-efficiency advantage lacks quantitative evidence. The method starts from a pretrained model and requires importance scoring followed by three-stage training, yet the paper does not report the total GPU hours relative to independently training the target model sizes or clarify how pretraining costs are accounted for. Since avoiding repeated training is a central motivation, accuracy comparisons alone are insufficient to substantiate the practical training-cost savings.
Questions for authors
  1. What are the specific technical differences between N-Spikformer and related methods such as Flextron and NESTformer? What evidence demonstrates that current-based scoring and progressive training outperform these methods or their adaptations to the same backbone?
  2. On the same hardware, how many GPU hours are required for importance scoring and all three training stages, compared with independently training the four target model sizes? Please clarify whether pretraining costs are included and whether the accuracy comparisons use matched training budgets.
1 comment
AuthorsOpenReview ↗

Thank You and a Brief Follow-up

Dear Reviewer cNip,

We understand that your time is valuable. Please allow us to thank you once again for your time and effort.

As the discussion deadline approaches, we would be grateful for any further feedback on our responses. Please let us know if any concerns or questions remain.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of strengths and rebuttal for the Area Chairs

Dear Area Chairs,

We sincerely thank you, as well as the reviewers, for your time and effort in evaluating our paper. We are grateful to the reviewers for recognizing the practical motivation, clear presentation, and extensive experimental validation of our work (N-Spikformer). The main concerns center on the novelty and training efficiency of N-Spikformer. We have provided detailed, point-by-point responses and summarize them below.

Reviewer cNip

  • W1&Q1: The additional contributions over Flextron and NESTformer are insufficiently established.

    • We clarified our differences from Flextron in training, inference, and ranking, and from NESTformer in progressive training and current-based scoring. We also provided ablations to demonstrate the benefits of these designs.
  • W2&Q2: Training-efficiency claims lack measured GPU-hours and pretraining-cost accounting.

    • We reported 178.24 GPU-hours for the complete training process of N-Spikformer, compared with 554.59 GPU-hours for independently training four sizes, representing a 67.86% reduction. Including the pretraining cost, our total remains 42.79% reduction. Moreover, the advantage in training efficiency increases linearly with the number of required model sizes.

Reviewer FJGb

  • W1&Q1: The additional contributions over Flextron and MatFormer are insufficiently established.

    • We clarified our differences from Flextron in training, inference, and ranking, and from MatFormer in nested architecture, training, scoring and reordering. We also provided ablations to demonstrate the benefits of these designs.
  • W2&Q2: The pretrained initialization and actual computational savings are unclear.

    • We clarified the pretrained initialization. We also reported 178.24 GPU-hours for the complete scoring and training process of N-Spikformer, compared with 554.59 GPU-hours for independently training four sizes, representing a 67.86% reduction. Including the pretraining cost, our total remains 42.79% lower. Moreover, the advantage in training efficiency increases linearly with the number of required model sizes.

Reviewer xEGS

  • W1&Q1: The scoring and progressive training resemble Flextron, OFA, and EA-ViT.

    • We clarified our differences from Flextron in training, inference, and ranking; from OFA in training, scoring, and nested architecture; and from EA-ViT in training and sub-model selection. We also provided ablations to demonstrate the benefits of these designs.
  • W2&Q2: End-to-end training-cost savings are not quantified.

    • We reported 178.24 GPU-hours for the complete scoring and training process of N-Spikformer, compared with 554.59 GPU-hours for independently training four sizes, representing a 67.86% reduction. Including the pretraining cost, our total remains 42.79% lower. Moreover, the advantage in training efficiency increases linearly with the number of required model sizes.

In summary, we sincerely thank the reviewers again for their time and positive feedback on N-Spikformer. Meanwhile, we have also provided timely and detailed responses to all weaknesses and questions. We believe that our responses have adequately addressed their concerns.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.