- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes the Nested Spiking Transformer (N-Spikformer). It constructs a nested spiking Transformer architecture by reordering attention heads and MLP hidden neurons according to their magnitude. It also proposes a multi-stage training strategy for N-Spikformer. Experimental results show that nested sub-models achieve comparable performance to the retrained sub-models.
Critical strengths
- The nested sub-models achieve comparable performance to the retrained sub-models.
- This paper is well-supported by experiments. The effectiveness of the proposed method is validated across various tasks.
Critical weaknesses
- This paper offers limited innovation. The N-Spikformer proposed in this paper appears to simply adapt the nested network technique from ANNs to SNNs. The technical workflow of N-Spikformer is similar to that of some existing work. First, the importance scoring and parameter reordering processes in Flextron [1] and N-Spikformer are particularly similar. Flextron starts with an existing pre-trained LLM, uses a small number of samples to calculate the magnitudes of the attention head outputs and the corresponding MLP responses, and reorders the corresponding weights in descending order of importance. It then directly selects the first few heads or hidden neurons in each layer to form nested submodels. Second, the gradual training strategy has been explored in previous work, including OFA [2] and EA-ViT [3]. The progressive shrinking training in OFA first optimizes the large network and then gradually supports smaller structures to minimize the interference caused by smaller models on the optimization of the large model. EA-ViT also starts training with the largest model and then gradually lowers the lower bound of the sampling structure to incorporate smaller models into the training process.
- This paper does not provide a comprehensive comparison of end-to-end training cost. Therefore, it is unclear exactly how much training cost is actually saved.
[1] Cai, Ruisi, et al. Flextron: Many-in-one flexible large language model. International Conference on Machine Learning. 2024.
[2] Cai, Han, et al. Once-for-all: Train one network and specialize it for efficient deployment.International Conference on Learning Representations. 2020.
[3] Zhu, Chen, et al. EA-ViT: Efficient adaptation for elastic vision transformer. International Conference on Computer Vision. 2025.
Questions for authors
- Please explain how the proposed method is innovative compared to existing methods.
- Please provide a comprehensive comparison of end-to-end training cost.
2 comments
Sorry for the late reply. The rebuttal has addressed my concerns regarding the end-to-end training cost. Furthermore, the rebuttal has detailed the differences between the proposed N-Spikformer and existing methods. Therefore, I am willing to raise my rating.
However, I have also noted that other reviewers have raised similar concerns regarding the novelty of the proposed N-Spikformer. While it is true that the proposed N-Spikformer differs from each of the existing methods individually, it is more likely a combination of existing methods in terms of importance scoring, parameter reordering, and training strategy. The specific design related to spiking neural networks appears to be limited to changing the ranking strategy from one based on ANN response magnitudes to one based on SNN input currents aggregated across timesteps. Therefore, my concerns regarding the novelty of the proposed method remain.
Thank You and a Brief Follow-up
Dear Reviewer xEGS,
We understand that your time is valuable. Please allow us to thank you once again for your time and effort.
As the discussion deadline approaches, we would be grateful for any further feedback on our responses. Please let us know if any concerns or questions remain.
Best regards,
The Authors