acceptodds
Under review as a conference paper at ICLR 2027

MiST: Memory-inspired Token-Efficient Spiking Transformer

Abstract

Transformer-based Spiking Neural Networks (SNNs) combine the high performance of Transformers with the energy efficiency of SNNs, offering a promising paradigm for intelligent edge computing. However, they remain bottlenecked by the token-intensive processing inherited from conventional vision transformers, which hinders their deployment on resource-constrained edge devices. Inspired by the key-value memory mechanism of the human brain, we propose MiST, a memory-inspired token-efficient spiking transformer that exploits the inherent efficiency of transformer-based SNNs while maintaining competitive performance. Specifically, MiST decouples the full feature map into a memory repository and a small set of propagated query tokens, enabling forward inference to operate on only a fraction of the original tokens. The core component in MiST is the bidirectional spike-driven cross-attention (Bi-SDCA) module, which comprises two pathways: a forward pathway that retrieves semantic content from the repository into query tokens, and an inverse pathway that propagates enriched token representations back to update the repository. Furthermore, we observe that the memory repository in MiST exhibits high semantic disorder during early training, which slows convergence. To address this, we propose a semantic-guided self-distillation (SGSD) strategy that drives the repository to form compact semantic anchors. Extensive experiments on image classification and semantic segmentation show that MiST achieves accuracy comparable to token-intensive baselines using only 2040% of tokens, establishing a new efficiency frontier for spiking transformer architectures.

Primary area:
applications to neuroscience & cognitive science
Keywords:
spiking quantization, spiking pruning, spiking token pruning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JiwuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MiST, a token-efficient Spiking Transformer that maintains a full feature map as a memory repository while propagating only a small set of query tokens. It introduces bidirectional spike-driven cross-attention (Bi-SDCA) for memory retrieval and updating, along with semantic-guided self-distillation (SGSD) to improve training. Experiments demonstrate competitive accuracy with reduced token processing across image classification and semantic segmentation tasks.

Critical strengths
  1. The bidirectional memory update provides a reasonable approach to preserving information under aggressive token compression.
  2. The experiments cover multiple datasets and architectures, demonstrating competitive accuracy and improved throughput.
Critical weaknesses
  1. Limited discussion of related architectures: The core design of using sparse queries to retrieve information from a full input representation closely resembles Perceiver [1] and Perceiver IO [2], neither of which is discussed in the paper.

  2. Insufficient justification for input-dependent queries: It remains unclear whether the sampled query tokens are essential or whether the full memory repository primarily accounts for the preserved accuracy. The existing random-sampling ablation does not address this question.

  3. Insufficient implementation details for segmentation: The downsampling and restoration of memory representations described in Section 5.1 are not sufficiently explained.

  4. Confusing compression ratio notation: The paper uses terms such as "0.05× compression ratio" to indicate that 5% of tokens are retained. This corresponds to a 20× compression factor.

[1] Jaegle, Andrew, et al. "Perceiver: General perception with iterative attention." International Conference on Machine Learning. PMLR, 2021.

[2] Jaegle, Andrew, et al. "Perceiver IO: A general architecture for structured inputs and outputs." arXiv preprint arXiv:2107.14795 (2021).

Questions for authors
  1. Can the authors provide a more comprehensive ablation study on the role of input-dependent query tokens? Although Appendix F evaluates different sampling strategies, an experiment using input-independent, content-free queries (ex, a fully random or identity matrix) would help clarify whether the query content is essential or whether the full memory repository primarily accounts for the preserved accuracy.

  2. Can the authors provide a component-wise inference latency breakdown to clarify which modules contribute most to the speedup from reduced token processing and quantify the overhead introduced by Bi-SDCA?

5 comments

Dear Reviewer Jiwu,

We sincerely thank you for the time you have devoted to reviewing our work. Your technically insightful comments have been helpful in clarifying our contributions and strengthening our evaluation. To better address your concerns, we have devoted substantial effort, including an additional query ablation, a component-wise latency analysis, and further explanations of the architecture and implementation. We would be grateful for your consideration of our responses and the additional results below. If any questions remain, we would be happy to clarify them and discuss them further with you. We wish you all the best.

Weaknesses 1: Limited discussion of related studies

Response: Thank you for pointing out these relevant architectures. We agree that Perceiver, Perceiver IO, and MiST share the use of cross-attention between a compact query set and a larger feature set. We will cite and discuss these works in the revised manuscript and clarify their relationship to MiST from the following perspectives.

  1. Motivation and design.

    • Perceiver/Perceiver IO mainly targets general-purpose multimodal processing. Many models rely on modality-specific priors, such as the 2D grid structure used by CNNs, making them difficult to apply uniformly across modalities. Although Transformers are more general, processing pixel-level images or raw audio is computationally expensive. To address this, Perceiver and Perceiver IO use cross-attention to encode an arbitrary-sized input XX into a compact latent array ZZ, where most subsequent computation is performed without redesigning the backbone for each input modality.

    • MiST targets the token-intensive computation of Spiking Transformers. Instead of compressing the input upfront like Perceiver, MiST applies a Spike Patch Splitting (SPS) module to the input to generate a full spiking feature map M0M_0, from which a sparse query set is sampled. Inspired by biological memory mechanisms, MiST treats M0M_0 as a memory repository and redesigns the Transformer block using Bidirectional Spike-Driven Cross-Attention. Through this redesign, MiST maintains more semantic while computing only a small subset of tokens, significantly reducing computational costs.

  2. Query construction.

    • Perceiver/Perceiver IO: The initial encoding latent array ZZ is learnable, shared across samples, and independent of the current input.

    • MiST: SPS generates a full spiking feature map M0∈RT×H×W×DM_0\in\mathbb{R}^{T\times H\times W\times D}, from which a compact query set X0∈RT×kH×kW×DX_0\in\mathbb{R}^{T\times kH\times kW\times D} is sampled. Thus, MiST introduces no additional learnable latent array, and its queries are input-dependent features that carry visual content and spatial correspondence.

  3. Cross-attention and information flow.

    • In Perceiver, because the latent array ZZ is initialized independently of the input, the cross-attention serves as a input-reading module separate from Transformer blocks. The cross-attention is repeatedly inserted between several Transformer blocks, allowing the updated latent array Z′Z' to revisit the input. In Perceiver, the information flow remains unidirectional (input →\rightarrow latent array), leaving the input representation unupdated. Perceiver IO extends this by adding a decoder cross-attention at the output layer, but the core backbone largely relies on this.

    • MiST directly embeds cross-attention into each Spiking Transformer block, extending the traditional pipeline into a four-stage process: Forward-SDCA→SSA→FFN→Inverse-SDCA\text{Forward-SDCA}\rightarrow\text{SSA}\rightarrow\text{FFN}\rightarrow\text{Inverse-SDCA}. Within it, Forward-SDCA ensures that sparse query tokens retain the key semantic representations of the input, whereas Inverse-SDCA feeds the enhanced query information back to the input memory repository. Thus, MiST establishes a bidirectional query–memory interaction with a dynamic reconsolidation mechanism, rather than a unidirectional input-to-latent projection as in Perceiver.

Weaknesses 2 & Questions 1: Role of Input-Dependent Query Tokens

Response: Thank you for this helpful comment. In Appendix F, we replace L1-norm sampling with random sampling, both of which select tokens from the input feature map. This ablation evaluates robustness to the sampling criterion, but does not isolate the role of query content, since both strategies select tokens containing input information. To better address your question, we additionally retrain MiST with a fixed random matrix as the initial queries, independent of the input image. This experiment uses SDT-V3 on ImageNet-1K with k=0.05k=0.05 and T=4T=4. Apart from the initial query construction, all architecture and training settings remain identical.

As shown below, the fixed-random-query variant achieves 76.20% accuracy, compared with 78.06% for the original MiST. *Its relatively high accuracy supports the important role of the memory repository in retaining input information. The 1.86-percentage-point gap also indicates an additional benefit from input-dependent queries under the same training conditions. We sincerely hope these additional results can help address your concern.

Initial query constructionInput-dependentMemory repositoryAccuracy (%)
Sampled input featuresYesFull78.06
Fixed random matrixNoFull76.20

Weaknesses 3: Memory Downsampling and Restoration in Segmentation

Response: Thank you for this helpful suggestion. The memory repository is only utilized within the backbone to provide semantic guidance for feature extraction; it is not directly passed to the segmentation head and thus requires no spatial downsampling or restoration during decoding. Specifically, in the segmentation task, for a fair comparison, we replace the original SDT-V3 backbone with our MiST while retaining the original spike-driven Semantic FPN as the task head. Therefore, the segmentation head receives only the compact semantic features produced by MiST, without involving any repository-specific downsampling or restoration. We will clarify these details in the main text to improve the reproducibility of the results.

Weaknesses 4: Clarification of Compression Ratio Notation

Response: Thank you for noting this ambiguity. In our paper, 0.050.05 refers to the token retention ratio, i.e., retaining 0.05N0.05N tokens, and its compression factor is 20×20\times. To avoid confusion, we will update the notation across the revised manuscript accordingly.

Questions 2: Component-Wise Latency and Bi-SDCA Overhead

Response: Thank you for this helpful suggestion. Following your comment, we provide a component-wise inference latency breakdown for Spikformer-8-768 and MiST (k=0.25k=0.25), both using T=4T=4 under the same hardware and evaluation settings. As shown below, MLP provides the largest runtime reduction of 0.76 ms/image, followed by SSA with a reduction of 0.60 ms/image. F-SDCA and I-SDCA introduce an additional 0.29 and 0.14 ms/image, respectively, totaling 0.43 ms/image. The sum of component times decreases from 2.62 to 1.70 ms/image, a reduction of 35.11%. These results show that the runtime savings in SSA and MLP outweigh the additional Bi-SDCA overhead in this configuration.

ComponentSpikformer (ms/image)MiST (k=0.25k=0.25) (ms/image)
SPS0.910.92
F-SDCA—0.29
SSA0.720.12
MLP0.990.23
I-SDCA—0.14
Classification head0.000.00
Sum of components2.621.70

Dear Reviewer Jiwu,

We sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions. As the author-reviewer discussion phase is drawing to a close, we would like to confirm whether our responses have effectively addressed your concerns. We have provided detailed responses to your concerns, and we hope they have adequately addressed your issues. If you require further clarification or have any additional concerns, please do not hesitate to contact us. We are more than willing to continue our communication with you.

Best regards.

Reviewer_zaooOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

MiST is introduced to address the computational overhead of processing large numbers of tokens in Spiking Transformers. Its Bi-SDCA module links compressed tokens to the full feature map through forward semantic retrieval and backward memory updates. Semantic-Guided Self-Distillation (SGSD) provides auxiliary supervision to improve the stability of semantic representations learned by the memory repository. Experimental results demonstrate that MiST achieves strong performance while processing fewer tokens.

Critical strengths
  1. MiST reduces token redundancy in Spiking Transformers by retaining full features in memory and processing fewer tokens. This helps preserve semantic information that token pruning or merging may lose.
  2. The design is well motivated. Bi-SDCA connects query tokens and memory through retrieval and updates, while SGSD improves memory representations using classification supervision and knowledge distillation.
  3. The evaluation covers static image classification, event-based visual classification, and semantic segmentation. The method is also tested on several Spiking Transformer architectures.
Critical weaknesses
  1. Using a few query tokens to attend to the full feature set is similar to existing ANN token-pruning approaches.
  2. Despite using fewer query tokens, MiST retains full key and value representations in memory and adds bidirectional cross-attention to every Transformer block.
  3. The paper reports GPU throughput but lacks a detailed latency breakdown, including module runtimes and additional attention overhead. Practical speedups are therefore not fully established.
Questions for authors
  1. Please clarify how MiST differs from existing token selection methods and identify its novel contributions.
  2. Accessing and updating the full memory at every Transformer block may increase bandwidth requirements and memory access costs. How do these costs affect inference latency?
  3. Please report single-image inference latency for the complete model, along with the time spent in each component.
  4. Please quantify the additional training cost of SGSD.
4 comments

Dear Reviewer zaoo,

Thank you for your constructive review and positive assessment of our work. Your comments have helped us clarify MiST’s novelty and practical efficiency. In response, we provide further comparisons with existing methods, analyses of memory-access costs and SGSD overhead, and a component-wise latency breakdown. We hope these additions address your concerns and welcome any further questions or discussion.

Weaknesses 1 & Questions 1: Novelty of MiST Compared with Existing Token Selection Methods

Response: Thank you for this helpful comment. We agree that using a small query set to attend to a larger feature set has been widely studied in ViTs, our MiST belongs to this broad token-efficient family, but it introduces a distinct token-processing paradigm. In particular, MiST is inspired by biological memory, and reorganizes the dataflow within the spiking Transformer into three complementary processes: memory formation, retrieval, and reconsolidation.

  • Memory formation: retaining global context. MiST retains the full feature map as a memory repository that generates keys for indexing and values for semantic storage. This design keeps their information accessible to the sparse query tokens throughout subsequent processing.

  • Memory retrieval: enriching sparse queries. In F-SDCA, sparsely sampled tokens act as incomplete perceptual cues. They are matched with memory keys through QK⊤QK^\top, and the resulting scores are applied to the memory values to retrieve relevant contextual information. The retrieved representations are further refined by SSA and FFN, which operate only on the compact query set.

  • Memory reconsolidation: updating the repository. Through I-SDCA, query-side information is written back into the memory repository, allowing it to evolve across blocks rather than remain a static feature source. This pathway reuses the attention map from F-SDCA, establishing bidirectional query–memory interaction without recomputing the similarity scores.

In addition, we propose SGSD to encourage the repository to form compact and stable semantic anchors, providing more reliable semantic guidance for sparse queries during training. In summary, MiST’s core contribution is a memory-inspired token-processing paradigm that couples sparse query computation with a full memory repository updated across Transformer blocks.

Weaknesses 2: Additional Computational Overhead of Bi-SDCA in MiST

Response: Yes, MiST retains full-resolution K/V representations and introduces bidirectional cross-attention in each Transformer block. However, the savings from token compression outweigh this additional cost. We illustrate this from both theoretical analysis and experimental results.

  • Theoretical analysis. We compare the per-block FLOPs of a vanilla spiking Transformer (VST) and MiST, including your mentioned additional Bi-SDCA cost. As shown in the table below, NN is the original token count and kk is the query retention ratio. For N=196N=196, Din=D=768D_{\mathrm{in}}=D=768, and Dhid=4DD_{\mathrm{hid}}=4D, summing the three module costs shows that VST requires approximately 1.601.60–3.48×3.48\times as many FLOPs as MiST for k∈[0.12,0.41]k\in[0.12,0.41]. Thus, under these commonly used settings, the savings in SSA and FFN outweigh the additional Bi-SDCA cost.
ModuleOperationVSTMiST
Bi-SDCAQ, K, V00(2k+2)ND2(2k+2)ND^2
Attention002kN2D2kN^2D
Projection00(k+1)ND2(k+1)ND^2
SSAQ, K, V3ND23ND^23kND23kND^2
AttentionN2DN^2Dk2N2Dk^2N^2D
ProjectionND2ND^2kND2kND^2
FFNLinear 12NDinDhid2ND_{\mathrm{in}}D_{\mathrm{hid}}2kNDinDhid2kND_{\mathrm{in}}D_{\mathrm{hid}}
Linear 22NDinDhid2ND_{\mathrm{in}}D_{\mathrm{hid}}2kNDinDhid2kND_{\mathrm{in}}D_{\mathrm{hid}}
  • Experimental results. In addition to the theoretical analysis above, we evaluate MiST on ImageNet-1K to examine whether its computational savings translate into practical inference efficiency. We compare MiST with the corresponding uncompressed baselines on Spikformer-8-768 and SDT-V3, using query retention ratios of k=0.25k=0.25 and k=0.05k=0.05, respectively. Throughput is evaluated on an NVIDIA H100 GPU, with the same batch size for each baseline–MiST pair. As shown below, MiST reduces Transformer-block SOPs by 73.3% and 41.4%, while increasing throughput by 50.0% and 27.1%, respectively. These results demonstrate net efficiency gains despite the additional cost of retaining and updating the memory repository.
Backbone and MiST settingTransformer-block SOPs (G): baseline → MiSTThroughput (images/s): baseline → MiST
Spikformer-8-768, k=0.25k=0.2518.91 → 5.05392 → 588
SDT-V3, k=0.05k=0.052.66 → 1.56595 → 756

Weaknesses 3 & Questions 3: Component-Wise Latency of MiST

Response: Thank you for this helpful suggestion. As per your suggestion, we provide a component-wise latency breakdown for Spikformer-8-768 and MiST (k=0.25k=0.25), both using T=4T=4 and a batch size of 1. As shown below, F-SDCA and I-SDCA add 0.43 ms/image, while SSA and MLP together save 1.36 ms/image. Summing all components, MiST reduces the end-to-end latency from 2.60 ms to 1.70 ms per image, corresponding to a 34.62% latency reduction. This breakdown shows that the runtime savings in our SSA and MLP outweigh the additional cross-attention overhead.

ComponentSpikformer (ms/image)MiST (k=0.25k=0.25) (ms/image)
SPS0.910.92
F-SDCA—0.29
SSA0.720.12
MLP0.990.23
I-SDCA—0.14
Classification head0.000.00
Sum of components2.621.70

Questions 2: Bandwidth Requirements, Memory Access Costs, and Inference Latency of MiST

Response: Thank you for raising this concern. Following your suggestion, we clarify the MiST's repository read/write traffic and its runtime latency in detail.

First, bandwidth requirements and memory-access costs. We quantify the repository’s data-movement overhead through its per-image read/write traffic and corresponding bandwidth requirement. The repository contains TNDTND binary spike activations, corresponding to a binary-encoded data volume of SM=TND/8S _M=TND/8 bytes per full access. For N=196N=196 spatial tokens, D=768D=768 channels, and T=4T=4 timesteps, this gives SM=73.5S_M=73.5 KiB.

Our access model counts two full repository reads for key/value generation and one full write after the memory update in each block. Across L=8L=8 blocks, the read and write traffic are Vread=2LSM≈1.15V_ {\mathrm{read}}=2LS_M\approx1.15 MiB/image and Vwrite=LSM≈0.57V_{\mathrm{write}}=LS_M\approx0.57 MiB/image, respectively. The total repository traffic is therefore Vtotal=3LSM≈1.72V_{\mathrm{total}}=3LS_M\approx1.72 MiB/image.

At a processing rate of rr images/s, the corresponding estimated binary-data bandwidth requirement is Brepo=VtotalrB_{\mathrm{repo}}=V_{\mathrm{total}}r. At r=588r=588 images/s, this gives approximately 0.99 GiB/s, accounting for repository reads and updates across all eight blocks.

Second, impact on inference latency. We provide the component-wise latency breakdown for Spikformer-8-768 and MiST (k=0.25k=0.25), both using T=4T=4. F-SDCA and I-SDCA together take 0.43 ms/image, including their computation and memory access, while SSA and MLP together save 1.36 ms/image. Summing all components, latency decreases from 2.62 to 1.70 ms/image, a reduction of 35.11%. This breakdown shows that, MiST's runtime savings from sparse query processing outweigh the additional memory-interaction cost.

ComponentSpikformer (ms/image)MiST (k=0.25k=0.25) (ms/image)
SPS0.910.92
F-SDCA—0.29
SSA0.720.12
MLP0.990.23
I-SDCA—0.14
Classification head0.000.00
Sum of components2.621.70

Questions 4: Additional Training Cost of SGSD.

Response: Thank you for raising this question. As per your suggestion, we quantify the additional computational cost of SGSD using synaptic operations (SOPs) for Spikingformer-8-768 (k=0.25k=0.25, T=4T=4) on ImageNet-1K. As shown below, SGSD adds 0.000768 GSOPs, corresponding to a 0.0152% increase in the reported SOP count. This cost is minimal because SGSD requires neither an teacher model nor several feature-map alignment, and adds only a projection head and logit-level supervision. Notably, MiST is designed to optimize SNN inference efficiency, but SGSD operates only in training, ensuring no additional inference overhead.

SettingSOPs (G)Additional SOPs (G)
Without SGSD5.0518—
With SGSD5.05260.000768

The authors have addressed all my concerns. The memory-inspired design of the spiking Transformer is interesting, and the experimental results are good. I will therefore raise my score.

Dear Reviewer zaoo,

We sincerely appreciate your recognition of our work. We will thoroughly revise the manuscript in accordance with your suggestions to enhance its quality.

Reviewer_jbcWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MiST, a memory-inspired token-efficient spiking transformer. Specifically, MiST split the input tokens into memory tokens and sampled sparse query tokens, and only the sampled sparse query tokens are processed by the spiking transformer blocks, while the memory tokens are updated using information from the sparse query tokens.

Critical strengths

The proposed mechanism of using SDCA for memory retrieval and Inverse-SDCA for memory update is interesting.

Critical weaknesses

The self-distillation strategy seems can be only used for classfication.

The dataset used for evaluation are rather small compared with current LLM/LVM models.

Only FLOPs are reported without actual speed evaluation.

The SDCA and I-SDCA introduce substantial‌ computational overhead. Thus it it not fair to compare token pruning methods under the same pruning ratio.

No strong evidence is shown of the proposed method over baseline methods, especially in accuracy (for example, the results in table 3).

Questions for authors

Figure 1 is rather difficult to understand. Could you provide some explanation?

5 comments

Dear Reviewer jbcW,

Thank you for your careful review and constructive comments. Your feedback has helped us clarify MiST’s scope and better explain its experimental results. Below, we provide further details on SGSD and the evaluation settings, present measurements of inference speed and performance comparisons, and explain Figure 1. We hope these clarifications address your concerns and welcome any further questions or discussion.

Weaknesses 1: Applicability of SGSD Beyond Classification

Thank you for this professional comment. The self-distillation strategy SGSD is designed to support the pretraining of a reusable visual backbone, rather than provide a distillation objective for every downstream task. Specifically, MiST is designed as an efficient visual backbone for resource-constrained edge devices. Following standard practice, we pretrain MiST on ImageNet and then adapt the pretrained backbone to downstream tasks using task-specific heads. SGSD supports this pretraining stage by providing additional semantic supervision to help the compact backbone learn informative feature representations. The pretrained backbone is then fine-tuned for downstream tasks, such as semantic segmentation, using task-specific heads and losses, without requiring SGSD. We will clarify this training pipeline and the role of SGSD in the revised manuscript.

Weaknesses 2: Scale of Evaluation Datasets

Response: Yes, token compression research in LLMs and LVMs typically focuses on large token-intensive ANN models and much larger datasets. In contrast, MiST studies further token compression in spiking Transformers that already exploit spike activations and event-driven computation. For fair and direct comparisons with related methods, we choose visual benchmarks commonly adopted in SNN token-compression research. For example, STATA (ICML 2024) uses classification benchmarks such as ImageNet, while TP-Spikformer (ICLR 2026) evaluates on ImageNet and ADE20K, among other datasets. Our evaluation includes ImageNet-1K, with over one million training images, and covers three spiking Transformer backbones across static-image classification, event-based classification, and semantic segmentation. Notably, on ImageNet with SDT-V1, MiST uses 12% of tokens as queries and requires 2.42 G Transformer-block SOPs, whereas TP-Spikformer retains 51% of tokens and requires 4.71 G SOPs. At comparable accuracy (74.86% versus 74.79%), MiST reduces SOPs by 48.6%, demonstrating more aggressive query compression and lower computational cost while preserving performance.

Weaknesses 3: Practical Inference Speed Evaluation

Response: Thank you for this helpful comment. Following your suggestion, we supplement our analysis with GPU throughput measurements and a component-wise latency breakdown to evaluate MiST’s practical inference speed.

  • Throughput. We measure throughput on ImageNet-1K using an NVIDIA H100 GPU, with T=4T=4 and the same batch size for each baseline–MiST pair. As shown below, MiST increases throughput by 50.0%, 50.1%, and 27.1% on Spikformer-8-768, SDT-V1, and SDT-V3, respectively. These measurements demonstrate practical inference speedups across three classical spiking Transformer backbones.
BackboneMiST query retention ratioBaseline (imgs/s)MiST (imgs/s)
Spikformer-8-768k=0.25k=0.25392588
SDT-V1-8-768k=0.12k=0.12457686
SDT-V3k=0.05k=0.05595756
  • Component-wise latency. We further provide a latency breakdown for Spikformer-8-768 and MiST (k=0.25k=0.25), both using T=4T=4 under identical hardware and evaluation settings. MLP provides the largest runtime reduction of 0.76 ms/image, followed by SSA with a reduction of 0.60 ms/image. F-SDCA and I-SDCA introduce an additional 0.29 and 0.14 ms/image, respectively, totaling 0.43 ms/image. This reflects a deliberate design trade-off: Bi-SDCA trades some potential efficiency gains to help preserve model accuracy under aggressive token compression. Even with this overhead, the sum of component times decreases from 2.62 to 1.70 ms/image, a reduction of 35.11%, showing that the runtime savings of MiST outweigh the additional Bi-SDCA cost.
ComponentSpikformer (ms/image)MiST (k=0.25k=0.25) (ms/image)
SPS0.910.92
F-SDCA—0.29
SSA0.720.12
MLP0.990.23
I-SDCA—0.14
Classification head0.000.00
Sum of components2.621.70

Weaknesses 4: Fairness of Token-Compression Comparisons

Response: Thank you for raising this concern. We would like to clarify that we do not compare all token-compression methods at the same token retention ratio. In our experimental setup, MiST uses substantially lower retention ratios than the corresponding methods. Thus, we report both accuracy and SOPs for a fair comparison. The reported SOPs cover all Transformer blocks, including the additional computation introduced by F-SDCA and I-SDCA.

More specifically, on ImageNet-1K, MiST uses query retention ratios of 25% and 5% on SDT-V1 and SDT-V3, compared with 51% and 56% for TP-Spikformer, respectively. On SDT-V1, MiST improves accuracy by 1.00 percentage point while reducing SOPs from 4.71 to 3.60 G. On SDT-V3, it improves accuracy by 0.51 percentage points at a comparable computational cost of 1.56 versus 1.55 G SOPs. These ImageNet results show that MiST achieves higher accuracy at lower or comparable computational costs, with the Bi-SDCA overhead included. We will clarify these comparison settings in the revised manuscript.

BackboneMethodToken retention ratioAccuracy (%)Transformer-block SOPs (G)
SDT-V1-8-768TP-Spikformer0.5174.794.71
SDT-V1-8-768MiST0.2575.793.60
SDT-V3TP-Spikformer0.5677.551.55
SDT-V3MiST0.0578.061.56

Weaknesses 5: Performance Advantages over Baselines

Response: Thank you for this helpful comment. MiST is designed for resource-constrained edge devices, where our goal is not to maximize accuracy alone, but to achieve high performance within limited computational and energy budgets. Below, we clarify MiST’s advantages through comparisons with both uncompressed baselines and existing token-compression methods.

  • Comparison with uncompressed baselines. On ImageNet-1K, MiST (k=0.41k=0.41) reduces Spikformer’s Transformer-block SOPs from 18.91 to 7.17 G, a reduction of 62.1%, while accuracy decreases only from 74.81% to 74.66%. For semantic segmentation in Table 3, MiST (T=4T=4, k=0.20k=0.20) achieves 35.6% mIoU at an estimated energy cost of 18.2 mJ/image, compared with 31.9% mIoU at 18.4 mJ/image for uncompressed SDT-V3 (T=2T=2). This represents a 3.7-percentage-point improvement in mIoU at a comparable estimated energy budget. These results illustrate MiST’s benefits for both classification and dense prediction under resource constraints.

  • Comparison with existing token-compression methods. We primarily compare with TP-Spikformer, published at ICLR 2026, as a recent advanced baseline for SNN token compression. On CIFAR-10/100, MiST uses 12% of tokens as queries and achieves 95.33%/78.69% accuracy, whereas TP-Spikformer retains 25%/60% of tokens and achieves 95.16%/78.48%, respectively. On ImageNet-1K, Table 5 shows that MiST improves accuracy by 1.00 percentage point on SDT-V1 while reducing SOPs from 4.71 to 3.60 G. On SDT-V3, MiST improves accuracy by 0.51 percentage points at a comparable cost of 1.56 versus 1.55 G SOPs. These counts include the additional Bi-SDCA computation. These comparisons demonstrate progress in SNN token compression through higher accuracy at lower or comparable computational costs. We will make these results and comparison settings clearer in the revised manuscript.

Questions 1: Clarification of Figure 1

Response: Thank you for this helpful comment. Figure 1 compares MiST with three classical token-intensive spiking Transformers, including Spikformer, SDT-V1, and SDT-V3, in terms of performance, energy consumption, and throughput. Below, we explain the purpose and details of each panel in turn.

  • Figure 1(a): Performance and computation. This panel examines how query token reduction affects predictive performance and computational cost. The four plots show ImageNet classification on SDT-V1, Spikformer, and SDT-V3, and semantic segmentation on ADE20K. Blue bars show accuracy or mIoU on the left axis; purple bars show SOPs on the right axis. The horizontal labels indicate query token retention ratios: 1.01.0 denotes the uncompressed baseline, while 0.250.25 means using 25% of the original tokens as queries. These results show that MiST substantially reduces computation while largely preserving baseline performance.

  • Figure 1(b): Energy consumption. This panel examines how energy consumption changes as fewer query tokens are retained. Each curve represents one backbone, including Spikformer, SDT-V1, and SDT-V3. The horizontal axis shows query token retention ratios, decreasing from left to right, and the vertical axis shows estimated energy in mJ. For example, SDT-V3’s energy decreases from 5.90 to 4.29 mJ at k=0.05k=0.05. These results show lower estimated energy consumption across all three backbones as the query token count decreases.

  • Figure 1(c): Throughput. This panel examines whether query token reduction produces practical inference speedups. The horizontal axis shows query token retention ratios, with smaller values indicating fewer query tokens. The vertical axis shows throughput measured on an NVIDIA H100 GPU. For example, SDT-V3’s throughput increases from 595 to 756 images/s at k=0.05k=0.05. These measurements demonstrate that MiST’s reduced query processing translates into practical inference speedups across all three backbones.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.