- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes MiST, a token-efficient Spiking Transformer that maintains a full feature map as a memory repository while propagating only a small set of query tokens. It introduces bidirectional spike-driven cross-attention (Bi-SDCA) for memory retrieval and updating, along with semantic-guided self-distillation (SGSD) to improve training. Experiments demonstrate competitive accuracy with reduced token processing across image classification and semantic segmentation tasks.
Critical strengths
- The bidirectional memory update provides a reasonable approach to preserving information under aggressive token compression.
- The experiments cover multiple datasets and architectures, demonstrating competitive accuracy and improved throughput.
Critical weaknesses
-
Limited discussion of related architectures: The core design of using sparse queries to retrieve information from a full input representation closely resembles Perceiver [1] and Perceiver IO [2], neither of which is discussed in the paper.
-
Insufficient justification for input-dependent queries: It remains unclear whether the sampled query tokens are essential or whether the full memory repository primarily accounts for the preserved accuracy. The existing random-sampling ablation does not address this question.
-
Insufficient implementation details for segmentation: The downsampling and restoration of memory representations described in Section 5.1 are not sufficiently explained.
-
Confusing compression ratio notation: The paper uses terms such as "0.05× compression ratio" to indicate that 5% of tokens are retained. This corresponds to a 20× compression factor.
[1] Jaegle, Andrew, et al. "Perceiver: General perception with iterative attention." International Conference on Machine Learning. PMLR, 2021.
[2] Jaegle, Andrew, et al. "Perceiver IO: A general architecture for structured inputs and outputs." arXiv preprint arXiv:2107.14795 (2021).
Questions for authors
-
Can the authors provide a more comprehensive ablation study on the role of input-dependent query tokens? Although Appendix F evaluates different sampling strategies, an experiment using input-independent, content-free queries (ex, a fully random or identity matrix) would help clarify whether the query content is essential or whether the full memory repository primarily accounts for the preserved accuracy.
-
Can the authors provide a component-wise inference latency breakdown to clarify which modules contribute most to the speedup from reduced token processing and quantify the overhead introduced by Bi-SDCA?
5 comments
Weaknesses 2 & Questions 1: Role of Input-Dependent Query Tokens
Response: Thank you for this helpful comment. In Appendix F, we replace L1-norm sampling with random sampling, both of which select tokens from the input feature map. This ablation evaluates robustness to the sampling criterion, but does not isolate the role of query content, since both strategies select tokens containing input information. To better address your question, we additionally retrain MiST with a fixed random matrix as the initial queries, independent of the input image. This experiment uses SDT-V3 on ImageNet-1K with and . Apart from the initial query construction, all architecture and training settings remain identical.
As shown below, the fixed-random-query variant achieves 76.20% accuracy, compared with 78.06% for the original MiST. *Its relatively high accuracy supports the important role of the memory repository in retaining input information. The 1.86-percentage-point gap also indicates an additional benefit from input-dependent queries under the same training conditions. We sincerely hope these additional results can help address your concern.
| Initial query construction | Input-dependent | Memory repository | Accuracy (%) |
|---|---|---|---|
| Sampled input features | Yes | Full | 78.06 |
| Fixed random matrix | No | Full | 76.20 |
Weaknesses 3: Memory Downsampling and Restoration in Segmentation
Response: Thank you for this helpful suggestion. The memory repository is only utilized within the backbone to provide semantic guidance for feature extraction; it is not directly passed to the segmentation head and thus requires no spatial downsampling or restoration during decoding. Specifically, in the segmentation task, for a fair comparison, we replace the original SDT-V3 backbone with our MiST while retaining the original spike-driven Semantic FPN as the task head. Therefore, the segmentation head receives only the compact semantic features produced by MiST, without involving any repository-specific downsampling or restoration. We will clarify these details in the main text to improve the reproducibility of the results.
Weaknesses 4: Clarification of Compression Ratio Notation
Response: Thank you for noting this ambiguity. In our paper, refers to the token retention ratio, i.e., retaining tokens, and its compression factor is . To avoid confusion, we will update the notation across the revised manuscript accordingly.
Questions 2: Component-Wise Latency and Bi-SDCA Overhead
Response: Thank you for this helpful suggestion. Following your comment, we provide a component-wise inference latency breakdown for Spikformer-8-768 and MiST (), both using under the same hardware and evaluation settings. As shown below, MLP provides the largest runtime reduction of 0.76 ms/image, followed by SSA with a reduction of 0.60 ms/image. F-SDCA and I-SDCA introduce an additional 0.29 and 0.14 ms/image, respectively, totaling 0.43 ms/image. The sum of component times decreases from 2.62 to 1.70 ms/image, a reduction of 35.11%. These results show that the runtime savings in SSA and MLP outweigh the additional Bi-SDCA overhead in this configuration.
| Component | Spikformer (ms/image) | MiST () (ms/image) |
|---|---|---|
| SPS | 0.91 | 0.92 |
| F-SDCA | — | 0.29 |
| SSA | 0.72 | 0.12 |
| MLP | 0.99 | 0.23 |
| I-SDCA | — | 0.14 |
| Classification head | 0.00 | 0.00 |
| Sum of components | 2.62 | 1.70 |
Dear Reviewer Jiwu,
We sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions. As the author-reviewer discussion phase is drawing to a close, we would like to confirm whether our responses have effectively addressed your concerns. We have provided detailed responses to your concerns, and we hope they have adequately addressed your issues. If you require further clarification or have any additional concerns, please do not hesitate to contact us. We are more than willing to continue our communication with you.
Best regards.
Dear Reviewer Jiwu,
We sincerely thank you for the time you have devoted to reviewing our work. Your technically insightful comments have been helpful in clarifying our contributions and strengthening our evaluation. To better address your concerns, we have devoted substantial effort, including an additional query ablation, a component-wise latency analysis, and further explanations of the architecture and implementation. We would be grateful for your consideration of our responses and the additional results below. If any questions remain, we would be happy to clarify them and discuss them further with you. We wish you all the best.
Response: Thank you for pointing out these relevant architectures. We agree that Perceiver, Perceiver IO, and MiST share the use of cross-attention between a compact query set and a larger feature set. We will cite and discuss these works in the revised manuscript and clarify their relationship to MiST from the following perspectives.
Motivation and design.
Perceiver/Perceiver IO mainly targets general-purpose multimodal processing. Many models rely on modality-specific priors, such as the 2D grid structure used by CNNs, making them difficult to apply uniformly across modalities. Although Transformers are more general, processing pixel-level images or raw audio is computationally expensive. To address this, Perceiver and Perceiver IO use cross-attention to encode an arbitrary-sized input X into a compact latent array Z, where most subsequent computation is performed without redesigning the backbone for each input modality.
MiST targets the token-intensive computation of Spiking Transformers. Instead of compressing the input upfront like Perceiver, MiST applies a Spike Patch Splitting (SPS) module to the input to generate a full spiking feature map M0, from which a sparse query set is sampled. Inspired by biological memory mechanisms, MiST treats M0 as a memory repository and redesigns the Transformer block using Bidirectional Spike-Driven Cross-Attention. Through this redesign, MiST maintains more semantic while computing only a small subset of tokens, significantly reducing computational costs.
Query construction.
Perceiver/Perceiver IO: The initial encoding latent array Z is learnable, shared across samples, and independent of the current input.
MiST: SPS generates a full spiking feature map M0∈RT×H×W×D, from which a compact query set X0∈RT×kH×kW×D is sampled. Thus, MiST introduces no additional learnable latent array, and its queries are input-dependent features that carry visual content and spatial correspondence.
Cross-attention and information flow.
In Perceiver, because the latent array Z is initialized independently of the input, the cross-attention serves as a input-reading module separate from Transformer blocks. The cross-attention is repeatedly inserted between several Transformer blocks, allowing the updated latent array Z′ to revisit the input. In Perceiver, the information flow remains unidirectional (input → latent array), leaving the input representation unupdated. Perceiver IO extends this by adding a decoder cross-attention at the output layer, but the core backbone largely relies on this.
MiST directly embeds cross-attention into each Spiking Transformer block, extending the traditional pipeline into a four-stage process: Forward-SDCA→SSA→FFN→Inverse-SDCA. Within it, Forward-SDCA ensures that sparse query tokens retain the key semantic representations of the input, whereas Inverse-SDCA feeds the enhanced query information back to the input memory repository. Thus, MiST establishes a bidirectional query–memory interaction with a dynamic reconsolidation mechanism, rather than a unidirectional input-to-latent projection as in Perceiver.