- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper enhances the design of Spiking Transformers to address their intrinsic temporal modeling issue. The authors introduce multi-threshold spiking neurons (MTSN) in the embedding module to generate different responses to the same input. In the attention module, they further construct a modulation mechanism for the value branch, termed EDVM, by leveraging the variance across time steps.
Results on multiple benchmarks show that the proposed method can achieve certain improvements over the baseline. However, the choice and reproduction of the baselines raise concerns about the reliability of the experimental results. The degree of coupling between MTSN and EDVM is also unclear. In addition, the hardware friendliness of the multi-level integer response mechanism in MTSN has not been sufficiently evaluated, and the analysis of some ablation experiments is overly coarse-grained. Overall, I believe that the paper still has considerable room for improvement.
Critical strengths
-
The paper improves model performance from the perspective of temporal characteristics, and the proposed enhancement is model-agnostic.
-
The proposed method does not introduce additional parameters and therefore avoids excessive computational overhead or complex operations.
-
EDVM considers different modulation strategies for event-based and static temporal characteristics, making it adaptable to cases with either pronounced or less significant temporal dynamics.
Critical weaknesses
-
The baselines in the paper (except for SDT) are all taken from STAtten. Why are the results not cited from their corresponding original papers? For some models, such as Spikformer and SpikingResFormer, even with the improvement brought by SNTM, their performance still fails to reach the results reported in the original papers.
-
The coupling between EDVM and MTSN has not been analyzed. As shown in Tab. 2, the improvement brought by EDVM is relatively significant, whereas MTSN only brings a 0.17 improvement on SDT.
-
The ablation experiments are limited to component-level analysis. The sensitivity to different thresholds in MTSN has not been investigated. Similarly, it remains unclear why EDVM adopts first-order adjacent temporal differences instead of higher-order temporal differences.
-
The modulation results on the value branch lack visualization and more specific analysis.
Questions for authors
see Weakness.