- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates the efficiency limits of conditional computation based on Query-side Routing in Spiking Speech Transformers. Through theoretical analysis, the authors demonstrate that the computational savings achievable by skipping Attention alone are inherently limited due to shared projections and cross-frame dependencies. Through ablation studies and a revised energy estimation methodology, the authors find that the substantial energy savings previously reported primarily stem from static reductions in network width rather than the dynamic routing mechanism itself. Furthermore, experimental results show that skipping entire Transformer blocks yields significantly greater energy reductions than skipping Attention alone.
Critical strengths
-
The paper provides a theoretical analysis explaining why skipping Attention alone is unlikely to yield substantial computational savings, and further validates the theoretical findings through experiments.
-
The authors conduct comprehensive ablation studies on the SHD and SSC datasets, examining the impact of different modules and routing strategies on energy consumption, thereby strengthening the credibility of the experimental conclusions.
Critical weaknesses
- This paper is primarily analytical in nature and does not propose a fundamentally new method. While this is not necessarily a weakness, as critically examining existing approaches is important, the method proposed based on the analytical findings offers very limited novelty.
- Although the authors conduct multiple experiments on the SHD and SSC datasets, the most critical comparison of different routing positions is performed only on SHD using a single random seed. Therefore, the generalizability and robustness of the experimental findings require further validation.
- Since the authors focus exclusively on Speech Spiking Transformers, the generalizability of both the analysis and the proposed method remains limited. It is unclear whether similar issues arise in other tasks and whether the proposed approach can effectively address them. Extending the investigation to Spiking Transformers more broadly, rather than restricting it to speech-related tasks, would significantly enhance the value and broader applicability of this work.
- The energy consumption results reported in the paper are primarily based on theoretical estimates, without validation on real hardware. Therefore, it remains unclear whether comparable energy savings can be achieved in practical deployment scenarios.
Questions for authors
The authors are advised to address the issues raised in the Weaknesses section.