- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper uses a heuristic method to reduce KV reads and attention computation, but this acceleration comes with a clear performance drop. I did not gain any unexpected or new knowledge from the paper. The validation of the paper is attacked by its own Limitations section and Appendix G, and in my view, this issue is fatal.
Critical strengths
None.
Critical weaknesses
The paper gives a relatively heuristic solution, but it does not justify why this speed-quality trade-off is acceptable. In fact, I think works such as FlexPrefill[1] and ProxyAttn[2] have demonstrated trade-offs that are entirely above BF1. The claims of the paper are deliberately narrowed, which is generally a good thing because it avoids creating excessive expectations about the paper. However, the contribution demonstrated by the claims themselves is not sufficient to meet the contribution standard of ICLR. The theory is very shallow, consisting mostly of simple implications directly from the construction itself. The method does not show obvious novelty, and the experiments also fail to demonstrate sufficient advantages.
The paper is very difficult to read. For example, the abstract and introduction almost completely fail to explain the motivation of this work, what need in the field it is responding to, and how it is supposed to attract researchers' interest. In fact, around 70% of the abstract is spent reporting experimental result numbers, which I find confusing. BTW, the title on OpenReview is inconsistent with the title in the PDF.
[1] Lai, X., Lu, J., Luo, Y., Ma, Y., & Zhou, X. (2025). Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference. ICLR 2025.
[2] Wang, Y., He, H., Bao, S., Wang, H., Zhu, Q., & Che, W. (2026). Proxyattn: Guided sparse attention via representative heads. ICLR 2026.
Questions for authors
I have no questions for the authors, because my concerns are mainly about the contribution and overall presentation, and these may be difficult to address during the rebuttal period.