- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes SubCenSA, a training-free block-sparse attention method that uses count-weighted sub-centroids to estimate block attention mass. The authors provide theoretical analyses of mass estimation and block selection, along with experiments on long-context benchmarks.
Critical strengths
- The paper addresses an important problem in efficient long-context LLM inference.
- The proposed sub-centroid representation is a reasonable extension of conventional centroid-based block selection, with theoretical guarantees under specified conditions.
Critical weaknesses
1. The manuscript is seriously incomplete.
Numerous experimental results remain marked as [TBD], including results essential to the paper's central claims:
- Table 2: All selector-fidelity results are missing.
- Table 3: The COBS-Index and LOCKS-Index baseline results are missing.
- Table 5: All end-to-end decoding latency and speedup results are missing.
- Section 4.3: Index-construction and incremental-update overheads remain unspecified.
Most concerningly, the abstract itself contains [TBD] in its main efficiency claim. These omissions leave several central empirical claims insufficiently substantiated and raise serious concerns about the manuscript's readiness for peer review.
2. The manuscript contains numerous writing and formatting errors.
The paper contains repeated malformed references such as "Eq. equation 8" and missing explicit cross-references to important tables (e.g., Table 2).
The frequency of these elementary errors, together with the unresolved placeholders, suggests insufficient manuscript preparation and quality control.
3. Key empirical claims remain unsubstantiated.
The missing selector-fidelity results prevent direct empirical verification that improved mass estimation leads to better block selection. Similarly, operator-level speedups alone are insufficient to establish end-to-end inference efficiency.
Questions for authors
- Why are numerous essential experimental results still marked as
[TBD]in the submitted manuscript? - Can the authors provide complete selector-fidelity and end-to-end efficiency results, including index-construction and maintenance overheads?
4 comments
Follow-up from the authors
Dear Reviewer mrvE,
We hope this message finds you well, and we would like to sincerely follow up on your careful review, for which we are very grateful.
All previously missing results have been completed and uploaded: the selector-fidelity diagnostics (Table 2), where SubCenSA leads on all four metrics; the end-to-end TPOT speedups over FlashAttention-2, up to 3.08× (Tables 5 and 10); the matched-storage comparison (Table 3); and the fully specified index construction and maintenance costs (Table 1 and Section 4.3). The writing and formatting issues you flagged have also been revised page by page: malformed references such as "Eq. equation 8" were checked and removed throughout, explicit in-text cross-references to the key tables were added, and no [TBD] placeholders or errors of the noted kinds remain. In addition, following a reviewer's suggestion, we added from-scratch pretraining experiments demonstrating that SubCenSA is trainable (see the common message and Appendix E).
May we kindly ask whether these revisions address your concerns, or whether anything remains that we could further clarify or improve? We would be more than willing to do our best to resolve any remaining concerns within the remaining discussion time.
Thank you for your response. It has solved some of my problems. However, I believe that this paper has not met the submission standards before the submission process was completed and there are serious issues. Therefore, I will keep my score.
Thank you
Dear Reviewer mrvE, Thank you for your time and for confirming that the additions resolved part of your concerns. We sincerely apologize that the initial submission did not meet the completeness standards — the responsibility is fully ours, and we respect your assessment. We have taken this lesson seriously: completeness will be a first-order requirement in all our future submissions, and we are grateful for the rigor that caught it.
Response to Reviewer mrvE
Dear Reviewer mrvE,
We sincerely thank you for taking the time to review our paper carefully. The concerns you raised have all been addressed, and your comments prompted these improvements.
Q1: Numerous essential experimental results were still marked as [TBD] in the submitted manuscript.
A: We sincerely apologize for the incomplete experimental results in our initial submission and for the inconvenience this caused to your review. Some results were not uploaded at the submission deadline and were left as [TBD] placeholders; all 39 have now been uploaded. Each spot you identified is now filled in:
The updated manuscript contains no [TBD] placeholders.
Q2a: The missing selector-fidelity results prevent direct empirical verification that improved mass estimation leads to better block selection.
A: The complete results are shown below. SubCenSA leads on every metric: MassRatio_total — the retained fraction of dense attention mass versus the optimal selection — leads by +0.0325 over the strongest baseline; MassRatio_optional — the same ratio over optional blocks — by +0.0914; Recall@m — the fraction of true top-m blocks recovered — by +0.0937; and Mass-NDCG — the mass-weighted ranking quality — by +0.1220. These selector-level leads are consistent with SubCenSA's leading RULER scores, by 2.00-9.08 points across all 8 settings.
Selector Fidelity (Qwen3.5-9B, RULER-128K, K_c=2, H20, 15% KV budget)
Q2b: Operator-level speedups alone are insufficient to establish end-to-end inference efficiency.
A: The complete end-to-end results are shown below. With KV-split enabled, SubCenSA is notably faster than FlashAttention-2 across all batch sizes, with end-to-end TPOT (Time Per Output Token) speedup up to 3.08x.
Matched Decoding Efficiency (Qwen3-30B-A3B, H20, 15% KV budget, dense baseline = FlashAttention-2)
(The complete batch sweep is in Table 10 in the appendix of the paper.)
Q2c: Index-construction and maintenance overheads remain unspecified.
A: These overheads are now fully specified: construction takes 97.952/195.302 ms (64K/128K), incremental maintenance occupies only 0.499%/0.451% of GPU decoding time, and the storage side is tabulated in Table 1: the persistent index costs 516 bytes per block and KV head, 1.57% of the 32,768-byte KV block.
W2: The manuscript contains numerous writing and formatting errors.
A: We apologize for these oversights and especially appreciate your careful proofreading. We have revised the updated manuscript page by page: repeated malformed references such as "Eq. equation 8" have been checked and removed throughout, and explicit in-text cross-references to the important tables (including Table 2, Table 5, and the ablation table) have been added. No [TBD] placeholders or formatting errors of the noted kinds remain.
Beyond the requested results. Following another reviewer's suggestion on trainability, we pretrained five architecturally matched ~470M models from scratch: SubCenSA, HiLS, DASH (DashAttention), InfLLM, and MeanPool. SubCenSA converges with the lowest training loss and the best commonsense average, demonstrating native trainability; please see the common message for details.