- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
RBS-Attention is a sparse prefill design that adds a rescue path besides the traditional critique based only on the relevance between the query and the centroid of KV clusters. The rescue path additionally includes a term depending on the radius of the KV cluster, implying that a more diversified cluster may contain a relevant KV even when the centroid seems irrelevant. This term is further weighted by the position of the cluster radius in the overall radius distribution.
Critical strengths
- A simple and effective extension to sparse prefill method.
- The evaluation includes SOTA designs and shows high speedups and slightly better accuracy.
Critical weaknesses
- The key idea is relatively incremental. Also, it may be overly simplified to cover the claimed mean dilution issue, which is a critical limitation. Specifically, by only capturing the radius magnitude in the equation, the rescue path does not consider the direction of the diversification. For a token that is diversified from its cluster mean and goes in the same direction as the query, and another token that goes in the opposite direction, the proposed method would treat them the same, though the second one should not be included.
- For the results, the accuracy results are evaluated at different sparsity levels, but the speedups are presented without specifying the clear accuracy/sparse conditions. The accuracy and speedup results need to be shown in a better and combined way to understand the tradeoffs.
- The ablation study vs. the centroid-only baseline needs to be better explained.
Questions for authors
-
As explained in the weakness part, a critical concern of the proposed method is that it only considers the magnitude of the radius but not the direction of the diversified token. How can your design distinguish the two tokens that both diversify from the centroid but go in opposite directions compared to the query?
-
What is the accuracy level of Fig 2? Do all the designs achieve the same accuracy at the shown speedups? The speedups are significant, e.g., 2x over the best baseline, in this figure. This speedup should mainly come from the saved KVs, i.e., more sparsity at the same accuracy. But the later tables about the accuracy show that the accuracy gain is slight at each same sparsity level. The two types of results therefore do not come together in a clear way. Please better explain this accuracy vs. speedup tradeoff.
-
Please add more explanation and analysis against the centroid-only baseline in the ablation study. Table 7 shows only brief results. A more clear comparison should be, when achieving the same accuracy, how many KVs does each method need to include (sparsity), and the corresponding speedups.
-
In Fig 1(b), RBS improves Q4 and Q5, but it also worsens Q1, Q2, and Q3. What counter-measurements are used to prevent this problem from degrading the performance of RBS?
Thank you for highlighting directionality and the quality-cost comparison. Our new experiments connect the selection mechanism to retained information, answer quality, and measured system cost.
1. Centroid at comparable quality
We evaluated a quality-targeted centroid configuration and RBS on full RULER-128K using Qwen3-30B-A3B-Instruct-2507-FP8. Timing uses 2 A100s, TP=2, batch=1, and chunked prefill.
The budget measures retained causal block pairs. At an accuracy difference of only 0.35 pp, RBS uses 35.2% of the centroid configuration's budget and is 2.32x faster end to end, including selection overhead. The extra selection work pays for itself through a substantially smaller retained-attention workload at comparable answer quality.
2. Directionality and the contribution
RBS separates average relevance from mean-dilution risk:
ℓbase=q⊤cb,ℓrescue=q⊤cb+∥q∥2rbβb.
The centroid term contains query direction; radius summarizes possible token-level deviations. Blocks with identical centroid/radius can tie despite different internal directions. Radius affects candidate selection only: selected tiles use the original Q/K/V and token-level softmax. A negatively aligned token receives no radius bonus in its actual attention logit.
The contribution is a radius-adaptive dual-path allocation policy. Independent centroid selection preserves ordinary relevance, while rescue gives dispersed blocks another opportunity to contribute. Distribution-adaptive βb concentrates the radius correction on blocks with greater mean-dilution risk.
We tested the value of these opportunities on shared dense Q/K traces. Centroid extension and RBS rescue start from the same base/forced mask and add exactly the same number of blocks per query block:
Centroid extension adds 0.6192 pp of dense probability mass; RBS rescue adds 4.6560 pp. At the same additional budget, rescue recovers 4.0368 pp more actual attention mass. This directly validates the allocation mechanism, and the first table establishes its value in system cost.
3. Fig. 2 and quality-speed comparisons
Original Fig. 2 uses method-specific operating configurations, each normalized to Dense on the same H100 setup; quality and density vary across those configurations. Fixed-budget accuracy and comparable-quality latency examine different directions of the quality-budget relationship. A modest accuracy gap at one density can correspond to a substantial budget gap when targeting comparable quality. The centroid experiment above measures this relationship directly.
We also added a joint LongBench-v2 comparison on Qwen3-30B-A3B-Instruct-2507-FP8, one A100, TP=1. TTFT runs from vLLM engine entry to the first generated token. Same-GPU speedups use Dense as the reference; sparsity is the discarded proportion.
RBS achieves 43.88% accuracy with 93.98% sparsity and the highest same-GPU speedup, 3.99x. Reporting these quantities together directly exhibits the quality-efficiency trade-off.
4. Protecting Q1-Q3
The independent base mask provides explicit protection:
Mfinal=Mbase∪Mrescue∪Mforced.
Radius uplift can change the rescue branch's maximum and relative threshold, while the union preserves every block selected by the fixed base branch. Fig. 1(b) describes score-ranking changes; the final algorithm makes its decision through these independently selected masks.
The fixed-base diagnostic measures retained dense-attention mass within each radius quintile:
Coverage improves throughout Q1-Q3, with the largest gains in Q4/Q5. This demonstrates the intended division of labor: preserve ordinary relevance and recover more evidence in the mean-dilution regime.