- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes WASP, a training-free sparse attention framework that accelerates both prefilling and decoding through asymmetric block attention. It introduces inverse density weighted (IDW) pooling and Bernoulli sampling for dynamic attention selection, and evaluates quality and efficiency across text and multimodal long-context benchmarks.
Critical strengths
-
The paper considers the different computational and memory bottlenecks of prefilling and decoding, providing a unified framework to accelerate both stages.
-
The evaluation covers text and multimodal tasks, including long-context retrieval, summarization, and video understanding, with quality-speed comparisons across several baselines.
Critical weaknesses
-
The paper mainly compares WASP against methods that accelerate only prefilling or decoding, which does not sufficiently justify the claimed advantage of its asymmetric design. Combinations such as MInference + Quest or FlexPrefill + Quest are necessary to establish the actual contribution.
-
The reported end-to-end speedups are derived from separately measured prefilling and decoding performance rather than directly measured full-request latency. Experiments are also limited to a single H200 with batch size 1, making the practical serving advantages unclear.
-
The density defined in Section 3.2 can be negative, causing the inverse weighting to depend heavily on the numerical clipping threshold. The paper does not sufficiently analyze the resulting weight distribution or numerical stability.
-
The linear approximation of the inclusion probability can differ substantially from the original probability. The comparison with exact sampling does not clearly establish whether the observed quality differences remain under matched realized selection budgets.
-
On Qwen3-8B RULER, MInference achieves both higher accuracy and faster inference than WASP at certain operating points. This raises concerns about the generality of the claimed Pareto improvements.
Questions for authors
-
Could the authors report directly measured end-to-end latency, TTFT, TPOT, and throughput under different batch sizes?
-
How does IDW pooling handle negative density values, and what is the impact of numerical clipping on selection quality and stability?
4 comments
W2 and Q1. Directly measured latency, TTFT, TPOT and throughput
Thank you for this request. In the paper, prefilling and decoding are each timed directly (synchronised wall clock for prefilling, CUDA-graph replays for decoding) and the end-to-end number combines them with the dense model's average output length, so that no method gains from generating fewer tokens. We have now also measured whole requests directly, under batched serving on one H200 with Qwen3-8B and six LongMemEval prompts (104k to 108k tokens), at batch sizes 1, 2, 4 and 6. WASP runs at τ = 2.5 with U = 1024, the operating point of the paper's main tables, where decoding keeps about a quarter of the key blocks of these prompts (1544 of 6655 blocks per KV head and decoding step on 24 LongMemEval questions, see our reply to Reviewer 1NeG, Q1), and it runs all attention of prefilling on FlashAttention-2 like the dense model (WASP-FA2, see our reply to Reviewer 1NeG, Q3). Every speed in this answer is at this setting. A full serving system with continuous batching and interleaved prefilling and decoding adds considerable engineering and configuration complexity (finding the best configuration for production serving is a hard problem in itself), so we measure static batches rather than requests that arrive at random times. The requests of a batch arrive together, each is prefilled alone, and the batch then decodes together under one CUDA graph. We time every decoding step on the GPU (CUDA events between steps) during actual generation, so each step reads a cache that has grown by the tokens generated before it. As in the paper's end-to-end numbers, every request produces LongMemEval's mean output length, 34 tokens.
| 100k tokens, 34 output tokens | B = 1 | B = 2 | B = 4 | B = 6 |
|---|---|---|---|---|
| TTFT (mean), dense (s) | 12.4 | 18.6 | 30.9 | 43.5 |
| TTFT (mean), WASP (s) | 6.3 | 9.3 | 15.4 | 21.5 |
| prefilling speedup | 1.98× | 2.00× | 2.00× | 2.02× |
| TPOT, dense (ms) | 13.0 | 17.0 | 23.8 | 33.2 |
| TPOT, WASP (ms) | 10.3 | 12.3 | 15.8 | 19.3 |
| decoding speedup | 1.26× | 1.38× | 1.51× | 1.72× |
| request latency, dense (s) | 12.8 | 25.5 | 50.0 | 75.9 |
| request latency, WASP (s) | 6.6 | 12.8 | 25.2 | 37.7 |
| end-to-end speedup | 1.94× | 1.99× | 1.98× | 2.01× |
| throughput, dense (requests/min) | 4.7 | 4.7 | 4.8 | 4.7 |
| throughput, WASP (requests/min) | 9.1 | 9.4 | 9.5 | 9.6 |
TTFT (mean) averages the time to first token over a batch whose requests arrive together and are prefilled one after another. TPOT is the time of one decoding step of the whole batch, and throughput counts completed requests. Every entry is measured directly on the same prompts. The end-to-end speedup at B = 1, 1.94×, is close to the 2.04× that the revised paper reports on LongMemEval from separately timed prefilling and decoding, and lower than the 2.36× of the previous version, mostly because WASP-FA2 also runs the sampled keys on FlashAttention-2, where the previous version used our own kernel, which is faster than FlashAttention-2 (see our reply to Reviewer 1NeG, Q3). A larger batch mainly raises decoding throughput. Prefilling a 100k-token prompt is compute-bound and already fills the GPU, so prefilling several requests in parallel would not make it faster, and WASP's prefilling speedup stays at about 2.0× at every batch size. Decoding is bound by memory traffic: each decoding step reads the model weights and the KV cache of every sequence from GPU memory and produces one token per sequence, so a larger batch produces more tokens per read of the weights. The KV caches, however, are read once per sequence, so their share of the traffic grows with the batch size. WASP reads only about a quarter of each cache, so its decoding speedup grows from 1.26× at B = 1 to 1.72× at B = 6. The revised paper summarises them in the main text (Table 2) and gives the full table in the appendix.
W3 and Q2. Negative IDW density and clipping
The dot-product density can indeed be negative, and we clip it at 10⁻⁶ before inverting. We measured how often this happens on real activations (Qwen3-8B, one 106k-token LongMemEval prompt):
| layer | query tokens with d ≤ 0 | (block, head) pairs containing one |
|---|---|---|
| 0 | 0.077% | 6.0% |
| 8 | 0.0013% | 0.51% |
| 20 | < 0.0001% | 0.03% |
| 30 | 0 | 0 |
Negative densities are therefore possible but very rare, and essentially absent beyond the first layers, so the clipping threshold affects the pooled query of only a small fraction of blocks. We use the dot-product density because it is simple to compute: it factorises over the block into one product with the block sum, so density and pooling together take about 1.2 ms per layer.
W4. Linear against exact inclusion probability
You are right that the linear approximation min(1, Np) can differ substantially from the exact inclusion probability 1 − (1 − p)^N, and that the comparison at equal τ in the paper is not a comparison at equal budget, since at the same τ the exact form keeps fewer keys. Matching the budget exactly is difficult, because the attention distribution, and with it the number of kept keys, differs from layer to layer and from head to head. We therefore ran the exact form over the full temperature range and compare the two as accuracy–speed Pareto fronts, which are included in the revised paper. At matched speed the linear approximation is ahead by 1 to 2.4 points at the higher temperatures we use (τ ≥ 2.5), and the two are on par at low temperatures.
W5. MInference on Qwen3-8B RULER
We acknowledge that on Qwen3-8B RULER, MInference is more accurate than WASP at matched speed. On the other text benchmarks WASP is ahead: with the per-head patterns from its released search, MInference reaches 40.2 on LongMemEval and 31.8 on En.Sum at 1.85× and 1.44×, against 42.5 and 32.4 for WASP at τ = 2.5 (mean of five seeds) at 2.04× and 1.70×, since MInference leaves decoding dense.
MInference relies on attention patterns searched offline for each head, in particular the vertical-slash pattern. As discussed in the paper (Limitation 1, "Attention structures are architecture artifacts"), these structures are tied to the architecture: slash lines arise from RoPE (Barbero et al., 2025; Cheng et al., 2026; Yang et al., 2026), and video inputs produce different, grid-shaped structures and modality boundaries that the text patterns do not capture, so that MInference's extension to vision-language models needs new pattern types and a new offline search (Li et al., 2025). WASP assumes no attention pattern: it scores key blocks with pooled queries at run time and needs no offline search, so the same method applies directly to the vision-language model, as our Video-MME, LVBench and OmniDocBench results show.
Presentation
We take the presentation score seriously and have revised the method section. The revised paper (1) collects every hyperparameter in one table in a new appendix, as listed in our general response, (2) states explicitly what each component is responsible for, with a pointer to the experiment or ablation that isolates it, and (3) keeps implementation details such as kernel fusion and CUDA graphs in the appendix.
Thank you for the additional experiments, which have substantially addressed my concerns regarding W1 and W2.
I still have some remaining concerns. For W3, an analysis of clipping threshold sensitivity would be helpful. For W4, the matched-speed comparison does not fully address the question of matched selection budgets. For W5, FlexPrefill still achieves both higher accuracy and faster inference than WASP on RULER.
I appreciate the authors' considerable efforts during the rebuttal and will raise my score from 1 to 2 in recognition of these improvements. Nevertheless, I believe the paper would benefit from further refinement before meeting the acceptance standard of ICLR.
W1 Combined methods
Thank you for suggesting this comparison, which is a fair test of the asymmetric design. We have added two combinations that pair a sparse-prefilling baseline with Quest's sparse decoding, MInference + Quest and FlexPrefill + Quest, on Qwen3-8B. We used MInference with the per-head patterns returned by its released search, FlexPrefill at its released defaults (γ = 0.9, τ_JS = 0.1, minimum budget 1024 tokens), and Quest with a budget of 2048 tokens (128 pages of 16 tokens, with the first two layers dense). All runs use the same harness, judge and seeds as the paper.
Combining the two approximations lowers accuracy: each combination is less accurate than the better of its two components on every benchmark, by 0.2 to 6.3 points. In return Quest speeds up decoding by only 1.09× at 100k tokens and 1.14× on En.Sum, which raises the end-to-end speedup of the prefilling method by at most 0.13×. As a result, both combinations fall below the accuracy–speed Pareto front on every benchmark. We have added both combinations to the tables and Pareto fronts of the revised paper. Speedups use the paper's measured components and formula, with WASP re-timed as WASP-FA2 (see our general response).