acceptodds
Under review as a conference paper at ICLR 2027

WASP: Weighted Asymmetric Sparse Attention to Accelerate both Prefilling and Decoding in Long-context Models

Abstract

Long-context inference stresses attention in two different ways: prefilling is compute-bound, while decoding is bound by memory bandwidth. Sparse attention methods mitigate this problem, but usually deal with only one of the two bottlenecks. Many of them also rely on attention patterns that exists in specific architectures, on probes that overfit the question-at-the-end format of common benchmarks, or on rigid top- budgets that fail to reflect how widely sparsity varies across layers, heads, tasks, and modalities. To this end, we introduce Weighted Asymmetric Sparse (WASP) attention, a training-free sparse attention designed to asymmetrically sparsify the two stages: block attention pools queries during prefilling and keys during decoding, Two primitives make WASP accurate: inverse density weighted (IDW) pooling, a TF–IDF-style summary that preserves the distinctive vectors that otherwise classical mean pooling would dilute, and an approximate Bernoulli sampler that adapts each head's budget to its own score distribution with no sorting. Across six text and multi-modal long-context benchmarks on Qwen3-8B, Llama-3.1-8B, and Qwen3-VL-8B, WASP matches the accuracy of dense attention on five of them while its attention layer runs faster at prefilling and at decoding, achieving an overall speedup that grows further with context length.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse attention, long-context inference, efficient LLM inference, training-free, KV cache, vision-language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZtUSOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes WASP, a training-free sparse attention framework that accelerates both prefilling and decoding through asymmetric block attention. It introduces inverse density weighted (IDW) pooling and Bernoulli sampling for dynamic attention selection, and evaluates quality and efficiency across text and multimodal long-context benchmarks.

Critical strengths
  1. The paper considers the different computational and memory bottlenecks of prefilling and decoding, providing a unified framework to accelerate both stages.

  2. The evaluation covers text and multimodal tasks, including long-context retrieval, summarization, and video understanding, with quality-speed comparisons across several baselines.

Critical weaknesses
  1. The paper mainly compares WASP against methods that accelerate only prefilling or decoding, which does not sufficiently justify the claimed advantage of its asymmetric design. Combinations such as MInference + Quest or FlexPrefill + Quest are necessary to establish the actual contribution.

  2. The reported end-to-end speedups are derived from separately measured prefilling and decoding performance rather than directly measured full-request latency. Experiments are also limited to a single H200 with batch size 1, making the practical serving advantages unclear.

  3. The density defined in Section 3.2 can be negative, causing the inverse weighting to depend heavily on the numerical clipping threshold. The paper does not sufficiently analyze the resulting weight distribution or numerical stability.

  4. The linear approximation of the inclusion probability can differ substantially from the original probability. The comparison with exact sampling does not clearly establish whether the observed quality differences remain under matched realized selection budgets.

  5. On Qwen3-8B RULER, MInference achieves both higher accuracy and faster inference than WASP at certain operating points. This raises concerns about the generality of the claimed Pareto improvements.

Questions for authors
  1. Could the authors report directly measured end-to-end latency, TTFT, TPOT, and throughput under different batch sizes?

  2. How does IDW pooling handle negative density values, and what is the impact of numerical clipping on selection quality and stability?

4 comments

W1 Combined methods

Thank you for suggesting this comparison, which is a fair test of the asymmetric design. We have added two combinations that pair a sparse-prefilling baseline with Quest's sparse decoding, MInference + Quest and FlexPrefill + Quest, on Qwen3-8B. We used MInference with the per-head patterns returned by its released search, FlexPrefill at its released defaults (γ = 0.9, τ_JS = 0.1, minimum budget 1024 tokens), and Quest with a budget of 2048 tokens (128 pages of 16 tokens, with the first two layers dense). All runs use the same harness, judge and seeds as the paper.

methodLongMemEval stdLongMemEval q-agnEn.SumRULER stdRULER q-agnend-to-end speedup (RULER / LME / En.Sum)
MInference + Quest38.238.231.065.961.21.72× / 1.86× / 1.56×
FlexPrefill + Quest39.437.831.865.561.01.97× / 2.19× / 1.62×
MInference (official cfg.)40.241.231.871.467.51.71× / 1.85× / 1.44×
FlexPrefill (γ = 0.9)42.437.432.069.464.21.96× / 2.17× / 1.49×
Quest40.040.431.667.167.11.00× / 1.01× / 1.05×
WASP (τ = 2.5, mean of five seeds)42.543.332.463.864.41.82× / 2.04× / 1.70×

Combining the two approximations lowers accuracy: each combination is less accurate than the better of its two components on every benchmark, by 0.2 to 6.3 points. In return Quest speeds up decoding by only 1.09× at 100k tokens and 1.14× on En.Sum, which raises the end-to-end speedup of the prefilling method by at most 0.13×. As a result, both combinations fall below the accuracy–speed Pareto front on every benchmark. We have added both combinations to the tables and Pareto fronts of the revised paper. Speedups use the paper's measured components and formula, with WASP re-timed as WASP-FA2 (see our general response).

W2 and Q1. Directly measured latency, TTFT, TPOT and throughput

Thank you for this request. In the paper, prefilling and decoding are each timed directly (synchronised wall clock for prefilling, CUDA-graph replays for decoding) and the end-to-end number combines them with the dense model's average output length, so that no method gains from generating fewer tokens. We have now also measured whole requests directly, under batched serving on one H200 with Qwen3-8B and six LongMemEval prompts (104k to 108k tokens), at batch sizes 1, 2, 4 and 6. WASP runs at τ = 2.5 with U = 1024, the operating point of the paper's main tables, where decoding keeps about a quarter of the key blocks of these prompts (1544 of 6655 blocks per KV head and decoding step on 24 LongMemEval questions, see our reply to Reviewer 1NeG, Q1), and it runs all attention of prefilling on FlashAttention-2 like the dense model (WASP-FA2, see our reply to Reviewer 1NeG, Q3). Every speed in this answer is at this setting. A full serving system with continuous batching and interleaved prefilling and decoding adds considerable engineering and configuration complexity (finding the best configuration for production serving is a hard problem in itself), so we measure static batches rather than requests that arrive at random times. The requests of a batch arrive together, each is prefilled alone, and the batch then decodes together under one CUDA graph. We time every decoding step on the GPU (CUDA events between steps) during actual generation, so each step reads a cache that has grown by the tokens generated before it. As in the paper's end-to-end numbers, every request produces LongMemEval's mean output length, 34 tokens.

100k tokens, 34 output tokensB = 1B = 2B = 4B = 6
TTFT (mean), dense (s)12.418.630.943.5
TTFT (mean), WASP (s)6.39.315.421.5
prefilling speedup1.98×2.00×2.00×2.02×
TPOT, dense (ms)13.017.023.833.2
TPOT, WASP (ms)10.312.315.819.3
decoding speedup1.26×1.38×1.51×1.72×
request latency, dense (s)12.825.550.075.9
request latency, WASP (s)6.612.825.237.7
end-to-end speedup1.94×1.99×1.98×2.01×
throughput, dense (requests/min)4.74.74.84.7
throughput, WASP (requests/min)9.19.49.59.6

TTFT (mean) averages the time to first token over a batch whose requests arrive together and are prefilled one after another. TPOT is the time of one decoding step of the whole batch, and throughput counts completed requests. Every entry is measured directly on the same prompts. The end-to-end speedup at B = 1, 1.94×, is close to the 2.04× that the revised paper reports on LongMemEval from separately timed prefilling and decoding, and lower than the 2.36× of the previous version, mostly because WASP-FA2 also runs the sampled keys on FlashAttention-2, where the previous version used our own kernel, which is faster than FlashAttention-2 (see our reply to Reviewer 1NeG, Q3). A larger batch mainly raises decoding throughput. Prefilling a 100k-token prompt is compute-bound and already fills the GPU, so prefilling several requests in parallel would not make it faster, and WASP's prefilling speedup stays at about 2.0× at every batch size. Decoding is bound by memory traffic: each decoding step reads the model weights and the KV cache of every sequence from GPU memory and produces one token per sequence, so a larger batch produces more tokens per read of the weights. The KV caches, however, are read once per sequence, so their share of the traffic grows with the batch size. WASP reads only about a quarter of each cache, so its decoding speedup grows from 1.26× at B = 1 to 1.72× at B = 6. The revised paper summarises them in the main text (Table 2) and gives the full table in the appendix.

W3 and Q2. Negative IDW density and clipping

The dot-product density can indeed be negative, and we clip it at 10⁻⁶ before inverting. We measured how often this happens on real activations (Qwen3-8B, one 106k-token LongMemEval prompt):

layerquery tokens with d ≤ 0(block, head) pairs containing one
00.077%6.0%
80.0013%0.51%
20< 0.0001%0.03%
3000

Negative densities are therefore possible but very rare, and essentially absent beyond the first layers, so the clipping threshold affects the pooled query of only a small fraction of blocks. We use the dot-product density because it is simple to compute: it factorises over the block into one product with the block sum, so density and pooling together take about 1.2 ms per layer.

W4. Linear against exact inclusion probability

You are right that the linear approximation min(1, Np) can differ substantially from the exact inclusion probability 1 − (1 − p)^N, and that the comparison at equal τ in the paper is not a comparison at equal budget, since at the same τ the exact form keeps fewer keys. Matching the budget exactly is difficult, because the attention distribution, and with it the number of kept keys, differs from layer to layer and from head to head. We therefore ran the exact form over the full temperature range and compare the two as accuracy–speed Pareto fronts, which are included in the revised paper. At matched speed the linear approximation is ahead by 1 to 2.4 points at the higher temperatures we use (τ ≥ 2.5), and the two are on par at low temperatures.

W5. MInference on Qwen3-8B RULER

We acknowledge that on Qwen3-8B RULER, MInference is more accurate than WASP at matched speed. On the other text benchmarks WASP is ahead: with the per-head patterns from its released search, MInference reaches 40.2 on LongMemEval and 31.8 on En.Sum at 1.85× and 1.44×, against 42.5 and 32.4 for WASP at τ = 2.5 (mean of five seeds) at 2.04× and 1.70×, since MInference leaves decoding dense.

MInference relies on attention patterns searched offline for each head, in particular the vertical-slash pattern. As discussed in the paper (Limitation 1, "Attention structures are architecture artifacts"), these structures are tied to the architecture: slash lines arise from RoPE (Barbero et al., 2025; Cheng et al., 2026; Yang et al., 2026), and video inputs produce different, grid-shaped structures and modality boundaries that the text patterns do not capture, so that MInference's extension to vision-language models needs new pattern types and a new offline search (Li et al., 2025). WASP assumes no attention pattern: it scores key blocks with pooled queries at run time and needs no offline search, so the same method applies directly to the vision-language model, as our Video-MME, LVBench and OmniDocBench results show.

Presentation

We take the presentation score seriously and have revised the method section. The revised paper (1) collects every hyperparameter in one table in a new appendix, as listed in our general response, (2) states explicitly what each component is responsible for, with a pointer to the experiment or ablation that isolates it, and (3) keeps implementation details such as kernel fusion and CUDA graphs in the appendix.

Thank you for the additional experiments, which have substantially addressed my concerns regarding W1 and W2.

I still have some remaining concerns. For W3, an analysis of clipping threshold sensitivity would be helpful. For W4, the matched-speed comparison does not fully address the question of matched selection budgets. For W5, FlexPrefill still achieves both higher accuracy and faster inference than WASP on RULER.

I appreciate the authors' considerable efforts during the rebuttal and will raise my score from 1 to 2 in recognition of these improvements. Nevertheless, I believe the paper would benefit from further refinement before meeting the acceptance standard of ICLR.

Reviewer_fCEWOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

WASP proposes hybrid sparse attention, using both blocked query attention and blocked key attention.

Critical strengths

The authors are very intuitive and have clear claims: """Prefill is compute-bound. Since FLOPs scale with the number of queries, the query side must be reduced to speed it up. Decode is memory-bandwidth-bound. Since the entire KV cache is read from HBM at every step, the number of keys read must be reduced to speed it up.""". Sounds simple, but effective.

The limitations of prior work they point out are also very intuitive, and they correctly identify critical problems people actually run into in practice. I can relate.

Critical weaknesses

Sparse attention is not that useful anymore, now that GDN hybrid attention has become the default. This is a fundamental limitation of sparse attention research and not directly related to the paper, but its impact on hybrid networks should also be examined.

Questions for authors

Please check the weakness section to resolve my concerns.

1 comment

W1 (GDN networks)

Hybrid networks. Thank you for raising this point, which identifies a limitation of our study. We acknowledge that recent open models are predominantly hybrid. Most of their layers use linear attention, such as Gated DeltaNet, or sliding-window attention, neither of which WASP accelerates, and full attention is retained in only a small fraction of the layers, for example 8 of the 32 layers of Qwen3.5-9B.

At the same time, the context lengths of these models continue to grow: Qwen3.5 supports 262k tokens natively, compared with 32k for Qwen3. Although full attention occupies only a small fraction of the layers, its cost grows quadratically with the context length, so at very long contexts it still accounts for a large share of the computation. By our estimate from the public configuration of Qwen3.5-9B (its parameter count and attention shapes), its 8 full-attention layers account for about 24% of the prefilling FLOPs at 64k tokens, 38% at 128k and 55% at 256k.

We therefore believe that sparse attention for the remaining full-attention layers, which carry the long-range dependencies of these models, remains relevant, and WASP applies to these layers unchanged and without retraining. As we have not yet evaluated WASP on a hybrid model, we have added this discussion to the limitations of the revised paper.

Reviewer_1NeGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes WASP, a training-free sparse attention that applies one block-attention template asymmetrically: prefilling pools queries (B_Q=1024) so one sampled key set serves a whole query block, and decoding pools keys (B_K=16) so each step reads only the sampled blocks. Two components support it — IDW pooling, which weights each vector by the inverse of its summed dot product with its block-mates, and a sorting-free keep rule min(1, max(i·p_j, U/i)) that lets each head's budget follow its own score distribution under a single temperature τ. The paper also introduces a question-agnostic prefilling protocol, where the context is encoded before the question arrives. Across six text and video benchmarks on Qwen3-8B, Llama-3.1-8B and Qwen3-VL-8B on one H200, it reports near-dense accuracy on five at about 1.8–2.4× end-to-end speedup, attention-layer speedups of 3.9× prefilling and 2.7× decoding, and at most 2 points of accuracy movement under the question-agnostic protocol against up to 7 for probe-based and 17 for eviction baselines.

Critical strengths

1. One template, two phases, opposite axes. Prefilling is compute-bound and decoding is bandwidth-bound, so WASP pools queries at prefilling (B_Q=1024) and keys at decoding (B_K=16). Most prior work accelerates one phase and leaves the other unchanged.

2. IDW pooling is well-motivated and portable. Weighting each vector by the inverse of its summed similarity to its block-mates stops redundant vectors from dominating the block representative. Cheap, training-free, and droppable into other block-selection methods.

3. Per-head budget without a sort. min(1, max(i·p_j, U/i)) lets sparsity follow each head's own score distribution under a single temperature, with no top-k kernel on the critical path and a floor that guarantees tail coverage.

4. Broad evaluation, candid reporting. Three backbones including a VLM, six 100k+ benchmarks across text, video and documents, wall-clock end-to-end speedups on one H200 rather than FLOPs. The weak case is reported (1.13× on OmniDocBench) and the omitted baseline is justified.

Critical weaknesses

1. Block granularity is never swept. B_Q=1024 and B_K=16 appear only as defaults, with no sensitivity study anywhere in the main text or appendices. The paper itself attributes its RULER deficit against MInference to "pooled representatives that low-pass positional structure" — a block-granularity diagnosis it never tests against block size.

2. The configuration is under-specified. The sink count and the recent-window length are never given numerically, although both phases always attend them. U=1024 is also fixed without a sweep. This is a reproducibility gap in an otherwise meticulous experimental section.

3. The batched-serving regime is untested. All measurements are batch size 1 on a single H200. The paper acknowledges that batch-1 decoding is capped by streaming 16.4 GB of weights and that WASP does not reduce KV-cache memory — together these leave the multi-request setting, where the bandwidth argument should matter most and where KV capacity binds, entirely unevaluated.

Questions for authors

Clarification

  1. The constant 1.13× on OmniDocBench. In Table 6, WASP's OCR speedup is 1.13× at every temperature from τ=4.0 down to τ=1.0 — identical to three digits - while the edit distance on those same rows moves (0.165, 0.158, 0.145, 0.150, 0.174, 0.178) and the video columns of the same table move from 1.46× to 2.40×. If accuracy changes, the retained fraction changes, so decode attention time cannot be invariant; Quest and FastV show 0.97–0.98× and 1.01–1.02× jitter in that same column. Section 4.2 explains the magnitude but not the invariance. Is this number measured per temperature, or derived from a τ-independent quantity such as weight streaming or non-attention time?

  2. Settings needed to reproduce. Please report: (a) the number of sink tokens and the recent-window size in each phase - both are always attended but never given numerically; (b) how keys generated during decoding are represented, given that representatives are computed once at the end of prefilling - pooled when a block fills, or held in the window? This matters on En.Sum (900 generated tokens) and OmniDocBench (1.1k); (c) the RoPE scaling (e.g. YaRN) used for Qwen3-8B at 100–126k, which affects the dense reference itself; (d) how τ=2.5, "the operating point of the main tables", was fixed, and whether any test-split scores informed that choice.

  3. A dense reference at the current kernel generation. Section I establishes parity by holding every method at the FlashAttention-2 generation, and shows that swapping an FA3-class kernel into WASP's attend moves prefilling by only −1.5% to +3.8%, so internal parity is not my concern. The external question is the dense reference itself: a production deployment on H200 would run an FA3-class dense path, not FA2. How much of the reported end-to-end speedup survives against such a baseline? Please re-time dense and report Tables 5, 7 and 14 against it.

5 comments

W1 (block granularity swep)

We swept the query block size B_Q on RULER at 100k tokens with τ = 2.5. The accuracy does not change much over the sizes we tested.

B_Q=256512102420484096
Llama-3.1-8B, RULER (100 per task)73.674.974.374.774.7
Llama-3.1-8B, prefilling (k tokens/s)16.918.519.219.017.9

A plausible reason is that two effects cancel. A smaller block gives a more specific pooled query and therefore a more precise selection of earlier keys. But the attention inside a block is dense, the larger the block size the closer to dense attention. Since accuracy does not separate the settings, we chose B_Q by speed. This sweep was run before the uniform floor was added, and B_Q was not re-tuned afterwards. You are right that our remark on RULER was a hypothesis we did not test. The sweep above does not support it on the query side, since smaller query blocks do not close the gap to MInference.

We did not sweep the key block size B_K. The only comparison we have comes from an earlier variant of the method (no temperature, no uniform floor), where B_K = 8 and B_K = 16 gave 37.4 and 38.6 on LongMemEval with Qwen3-8B. The key side of the diagnosis is therefore untested. We have removed the attribution from the paper and state the gap without an explanation.

W2 and Q2(a) (Number of sink tokens, recent window size, and U)

We manually set the sink count, recent window size, and U by hand and did not sweep them.

Following the configuration used in StreamingLLM, we set the first 4 tokens as sink tokens. The recent window sizes differ between the prefilling and decoding. During the prefilling, the size of the recent window is 0, every query attends densely to the earlier tokens inside its own block, and the sampled tokens over previous blocks. During the decoding, we set the recent window as 1024 for text benchmarks and 128 for visual benchmarks. For the parameter U, we adopt the setting from FlexPrefill and set it to 1024.

W3 (Batched serving)

We have now measured batched serving on one H200 with Qwen3-8B and six LongMemEval prompts (104k to 108k tokens), at batch sizes 1, 2, 4 and 6. WASP runs at τ = 2.5 with U = 1024, the operating point of the paper's main tables, where decoding keeps about a quarter of the key blocks of these prompts (1544 of 6655 blocks per KV head and decoding step on 24 LongMemEval questions , see Q1), and it runs all attention of prefilling on FlashAttention-2 like the dense model (WASP-FA2, see Q3). Every speed in this answer is at this setting. A full serving system with continuous batching and interleaved prefilling and decoding adds considerable engineering and configuration complexity (finding the best configuration for production serving is a hard problem in itself), so we measure static batches rather than requests that arrive at random times. The requests of a batch arrive together, each is prefilled alone, and the batch then decodes together under one CUDA graph. We time every decoding step on the GPU (CUDA events between steps) during actual generation, so each step reads a cache that has grown by the tokens generated before it. As in the paper's end-to-end numbers, every request produces LongMemEval's mean output length, 34 tokens.

100k tokens, 34 output tokensB = 1B = 2B = 4B = 6
TTFT (mean), dense (s)12.418.630.943.5
TTFT (mean), WASP (s)6.39.315.421.5
prefilling speedup1.98×2.00×2.00×2.02×
TPOT, dense (ms)13.017.023.833.2
TPOT, WASP (ms)10.312.315.819.3
decoding speedup1.26×1.38×1.51×1.72×
request latency, dense (s)12.825.550.075.9
request latency, WASP (s)6.612.825.237.7
end-to-end speedup1.94×1.99×1.98×2.01×
throughput, dense (requests/min)4.74.74.84.7
throughput, WASP (requests/min)9.19.49.59.6

TTFT (mean) averages the time to first token over a batch whose requests arrive together and are prefilled one after another. TPOT is the time of one decoding step of the whole batch, and throughput counts completed requests. Every entry is measured directly on the same prompts. The end-to-end speedup at B = 1, 1.94×, is close to the 2.04× that the revised paper reports on LongMemEval from separately timed prefilling and decoding, and lower than the 2.36× of the previous version, mostly because WASP-FA2 also runs the sampled keys on FlashAttention-2, where the previous version used our own kernel, which is faster than FlashAttention-2 (see Q3). A larger batch mainly raises decoding throughput. Prefilling a 100k-token prompt is compute-bound and already fills the GPU, so prefilling several requests in parallel would not make it faster, and WASP's prefilling speedup stays at about 2.0× at every batch size. Decoding is bound by memory traffic: each decoding step reads the model weights and the KV cache of every sequence from GPU memory and produces one token per sequence, so a larger batch produces more tokens per read of the weights. The KV caches, however, are read once per sequence, so their share of the traffic grows with the batch size. WASP reads only about a quarter of each cache, so its decoding speedup grows from 1.26× at B = 1 to 1.72× at B = 6. The revised paper summarises them in the main text (Table 2) and gives the full table in the appendix.

Q1(The constant 1.13x on OmniDocBench)

The speedup in the previous version was is measured separately at every temperature, and the measured values ranged from 1.129× to 1.135× end to end (decoding 1.155× to 1.160×), all of which round to 1.13×. Re-measured with WASP-FA2 (see Q3) on the same nine pages at every temperature, the range is similarly narrow: from 1.132× to 1.140× end to end and from 1.160× to 1.168× in decoding (table below). The reason is that on OmniDocBench the number of key blocks kept in decoding barely changes with the temperature. We measured how many 16-token key blocks BKA keeps per KV head and decoding step, on 16 random OmniDocBench pages and 24 random LongMemEval questions:

τOmniDocBench, end-to-end speedupOmniDocBench, decoding speedupOmniDocBench, kept blocks (share of candidates)LongMemEval, kept blocks (share of candidates)
4.01.136×1.161×120 (41%)3186 (48%)
3.01.139×1.164×101 (36%)2099 (32%)
2.51.140×1.165×92 (33%)1544 (23%)
2.01.132×1.162×84 (31%)990 (15%)
1.51.140×1.167×77 (29%)537 (8.1%)
1.01.137×1.163×71 (27%)229 (3.4%)
0.51.134×1.160×67 (25%)91 (1.4%)
0.251.140×1.168×66 (25%)71 (1.1%)
0.11.135×1.160×65 (25%)66 (1.0%)

On OmniDocBench the number of blocks kept at τ = 4.0 is only about twice that at τ = 0.1, whereas on LongMemEval it is 48 times higher.

Q2(b) (Keys generated during decoding)

The representatives of the prompt’s key blocks are computed once, when prefilling ends. During the decoding, once a block of 16 tokens (the oldest token) leaves the recent window, it is pooled into a new representative and becomes selectable like a prompt block, so the recent window size during the decoding lies between the configured recent window size and the configured size plus 15.

Q2(c) (RoPE scaling)

Qwen3-8B runs with YaRN at factor 4, extending its original 32,768-token context to 131,072, the setting recommended by the official Qwen3 model card for long inputs. The dense reference and every baseline use the same configuration. Only in the length-scaling curves beyond 131k tokens is the factor raised to the context length divided by 32,768. Llama-3.1-8B runs with its released RoPE configuration (128k context) and no additional scaling, in every accuracy and speed measurement.

Q2(d) (How tau was fixed)

We did not tune τ on a held-out split. We swept τ on the evaluation benchmarks during development. The paper therefore reports the whole τ sweep as a Pareto frontier for every model and benchmark, and every baseline's budget knob is swept in the same way on the same benchmarks, so no comparison rests on a single operating point. τ = 2.5 is the point we chose from these sweeps for the main tables. It was fixed once and used unchanged for Qwen3-8B on all three text benchmarks and for Qwen3-VL-8B on all three vision benchmarks. Llama-3.1-8B uses τ = 2.0, chosen from its own sweep. The seed study shows a run-to-run standard deviation of 0.1 to 0.8 points at these settings, which is the scale of optimism that choosing τ on the same data can introduce. The revised paper states this explicitly.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their detailed and constructive reviews. This comment summarises a correction to our speed numbers, the new measurements in this response and the changes in the revised paper, which we have uploaded. The point-by-point answers are in the replies to each review. At the end we add one result that no review asked for, the run-to-run variation of WASP over sampler seeds.

A correction to WASP's speed numbers. While answering Reviewer 1NeG (Q3), we found that WASP's prefilling in the previous version mixed two kernel generations: attention within each query block ran on FlashAttention-2, but attention over the sampled keys ran on a Triton kernel of our own whose speed lies between FlashAttention-2 and FlashAttention-3. Since the dense reference and all baselines run at the FlashAttention-2 generation, we now run all of WASP's prefilling attention on FlashAttention-2 (WASP-FA2). The revised paper re-times all of WASP's speeds with WASP-FA2. The speedups are lower than in the previous version: on LongMemEval at τ = 2.5 the end-to-end speedup is 2.04×, against 2.36× reported previously, and in batched serving, measured directly on whole requests, it stays at about 2.0× from batch size 1 to 6. WASP nevertheless remains the only single method that accelerates both prefilling and decoding. Under question-agnostic prefilling, where the context is encoded before the question arrives, as with prefix caching, WASP-FA2 is both faster and more accurate than every baseline of our main text table on LongMemEval (Qwen3-8B), and on RULER its point at τ = 4.0 comes within 0.1 points of the most accurate baseline at a higher speed.

New measurements in this response.

  • 1NeG W1: a query block size sweep on RULER for Llama-3.1-8B, with the prefilling throughput of each block size.

  • 1NeG W3, ZtUS W2 and Q1: · request latency, TTFT, TPOT and throughput at batch sizes 1, 2, 4 and 6 at 100k tokens, measured directly on whole requests, with both WASP-FA2 and WASP-FA3 against dense-FA2 and dense-FA3.

  • 1NeG Q1: the OmniDocBench speedup at nine temperatures with WASP-FA2, and the number of key blocks that decoding keeps per step on OmniDocBench and LongMemEval.

  • 1NeG Q3: WASP-FA2 and WASP-FA3, a FlashAttention-3 build, each against the dense model of the same kernel generation.

  • ZtUS W1: the combined baselines MInference + Quest and FlexPrefill + Quest on LongMemEval, En.Sum and RULER.

  • ZtUS W3 and Q2: how often the IDW density is non-positive on real activations.

  • ZtUS W4: the exact inclusion probability as a full accuracy–speed front on LongMemEval, against the linear approximation.

Changes in the revised paper.

  1. A new appendix on the hyperparameters, with one table of every fixed setting: sink tokens, the recent window of each phase, U, B_Q, B_K, τ and the RoPE configuration, and how keys generated during decoding are handled (1NeG W2 and Q2).

  2. A revised method section that states what each component is responsible for, with a pointer to the experiment that isolates it (ZtUS presentation).

  3. An explicit statement that τ was chosen on the evaluation benchmarks (1NeG Q2(d)).

  4. Section I corrected, and every WASP speed in the tables and Pareto fronts re-timed with WASP-FA2 (1NeG Q3).

  5. The untested attribution of the RULER gap to pooling removed (1NeG W1).

  6. The combined baselines in the tables and Pareto fronts (ZtUS W1).

  7. A limitations paragraph on hybrid models (fCEW).

  8. Mean ± standard deviation over seeds in the tables, and error bars on the Pareto fronts (see below).

  9. The exact and linear inclusion probabilities as two accuracy–speed fronts (ZtUS W4).

  10. The batched-serving measurements in the main text, and a new appendix section with the block-size sweep, the kept-block counts, the full batched-serving and FlashAttention-3 comparisons and the IDW density statistics (1NeG W1, W3, Q1, Q3, ZtUS W2, W3).

Run-to-run variation (not raised in the reviews). The tables of the previous version report one sampler seed per cell, as there was not enough time to run more seeds before the deadline. Because WASP samples the keys it attends, its results could vary from run to run, so we have since re-run every headline cell with five sampler seeds as evidence of the method's stability. The standard deviation is at most 0.8 points (0.006 edit distance on OmniDocBench). We have added the mean and standard deviation to the tables, and error bars to the Pareto fronts, of the revised paper. With this variation shown, one can judge more clearly whether WASP's lead on a Pareto front exceeds run-to-run noise, rather than reading it from a single run.

Area_Chair_SCDDOpenReview ↗

Response to the rebuttal.

Dear reviewers,

The authors have submitted rebuttal responses. Please read through their replies and update your review as you see fit. Thank you for your work.

AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.