acceptodds
Under review as a conference paper at ICLR 2027

The Model Knows Where to Look: Distilling Attention Selectors for Long-Context Inference

Abstract

Language models now process whole documents, conversations and codebases, but generating each token requires reading a growing key-value cache. Attention often concentrates on a small fraction of that context, allowing sparse attention to reduce these reads. Selecting the relevant keys, however, introduces its own cost: the selector must inspect the cache or score a compressed representation of it. We present Highlight, a lightweight selector that skims the full context and highlights the relevant blocks for the model to attend to. It scores the context with a few narrow attention heads over its own small key cache, and selects 16-token blocks. It learns from the frozen model's own attention, through KL divergence on the tokens it selects. On five full-attention and hybrid models from 4B to 32B, accuracy stays close to dense attention on long-context benchmarks up to 128K tokens. On Qwen3-8B, every speedup we report comes from a configuration that stays within two points of dense or above it on each benchmark. At KV-cache saturation, selecting a quarter of the context yields 1.4× to 1.8× dense throughput. Adaptive top-p selection sets a budget for each query and layer, retaining 12 to 19 percent of the context and reaching up to 1.9× throughput. Quantizing the selector's key cache to four bits raises throughput up to 2.5× dense. Highlight also selects blocks up to 8 times faster than Twilight, an adaptive top-p selector, while both stay within two points of dense on held-out examples.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse attention, long-context inference, KV cache, attention distillation, efficient inference, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TtUnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Highlight, a lightweight attention selector that enables sparse decoding without updating the pretrained language model. The selector maintains a compact key cache and learns from the frozen model’s attention through KL divergence over selected tokens. At inference, it selects 16-token blocks from the original KV cache using either a fixed top-K budget or an adaptive dual top-p policy. Experiments on several full-attention and hybrid models show accuracy close to dense attention in many evaluated settings, including context-length generalization from 16K training to 128K evaluation on Qwen3.5.

Critical strengths
  1. Highlight integrates sparse selection into vLLM and measures whole-model decoding throughput while accounting for selector-cache memory overhead. This provides practical evidence of its deployment benefits.

  2. Experiments cover both full-attention and hybrid models. Ablations on the distillation objective and tile-shared versus per-query selection during training provide useful support for the method’s design.

Critical weaknesses
  1. The methodological novelty appears incremental: the scoring mechanism follows DSA, selected-token KL distillation has a precedent in DSA’s sparse training stage, and frozen-model selector distillation for decoding has been explored by SeerAttention-R. Although the serving integration is valuable, there is no direct comparison with the closest learned selector. Moreover, the runtime comparisons with Quest and Twilight measure selector latency rather than whole-model decoding throughput. Given their different retained-token counts (Table 26), Highlight’s overall accuracy–efficiency advantage over these sparse baselines remains insufficiently established.

  2. The benefit of adaptive budget allocation is not isolated from simply reading fewer tokens. In Section 5.1, the Qwen3-8B comparison with BF16 selector keys uses a fixed 25% budget versus approximately 12%–19% retention under dual top-p. Appendix K.3 evaluates smaller fixed budgets only on RULER, leaving it unclear whether adaptive allocation provides a better accuracy–efficiency trade-off at comparable cost.

Questions for authors
  1. Which design changes beyond existing approaches are essential to Highlight’s gains? Can a controlled comparison with SeerAttention-R on a common backbone establish their contribution to accuracy and efficiency?

  2. At matched accuracy targets and under the same GPU memory limit, how do Highlight, Quest, and Twilight compare in whole-model decoding throughput, accounting for downstream attention and selector-cache overhead?

  3. How does dual top-p compare with fixed top-K across RULER, BABILong, and LongBench-v2 when average KV-read budgets or measured decoding latency are matched?

4 comments

Thank you for the review. We report new accuracy and whole-model decoding comparisons with SeerAttention-R, Quest and Twilight on Qwen3-8B, and compare dual top-p with fixed top-K at matched read budgets.

W1 and Q1. Which design choices matter, and SeerAttention-R on a common backbone.

The scorer follows DSA's lightning indexer. DSA's sparse stage also uses a KL over the selected tokens, but it trains the whole model. SeerAttention-R also distills a frozen model, with a gate over 64-token blocks. Highlight's change is to train the scorer with the selected-token KL alone, on a frozen model with no dense warm-up, and to read its selection in 16-token blocks. Two ablations and one direct comparison show which choices matter.

  • Loss over selected tokens: essential. Appendix K.2 compares the full-distribution and selected-token objectives on Qwen3-4B (RULER accuracy, %, 32K, fixed budget):
Objective512 tokens1024 tokens2048 tokens
Full distribution (DSA warm-up objective)68.277.687.0
Selected tokens only (Highlight)94.396.996.9
Dense attention95.895.895.8
  • Query tile: no accuracy cost. Selecting once per 16-query tile instead of per query changes no RULER outcome and moves perplexity by at most 0.013 (Appendix K.1).
  • Selector: compared directly with SeerAttention-R. We ran SeerAttention-R on Qwen3-8B with its released gates and TileLang sparse-decode kernel, at the same token budgets. Accuracy (%) is the mean over five sampling seeds on the 435 held-out examples of Table 2 (126 RULER, 240 BABILong and 69 LongBench-v2; prompts of 12K to 32K tokens). Selector keys are BF16. Bold marks a paired difference whose 95% interval excludes zero:
BenchmarkBudget (tokens)SeerAttention-RHighlight
RULER (dense 99.4)204884.695.4
409696.298.6
819299.098.6
BABILong (dense 32.8)204827.034.1
409629.732.9
819231.032.8

Highlight's selector matters most when the budget is tight. At 2048 tokens it leads by 10.8 points on RULER and 7.1 on BABILong, and both differences hold after a Holm correction across all nine budget and benchmark comparisons, LongBench-v2 included. At 4096 tokens, the budget we fixed before running the comparison, the 2.4- and 3.2-point leads fall short of significance, but only Highlight stays within the paper's two-point tolerance of dense on RULER and BABILong. Highlight is never significantly less accurate, and LongBench-v2 differs by at most 0.29 points. At 4096 tokens with MXFP4 selector keys, which also stay within two points of dense on the 435 examples (RULER −0.95, BABILong +1.75 and LongBench-v2 0.00 points), Highlight reaches 1.31× SeerAttention-R's decode throughput at 32K and 1.35× at 64K, measured as in Q2. The two selectors were trained on different data, so the comparison is between released selectors.

Q2. Whole-model throughput against Quest and Twilight, at matched accuracy and memory.

On Qwen3-8B, Highlight, Quest and Twilight all stay within two percentage points of dense on each of RULER, BABILong and LongBench-v2 over the 435 held-out examples.

We measured whole-model decoding in one fixed-batch runner using the vLLM Qwen3-8B backbone. Highlight uses a 4096-token budget with BF16 selector keys; Quest uses 4096 tokens per query head; Twilight uses 8192 candidates and p = 0.85, with the repairs of Appendix I.1. We use Highlight's paged FlashAttention path and Twilight's released FlashInfer sparse decoder; Quest uses the same sparse decoder. All methods share the Qwen3-8B backbone, workload and memory limit, and timing includes all implementation costs.

We report decode tokens per second, averaged over three runs on one B200 per context length, with 256 output tokens per request and prefill excluded. Each method uses its largest batch within the same 85% GPU memory limit, counting selector state and workspace:

Highlight throughput gain16K32K
over dense1.73×1.89×
over Quest (4096 tokens)1.57×1.26×
over Twilight3.27×2.54×

At matched batch sizes, subtracting the identified adapter-kernel times from the baseline profiles leaves Quest at 1.30× and 1.13× Highlight's total profiled GPU time, and Twilight at 2.46× and 2.07×, at 16K and 32K respectively. Highlight's total includes all its adapter work. We will report the breakdown alongside the measured throughput.

W2 and Q3. Dual top-p against fixed top-K at matched reads.

We ran this comparison on all three benchmarks. For each benchmark, we set top-K to the fraction of the context that top-p reads on the 114 development examples, which are disjoint from the 435: 36 to 39% for p = 0.9 and 22 to 26% for p = 0.8. Both then ran once on the 435 held-out examples with BF16 selector keys, in the harness of Table 2. On LongBench-v2 the two give the same score on every example:

SettingRULERBABILongLongBench-v2
p = 0.90.00+1.670.00
p = 0.8−1.590.000.00

Values are top-p minus matched top-K in percentage points. All paired 95% confidence intervals include zero, so the comparison establishes no accuracy advantage for either policy.

Dual top-p uses one threshold to adjust the read share across workloads; matching it with fixed top-K required a separate fraction calibrated for each benchmark on development data. We will add the matched-read table to Section 3 and clarify the budget comparison in Section 5.

Thank you for the response. Regarding my concern to W1, the authors state: “Highlight's change is to train the scorer with the selected-token KL alone, on a frozen model with no dense warm-up, and to read its selection in 16-token blocks.” However, DSA does not inherently require joint training of the backbone and the indexer. Its S1 stage likewise freezes the backbone and trains only the indexer, while its S2 stage unfreezes the backbone to further improve model capabilities. The decision to unfreeze the backbone in S2 does not demonstrate that DSA’s indexer cannot be trained with a frozen backbone. Furthermore, token-level selection is not a novel contribution of this paper. Therefore, I still find the methodological novelty limited and will maintain my original score.

Thank you. You are right: DSA's first stage freezes the backbone, and token-level scoring is DSA's. We claim neither.

But DSA reports no sparse decoding on the frozen model. We ran it, with both of its losses (Appendix K.2, Qwen3-4B, RULER at 32K, dense 95.8%):

Loss, frozen backbone512 tokens2048 tokens
DSA's first-stage objective68.287.0
Selected tokens (Highlight)94.396.9

The first-stage objective loses up to 27.6 points. The selected-token loss, which DSA uses only while retraining the whole model, stays within 1.5 points of dense on the original weights. That result, with the serving evidence, is our contribution, and we will credit DSA for the indexer and both losses.

Reviewer_nUvyOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the cost of reading the KV cache during long-context decoding. It argues that attention is sparse, but that deciding which tokens to read is itself expensive, because existing selectors either hand-build relevance estimates from the keys or require training the model for sparse attention. The authors propose Highlight, a small per-layer selector that is distilled from the frozen model's own attention: four narrow query heads of width 128 score a separate selector key cache, a KL loss is applied only on the tokens the selector selects, and training uses 142M tokens of general pretraining text at 16K sequence length for 540 steps, adding at most 2.4% parameters. The architecture follows the DSA lightning indexer, and the model itself is never further trained. At decode time the selector picks 16-token blocks under a fixed top-K budget or an adaptive dual top-p budget, attention reads the original keys and values of those blocks, and the selector keys can be stored in FP8, INT8 or MXFP4.

They evaluate seven models (Qwen3 at 4B, 8B, 14B and 32B; Qwen3.5 at 4B and 9B; Qwen3.6-27B, plus a Gemma-4-E4B accuracy control) on RULER, BABILong, InfiniteBench and LongBench-v2 up to 128K tokens, and measure serving throughput with a vLLM plugin on one B200 at KV-cache saturation. The main results are that accuracy stays within roughly two points of dense attention for the configurations that pass a pre-registered held-out check, that reading a quarter of the context gives 1.4x to 1.8x dense throughput and up to 2.5x with adaptive budgets and four-bit selector keys, and that the selector call is 6.5x to 8x faster than Twilight's and 1.4x to 3.9x faster than Quest's. The authors conclude that a frozen model's attention can be distilled into a cheap plug-in selector that makes decoding faster without retraining the model. The contribution is primarily systems and empirical.

Critical strengths
  1. A cheap, frozen-model recipe that transfers across model families without any change to the base weights, including hybrid linear-attention architectures. Evidence: Table 3 and Appendix B give one unchanged recipe (4 heads, width 128, 16-token blocks, 540 AdamW steps, 141,557,760 tokens, seed 42) applied to seven checkpoints. Tables 10 and 11 give per-benchmark accuracy for Qwen3 and Qwen3.5, and Appendix K.4 Table 33 applies the same recipe unchanged to Gemma-4-E4B, whose sliding-window and cache-sharing layers are handled by attaching selectors only to the four full-attention layers that own a cache; the pooled difference from dense is +0.7 points at K=0.5L and 0.0 at p=0.9. Why it is consequential: This is what makes the result usable rather than merely interesting. A method that needs continued training, like DSA, or architecture-specific sparse attention, like NSA and MoBA, cannot be applied to an existing deployment. A 540-step distillation that adds at most 2.4% parameters and survives a transfer to a fourth model family is a materially lower adoption barrier, and the Gemma control shows the recipe is not tuned to one family.
  2. Serving-level throughput evidence with a pre-registered accuracy gate and honest accounting of the selector's own memory cost. Evidence: Section 5 and Appendix G. Each configuration fills its own KV pool, so the selector key cache is charged against the requests in flight; at 8K this means 89 requests instead of 112. A barrier holds decoding until every request has prefilled, and the measurement window excludes prefill and drain. Appendix G reports 90 accepted cells for Qwen3-8B after removing the twelve that fail the accuracy check, and Appendix I.1 discloses the four repairs made to Twilight's released kernels. Why it is consequential: Sparse-attention papers commonly report kernel-level speedups and accuracy on different configurations, which makes the practical benefit unreadable. Tying each served configuration to a quality check on the same engine, and counting the selector's memory as a deployment cost, is the evidence a practitioner actually needs, and it is the gap SeerAttention-R left open.
Critical weaknesses

Weakness 1: The paper does not establish that learning the selector buys accuracy over training-free selectors at a matched budget, so the contribution reduces to selector-call latency, which is never converted into an end-to-end serving comparison. Evidence: Appendix F.1, Table 10 runs Quest and Twilight in the same harness on the same examples at budgets of 0.25L, 0.5L and 0.75L. On Qwen3-4B RULER, Quest at K=0.25L scores 98.1 (1.1) against dense 98.1 (1.1), while Highlight at the same budget scores 96.2 (1.5); on Qwen3-8B, Quest scores 99.4 (0.6) against Highlight's 98.1 (1.1). On LongBench-v2 the ordering is mixed in both directions. The learned-selector competitors named as closest in Appendix L, SeerAttention-R and SAS, are not evaluated at all. The speed advantage is measured as an isolated per-layer selector call at batch size 1 (Section 5.3, Table 2, Figure 5), and Quest and Twilight do not appear in any throughput table in Appendix G (Tables 17 to 20). Why it is consequential: This determines what the paper is actually claiming. If a training-free page-min/max bound matches a distilled token-level selector on accuracy at the same budget, then the argument for Highlight is purely that its selector is cheaper to run, and the reader is left to infer the end-to-end consequence from a batch-1 microbenchmark on one component. The abstract's framing, which pairs "accuracy stays close to dense" with "8 times faster than Twilight", invites exactly that inference without supporting it. It also means the extra training cost is not shown to pay for itself in quality. What would resolve it: Either an end-to-end throughput comparison with Quest and Twilight inside the same vLLM stack at KV saturation, at one or two context lengths, under the same two-point accuracy gate; or an accuracy comparison against a SeerAttention-R-style distilled block gate trained on the same budget, which would isolate what token-level scoring with 16-token blocks adds over 64-token block gating. A shorter fix would be to state plainly in Section 4 and the abstract that Highlight's advantage over Quest is selector cost rather than selection quality at matched budgets.

Weakness 2: The conceptual delta over DSA's warm-up stage, Twilight's budgeting and SeerAttention-R's frozen-model distillation is small, and the main text does not say so as directly as the appendix does. Evidence: Section 2.1 states that Highlight "follows the design of the DSA lightning indexer"; Appendix L and Table 34 place the method against DSA, SeerAttention-R, SAS, MISA and IndexCache and describe the training signal as KL to dense attention restricted to selected tokens, which is the DSA warm-up recipe. The Introduction's first contribution bullet, "Distill. We distill a frozen model's attention into Highlight", does not mention that DSA already performs this distillation with the model frozen before continuing to train it. Related scope limits compound this: Appendix G reports that Qwen3-4B, Qwen3-32B and Qwen3.6-27B are timed without any accuracy check; the 16K-to-128K generalization claim is demonstrated only on the two Qwen3.5 hybrids, with Qwen3 evaluated to 32K; prefill stays dense; and all benchmarks are English retrieval or QA tasks, with no long-chain-of-thought evaluation, which is the setting SeerAttention-R targets. Why it is consequential: Novelty at ICLR is judged against the closest prior work, and here the honest description is a well-engineered combination plus a measurement study rather than a new mechanism. That is publishable, but only if the paper claims it. As written, a reader of the first two pages will overestimate the conceptual contribution and underestimate how much of the value lies in the systems evidence. The ungated throughput cells and the absence of any reasoning-trace evaluation also leave the generality of "accuracy stays close to dense" weaker than the abstract implies, since Table 1 itself shows Qwen3-32B falling from 59.9 to 51.6 at top-p=0.7. What would resolve it: Restate the delta explicitly in the Introduction, naming DSA's warm-up stage as the source of the objective and specifying what is new here, namely that warm-up alone suffices and that the selector is deployed in a serving engine. Add an accuracy check or an explicit caveat for the three ungated models in Appendix G. One reasoning-trace or long-generation evaluation, even small, would address the most obvious generalization worry. A capacity or training-budget ablation in Appendix K would substantiate the "same recipe everywhere" claim.

Questions for authors

(Re: W1) [Clarification] In Table 10, Quest at K=0.25L matches or exceeds Highlight on RULER for Qwen3-4B (98.1 vs 96.2) and Qwen3-8B (99.4 vs 98.1). Do you agree that at matched fixed budgets the training-free selector is not worse on this benchmark, and if so, should Section 4 and the abstract frame Highlight's advantage as selector cost rather than selection quality? (Re: W1) [Clarification] Can you report end-to-end throughput for Quest and Twilight in the same vLLM configuration as Tables 17 to 20, at one or two context lengths and under the same two-point gate? If that is infeasible in the rebuttal window, what is your estimate of the end-to-end margin implied by the batch-1 selector-call timings in Table 2, and what fraction of a decode step does the selector call represent at each length? (Re: W1) [Clarification] Why were SeerAttention-R and SAS, which Appendix L identifies as the closest prior work, not included in the accuracy comparison? Was it a matter of available checkpoints for Qwen3, or of block-size and supervision mismatch? (Re: W2) [Potential correction] Appendix G states that Qwen3-4B, Qwen3-32B and Qwen3.6-27B "have no accuracy check and are timed only". Given that Table 1 shows an 8-point mean drop for Qwen3-32B at top-p=0.7, how do you know the served configurations for these three models are quality-acceptable, and should the throughput results for them carry a caveat? (Re: W2) [Clarification] The 16K-to-128K transfer is shown only for the two Qwen3.5 hybrids, and the 128K RULER bins hold 9 to 14 examples. Do you have any evidence that selectors trained at 16K hold up beyond 32K on the full-attention Qwen3 models, or is the claim restricted to the hybrid family? (Re: W2) [Clarification] Appendix K varies the selection unit, the loss support, the smallest budget and the model family. Did you test sensitivity to selector capacity (number of heads or head width), to the 142M training-token budget, or to the 16-token block size? If any of these were tested and dropped, what did they show?

2 comments

Thank you for the review. We clarify that Highlight's advantage over Quest at matched budgets is decoding cost, and report new whole-model comparisons with Quest, Twilight and SeerAttention-R.

Q1. Quest at matched fixed budgets.

Yes. Quest matches or exceeds Highlight's RULER scores at the same fixed budgets in Table 10. On the 435 held-out examples of Table 2, with Qwen3-8B and a 4096-token budget, Highlight minus Quest is −0.79 [−2.38, 0.00] points on RULER, +0.58 [−1.08, +2.33] on BABILong and −0.29 [−0.87, 0.00] on LongBench-v2 (five sampling seeds, paired 95% intervals). We will clarify in Section 4 and the abstract that Highlight's advantage over Quest at a fixed budget is decoding speed: 1.26× to 1.57× Quest's throughput (Q2).

Q2. Decode throughput against Quest and Twilight.

Highlight, Quest and Twilight all stay within two percentage points of dense on each of RULER, BABILong and LongBench-v2 over the 435 held-out Qwen3-8B examples.

We measured whole-model decoding in one fixed-batch runner using the vLLM Qwen3-8B backbone, separate from the serving engine of Tables 17 to 20. Highlight uses a 4096-token budget with BF16 selector keys; Quest uses 4096 tokens per query head; Twilight uses 8192 candidates and p = 0.85, with the repairs of Appendix I.1. Dense and Highlight use FlashAttention 4; Quest and Twilight use the released FlashInfer-based sparse-attention kernels. Timing includes all cache updates, selection, attention and integration work.

We report decode tokens per second, averaged over three runs on one B200 per context length, with 256 output tokens per request and prefill excluded. Each method uses its largest batch within the same 85% GPU memory limit, counting selector state and workspace:

Highlight throughput gain16K32K
over dense1.73×1.89×
over Quest (4096 tokens)1.57×1.26×
over Twilight3.27×2.54×

At matched batch sizes, subtracting the identified adapter-kernel times from the baseline profiles leaves Quest at 1.30× and 1.13× Highlight’s total profiled GPU time, and Twilight at 2.46× and 2.07×, at 16K and 32K respectively. We will report the breakdown alongside the measured throughput.

In the serving engine of Tables 17 to 20, we estimate selector cost by subtracting the dense step time from the step time with every block selected, at the same batch. At KV saturation, this costs 4.1 to 6.5 ms, or 29% to 41% of the top-K 25% step across 8K to 255.5K.

Q3. SeerAttention-R and SAS.

We now compare Highlight with SeerAttention-R on Qwen3-8B using its released gates and TileLang sparse-decode kernel. SeerAttention-R uses 64-token blocks and was trained on reasoning traces; Highlight was trained on general text. Accuracy (%) is the mean over five sampling seeds on the 435 held-out examples, with BF16 selector keys; bold marks a paired difference whose 95% interval excludes zero:

BenchmarkBudget (tokens)SeerAttention-RHighlight
RULER (dense 99.4)204884.695.4
409696.298.6
819299.098.6
BABILong (dense 32.8)204827.034.1
409629.732.9
819231.032.8

At 2048 tokens, Highlight leads by 10.8 points on RULER and 7.1 on BABILong; both differences remain significant after a Holm correction across all nine budget and benchmark comparisons. LongBench-v2 differs by at most 0.29 points. At 4096 tokens, the budget fixed before the comparison, the leads are not statistically significant, but only Highlight stays within two points of dense on all three benchmarks. With MXFP4 selector keys at 4096 tokens, Highlight also stays within two points of dense on the 435 examples (RULER −0.95, BABILong +1.75 and LongBench-v2 0.00 points) and reaches 1.31× SeerAttention-R's decode throughput at 32K and 1.35× at 64K, measured as in Q2.

SAS appeared on 11 September 2026, with Qwen3 selector weights released on 14 September, shortly before submission. We discuss this contemporaneous work in Appendix L. We focused the new experiments on SeerAttention-R because it is the closest baseline using frozen-model attention distillation. We will add the Q2 and Q3 comparisons to the paper and place the end-to-end Twilight ratios next to the abstract's selector-call figure.

Q4. Timed-only models.

The Qwen3-4B, Qwen3-32B and Qwen3.6-27B serving results are timing-only measurements. Table 10 reports accuracy for Qwen3-4B and Qwen3-32B in the evaluation harness; the abstract's throughput claims use Qwen3-8B configurations that pass the serving-engine accuracy check. We will label these cells and the backbone comparison as timing only in Appendix G and include the corresponding Table 10 accuracy where available.

Q5. Beyond 32K on full-attention Qwen3.

Accuracy evaluation on the full-attention Qwen3 models covers contexts up to 32K. Their checkpoint window is 40,960 tokens; the 255.5K serving cells use YaRN and report timing only. The evidence for accuracy transfer from 16K training to 128K evaluation comes from the Qwen3.5 hybrids. We will state this scope in the contributions and Section 4 and report intervals for the 128K RULER bins, which hold 9 examples each.

Q6. Capacity, training budget and block size.

We used four selector heads of width 128, 142M training tokens and 16-token blocks. We did not vary head count, head width, training-token budget or block size. A 16-token block matches one page of vLLM's KV cache, allowing FlashAttention to read the selected blocks through its block table. We will state these fixed settings and the scope of the ablations in the limitations.

W2. How Highlight differs from DSA and SeerAttention-R.

DSA's warm-up trains the indexer with KL to the full attention distribution. Its sparse stage uses KL over selected tokens while updating the whole model. Highlight uses the selected-token KL alone to train a frozen model's selector, without a warm-up stage. On frozen Qwen3-4B at 32K, the full-distribution objective gives 68.2% RULER accuracy against 94.3% for the selected-token KL at 512 tokens, and 87.0% against 96.9% at 2048, with dense at 95.8% (Appendix K.2). SeerAttention-R also freezes the backbone, but distills block-pooled attention into a 64-token block gate. We will revise the contributions to credit DSA for the indexer and both objectives, Twilight for top-p budgets and SeerAttention-R for frozen-model distillation. We will also state the evaluation scope in the limitations: dense prefill, English benchmarks and no reasoning-trace evaluation.

Reviewer_3yQYOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The KV cache has become a major bottleneck for modern LLMs. This paper presents Highlight, a block predictor distilled from the model’s dense attention, and uses it to perform top-k or top-P selection. Evaluation shows that Highlight has lower selector latency with comparable or higher accuracy.

Critical strengths
  • The proposed indexer is low overhead compared to existing solutions
Critical weaknesses
  • Very similar to DSA and SeerAttention while not compared to SeerAttention in the evaluation
  • Need training
Questions for authors

Thanks for submitting the paper. I have some concern about the paper.

  1. What is the major difference between Highlight and SeerAttention? It seems that both train a gate and use it for selection. Additionally, Highlight seems very similar to DeepSeek Sparse Attention; essentially, you turn DSA’s online training into offline training. Overall, the novelty of Highlight is very limited: it mostly stitches together SeerAttention’s training, DSA’s indexer, and Quest/Twilight’s top-k and top-P selection.
  2. In the evaluation, why not compare against SeerAttention?
  3. The accuracy seems very similar to that of existing systems, and I am not sure how the selector speedup translates into end-to-end system speedup. Considering the need for training, the value of the work is weakened.
  4. The extra memory usage seems to be a big concern, especially as KV cache quantization becomes more and more popular. Given FP8 and even NVFP4 KV caches, the room for further quantizing the indexer seems limited.
2 comments

Thank you for the review. We answer each question below, with new whole-model decoding measurements against SeerAttention-R, Quest and Twilight on Qwen3-8B.

Q1. Difference from SeerAttention and DSA.

SeerAttention learns block gates for sparse prefill. Highlight sparsifies decoding, so the matching baseline is SeerAttention-R, which extends the gate to decoding: it pools keys over 64-token blocks, maps each GQA group's query heads to one gate query, distills block-pooled attention, and leaves end-to-end serving to future work. Highlight scores every cached token with four 128-dimensional heads, distills token-level attention on the tokens it selects, reads 16-token blocks, and runs end to end in vLLM. Q2 compares the two directly.

Our selector is inspired by DSA's lightning indexer. DSA trains it in two stages: a warm-up on the frozen model with KL to the full attention distribution, then KL over the selected tokens while the whole model is updated. We ask whether a frozen model's attention can be distilled into a plug-in selector, and with which objective. Highlight uses the selected-token KL alone, on a frozen model and without the warm-up. Appendix K.2 trains two Qwen3-4B selectors that differ only in the loss and reports RULER accuracy (%) at 32K under a fixed budget:

Objective512 tokens1024 tokens2048 tokens
Full distribution (DSA warm-up)68.277.687.0
Selected tokens (ours)94.396.996.9
Dense attention95.895.895.8

Turning DSA's training offline keeps only its warm-up stage, since the second stage updates the model. The warm-up loss trails dense by 8.9 to 27.6 points; the selected-token loss stays within 1.6 points of dense at every budget.

We also select 16-token blocks instead of single tokens, so that attention runs on tensor cores with the standard paged kernel: each block is one page of vLLM's KV cache, which FlashAttention reads through its block table. Scores, attention and the loss stay per token.

Top-K and top-p (nucleus) selection are standard, and we do not claim them. We will revise the contributions to name each source and to state what is ours: the selected-token objective on a frozen model, token scoring read in 16-token blocks, and the end-to-end serving evidence.

Q2. Comparison with SeerAttention-R.

We now include SeerAttention-R. We run it on Qwen3-8B with its released gates and TileLang sparse-decode kernel, next to Highlight, in one fixed-batch runner around the vLLM backbone. Accuracy (%) is the mean over five sampling seeds on the 435 held-out examples of Table 2 (prompts of 12K to 32K tokens), with BF16 selector keys; bold marks a paired difference whose 95% interval excludes zero:

BenchmarkBudget (tokens)SeerAttention-RHighlight
RULER (dense 99.4)204884.695.4
409696.298.6
819299.098.6
BABILong (dense 32.8)204827.034.1
409629.732.9
819231.032.8

The gap widens as the budget shrinks. At 2048 tokens, Highlight leads by 10.8 points on RULER and 7.1 on BABILong; both differences hold after a Holm correction across all nine budget and benchmark comparisons, LongBench-v2 included. At 4096 tokens, the budget we fixed before running the comparison, Highlight leads by 2.4 and 3.2 points, short of significance; there Highlight stays within two points of dense, while SeerAttention-R falls more than three points below dense on both benchmarks. At 8192 no difference is significant. Highlight is never significantly less accurate, and LongBench-v2 differs by at most 0.29 points.

Throughput is decode tokens per second on one B200, the mean of three runs, with 256 output tokens per request and prefill excluded. Each method runs the largest batch that fits in 85% of GPU memory, with its selector cache counted. At 4096 tokens with MXFP4 selector keys, Highlight reaches 1.31× SeerAttention-R's throughput at 32K and 1.35× at 64K. On the same 435 examples, MXFP4 keys at 4096 tokens stay within two points of dense: RULER −0.95, BABILong +1.75 and LongBench-v2 0.00 points.

SeerAttention-R's gates were trained on reasoning traces and ours on general text, so this compares released selectors.

Q3. Accuracy, end-to-end speed and training cost.

On Qwen3-8B, Highlight, Quest and Twilight all stay within two percentage points of dense on each of RULER, BABILong and LongBench-v2 over the 435 held-out examples. We compare decoding cost among settings that meet this common accuracy target.

We now measure complete model decoding in a common fixed-batch runner around the vLLM Qwen3-8B backbone. Highlight uses 4096 tokens with BF16 selector keys; Quest uses 4096 tokens per query head; Twilight uses 8192 candidates and p = 0.85, with the repairs of Appendix I.1. Dense and Highlight use FlashAttention 4, and Quest and Twilight use the released FlashInfer-based sparse-attention kernels. Timing includes all cache updates, selection, attention and integration work.

Throughput is the mean of three runs on one B200 per context length, with 256 output tokens per request and prefill excluded. Each method uses its largest batch within the same 85% GPU memory limit, counting selector state and workspace:

Highlight throughput gain16K32K
over dense1.73×1.89×
over Quest (4096 tokens)1.57×1.26×
over Twilight3.27×2.54×

We also profile integration costs at fixed batches of 46 (16K) and 22 (32K). Subtracting all identified adapter-kernel time from the baseline profiles, while retaining Highlight's complete GPU cost, leaves Quest at 1.30× and 1.13× Highlight's cost, and Twilight at 2.46× and 2.07×, respectively. We will report the component breakdown alongside the measured throughput.

Training is a one-time cost per backbone: 540 steps on 142M tokens of general text, with the model frozen and no task-specific training data. It takes about two hours for Qwen3-8B and under seven hours for Qwen3-32B, each on four B200 GPUs. The same trained selector is reused across evaluation tasks.

Q4. Selector memory under quantized KV caches.

The selector's share depends on its precision relative to the KV cache. On Qwen3-8B, a selector key row holds 512 values per token and layer and the KV cache 2048, so at equal precision the selector adds a quarter of the KV cache's size. Selector bytes as a share of KV bytes, per token and layer (bold: equal precision):

KV cacheBF16 selector keys (1024 B)FP8 (528 B)MXFP4 (272 B)
BF16 (4096 B)25%13%6.6%
FP8 (2048 B)50%26%13%
NVFP4 (1152 B)89%46%24%

Quantizing the KV cache further than the selector keys increases the selector's relative cost; matching their precision keeps it at about a quarter. Our measurements use BF16 KV caches: MXFP4 selector keys occupy 6.6% of KV-cache storage, pass the accuracy rule and achieve 1.82× to 2.29× dense throughput with selector memory counted (Appendix H). The FP8 and NVFP4 rows show memory ratios; the reported accuracy and throughput use BF16 attention KV caches with quantized selector keys. We will add the table and clarify this measurement scope in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.