- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Highlight, a lightweight attention selector that enables sparse decoding without updating the pretrained language model. The selector maintains a compact key cache and learns from the frozen model’s attention through KL divergence over selected tokens. At inference, it selects 16-token blocks from the original KV cache using either a fixed top-K budget or an adaptive dual top-p policy. Experiments on several full-attention and hybrid models show accuracy close to dense attention in many evaluated settings, including context-length generalization from 16K training to 128K evaluation on Qwen3.5.
Critical strengths
-
Highlight integrates sparse selection into vLLM and measures whole-model decoding throughput while accounting for selector-cache memory overhead. This provides practical evidence of its deployment benefits.
-
Experiments cover both full-attention and hybrid models. Ablations on the distillation objective and tile-shared versus per-query selection during training provide useful support for the method’s design.
Critical weaknesses
-
The methodological novelty appears incremental: the scoring mechanism follows DSA, selected-token KL distillation has a precedent in DSA’s sparse training stage, and frozen-model selector distillation for decoding has been explored by SeerAttention-R. Although the serving integration is valuable, there is no direct comparison with the closest learned selector. Moreover, the runtime comparisons with Quest and Twilight measure selector latency rather than whole-model decoding throughput. Given their different retained-token counts (Table 26), Highlight’s overall accuracy–efficiency advantage over these sparse baselines remains insufficiently established.
-
The benefit of adaptive budget allocation is not isolated from simply reading fewer tokens. In Section 5.1, the Qwen3-8B comparison with BF16 selector keys uses a fixed 25% budget versus approximately 12%–19% retention under dual top-p. Appendix K.3 evaluates smaller fixed budgets only on RULER, leaving it unclear whether adaptive allocation provides a better accuracy–efficiency trade-off at comparable cost.
Questions for authors
-
Which design changes beyond existing approaches are essential to Highlight’s gains? Can a controlled comparison with SeerAttention-R on a common backbone establish their contribution to accuracy and efficiency?
-
At matched accuracy targets and under the same GPU memory limit, how do Highlight, Quest, and Twilight compare in whole-model decoding throughput, accounting for downstream attention and selector-cache overhead?
-
How does dual top-p compare with fixed top-K across RULER, BABILong, and LongBench-v2 when average KV-read budgets or measured decoding latency are matched?
4 comments
Q2. Whole-model throughput against Quest and Twilight, at matched accuracy and memory.
On Qwen3-8B, Highlight, Quest and Twilight all stay within two percentage points of dense on each of RULER, BABILong and LongBench-v2 over the 435 held-out examples.
We measured whole-model decoding in one fixed-batch runner using the vLLM Qwen3-8B backbone. Highlight uses a 4096-token budget with BF16 selector keys; Quest uses 4096 tokens per query head; Twilight uses 8192 candidates and p = 0.85, with the repairs of Appendix I.1. We use Highlight's paged FlashAttention path and Twilight's released FlashInfer sparse decoder; Quest uses the same sparse decoder. All methods share the Qwen3-8B backbone, workload and memory limit, and timing includes all implementation costs.
We report decode tokens per second, averaged over three runs on one B200 per context length, with 256 output tokens per request and prefill excluded. Each method uses its largest batch within the same 85% GPU memory limit, counting selector state and workspace:
| Highlight throughput gain | 16K | 32K |
|---|---|---|
| over dense | 1.73× | 1.89× |
| over Quest (4096 tokens) | 1.57× | 1.26× |
| over Twilight | 3.27× | 2.54× |
At matched batch sizes, subtracting the identified adapter-kernel times from the baseline profiles leaves Quest at 1.30× and 1.13× Highlight's total profiled GPU time, and Twilight at 2.46× and 2.07×, at 16K and 32K respectively. Highlight's total includes all its adapter work. We will report the breakdown alongside the measured throughput.
W2 and Q3. Dual top-p against fixed top-K at matched reads.
We ran this comparison on all three benchmarks. For each benchmark, we set top-K to the fraction of the context that top-p reads on the 114 development examples, which are disjoint from the 435: 36 to 39% for p = 0.9 and 22 to 26% for p = 0.8. Both then ran once on the 435 held-out examples with BF16 selector keys, in the harness of Table 2. On LongBench-v2 the two give the same score on every example:
| Setting | RULER | BABILong | LongBench-v2 |
|---|---|---|---|
| p = 0.9 | 0.00 | +1.67 | 0.00 |
| p = 0.8 | −1.59 | 0.00 | 0.00 |
Values are top-p minus matched top-K in percentage points. All paired 95% confidence intervals include zero, so the comparison establishes no accuracy advantage for either policy.
Dual top-p uses one threshold to adjust the read share across workloads; matching it with fixed top-K required a separate fraction calibrated for each benchmark on development data. We will add the matched-read table to Section 3 and clarify the budget comparison in Section 5.
Thank you for the response. Regarding my concern to W1, the authors state: “Highlight's change is to train the scorer with the selected-token KL alone, on a frozen model with no dense warm-up, and to read its selection in 16-token blocks.” However, DSA does not inherently require joint training of the backbone and the indexer. Its S1 stage likewise freezes the backbone and trains only the indexer, while its S2 stage unfreezes the backbone to further improve model capabilities. The decision to unfreeze the backbone in S2 does not demonstrate that DSA’s indexer cannot be trained with a frozen backbone. Furthermore, token-level selection is not a novel contribution of this paper. Therefore, I still find the methodological novelty limited and will maintain my original score.
Thank you. You are right: DSA's first stage freezes the backbone, and token-level scoring is DSA's. We claim neither.
But DSA reports no sparse decoding on the frozen model. We ran it, with both of its losses (Appendix K.2, Qwen3-4B, RULER at 32K, dense 95.8%):
| Loss, frozen backbone | 512 tokens | 2048 tokens |
|---|---|---|
| DSA's first-stage objective | 68.2 | 87.0 |
| Selected tokens (Highlight) | 94.3 | 96.9 |
The first-stage objective loses up to 27.6 points. The selected-token loss, which DSA uses only while retraining the whole model, stays within 1.5 points of dense on the original weights. That result, with the serving evidence, is our contribution, and we will credit DSA for the indexer and both losses.
Thank you for the review. We report new accuracy and whole-model decoding comparisons with SeerAttention-R, Quest and Twilight on Qwen3-8B, and compare dual top-p with fixed top-K at matched read budgets.
W1 and Q1. Which design choices matter, and SeerAttention-R on a common backbone.
The scorer follows DSA's lightning indexer. DSA's sparse stage also uses a KL over the selected tokens, but it trains the whole model. SeerAttention-R also distills a frozen model, with a gate over 64-token blocks. Highlight's change is to train the scorer with the selected-token KL alone, on a frozen model with no dense warm-up, and to read its selection in 16-token blocks. Two ablations and one direct comparison show which choices matter.
Highlight's selector matters most when the budget is tight. At 2048 tokens it leads by 10.8 points on RULER and 7.1 on BABILong, and both differences hold after a Holm correction across all nine budget and benchmark comparisons, LongBench-v2 included. At 4096 tokens, the budget we fixed before running the comparison, the 2.4- and 3.2-point leads fall short of significance, but only Highlight stays within the paper's two-point tolerance of dense on RULER and BABILong. Highlight is never significantly less accurate, and LongBench-v2 differs by at most 0.29 points. At 4096 tokens with MXFP4 selector keys, which also stay within two points of dense on the 435 examples (RULER −0.95, BABILong +1.75 and LongBench-v2 0.00 points), Highlight reaches 1.31× SeerAttention-R's decode throughput at 32K and 1.35× at 64K, measured as in Q2. The two selectors were trained on different data, so the comparison is between released selectors.