- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper highlights a key mismatch between KL-trained indexers and top-p sparse attention: accurate token rankings do not necessarily translate into well-calibrated cumulative attention mass, potentially leading to excessive token selection. To address this issue, the authors propose a budget-calibration loss that requires no additional model parameters. The proposed approach is supported by a theoretical construction, a two-stage training strategy, ablation studies across multiple architectures, and empirical evaluations of inference efficiency.
Critical strengths
- The paper is well motivated, with clear preliminary experiments that illustrate the limitations of existing KL-trained indexers. The authors also provide a convincing explanation of why accurate token rankings and low KL divergence do not necessarily guarantee an appropriate top-p attention budget. These observations are further supported by targeted ablation studies.
- The experimental evaluation is comprehensive, covering diverse model architectures and scales. The results demonstrate the effectiveness of the proposed method in adaptively allocating attention budgets across different context lengths, selecting fewer keys for shorter contexts while allocating more keys when necessary to improve retrieval performance at 128K context lengths.
Critical weaknesses
The gap between the attention budgets selected by the trained CaliTopP and the teacher oracle remains unclear. In particular, the paper does not reproduce the context-length sweep for DeepSeek-V3.2 shown in Figure 1(b) after calibration, making it difficult to assess how much median and tail over-selection persists, especially at longer context lengths.
Questions for authors
- What is the latency breakdown of scoring, selection, and attention costs when comparing the calibrated and KL-only top-p models?
- What is the budget gap between the trained CaliTopP and the teacher oracle? Could you repeat the analysis in Figure 1(b) after calibration?
- Have you evaluated fitted global or per-layer score temperatures as simpler calibration controls?
- At long context lengths, are the increased token budgets and performance gains over fixed top-k consistent across different models and task families?
6 comments
Q1
We first break down the index selection overhead in Figure 3 into indexer score computation and token selection, and evaluate the DeepSeek-V3.2 top-p model trained with KL only under the same configuration. The results are as follows:
Prefill
| Top-k normalized execution time with Full = 1 (indexer computation / top-k selection / sparse attention) | Top-p normalized execution time with Full = 1 (indexer computation / top-p selection / sparse attention) | Top-p KL only normalized execution time with Full = 1 (indexer computation / top-p selection / sparse attention) |
|---|---|---|
| 0.7226 / 0.0405 / 0.5437 | 0.6872 / 0.1579 / 0.1398 | 0.6991 / 0.1598 / 0.4503 |
| 0.4586 / 0.0442 / 0.3715 | 0.4413 / 0.1027 / 0.0853 | 0.4593 / 0.1343 / 0.2302 |
| 0.2736 / 0.0334 / 0.2134 | 0.2648 / 0.0694 / 0.0498 | 0.2623 / 0.0685 / 0.1718 |
| 0.1728 / 0.0236 / 0.1127 | 0.1702 / 0.0544 / 0.0370 | 0.1916 / 0.0719 / 0.0789 |
| 0.1180 / 0.0180 / 0.0567 | 0.1160 / 0.0507 / 0.0299 | 0.1189 / 0.0549 / 0.0635 |
| 0.0889 / 0.0136 / 0.0273 | 0.0875 / 0.0546 / 0.0257 | 0.0878 / 0.0570 / 0.0413 |
Decode
| Top-k normalized execution time with Full = 1 (indexer computation / top-k selection / sparse attention) | Top-p normalized execution time with Full = 1 (indexer computation / top-p selection / sparse attention) | Top-p KL only normalized execution time with Full = 1 (indexer computation / top-p selection / sparse attention) |
|---|---|---|
| 0.1664 / 0.0566 / 0.1500 | 0.0968 / 0.0572 / 0.1260 | 0.1734 / 0.0682 / 0.1278 |
| 0.1640 / 0.0690 / 0.1480 | 0.1011 / 0.0739 / 0.1250 | 0.1458 / 0.0687 / 0.1286 |
| 0.1708 / 0.0972 / 0.1410 | 0.1244 / 0.1006 / 0.1220 | 0.1328 / 0.1270 / 0.1258 |
| 0.1975 / 0.1045 / 0.1360 | 0.1576 / 0.1134 / 0.1230 | 0.1716 / 0.1295 / 0.1314 |
| 0.2470 / 0.1200 / 0.1270 | 0.2303 / 0.1277 / 0.1250 | 0.2474 / 0.1412 / 0.1395 |
| 0.3113 / 0.1657 / 0.1180 | 0.3356 / 0.1794 / 0.1330 | 0.3353 / 0.2078 / 0.1421 |
These experiments show that the KL-only model has an index selection time comparable to that of Top-p, but takes longer to compute sparse attention because it selects more tokens. Meanwhile, top-p selection incurs noticeably greater overhead than top-k selection. We believe this is because current mainstream sparse attention methods all use top-k retrieval, so the official vLLM implementation has relatively mature optimizations for top-k operators. For example, vLLM uses different top-k operators for prefill and decode, whereas top-p uses our own implementation, resulting in a performance gap. As research on top-p sparse attention grows, more high-performance top-p operators have emerged, such as those in Twilight and RaBitQCache. We believe that further improving the efficiency of top-p selection is also a worthwhile direction for future research.
Q3
To address your concern about the lack of a simple baseline that fits a global or per-layer score temperature, we trained a per-layer temperature calibration baseline on DeepSeek-V2-Lite. It learns a scalar parameter for each layer, initialized to 1. To facilitate comparison, we keep the backbone frozen. The indexer is optimized using the original row-weighted KL divergence objective, while the temperature parameters are optimized only by the same budget loss , with the stop-gradient log probabilities divided by the temperature during computation. Specifically, the original indexer scores are
We additionally apply the following to each layer:
This method also starts from the same KL-only checkpoint after 900 updates and performs 600 additional Stage 1 updates, using the same training data and budget loss configuration. We evaluate both methods using the same top- threshold and LongBench v1 evaluation protocol. The results are as follows:
| Subtask | Temperature baseline score | CaliTopP (S1) score | Temperature baseline tokens | Our tokens | Our token reduction |
|---|---|---|---|---|---|
| NarrativeQA | 21.43 | 21.75 | 4016 | 3626 | 9.72% |
| Qasper | 33.77 | 34.83 | 1467 | 1409 | 4.00% |
| MultiFieldQA-en | 46.79 | 47.19 | 2097 | 1929 | 8.03% |
| HotpotQA | 26.31 | 26.76 | 3332 | 2763 | 17.09% |
| 2WikiMQA | 19.67 | 18.98 | 2066 | 1763 | 14.63% |
| MuSiQue | 12.32 | 12.06 | 3790 | 3068 | 19.07% |
| GovReport | 33.02 | 32.68 | 2740 | 2570 | 6.21% |
| QMSum | 23.47 | 23.60 | 3441 | 3175 | 7.72% |
| MultiNews | 25.77 | 25.83 | 876 | 830 | 5.19% |
| TriviaQA | 84.53 | 85.17 | 2654 | 2028 | 23.59% |
| Passage Retrieval-en | 76.50 | 76.50 | 3198 | 2480 | 22.47% |
| LCC | 63.39 | 63.82 | 1088 | 1019 | 6.33% |
| RepoBench-P | 65.63 | 65.27 | 2781 | 2444 | 12.12% |
| Average | 40.97 | 41.11 | 2580 | 2239 | 13.25% |
Our method achieves a higher score while reducing the number of retained tokens on every task, with an average reduction of 13.25% compared with this baseline. This demonstrates that assigning a shared temperature coefficient to each layer cannot achieve the budget efficiency obtained by applying directly to the indexer weights.
Q4
Regarding your question about whether the gains are consistent across different models and tasks, the experimental results in the paper indicate that they are not. Budget increases and performance improvements vary by model and task category. For example, on LongBench v1, the average budget is 2442 for DeepSeek-V2-Lite and 3110 for Llama-3.1-8B. Differences also occur across tasks: for DeepSeek-V3.2 at a context length of 32K, the median budget required to cover 90% of the teacher's attention mass is 514 tokens for multi-key retrieval and 868 tokens for question answering. The token counts required by the same layer can differ by a median factor of 5 across task categories. These differences also demonstrate that a single fixed cannot efficiently accommodate the diverse requirements of different models, tasks, and inputs. Our proposed top- method dynamically adjusts the number of selected tokens based on the estimated attention distribution for each query, allocating more tokens where needed and fewer where attention is concentrated, with no noticeable degradation in model capability across all scenarios.
We sincerely thank the reviewer for the detailed and constructive comments. We hope that the responses below address your concerns.
W1&Q2
We repeated the same experiment on our trained DeepSeek-V3.2 model using the same configuration as Figure 1(b). The results are as follows:
We have also added a comparison plot of the model before and after training, using the same statistics as Figure 1(b), as shown in Figure 11 of the paper. Our method effectively reduces token over-selection, and the ratios of its selected token counts to the teacher oracle counts are all close to 1. Meanwhile, top-p recall does not decrease noticeably and remains close to the specified p=0.9. This demonstrates that our method can indeed achieve relatively accurate top-p recall.