acceptodds
Under review as a conference paper at ICLR 2027

CaliTopP: Natively Trainable Top- Sparse Attention via Set-Mass Calibration

12 views

Abstract

As context windows grow, the quadratic cost of full attention becomes the main bottleneck of long-context LLM inference, motivating sparse attention. Learning lightweight indexers for token selection is emerging as a mainstream approach because it maintains consistency between training and inference. However, these indexers almost universally use fixed top-k budgets, overlooking the vast differences in token requirements across tasks and models. In contrast, top-p selection retains a target fraction of attention mass, providing a more reliable accuracy guarantee and an input-adaptive budget. Top-k selection depends only on token rankings, whereas top-p additionally depends on cumulative probability mass. We show theoretically and empirically that KL-distilled top-k indexers can accurately rank tokens yet underestimate the cumulative mass of important tokens, leading to severe over-selection when reused for top-p selection. Based on this finding, we propose CaliTopP, a trainable token-level top-p sparse-attention framework. Its parameter-free budget-calibration loss aligns the indexer's selected budget with the model's true requirement without adding parameters or reducing the recall of important tokens. With only two lightweight training stages, CaliTopP transfers across diverse architectures and achieves up to a speedup over dense attention while preserving generation quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Top-p Sparse Attention

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pBGSOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper highlights a key mismatch between KL-trained indexers and top-p sparse attention: accurate token rankings do not necessarily translate into well-calibrated cumulative attention mass, potentially leading to excessive token selection. To address this issue, the authors propose a budget-calibration loss that requires no additional model parameters. The proposed approach is supported by a theoretical construction, a two-stage training strategy, ablation studies across multiple architectures, and empirical evaluations of inference efficiency.

Critical strengths
  1. The paper is well motivated, with clear preliminary experiments that illustrate the limitations of existing KL-trained indexers. The authors also provide a convincing explanation of why accurate token rankings and low KL divergence do not necessarily guarantee an appropriate top-p attention budget. These observations are further supported by targeted ablation studies.
  2. The experimental evaluation is comprehensive, covering diverse model architectures and scales. The results demonstrate the effectiveness of the proposed method in adaptively allocating attention budgets across different context lengths, selecting fewer keys for shorter contexts while allocating more keys when necessary to improve retrieval performance at 128K context lengths.
Critical weaknesses

The gap between the attention budgets selected by the trained CaliTopP and the teacher oracle remains unclear. In particular, the paper does not reproduce the context-length sweep for DeepSeek-V3.2 shown in Figure 1(b) after calibration, making it difficult to assess how much median and tail over-selection persists, especially at longer context lengths.

Questions for authors
  1. What is the latency breakdown of scoring, selection, and attention costs when comparing the calibrated and KL-only top-p models?
  2. What is the budget gap between the trained CaliTopP and the teacher oracle? Could you repeat the analysis in Figure 1(b) after calibration?
  3. Have you evaluated fitted global or per-layer score temperatures as simpler calibration controls?
  4. At long context lengths, are the increased token budgets and performance gains over fixed top-k consistent across different models and task families?
6 comments

We sincerely thank the reviewer for the detailed and constructive comments. We hope that the responses below address your concerns.

W1&Q2

We repeated the same experiment on our trained DeepSeek-V3.2 model using the same configuration as Figure 1(b). The results are as follows:

LengthSelected middle-token count: before → afterOracle countMiddle-token ratio: before → afterOriginal top-p coverage: before → after
4K159 → 103.5991.605× → 1.193×91.28% → 89.08%
8K309.5 → 156173.51.611× → 1.061×91.43% → 89.02%
16K762 → 356422.51.610× → 0.957×91.46% → 88.95%
32K1540.5 → 7177811.782× → 1.038×91.87% → 89.41%
64K2442.5 → 1101.59801.966× → 1.129×92.25% → 89.73%
128K3350.5 → 15541382.51.937× → 1.178×92.00% → 89.82%

We have also added a comparison plot of the model before and after training, using the same statistics as Figure 1(b), as shown in Figure 11 of the paper. Our method effectively reduces token over-selection, and the ratios of its selected token counts to the teacher oracle counts are all close to 1. Meanwhile, top-p recall does not decrease noticeably and remains close to the specified p=0.9p=0.9. This demonstrates that our method can indeed achieve relatively accurate top-p recall.

Q1

We first break down the index selection overhead in Figure 3 into indexer score computation and token selection, and evaluate the DeepSeek-V3.2 top-p model trained with KL only under the same configuration. The results are as follows:

Prefill

Top-k normalized execution time with Full = 1 (indexer computation / top-k selection / sparse attention)Top-p normalized execution time with Full = 1 (indexer computation / top-p selection / sparse attention)Top-p KL only normalized execution time with Full = 1 (indexer computation / top-p selection / sparse attention)
0.7226 / 0.0405 / 0.54370.6872 / 0.1579 / 0.13980.6991 / 0.1598 / 0.4503
0.4586 / 0.0442 / 0.37150.4413 / 0.1027 / 0.08530.4593 / 0.1343 / 0.2302
0.2736 / 0.0334 / 0.21340.2648 / 0.0694 / 0.04980.2623 / 0.0685 / 0.1718
0.1728 / 0.0236 / 0.11270.1702 / 0.0544 / 0.03700.1916 / 0.0719 / 0.0789
0.1180 / 0.0180 / 0.05670.1160 / 0.0507 / 0.02990.1189 / 0.0549 / 0.0635
0.0889 / 0.0136 / 0.02730.0875 / 0.0546 / 0.02570.0878 / 0.0570 / 0.0413

Decode

Top-k normalized execution time with Full = 1 (indexer computation / top-k selection / sparse attention)Top-p normalized execution time with Full = 1 (indexer computation / top-p selection / sparse attention)Top-p KL only normalized execution time with Full = 1 (indexer computation / top-p selection / sparse attention)
0.1664 / 0.0566 / 0.15000.0968 / 0.0572 / 0.12600.1734 / 0.0682 / 0.1278
0.1640 / 0.0690 / 0.14800.1011 / 0.0739 / 0.12500.1458 / 0.0687 / 0.1286
0.1708 / 0.0972 / 0.14100.1244 / 0.1006 / 0.12200.1328 / 0.1270 / 0.1258
0.1975 / 0.1045 / 0.13600.1576 / 0.1134 / 0.12300.1716 / 0.1295 / 0.1314
0.2470 / 0.1200 / 0.12700.2303 / 0.1277 / 0.12500.2474 / 0.1412 / 0.1395
0.3113 / 0.1657 / 0.11800.3356 / 0.1794 / 0.13300.3353 / 0.2078 / 0.1421

These experiments show that the KL-only model has an index selection time comparable to that of Top-p, but takes longer to compute sparse attention because it selects more tokens. Meanwhile, top-p selection incurs noticeably greater overhead than top-k selection. We believe this is because current mainstream sparse attention methods all use top-k retrieval, so the official vLLM implementation has relatively mature optimizations for top-k operators. For example, vLLM uses different top-k operators for prefill and decode, whereas top-p uses our own implementation, resulting in a performance gap. As research on top-p sparse attention grows, more high-performance top-p operators have emerged, such as those in Twilight and RaBitQCache. We believe that further improving the efficiency of top-p selection is also a worthwhile direction for future research.

Q3

To address your concern about the lack of a simple baseline that fits a global or per-layer score temperature, we trained a per-layer temperature calibration baseline on DeepSeek-V2-Lite. It learns a scalar parameter τ\tau for each layer, initialized to 1. To facilitate comparison, we keep the backbone frozen. The indexer is optimized using the original row-weighted KL divergence objective, while the temperature parameters are optimized only by the same budget loss LbudL_{bud}, with the stop-gradient log probabilities divided by the temperature during computation. Specifically, the original indexer scores are

sij=∑haih ReLU(qih⊤kj),ai=Wweightsxi.s_{ij}=\sum_h a_{ih}\,\mathrm{ReLU}(q_{ih}^{\top}k_j), a_i=W_{\mathrm{weights}}x_i .

We additionally apply the following to each layer:

Wweights′=WweightsτW_{\mathrm{weights}}'=\frac{W_{\mathrm{weights}}}{\tau}

This method also starts from the same KL-only checkpoint after 900 updates and performs 600 additional Stage 1 updates, using the same training data and budget loss configuration. We evaluate both methods using the same top-pp threshold and LongBench v1 evaluation protocol. The results are as follows:

SubtaskTemperature baseline scoreCaliTopP (S1) scoreTemperature baseline tokensOur tokensOur token reduction
NarrativeQA21.4321.75401636269.72%
Qasper33.7734.83146714094.00%
MultiFieldQA-en46.7947.19209719298.03%
HotpotQA26.3126.763332276317.09%
2WikiMQA19.6718.982066176314.63%
MuSiQue12.3212.063790306819.07%
GovReport33.0232.68274025706.21%
QMSum23.4723.60344131757.72%
MultiNews25.7725.838768305.19%
TriviaQA84.5385.172654202823.59%
Passage Retrieval-en76.5076.503198248022.47%
LCC63.3963.82108810196.33%
RepoBench-P65.6365.272781244412.12%
Average40.9741.112580223913.25%

Our method achieves a higher score while reducing the number of retained tokens on every task, with an average reduction of 13.25% compared with this baseline. This demonstrates that assigning a shared temperature coefficient to each layer cannot achieve the budget efficiency obtained by applying LbudL_{bud} directly to the indexer weights.

Q4

Regarding your question about whether the gains are consistent across different models and tasks, the experimental results in the paper indicate that they are not. Budget increases and performance improvements vary by model and task category. For example, on LongBench v1, the average budget is 2442 for DeepSeek-V2-Lite and 3110 for Llama-3.1-8B. Differences also occur across tasks: for DeepSeek-V3.2 at a context length of 32K, the median budget required to cover 90% of the teacher's attention mass is 514 tokens for multi-key retrieval and 868 tokens for question answering. The token counts required by the same layer can differ by a median factor of 5 across task categories. These differences also demonstrate that a single fixed kk cannot efficiently accommodate the diverse requirements of different models, tasks, and inputs. Our proposed top-pp method dynamically adjusts the number of selected tokens based on the estimated attention distribution for each query, allocating more tokens where needed and fewer where attention is concentrated, with no noticeable degradation in model capability across all scenarios.

Reviewer_KMBWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CaliTopP, a trainable top-p sparse attention mechanism. The paper first empirically and theoretically shows that existing methods that only use KL-distilled indexer perform not well because they can underestimate the budget mass and thus lead to over-selection. Then it proposes the new design, adding a budget-calibration loss to the KL-based indexer. The results demonstrate some improvements in terms of accuracy and speedup over the baselines.

Critical strengths
  • Good observation on the limitation of KL indexer.
  • Both theoretical and empirical results for the motivation.
  • The evaluation is relatively comprehensive with multiple models and datasets.
Critical weaknesses
  • The paper only compares with training-free top-k baselines while ignoring the more relevant training-free top-p works.
  • The speedup results at long contexts need more explanation and analysis.
Questions for authors
  1. It would be great if the authors can further compare with some training-free top-p designs in the evaluation, such as Twilight. The evaluation currently includes many top-k baselines, which is good, but misses the more relevant top-p design. I believe the benefit will preserve as CaliTopP accelerates both prefill and decoding while training-free top-p only improves decoding.

  2. In Fig 3, CaliTopP fails to preserve speedups at long contexts. This result needs more explanation. The main target scenario of top-p is at long contexts, so this performance result is critical. Are you suggesting that the baseline top-k is faster but with worse accuracy? If so please show the accuracy-speedup tradeoff clearly, or using larger k values. On the other hand, maybe using a top-p baseline (as requested above) will automatically adjust the k value and thus achieve a more explainable outcome for this speedup experiment.

  3. What is the final selected value of lambda? Fig 4 seems to show that larger lambda is always better, but it was supposed to balance the two losses and thus a proper value needs to be decided.

  4. I appreciate the theoretical proofs in Appendix A. But I think some more explanation of the quantitative propositions (e.g., proposition 1) and linking the theorems to the qualitative description would be helpful for understanding.

6 comments

We sincerely thank the reviewer for the detailed and constructive comments. We hope that the responses below address your concerns.

W1 & Q1

Regarding your concern about the lack of comparisons with training-free top-p methods, we would first like to clarify that the baselines already included in our paper are not limited to top-k methods. We also compare against RaBitQCache (ICML 26), a recent training-free top-p baseline. We have also added experiments with Twilight using Llama-3.1-8B on LongBench v1. We use the best-performing configuration in the Twilight paper, DS+Twilight (K=8192, p=0.95). The results are as follows:

SubtaskTwilight scoreTwilight tokensCaliTopP scoreCaliTopP tokens
NarrativeQA29.77547255172
Qasper46294939.71887
MultiFieldQA-en52.5368753.92559
HotpotQA53.5542757.94368
2WikiMQA45.7356146.62522
MuSiQue27.9621431.45224
GovReport34.2457836.23432
QMSum25.2546725.44188
MultiNews27.2162728.2983
TriviaQA90.8364091.83530
Passage Retrieval-en99.5372299.54094
LCC40.5165172.71189
RepoBench-P48.7422470.33476
Average47.8394852.193110

The results show that our method outperforms both training-free top-p baselines, RaBitQCache and Twilight. We believe this is because a trainable method allows the model to adapt to this form of sparse inference during training. Compared with training-free methods, another advantage, as the reviewer noted, is that it can accelerate both the prefill and decode stages. This is also one of our motivations for exploring a trainable top-p method in this paper.

W2&Q2

Regarding your concern that our method does not maintain its speedups in long contexts, your understanding is correct: this is a trade-off made to preserve model capabilities. To verify this, we used DSA to train a Qwen3.8-27B with a top-k indexer and conducted experiments on LongBench v2. All frontier models using DSA (DeepSeek-V3.2, GLM 5.3, and KIMI K3.1 pro) use a top-k value of 2048, so we also set k to 2048. To ensure consistency in model capabilities, we froze the parameters of our Stage 2 backbone and trained only the indexer parameters. The results are shown below:

CategoryOriginal top-ptop-k=2048Difference
Overall54.08%34.39%−19.68
Short61.67%40.00%−21.67
Medium49.30%30.23%−19.07
Long50.93%33.33%−17.59
Easy57.29%34.38%−22.92
Hard52.09%34.41%−17.68

We can see that a fixed budget of 2048 tokens leads to a significant performance drop in extremely long contexts. Although top-k achieves greater speedups than top-p in this case, we consider this cost unacceptable. This is also the motivation behind our top-p method: to dynamically adjust the budget according to the task and find the minimum number of tokens required.

Q3

We describe our training configuration in Appendix C, where the final value of λ\lambda is 0.4. We would also like to clarify that the ablation study in Figure 4 does not indicate that a larger λ\lambda is always better. Figure 4 shows that a larger λ\lambda reduces the token budget, but the budget ultimately only approaches the oracle count, because when it falls below the oracle count, LbudL_{bud} increases the number of tokens retrieved by the indexer. However, a very large λ\lambda also reduces model capabilities, because the LbudL_{bud} term only adjusts the number of tokens without accounting for their importance ranking, which requires the KL term. Accordingly, Figure 4 shows that model performance is lower with λ=1\lambda=1 than with λ=0.4\lambda=0.4. We therefore chose λ=0.4\lambda=0.4 after weighing efficiency against quality. We also took this into account in our Stage 1 setup: we first train using only KL for a period of time, allowing the indexer to accurately estimate the token importance ranking, and then gradually increase λ\lambda to constrain the number of tokens.

Q4

Thank you for this valuable suggestion. We have added further explanations of the propositions in the paper and connected them to the qualitative descriptions in the text. For example, Proposition 1 shows that even correct rankings and a small KL divergence do not guarantee a well-calibrated top-p budget, even when the global KL optimum is attained within a restricted indexer family. Intuitively, if the probability mass of important tokens is slightly below the threshold, the indexer must make up the missing probability mass using a large number of low-probability background tokens, thereby substantially increasing the budget. For example, in our finite theoretical construction, the teacher's probability mass for the first 512 tokens is 0.905, while the indexer assigns a probability mass of only 0.895. When p=0.9p=0.9, this small shortfall requires an additional 1536 background tokens, resulting in 2048 selected tokens, whereas the teacher selects only 507 tokens, even though the KL divergence is below 0.001. This also explains the description of a “flatter distribution” in Section 3: underestimated cumulative probability mass pushes the top-p boundary deeper into the background region. This motivates us to introduce LbudL_{bud} to explicitly supervise the selection budget in addition to the KL loss.

Reviewer_8Uw4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CaliTopP trains an attention indexer for adaptive top-p selection. The paper shows that a KL-trained indexer can rank tokens well yet underestimate their cumulative attention mass, causing top-p to select too many tokens. It adds a budget-calibration loss that matches the selected count to a teacher-derived target, followed by joint training with the backbone. Experiments cover four model architectures, with reported speedups of up to 5.36× for prefill and 2.05× for decoding on DeepSeek-V3.2.

Critical strengths
  1. The distinction between ranking and mass calibration is well motivated. Figure 1 separates the cost of ranking errors from probability miscalibration, and Appendix A gives a concrete construction where small KL error still allows large budget inflation. This makes the case for calibration more convincing than benchmark gains alone.
  2. The calibration loss gives measurable savings without adding indexer parameters. On frozen-backbone DeepSeek-V2-Lite, increasing its weight reduces retained keys by 16.3%, while LongBench scores remain similar: 40.90 versus 40.73. Figure 5 also compares against lowering the threshold in a KL-only indexer, rather than relying only on fixed-p comparisons.
  3. The evaluation includes useful controls. Dense models trained on the same data help separate sparse-attention effects from further training. The frozen-backbone DeepSeek-V3.2 experiment is particularly useful: it preserves LongBench performance while reducing the average budget from 2,048 to 1,180 keys.
Critical weaknesses
  1. A simpler calibration baseline is still missing. Appendix A.3 explains why a shared temperature cannot independently fix every query, but this does not establish how well global or per-layer temperature calibration would work in practice. Figure 5 tests threshold adjustment, which is useful but different from rescaling the logits. This comparison matters for judging how much the proposed training objective adds.

  2. Budget calibration beyond the training length is not directly evaluated. LongBench v2 provides accuracy results, but not retained-token counts, teacher-mass coverage, or budget inflation by length. Table 6 shows a 4.7-point gap from the dense twin on the long split. Without the budget measurements and actual processed lengths, it is hard to tell whether the method retains its efficiency as context grows. Sensitivity to the smoothing width in Equation (3) is also untested.

  3. The runtime results need more detail to establish the practical tradeoff. Figure 3 includes selection overhead, but inference batch size and output length are not specified. Appendix F also leaves out the sorting and cumulative-sum costs from its time analysis and describes padded indices whose storage depends on the largest selected budget. These details matter for variable-budget execution. The paper appropriately reports that CaliTopP becomes 17% slower than fixed top-k at 128K; the headline speedups are therefore relative to dense attention, not a general speed advantage over DSA.

Questions for authors
  1. How does CaliTopP compare with global or per-layer temperature calibration of the KL-only indexer, evaluated at matched quality or retained-token budgets?
  2. Could you report processed context lengths, retained keys, and teacher-mass coverage by length on LongBench v2? How sensitive are these results to the smoothing-width schedule?
  3. What batch size and output length were used for Figure 3? Could you separate scoring from sorting/cut overhead and report peak memory and performance at several batch sizes?
6 comments

We sincerely thank the reviewer for the detailed and constructive comments. We hope that the responses below address your concerns.

W1&Q1

To address your concern about the lack of a simple baseline using global or per-layer temperature calibration, we trained a per-layer temperature calibration baseline on DeepSeek-V2-Lite. It learns a scalar parameter τ\tau for each layer, initialized to 1. To facilitate comparison, we keep the backbone frozen. The indexer is optimized using the original KL divergence objective, while the temperature parameters are optimized only by the same budget loss LbudL_{bud}, with the log probabilities divided by the temperature during computation. Specifically, the original indexer scores are sij=∑haih ReLU(qih⊤kj),ai=Wweightsxi.s_{ij}=\sum_h a_{ih}\,\mathrm{ReLU}(q_{ih}^{\top}k_j), a_i=W_{\mathrm{weights}}x_i .

We additionally apply the following to each layer:

Wweights′=WweightsτW_{\mathrm{weights}}'=\frac{W_{\mathrm{weights}}}{\tau}

This method also starts from the same KL-only checkpoint after 900 updates and performs 600 additional Stage 1 updates, using the same training data and budget loss configuration. We evaluate both methods using the same top-pp threshold and LongBench v1 evaluation protocol. The results are as follows:

SubtaskTemperature baseline scoreCaliTopP (S1) scoreTemperature baseline tokensOur tokensOur token reduction
NarrativeQA21.4321.75401636269.72%
Qasper33.7734.83146714094.00%
MultiFieldQA-en46.7947.19209719298.03%
HotpotQA26.3126.763332276317.09%
2WikiMQA19.6718.982066176314.63%
MuSiQue12.3212.063790306819.07%
GovReport33.0232.68274025706.21%
QMSum23.4723.60344131757.72%
MultiNews25.7725.838768305.19%
TriviaQA84.5385.172654202823.59%
Passage Retrieval-en76.5076.503198248022.47%
LCC63.3963.82108810196.33%
RepoBench-P65.6365.272781244412.12%
Average40.9741.112580223913.25%

Our method achieves a higher score while reducing the number of retained tokens on every task, with an average reduction of 13.25% compared with this baseline. This demonstrates that assigning a shared temperature coefficient to each layer cannot achieve the budget efficiency obtained by applying LbudL_{bud} directly to the indexer weights.

W2&Q2

To address your question about budget calibration beyond the training length, we measured the retained keys and attention mass coverage of our trained Qwen3.8-27B model on LongBench v2. The results are as follows:

Oracle token countActually selectedAttention coverageAverage length (tokens)
Short2,427.412565.7193.41%37,669.19
Medium6,991.017980.9889.73%139,115.63
Long23,018.7928064.1889.41%579,845.43

The number of tokens actually selected remains at a relatively accurate ratio: even at an average length of 580K, the over-selection ratio is only 1.22×1.22\times. However, attention coverage decreases noticeably at lengths not seen during training, leading to lower model quality than within the training length. We believe this is because the sharpness of a given model's attention-score distribution does not change substantially with sequence length, allowing LbudL_{bud} to generalize well to lengths not seen during training. However, the best recall of important tokens can only be guaranteed within the training length. To support this view, we also repeated the experiment in Figure 1(b) using our trained DeepSeek-V3.2 model. The results are as follows:

LengthSelected middle-token count: before → afterOracle countMiddle-token ratio: before → afterOriginal top-p coverage: before → after
4K159 → 103.5991.605× → 1.193×91.28% → 89.08%
8K309.5 → 156173.51.611× → 1.061×91.43% → 89.02%
16K762 → 356422.51.610× → 0.957×91.46% → 88.95%
32K1540.5 → 7177811.782× → 1.038×91.87% → 89.41%
64K2442.5 → 1101.59801.966× → 1.129×92.25% → 89.73%
128K3350.5 → 15541382.51.937× → 1.178×92.00% → 89.82%

These results demonstrate that the above findings also hold across different models, and that recall becomes more accurate as the sequence length approaches the training length.

The smoothing width primarily affects training stability and is introduced to address the problem that integer-valued counts do not allow gradient propagation. In theory, a smaller value gives a closer approximation to the correct count. However, setting it too small makes the magnitude of each gradient update too small. We therefore use τ=0.02 to 0.005\tau=0.02 \text{ to } 0.005. With this setting, training is very stable and ultimately achieves the intended effect. We observe almost no oscillations during training and therefore consider the current choice of smoothing width reasonable.

W3&Q3

Thank you for your comments. We acknowledge that the original manuscript did not provide enough detail about the configuration of the efficiency experiments. In Figure 3, we use a batch size of 1, and each request generates exactly 64 tokens with ignore_eos=True, so generation does not terminate early due to EOS. We then calculate throughput. For each run, we first perform one complete warm-up, followed by four measurements, and report the arithmetic mean of the four throughput measurements. We have also updated the corresponding section of the paper to address the shortcomings in the analysis in Appendix F. We further break down the index selection overhead in Figure 3 into indexer score computation and token selection. The results are as follows:

Prefill

LengthTop-k normalized execution time with Full = 1 (indexer computation / top-k selection / sparse attention)Top-p normalized execution time with Full = 1 (indexer computation / top-p selection / sparse attention)
4K0.7226 / 0.0405 / 0.54370.6872 / 0.1579 / 0.1398
8K0.4586 / 0.0442 / 0.37150.4413 / 0.1027 / 0.0853
16K0.2736 / 0.0334 / 0.21340.2648 / 0.0694 / 0.0498
32K0.1728 / 0.0236 / 0.11270.1702 / 0.0544 / 0.0370
64K0.1180 / 0.0180 / 0.05670.1160 / 0.0507 / 0.0299
128K0.0889 / 0.0136 / 0.02730.0875 / 0.0546 / 0.0257

Decode

LengthTop-k normalized execution time with Full = 1 (indexer computation / top-k selection / sparse attention)Top-p normalized execution time with Full = 1 (indexer computation / top-p selection / sparse attention)
4K0.1664 / 0.0566 / 0.15000.0968 / 0.0572 / 0.1260
8K0.1640 / 0.0690 / 0.14800.1011 / 0.0739 / 0.1250
16K0.1708 / 0.0972 / 0.14100.1244 / 0.1006 / 0.1220
32K0.1975 / 0.1045 / 0.13600.1576 / 0.1134 / 0.1230
64K0.2470 / 0.1200 / 0.12700.2303 / 0.1277 / 0.1250
128K0.3113 / 0.1657 / 0.11800.3356 / 0.1794 / 0.1330

The results show that top-p selection indeed incurs greater overhead than top-k selection, and that the performance decline at 128K is also mainly due to this greater selection overhead. This is also because current mainstream sparse attention methods all use top-k retrieval, so the official vLLM implementation has relatively mature optimizations for top-k operators. For example, vLLM uses different operators for prefill and decode, whereas top-p uses our own implementation. As research on top-p sparse attention grows, more high-performance top-p operators have emerged, such as those in Twilight and RaBitQCache. We believe that further improving the efficiency of top-p selection is also a worthwhile direction for future research.

Comments on the paper

1 comment

Summary of Responses

We thank the reviewers for their valuable comments. Our revised paper and rebuttal provide additional experiments and clarifications supporting CaliTopP’s central contribution: explicitly supervising the token-selection budget within an existing indexer, rather than merely replacing top-kk with top-pp.

  • Calibration beyond a shared temperature. We added a controlled per-layer temperature-calibration baseline with matched initialization, training data, training budget, and a frozen backbone. On DeepSeek-V2-Lite, CaliTopP achieves higher aggregate LongBench v1 performance while reducing retained tokens by 13.25% on average across tasks. This supports the benefit of learning input-dependent calibration through the existing indexer weights rather than relying on a shared per-layer scale.

  • Direct measurements of budget calibration and length generalization. We repeated the DeepSeek-V3.2 context-length sweep after calibration and added the comparison in Figure 11. At 128K, the median middle-token budget ratio decreases from 1.94× to 1.18×, while teacher-mass coverage remains approximately 89.8%, close to the target p=0.9p=0.9. We additionally report processed lengths, retained keys, and teacher-mass coverage on LongBench v2. On unseen contexts averaging approximately 580K tokens, Qwen3.8-27B maintains a retained-to-oracle budget ratio of 1.22×, demonstrating that our token-budget calibration remains effective well beyond the training range.

  • Broader comparisons with training-free top-pp methods. In addition to the existing RaBitQCache baseline, we added Twilight on Llama-3.1-8B. CaliTopP achieves higher aggregate LongBench v1 performance while retaining fewer tokens. CaliTopP also supports sparse attention during both prefill and decoding.

  • A clearer quality–efficiency trade-off and runtime analysis. On the same frozen Qwen3.8-27B Stage-2 backbone, CaliTopP achieves higher LongBench v2 performance than a separately trained top-k=2048k=2048 indexer. These results demonstrate that the acceleration achieved by a small fixed token budget in long contexts comes at the cost of substantial performance degradation, whereas CaliTopP adapts to different context lengths using the same model parameters and selection threshold. We also provide scoring, selection, and attention breakdowns, batch-size measurements, and indexer-memory accounting, and revise Appendix F to include sorting and cumulative-sum costs. These additions help readers better understand the sources of CaliTopP’s speedups and the factors affecting its runtime.

  • Clearer theoretical intuition and training choices. We expanded the explanation of how a small cumulative-mass deficit can cause substantial budget inflation despite accurate rankings and small KL error. We also clarify the quality–efficiency rationale for λ=0.4\lambda=0.4 and the smoothing-width schedule from 0.020.02 to 0.0050.005.

We again thank the reviewers and the AC for their careful evaluation, time, and effort. We welcome any further questions and would be happy to provide additional clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.