- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Double-P is a sparse attention design for long-context LLMs, which uses top-p twice for both selecting and pruning, in contrast to just once in previous work only for pruning. More specifically, Double-P is built upon cluster-based sparse methods, and first identifies the top-p clusters by estimating attention masses between the query and each KV cluster. Then it further applies another top-p on the selected clusters to split them into an exact subset and an approximate subset only using the cluster centroids. Double-P is shown to outperform the baseline that directly uses top-p on top of a cluster-based method.
Critical strengths
- The key idea of using top-p also for selecting on top of cluster-based methods is promising.
- The paper has compared with SOTA baselines including traditional top-p and cluster + top-p.
- Writing is easy to follow.
Critical weaknesses
- The performance of the proposed design and the baselines would heavily depend on the parameter p, so it would be more convincing to show the overall accuracy-speedup tradeoff curves.
- It is unclear whether the double top-p construction can be used beyond cluster-based methods.
Questions for authors
-
The main baseline to compare with is RetroInfer+Twilight. The results show that Double-P has similar accuracy with RetroInfer+Twilight but higher speedup. From Fig 9 the speedup is mainly from the final attention stage, in which Double-P uses the centroids to approximate some attentions while RetroInfer+Twilight would use accurate attention for this part. But how large this part is would depend on the setting of parameter p. Double-P approximates this region. If RetroInfer+Twilight uses a smaller p, would its performance be better while not sacrificing the accuracy?
-
Continuing on the question of 1, the choices of parameter p would highly affect the accuracy and speedup of both the baselines and the proposed design. Could you show the accuracy-speedup tradeoff curves (i.e., using accuracy and speedup for the x and y axes, and plot points with different p values for each design as a curve) for them? Double-P and RetroInfer+Twilight would be necessary to include. It would be good if other baselines can also be shown.
-
If I understand correctly, the key reason that top-p can be used also for selecting in Double-P is that the cluster-based methods provides the unit of clusters as a cheap way to estimate attention scores. While recent works as RetroInfer and ClusterKV show this paradigm is promising, most sparse attention designs are not based on clustering. Can the idea of double top-p be generalized beyond cluster-based methods? For example, can it be built on Quest?
-
The results in Table 3 miss the most important baseline RetroInfer+Twilight. Can you add it?
3 comments
Q4 — Add RetroInfer+Twilight to Table 3
We have evaluated the missing accuracy evaluations for RetroInfer+Twilight, and will add these results to Table 3.
Adding RetroInfer+Twilight preserves our main conclusion: Double-P remains on par with the strongest sparse baselines in accuracy while providing lower decode latency. The added Qwen3-8B RULER result is close to Double-P’s reported number in Table 3. On Llama-3.1-8B LongBench, Double-P scores 39.06, compared with 38.590 for RetroInfer+Twilight.
In the 32K timing comparison in Q2, Double-P takes 19.421 ms/token versus 35.461 ms/token for RetroInfer+Twilight at P=0.95—a 1.83× decode speedup. Together, the accuracy results and latency measurements support the benefit of selecting by top-p at the group level, which avoids the token-level candidate scoring and pruning overhead discussed in Q1.
| Model | Method | RULER 16K | RULER 32K | RULER 64K | RULER mean | LongBench |
|---|---|---|---|---|---|---|
| Qwen3-8B | RetroInfer+Twilight | 91.95 | 90.70 | 79.19 | 87.28 | 33.40 |
| Qwen3-8B | Double-P | 91.02 | 90.36 | 79.26 | 86.88 | 35.78 |
| Llama-3.1-8B | RetroInfer+Twilight | 92.83 | 89.95 | 85.21 | 89.33 | 38.59 |
| Llama-3.1-8B | Double-P | 92.87 | 90.31 | 84.55 | 89.24 | 39.06 |
Thanks for the new experimental results added. I appreciate the authors' efforts for the rebuttal. Most of the results match the expectation. One remaining issue is that the work is heavily based on RetroInfer, making its contributions a bit limited. So I would like to keep the original score, as weak accept.
We thank the reviewer for recognizing the promise of applying top-p to cluster selection, our comparisons with state-of-the-art baselines, and the clarity of our writing. We also appreciate the constructive suggestions and thoughtful concerns, which has helped us clarify and strengthen our work. We address the reviewer’s concerns point by point below.
W1 / Q1 — Can a smaller RetroInfer+Twilight threshold close the gap?
Lowering RetroInfer+Twilight’s pruning threshold does not close the measured latency gap. It reduces the final attention workload, but the same 25% candidate-cluster budget must still be retrieved and scored before pruning. This cost remains even when very few tokens survive.
At 32K, we evaluate across all 13 RULER tasks:
We separate candidate retrieval/scoring, token pruning, and final attention in the performance comparison below.
Reducing P from 0.95 to 0.75 saves only 0.419 ms/token in gather + attention, while candidate retrieval/scoring and pruning still cost approximately 17.3 ms/token. The corresponding accuracy evaluation also shows a 1.686-point loss. The experiment supports applying top-p during group selection: reducing token work before retrieval and scoring addresses an overhead that a lower post-retrieval pruning threshold leaves in place.
W1 / Q2 — Accuracy–speedup curves over threshold settings
Several Double-P settings match full attention’s observed accuracy while reducing decode latency. We have completed the 32K threshold study with 100 inputs per task across all 13 RULER tasks, using the same accuracy cohort at every Double-P point and for full attention.
Threshold settings and latency at 32K.
Five of the six Double-P settings have observed RULER scores above the full-attention reference and decode speedups of 1.13–1.29×. More aggressive approximation has a measurable quality cost: reducing p2 from 0.70 to 0.50 at p1=0.95 lowers the score from 91.162 to 90.109. The sweep supports the two thresholds as practical controls for the accuracy–compute tradeoff.
W2 / Q3 — Can the construction extend beyond clustering, for example to Quest?
The page experiment demonstrates that Double-P’s two-threshold construction extends beyond clustering. We apply it to 16-token contiguous pages: the first threshold selects pages using estimated attention mass, and the second divides selected pages between exact-token attention and value-summary contributions. This uses Quest’s page organization with our mass estimator in place of its original page-bound ranking.
The table compares results at 32K on Llama-3.1-8B-Instruct. Full attention and both Double-P variants use the same 100 inputs per task across all 13 RULER tasks.
The page extension reaches 89.732, compared with the paper’s Quest score of 85.950 and Quest+Twilight score of 86.500. Its 17.520 ms/token decode latency gives a 1.26× speedup over full attention. These results establish a working page-based extension with strong accuracy and lower decode cost than full attention.
Clustering remains the more accurate grouping choice in this comparison, improving the score by 1.430 points on the same inputs. Similar keys yield more representative group summaries and better attention-mass estimates than contiguous pages.