- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
GroupIndex accelerates sparse-attention indexing by sharing one prefix scan across neighboring queries. It learns query aggregation and scoring projections to retrieve shared candidates, which the native indexer re-ranks for each query. It also selects group widths and candidate budgets per layer based on coverage and GPU latency. Experiments on DeepSeek-V4-Flash report near-baseline accuracy on three long-context benchmarks up to 512K tokens and up to 1.9× end-to-end prefill speedup at 1M tokens.
Critical strengths
-
Clear motivation and targeted design. The paper clearly identifies the limitations of mean-pooled query sharing and motivates a learned surrogate trained to cover the union of members’ selections. The connection between the analysis, method, and ablations makes the contribution easy to understand.
-
Meaningful practical acceleration. GroupIndex demonstrates a favorable accuracy–latency tradeoff on DeepSeek-V4-Flash, maintaining near-baseline accuracy across three long-context benchmarks up to 512K tokens. The reported end-to-end prefill speedup of up to 1.9× at 1M tokens shows that the indexing improvements translate into practical system gains.
Critical weaknesses
The conceptual novelty needs clearer positioning. The motivation that mean aggregation can suppress keys important to only a few queries is closely related to DefensiveKV’s defensive aggregation perspective. Quest also demonstrates training-free, query-aware candidate selection using min/max key summaries. Although GroupIndex addresses a distinct computational constraint through learned cross-query aggregation, the paper should discuss these connections and clarify its contribution beyond existing aggregation and candidate-selection principles.
Questions for authors
Could a Quest-inspired, training-free query aggregation using per-dimension minima and maxima achieve comparable accuracy at similar inference cost? This differs from the coordinate-wise max pooling evaluated in the paper. Such a comparison would help establish whether learned aggregation provides a meaningful advantage over a stronger training-free alternative.
4 comments
Response to Reviewer kXCz (2/3): a Quest-style bound over the group, built and measured
Q (a Quest-inspired, training-free aggregation at similar inference cost). This is a comparison worth having; we built it and measured it offline and end to end. At similar scoring cost a sizable gap to the learned surrogate remains, 20–36 points. Below are the numbers, the task results and the cause we find.
(1) We ported the bound you describe and looked for its strongest form in this setting.
- The rule: per head, the per-dimension minima and maxima of the group's g queries bound every member's score, , scored as . It is a strict upper bound on the union score . The box spans the group, not a page of keys, so g is the only size knob; keys keep their native chunking.
- Its cost is measured, not assumed: one GEMM of inner dimension 2d, so a Quest-style scan at width 2g costs what a one-scan rule spends at width g. At 512K the scoring stage runs 3.65× faster than DSA for Quest@8 and 3.66× for PIVOT-Refine g = 4, the pair our cost model calls equal; GroupIndex g = 16 reaches 8.80×.
- The four variants:
- Coordinates: before or after the Hadamard transform; it is orthogonal and self-inverse, so the bound holds either way and only its tightness differs.
- Head weights: the group mean, or the positive part of the per-head maximum, which makes it a strict bound.
- What we report: all four in the table below and the best of them in every comparison; its queries stay in fp32 while GroupIndex quantizes its own.
(2) At matched scoring cost it is 20–36 points behind (100 test ProLong documents, worst layer of the 256K–512K bucket, c = 2048, top-p with top-k in parentheses).
| Scoring rule | Cost/query | g = 4 | g = 8 | g = 16 | g = 32 | g = 64 |
|---|---|---|---|---|---|---|
| Quest-style, before Hadamard, mean w ★ | 2/g | 80.75 (75.77) | 70.17 (62.13) | 59.87 (51.55) | 51.06 (42.48) | 43.92 (31.44) |
| Quest-style, after Hadamard, mean w | 2/g | 79.21 (73.89) | 68.71 (58.35) | 58.77 (47.05) | 50.90 (38.50) | 44.78 (28.27) |
| Quest-style, before Hadamard, max⁺ w (strict) | 2/g | 54.06 (46.88) | 39.57 (32.19) | 27.34 (21.67) | 18.35 (15.12) | 12.91 (12.49) |
| Quest-style, after Hadamard, max⁺ w (strict) | 2/g | 54.36 (46.51) | 38.96 (27.12) | 26.04 (17.50) | 18.04 (12.36) | 13.48 (10.15) |
| Max pooling, hidden states | 1/g | 87.40 (84.98) | 78.33 (57.84) | 69.06 (46.88) | 63.76 (38.61) | 59.70 (38.97) |
| Mean pooling (PIVOT-Refine) | 1/g | 89.75 (87.79) | 82.44 (79.59) | 74.50 (73.63) | 70.09 (68.69) | 67.08 (62.46) |
| GroupIndex | 1/g | — | 94.46 (88.92) | 87.33 (80.74) | 80.38 (73.69) | 73.08 (66.88) |
★ The strongest Quest variant, best in every cell except g = 64 top-p, where the after-Hadamard row is 0.86 higher.
- At equal width: at twice the cost, even the best variant stays below PIVOT-Refine; the two strict variants, which keep the guarantee, sit far lower.
- At matched cost (its g against the others' g/2): it is below the best one-scan rule at that cost by 20–36 points of worst-layer top-p coverage and 26–42 of top-k coverage.
Tasks, the cause and the scope follow in the next comment.
Response to Reviewer kXCz (3/3): task accuracy, the cause of the gap, and what learning adds
Q, continued.
(3) On tasks RULER separates the arms, and the other two benchmarks agree in direction. RULER at c = 2048, 13 tasks, 100 prompts per task (50 at 1M); the rerun arms share one cluster and code version, and Quest uses one fp8 implementation for accuracy and speed.
| Method | Scan cost | LongBench-v2 (Overall) | LME-s | LME-m | RULER 256K | RULER 512K | RULER 1M | Indexer kernel, 512K / 1M |
|---|---|---|---|---|---|---|---|---|
| Native indexer (DSA), reference | 1 | 52.5 | 85.6 | 40.4 | 89.8 | 88.3 | 87.72 | 1.00× / 1.00× |
| IndexCache (Reuse, g = 4) | 6 of 21 layers | 49.9 | 41.0 | 88.2 | 86.6 | 83.69 | 3.50× / 3.50× | |
| IndexCache (Refine, g = 4) | 6 of 21 layers | 51.3 | 86.6 | 40.8 | 89.0 | 86.29 | 3.19× / 3.26× | |
| Quest-style, g = 8 | 1/4 | 53.5 | 84.8 | 38.0 | 85.34 | 83.44 | 75.03 | 3.65× / 3.89× |
| PIVOT-Refine, g = 4 | 1/4 | 53.3 | 85.2 | 87.8 | 86.82 | 3.66× / 3.72× | ||
| PIVOT-Refine, g = 8 | 1/8 | 53.9 | 85.4 | 86.5 | 83.0 | 79.98 | 6.75× / 7.18× | |
| GroupIndex, g = 16 | 1/16 | 87.8 | 41.8 | 89.6 | 88.2 | × / × | ||
| ↳ + layer spec. | per layer | 53.1 | 85.8 | 89.9 | 88.2 | 87.38 | 10.66× / 15.67× |
- RULER: at the same cost as PIVOT-Refine, the Quest-style arm scores 11.79 points below it at 1M and 12.29 below the faster GroupIndex.
- LongBench-v2 and LongMemEval: they do not separate the arms, but point the same way.
(4) Where the gap comes from: the bound favors large keys rather than wanted ones. To hold for every query in the group, it adds to each key a margin that grows with the key's magnitude, so the ranking tilts toward large keys.
(5) On your question of how far training-free aggregation goes and how much learning adds.
- Training-free: the strongest training-free rule we measured is mean pooling (PIVOT-Refine); the Quest-style bound is less accurate at the same cost and slower when given more budget.
- What learning adds:
- Coverage: at the same width and one scan, GroupIndex recovers 71–76% of the worst-layer top-p coverage that mean pooling leaves below the capacity ceiling (first comment's table, g = 8–32).
- Accuracy: on RULER at 512K and g = 16, mean pooling reaches 75.6 and GroupIndex 88.2; with or without layer specialization, GroupIndex stays on par with DSA at 512K / 1M.
- Speed: the scoring stage runs 8.80× / 11.23× faster at 512K / 1M, and 10.66× / 15.67× with layer specialization.
The revised Appendix E.3 extends Table 12 with these rows and a scoring-cost column, and adds the bound's definition, its kernel and this diagnosis.
We hope these answers resolve your questions and would be glad to clarify anything further.
Thanks for the response. I’m keeping my original score.
Response to Reviewer kXCz (1/3): what GroupIndex adds to DefensiveKV and Quest
Thank you for pointing us to DefensiveKV and Quest; §2 now discusses both. Below we set out what GroupIndex contributes beyond the existing aggregation and candidate-selection principles. Changes in the revised PDF are in blue.
W (positioning relative to DefensiveKV and Quest).
(1) We reach the same diagnosis as DefensiveKV, and the revision cites it at O4. Porting the principle here leaves the maximum only two places to sit: outside the ReLU, after scoring each member, or inside it, over the queries before one scoring pass. We built both.
Outside the ReLU, in score space: the union score of Eq. (4), maxj∑hwj,hReLU(qj,h⋅kn). Exact, but g full-prefix scans per group, so it cannot be deployed; we report it only as a reference.
Inside the ReLU, in query space: 1 scan, and deployable. There, however, the advantage of the maximum over the mean does not survive the approximation; the coordinate-wise max pooling evaluated in the paper is this branch.
This trade-off does not arise in either prior setting: eviction already has the per-query importance, and Quest summarizes the key side. Summarizing the query side instead, by the per-dimension minima and maxima you suggest, also falls in the second branch; we built four variants of it, reported next.
(2) This changes the target, not only its cost. Selection is two-stage: the shared candidates only bound the range, and each position re-ranks them with its own native query. So the surrogate never has to reproduce a score, only to rank the whole union ahead of every other key.
The plain maximum does not: it never puts the group's queries on the same footing, so a low-scoring member sees its own keys rank below keys no other member needs. At equal cost it gives up 3–7 points of top-p coverage.
A maximum taken after normalizing by each member's own threshold does: it puts the entire union ahead of every other key. It reaches the ceiling exactly wherever the budget holds the union, and comes within a fraction of a point otherwise. Belonging to the union is exactly the label GroupIndex is trained on (Appendix C.1).
Measured on two sets no selection step has seen: 100 ProLong documents disjoint from training (48 books, 48 code repos, 4 textbooks) and all 462 LongBench-v2 contexts. Worst layer in the 256K–512K bucket, c = 2048 as deployed, top-p with top-k in parentheses.
(3) What is ours, against the two principles you name.
Your question follows in the next comment.