acceptodds
Under review as a conference paper at ICLR 2027

GroupIndex: Accelerating Sparse Attention via Learned Cross-Query Indexing

Abstract

DeepSeek Sparse Attention (DSA) makes attention cheap by keeping only a few keys per query, but its lightweight indexer still scans the whole prefix for every query, and this scan comes to dominate long-context prefill. Existing accelerators reuse selections across layers, activate fewer indexer heads or filter key blocks. Sharing across layers and heads is capped by the fixed numbers of indexer layers and heads, and accuracy grows harder to keep as it nears that cap. Filtering key blocks instead trades the resolution of their summaries against recall. Sharing one scan across neighboring queries has no such fixed cap, since how many can share depends on their similarity and on context length. Existing query-group indexing, however, averages the queries of each group and loses accuracy as the group widens. We show that this width limit is representational rather than a matter of capacity. A shared scan must keep every key that any member selects, which calls for the maximum of the members' scores, whereas averaging their queries dilutes each member's own needs and loses keys that are important to individual members. We introduce GroupIndex, which jointly learns head-wise query aggregation and scoring projections in a Surrogate Indexer, and applies hardware-aware layer specialization to select each layer's query-group width and candidate budget. It indexes each query group with one scan whose surrogate query is learned rather than averaged. Training targets the group's selection union, uses the native indexer's own selections as labels, and runs offline on activations cached once, without loading the backbone. At inference the Surrogate Indexer runs as a sidecar ahead of the native indexer and proposes candidates for each group, among which the native indexer still makes each query's final selection. On DeepSeek-V4-Flash, GroupIndex matches the dense indexer's accuracy on RULER, LongBench-v2 and LongMemEval from 64K to 512K tokens, and at 1M tokens it speeds up the indexer about 11 and end-to-end prefill up to 1.9.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sparse attention, Efficient long-context inference, Learning the selector, Sharing across queries

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kXCzOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GroupIndex accelerates sparse-attention indexing by sharing one prefix scan across neighboring queries. It learns query aggregation and scoring projections to retrieve shared candidates, which the native indexer re-ranks for each query. It also selects group widths and candidate budgets per layer based on coverage and GPU latency. Experiments on DeepSeek-V4-Flash report near-baseline accuracy on three long-context benchmarks up to 512K tokens and up to 1.9× end-to-end prefill speedup at 1M tokens.

Critical strengths
  1. Clear motivation and targeted design. The paper clearly identifies the limitations of mean-pooled query sharing and motivates a learned surrogate trained to cover the union of members’ selections. The connection between the analysis, method, and ablations makes the contribution easy to understand.

  2. Meaningful practical acceleration. GroupIndex demonstrates a favorable accuracy–latency tradeoff on DeepSeek-V4-Flash, maintaining near-baseline accuracy across three long-context benchmarks up to 512K tokens. The reported end-to-end prefill speedup of up to 1.9× at 1M tokens shows that the indexing improvements translate into practical system gains.

Critical weaknesses

The conceptual novelty needs clearer positioning. The motivation that mean aggregation can suppress keys important to only a few queries is closely related to DefensiveKV’s defensive aggregation perspective. Quest also demonstrates training-free, query-aware candidate selection using min/max key summaries. Although GroupIndex addresses a distinct computational constraint through learned cross-query aggregation, the paper should discuss these connections and clarify its contribution beyond existing aggregation and candidate-selection principles.

Questions for authors

Could a Quest-inspired, training-free query aggregation using per-dimension minima and maxima achieve comparable accuracy at similar inference cost? This differs from the coordinate-wise max pooling evaluated in the paper. Such a comparison would help establish whether learned aggregation provides a meaningful advantage over a stronger training-free alternative.

4 comments
AuthorsOpenReview ↗

Response to Reviewer kXCz (1/3): what GroupIndex adds to DefensiveKV and Quest

Thank you for pointing us to DefensiveKV and Quest; §2 now discusses both. Below we set out what GroupIndex contributes beyond the existing aggregation and candidate-selection principles. Changes in the revised PDF are in blue.

W (positioning relative to DefensiveKV and Quest).

(1) We reach the same diagnosis as DefensiveKV, and the revision cites it at O4. Porting the principle here leaves the maximum only two places to sit: outside the ReLU, after scoring each member, or inside it, over the queries before one scoring pass. We built both.

  • Outside the ReLU, in score space: the union score of Eq. (4), max⁡j∑hwj,hReLU⁡(qj,h⋅kn)\max_j \sum_h w_{j,h}\operatorname{ReLU}(q_{j,h}\cdot k_n). Exact, but g full-prefix scans per group, so it cannot be deployed; we report it only as a reference.

  • Inside the ReLU, in query space: 1 scan, and deployable. There, however, the advantage of the maximum over the mean does not survive the approximation; the coordinate-wise max pooling evaluated in the paper is this branch.

    This trade-off does not arise in either prior setting: eviction already has the per-query importance, and Quest summarizes the key side. Summarizing the query side instead, by the per-dimension minima and maxima you suggest, also falls in the second branch; we built four variants of it, reported next.


(2) This changes the target, not only its cost. Selection is two-stage: the shared candidates only bound the range, and each position re-ranks them with its own native query. So the surrogate never has to reproduce a score, only to rank the whole union ahead of every other key.

  • The plain maximum does not: it never puts the group's queries on the same footing, so a low-scoring member sees its own keys rank below keys no other member needs. At equal cost it gives up 3–7 points of top-p coverage.

  • A maximum taken after normalizing by each member's own threshold does: it puts the entire union ahead of every other key. It reaches the ceiling exactly wherever the budget holds the union, and comes within a fraction of a point otherwise. Belonging to the union is exactly the label GroupIndex is trained on (Appendix C.1).

    Measured on two sets no selection step has seen: 100 ProLong documents disjoint from training (48 books, 48 code repos, 4 textbooks) and all 462 LongBench-v2 contexts. Worst layer in the 256K–512K bucket, c = 2048 as deployed, top-p with top-k in parentheses.

Scoring a group's keysProLong g = 8ProLong g = 16ProLong g = 32LB-v2 g = 8LB-v2 g = 16LB-v2 g = 32
Reference: an oracle, or g scans per group
Capacity ceiling (the most any c keys can cover)99.34 (96.03)91.39 (85.73)84.22 (78.34)99.01 (96.03)90.34 (85.65)82.49 (77.62)
Max of members' scores, Eq. (4)92.53 (86.49)84.96 (77.98)75.93 (69.50)91.54 (86.05)82.96 (77.56)73.44 (65.64)
Mean of members' scores85.36 (80.51)77.55 (74.44)72.22 (69.45)84.28 (80.49)75.88 (74.61)69.62 (65.90)
Max after per-member thresholds99.13 (95.69)89.42 (82.70)79.17 (72.25)98.79 (95.70)88.09 (82.50)77.30 (69.12)
Deployable: one scan per group
Mean pooling (PIVOT-Refine)82.44 (79.59)74.50 (73.63)70.09 (68.69)80.50 (79.53)72.73 (73.78)67.41 (63.99)
Max pooling, hidden states78.33 (57.84)69.06 (46.88)63.76 (38.61)76.34 (57.65)67.50 (46.60)60.97 (56.77)
GroupIndex (learned)94.46 (88.92)87.33 (80.74)80.38 (73.69)93.21 (88.56)85.24 (80.41)77.47 (71.94)

(3) What is ours, against the two principles you name.

  • Beyond aggregation: the principle combines the members' scores once they exist; we combine before scoring. Two things follow:
    • Cost: one scan per group, not g.
    • Quality: an average leaves a direction only one member wants at 1/g, and reweighting cannot repair that, since averaging commutes with the projections. So the head-wise aggregation and the scoring projections are trained together.
  • Beyond candidate selection: we are two-stage as well, shared candidates then per-query rescoring; the difference is where the candidates come from.
    • Theirs is a guarantee: the bound must hold for every member, a worst-case constraint paid for with a box enclosing the group's queries: the wider the group, the bigger the box, until the order follows the overshoot, not importance.
    • Ours gives it up: a ranker carries no worst case; it only has to get the final order right. The same budget goes into ranking rather than into covering every member: at equal cost the bound sits 20–36 points of worst-layer coverage below the best one-scan rule.

Your question follows in the next comment.

AuthorsOpenReview ↗

Response to Reviewer kXCz (2/3): a Quest-style bound over the group, built and measured

Q (a Quest-inspired, training-free aggregation at similar inference cost). This is a comparison worth having; we built it and measured it offline and end to end. At similar scoring cost a sizable gap to the learned surrogate remains, 20–36 points. Below are the numbers, the task results and the cause we find.

(1) We ported the bound you describe and looked for its strongest form in this setting.

  • The rule: per head, the per-dimension minima and maxima of the group's g queries bound every member's score, UBh(n)=qmax⁡,h⋅kn++qmin⁡,h⋅kn−≥max⁡jqj,h⋅knUB_h(n) = q_{\max,h}\cdot k_n^{+} + q_{\min,h}\cdot k_n^{-} \ge \max_j q_{j,h}\cdot k_n, scored as ∑hwˉhReLU⁡(UBh(n))\sum_h \bar w_h \operatorname{ReLU}(UB_h(n)). It is a strict upper bound on the union score max⁡j∑hwj,hReLU⁡(qj,h⋅kn)\max_j \sum_h w_{j,h}\operatorname{ReLU}(q_{j,h}\cdot k_n). The box spans the group, not a page of keys, so g is the only size knob; keys keep their native chunking.
  • Its cost is measured, not assumed: one GEMM of inner dimension 2d, so a Quest-style scan at width 2g costs what a one-scan rule spends at width g. At 512K the scoring stage runs 3.65× faster than DSA for Quest@8 and 3.66× for PIVOT-Refine g = 4, the pair our cost model calls equal; GroupIndex g = 16 reaches 8.80×.
  • The four variants:
    • Coordinates: before or after the Hadamard transform; it is orthogonal and self-inverse, so the bound holds either way and only its tightness differs.
    • Head weights: the group mean, or the positive part of the per-head maximum, which makes it a strict bound.
    • What we report: all four in the table below and the best of them in every comparison; its queries stay in fp32 while GroupIndex quantizes its own.

(2) At matched scoring cost it is 20–36 points behind (100 test ProLong documents, worst layer of the 256K–512K bucket, c = 2048, top-p with top-k in parentheses).

Scoring ruleCost/queryg = 4g = 8g = 16g = 32g = 64
Quest-style, before Hadamard, mean w ★2/g80.75 (75.77)70.17 (62.13)59.87 (51.55)51.06 (42.48)43.92 (31.44)
Quest-style, after Hadamard, mean w2/g79.21 (73.89)68.71 (58.35)58.77 (47.05)50.90 (38.50)44.78 (28.27)
Quest-style, before Hadamard, max⁺ w (strict)2/g54.06 (46.88)39.57 (32.19)27.34 (21.67)18.35 (15.12)12.91 (12.49)
Quest-style, after Hadamard, max⁺ w (strict)2/g54.36 (46.51)38.96 (27.12)26.04 (17.50)18.04 (12.36)13.48 (10.15)
Max pooling, hidden states1/g87.40 (84.98)78.33 (57.84)69.06 (46.88)63.76 (38.61)59.70 (38.97)
Mean pooling (PIVOT-Refine)1/g89.75 (87.79)82.44 (79.59)74.50 (73.63)70.09 (68.69)67.08 (62.46)
GroupIndex1/g—94.46 (88.92)87.33 (80.74)80.38 (73.69)73.08 (66.88)

★ The strongest Quest variant, best in every cell except g = 64 top-p, where the after-Hadamard row is 0.86 higher.

  • At equal width: at twice the cost, even the best variant stays below PIVOT-Refine; the two strict variants, which keep the guarantee, sit far lower.
  • At matched cost (its g against the others' g/2): it is below the best one-scan rule at that cost by 20–36 points of worst-layer top-p coverage and 26–42 of top-k coverage.

Tasks, the cause and the scope follow in the next comment.

AuthorsOpenReview ↗

Response to Reviewer kXCz (3/3): task accuracy, the cause of the gap, and what learning adds

Q, continued.

(3) On tasks RULER separates the arms, and the other two benchmarks agree in direction. RULER at c = 2048, 13 tasks, 100 prompts per task (50 at 1M); the rerun arms share one cluster and code version, and Quest uses one fp8 implementation for accuracy and speed.

MethodScan costLongBench-v2 (Overall)LME-sLME-mRULER 256KRULER 512KRULER 1MIndexer kernel, 512K / 1M
Native indexer (DSA), reference152.585.640.489.888.387.721.00× / 1.00×
IndexCache (Reuse, g = 4)6 of 21 layers49.987.0‾\underline{87.0}41.088.286.683.693.50× / 3.50×
IndexCache (Refine, g = 4)6 of 21 layers51.386.640.889.087.9‾\underline{87.9}86.293.19× / 3.26×
Quest-style, g = 81/453.584.838.085.3483.4475.033.65× / 3.89×
PIVOT-Refine, g = 41/453.385.241.4‾\underline{41.4}89.7‾\underline{89.7}87.886.823.66× / 3.72×
PIVOT-Refine, g = 81/853.985.441.4‾\underline{41.4}86.583.079.986.75× / 7.18×
GroupIndex, g = 161/1653.7‾\underline{53.7}87.841.889.688.287.32‾\underline{87.32}8.80‾\underline{8.80}× / 11.23‾\underline{11.23}×
↳ + layer spec.per layer53.185.841.4‾\underline{41.4}89.988.287.3810.66× / 15.67×
  • RULER: at the same cost as PIVOT-Refine, the Quest-style arm scores 11.79 points below it at 1M and 12.29 below the faster GroupIndex.
  • LongBench-v2 and LongMemEval: they do not separate the arms, but point the same way.

(4) Where the gap comes from: the bound favors large keys rather than wanted ones. To hold for every query in the group, it adds to each key a margin that grows with the key's magnitude, so the ranking tilts toward large keys.


(5) On your question of how far training-free aggregation goes and how much learning adds.

  • Training-free: the strongest training-free rule we measured is mean pooling (PIVOT-Refine); the Quest-style bound is less accurate at the same cost and slower when given more budget.
  • What learning adds:
    • Coverage: at the same width and one scan, GroupIndex recovers 71–76% of the worst-layer top-p coverage that mean pooling leaves below the capacity ceiling (first comment's table, g = 8–32).
    • Accuracy: on RULER at 512K and g = 16, mean pooling reaches 75.6 and GroupIndex 88.2; with or without layer specialization, GroupIndex stays on par with DSA at 512K / 1M.
    • Speed: the scoring stage runs 8.80× / 11.23× faster at 512K / 1M, and 10.66× / 15.67× with layer specialization.

The revised Appendix E.3 extends Table 12 with these rows and a scoring-cost column, and adds the bound's definition, its kernel and this diagnosis.


We hope these answers resolve your questions and would be glad to clarify anything further.


Thanks for the response. I’m keeping my original score.

Reviewer_iNDUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GroupIndex reduces sparse-attention indexing cost by learning a shared representation for neighboring queries. One scan retrieves group-level candidates, followed by native per-query reranking. The backbone stays frozen while the surrogate indexer and layer configurations are optimized. On DeepSeek-V4-Flash, it preserves near-native quality and reports approximately 1.9x end-to-end prefill speedup at 1M context.

Critical strengths
  1. Learned query aggregation addresses a clear limitation of PIVOT-style mean pooling, enabling larger sharing groups with less quality degradation.
  2. Detailed component ablations and whole-prefill measurements provide a convincing account of the design and its practical acceleration.
Critical weaknesses
  1. Pooling the entire query group may let later queries affect earlier selections, even when keys are restricted to the past. Please test prefix invariance and chunked-prefill consistency, and clarify the intended causal semantics. This concern does not itself establish answer leakage.
  2. Coverage measured on cached native activations may change when GroupIndex is enabled throughout the model. Please separate calibration from final testing and measure coverage on actual deployment activations.
  3. Quality is evaluated only up to 512K, while the strongest speedup is at 1M. Please validate quality at 1M; testing another checkpoint or domain would also strengthen evidence beyond the single-backbone setting.
Questions for authors

Please check the weakness above.

6 comments
AuthorsOpenReview ↗

Response to Reviewer iNDU (1/3): causal semantics, measured in the serving engine

Thank you for three specific and actionable requests. We ran every check they ask for and answer one weakness per comment; changes in the revised PDF are marked in blue.

W1 (causal semantics, prefix invariance, chunked prefill). Thank you for separating this from answer leakage.

(1) Short answer: we agree that key visibility does not imply prefix consistency. GroupIndex satisfies the former strictly: no position ever attends to a future key. For the latter, Eq. (5) aggregates the queries of the whole group, so later tokens in j's group can change j's shared candidates; both the reach and the size of this influence are bounded (items 2–4 below). The revised Appendix B.2 now treats the two properties separately.


(2) The intended semantics is block-level prefix consistency. Every G consecutive tokens form a block, where G is the largest group width. Only later tokens of j's own block can affect j, and nothing after that block can.

Items 3–5 are measured in the serving engine (new Appendix B.3): 20 documents at each length, books and code repositories, disjoint from both the training and the validation documents and hence from checkpoint and layer-plan selection (the independent test set of W2), six lengths from 32K to beyond 512K, 960 cuts of each kind, both plans (g = 16, c = 2048; τ = 96%). The engine varies more between runs than the effect under test, for the native indexer too, so every comparison is made within one prefill pass.

(3) Prefix invariance holds. Noise floor: recomputing with identical inputs already flips a few selections (the top-k kernel breaks ties by scan order). Replacing every token after a block boundary leaves the rate at that floor.

Changes per million (position, layer) pairsg = 16, c = 2048Layer spec. (τ = 96%)
After the boundary replaced3.9 (298 of 75.6M)85 (5,545 of 65.1M)
Identical inputs (noise floor)3.9 (298)84 (5,460)

(4) Within a block, the influence is also limited.

  • Path: the influence passes only through the shared candidate set. j's final keys are still chosen by its own native query, so whenever the candidates contain j's native top-k, j's selection is exactly native.
  • Magnitude: later tokens can change only the part of j's native top-k that the candidates miss, and this part is small and carries little weight. Cutting inside a group and permuting everything after the cut, harsher than any real edit, changes 2.0–6.8% of an earlier member's keys at g = 16, but only 0.55–1.9% of their native indexer mass: by mass the change is about a third of what it is by count, so the keys that move come mostly from the long tail (ranges over the six lengths).
Earlier members of the cut's groupg = 16, c = 2048Layer spec. (τ = 96%)
Keys that change2.0–6.8%0.2–3.3%
Their share of the indexer mass0.55–1.9%0.07–0.93%
Top-p coverage drop0.4–1.0 pt≤ 0.3 pt

(5) Chunked prefill changes neither the grouping nor the selections. Groups are aligned to absolute positions, and SGLang ends chunks on KV-cache page boundaries (every 256 tokens, a multiple of G), so no group is ever split; vLLM implements the same chunking for hybrid models, which can be enabled for GroupIndex. Measured: at chunk sizes 4K / 6K / 8K the grouping is identical (0 mismatches in 46.1M (window, layer) pairs); re-splitting a chunk within one pass, with the activations unchanged, leaves selections at the noise floor of item 3.


(6) Cached prefixes are reused exactly. Both engines reuse only whole pages, so a reused prefix never ends inside a block. By item 3, a prefix cached during prefill (e.g., a shared system prompt or document) then equals a fresh prefill of it and carries nothing from the request that produced it.


W2 and W3 follow in the next two comments.

AuthorsOpenReview ↗

Response to Reviewer iNDU (2/3): independent test documents and deployment activations

W2 (calibration vs. final testing; coverage on deployment activations).

(1) Short answer: we now report coverage on documents no selection step has seen, and on activations produced with GroupIndex running in all 21 layers. Both are new experiments; the downstream benchmarks were already independent of all selection steps.


(2) Calibration and final testing are now separate. In the submission, checkpoint selection, layer-plan selection and every reported coverage used the same validation documents, and Appendix D.2's "held-out" referred to training only; we have corrected that wording.

DataDocumentsWindowsUsed for
Training900 per widthall (about 0.5M per width)training
Validation100 per width, one set600 (100 per length bucket); 96 (16 per bucket) for plan selectioncheckpoint selection, all coverage in the submission, layer-plan selection
Test (new)100 ProLong, disjoint (48 books, 48 code repos, 4 textbooks), + all 462 LongBench-v2 contexts (6 domains, 20 subdomains, 10K–4.6M tokens)600 each (100 per bucket)independent evaluation only

(3) Table 2's averages transfer (new Appendix E.7). Re-running on the validation documents reproduces every Table 2 entry exactly. On the test ProLong documents all eight configurations stay within 0.5 points (top-p) and 0.8 (top-k) of their validation values, and none of the differences is significant. Outside the training domains the top-p drop stays within 1.5 points.

Coverage: top-p (top-k)Validation = Table 2Test ProLongLongBench-v2, training domainsLongBench-v2, other domains (QA, ICL, tables, dialogue)
g = 8, c = 204898.53 (96.78)98.39 (96.50)98.46 (96.67)98.13 (96.09)
g = 16, c = 204896.74 (93.18)96.66 (93.09)96.65 (93.09)96.21 (92.32)
g = 32, c = 204895.00 (89.98)94.69 (89.48)94.66 (89.44)93.91 (88.17)
g = 64, c = 204892.78 (86.73)92.29 (85.99)92.09 (85.66)91.32 (84.47)

(4) The plan's worst layer on independent documents. Per-layer coverage as a long-context average (each layer averaged over the four length buckets above 65K, then the lowest of the 21 layers): the worst layer is 96.6% on test ProLong (95% CI 96.1–97.0, document bootstrap) and 94.4% outside the training domains.


(5) Enabling GroupIndex in all 21 layers moves coverage by less than a tenth of a point. On the same 100 test documents and the same windows, we run each document twice in one build. In both runs GroupIndex computes its selection and we score it against the native indexer's top-512 recomputed on that run's activations; the runs differ only in which selection drives attention, and therefore in the activations the later layers see. Brackets are 95% confidence intervals, bootstrapped over documents.

Top-p coverage (%), 100 test documentsResult
Forward pass uses the native selections (native activations)98.30 [98.08, 98.51]
GroupIndex drives attention in all 21 layers (deployment activations)98.22 [97.99, 98.45]
Difference−0.08 points [−0.10, −0.06]
Largest difference at any single layer0.25 points
Difference at the worst layer (long-context average, as above)0.15 points

The gap does not grow with depth (slope −0.003 points per layer). The task results in Table 1 were always measured with GroupIndex running in all 21 layers.


W3 follows in the next comment.

AuthorsOpenReview ↗

Response to Reviewer iNDU (3/3): accuracy at 1M, outside the training domains, and a second training

W3 (quality at 1M; another checkpoint or domain).

(1) At 1M, where the speedup is largest, GroupIndex stays within noise of DSA while the indexer stage runs nearly 16× faster: the most accurate and the fastest of all accelerated methods.

RULER at 1M (13 tasks, 50 prompts each)ScoreIndexer speedup at 1M (H20)
DSA87.721.00×
IndexCache (Reuse, g = 4)83.693.50×
IndexCache (Refine, g = 4)86.293.26×
Quest-style bound (g = 8)75.033.89×
PIVOT-Refine (g = 4)86.823.72×
PIVOT-Refine (g = 8)79.987.18×
GroupIndex (g = 16)87.3211.23×
↳ + layer spec.87.3815.67×

(2) Outside the training domains, GroupIndex does not degrade. The surrogate is trained only on books and code repositories; most LongBench-v2 documents are not.

LongBench-v2QuestionsDSAGroupIndexTop-p coverage, g = 16, c = 2048
Training domains (code repos, novels)12251.651.696.65
Other domains (multi-doc QA, reports, ICL, dialogue, tables)38152.854.396.21

(3) Another checkpoint (same backbone): a second training changes nothing. We trained the g = 16 surrogate twice from scratch, independently.

g = 16, c = 2048, two independent trainingsLongBench-v2 (all / S / M / L)RULER 512KRULER 1M
Training A53.7 / 56.7 / 49.8 / 56.588.1787.32
Training B53.7 / 56.1 / 49.3 / 58.388.1387.23

The two trainings also agree on 481 of 503 LongBench-v2 questions, and both RULER gaps (0.04 and 0.09) are the size of re-running a single checkpoint (0.11).


We hope these results resolve your three concerns. If they do, we would be grateful if you would consider updating your score; if anything still stands in the way, please tell us and we will run further checks before the discussion closes.


Reviewer_aQwKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GroupIndex targets the repeated full-prefix scans performed by the indexer during sparse-attention prefill. It jointly learns head-wise aggregation of neighboring queries and scoring projections, using the union of the native indexer's top-k selections as supervision. The frozen native indexer then re-ranks the proposed candidates separately for each query. Hardware-aware layer specialization selects each layer's group width and candidate budget using profiled latency and coverage constraints. On DeepSeek-V4-Flash, the method retains near-baseline accuracy on RULER from 64K to 512K tokens and on LongBench-v2 and LongMemEval within a 512K-token input budget. At 1M tokens, the g=16 configuration reports an 11.2x scoring-stage speedup, while the layer-specialized configuration achieves a 1.92x end-to-end prefill speedup (Figure 5).

Critical strengths
  1. A well-motivated learned representation supported by controlled comparisons. Separating candidate-budget loss from the surrogate gap explains why mean-pooled query sharing degrades as groups widen. Union supervision directly targets candidate recall while retaining native per-query refinement. At g=16 and c=2048, Table 2 reports a 512K RULER score of 88.2 for GroupIndex versus 75.6 for PIVOT-Refine, close to the native baseline's 88.3 in Table 1. Initialization from the mean-pooling baseline and the aggregation-only and projection-only ablations in Table 3 further support joint training, particularly in deeper layers. These comparisons attribute the accuracy improvement to the learned representation at a fixed candidate budget.

  2. Practical acceleration supported by kernel and end-to-end measurements. The added surrogate operates with a frozen backbone and retains the native attention interface. Selecting group widths and budgets from measured hardware latency exploits differences in sharing tolerance across layers. Table 1 reports a 10.66x scoring-stage speedup at 512K for the layer-specialized configuration, with a RULER score of 88.2 versus 88.3 for the baseline. Its 1.92x end-to-end prefill speedup at 1M shows that indexing savings translate into overall prefill gains. Appendix E.2 also reports smaller gains at shorter contexts, helping define the method's practical scope.

Critical weaknesses
  1. The causality claim needs to distinguish key visibility from prefix consistency. Appendix B.2 restricts shared candidates to keys visible to all group members and adds a near window cut at each query's causal limit. However, Equation (5) aggregates representations from the entire query group. As written, this permits later positions to influence which historical keys are proposed for an earlier position; masking future keys does not rule out that dependency. The formulation therefore leaves strict prefix consistency unproven. It does not by itself invalidate generation after a complete prompt, but the paper should clarify whether earlier representations can change with later tokens or chunk boundaries, and what this implies for prefix reuse.

  2. Independent coverage validation and task accuracy at 1M remain unclear. Algorithm 1 and Table 9 select checkpoints using 600 validation windows, and Table 15 explicitly reports coverage on validation windows. Appendix D.2's description of 600 held-out windows therefore does not establish a separate test after checkpoint selection. It also states that Table 4's coverage floors use 16 windows per length bucket. The sample sizes for selecting and independently evaluating the main layer-specialized plan should be stated explicitly. Downstream benchmarks provide separate evidence, but task accuracy stops at 512K while the largest measured latency gains occur at 1M. Results on one backbone and H20 hardware establish a useful case study, while leaving quality preservation at 1M and broader portability unverified.

Questions for authors
  1. With the prefix through position j fixed, can changing later tokens in its query group alter j's candidate set or output? Please clarify the guarantees for complete-prompt prefill, changes in chunk boundaries and reuse of cached prefixes.

  2. Which coverage results, if any, use documents excluded from checkpoint and layer-plan selection? Please specify the splits and sample sizes used to select and independently evaluate the main layer-specialized plan, including whether its worst-layer coverage meets the intended floor on those independent documents.

  3. What task accuracy does the exact layer-specialized configuration timed at 1M achieve at that length relative to the native baseline? This would connect the largest measured prefill gain to the quality-preservation claim.

3 comments
AuthorsOpenReview ↗

Response to Reviewer aQwK (1/2): causal semantics, measured in the serving engine

Thank you for the careful review. We answer your three questions in order; changes in the revised PDF are marked in blue.

W1 / Q1 (causal semantics). Thank you for separating the two notions.

(1) Short answer: we agree that key visibility does not imply prefix consistency. GroupIndex satisfies the former strictly: no position ever attends to a future key. For the latter, Eq. (5) aggregates the queries of the whole group, so later tokens in j's group can change j's candidate set; both the reach and the size of this influence are bounded (items 2 and 3 below). The revised Appendix B.2 now treats the two properties separately.

Items 2 and 3 are measured in the serving engine (new Appendix B.3), and item 4 follows from them: 20 documents at each length, books and code repositories, disjoint from both the training and the validation documents and hence from checkpoint and layer-plan selection (the independent test set of Q2), six lengths from 32K to beyond 512K, 960 cuts of each kind, both plans (g = 16, c = 2048; τ = 96%). The engine varies more between runs than the effect under test, for the native indexer too, so every comparison is made within one prefill pass.

(2) What holds: block-level prefix consistency. Every G consecutive tokens form a block, where G is the largest group width. Only later tokens of j's own block can affect j, and nothing after that block can. Noise floor: recomputing with identical inputs already flips a few selections (the top-k kernel breaks ties by scan order). Replacing every token after a block boundary leaves the rate at that floor.

Changes per million (position, layer) pairsg = 16, c = 2048Layer spec. (τ = 96%)
After the boundary replaced3.9 (298 of 75.6M)85 (5,545 of 65.1M)
Identical inputs (noise floor)3.9 (298)84 (5,460)

(3) Within a block, the influence is also limited.

  • Path: the influence passes only through the shared candidate set. j's final keys are still chosen by its own native query, so whenever the candidates contain j's native top-k, j's selection is exactly native.
  • Magnitude: later tokens can change only the part of j's native top-k that the candidates miss, and this part is small and carries little weight. Cutting inside a group and permuting everything after the cut, harsher than any real edit, changes 2.0–6.8% of an earlier member's keys at g = 16, but only 0.55–1.9% of their native indexer mass: by mass the change is about a third of what it is by count, so the keys that move come mostly from the long tail (ranges over the six lengths).
Earlier members of the cut's groupg = 16, c = 2048Layer spec. (τ = 96%)
Keys that change2.0–6.8%0.2–3.3%
Their share of the indexer mass0.55–1.9%0.07–0.93%
Top-p coverage drop0.4–1.0 pt≤ 0.3 pt

(4) Your three cases.

  • (i) Complete-prompt prefill: the output depends only on the prompt, which is given all at once: the within-block dependence uses only the prompt's own tokens, so the model sees nothing beyond its input; tokens generated during decoding cannot change any prefill result (candidate sets or KV cache) and select their keys with the native indexer. All three benchmarks are evaluated this way.
  • (ii) Chunk boundaries: no effect. Groups are aligned to absolute positions, and SGLang ends chunks on KV-cache page boundaries (every 256 tokens, a multiple of G), so no group is ever split; vLLM implements the same chunking for hybrid models, which can be enabled for GroupIndex. Measured: at chunk sizes 4K / 6K / 8K the grouping is identical, with 0 mismatches in 46.1M (window, layer) pairs; and re-splitting a chunk within one pass, with the activations unchanged, leaves selections at the noise floor of item 2.
  • (iii) Cached prefixes: reused exactly. SGLang (radix cache) and vLLM (automatic prefix caching) reuse only whole pages, so a reused prefix never ends inside a block. By item 2, a prefix cached during prefill (e.g., a shared system prompt or document) then equals a fresh prefill of that prefix and carries nothing from the request that produced it.

W2 / Q2 and Q3 follow in the next comment.

AuthorsOpenReview ↗

Response to Reviewer aQwK (2/2): independent test documents and accuracy at 1M

W2 / Q2 (splits and sample sizes).

(1) The splits. Checkpoint selection, layer-plan selection and every reported coverage did share one set of validation documents, and Appendix D.2's "held-out" referred to training only; we have corrected that wording. The downstream benchmarks were never part of these steps. We now also evaluate on two sets no selection step has seen (Appendix D.2, results in the new Appendix E.7): 100 new ProLong documents, disjoint from the 1,010 used so far, and all 462 LongBench-v2 contexts.

DataDocumentsWindowsUsed for
Training900 per widthall (about 0.5M per width)training
Validation100 per width, one set600 (100 per length bucket); 96 (16 per bucket) for plan selectioncheckpoint selection, all coverage in the submission, layer-plan selection
Test (new)100 ProLong, disjoint (48 books, 48 code repos, 4 textbooks), + all 462 LongBench-v2 contexts (6 domains, 20 subdomains, 10K–4.6M tokens)600 each (100 per bucket)independent evaluation only

(2) Table 2's averages transfer. Re-running on the validation documents reproduces every Table 2 entry exactly. On the test ProLong documents all eight configurations stay within 0.5 points (top-p) and 0.8 (top-k) of their validation values, and none of the differences is significant. Outside the training domains the top-p drop stays within 1.5 points.

Coverage: top-p (top-k)Validation = Table 2Test ProLongLongBench-v2, training domainsLongBench-v2, other domains (QA, ICL, tables, dialogue)
g = 8, c = 204898.53 (96.78)98.39 (96.50)98.46 (96.67)98.13 (96.09)
g = 16, c = 204896.74 (93.18)96.66 (93.09)96.65 (93.09)96.21 (92.32)
g = 32, c = 204895.00 (89.98)94.69 (89.48)94.66 (89.44)93.91 (88.17)
g = 64, c = 204892.78 (86.73)92.29 (85.99)92.09 (85.66)91.32 (84.47)

(3) The plan's worst layer on independent documents. Per-layer top-p coverage, as a long-context average.

Layer-specialized plan, test setProLong, 100 docsLongBench-v2, training domainsLongBench-v2, other domains
Worst layer [95% CI]96.6 [96.1, 97.0]95.4 [94.4, 96.3]94.4 [92.7, 95.3]
Mean over the 21 layers97.797.596.4

(4) On τ = 96%. τ is the threshold the solver applies to the calibration table: for each layer it takes the cheapest (g_ℓ, c_ℓ) whose coverage clears τ, which fixes one operating point on the coverage–speed trade-off that Table 4 sweeps.


W2 / Q3 (accuracy at 1M). Table 1 of the revised PDF now reports RULER at 1M for every method, with its speedup column moved to 1M as well.

RULER at 1M (13 tasks, 50 prompts each)ScoreIndexer speedup at 1M (H20)
DSA87.721.00×
IndexCache (Reuse, g = 4)83.693.50×
IndexCache (Refine, g = 4)86.293.26×
Quest-style bound (g = 8)75.033.89×
PIVOT-Refine (g = 4)86.823.72×
PIVOT-Refine (g = 8)79.987.18×
GroupIndex (g = 16)87.3211.23×
↳ + layer spec.87.3815.67×

GroupIndex at g = 16, alone and with the layer plan, stays within noise of DSA at 1M, where the prefill gain is largest: the paired differences are −0.41 ± 0.44 and −0.35 ± 0.54. It is the most accurate and the fastest arm in the table.


We hope these answers resolve your questions and would be glad to clarify anything further.


Reviewer_aQwKOpenReview ↗

Official Comment by Reviewer aQwK

Thank you for the detailed response and additional experiments. The clarification of block-level prefix consistency, the independent coverage evaluations, and the new RULER results at 1M tokens adequately address my main concerns. I maintain my score of 3/4.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for three careful reviews, and we are glad they recognise the core of the contribution: that learning the query aggregation lifts a real limit of mean pooling, so the sharing group can grow with less degradation (iNDU); that the analysis, the method and the ablations form a single account (kXCz); and that the component ablations and the whole-prefill measurements make this a system gain (iNDU, kXCz): at 1M GroupIndex matches the native indexer's accuracy while the indexer stage runs about 15.7× faster and end-to-end prefill up to 1.92×.

To answer the reviews in full, we ran six additional sets of experiments, grouped below under the three concerns the reviews share and given in detail in the individual replies. A revised PDF is uploaded, with all changes in blue.


(1) Causal semantics (aQwK, iNDU). We agree that key visibility does not imply prefix consistency, and the revision now treats the two separately (Appendix B.2).

  • Key visibility holds strictly: no position ever attends to a future key.
  • Prefix consistency holds per block of G consecutive tokens, where G is the largest group width: only later tokens of a position's own block can affect it.
  • Measured in the serving engine (new Appendix B.3): replacing everything after a block boundary leaves the selections at the noise floor of an identical rerun; chunk sizes of 4K / 6K / 8K give identical grouping; cached prefixes are reused exactly.
  • Within a block the effect is small: the influence passes only through the shared candidate set, and even permuting everything after a cut inside a group moves only 2.0–6.8% of an earlier member's keys and 0.55–1.9% of their indexer mass.

(2) Coverage: documents no selection step has seen, and deployment activations (aQwK, iNDU).

  • What was shared: in the submission, checkpoint selection, layer-plan selection and every reported coverage used one set of validation documents, and Appendix D.2's "held-out" meant held out from training only; we have corrected that wording.
  • What is new: 100 new ProLong documents and all 462 LongBench-v2 contexts (Appendix D.2, results in the new Appendix E.7).
  • Table 2's averages transfer: every configuration stays within 0.5 points of its validation coverage under top-p and 0.8 under top-k, and none of the differences is significant.
  • The plan's worst layer holds up: 96.6% on the new documents, and 94.4% outside the training domains.
  • Under deployment activations (iNDU): running the same documents with GroupIndex driving attention in all 21 layers moves top-p coverage by −0.08 points [−0.10, −0.06], and the gap does not grow with depth.

(3) Quality at 1M, where the speedup is largest (aQwK, iNDU, kXCz). Task accuracy in the submission stopped at 512K. Table 1 of the revised PDF now reports RULER at 1M for every method, and its speedup column moves to 1M as well. It also includes the Quest-style bound kXCz asked us to test, built in four variants and reported at its best. Bold and underline mark the best and second best, excluding the reference.

MethodLB-v2 / LME-s / LME-mRULER 512KRULER 1MIndexer speedup at 1M
Native indexer (DSA)52.5 / 85.6 / 40.488.387.721.00×
IndexCache (Reuse, g = 4)49.9 / 87.0‾\underline{87.0} / 41.086.683.693.50×
IndexCache (Refine, g = 4)51.3 / 86.6 / 40.887.9‾\underline{87.9}86.293.26×
Quest-style bound (g = 8)53.5 / 84.8 / 38.083.4475.033.89×
PIVOT-Refine (g = 4)53.3 / 85.2 / 41.4‾\underline{41.4}87.886.823.72×
PIVOT-Refine (g = 8)53.9 / 85.4 / 41.4‾\underline{41.4}83.079.987.18×
GroupIndex (g = 16)53.7‾\underline{53.7} / 87.8 / 41.888.287.32‾\underline{87.32}11.23‾\underline{11.23}×
↳ + layer spec.53.1 / 85.8 / 41.4‾\underline{41.4}88.287.3815.67×

At 1M, where the arms finally separate, GroupIndex stays within noise of the native indexer and is the fastest arm in the table.


We hope these answers resolve the reviewers' questions and would be glad to clarify anything further.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.