acceptodds
Under review as a conference paper at ICLR 2027

Oracle-Guided Sparse Prefill: Separating Oracle, Indexer, and Runtime Gaps in Long-Context GQA

Abstract

Long-context prefill remains expensive because global attention layers, including those using grouped-query attention (GQA), must compute pairwise query–key scores across the entire history, incurring quadratic cost in sequence length even in hybrid models that combine global attention with local, linear, or recurrent components. We study when this computation can be sparsified without materially changing task performance. We use an attention-mass top-k oracle that selects a head-averaged token support from dense attention. This non-deployable reference separates sparse-budget feasibility from indexer error and runtime realization effects. We evaluate a head-collapsed indexer trained by KL distillation with the backbone frozen. On Qwen3.5-27B RULER samples from 32K to 128K, matched per-query and 32-query-shared oracle and indexer configurations remain within one observed macro-accuracy point of dense. At 128K, the 32-query-shared indexer scores 0.61 points below dense with a 1.46 single-request TTFT speedup. On sampled tail queries, the indexer recovers 94–98% of the attention mass retained by the oracle, yet task-level failures coexist with this high aggregate fidelity. Key-block experiments show that oracle-feasible support structures can still fail under learned selection. These observations motivate configuration-aware support selection and establish a diagnostic study of sparse-prefill quality–latency tradeoffs.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
long-context inference, sparse attention, grouped-query attention, sparse prefill, attention oracle, learned indexer, time to first token

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_GZqvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates sparse prefill in frozen GQA models through comparisons among dense attention, oracle selection, and learned indexers. It constructs a top-k oracle from head-averaged dense attention distributions and trains a single-head indexer via KL distillation while keeping the backbone frozen. By comparing task performance and runtime efficiency across different support-selection granularities, the paper examines the quality degradation introduced by sparse selection and the speedups achieved in practice.

Critical strengths

On Qwen3.5-27B with a 128K context, the paper reports a 1.46× single-request TTFT speedup, providing some empirical support for the practical value of the evaluated configuration.

Critical weaknesses

1.Limited methodological novelty and an unclear core technical contribution.

The method largely follows the established approach of freezing the backbone, distilling an indexer from dense attention, and using it for sparse selection. The authors also explicitly acknowledge that their training objective is closely related to DSA’s warm-up objective. The paper does not sufficiently clarify what substantive new mechanism it introduces beyond existing indexer-distillation approaches. The single-head indexer appears more like a particular implementation choice than a well-substantiated methodological innovation.

2.The diagnostic analysis remains descriptive and does not yield sufficiently deep new insights.

The paper organizes its analysis around the oracle, indexer, and realization gaps. However, a central finding remains insufficiently explained: some key-block configurations preserve task performance under oracle selection but degrade substantially when using the learned indexer. The authors do not provide targeted controlled experiments to identify the source of this discrepancy, nor a stage-by-stage runtime breakdown to explain why the key-block implementation does not deliver additional speedups.

Questions for authors

1.Beyond applying frozen-backbone KL distillation with a single-head indexer, what substantive advance does this work introduce over existing indexer-based sparse attention approaches?

2.Why does learned key-block selection suffer a substantial accuracy drop when the corresponding oracle maintains strong task performance?

2 comments
AuthorsOpenReview ↗

Authors' response: controlled key-block experiments, fidelity diagnostics, and runtime profiling

Thank you for distinguishing a useful diagnostic organization from the need for deeper explanations. We agree that the key-block discrepancy requires more than a descriptive observation.

  • Advance beyond the established training recipe:

Frozen-backbone KL distillation and top-k selection are established components. We do not present the single-head implementation alone as the core novelty. The contribution is the empirical investigation of oracle feasibility, learned-selection error, and runtime under explicit support granularities and budgets.

  • Why learned key-block selection loses accuracy:

We added a matched-budget teacher/student by token/key-block control with query sharing fixed to 32 and key-block width 128. On three selected tasks (niah_multikey_2, niah_multikey_3, cwe; ten exactly hash-matched prompts each), the respective three-task mean scores are:

Selection32K, k=102464K, k=2048
Teacher token100.00100.00
Teacher key-block99.33100.00
Student token100.00100.00
Student key-block70.0066.00

These controls show that the teacher–student discrepancy persists for key blocks when query sharing and token budget are held fixed. They strengthen the evidence that the evaluated block support is feasible but the deployed learned selector fails to realize it. The scores cover selected tasks, not the full RULER macro. They do not yet distinguish score-estimation error, block aggregation, and training-objective mismatch; we retain these as hypotheses rather than claim a causal mechanism.

A further diagnostic holds dense trajectories fixed on nine shared prompts, three layers, and 32 tail queries at context 61,440. Student/teacher AMR is 0.4624/0.4769 for token support and 0.3599/0.3531 for key blocks; relative AMR is 0.9694 and 1.0192. Thus, these sampled aggregate metrics do not explain the key-block failure by lower relative mass retention. MAX-over-query selection need not maximize mean per-query AMR, so a ratio above one is permitted. This motivates examining task-critical support and recursive sparse trajectories, but does not establish either as the cause.

  • Why key blocks do not yield extra runtime benefit:

We now profile the current normalized two-pass implementation on one synthetic 64K request after warmup, with matched budget 2048, query sharing 32, and 4K chunks. Summed GPU kernel durations for normalization plus candidate scoring are 0.941 seconds for token support and 0.943 for key blocks. Sparse attention takes 1.294 and 1.386 seconds; named top-k/gather kernels take 0.097 and 0.053 seconds. All-model GEMMs contribute about 68% of summed kernel duration in both sparse cells. The revised appendix provides the breakdown and classification method.

In this trace, reduced top-k/gather time is offset by other costs, with no meaningful overall key-block advantage. We do not infer a general kernel limitation or a statistically established speed ranking from one instrumented synthetic request. These profiles are distinct from the real-prompt baseline timing protocol and do not isolate auxiliary projection time from backbone GEMMs.

Thank you for your response,and I will keep my rating.

Reviewer_tqQ1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies post-hoc sparse prefill for pretrained GQA language models with frozen backbone weights. It introduces an attention-mass top-k oracle to assess sparse-support feasibility and trains an auxiliary indexer through KL distillation to predict token importance. By comparing dense attention, oracle-selected sparsity, and learned selection across support-sharing configurations, the paper examines quality losses and runtime tradeoffs. On Qwen3.5-27B at 128K context, the proposed indexer achieves a 1.46× single-request TTFT speedup with a 0.61-percentage-point decrease in RULER macro accuracy. Additional experiments highlight limitations in key-block selection and transfer across tasks and models.

Critical strengths
  1. Clear diagnostic framework. The paper separates sparse-support feasibility, learned-selection error, and runtime effects through matched dense–oracle–indexer comparisons. This provides a useful structure for investigating where sparse-prefill quality losses arise.

  2. Concrete quality–latency results. On Qwen3.5-27B at 128K context, the block-shared indexer achieves a 1.46× single-request TTFT speedup with a 0.61-percentage-point decrease in RULER macro accuracy, demonstrating a practical operating point under the evaluated conditions.

  3. Informative ablations and failure cases. The query-sharing and key-block experiments reveal that oracle-feasible sparsity does not necessarily translate into successful learned selection. The finding that high aggregate attention-mass retention can coexist with task-level failures also highlights a limitation of attention-fidelity metrics.

Critical weaknesses
  1. Limited novelty. The core approach combines a frozen backbone, an auxiliary indexer trained through KL distillation, and top-k sparse attention. These components closely follow established approaches, including the DSA warm-up procedure acknowledged by the authors. The dense–oracle–indexer comparisons provide useful diagnostic evidence, but the paper does not clearly establish a sufficiently novel methodological or analytical contribution.

  2. Missing competitive baselines. The evaluation primarily compares the proposed indexer against dense attention and its own oracle reference. It lacks matched comparisons with existing sparse-attention and sparse-prefill methods in both task performance and latency. Comparisons with simple selection heuristics, such as attention sinks combined with a recent-token window, are also missing. Consequently, the reported speedup does not establish whether the proposed approach offers a competitive quality–latency tradeoff.

  3. Limited task coverage. The main evaluation relies heavily on RULER. Although the appendix includes sampled InfiniteBench and LongBench-v2 evaluations, these provide limited evidence for more complex workloads. Evaluations on long-context coding and agentic reasoning tasks, with corresponding latency measurements, would better establish practical utility.

  4. Limited validation across models. The matched evaluations of distilled-indexer quality and latency are concentrated on Qwen3.5-27B. The additional model evaluations primarily examine oracle feasibility, which does not establish that the learned indexer achieves comparable quality or speedups on those models. Evaluating trained indexers across multiple model families and sizes is necessary to substantiate the generality of the approach.

Questions for authors

see weaknesses

1 comment
AuthorsOpenReview ↗

Authors' response: competitive baselines and selector controls; clarified contributions and generalization limits

Thank you for recognizing the diagnostic framework, concrete quality–latency results, and informative failure cases. We agree that the original comparison did not sufficiently establish competitiveness or the scope of generalization.

  • Competitive quality–latency comparisons:

We have added MInference, FlexPrefill, Prism, and XAttention on the same 650 RULER prompts per length, measuring accuracy and engine prefill time. We also tune MInference, FlexPrefill, and both indexer configurations on a disjoint seed-1234 set, using an accuracy-constrained selection rule. Default and tuned results are both reported.

At 128K, the selected per-query k=512 configuration achieves 96.31 accuracy versus 96.46 for dense, at 39.87 versus 59.36 seconds of mean engine prefill time (1.49×). Tuned MInference has comparable latency (39.49 seconds) with lower accuracy (94.63); tuned FlexPrefill reaches 95.85 accuracy at 48.84 seconds. The revised appendix discloses chunk differences, paired dense references, and the XAttention kernel adaptation. These results support competitiveness under the evaluated settings, rather than universal superiority. A completed 650-prompt fixed-support extension at 64K scores 55.89 for 64 sinks plus a 4032-token window, versus 97.69 for dense and 97.54 for the per-query indexer on exactly hash-matched inputs. GDN and dense decode remain unchanged; the wider window was selected after exploratory screening.

  • What the additional comparison reveals:

Budget tuning also qualifies the benefit of query sharing. At 128K, per-query k=512 has higher observed accuracy and slightly shorter prefill time than the evaluated block-shared k=2048 configuration (95.54 accuracy, 40.81 seconds). Granularity and budget therefore need joint evaluation; query sharing is not an unconditional quality–latency improvement. These are descriptive operating points, not statistically established rankings.

  • Methodological and analytical contribution:

We agree that the inherited training components do not establish methodological novelty by themselves. The revision separates that recipe from the empirical contribution: comparing support feasibility, learned selection, and realization under explicit configurations. Oracle-feasible key blocks and high aggregate attention fidelity can coexist with task failures. We present these as diagnostic findings and do not claim that their mechanisms have already been isolated.

  • Task and model coverage:

The strongest matched evidence remains on RULER and Qwen3.5-27B. Existing sampled InfiniteBench and LongBench-v2 results delimit transfer but do not establish broad coding or agentic utility. Oracle-only evaluations on additional models establish feasibility under those configurations, not trained-indexer quality or speed generalization. We retain these boundaries explicitly. Larger non-retrieval evaluations, simple-selector controls, and a matched trained-indexer frontier on another model remain necessary to support broader claims.

Reviewer_gbUJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper analyzes the effect of sparsifying the dense attention layers in a GDN hybrid attention network.

Critical strengths

They converted dense attention into sparse attention.

Critical weaknesses
  • Limited evaluation: RULER (NIAH) can be largely solved by GDN layers alone, so it is not suitable for evaluating sparse attention in a hybrid model. Comparison with training-free sparse attention methods is also missing.
  • Limited novelty: Learning-based sparse attention (e.g., SEA: Sparse estimated attention, DSA: Deepseek sparse attention) and oracle-based sparsification (e.g., TokenSelect) have already been studied.
  • Significantly under-analyzed: In a hybrid model, the effect of converting attention is limited, so end-to-end efficiency should be analyzed.
Questions for authors

Please check the weakness section for my concerns.

1 comment
AuthorsOpenReview ↗

Authors' response: baseline comparisons, hybrid-model controls, and runtime analysis

Thank you for raising concerns about evaluation validity, competitive baselines, novelty, and end-to-end efficiency.

  • Competitive training-free baselines:

We have added MInference, FlexPrefill, Prism, and XAttention, evaluating both task accuracy and engine prefill time on the same 650 prepared RULER prompts at each of 32K, 64K, and 128K. We report default configurations and additionally tune MInference, FlexPrefill, and the indexers on a disjoint tuning set.

At 128K, the per-query indexer with k=512 achieves 96.31 macro accuracy compared with 96.46 for dense, while reducing mean engine prefill time from 59.36 to 39.87 seconds. Tuned MInference reaches comparable latency (39.49 seconds) with 94.63 accuracy; tuned FlexPrefill reaches 48.84 seconds with 95.85 accuracy. The revised-paper appendix gives the full tables and tuning protocol.

  • RULER evaluation in a hybrid architecture:

We agree that dense-relative preservation alone does not isolate the role of global attention. We now add fixed-support controls, leaving GDN and dense decode unchanged. After screening three supports on 130 prompts, we extend the strongest observed fixed support (64 sinks plus 4032 recent tokens) to all 650 prompts at 64K. It scores 55.89, versus 97.69 for dense and 97.54 for the per-query indexer on exactly hash-matched inputs. Thus, quality preservation is not automatic merely because GDN is unchanged: restricting global-attention prefill support produces a 41.80-point loss, despite twice the indexer token budget. This tests support dependence, not GDN alone or the necessity of every global-attention layer. The wider window was selected after screening; the 650-prompt set includes the screening samples.

  • End-to-end efficiency:

The supplementary measurements include selection, sparse-attention execution, and the remaining model computation up to the first token on real prompts. They therefore quantify model-level prefill benefits in the hybrid architecture. This does not by itself provide a causal stage-level explanation of every runtime difference. Indexers use 4K chunks; baseline implementations use whole prompts at 32K/64K and two chunks at 128K because of memory constraints. We disclose these differences and report absolute times and both paired and common dense references.

  • Novelty and positioning:

We agree that learned selection and oracle-based sparsification have precedents. Frozen-backbone KL distillation and top-k selection are not claimed as new primitives. Our intended contribution is a matched empirical study of oracle feasibility, learned-selection error, and runtime realization across support configurations. The additional competitive results strengthen practical evidence, but do not on their own resolve novelty. We will sharpen the comparison with SEA, DSA, and TokenSelect and distinguish established components from the diagnostic findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.