acceptodds
Under review as a conference paper at ICLR 2027

SCOPE: Stored Chunk-Level Offline Priors for Efficient Long-Context Inference

Abstract

Long-context inference for large language models is limited by the repeated online scoring used to select important key-value (KV) blocks. We introduce SCOPE (Stored Chunk-Level Offline Priors), an efficient chunk-level method that reuses offline content priors across queries within a deployment domain. A store of offline chunk scores is built in the scheme of exponential moving average (EMA). Then unseen chunks are predicted with a lightweight ridge regressor, and finally intra- and inter-chunk refinement is applied at inference time with constant cost. This design replaces the dense scoring path of Flash Sparse Attention (FSA) with an expected- hash-table lookup per chunk followed by lightweight refinement. Although SCOPE uses static routing instead of the original query-dependent selector, it is still compatible with FSA’s sparse-kernel interface. SCOPE adds no trainable parameters and requires only a small amount of auxiliary storage. Its store construction and lookup path can also run on CPUs alongside GPU computation, reducing inference latency without extending GPU pretraining. On Qwen-3.5-2B and Llama-3.2-1B, SCOPE improves median latency in 31 of 32 released-backbone cells, with average median and P99 reductions of 11.5% and 12.8%, respectively. On the larger Qwen2.5-7B model with a 32K context, SCOPE achieves a 15.3% average median-latency reduction. These results show that offline chunk priors provide an efficient alternative to repeated online scoring for long-context inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-context inference, Sparse attention, Efficient attention, Large language models, Inference acceleration, KV block selection, Offline chunk priors

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nN15OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SCOPE, which replaces the query-dependent online block scoring used by FSA with stored, chunk-level offline priors. Chunk scores are built from a domain corpus and stored in a hash table; unseen chunks use a lightweight ridge predictor, followed by local refinement. The resulting scores are converted into FSA-compatible top-K block indices, allowing the existing sparse-attention kernel to remain unchanged.

Experiments on Qwen-3.5-2B, Llama-3.2-1B, and Qwen2.5-7B report roughly 11–15% latency reductions over FSA, with larger gains at longer contexts. The paper also evaluates perplexity, MedQA accuracy, domain transfer, and several components through ablations.

Critical strengths
  1. The method targets a real bottleneck that remains after sparse-attention kernel optimization. Replacing repeated query-dependent scoring with a cheap reusable prior is simple and potentially practical, especially in domain-specific serving workloads.

  2. The latency improvement is reasonably consistent and not merely a very small numerical gain. The reported average improvement is around 11–15%, becomes larger at longer sequence lengths, and is observed across multiple backbones. The paper also includes useful operator-level analysis and domain-shift experiments rather than reporting only one favorable configuration.

Critical weaknesses
  1. The long-context quality evaluation is not strong enough for a method that replaces query-dependent attention routing with largely query-independent offline scores. This is exactly the setting where an important chunk may become relevant only for a particular query. However, the paper mainly relies on perplexity and a MedQA accuracy check, while standard long-context retrieval/reasoning benchmarks such as RULER and LongBench v2 are explicitly left to future work. This is consequential: good perplexity does not establish that SCOPE preserves retrieval of query-specific information from a long context. The relatively large PPL degradation on some Llama/MedQA settings also suggests that this deserves closer study.

  2. The current latency measurements do not fully establish an end-to-end serving improvement. SCOPE moves the scoring path from the GPU to CPU-side chunking, hash lookup/regression, refinement, and host-to-device transfer, but the evaluation relies primarily on CUDA-event measurements and does not clearly report these CPU-side costs as end-to-end wall-clock overhead. The serving setup is also batch size 1, single process, without paged KV or a production serving stack. Given that the reported overall gain is mostly 10–15%, these omitted costs could materially affect the conclusion.

  3. FSA is a sensible primary baseline, and the appendix includes HashAttention, which is useful. However, the HashAttention comparison is limited to median latency at 8K on one backbone and does not include matched quality or longer-context results. Other closely related online-routing methods discussed by the paper, such as SeerAttention and TokenButler, are not experimentally compared. I would not require comparison to every sparse-attention method, but a more complete comparison with the closest scoring alternatives would make the claimed Pareto advantage much more convincing.

Questions for authors
  1. Can the authors evaluate SCOPE on RULER, LongBench v2, or another query-dependent long-context retrieval benchmark? I would particularly like to see cases where the relevant evidence occurs far from the query and has low prior importance independent of that query.

  2. Can the authors report true end-to-end wall-clock latency, including CPU chunking, hash lookup/regression, refinement, host-to-device transfer, and synchronization? How much of the reported 11–15% gain remains under this measurement?

  3. Could the authors provide a matched latency–quality comparison with HashAttention at 16K/32K, and preferably one additional closely related routing baseline? The current 8K latency-only comparison makes it difficult to judge the SOTA advantage.

  4. How sensitive is SCOPE to domain shift in terms of quality, not only latency? The law-domain experiment already shows that a high hit ratio does not necessarily imply a speed advantage; it would be useful to understand when offline chunk importance ceases to approximate query-dependent routing well enough.

1 comment

We thank Reviewer nN15 for asking for query-dependent long-context quality, complete serving-path latency, and closer routing baselines. On Llama-3.2-1B, SCOPE obtains 25.23% accuracy on 400/503 LongBench-v2 examples, compared with 24.58% for FSA (+0.65 percentage points). This reports task accuracy on non-repeated long-context examples rather than relying on perplexity alone.

On the Law transfer workload, synchronized end-to-end wall-clock latency includes CPU chunking, hash lookup/regression, refinement, H2D transfer, and synchronization. SCOPE reduces median latency from 811.0 to 755.8 ms (6.8%) and P99 from 898.1 to 851.4 ms (5.2%); the ChunkStore hit ratio is 93.5%.

At 16K, SCOPE's median/P99 latency is 8.5%/6.2% lower than HashAttention and 15.3%/12.1% lower than SeerAttention. We limit our conclusions to the measured workloads and report task quality separately from latency.

Reviewer_EQ4dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SCOPE replaces FSA's query-dependent scoring with offline chunk priors, a regressor and refinement. Within a forward pass, the same top-K index set is reused across the sparse-attention layers using SCOPE and shared by all KV heads and query positions. On Qwen-3.5-2B and Llama-3.2-1B at 8K–16K, attention latency improves in 31 of 32 cells, with a mean median reduction of 11.5%. Quality is assessed using PPL on cyclically tiled documents and a MedQA test at 3,072 tokens.

Critical strengths
  1. The method targets a substantial measured scoring cost: on the self-built Transformer, reported score+TopK time is about 40.4 ms per call for FSA versus 0.19–0.20 ms for SCOPE with cross-layer index reuse (Table 22; Appendix B.6). SCOPE retains FSA's sparse kernels.
  2. The paper reports per-configuration results and several ablations, including unfavorable cases.
Critical weaknesses
  1. The quality tests do not establish use of distant evidence: the 8K–16K inputs tile short documents (at most 964 tokens under the Qwen tokenizer), and MedQA uses random PubMed prefixes. Appendix B.8 attributes near-chance accuracy to the 1,000-step training budget, but these near-chance results are insufficient to establish capability preservation. Llama's MedQA PPL also rises from 2.90 to 5.00 at 8K (Table 20), which warrants qualifying the quality claim in Section 5.2.
  2. Table 15 leaves the benefit of inference-time ChunkStore lookup unclear. All-regressor lowers mean MedLat/P99 from 786.6/851.4 to 755.8/783.1 ms with ΔPPL +0.032. Randomizing miss scores also changes PPL little (+0.016), but MedQA P99 rises to 1079 ms; that variant keeps the store, so it does not separate the store's contribution from the rest of the pipeline. AvgScr alone does not establish a task-quality advantage for explicit lookup.
  3. The MedQA fidelity argument needs a statistical basis. Llama's observed accuracy difference is −1.00 pp (Table 13). Comparing it with individual binomial intervals does not establish equivalence. Please justify the 1 pp tolerance and report a confidence interval for the paired accuracy difference; if statistical equivalence is claimed, provide the corresponding test.
Questions for authors
  1. For PPL, is routing built from the full scored sequence? Please test whether changing a suffix while keeping its length fixed changes prefix logits. If it does, rerun with prefix-causal routing; masking future K/V alone would not remove that dependence.
  2. Can you compare the original backbone, FSA and SCOPE on one non-repeated long-context task whose answer depends on a distant passage?
  3. Which CPU preparation and H2D costs fall inside the reported timing intervals (Appendix B.5)? Please provide synchronized wall-clock latency covering the complete CPU–GPU path.
2 comments

We thank Reviewer EQ4d for requesting task-level long-context evaluation, paired statistics, a prefix-causality check, and complete wall-clock timing. On Llama-3.2-1B, SCOPE obtains 24.12% accuracy on 400/503 LongBench-v2 examples versus 24.58% for FSA (paired difference: -0.46 percentage points; 95% CI: [-0.85, -0.12]).

On 500 paired MedQA questions, SCOPE reaches 73.4% accuracy versus 72.0% for FSA (+1.4 points); the exact paired McNemar test gives p=0.04. We report this as evidence on the evaluated set, not as a formal claim of statistical equivalence. In a fixed-length suffix perturbation test with strict prefix-causal routing, the maximum prefix hidden-state and probe-logit differences are 0.0004 and 0.0007, mean JS divergence is 5e-7 nats, and task accuracy changes by 0.00 points.

For the Law timing, synchronized end-to-end wall-clock measurement includes CPU chunking, lookup/regression, H2D transfer, and synchronization: median latency falls from 811.0 to 755.8 ms and P99 from 898.1 to 851.4 ms.

Thanks for the response and experiments but I don't think my major concerns are addressed so I remain my score.

Reviewer_ZmL6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a sparse-attention routing method that replaces FSA's query-dependent block scoring with offline priors. Basically, the authors use some heuristics to compute chunk importance scores and then train a ridge regressor to predict these scores for unseen chunks. During inference, they further refine these scores before selecting the top-K blocks. They evaluate the method on three language models and report around 11.5% to 15.3% reduction in attention latency, although the results on perplexity are mixed. Overall, I think the idea is interesting, but the experiments raise more questions than they answer.

Critical strengths

I think using offline priors could be useful in some specific applications, especially when the documents follow relatively fixed structures or templates, like medical or financial records. In these cases, the importance of different chunks might follow some recurring patterns, and we could use that to reduce the amount of computation needed at inference time. So, I think there is some novelty in this idea.

Critical weaknesses

There are too many heuristic design choices in the paper. These are the ones that caught my eye:

  • seven hand crafted regression features
  • IDF position scoring weights and thresholds
  • EMA smoothing coefficient
  • intra-chunk refinement weights
  • inter-chunk neighbor bonuses and isolation penalties
  • top-K block selection,
  • disabling the compressed-attention branch

The paper doesn't provide enough evidence that these heuristic scores can reliably identify important attention blocks. Also, interestingly enough, the ablation results show that removing the ChunkStore actually improves latency with almost no effect on perplexity. So this raises the question of whether ChunkStore is even necessary in the first place.

Another issue I have is with the evaluation. The 8K to 16K experiments are done by cyclically repeating short documents, which introduces a lot of redundancy. I don't think this is a good way to evaluate long-context performance, as it doesn't tell us whether the method can actually retrieve important information from long documents. There are also no standard long-context retrieval or reasoning benchmarks. On top of that, some of the perplexity regressions are quite large. I'm not convinced that the experiments provide enough evidence that the method can preserve quality while reducing latency.

Questions for authors
  1. What evidence do we have that IDF and position based chunk importance remains stable across different queries, attention heads, and layers? I think this is one of the main assumptions of the paper, but I don't see enough experiments supporting it.

  2. Given that removing the ChunkStore actually improves latency with almost no effect on perplexity, how would SCOPE perform against simpler baselines? For example, what if we just used position-only, IDF-only, or even random static block selection?

  3. Can the authors evaluate SCOPE on actual long-context benchmarks like RULER or LongBench v2 instead of repeating short documents? I think this would give us a much better picture of whether the method can preserve retrieval and reasoning performance, especially when the relevant information appears only once.

3 comments

We thank Reviewer ZmL6 for emphasizing the need to evaluate long-context retrieval and the stability of offline routing. On Llama-3.2-1B, SCOPE achieves 25.23% accuracy on 400/503 LongBench-v2 examples, compared with 24.58% for FSA (+0.65 percentage points). This adds a non-repeated long-context task evaluation beyond perplexity on tiled inputs.

We also compared online FSA scores with SCOPE priors across the evaluated queries, heads, and layers. Mean Spearman correlation is 0.58, with 56.2% top-12 overlap and 0.39 Jaccard similarity; the worst layer/head values are 0.41, 42.1%, and 0.28. These results indicate partial, non-uniform alignment, so we will frame SCOPE as a lower-cost approximate routing strategy rather than claim that offline priors reproduce every query-dependent route.

On the common 118-example ablation subset, SCOPE scores 20.34%, versus 17.80% for position-only, 17.80% for IDF-only, 18.64% for random static selection, and 17.80% for the all-regressor variant. This comparison indicates that the combined routing design improves task accuracy over these simpler controls on this subset.

Thanks for the additional experiments. I think the rebuttal addresses some of my concerns, but two major issues remain for me.

First, the simpler baseline comparison is only on 118 examples so the differences between SCOPE and other variants are quite small in absolute terms. I don’t think this is enough to justify that the ChunkStore-based design is necessary.

Second, while I appreciate the LongBench experiment, both SCOPE (25.23%) and FSA (24.58%) are essentially at chance-level performance for a four-choice task. So this does not convince me that SCOPE preserves meaningful long-context retrieval.

For these concerns, I maintain my score.

Thank you for the detailed feedback. FSA has been established as a reliable sparse attention model, so comparison with FSA is indeed necessary. The near-chance performance is because our model is built for and applied to a specialized domain (currently medicine). Therefore, benchmarks like LongBench, which involve diverse domains, are not well-suited for our model, resulting in the ~25% accuracy. We will apply it to medical-specific datasets such as LongMedBench to demonstrate that our model possesses long-context retrieval capability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.