acceptodds
Under review as a conference paper at ICLR 2027

ReFresh: Learning When to Retrieve for Long-Context Prefilling

Abstract

Long-context language models enable retrieval and reasoning over large documents, but quadratic self-attention makes prefilling a major bottleneck. Hybrid attention mitigates this cost by combining a few full-attention heads with cheaper streaming heads, yet its efficiency remains limited by the full-attention budget, whose reduction can hurt long-range retrieval. We observe that full attention serves two roles—computing attention outputs and identifying relevant context regions—and that the latter exhibits substantial redundancy across layers: the same key/value (KV) head in neighboring layers often attends to highly overlapping context regions. Existing cross-layer reuse methods exploit this redundancy, but typically reuse indices or attention according to fixed or heuristic schedules. Building on this observation, we introduce ReFresh, a hybrid attention framework that learns where global retrieval should be refreshed or reused across layers, yielding a binary Anchor/Reuse map through dense-to-sparse distillation with the language model frozen. Anchor slots perform full attention, derive block scores from the resulting attention probabilities, and cache the selected KV-block indices for each query block. Reuse slots attend sparsely using the latest indices from an earlier Anchor of the same KV head, preserving global access to distant blocks without recomputing selection—unlike streaming heads, whose access is restricted to attention sinks and a recent window. Across Qwen3-8B and Llama-3.1-8B on LongBench, LongBench v2, and RULER, ReFresh preserves long-context quality with 80% Reuse and achieves a prefill speedup for Qwen3-8B at 256K tokens over FlashAttention-2.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Long Context LLMs, Sparse Attention Mechanism

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bzamOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReFresh, a hybrid attention framework designed to accelerate long-context prefilling by exploiting cross-layer redundancy in attention patterns. The key observation is that neighboring Transformer layers often attend to similar KV blocks, making repeated global retrieval unnecessary.

ReFresh introduces an Anchor/Reuse mechanism at the layer–KV-head level. Anchor slots perform full attention and identify important KV blocks based on attention probabilities, while Reuse slots reuse the previously selected block indices to perform block-sparse attention using the current layer's queries, keys, and values. To determine where retrieval should be refreshed, the authors propose a learned routing schedule based on Hard-Concrete gates and dense-to-sparse distillation, while keeping the pretrained language model weights frozen. The learned schedule is subsequently converted into a static binary routing map for inference.

The authors evaluate ReFresh on Qwen3-8B and Llama-3.1-8B using LongBench, LongBench v2, and RULER. Experimental results show that ReFresh largely preserves full-attention accuracy with 80% of layer–KV-head slots operating in Reuse mode. Furthermore, ReFresh achieves up to 2.62× prefill speedup over FlashAttention-2 at 256K context length on an NVIDIA H800 GPU.

Critical strengths
  1. Effective exploitation of cross-layer attention redundancy.

The paper presents a well-motivated Anchor/Reuse mechanism that exploits the redundancy of attention patterns across neighboring Transformer layers. By reusing previously selected KV-block indices, ReFresh avoids redundant retrieval operations and reduces attention computation while preserving access to globally relevant context.

  1. Practical and lightweight learning-based routing design.

ReFresh learns layer–KV-head-level refresh schedules through dense-to-sparse distillation without modifying the pretrained language model weights. The inference-consistent index propagation and static routing map provide a practical approach to integrating learned refresh scheduling into existing LLM inference pipelines without additional runtime routing decisions.

  1. Promising accuracy–efficiency trade-off with measurable prefill acceleration.

The paper evaluates ReFresh on two language models across LongBench, LongBench v2, and RULER, demonstrating that it largely preserves full-attention quality with 80% of slots operating in Reuse mode. Furthermore, the implementation achieves up to 2.62× prefill speedup over FlashAttention-2 at 256K tokens on an NVIDIA H800 GPU, demonstrating the practical potential of the proposed approach.

Critical weaknesses
  1. Insufficient controlled evaluation of the learned refresh schedule.

The paper attributes the effectiveness of ReFresh primarily to its learned Anchor/Reuse placement. However, the ablation study in Table 3 does not provide a sufficiently controlled comparison against alternative routing strategies. Although the learned schedule achieves better accuracy than fixed-interval schedules despite using fewer Anchor slots, the comparisons are conducted under different Anchor budgets, making it difficult to isolate the contribution of the proposed learning-based optimization.

To better demonstrate the effectiveness of learned routing, the authors should compare the following strategies under the same Anchor budget (e.g., 20.1%) and identical block-selection configurations: Random Routing (Randomly select 20.1% of layer–KV-head slots as Anchors), Uniform Routing (Distribute Anchor slots uniformly across layers while maintaining the same 20.1% budget), Heuristic Routing (Determine Anchor placement based on cross-layer attention-pattern similarity, refreshing indices when the attention patterns change significantly), Learned Routing (Use the proposed learned Anchor/Reuse schedule).

Such budget-matched comparisons would more convincingly establish whether the performance gains arise from the learned routing strategy itself rather than the general benefits of cross-layer index reuse or differences in Anchor allocation.

  1. Missing comparisons with closely related cross-layer index reuse methods.

The paper identifies Kascade and IndexCache as closely related approaches that exploit cross-layer redundancy in sparse attention indices. However, neither method is included in the experimental comparisons. Instead, the evaluation primarily focuses on conventional sparse and hybrid attention baselines, such as FlexPrefill, XAttention, and DuoAttention. Since the central contribution of ReFresh is learning when to refresh cached indices rather than introducing cross-layer index reuse itself, comparisons with existing index-reuse methods are essential to establish its incremental benefits. Without such comparisons, it remains unclear whether the learned refresh schedule offers meaningful improvements over existing fixed or heuristic reuse strategies.

Questions for authors

Q1. Could the authors provide a budget-matched comparison of different Anchor/Reuse routing strategies, including random routing, uniform routing, heuristic routing based on cross-layer attention similarity, and the proposed learned routing? All strategies should use the same Anchor ratio (e.g., 20.1%) and identical block-selection configurations. This would help isolate the contribution of the learned refresh schedule from the general benefits of cross-layer index reuse.

Q2. Since Kascade and IndexCache also exploit cross-layer attention redundancy through index reuse, could the authors provide direct comparisons with these closely related methods under comparable experimental settings? In particular, how much additional improvement does the learned refresh schedule provide over their fixed or heuristic reuse strategies in terms of both accuracy and prefill latency?

Q3. The learned Anchor/Reuse schedule is fixed after offline distillation and shared across different inputs and context lengths. How does the effectiveness of this static schedule vary across tasks with different retrieval and reasoning requirements? Could the authors analyze whether the learned routing remains effective when cross-layer attention patterns differ substantially from those observed in the synthetic multi-passkey training data?

Q4. Section 3.1 states that the final query block always performs full attention, even in Reuse slots. Since many long-context benchmarks place the question near the end of the input, to what extent does this special treatment contribute to the reported accuracy? Could the authors provide an ablation in which the final query block also uses cached sparse indices, to better isolate the effectiveness of cross-layer index reuse?

6 comments

We thank the reviewer for the valuable feedback. We address the concerns below.

W1/Q1: Budget-Matched Comparison of Routing Strategies

Following the reviewer's suggestion, we additionally compare random, uniform, heuristic, and learned routing under closely matched Anchor budgets on Llama-3.1-8B-Instruct at 128K. All methods use identical block-selection settings: top-p = 0.9, mmax⁡=48m_{\max}=48, mmin⁡=8m_{\min}=8, and full attention for the final query block.

Routing StrategyConfigurationAnchor SlotsRULER-128K ↑
Random3 seeds (0/1/2)5152.89 ± 4.74
UniformInterval = 64850.07
UniformInterval = 55662.95
HeuristicMass-recall chain5165.49
Learned (Ours)Hard-Concrete5173.60
FlashAttention-2——74.20

Random, heuristic, and learned routing use the same budget of 51 Anchor slots (approximately 20%). Uniform routing operates at layer granularity, so its budget must be a multiple of 8 KV heads. We therefore report the two closest settings, 48 and 56 slots, which bracket the learned schedule's budget. All strategies retain every KV head in layer 0 as an Anchor.

The routing strategies are constructed as follows:

  • Random: The remaining 43 Anchor slots are sampled uniformly at random from layers 1–31, using three random seeds.
  • Uniform: All KV heads refresh every 6 or 5 layers, corresponding to layers {0, 6, ..., 30} or {0, 5, ..., 30}, respectively.
  • Heuristic (mass-recall chain): Following the reviewer's suggestion, we refresh indices when the attention pattern changes significantly. For each KV head, the cached block selection from its most recent Anchor layer aa is reused until its coverage of the current layer ℓ\ell falls below a threshold τ\tau:
\frac{1}{N_{\mathrm{qb}}} \sum_q\sum_{k\in S_{a,h,q}}M_{\ell,h}(q,k).$$ Here, $S_{a,h,q}$ denotes the cached KV-block selection for query block $q$, and $M_{\ell,h}(q,k)$ denotes the exact block-level attention mass. Coverage is averaged over 16 multi-passkey calibration samples (32K–128K), drawn from the same distribution used for ReFresh distillation. Under the same Anchor budget, learned routing outperforms random and heuristic routing by 20.71 and 8.11 points, respectively. It also exceeds uniform routing with 56 Anchor slots by 10.65 points. These results demonstrate that the learned Anchor placement provides substantial improvements beyond the general benefits of index reuse or differences in Anchor allocation. ### W2/Q2: Comparison with IndexCache and Kascade IndexCache was originally proposed for indexer-based sparse attention, and we could not find a complete runnable implementation of its training-free greedy search. We therefore reimplement its greedy layer-search procedure within our block-selection framework, so that the routing schedule is the primary difference from ReFresh. Following IndexCache, the schedule is layer-granular. Each layer is either an F (Full) layer, where all KV heads recompute their block selections, or an S (Shared) layer, where all KV heads reuse the selections from the most recent F layer. Layer 0 is always F. Starting from an all-F model, the greedy search removes one F layer at each step. We use the same 16 multi-passkey calibration samples (32K–128K) as the heuristic baseline. Each forward pass uses the actual ReFresh prefill implementation with identical block-selection settings (top-p = 0.9, $m_{\max}=48$, $m_{\min}=8$, and full attention for the final query block). Because the schedule is layer-granular, the Anchor budget must be a multiple of 8 KV heads. We therefore evaluate the two configurations bracketing ReFresh's 51 Anchor slots: 6 F layers (48 slots) and 7 F layers (56 slots). | Method | Anchor Slots | RULER-128K ↑ | |:---|---:|---:| | ReFresh (Ours) | 51 (20%) | **73.60** | | Greedy Layer Reuse (7 F layers) | 56 (21.9%) | 68.95 | | Greedy Layer Reuse (6 F layers) | 48 (18.75%) | 64.77 | ReFresh outperforms greedy layer reuse by 8.83 points relative to the 6-F-layer configuration. Even when the greedy schedule is given more Anchor slots than ReFresh (56 vs. 51), ReFresh remains 4.65 points higher. These results indicate that the advantage comes from the learned head-level refresh schedule rather than a larger refresh budget. This controlled comparison evaluates the routing strategies under a common implementation rather than the original IndexCache system's end-to-end latency. We also attempt to reproduce Kascade. Our reproduction completes up to 64K context length, with the following RULER results: | Method | 8K | 16K | 32K | 64K | 128K | |:---|---:|---:|---:|---:|---:| | Kascade | 90.79 | 89.45 | 83.15 | 74.83 | — | The 128K run could not be completed under our available environment and GPU memory constraints. Therefore, we do not report a 128K result or an overall average.

Q3: Generalization Across Different Task Types

We clarify that only the Anchor/Reuse routing schedule is static across inputs and context lengths. Specifically, it determines which layer–KV-head slots refresh their indices and which reuse previously selected indices. The actual KV-block indices are computed dynamically for each input by Anchor slots using bounded top-p selection, and the number of selected blocks also adapts to the input.

Although the routing schedule is distilled using synthetic multi-passkey data, we evaluate it across diverse retrieval, reasoning, aggregation, and real-text QA tasks. The following table summarizes RULER-128K results on Llama-3.1-8B-Instruct.

Task TypeReFreshFlashAttention-2
Single-needle retrieval99.599.8
Multi-key / multi-value / multi-query retrieval83.687.2
Multi-hop tracing (VT)74.057.4
Aggregation (CWE, FWE)27.630.0
Real-text QA (SQuAD, HotpotQA)55.855.8

ReFresh maintains quality close to FlashAttention-2 across several task categories, including real-text QA, and outperforms FlashAttention-2 on multi-hop tracing.

The results on LongBench and LongBench v2 further support generalization beyond synthetic retrieval. On Llama-3.1-8B-Instruct, ReFresh achieves 38.38 versus 38.23 for FlashAttention-2 on LongBench, and 28.63 versus 29.22 on LongBench v2. On Qwen3-8B, the corresponding scores are 34.60 versus 35.45 and 31.81 versus 31.01, respectively (Tables 1 and 2).

These results suggest that the learned routing schedule transfers beyond the synthetic multi-passkey data used for distillation, although its relative advantage varies across task types.

Q4: Ablation of Full Attention for the Final Query Block

We thank the reviewer for this suggestion. As described in Section 3.1, ReFresh applies full attention to the final query block in Reuse slots rather than using cached sparse indices. We additionally disable this special treatment, allowing the final query block to use cached indices while keeping all other configurations unchanged.

The following paired ablation results use Llama-3.1-8B-Instruct.

Benchmarkw/o Final-Block Fullw/ Final-Block FullGain
RULER 128K72.2373.60+1.37
LongBench37.9038.38+0.48
LongBench v229.8228.63−1.19

The contribution of full attention for the final query block is benchmark-dependent. It improves performance on RULER 128K and LongBench by 1.37 and 0.48 points, respectively, while reducing LongBench v2 accuracy by 1.19 points. On RULER 128K, the largest improvements occur in multi-key retrieval, while other subtasks exhibit mixed changes.

These results suggest that full attention for the final query block provides an additional safeguard for certain retrieval tasks, but is not the primary source of ReFresh's effectiveness. ReFresh maintains competitive performance even when the final query block uses cached sparse indices in Reuse slots, demonstrating the effectiveness of cross-layer index reuse beyond this special treatment.

Dear Reviewer bzam,

We thank you again for your valuable feedback. If you have any further questions or concerns, we would be happy to address them before the discussion period ends. If our responses have adequately addressed your concerns, we would appreciate it if you could reconsider your assessment.

Reviewer_vvBGOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a new long-context prefilling method that reuses global retrieval across layers by learning anchor and reuse slots. Empirical results across models on three long-context benchmarks show substantial speedups over baselines.

Critical strengths
  1. The method is well motivated and easy to implement efficiently.

  2. The proposed method shows moderate gains on long-context benchmarks with substantial efficiency gains.

Critical weaknesses
  1. My main concern is that Refresh might only achieve substantial gains over baselines on certain tasks where cross-layer redundancy in global retrieval is frequently observed, which might limit the applicable scope of the method. For example, in Table 1, the main gains of Refresh over FlexPrefill and XAttention come from the synthetic set for the Llama-3.1 model, and the synthetic set in LongBench is constructed to be more templated. This might also explain the moderate gains of Refresh over baseline methods across tasks in Table 2. The prevalence of such redundancy is not shown quantitatively either; the only evidence is a single anecdotal RULER example in Figure 1.

  2. The ablation only analyzes one component of the proposed method. Other components, such as the hard concrete gates, should also be examined to evaluate the importance of the full proposed pipeline.

Questions for authors

Table 2: Why is Qwen3-8B so much worse than Llama-3.1-8B on the Code category in LongBench, while it is on par on all other tasks? Are these typos, or potentially degenerate or malformatted responses?

5 comments

We thank the reviewer for the valuable feedback. We address the concerns below.

W1: Generalization and Prevalence of Cross-Layer Redundancy

Generalization beyond synthetic tasks. We agree that ReFresh's relative accuracy gains over sparse baselines are more pronounced on synthetic retrieval tasks. However, ReFresh also maintains quality close to FlashAttention-2 on non-synthetic benchmarks. For example, on Llama-3.1-8B-Instruct, ReFresh achieves 49.40 vs. 48.87 on Single-Doc QA, 41.93 vs. 41.78 on Multi-Doc QA, and 17.79 vs. 17.80 on Summarization (Table 1). On LongBench v2, ReFresh achieves 28.63 vs. 29.22 on Llama-3.1-8B-Instruct and 31.81 vs. 31.01 on Qwen3-8B (Table 2). These results show that the method's applicability is not limited to synthetic retrieval, although its relative advantage varies across tasks.

Quantifying cross-layer redundancy. We agree that the single example in Figure 1 is insufficient to establish prevalence. We therefore conduct a large-scale analysis across all 13 RULER subtasks on Llama-3.1-8B-Instruct, covering 156 prompts and 620,496 sampled (layer, KV head, query block) instances. For each prompt, we uniformly sample approximately 16 query blocks. Each sampled query block is evaluated across 31 layers (ℓ≥1\ell \geq 1) and 8 KV heads.

For each layer ℓ≥1\ell\geq1, we measure the fraction of its exact attention mass covered by reusing the preceding layer's bounded top-p block selection (p=0.9p=0.9, mmin⁡=8m_{\min}=8, mmax⁡=48m_{\max}=48, block size 128).

ContextPromptsInstancesReuse Coverage≥0.8≥0.9Random BaselineExcl. Sink/Local
32K104414,1600.873 ± 0.09581%48%0.2180.506
128K52206,3360.822 ± 0.12363%33%0.0850.453

At 32K and 128K, adjacent-layer reuse captures 87.3% and 82.2% of the attention mass on average, compared with 21.8% and 8.5% for equal-budget random selection, corresponding to 4.0× and 9.7× higher coverage. The effect is consistent across all 13 subtasks, with mean coverage ranging from 0.84–0.91 at 32K and 0.79–0.86 at 128K. After excluding sink and local blocks, substantial nonlocal attention mass remains covered (0.506 and 0.453).

These measurements quantify the prevalence of reusable block-level attention mass across tasks and context lengths, beyond the single example in Figure 1. They measure adjacent-layer reuse potential rather than the complete learned routing schedule.

W2: Ablation of the Learned Refresh Schedule

We thank the reviewer for the suggestion. The original paper already studies the effect of the maximum selected-block budget mmax⁡m_{\max} and fixed-interval routing (Table 3).

Following the reviewers' feedback, we further evaluate different Anchor ratios in our response to Reviewer Bf2R (W1/Q1), and conduct controlled comparisons of random, uniform, heuristic, and learned routing in our response to Reviewer bzam (W1/Q1).

These experiments replace the learned Anchor/Reuse decisions obtained through Hard-Concrete optimization with non-learned routing strategies while keeping the block-selection mechanism unchanged. Under the same 51-slot Anchor budget on Llama-3.1-8B-Instruct at RULER 128K, learned routing achieves 73.60, compared with 52.89 ± 4.74 for random routing and 65.49 for heuristic routing. Uniform routing with 56 Anchors achieves 62.95. We also evaluate an IndexCache-inspired greedy routing strategy in our response to Reviewer bzam (W2/Q2).

Together with the selected-block budget ablation in Table 3 and the final-query-block ablation in our response to Reviewer bzam (Q4), these experiments examine the main design choices of the proposed pipeline.

Hard-Concrete gates provide a differentiable relaxation of binary Anchor/Reuse decisions and enable sparsity-constrained optimization. Our central contribution is learned cross-layer refresh scheduling, rather than a particular gate parameterization. The routing comparisons isolate the benefit of the learned schedule, although they do not constitute a direct comparison among alternative differentiable gate parameterizations.

Q1: Low Code Scores of Qwen3-8B on LongBench

We thank the reviewer for pointing out this discrepancy. We inspected the generated responses and identified a model-specific output-format issue in OpenCompass rather than an issue caused by ReFresh.

OpenCompass's LongBenchCodeSimEvaluator evaluates code completion based on edit similarity of the extracted code line. On LCC and RepoBench-P, Qwen3-8B frequently generates conversational or Markdown preambles instead of directly producing code. Under the short generation budget, these preambles lead to near-zero scores.

A similar issue has been observed in prior OpenCompass-based evaluations. PBS-Attn (arXiv:2510.21270), which evaluates Qwen-2.5-7B-1M using OpenCompass 0.4.2, reports a Code score of 3.91 under Full Attention, compared with 24.77 for Llama-3.1-8B. Our results also show consistently low Code scores across attention methods, confirming that this issue is not specific to ReFresh.

To better assess performance beyond code completion, we additionally report a supplementary Non-Code Average over the remaining 19 LongBench tasks, excluding LCC and RepoBench-P. The original official average remains unchanged. The Non-Code Average is computed as the arithmetic mean of the 19 individual task scores.

ModelMethodNon-Code Avg. ↑
Llama-3.1-8B-InstructFlashAttention-239.57
DuoAttention38.02
FlexPrefill37.93
XAttention37.50
ReFresh39.70
Qwen3-8BFlashAttention-240.99
DuoAttention37.13
FlexPrefill37.20
XAttention40.03
ReFresh40.13

ReFresh achieves the highest Non-Code Average among sparse-attention methods on both models, demonstrating its effectiveness beyond code-completion tasks.

Dear Reviewer vvBG,

We thank you again for your valuable feedback. If you have any further questions or concerns, we would be happy to address them before the discussion period ends. If our responses have adequately addressed your concerns, we would appreciate it if you could reconsider your assessment.

Reviewer_Bf2ROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the cost of prefilling long contexts. It gives each layer and KV head one of two roles. Anchor slots run full attention and select a small set of KV blocks. Reuse slots attend only to the blocks selected by the most recent Anchor for the same KV head. The route map is learned offline with a Hard-Concrete gate and a frozen language model, then exported as a fixed binary map at a target reuse fraction. On Llama-3.1-8B and Qwen3-8B over LongBench, LongBench v2 and RULER, the method stays close to full attention with 80% reuse and reaches a 2.62x prefill speedup at 256K on an H800.

Critical strengths
  1. Simple global reuse: Reuse slots keep global access through cached indices, so the method does not fall back to a local window. The 2.62x prefill speedup at 256K is meaningful, and decoding is unchanged.

  2. Learned placement matters: At a similar anchor budget, the learned map scores 66.60 on RULER 128K, while a fixed interval with 25% anchors scores 59.85 (Table 3). This supports the claim that placement, not just the anchor count, drives the gain.

Critical weaknesses
  1. No matched-cost comparison: Each baseline uses its own recommended configuration, while ReFresh uses 80% reuse, so Figure 3 compares one point per method rather than their Pareto frontiers. The paper also reports only one reuse fraction (0.8) and no quality at 256K, where the speedup is measured.

  2. Missing close baselines: MInference [1] and Kascade [2] target sparse prefill and are not compared. The dense baseline is FlashAttention-2, while FlashAttention-3 [3] is the natural Hopper baseline for the H800 GPU.

  3. Unreported index-cache cost: The paper does not report the size of the per-query-block index cache at 128K or 256K, so the memory side of the efficiency claim is incomplete.

References: [1] Jiang, et al. MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention, NeurIPS, 2024. [2] Deshmukh, et al. Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference, MLSys, 2026. [3] Shah, et al. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision, NeurIPS, 2024.

Questions for authors
  1. Can you report quality and latency for several reuse fractions, and match the prefill cost of each baseline, so the trade-off is compared at the same cost?

  2. What is the quality at 256K, where the 2.62x speedup is measured?

  3. What is the average number of selected KV blocks and the memory cost of the index cache at 256K?

4 comments

We thank the reviewer for the valuable feedback. We address the concerns below. Unless otherwise specified, all additional experiments reported in this rebuttal are conducted on Llama-3.1-8B-Instruct to maintain a consistent evaluation setting across our controlled experiments and ablations.

W1/Q1: Quality–Efficiency Trade-off and Additional Baselines

Quality comparison. Following the reviewer's suggestion, we additionally evaluate different Anchor ratios and sparse-attention baselines. The following table reports RULER scores at different context lengths.

MethodConfigurationRULER ↑ (8K / 16K / 32K / 64K / 128K)
FlashAttention-2—93.71 / 91.59 / 85.07 / 83.67 / 74.20
ReFreshAnchor=20% (default)93.77 / 92.54 / 85.36 / 83.38 / 73.60
ReFreshAnchor=30%93.71 / 92.61 / 85.69 / 83.76 / 74.93
ReFreshAnchor=40%93.71 / 92.48 / 85.75 / 83.70 / 75.41
FlexPrefillγ=0.9593.45 / 91.67 / 87.88 / 83.31 / 74.23
FlexPrefillγ=0.992.87 / 90.84 / 86.23 / 82.64 / 73.45
MInferenceDefault93.52 / 91.39 / 85.00 / 83.79 / 73.12
XAttentionstride=893.45 / 91.36 / 86.89 / 81.60 / 69.98
XAttentionstride=1693.54 / 90.60 / 84.62 / 78.62 / 65.98
DuoAttentionsp=0.2593.76 / 92.15 / 86.63 / 83.50 / 70.42
DuoAttentionsp=0.592.11 / 91.88 / 88.29 / 80.41 / 63.18
KascadeDefault90.79 / 89.45 / 83.15 / 74.83 / —

Prefill efficiency across context lengths. We additionally evaluate prefill latency from 8K to 128K. All measurements use a single H800 GPU with batch size 1. Each entry reports latency in milliseconds and speedup relative to FlashAttention-2 at the same context length.

MethodConfiguration8K16K32K64K128K
FlashAttention-2—285 (1.00×)656 (1.00×)1,707 (1.00×)4,961 (1.00×)16,634 (1.00×)
FlashAttention-3—262 (1.09×)566 (1.16×)1,353 (1.26×)3,524 (1.41×)10,357 (1.61×)
ReFreshAnchor=20%356 (0.80×)650 (1.01×)1,392 (1.23×)3,249 (1.53×)8,174 (2.04×)
ReFreshAnchor=30%362 (0.79×)672 (0.98×)1,494 (1.14×)3,634 (1.37×)9,642 (1.73×)
ReFreshAnchor=40%374 (0.76×)698 (0.94×)1,593 (1.07×)3,974 (1.25×)11,217 (1.48×)
ReFresh+FA3Anchor=20%290 (0.98×)543 (1.21×)1,165 (1.47×)2,624 (1.89×)6,513 (2.55×)
FlexPrefillγ=0.95458 (0.62×)779 (0.84×)1,485 (1.15×)3,334 (1.49×)8,217 (2.02×)
FlexPrefillγ=0.90437 (0.65×)754 (0.87×)1,371 (1.24×)2,947 (1.68×)6,917 (2.40×)
MInferenceDefault1,860 (0.15×)2,388 (0.27×)3,333 (0.51×)5,480 (0.91×)9,405 (1.77×)
XAttentionstride=8306 (0.93×)608 (1.08×)1,377 (1.24×)3,328 (1.49×)9,332 (1.78×)
XAttentionstride=16300 (0.95×)600 (1.09×)1,344 (1.27×)3,179 (1.56×)8,626 (1.93×)
DuoAttentionsp=0.25291 (0.98×)634 (1.04×)1,559 (1.09×)4,263 (1.16×)13,444 (1.24×)
DuoAttentionsp=0.50285 (1.00×)599 (1.10×)1,390 (1.23×)3,522 (1.41×)10,327 (1.61×)

Anchor ratio sensitivity. The results demonstrate a clear quality–efficiency trade-off. With 20% Anchors, ReFresh achieves a 2.04× prefill speedup at 128K while maintaining quality close to FlashAttention-2. Increasing the Anchor ratio to 30% and 40% improves RULER-128K accuracy to 74.93 and 75.41, respectively, while retaining 1.73× and 1.48× speedups.

Matched-cost comparison. ReFresh with 20% Anchors achieves nearly identical 128K prefill latency to FlexPrefill (γ=0.95) (8,174 vs. 8,217 ms). ReFresh with 30% Anchors achieves higher RULER accuracy than MInference (74.93 vs. 73.12) at comparable latency (9,642 vs. 9,405 ms). These results further characterize ReFresh's quality–efficiency trade-off beyond the default configuration.

Additional baselines. We additionally evaluate MInference and attempt to reproduce Kascade. Our Kascade reproduction completes up to 64K context length. The 128K run could not be completed under our available environment and GPU memory constraints; therefore, we do not report a 128K result or an overall average.

W2: Choice of FlashAttention Baseline

We thank the reviewer for pointing out that FlashAttention-3 (FA3) is a more appropriate dense baseline for Hopper GPUs. We therefore additionally implement ReFresh on top of FA3 and compare it against both FA2 and FA3 on H800.

The FA3 implementation replaces both attention paths:

  1. Anchor heads: We replace the original fused Triton kernel with FA3 dense attention followed by a separate block-scoring kernel.
  2. Sparse heads: We replace the Triton indexed sparse-attention kernel with the FA3 paged-KV kernel, using the page table to represent the selected KV blocks.

The resulting ReFresh+FA3 implementation is a reference port without deep kernel fusion. In particular, the Anchor path currently executes separate passes for attention output and block scoring. To isolate the benefit of cross-layer index reuse from the underlying attention kernel, we compare ReFresh+FA3 directly against FA3 using the same FA3 kernel family.

Comparison8K16K32K64K128K
ReFresh+FA3 / FA30.90×1.04×1.16×1.34×1.59×

At 128K, ReFresh+FA3 achieves 1.59× speedup over FA3, demonstrating that the efficiency gains persist even with a Hopper-optimized dense-attention baseline. The benefit increases with context length, while the sparse-attention overhead slightly outweighs the savings at 8K.

The current implementation has not yet fused block scoring into the FA3 attention kernel. Integrating block scoring into FA3 could eliminate the additional QK pass in the Anchor path and further reduce overhead.

Thus, the improvement is not merely an artifact of using FA2 as the baseline: ReFresh provides additional acceleration on top of FA3 through cross-layer index reuse.

Q2: Quality at 256K

Following the reviewer's suggestion, we additionally evaluate RULER at 256K on Llama-3.1-8B-Instruct, sampling 50 examples per RULER subtask.

MethodRULER-256K ↑
FlashAttention-271.18
ReFresh66.94
XAttention65.22

At 256K, ReFresh achieves a RULER score of 66.94, outperforming XAttention by 1.72 points.

256K prefill efficiency. We additionally report 256K prefill latency using a single H800 GPU with batch size 1. Speedup is normalized to FlashAttention-2.

MethodConfiguration256K Prefill (ms) ↓Speedup ↑
FlashAttention-2—59,8971.00×
ReFreshAnchor=20%23,3272.57×
FlexPrefillγ=0.95OOM—
FlexPrefillγ=0.90OOM—
XAttentionstride=826,8392.23×
XAttentionstride=1623,9722.50×

At 256K, ReFresh achieves 2.57× prefill speedup over FlashAttention-2, outperforming both evaluated XAttention configurations. Both FlexPrefill configurations run out of memory.

W3/Q3: Selected KV Blocks and Index Cache Memory

Average selected KV blocks. We count the number of KV blocks read from the index cache for each sparse-head/query-block pair, including the selected top-p blocks and the mandatory sink and diagonal blocks. We randomly select three samples from each of the 13 RULER subtasks, resulting in 39 samples in total. The same set of samples is used for both 128K and 256K context lengths. We first compute the average for each sample across all layers, sparse heads, and query blocks, and then report the mean and standard deviation across the 39 RULER samples.

Context LengthSparse-head Avg. Selected Blocks
128K41.49 ± 1.53
256K45.27 ± 0.74

Index-cache memory. The cache stores at most Kmax⁡=50K_{\max}=50 block indices (48 candidate blocks plus the mandatory sink and diagonal blocks) and one int32 counter for each query-block/KV-head pair. Its memory footprint is:

N_{\mathrm{qb}}\times H_{\mathrm{kv}} \times (K_{\max}+1)\times 4\ \mathrm{Bytes}.$$ For Llama-3.1-8B-Instruct, with 8 KV heads and a block size of 128, the resulting memory costs are: | Context | Index Cache | BF16 KV Cache | Ratio | |:---|---:|---:|---:| | 128K | 1.59 MiB | 16 GiB | 0.0097% | | 256K | 3.19 MiB | 32 GiB | 0.0097% | The index cache is allocated only once and shared across all layers: Anchor slots update the cache, while Reuse slots directly consume the cached indices. Thus, its memory footprint does not grow with the number of Transformer layers. These results show that the additional memory required for cross-layer index reuse is negligible compared with the BF16 KV cache.

Thank you for your response. The additional results and explanations have addressed my concerns; thus, I am increasing my score to 3 to reflect this.

Dear Reviewer Bf2R,

We sincerely appreciate your meticulous evaluation and constructive feedback. We are glad that our additional experiments and clarifications have addressed your concerns. Your opinion has helped us enhance the empirical assessment of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.