- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ReFresh, a hybrid attention framework designed to accelerate long-context prefilling by exploiting cross-layer redundancy in attention patterns. The key observation is that neighboring Transformer layers often attend to similar KV blocks, making repeated global retrieval unnecessary.
ReFresh introduces an Anchor/Reuse mechanism at the layer–KV-head level. Anchor slots perform full attention and identify important KV blocks based on attention probabilities, while Reuse slots reuse the previously selected block indices to perform block-sparse attention using the current layer's queries, keys, and values. To determine where retrieval should be refreshed, the authors propose a learned routing schedule based on Hard-Concrete gates and dense-to-sparse distillation, while keeping the pretrained language model weights frozen. The learned schedule is subsequently converted into a static binary routing map for inference.
The authors evaluate ReFresh on Qwen3-8B and Llama-3.1-8B using LongBench, LongBench v2, and RULER. Experimental results show that ReFresh largely preserves full-attention accuracy with 80% of layer–KV-head slots operating in Reuse mode. Furthermore, ReFresh achieves up to 2.62× prefill speedup over FlashAttention-2 at 256K context length on an NVIDIA H800 GPU.
Critical strengths
- Effective exploitation of cross-layer attention redundancy.
The paper presents a well-motivated Anchor/Reuse mechanism that exploits the redundancy of attention patterns across neighboring Transformer layers. By reusing previously selected KV-block indices, ReFresh avoids redundant retrieval operations and reduces attention computation while preserving access to globally relevant context.
- Practical and lightweight learning-based routing design.
ReFresh learns layer–KV-head-level refresh schedules through dense-to-sparse distillation without modifying the pretrained language model weights. The inference-consistent index propagation and static routing map provide a practical approach to integrating learned refresh scheduling into existing LLM inference pipelines without additional runtime routing decisions.
- Promising accuracy–efficiency trade-off with measurable prefill acceleration.
The paper evaluates ReFresh on two language models across LongBench, LongBench v2, and RULER, demonstrating that it largely preserves full-attention quality with 80% of slots operating in Reuse mode. Furthermore, the implementation achieves up to 2.62× prefill speedup over FlashAttention-2 at 256K tokens on an NVIDIA H800 GPU, demonstrating the practical potential of the proposed approach.
Critical weaknesses
- Insufficient controlled evaluation of the learned refresh schedule.
The paper attributes the effectiveness of ReFresh primarily to its learned Anchor/Reuse placement. However, the ablation study in Table 3 does not provide a sufficiently controlled comparison against alternative routing strategies. Although the learned schedule achieves better accuracy than fixed-interval schedules despite using fewer Anchor slots, the comparisons are conducted under different Anchor budgets, making it difficult to isolate the contribution of the proposed learning-based optimization.
To better demonstrate the effectiveness of learned routing, the authors should compare the following strategies under the same Anchor budget (e.g., 20.1%) and identical block-selection configurations: Random Routing (Randomly select 20.1% of layer–KV-head slots as Anchors), Uniform Routing (Distribute Anchor slots uniformly across layers while maintaining the same 20.1% budget), Heuristic Routing (Determine Anchor placement based on cross-layer attention-pattern similarity, refreshing indices when the attention patterns change significantly), Learned Routing (Use the proposed learned Anchor/Reuse schedule).
Such budget-matched comparisons would more convincingly establish whether the performance gains arise from the learned routing strategy itself rather than the general benefits of cross-layer index reuse or differences in Anchor allocation.
- Missing comparisons with closely related cross-layer index reuse methods.
The paper identifies Kascade and IndexCache as closely related approaches that exploit cross-layer redundancy in sparse attention indices. However, neither method is included in the experimental comparisons. Instead, the evaluation primarily focuses on conventional sparse and hybrid attention baselines, such as FlexPrefill, XAttention, and DuoAttention. Since the central contribution of ReFresh is learning when to refresh cached indices rather than introducing cross-layer index reuse itself, comparisons with existing index-reuse methods are essential to establish its incremental benefits. Without such comparisons, it remains unclear whether the learned refresh schedule offers meaningful improvements over existing fixed or heuristic reuse strategies.
Questions for authors
Q1. Could the authors provide a budget-matched comparison of different Anchor/Reuse routing strategies, including random routing, uniform routing, heuristic routing based on cross-layer attention similarity, and the proposed learned routing? All strategies should use the same Anchor ratio (e.g., 20.1%) and identical block-selection configurations. This would help isolate the contribution of the learned refresh schedule from the general benefits of cross-layer index reuse.
Q2. Since Kascade and IndexCache also exploit cross-layer attention redundancy through index reuse, could the authors provide direct comparisons with these closely related methods under comparable experimental settings? In particular, how much additional improvement does the learned refresh schedule provide over their fixed or heuristic reuse strategies in terms of both accuracy and prefill latency?
Q3. The learned Anchor/Reuse schedule is fixed after offline distillation and shared across different inputs and context lengths. How does the effectiveness of this static schedule vary across tasks with different retrieval and reasoning requirements? Could the authors analyze whether the learned routing remains effective when cross-layer attention patterns differ substantially from those observed in the synthetic multi-passkey training data?
Q4. Section 3.1 states that the final query block always performs full attention, even in Reuse slots. Since many long-context benchmarks place the question near the end of the input, to what extent does this special treatment contribute to the reported accuracy? Could the authors provide an ablation in which the final query block also uses cached sparse indices, to better isolate the effectiveness of cross-layer index reuse?
6 comments
Q3: Generalization Across Different Task Types
We clarify that only the Anchor/Reuse routing schedule is static across inputs and context lengths. Specifically, it determines which layer–KV-head slots refresh their indices and which reuse previously selected indices. The actual KV-block indices are computed dynamically for each input by Anchor slots using bounded top-p selection, and the number of selected blocks also adapts to the input.
Although the routing schedule is distilled using synthetic multi-passkey data, we evaluate it across diverse retrieval, reasoning, aggregation, and real-text QA tasks. The following table summarizes RULER-128K results on Llama-3.1-8B-Instruct.
| Task Type | ReFresh | FlashAttention-2 |
|---|---|---|
| Single-needle retrieval | 99.5 | 99.8 |
| Multi-key / multi-value / multi-query retrieval | 83.6 | 87.2 |
| Multi-hop tracing (VT) | 74.0 | 57.4 |
| Aggregation (CWE, FWE) | 27.6 | 30.0 |
| Real-text QA (SQuAD, HotpotQA) | 55.8 | 55.8 |
ReFresh maintains quality close to FlashAttention-2 across several task categories, including real-text QA, and outperforms FlashAttention-2 on multi-hop tracing.
The results on LongBench and LongBench v2 further support generalization beyond synthetic retrieval. On Llama-3.1-8B-Instruct, ReFresh achieves 38.38 versus 38.23 for FlashAttention-2 on LongBench, and 28.63 versus 29.22 on LongBench v2. On Qwen3-8B, the corresponding scores are 34.60 versus 35.45 and 31.81 versus 31.01, respectively (Tables 1 and 2).
These results suggest that the learned routing schedule transfers beyond the synthetic multi-passkey data used for distillation, although its relative advantage varies across task types.
Q4: Ablation of Full Attention for the Final Query Block
We thank the reviewer for this suggestion. As described in Section 3.1, ReFresh applies full attention to the final query block in Reuse slots rather than using cached sparse indices. We additionally disable this special treatment, allowing the final query block to use cached indices while keeping all other configurations unchanged.
The following paired ablation results use Llama-3.1-8B-Instruct.
| Benchmark | w/o Final-Block Full | w/ Final-Block Full | Gain |
|---|---|---|---|
| RULER 128K | 72.23 | 73.60 | +1.37 |
| LongBench | 37.90 | 38.38 | +0.48 |
| LongBench v2 | 29.82 | 28.63 | −1.19 |
The contribution of full attention for the final query block is benchmark-dependent. It improves performance on RULER 128K and LongBench by 1.37 and 0.48 points, respectively, while reducing LongBench v2 accuracy by 1.19 points. On RULER 128K, the largest improvements occur in multi-key retrieval, while other subtasks exhibit mixed changes.
These results suggest that full attention for the final query block provides an additional safeguard for certain retrieval tasks, but is not the primary source of ReFresh's effectiveness. ReFresh maintains competitive performance even when the final query block uses cached sparse indices in Reuse slots, demonstrating the effectiveness of cross-layer index reuse beyond this special treatment.
Dear Reviewer bzam,
We thank you again for your valuable feedback. If you have any further questions or concerns, we would be happy to address them before the discussion period ends. If our responses have adequately addressed your concerns, we would appreciate it if you could reconsider your assessment.
We thank the reviewer for the valuable feedback. We address the concerns below.
W1/Q1: Budget-Matched Comparison of Routing Strategies
Following the reviewer's suggestion, we additionally compare random, uniform, heuristic, and learned routing under closely matched Anchor budgets on Llama-3.1-8B-Instruct at 128K. All methods use identical block-selection settings: top-p = 0.9, mmax=48, mmin=8, and full attention for the final query block.
Random, heuristic, and learned routing use the same budget of 51 Anchor slots (approximately 20%). Uniform routing operates at layer granularity, so its budget must be a multiple of 8 KV heads. We therefore report the two closest settings, 48 and 56 slots, which bracket the learned schedule's budget. All strategies retain every KV head in layer 0 as an Anchor.
The routing strategies are constructed as follows:
- Random: The remaining 43 Anchor slots are sampled uniformly at random from layers 1–31, using three random seeds.
- Uniform: All KV heads refresh every 6 or 5 layers, corresponding to layers {0, 6, ..., 30} or {0, 5, ..., 30}, respectively.
- Heuristic (mass-recall chain): Following the reviewer's suggestion, we refresh indices when the attention pattern changes significantly. For each KV head, the cached block selection from its most recent Anchor layer a is reused until its coverage of the current layer ℓ falls below a threshold τ:
\frac{1}{N_{\mathrm{qb}}} \sum_q\sum_{k\in S_{a,h,q}}M_{\ell,h}(q,k).$$ Here, $S_{a,h,q}$ denotes the cached KV-block selection for query block $q$, and $M_{\ell,h}(q,k)$ denotes the exact block-level attention mass. Coverage is averaged over 16 multi-passkey calibration samples (32K–128K), drawn from the same distribution used for ReFresh distillation. Under the same Anchor budget, learned routing outperforms random and heuristic routing by 20.71 and 8.11 points, respectively. It also exceeds uniform routing with 56 Anchor slots by 10.65 points. These results demonstrate that the learned Anchor placement provides substantial improvements beyond the general benefits of index reuse or differences in Anchor allocation. ### W2/Q2: Comparison with IndexCache and Kascade IndexCache was originally proposed for indexer-based sparse attention, and we could not find a complete runnable implementation of its training-free greedy search. We therefore reimplement its greedy layer-search procedure within our block-selection framework, so that the routing schedule is the primary difference from ReFresh. Following IndexCache, the schedule is layer-granular. Each layer is either an F (Full) layer, where all KV heads recompute their block selections, or an S (Shared) layer, where all KV heads reuse the selections from the most recent F layer. Layer 0 is always F. Starting from an all-F model, the greedy search removes one F layer at each step. We use the same 16 multi-passkey calibration samples (32K–128K) as the heuristic baseline. Each forward pass uses the actual ReFresh prefill implementation with identical block-selection settings (top-p = 0.9, $m_{\max}=48$, $m_{\min}=8$, and full attention for the final query block). Because the schedule is layer-granular, the Anchor budget must be a multiple of 8 KV heads. We therefore evaluate the two configurations bracketing ReFresh's 51 Anchor slots: 6 F layers (48 slots) and 7 F layers (56 slots). | Method | Anchor Slots | RULER-128K ↑ | |:---|---:|---:| | ReFresh (Ours) | 51 (20%) | **73.60** | | Greedy Layer Reuse (7 F layers) | 56 (21.9%) | 68.95 | | Greedy Layer Reuse (6 F layers) | 48 (18.75%) | 64.77 | ReFresh outperforms greedy layer reuse by 8.83 points relative to the 6-F-layer configuration. Even when the greedy schedule is given more Anchor slots than ReFresh (56 vs. 51), ReFresh remains 4.65 points higher. These results indicate that the advantage comes from the learned head-level refresh schedule rather than a larger refresh budget. This controlled comparison evaluates the routing strategies under a common implementation rather than the original IndexCache system's end-to-end latency. We also attempt to reproduce Kascade. Our reproduction completes up to 64K context length, with the following RULER results: | Method | 8K | 16K | 32K | 64K | 128K | |:---|---:|---:|---:|---:|---:| | Kascade | 90.79 | 89.45 | 83.15 | 74.83 | — | The 128K run could not be completed under our available environment and GPU memory constraints. Therefore, we do not report a 128K result or an overall average.