- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ProbeRefine is an improved version of ProxyAttn. The key difference is how it “probes” attention: it probes the attention matrix from the last query tokens.
Critical strengths
The method remains training-free and simple. Probing with outlier queries makes sense.
Critical weaknesses
- Novelty of the method is limited
- The core sparsification algorithm is mostly from ProxyAttn. The only key difference is head probing, and I do not think this is significant enough to be published. I believe there should be more about probing.
- There is also existing research addressing the nuances of sharing across heads versus accounting for differences between them, neither of which is cited here: specifically, “TokenSelect” (which examines soft voting across heads and head-specific attention patterns) and “SharePrefill” (which focuses on sharing patterns among heads).
- Limited evaluation
- The cited papers and benchmark models are quite old, considering the speed of this kind of research. I strongly suggest updating the models to hybrid attention models, which are the mainstream of SoTA models, e.g., Qwen 3.8, DeepSeek 4.1 Flash, etc.
- Limited efficiency of implementation
- ProbeRefine is even slower than the previous baseline, ProxyAttn.
And here is a critical mistake in the paper:
- Table 6 should be titled Llama 3.1, but it is titled GLM. I am not sure which one is correct, but Tables 1 and 6 have exactly the same numbers while being claimed as different models, which makes no sense.
Questions for authors
Please check weakness section for answer my concerns.
4 comments
Response to Reviewer — Part 2/3: Efficiency and GLM Correction
We thank the reviewer for carefully examining the implementation efficiency and identifying the GLM reporting inconsistency. We clarify the measured trade-offs and the corrections below.
W3. Implementation Efficiency
(1) Runtime limitations and their source. We agree that ProbeRefine is slower than ProxyAttn at some lengths and settings. We retain ProxyAttn's low-cost shared estimation, but head-specific probing, importance aggregation, budget selection, and residual refinement add work. Fig. 5 indicates that the main additional overhead lies in selection, rather than layout construction. Timings include selection, layout, and sparse execution.
Appendix E reports attention-pipeline speedups over a common FlashAttention reference:
| Length | ProxyAttn | ProbeRefine |
|---|---|---|
| 32K | 1.4x | 1.7x |
| 64K | 2.7x | 2.3x |
| 128K | 3.6x | 2.5x |
ProbeRefine is faster at 32K but slower at 64K/128K under these settings; at 128K, the speedup ratio 3.6/2.5 implies approximately 44% higher pipeline latency. At 4K–16K its additional selection work is not yet amortized (Fig. 4). We acknowledge these limitations.
(2) Quality at comparable latency. To evaluate whether the extra cost can improve task quality, we swept ProbeRefine, ProxyAttn and FlexPrefill at five coverage thresholds and two lengths (64K/128K): 30 operating points, each using 416 RULER examples. Latency was measured on NVIDIA A800 with fixed real QKV from four layers, not full-model inference. Additional fine-grained ProxyAttn thresholds were chosen by latency before evaluating quality. Gamma is the cumulative-importance coverage threshold.
| Length | Method | Gamma | RULER | Latency | Active ratio |
|---|---|---|---|---|---|
| 64K | ProbeRefine | 0.950 | 87.26 | 88.74 ms | 0.3649 |
| 64K | ProxyAttn | 0.960 | 84.14 | 87.81 ms | 0.3483 |
| 128K | ProbeRefine | 0.950 | 79.09 | 290.94 ms | 0.3092 |
| 128K | ProxyAttn | 0.965 | 78.85 | 284.40 ms | 0.3374 |
The absolute latency mismatch divided by ProbeRefine latency is 1.05% at 64K and 2.25% at 128K, within our predefined 5% criterion. Relative to ProxyAttn's latency, ProbeRefine obtains +3.12 RULER points for +1.06% latency at 64K, and +0.24 points for +2.30% latency at 128K. We do not claim statistical significance for the smaller 128K gain. These points use different coverage settings from the original 44% comparison.
(3) Scope and optimization opportunities. These comparisons match runtime, not identical active-block budgets. ProbeRefine does not dominate every point: at 128K, ProxyAttn (gamma=0.90) scores 75.00 at 115.31 ms, while ProbeRefine (gamma=0.80) scores 74.28 at 119.87 ms.
In separate paired four-layer tests, swap alone adds only 1.14% (32K) and 0.68% (64K), so the overall gap cannot be attributed to replacement alone. Fig. 5 suggests concrete future optimization opportunities: fusing probe scoring with importance aggregation, reducing intermediate-memory traffic and budget-sorting overhead, and exploring GQA-aware K/V reuse where compatible with head-specific masks. These are future engineering directions, not measured speedups. We will clarify both current limitations and opportunities in the revision.
Critical Reporting Issue: GLM Table 6
We sincerely apologize for a transcription error in Appendix B, Table 6: the three Llama-3.1 rows from Table 1 were inadvertently copied into the GLM table. These duplicated entries are not independent GLM evidence. The corrected GLM-4-9B-Chat-1M RULER results are:
| Method | 4K | 8K | 16K | 32K | 64K | 128K | Avg. |
|---|---|---|---|---|---|---|---|
| FlexPrefill | 92.07 | 91.11 | 89.42 | 91.59 | 86.54 | 83.17 | 88.98 |
| ProxyAttn | 93.27 | 91.35 | 89.90 | 90.87 | 86.54 | 80.77 | 88.78 |
| ProbeRefine | 93.99 | 91.59 | 90.87 | 91.59 | 86.78 | 81.97 | 89.46 |
ProbeRefine has the highest six-length average (89.46), +0.68 over ProxyAttn, while FlexPrefill performs better at 128K (83.17 vs. 81.97). We will replace the duplicated entries and revise Appendix B to reflect these corrected findings.
Response to Reviewer — Part 3/3: Recent Hybrid-Attention Evaluation
We thank the reviewer for suggesting evaluation on recent hybrid-attention architectures. We added Qwen3.5-9B [8] results and an independent diagnostic of head-specific block refinement.
W2. Evaluation on a Recent Hybrid-Attention Model
(1) Model and adaptation. Qwen3.5-9B (2026) combines Gated DeltaNet and Gated Attention. We selected this 9B model to evaluate a recent hybrid architecture up to 128K within available resources. Its 32 text layers include 24 linear-attention and 8 full-attention layers. ProxyAttn [9], FlexPrefill [10], and ProbeRefine replace only the prefill computation of the 8 full-attention layers (zero-based indices: 3, 7, 11, 15, 19, 23, 27, 31). Linear-attention layers and decoding remain unchanged. Dense Attention denotes unmodified Qwen3.5-9B without sparse block selection.
(2) RULER results. All four methods were evaluated at six context lengths. Bold indicates the best results among the three sparse methods, including ties.
| Method | 4K | 8K | 16K | 32K | 64K | 128K | Avg. |
|---|---|---|---|---|---|---|---|
| Dense Attention | 93.41 | 96.56 | 96.40 | 96.87 | 95.89 | 96.05 | 95.86 |
| ProxyAttn [9] | 93.38 | 96.27 | 96.15 | 96.87 | 96.13 | 95.98 | 95.80 |
| FlexPrefill [10] | 92.44 | 96.51 | 95.67 | 96.63 | 96.45 | 96.22 | 95.65 |
| ProbeRefine | 93.02 | 96.51 | 96.32 | 96.87 | 95.92 | 95.86 | 95.75 |
ProbeRefine averages 95.75, close to Dense Attention (95.86), ProxyAttn (95.80), and FlexPrefill (95.65), demonstrating competitive task quality on a recent hybrid backbone.
(3) Independent dense-reference diagnostic. To isolate residual replacement beyond shared ranking and probing, we conducted a paired 64K analysis on 12 RULER examples from four tasks. No Swap and Swap (W=16) shared dense-forward Q/K activations, proxy rankings, Q16 probes, and per-head budgets. Only gated one-for-one replacement differed. Ranking, selection, and swap operated on 128-token blocks; the resulting masks were mapped to 64-token execution blocks for the Triton kernel at head_dim=256, without re-selection. Per-row block counts remained equal before and after mapping.
Metric definition. Held-out Dense Top-2 Recall measures the fraction of the two highest-importance non-protected, causally visible 128-token key blocks, identified using dense causal attention on query positions excluded from probing, retained by the sparse layout. Dense Attention Mass Retained measures the original dense-softmax probability mass covered by selected blocks, accounting for both inserted and removed blocks.
| Diagnostic | No Swap | Swap | Gain (95% bootstrap CI) |
|---|---|---|---|
| Held-out Dense Top-2 Recall | 88.98% | 89.38% | +0.405 pp [0.313, 0.498] |
| Dense Attention Mass Retained | 80.463% | 80.543% | +0.080 pp [0.070, 0.089] |
| Executed 64-token Causal Block Ratio | 53.83% | 53.83% | 0 |
At the unchanged 53.83% execution-block ratio, Swap recovered 304 previously missed Top-2 hits and lost 105 initially retained hits, a net gain of 199. Both diagnostic metrics improved in 7 of 8 full-attention layers; the first showed a small decrease. The confidence intervals use task-stratified, sample-level bootstrap resampling. The execution-block ratio is distinct from the 128-token selection-block ratio and does not imply measured speedup.
(4) Implications. The paired diagnostic provides modest, independent evidence that probe-guided replacement improves head-specific block coverage at an unchanged block budget. Alongside full-model RULER results, it tests which blocks are retained as well as downstream task quality. We will add these results and the adaptation and diagnostic protocols to the revised manuscript.
References
[8] Qwen Team. Qwen3.5: Towards Native Multimodal Agents[EB/OL]. 2026.
[9] Wang Y, He H, Bao S, et al. ProxyAttn: Guided Sparse Attention via Representative Heads[C]//ICLR. 2026.
[10] Lai X, Lu J, Luo Y, et al. FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference[C]//ICLR. 2025.
Dear Reviewer hfiV, We would appreciate your feedback on whether our rebuttal has adequately addressed your concerns. If any questions or concerns remain, we would be happy to provide further clarification and continue the discussion. Thank you for your time and consideration.
Response to Reviewer
Response to Reviewer — Part 1/3: Novelty and Related Work
We thank the reviewer for the careful assessment and helpful references. ProbeRefine builds on ProxyAttn but focuses on head-specific block correction after efficient cross-head sharing.
W1. Novelty and Relation to ProxyAttn
(1) Motivation. MInference [1] and FlexPrefill [2] sparsify sequence-axis interactions while considering head-dependent patterns or budgets. ProxyAttn [3] shares estimates across heads to reduce sparse-index selection cost. We retain this efficient coarse stage.
Yet head redundancy coexists with specialization [4,5]. ProxyAttn assigns per-head budgets, but each head takes a prefix of the shared ranking: budgets determine how many, not independently which, blocks are retained. We call this the shared-ranking bottleneck.
(2) Evidence and method distinction. Fig. 1(a) shows incomplete Proxy–Head and Head–Head Top-K overlap at 32K/64K. Fig. 1(b) shows W=16 covers 23.5%/12.6% of missed head-specific blocks; Fig. 1(c) shows that prefix expansion retains intervening blocks and increases budget.
Scale-aware probes estimate per-head importance for both budgets and candidate correction. Dual-gated one-for-one replacement revises block identities when justified by head-specific evidence. The contribution is a share-then-refine mechanism, not sharing alone.
(3) Experimental evidence. Tables 3 and 5 report controlled RULER probe ablations on Llama-3.1-8B-Instruct:
Outlier-Q16 exceeds Random-Q16 by 2.00 at 32K. At 128K, Bimodal-Q32 (24 outlier + 8 representative) exceeds Outlier-Q32 by 0.48 at equal probe count; at 64K, Outlier-Q16 is best. Thus, probe selection and composition matter.
In Table 4, swap improves probe-defined remote Top-2 recall by 4.64/2.95 pp at 32K/64K with nearly unchanged active ratios. Its isolated macro-average gains are modest (+0.24/0.00); we do not attribute all framework gains to swap. Relative to ProxyAttn, ProbeRefine gains 1.86/0.86/0.97 RULER points and 0.10/0.51/0.12 InfiniteBench points on Qwen2.5/Llama-3.1/Yi.
W1. Relation to Prior Work
We appreciate the suggestions of TokenSelect [6] and SharePrefill [7]. TokenSelect uses per-head importance and soft voting for token-level KV-cache selection; we address block-sparse prefill, correcting blocks after shared ranking. SharePrefill shares prefill sparse patterns among similar heads; our focus is the remaining within-group ranking disagreement and budget-preserving correction.
In the revised manuscript, we will cite both and expand Related Work with further recent studies on head sharing, heterogeneity, head-aware selection, and long-context attention, clarifying our contribution.
References
[1] Jiang H, Li Y, Zhang C, et al. MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention[C]//NeurIPS. 2024.
[2] Lai X, Lu J, Luo Y, et al. FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference[C]//ICLR. 2025.
[3] Wang Y, He H, Bao S, et al. ProxyAttn: Guided Sparse Attention via Representative Heads[C]//ICLR. 2026.
[4] Michel P, Levy O, Neubig G. Are Sixteen Heads Really Better than One?[C]//NeurIPS. 2019.
[5] Voita E, Talbot D, Moiseev F, et al. Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned[C]//ACL. 2019.
[6] Wu W, Pan Z, Fu K, et al. TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection[C]//EMNLP. 2025.
[7] Peng D, Fu Z, Ye Z, et al. Accelerating Prefilling for Long-Context LLMs via Sparse Pattern Sharing[EB/OL]. arXiv:2505.19578, 2025.