- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ReScope, a training-free sparse prefill method that combines segment-wise query observations with iterative refinement of temporary tail queries. The refined queries provide additional signals for identifying important attention candidates, while final sparse attention operates on the original , , and . The method also introduces GPU-level optimizations for candidate selection and index construction.
Experiments on synthetic retrieval, HotpotQA, NarrativeQA, and constructed 128K QASPER workloads demonstrate improvements in sparse-attention quality, particularly for question-first layouts. The paper reports F1 improvements of 3.0--4.9 points over sparse controls on natural tasks and approximately 15--20% lower prefill latency at selected operating points.
Critical strengths
S1. Meaningful query refinement beyond expanded observations. ReScope addresses an important challenge in representative-query sparse attention, particularly when task-relevant information is distributed across long contexts. The combination of segment-wise observations and iterative temporary query updates offers a principled way to improve candidate selection without modifying the original attention tensors. The comparisons against expanded original-query sampling, candidate replacement, and value perturbation provide useful evidence that query refinement contributes additional selection information beyond simply increasing observation coverage.
S2. Practical design and careful experimental analysis. The method requires no additional training and retains the original , , and full KV cache, making it compatible with existing model weights and attention implementations. The paper also pays attention to practical execution costs through GPU-level selection and index-construction optimizations. The inclusion of matched-support comparisons, paired statistical analyses, complete-prefill measurements, and results across different query layouts makes the empirical study informative.
Critical weaknesses
W1. Insufficient isolation of the efficiency gains attributable to query refinement. Although the optimized implementation reports promising latency improvements, the comparison in Table 15 uses selected optimized ReScope configurations against earlier baseline measurements. This makes it difficult to determine how much of the observed speedup comes from the refinement mechanism itself, as opposed to implementation optimizations or operating-point selection. Moreover, the original budget sweep in Tables 5--6 shows that ReScope does not consistently dominate simpler sparse alternatives in quality--latency trade-offs. A unified quality--latency comparison under comparable optimization settings would make the practical efficiency contribution more convincing.
W2. More evidence is needed to establish the consistency and magnitude of the refinement benefits. The natural-task results are encouraging, but the improvements vary across datasets, query layouts, models, and calibration settings. For example, the transfer results in Table 7 do not consistently favor ReScope over expanded original-query observations. Furthermore, some controlled comparisons in Appendix H report improvement thresholds rather than complete effect sizes and uncertainty estimates, making it difficult to assess the magnitude and consistency of the refinement contribution. More systematic analysis would help clarify when the additional refinement computation is most beneficial.
Questions for authors
Q1. Efficiency attribution. Could the authors provide a quality--latency Pareto comparison among ReScope, segment observations, expanded-query sampling, and FlexPrefill under comparable implementation optimizations? In particular, how much of the optimized ReScope speedup can be attributed to query refinement rather than shared kernel optimizations or budget selection?
Q2. Refinement contribution. Could the authors provide the full numerical results underlying the controlled comparisons in Appendix H, including effect sizes, confidence intervals, and experimental settings? It would be helpful to understand how consistently the refinement mechanism improves selection quality at matched support or selector cost.
Q3. Robustness and operating conditions. What factors explain the variations in ReScope's relative performance across natural tasks, models, and calibration settings, particularly the cases in Table 7 where expanded original-query observations perform comparably or better? Are there identifiable characteristics of the input distribution, attention patterns, or support budget that predict when refinement is most useful? Additionally, could the authors clarify the experimental configuration differences responsible for the quality changes between original and optimized ReScope in Table 2?