- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a training-free context-window extension method called PickPack. PickPack composes two existing operators inside a single FlashAttention prefill pass, i.e., PACK and PICK. The authors claim their contribution is addressing both the out-of-distribution problem of training-free extension and the quadratic prefill cost. On Qwen2.5/3/3.5/3.8 models at 128K–4M tokens (up to 16× the trained window), PickPack is reported to be 3.4–11 points more accurate than the YARN recipes shipped with released models and up to 1.2× faster.
Critical strengths
-
The in-window simulation error E_sim is a clever way to obtain a ground-truth reference for a setting that has none, and the result that PICK reduces it by 11–21% directly supports the entropy-dilution hypothesis.
-
The evaluation is comprehensive (alough only one model family) including eight model sizes, base and post-trained checkpoints, five released long-context recipes, three benchmarks, and lengths up to 4M.
Critical weaknesses
-
The paper claim 1.09–1.24× efficiency gain at ~50% tile sparsity. Fifty percent sparsity converting to ~1.2× means the skipped work is only about 17% of attention time. In serving economics this is a 15–20% cost reduction at best, which maybe meaningful but incremental, and far below what pattern-prior prefill sparsity or trained sparse attention achieves.
-
The accuracy gains are mostly attributable to PACK, not PICK, and this part actually attribute to previouse paper not this one.
-
Table 6 shows that on the post-trained Qwen3-8B, 60–78% sparsity costs 1.3–3.9 RULER-13 points while delivering 1.24–1.33×. Since production deployments run instruct models, the practical operating point is not the headline's faster and more accurate.
-
The paper motivates extension with agentic, multi-turn, token-stream workloads, yet decode and the KV cache are entirely untouched
Questions for authors
- What is the end-to-end latency (prefill + decode) improvement in a multi-turn agentic session at 256K–1M context, relative to the shipped recipe?
3 comments
Agentic motivation and decode
The extension to decoding is straightforward. As part of the rebuttal, we implemented decoding for PICK. A decode step consists of a single query row attending to the cached keys under the same PACK positions. The performance evaluation is detailed below.
We measured end-to-end request latency, including prefill at different context lengths followed by 8K generated tokens, for Qwen3-8B on an H100, comparing against dense attention under the same position map.
| Context | Prefill, dense (s) | Prefill, PickPack (s) | Prefill speedup | Decode, dense (s) | Decode, PickPack (s) | Decode speedup | Total, dense (s) | Total, PickPack (s) | Total speedup |
|---|---|---|---|---|---|---|---|---|---|
| 128K | 15.2 | 12.5 | 1.21× | 98.3 | 93.6 | 1.05× | 113.5 | 106.2 | 1.07× |
| 256K | 54.4 | 43.2 | 1.26× | 134.4 | 122.0 | 1.10× | 188.8 | 165.2 | 1.14× |
| 512K | 197.1 | 158.4 | 1.24× | 217.2 | 182.4 | 1.19× | 414.3 | 340.8 | 1.22× |
| 1M | 759.9 | 609.0 | 1.25× | 357.2 | 304.8 | 1.17× | 1117.1 | 913.7 | 1.22× |
Each turn in a multi-turn session consists of prefill over the newly added tokens followed by decoding. Therefore, we expect similar per-turn savings, within the range observed above.
We will include the decoding implementation details and these end-to-end results in the revised paper.
Dear Reviewer,
We hope we have adequately addressed your concerns through our responses. As we approach the end of the discussion period, we kindly urge you to consider our clarifications. Please let us know if there are any remaining concerns that we can address. If our responses have alleviated your concerns, we would appreciate it if you would consider revisiting your score and confidence.
Thank you again for your time and consideration.
Novelty
PickPack is a general framework for improving length generalization beyond a model’s context window. It builds on state-of-the-art research in positional methods for context extension and sparse attention, improving both the quality and efficiency of context extension. We believe the simplicity of the proposal is an asset. Our contributions are as follows:
By separating PICK and PACK, we can independently analyze the effects of the two operations and clearly understand how they work together to improve context-extension quality. Prior methods have either focused on positioning alone (e.g., DCA and JetLong) or combined sparsity and positioning (e.g., infLLM, ReAttention, and InfiniteHiP), making their individual effects difficult to disentangle. Our formulation outperforms these baselines while allowing us to isolate and study each component.
This separation also enables us to validate our hypothesis that sparsity is useful not only for improving performance through speedups, but also for improving model quality. To our knowledge, this observation has not previously been demonstrated through a conclusive evaluation, and it provides a valuable insight for the community.
PickPack allows advances in PICK and PACK to be developed independently while providing a seamless pathway for incorporating improved variants into the overall context-extension framework.
Our instantiation, PICK = BLASST, applies sparsity to repositioned attention weights. Unlike methods such as ReAttention, this design integrates cleanly with FlashAttention without additional overhead. Other sparse methods, such as infLLM and ReAttention, first determine the sparsity pattern and then launch an additional kernel to compute attention. Better choices for PICK may exist, and we plan to explore them in future work.
Lastly, and most importantly, our results demonstrate substantial improvements over previous baselines on the efficiency-quality pareto, both in the original paper and in our newly added comparisons with additional baselines.
Why 50% tile sparsity gives about 1.2×
The PickPack algorithm is generic and can be instantiated with different PICK operations. However, the speedups reported in the paper correspond to the particular sparsity algorithm used in our implementation, namely BLASST. BLASST still performs O(N2) computation: it computes QK⊤ for each tile and compares the resulting scores against a threshold to determine whether the tile should be included in the attention computation. Consequently, it only avoids the subsequent softmax and value matrix multiplication for skipped tiles. Given this, the observed improvements are reasonable, and we expect a better sparsity algorithm to unlock substantially larger gains. More importantly, 1.2x improvement in latency is a big win for real serving systems.
Regarding pattern-prior approaches (which we understand to mean fixed sparsity patterns), while they can be faster, they generally do not approximate attention well at inference time, as demonstrated in the sparse attention literature (e.g., MInference [Jiang et al., 2024]). Such patterns could potentially be learned through training. However, trained pattern-prior attention, or more generally trained sparse attention, is optimized for a particular training window. Whether such attention can reliably generalize beyond that window and support context extensibility remains an open question that warrants further investigation.
In any case, the central goal of PickPack is to provide a training-free, generic mechanism for improving both the quality and performance of long-context processing, including at context lengths beyond the model's training window.