acceptodds
Under review as a conference paper at ICLR 2027

PickPack: Scaling Context Extension to Millions of Tokens with Sparse Attention

Abstract

Due to a growing range of applications, long-context processing and long-form generation with AI are becoming increasingly common, particularly with the transition to agentic workloads. Agents are long-lived, continuously produce and consume streams of tokens, and base their subsequent responses on their entire interaction history; multi-agent systems can further amplify this token volume. Models capable of processing long contexts are therefore increasingly necessary, but training with larger context windows is expensive, so most models are trained with windows of 32K to 256K tokens. Application demands can extend to millions of tokens, which creates a need for training-free context extension. Training-free context extension presents two key challenges: (1) out-of-distribution context embeddings, where unseen context lengths can shift attention distributions away from those seen during training, and (2) computational complexity, since attention's quadratic cost makes processing millions of tokens expensive. We propose , which addresses both challenges with two composable operators in a single FlashAttention pass: maps every query–key pair to a relative position inside the trained window, and uses the attention scores to dynamically decide which key blocks to skip. At 50% sparsity, improves accuracy over dense attention under the same position map on seven of eight Qwen3 model sizes and context lengths, while being up to 1.2 faster. Against the recipes that released Qwen2.5, Qwen3.5, and Qwen3.8 models ship with, it is 3.4–11 points more accurate and up to 1.2 faster from 128K to one million tokens, and 11.6 points more accurate at four million tokens, 16 the trained window of Qwen3.5-9B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long-context inference, context extension, sparse attention, training-free, efficient inference, large language models, block-sparse attention, million-token context

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Gw8hOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a training-free context-window extension method called PickPack. PickPack composes two existing operators inside a single FlashAttention prefill pass, i.e., PACK and PICK. The authors claim their contribution is addressing both the out-of-distribution problem of training-free extension and the quadratic prefill cost. On Qwen2.5/3/3.5/3.8 models at 128K–4M tokens (up to 16× the trained window), PickPack is reported to be 3.4–11 points more accurate than the YARN recipes shipped with released models and up to 1.2× faster.

Critical strengths
  • The in-window simulation error E_sim is a clever way to obtain a ground-truth reference for a setting that has none, and the result that PICK reduces it by 11–21% directly supports the entropy-dilution hypothesis.

  • The evaluation is comprehensive (alough only one model family) including eight model sizes, base and post-trained checkpoints, five released long-context recipes, three benchmarks, and lengths up to 4M.

Critical weaknesses
  • The paper claim 1.09–1.24× efficiency gain at ~50% tile sparsity. Fifty percent sparsity converting to ~1.2× means the skipped work is only about 17% of attention time. In serving economics this is a 15–20% cost reduction at best, which maybe meaningful but incremental, and far below what pattern-prior prefill sparsity or trained sparse attention achieves.

  • The accuracy gains are mostly attributable to PACK, not PICK, and this part actually attribute to previouse paper not this one.

  • Table 6 shows that on the post-trained Qwen3-8B, 60–78% sparsity costs 1.3–3.9 RULER-13 points while delivering 1.24–1.33×. Since production deployments run instruct models, the practical operating point is not the headline's faster and more accurate.

  • The paper motivates extension with agentic, multi-turn, token-stream workloads, yet decode and the KV cache are entirely untouched

Questions for authors
  • What is the end-to-end latency (prefill + decode) improvement in a multi-turn agentic session at 256K–1M context, relative to the shipped recipe?
3 comments

Novelty

PickPack is a general framework for improving length generalization beyond a model’s context window. It builds on state-of-the-art research in positional methods for context extension and sparse attention, improving both the quality and efficiency of context extension. We believe the simplicity of the proposal is an asset. Our contributions are as follows:

  1. By separating PICK and PACK, we can independently analyze the effects of the two operations and clearly understand how they work together to improve context-extension quality. Prior methods have either focused on positioning alone (e.g., DCA and JetLong) or combined sparsity and positioning (e.g., infLLM, ReAttention, and InfiniteHiP), making their individual effects difficult to disentangle. Our formulation outperforms these baselines while allowing us to isolate and study each component.

  2. This separation also enables us to validate our hypothesis that sparsity is useful not only for improving performance through speedups, but also for improving model quality. To our knowledge, this observation has not previously been demonstrated through a conclusive evaluation, and it provides a valuable insight for the community.

  3. PickPack allows advances in PICK and PACK to be developed independently while providing a seamless pathway for incorporating improved variants into the overall context-extension framework.

  4. Our instantiation, PICK = BLASST, applies sparsity to repositioned attention weights. Unlike methods such as ReAttention, this design integrates cleanly with FlashAttention without additional overhead. Other sparse methods, such as infLLM and ReAttention, first determine the sparsity pattern and then launch an additional kernel to compute attention. Better choices for PICK may exist, and we plan to explore them in future work.

  5. Lastly, and most importantly, our results demonstrate substantial improvements over previous baselines on the efficiency-quality pareto, both in the original paper and in our newly added comparisons with additional baselines.

Why 50% tile sparsity gives about 1.2×

The PickPack algorithm is generic and can be instantiated with different PICK operations. However, the speedups reported in the paper correspond to the particular sparsity algorithm used in our implementation, namely BLASST. BLASST still performs O(N2)O(N^2) computation: it computes QK⊤QK^\top for each tile and compares the resulting scores against a threshold to determine whether the tile should be included in the attention computation. Consequently, it only avoids the subsequent softmax and value matrix multiplication for skipped tiles. Given this, the observed improvements are reasonable, and we expect a better sparsity algorithm to unlock substantially larger gains. More importantly, 1.2x improvement in latency is a big win for real serving systems.

Regarding pattern-prior approaches (which we understand to mean fixed sparsity patterns), while they can be faster, they generally do not approximate attention well at inference time, as demonstrated in the sparse attention literature (e.g., MInference [Jiang et al., 2024]). Such patterns could potentially be learned through training. However, trained pattern-prior attention, or more generally trained sparse attention, is optimized for a particular training window. Whether such attention can reliably generalize beyond that window and support context extensibility remains an open question that warrants further investigation.

In any case, the central goal of PickPack is to provide a training-free, generic mechanism for improving both the quality and performance of long-context processing, including at context lengths beyond the model's training window.

Agentic motivation and decode

The extension to decoding is straightforward. As part of the rebuttal, we implemented decoding for PICK. A decode step consists of a single query row attending to the cached keys under the same PACK positions. The performance evaluation is detailed below.

We measured end-to-end request latency, including prefill at different context lengths followed by 8K generated tokens, for Qwen3-8B on an H100, comparing against dense attention under the same position map.

ContextPrefill, dense (s)Prefill, PickPack (s)Prefill speedupDecode, dense (s)Decode, PickPack (s)Decode speedupTotal, dense (s)Total, PickPack (s)Total speedup
128K15.212.51.21×98.393.61.05×113.5106.21.07×
256K54.443.21.26×134.4122.01.10×188.8165.21.14×
512K197.1158.41.24×217.2182.41.19×414.3340.81.22×
1M759.9609.01.25×357.2304.81.17×1117.1913.71.22×

Each turn in a multi-turn session consists of prefill over the newly added tokens followed by decoding. Therefore, we expect similar per-turn savings, within the range observed above.

We will include the decoding implementation details and these end-to-end results in the revised paper.

Dear Reviewer,

We hope we have adequately addressed your concerns through our responses. As we approach the end of the discussion period, we kindly urge you to consider our clarifications. Please let us know if there are any remaining concerns that we can address. If our responses have alleviated your concerns, we would appreciate it if you would consider revisiting your score and confidence.

Thank you again for your time and consideration.

Reviewer_HEnZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a training-free method, called PickPack, that combines methods for context extension (Pack) with strategies for sparse prefill (Pick). More concretely, Pack remaps query-key positions back to the original model's training positional range via agnostic functions ϕ(i,j)\phi(i, j) that can be applied independently for queries and keys, while Pick applies a block skipping strategy inside FlashAttention to sparsify the computation and accelerate prefill. Experiments are carried under many different choices for Pack algorithms and different model sizes on RULER, LOFT, and PG19. The results show better quality-speed tradeoffs, with substantial improvements on RULER and up to 1.2x prefill speedup.

Critical strengths
  • I really like the simplicity and practicality of the idea: combining position remapping and sparse selection using the same mapped attention scores is useful in practice
  • The separation into PICK and PACK also makes the method composable with different position maps
  • The experimental coverage is good: the paper evaluates several model sizes and families, includes both base and post-trained checkpoints, and tests retrieval, real-corpus question answering, and language modeling
  • The comparisons with dense attention under the same PACK map help isolate the effect of sparsity
  • The related work is particularly well written and makes the connections to prior methods clear
Critical weaknesses
  • My major concern is that the methodological novelty seems somewhat limited. The best configuration combines Jet-Long's position map for Pack with BLASST's skipping rule for Pick. Both strategies come from previous papers. The novelty relies on the composable formulation and integration. While that is useful, the paper should more clearly identify the contribution beyond combining these existing components.
  • I think that experimenting with closely related methods would also strengthen the paper, especially methods that already combine selection and position remapping, e.g., InfiniteHiP and ReAttention.
  • The threshold λ\lambda is calibrated separately for each task and length using the evaluation prompts. Although no labels are used, it remains unclear how a threshold calibrated on specific data transfers to unseen tasks. Also, it is unclear how much the calibration phase actually costs.
  • The method also accelerates only prefill, while decoding retains the full-cache cost. That is ok, but I think this scope should be clearer from the introduction (which seems a bit off to me given the motivation around long-running agents).
  • It would also be valuable to lay out more kernel implementation details along with more workload timings .

Quick note: sparsity does not consistently improve accuracy: on the post-trained Qwen3-8B, the default target loses approximately 1.32 and 1.60 RULER points at 64K and 128K. These can still be reasonable quality-speed trade-offs, but should be described as such.

Questions for authors
  • Can you compare Pick with a matched, training-free temperature-scaling control under the same position map?
  • Can you provide more implementation details?
4 comments

Novelty

PickPack is a general framework for improving length generalization beyond a model’s context window. It builds on state-of-the-art research in positional methods for context extension and sparse attention, improving both the quality and efficiency of context extension. We believe the simplicity of the proposal is an asset. Our contributions are as follows:

  1. By separating PICK and PACK, we can independently analyze the effects of the two operations and clearly understand how they work together to improve context-extension quality. Prior methods have either focused on positioning alone (e.g., DCA and JetLong) or combined sparsity and positioning (e.g., infLLM, ReAttention, and InfiniteHiP), making their individual effects difficult to disentangle. Our formulation outperforms these baselines while allowing us to isolate and study each component.

  2. This separation also enables us to validate our hypothesis that sparsity is useful not only for improving performance through speedups, but also for improving model quality. To our knowledge, this observation has not previously been demonstrated through a conclusive evaluation, and it provides a valuable insight for the community.

  3. PickPack allows advances in PICK and PACK to be developed independently while providing a seamless pathway for incorporating improved variants into the overall context-extension framework.

  4. Our instantiation, PICK = BLASST, applies sparsity to repositioned attention weights. Unlike methods such as ReAttention, this design integrates cleanly with FlashAttention without additional overhead. Other sparse methods, such as infLLM and ReAttention, first determine the sparsity pattern and then launch an additional kernel to compute attention. Better choices for PICK may exist, and we plan to explore them in future work.

  5. Lastly, and most importantly, our results demonstrate substantial improvements over previous baselines on the efficiency-quality pareto, both in the original paper and in our newly added comparisons with additional baselines.

Comparison with selection-plus-remapping baselines

We added Llama-3-8B-Instruct, a different family with a trained context window of 8K tokens (4× smaller than Qwen3’s 32K) and a different RoPE base. We extend it to 32K and 64K, i.e., 4× and 8× its trained window. PICK improves over dense PACK_JL there by +8.1 and +5.2 points at 32K and 64K, respectively.

Llama-3-8B-InstructLMK2VTFWEQA1RULER-HARD-4 Mean
InfLLM32K10.034.050.022.029.0
ReAttention32K14.044.082.720.040.2
PACK_JL (dense)32K56.055.257.356.056.1
PICK ∘ PACK_JL (50%)32K64.062.470.760.064.3
InfLLM64K4.018.066.78.024.2
ReAttention64K2.042.086.714.036.2
PACK_JL (dense)64K22.038.846.756.040.9
PICK ∘ PACK_JL (50%)64K28.043.654.758.046.1

Observe,

  1. PickPack improves over the existing baselines on average (We do these comparisons on llama since original code of these methods provide support for llama). ReAttention is stronger only on frequent-word extraction, an aggregation task that favours its coarse token selection. This implies there is scope for improvement in PickPack which we leave for future work.

  2. Even in Llama family of models, PICK leads to improvements over dense PACK alone.

Threshold calibration: Transfer and Cost.

The calibration target is the desired sparsity, so calibration does not require labels.

  • RULER: For each task, the threshold is fitted using the first two prompts and then evaluated on all 50 prompts. The achieved sparsity across the 50 prompts stays within 0.492–0.509 at both 64K and 128K.
  • PG19: The threshold is fitted on two books and achieves a sparsity of 0.509–0.510 across all 100 books.

Calibration uses a log-space secant method to search for the appropriate threshold factor. Each attempt requires only two prefills, with no decoding, and most targets are reached within one or two attempts. Thus, calibration incurs only a small fraction of the cost of evaluating the corresponding task. We will report these calibration costs explicitly in Appendix A.

Decoding

We have also implemented PICK for decoding, extending it directly from the prefill implementation. A decode step consists of a single query row attending to the cached keys under the same PACK positions. PICK's test requires only that row's mapped scores on the current tile and its running maximum. Thus, the same rule applies inside the decode attention loop without modification.

Measured on Qwen3-8B with full requests of 8K generated tokens, PICK skips 66% of tiles during decoding and achieves 1.05–1.19× decoding speedup across context lengths from 128K to 1M tokens. As expected, decoding exhibits improvements consistent with those observed during prefill. Combined with PickPack prefill, a full request is 1.07–1.22× faster than dense attention under the same map.

ContextPrefill, dense (s)Prefill, PickPack (s)Prefill speedupDecode, dense (s)Decode, PickPack (s)Decode speedupTotal, dense (s)Total, PickPack (s)Total speedup
128K15.212.51.21×98.393.61.05×113.5106.21.07×
256K54.443.21.26×134.4122.01.10×188.8165.21.14×
512K197.1158.41.24×217.2182.41.19×414.3340.81.22×
1M759.9609.01.25×357.2304.81.17×1117.1913.71.22×

We will describe the decode path and these results in the revision.

Implementation Details and Kernel Profile

We implement PickPack with PACK = JL and PICK = BLASST

Kernels. With PACK_JL as the PACK recipe, Jet-Long (Tang et al., 2026) partitions each query's keys into two regions: a local window comprising the w0 most recent tokens, which are attended to at their native positions, and a far region comprising all earlier keys, which are attended to at grouped positions ⌊i/G⌋\lfloor i/G \rfloor and ⌊j/G⌋\lfloor j/G \rfloor. The local window is computed densely. PICK is applied only to the far-region attention call, inside the Hopper BLASST kernel (TensorRT-LLM's skip-softmax FMHA, using 64 × 128 tiles). The outputs of the two regions are merged exactly using their log-sum-exp statistics (Eq. 16).

Applying PACK. Under PACK_JL, the mapped positions depend independently on the query and key indices ⌊i/G⌋\lfloor i/G \rfloor and ⌊j/G⌋\lfloor j/G \rfloor, respectively). Therefore, RoPE is applied to queries and keys at their mapped positions before invoking the kernel. The attention kernel itself remains unchanged.

Threshold. The threshold is set to =F/N_far, where N_far is the number of keys in the far region.

What a skipped tile saves. With BLASST as our PICK, a skipped tile still computes QK, since these scores are needed to make the skip decision. However, it avoids computing the exponentials and the subsequent PV product.

End-to-end prefill profile. We report H100 end-to-end prefill latency for Qwen3-8B, with speedups measured relative to our own kernel at S=0.

Configuration64K128K
Modified Flash Attention, S=05.03 s15.49 s
Modified Flash Attention, S=0.54.45 s (1.13×)12.86 s (1.20×)
Modified Flash Attention, S=0.84.13 s (1.22×)11.56 s (1.34×)

The paper reports speedups relative to our own kernel at S=0, thereby isolating the performance gains attributable to sparsity. We will include this detailed kernel profile and a per-component latency breakdown in the revision.

Temperature Scaling vs. PICK Under the Same Position Map.

We additionally evaluate LogN query scaling [1], a standard training-free attention temperature adjustment, on Qwen3-8B base in the same setting as Table 1 (RULER-13 × 50 rows). LogN scales queries as q←q⋅max⁡(1,ln⁡(p+1)ln⁡W)q \leftarrow q \cdot \max\left(1, \frac{\ln(p+1)}{\ln W}\right). It does not modify the attention kernel. When combining LogN with PICK, we recalibrate PICK's threshold to maintain the target sparsity (achieved sparsity: 0.49–0.51).

Qwen3-8B base, RULER-1364K128K
PACK_JL84.1770.00
PACK_JL + LogN (temperature only)84.2675.32
PICK∘PACK_JL84.7172.05
PICK∘PACK_JL + LogN85.7778.88

The results show that LogN and PICK are complementary: combining the two consistently outperforms either approach alone at both context lengths. In particular, PICK∘PACK_JL + LogN achieves the best accuracy, reaching 85.77 at 64K and 78.88 at 128K.

[1] D. Chiang and P. Cholak. Overcoming a theoretical limitation of self-attention. ACL 2022. LogN scaling is also used in J. Bai et al., Qwen Technical Report, arXiv:2309.16609, 2023.

Sparsity and accuracy on the post-trained Qwen3-8B

We swept the target sparsity from 10% to 80% on Qwen3-1.7B and Qwen3-8B base models at context lengths of 64K and 128K. We evaluated on RULER-4 (defined in §5.1 of our paper) × 100 rows. Each cell reports the change in accuracy relative to dense PACK_JL, along with one paired standard error, and the H100 prefill speedup relative to S = 0.

S1.7B 64K1.7B 128K8B 64K8B 128K
0 (acc.)63.65 ± 1.9047.27 ± 1.7582.33 ± 1.5771.18 ± 1.85
0.1−1.88 ± 1.06, 1.02×+1.32 ± 0.76, 1.05×−0.05 ± 0.61, 1.03×−0.83 ± 0.85, 1.05×
0.2−0.88 ± 1.08, 1.05×+0.40 ± 0.78, 1.09×+0.83 ± 0.52, 1.06×+0.95 ± 0.83, 1.08×
0.3+0.07 ± 1.07, 1.08×+0.68 ± 0.83, 1.13×+0.55 ± 0.59, 1.08×+1.60 ± 0.94, 1.12×
0.4+0.37 ± 1.01, 1.09×+1.43 ± 0.90, 1.17×+1.28 ± 0.74, 1.10×+2.55 ± 1.13, 1.16×
0.5+2.33 ± 1.30, 1.12×+2.62 ± 1.10, 1.21×+2.13 ± 0.96, 1.13×+3.55 ± 1.21, 1.20×
0.6−0.52 ± 1.55, 1.15×+2.15 ± 1.38, 1.26×+2.43 ± 0.89, 1.16×+3.68 ± 1.36, 1.25×
0.7−2.38 ± 1.69, 1.18×+0.30 ± 1.54, 1.30×+1.22 ± 1.08, 1.18×+4.60 ± 1.37, 1.29×
0.8−3.80 ± 1.74, 1.21×−2.00 ± 1.73, 1.36×−1.77 ± 1.16, 1.22×+3.93 ± 1.60, 1.34×

We agree that sparsity does not improve accuracy at every operating point, and we will describe the post-trained results as a quality–speed trade-off. The best sparsity depends on the model and the context length. On the base models, accuracy improves over a range of targets, and the optimum grows with model size and length (50% for 1.7B, 60–70% for 8B)

Dear Reviewer,

We hope we have adequately addressed your concerns through our responses. As we approach the end of the discussion period, we kindly urge you to consider our clarifications. Please let us know if there are any remaining concerns that we can address. If our responses have alleviated your concerns, we would appreciate it if you would consider revisiting your score and confidence.

Thank you again for your time and consideration.

Reviewer_VvcCOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes PickPack, a training-free method for extending the context window of RoPE models to millions of tokens. It composes two operators inside a single FlashAttention pass: PACK remaps every query–key relative position into the model’s trained window (using recipes such as Jet-Long or DCA), while PICK dynamically skips key blocks whose mapped attention scores fall far below the running maximum (following a BLASST-style test). The authors argue that the combination addresses both out-of-distribution rotary phases and the quadratic cost of long prefill. Experiments on Qwen-family models show that 50% sparsity under the same position map improves accuracy over dense attention on most settings while being up to 1.2× faster, and that PickPack outperforms the official released recipes of Qwen2.5/3.5/3.8 by several points from 128K to 4M tokens.

Critical strengths

The dual challenge (OOD embeddings + quadratic cost) is clearly articulated and practically important for agentic and million-token workloads. Combining a position remapping with score-driven block skipping in one FlashAttention kernel is a clean, hardware-aware design. The formalization of PACK recipes (Definition 1) and FlashAttention-compatible PICK recipes (Definition 2) is precise. Proposition 1 correctly shows that the Jet-Long grouping map is a valid PACK recipe, and the composition is described as decoupled, which is a useful property. The empirical claims are strong and relevant. Gains over dense attention under the same map, improvements relative to Jet-Long and the official YaRN-style recipes of recent Qwen models, and positive results at 1M–4M tokens (including a 11.6-point QAMPARI gain at 4M) indicate practical value. Speed-ups of ~1.1–1.2× at these lengths are meaningful. The in-window simulation (Section 3) provides a reasonable micro-metric for the hypothesis that sparsity reduces attention-output error after position compression, connecting the method to prior observations on attention entropy (LM-Infinite, etc.).

Critical weaknesses

Novelty is primarily in the efficient composition and the empirical demonstration rather than in the individual components. PACK reuses existing grouping/remapping ideas (Jet-Long, DCA, Self-Extend), and PICK reuses a running-max score threshold already explored in SpargeAttn and BLASST. The paper should more sharply quantify what the joint design adds beyond “apply BLASST-style skipping on top of an existing remapping.” The interaction between the remapped positions and the skip decision is under-analyzed. Because scores are computed under the PACK map, the distribution of scores (and thus which blocks survive the λ threshold) can differ from the native-position case. It is not clear whether the sparsity pattern remains stable or whether certain important distant evidence is systematically dropped after remapping. The in-window simulation is a useful proxy but does not fully address this for true out-of-window lengths. Sparsity is reported mainly at a fixed 50% operating point. Sensitivity to the threshold λ, the resulting sparsity distribution across heads/layers, and whether an adaptive target sparsity would be preferable are not discussed in the main text. The local-window dense path is retained, but the trade-off between local fidelity and remote sparsity is only lightly explored. Model and task coverage is concentrated on the Qwen family. While results at extreme lengths are impressive, broader confirmation on Llama, Mistral, or other architectures (especially those with different RoPE bases or attention variants) would strengthen the generality claim. Comparisons with other recent training-free methods that also combine selection and remapping (e.g., InfLLM, InfiniteHiP) are mentioned in related work but appear limited in the experiments.

Questions for authors

How does the distribution of kept blocks under the PACK map differ from the same PICK rule applied at native positions? Are there systematic biases (e.g., over-pruning of certain distance bands)? Can you report accuracy and latency as a function of the sparsity level (or λ) rather than only at 50%? Is there a sweet spot that varies with context length or model size? In the 4M-token QAMPARI result, what fraction of the gain comes from the PACK remapping alone versus the addition of PICK? An ablation that isolates the two operators at these extreme lengths would be valuable. Does PickPack remain beneficial when the underlying checkpoint has already been trained or continued-pretrained to a longer window (e.g., Qwen2.5-1M), or does the relative gain diminish?

5 comments

We thank the reviewer for their careful assessment, in-depth questions, and support of our work. We hope the responses below address their concerns.

Novelty

PickPack is a general framework for improving length generalization beyond a model’s context window. It builds on state-of-the-art research in positional methods for context extension and sparse attention, improving both the quality and efficiency of context extension. We believe the simplicity of the proposal is an asset. Our contributions are as follows:

  1. By separating PICK and PACK, we can independently analyze the effects of the two operations and clearly understand how they work together to improve context-extension quality. Prior methods have either focused on positioning alone (e.g., DCA and JetLong) or combined sparsity and positioning (e.g., infLLM, ReAttention, and InfiniteHiP), making their individual effects difficult to disentangle. Our formulation outperforms these baselines while allowing us to isolate and study each component.

  2. This separation also enables us to validate our hypothesis that sparsity is useful not only for improving performance through speedups, but also for improving model quality. To our knowledge, this observation has not previously been demonstrated through a conclusive evaluation, and it provides a valuable insight for the community.

  3. PickPack allows advances in PICK and PACK to be developed independently while providing a seamless pathway for incorporating improved variants into the overall context-extension framework.

  4. Our instantiation, PICK = BLASST, applies sparsity to repositioned attention weights. Unlike methods such as ReAttention, this design integrates cleanly with FlashAttention without additional overhead. Other sparse methods, such as infLLM and ReAttention, first determine the sparsity pattern and then launch an additional kernel to compute attention. Better choices for PICK may exist, and we plan to explore them in future work.

  5. Lastly, and most importantly, our results demonstrate substantial improvements over previous baselines on the efficiency-quality pareto, both in the original paper and in our newly added comparisons with additional baselines.

Interaction between the map and the skip decision (W2, Q1).

We ran the ablation regarding which positions to use for deciding sparsity. The following table has same settings as that of Table 1 in the paper (RULER-13 × 50 rows, 50% sparsity):

Qwen3-8B Base, RULER-1364K128K
PACK_JL (dense)84.1770.00
PICK ∘ PACK_JL, decision under the map84.7172.05
PICK ∘ PACK_JL, decision at native positions84.6871.75

While the two decisions reach the same accuracy and both keep PICK's gain over dense PACK_JL, it was interesting to see the sparsity pattern in both cases as per your suggestion. The following table shows the average sparsity in buckets of size 16K at 50% overall sparsity.

Distance (q − k)0–16K16–32K32–48K48–64K64–80K80–96K96–112K112–128K
Decision under the map0.360.500.550.550.570.590.570.55
Decision at native positions0.480.550.540.570.500.400.330.28

We make the following observations:

  1. Under-the-map pruning does not over-prune any distance band. Tokens closer to the query are generally better preserved. Sparsity increases over the first ~30K tokens and then stabilizes at 0.5–0.6 through the beginning of the book. Notably, the attention-sink tile is never skipped. The skipped tiles account for only 4.9% (64K) and 5.4% (128K) of the far attention mass, on average.

  2. Native-position pruning exhibits a systematic distance-dependent sparsity pattern. In particular, the native rule skips a larger fraction of nearby keys, dropping 1.7× (64K) and 2.5× (128K) more attention mass than under-the-map pruning.

Overall, we believe under-the-map sparsity is better suited to our algorithm. It exhibits a natural preference for local tokens, consistent with the locality patterns typically observed in attention, without introducing additional position-dependent biases. In contrast, native-position sparsity exhibits a systematic positional bias that is not aligned with the natural distribution of attention scores

The in-window simulation is a proxy.

We agree. Since there is no real source of truth for attention beyond the context window, we wanted to develop a metric that could serve as a quick test for the quality of our approximation. The in-window simulation is an attempt in this direction. While it reproduces similar congestion patterns in both positional and embedding space, it cannot capture the true near-OOD behavior arising from context extension.

Although we see initial corroboration with downstream results, this metric should be interpreted with caution. We will make this caveat more explicit in the paper. Of course, downstream accuracy beyond the context window remains the primary evidence for evaluating our approach, as it is in our paper.

Sensitivity to and the sweet spot (W3, Q2).

We swept the target sparsity from 10% to 80% on Qwen3-1.7B and Qwen3-8B base models at context lengths of 64K and 128K. We evaluated on RULER-4 (defined in §5.1 of our paper) × 100 rows. Each cell reports the change in accuracy relative to dense PACK_JL, along with one paired standard error, and the H100 prefill speedup relative to S = 0.

S1.7B 64K1.7B 128K8B 64K8B 128K
0 (acc.)63.65 ± 1.9047.27 ± 1.7582.33 ± 1.5771.18 ± 1.85
0.1−1.88 ± 1.06, 1.02×+1.32 ± 0.76, 1.05×−0.05 ± 0.61, 1.03×−0.83 ± 0.85, 1.05×
0.2−0.88 ± 1.08, 1.05×+0.40 ± 0.78, 1.09×+0.83 ± 0.52, 1.06×+0.95 ± 0.83, 1.08×
0.3+0.07 ± 1.07, 1.08×+0.68 ± 0.83, 1.13×+0.55 ± 0.59, 1.08×+1.60 ± 0.94, 1.12×
0.4+0.37 ± 1.01, 1.09×+1.43 ± 0.90, 1.17×+1.28 ± 0.74, 1.10×+2.55 ± 1.13, 1.16×
0.5+2.33 ± 1.30, 1.12×+2.62 ± 1.10, 1.21×+2.13 ± 0.96, 1.13×+3.55 ± 1.21, 1.20×
0.6−0.52 ± 1.55, 1.15×+2.15 ± 1.38, 1.26×+2.43 ± 0.89, 1.16×+3.68 ± 1.36, 1.25×
0.7−2.38 ± 1.69, 1.18×+0.30 ± 1.54, 1.30×+1.22 ± 1.08, 1.18×+4.60 ± 1.37, 1.29×
0.8−3.80 ± 1.74, 1.21×−2.00 ± 1.73, 1.36×−1.77 ± 1.16, 1.22×+3.93 ± 1.60, 1.34×

We make the following observations:

  1. Speedup increases almost linearly with sparsity, as expected.

  2. Accuracy generally peaks and improves over dense at a range of sparsity. Higher sparsity appears to be more favorable for larger models and longer contexts.

Indeed, there is a sweet spot in sparsity with respect to quality. Ablations can help identify the favorable sparsity level for a given task, model, and expected context length.

Distribution across heads and layers, and adaptive targets (W3).

The sparsity-pattern experiment in our previous comment also records each head's sparsity under a single threshold (Qwen3-8B base, decision under the map, 1,152 heads):

Per-head sparsity< 0.10.1–0.30.3–0.50.5–0.70.7–0.9≥ 0.9
Share of heads, 64K23%16%11%10%15%25%
Share of heads, 128K24%15%10%10%16%25%
  1. The distribution is already adaptive to every head. This particular distribution belongs to the PICK rule we use, i.e. BLASST's running-maximum test. The per-head pattern depends on the PICK rule, and other rules would distribute sparsity differently. Under BLASST, at 50% overall sparsity, a quarter of the heads stay almost dense while another quarter skip over 90% of their tiles.

2.In our ablations, we also tried holding every head to the same sparsity under BLASST, and this degraded accuracy. We therefore keep only the overall sparsity as our target, so that heads which need dense attention stay dense, while heads which can be sparse become sparse. This is consistent with prior observations that a small set of retrieval heads must attend to the full context, while the remaining heads focus mostly on recent tokens and attention sinks and can be heavily sparsified (e.g., DuoAttention [Xiao et al., 2025], Retrieval Heads [Wu et al., 2024]).

Additionally, our calibration is also adaptive per task: the threshold factor is fitted per task and length, because a single factor does not hold the same sparsity across tasks (Appendix A, Table 8). Furthermore, the sweep over sparsity (mentioned above in "sensitivity to ") suggests that a target sparsity which grows with context length might be preferable to a fixed 50%. We will discuss this in the revision.

Local window sensitivity (W4)

The Jet-Long paper already reports that PACK_JL is robust to the local window size w0 from 1K to 8K, so we have reason to expect PICK ∘ PACK_JL to be similarly robust over this range. We are running these ablations as we post this response, and we will report the numbers shortly.

Model coverage (W5) & comparison with selection-plus-remapping baselines

We added Llama-3-8B-Instruct, a different family with a trained context window of 8K tokens (4× smaller than Qwen3’s 32K) and a different RoPE base. We extend it to 32K and 64K, i.e., 4× and 8× its trained window. PICK improves over dense PACK_JL there by +8.1 and +5.2 points at 32K and 64K, respectively.

Llama-3-8B-InstructLMK2VTFWEQA1RULER-HARD-4 Mean
InfLLM32K10.034.050.022.029.0
ReAttention32K14.044.082.720.040.2
PACK_JL (dense)32K56.055.257.356.056.1
PICK ∘ PACK_JL (50%)32K64.062.470.760.064.3
InfLLM64K4.018.066.78.024.2
ReAttention64K2.042.086.714.036.2
PACK_JL (dense)64K22.038.846.756.040.9
PICK ∘ PACK_JL (50%)64K28.043.654.758.046.1

Observe,

  1. PickPack improves over the existing baselines on average (We do these comparisons on llama since original code of these methods provide support for llama). ReAttention is stronger only on frequent-word extraction, an aggregation task that favours its coarse token selection. This implies there is scope for improvement in PickPack which we leave for future work.

  2. Even in Llama family of models, PICK leads to improvements over dense PACK alone.

4M QAMPARI: PACK alone vs. PICK (Q3).

Qwen3.5-9B at 4M tokens, QAMPARI score over 50 queries:

Methodscore
YaRN (released recipe)22.4
PACK_JL (dense)24.7
PICK . PACK_JL34.0

PickPack_JL achieves 34.0, improving over Dense PACK_JL (24.7) by 9.3 points and over YaRN (22.4) by 11.6 points. Thus, PICK accounts for 9.3 points of the total 11.6-point gain over YaRN.

Checkpoints already trained for long context (Q4).

In our understanding, the paper already addresses this setting in Appendix C. Specifically, Qwen2.5-7B-Instruct-1M is continued-pretrained to 256K, and we extend it to 512K and 1M (Appendix C, Table 9).

Against the shipped recipe (dense DCA). PickPack_JL improves needle-mean accuracy by 5.0 points at 512K (89.3 vs. 84.3) and 7.0 points at 1M (75.7 vs. 68.7).

Against dense PACK_JL under the same map. PICK contributes an additional 4.0 points at 512K (89.3 vs. 85.3), while performance is comparable at 1M (75.7 vs. 76.3, with n = 10 per task).

Thus, the improvement over the shipped recipe also holds for a checkpoint already trained for long context. The majority of the gain comes from replacing DCA with PACK_JL, with PICK providing an additional benefit at 512K. We will make this decomposition explicit in the paper.

We are not completely certain we understand the setting reviewer is talking about. If the question concerns a different setting, kindly let us know, we would be happy to clarify this distinction or add the corresponding experiment.

Comments on the paper

1 comment

We thank all the reviewers for their time and effort in evaluating our submission. Here we summarize the main concerns, our responses, and the additional evidence added during the discussion period.

We are grateful that the reviewers recognized the strengths of our work, including the simplicity and practicality of unifying position remapping and dynamic sparsity in a single FlashAttention pass (R1, R2), the composable formulation of PICK and PACK (R1, R2), the in-window simulation as a principled way to evaluate extension quality (R1, R3), and the breadth of our evaluation across models, checkpoints, benchmarks, and context lengths of up to 4M tokens (R1, R2, R3). The main concerns raised by the reviewers are summarized below, together with our responses. Other concerns are addressed in the individual responses.

ConcernRaised bySummary of response
Novelty beyond combining Jet-Long and BLASSTR1, R2PickPack is a general, training-free framework that separates where tokens are placed (PACK) from which tiles are computed (PICK). This separation lets us isolate each operation and show that sparsity improves model quality beyond the trained window, not only efficiency. Our instantiation runs inside the FlashAttention loop without an additional selection pass, and improved PICK and PACK variants can be incorporated directly.
Comparison with selection-plus-remapping methods and other model familiesR1, R2, R3We added InfLLM and ReAttention on Llama-3-8B-Instruct (8K trained window) at 32K and 64K. PickPack outperforms both on average, by 24–35 points at 32K and 10–22 points at 64K, and PICK improves over dense PACK_JL on this model family as well (+8.1 and +5.2).
Are the accuracy gains attributable to PACK?R1, R3Every PICK result is compared against dense attention under the same PACK map, so PACK's contribution is held fixed. At 4M tokens (Qwen3.5-9B, QAMPARI), dense PACK_JL reaches 24.7 against 22.4 for YaRN and 34.0 for PickPack, so PICK accounts for 9.3 of the 11.6-point gain.
Is PICK equivalent to a temperature-scaling control?R2We added LogN temperature scaling under the same position map (Qwen3-8B, RULER-13). LogN and PICK are complementary: PICK adds +1.51 (64K) and +3.56 (128K) on top of LogN, and their combination achieves the best accuracy (85.77 and 78.88).
Sensitivity to sparsity, and accuracy on the post-trained modelR1, R2, R3We swept sparsity from 10% to 80% on Qwen3-1.7B and 8B at 64K and 128K. Accuracy improves over dense across a range of sparsity levels, and the favorable level grows with model size and context length. We agree that sparsity does not improve accuracy at every operating point, and we will present such cases as a quality–speed trade-off.
Interaction between the position map and the skip decisionR1Implementing sparsity at native positions reaches similar accuracy, but introduces a systematic distance-dependent bias in which tiles are skipped. Deciding under the map shows no such bias, and never skips the attention-sink tile. Sparsity also adapts naturally across heads, consistent with prior findings on retrieval heads (DuoAttention): at 50% overall sparsity, about a quarter of the heads remain nearly dense (under 10% of tiles skipped), while another quarter are extremely sparse (over 90% of tiles skipped).
Sensitivity to the local windowR1Varying w0 ∈ {1024, 2048, 4096}, PICK improves over dense PACK_JL at every window size and both context lengths.
Modest speedup at 50% sparsityR3The ~1.2× speedup is specific to our current PICK operator, BLASST, which still computes QKᵀ for every tile to make the skip decision and avoids only the softmax and P·V work for skipped tiles. The speedup grows with sparsity (up to 1.34× at 80%) and with context length, and a cheaper PICK operator can unlock larger gains.
Decode and end-to-end latency for agentic workloadsR2, R3We implemented PICK for decoding. On Qwen3-8B (H100) with 8K generated tokens, decoding is 1.05–1.19× faster from 128K to 1M tokens, and a full request (prefill + decode) is 1.07–1.22× faster than dense attention under the same position map, with the gain growing with context length.
Threshold calibration: transfer and costR2Calibration targets a sparsity level, so it needs no labels. The threshold is fitted on the first two prompts of each task and holds 0.49–0.51 sparsity on all remaining prompts. Each attempt costs only two prefills, with no decoding, and most targets are reached within one or two attempts.
Implementation detailsR2We described how PACK_JL is applied before the kernel, how PICK runs inside the far-region call of the Hopper BLASST kernel, and what a skipped tile saves, together with an H100 prefill profile.

We will incorporate all of these results and clarifications into the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.