- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper identifies phase sensitivity in chunked KV-cache compression: retrieval accuracy depends on the source token’s position relative to compression windows, with period matching the compression stride. The phenomenon appears in DeepSeek-V4, V4.1, and all compressed pretrained models the authors train, but not in full-attention baselines. Using a Qwen3-derived family where chunked compression is the only substantial architectural change, the authors show that phase sensitivity persists across window/stride choices, kernels, RoPE removal, and uniform averaging. KV-head knockouts and gate analyses reveal phase specialization: some heads contribute disproportionately to retrieval at particular phases, and their learned gate preferences match those phases. Gate cycling shifts within-window weak phases, though not the boundary phase. Theory on an idealized bigram retrieval model shows that optimal compression is concentrated and that gradient flow drives static, head-specific phase selection. Evaluation is mainly synthetic NIAH and code-completion probes, and the paper argues for phase-resolved evaluation rather than average accuracy alone.
Critical strengths
1.The paper identifies phase sensitivity in chunked KV-cache compression: retrieval accuracy depends systematically on the source token's position modulo the compression stride. This pattern appears in DeepSeek-V4, DeepSeek-V4.1, and in Qwen3-derived models pretrained from scratch where chunked compression is the only substantial architectural change, while full-attention baselines remain flat. The controlled pretraining is especially consequential because it moves the claim beyond correlation in one model family and provides strong evidence that the compression stride itself causes the periodic weak spots. This matters for the overall assessment: it exposes a general evaluation blind spot, since averaged long-context metrics can hide large phase-dependent accuracy gaps.
2.The paper goes beyond documenting an anomaly. KV-head knockouts show phase-dependent contributions to retrieval, and some heads develop persistent gate preferences for particular offsets that match the phases where their knockout damage is strongest. Gate-cycling interventions largely shift within-window weak phases as predicted. The idealized induction analysis further explains why optimal compression may concentrate and why gradient flow can produce static, head-specific phase selection. This is consequential because it provides a plausible mechanism and testable predictions, making the phenomenon more than a benchmark curiosity and suggesting concrete directions for mitigation or phase-resolved evaluation.
Critical weaknesses
1.The evaluation relies mainly on synthetic needle-in-a-haystack retrieval and a few code-completion probes. There is no systematic evidence on standard long-context benchmarks such as LongBench, RULER, HELMET, or on downstream QA, summarization, or reasoning tasks. Without this, the paper cannot establish whether phase sensitivity materially affects real applications or is largely confined to synthetic diagnostics. This is consequential because the central claim is that averaged evaluation hides an important failure mode; if the failure does not translate into meaningful downstream degradation, its practical significance remains uncertain.
2.In the DeepSeek NIAH setup, moving the target key's phase also shifts all distractor records and changes absolute positions, because the whole record block moves together with outer fillers. The In-sequence Padding control that decouples distractor phases is only applied to the smaller self-trained models, not to DeepSeek-V4. This is consequential because the DeepSeek-V4 observations are a major part of the paper’s motivation and evidence. If those effects are partly driven by distractor phase or absolute position rather than the target’s phase, the claim that phase sensitivity is a general property of chunked compression in large models is weakened.
Questions for authors
-
How do the observed phase gaps translate into downstream task accuracy on standard long-context benchmarks such as LongBench, RULER, or GSM8K?
-
Can the DeepSeek NIAH evaluation be modified to vary the target phase independently of distractor phases, as In-sequence Padding does for the smaller models?
-
What mechanism causes the boundary phase failure, and why does gate cycling not restore it?
-
How robust are the theoretical conclusions under relaxed assumptions, such as approximate geometry, nonlinear gates, multi-token values, and noisy retrieval?
-
Can the authors provide identity or alternative control interventions for the knockouts, and can they ablate compressed memory separately from local memory?
-
Do the authors recommend a concrete phase-resolved evaluation protocol or a training-time fix that reduces phase sensitivity without sacrificing average performance?
-
Do the main findings hold at larger model scales, and are there plans to test larger compressed models beyond the 0.6B Qwen3 family?
6 comments
Official Response to Reviewer mLo1 (2/3)
Part 2 of 3
Weakness 2 / Question 2: Varying the target phase independently in DeepSeek
Moving the target key's phase in the DeepSeek setup also moves the distractor records and changes absolute positions, and the In-sequence Padding control is applied only to the smaller self-trained models.
Thank you for pointing this out. Following your suggestion we report the new result in part (a) of our first general response and summarize it here.
(a) Phase sensitivity persists in DeepSeek when only the target's phase varies. Following your suggestion, we rebuilt the 128K NIAH retrieval prompts with the In-sequence Padding schema. In this new evaluation set, each prompt contains 8,001 four-token key-value pairs. The pairs are separated by 8,000 shuffled gaps containing 0 to 7 tokens of “|”, with 1,000 gaps of each length. The target's position has the same distribution in every phase with a mean of 64,000 tokens, and the query position and prompt length are fixed. We evaluate 256 prompts per phase for each model.
We find that the DeepSeek V4 models still show large phase gaps (28.9 pp for V4-Flash-Base and 20.7 pp for V4-Pro-Base), and phases 0 and 3 of the four-token period remain the weak spots:
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | |
|---|---|---|---|---|---|---|---|---|
| DSV4-Flash-Base, Acc (%) | 62.5 | 64.8 | 76.6 | 47.7 | 60.5 | 68.0 | 73.4 | 50.0 |
| DSV4-Flash-Base, Acc diff from best (pp) | −14.1 | −11.7 | 0.0 | −28.9 | −16.0 | −8.6 | −3.1 | −26.6 |
| DSV4-Pro-Base, Acc (%) | 83.2 | 96.1 | 92.6 | 85.2 | 77.7 | 98.4 | 94.1 | 86.3 |
| DSV4-Pro-Base, Acc diff from best (pp) | −15.2 | −2.3 | −5.9 | −13.3 | −20.7 | 0.0 | −4.3 | −12.1 |
We note that the weak phases at 0 and 3 remains consistent as the original evaluation in the current draft.
In the revision, we will add this experiment to Section 2 and give the full protocol in Appendix C.
Question 3: Mechanism of the boundary-phase failure
What mechanism causes the boundary phase failure, and why does gate cycling not restore it?
Thank you for this interesting question. In summary, our mechanistic analysis suggests that the models tend to learn a single-layer retrieval solution for key-value pairs in the same compression chunk, but this solution does not apply when they are cross-chunk, so gate cycling only moves this performance within the window and does not transfer to the cross-chunk scenario.
In our analysis, the heads that matter for retrieval develop phase preferences in which the value gate peaks one position after the key gate (Appendix Figures 15 and 16). As a result, the compression kernel writes a key and the value that follows it into the same compressed entry. Compression thus already mixes each key with its value. A single attention layer then only has to perform the long-range copy. Our theory shows that the model can learn this one-layer induction solution (Section 5). Without compression, an induction head needs at least two layers [1].
Now consider windows that do not overlap, so that . The key at phase is then the last token of one window, and its value is the first token of the next. No compressed entry contains both, and thus retrieval across windows must rely on a different solution which we believe is more similar to the standard 2-layer induction head. This analysis is consistent with our experiments: boundary-phase accuracy stays at or below 2.9% in all 54 gate-shifted runs, and it recovers once we shift the compression boundaries across layers, which places the key and its value in one window in some layers (part (c) of our first general response).
In the revision, we will add a more comprehensive explanation to Section 4.
Question 4: Robustness of the theory under relaxed assumptions
How robust are the theoretical conclusions under relaxed assumptions such as approximate geometry, nonlinear gates, multi-token values and noisy retrieval?
Thank you for this question. Our optimality result (Theorem 1) allows approximate geometry. In particular, theorem 1 holds for any geometry error , which permits embedding dimension logarithmic in vocabulary size (Appendix B). Only distinct routes across heads and the gradient-flow result (Theorem 2) need . In the revision, we will state which results depend on which assumptions, and list exact-geometry dynamics, nonlinear gates, multi-token values and noise as open extensions in Section 5.
References
[1] Olsson et al. In-Context Learning and Induction Heads. Transformer Circuits Thread, 2022.
Our responses continue in Part 3
Official Response to Reviewer mLo1 (3/3)
Part 3 of 3
Question 5: Knockout controls and separating compressed from local memory
Can the authors provide identity or alternative control interventions for the knockouts, and can they ablate compressed memory separately from local memory?
Thank you for both suggestions. In short, we believe our experiments already include such controls, and that phase sensitivity persists with minimal local memory.
(a) The knockout sweeps already include identity and matched-random controls. With these controls, we find that the damage is specific to the head and is not a generic effect of perturbation. In the revision, we will move them from the appendix into the main knockout figure in Section 4.
(b) Phase sensitivity remains robust when local memory is reduced to the minimum. We agree that one may be concerned that the retrieval phenomenon could come from the local sliding-window branch instead of compressed attention. In the large models, the target sits 64,000 tokens before the query on average, which is far outside the local SWA windows (with 128 tokens). Therefore the local branch cannot read the target, and that the phase-dependent readout has to run through the compressed path.
For our controlled Qwen3-0.6B models, we have trained variants that shrink local memory as far as the architecture allows (see Figure 13a and 13b). The open-tail variants keep raw tokens only in the current incomplete stride block. At the last token of each block, the raw tail is therefore empty, and every earlier token is visible only through compressed entries. All of these runs use eight-token windows at stride eight, and we average the three evaluated positions that share each phase. We find that phase sensitivity persists with minimal local memory:
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | |
|---|---|---|---|---|---|---|---|---|
| Open tail, 1 KV head, Acc (%) | 67.3 | 91.0 | 85.0 | 84.3 | 70.8 | 69.1 | 74.1 | 58.9 |
| Open tail, 1 KV head, Acc diff from best (pp) | −23.7 | 0.0 | −6.0 | −6.7 | −20.2 | −21.8 | −16.9 | −32.0 |
| Open tail, 8 KV heads, Acc (%) | 49.2 | 73.6 | 31.6 | 67.2 | 43.4 | 60.9 | 71.4 | 27.0 |
| Open tail, 8 KV heads, Acc diff from best (pp) | −24.4 | 0.0 | −42.0 | −6.4 | −30.2 | −12.7 | −2.2 | −46.6 |
| In the revision, we will report the open-tail comparison in Section 3. |
Question 6: A phase-resolved evaluation protocol and a training-time fix
Do the authors recommend a concrete phase-resolved evaluation protocol or a training-time fix that reduces phase sensitivity without sacrificing average performance?
(a) For evaluation, we recommend the phase-resolved evaluation protocol described in our second general response. One can easily turn existing benchmarks into the prefix-shifted variants, and additionally reports robust accuracy, flip rate and log-probability variance next to mean accuracy. As we can see, one would more easily discover the hidden weak spots in DeepSeek V4-Flash-Base against V3.1-Base
(b) For training, shifting the compression boundaries across layers alleviates the boundary drop. We describe this remedy and its results in part (c) of our first general response. It alleviates the drop at the boundary phase without increasing validation loss, but a significant periodic oscillation remains. In general, we believe that a complete solution to remove phase-sensitivity in models with chunked KV compression remains to be a challenging open question.
Question 7: Phase-Sensitivity on Larger model scales
Do the main findings hold at larger model scales, and are there plans to test larger compressed models beyond the 0.6B Qwen3 family?
Beyond DeepSeek-V4 and V4.1, we now report GLM-5.3-Flash. This compressed model comes from a different developer and uses a different tokenizer and a four-token pooling indexer. We find that GLM-5.3-Flash shows a 22.7 pp phase gap with the same four-token period, and its weakest phases are 3 and 7 (part (b) of our first general response). We have not trained compressed models larger than 0.6B ourselves. In the revision, we will keep the behavioral and mechanistic lines of evidence clearly separated in Sections 2 and 4.
Thank you again for your constructive review. We hope these new results address your concerns, and we would be happy to answer any follow-up questions.
Official Response to Reviewer mLo1 (1/3)
Dear Reviewer, thank you for the careful and detailed review. We are glad that you found the controlled pretraining to be strong evidence that the compression stride itself causes the periodic weak spots.
We first summarize our responses below and then address each point in detail.
In the following threads, we will respond to the mentioned weakness and questions in more details.
Weakness 1 / Question 1: Evidence on standard long-context benchmarks
Thank you for raising this point, which brings us insights toward developing the new phase-resolved evaluation protocol based on existing long context benchmarks.
Standard benchmark scores average over phases, so they hide the periodic oscillation. Following your suggestion, we introduce a phase-resolved evaluation protocol for existing long-context benchmarks by appending a set of task-agnostic prefixes to the same problem instance and evaluate the consistency of the model behavior on the same instance when it is prepended with different prefixes. Please see the detailed explanation in our second general response.
On an augmented evaluation set derived from 366 retrieval problem instances from OpenAI MRCR, BABILong, HELMET, RULER and LongBench v2 (with length over 64k tokens), we find that DeepSeek-V4-Flash-Base trails DeepSeek-V3.1-Base by only 3.2 pp in average accuracy but by 8.2 pp in robust accuracy (the fraction of problem instances answered correctly under all prefixes). The answer log-probability variance of V4-Flash-Base is also 3.5 times that of V3.1-Base, and its flip rate is 27.9% against 16.9%. This shows that on standard long context benchmarks, models with compressed KV can exhibits less robust behavior despite matching on the average metrics.
In the revision, we will add the evaluation protocol and the corresponding results to Section 2 and Appendix C.
Our responses continue in Part 2/3