acceptodds
Under review as a conference paper at ICLR 2027

Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression

55 views

Abstract

Chunked KV-cache compression reduces the memory and attention costs of long-context inference by compressing windows of consecutive tokens into fewer cache entries at a fixed stride. Such compression also introduces a new positional coordinate: a token's phase, or its position relative to compression-window boundaries. We uncover a systematic asymmetry in models using such compression: the same information can be easy to retrieve at one phase and difficult at another. We call this periodic variation in retrieval performance phase-sensitivity. In the DeepSeek-V4 family, long context retrieval accuracy can differ by up to 40 percentage points across phases, revealing periodic weak spots that average benchmark scores can conceal. To investigate this behavior, we pretrain a family of transformers from scratch across multiple KV-compression designs, reproducing phase-sensitivity across the tested variants. Mechanistic analysis using causal interventions in these models and DeepSeek-V4 reveals phase specialization: different attention components contribute asymmetrically to retrieving information at different source phases. We further analyze idealized retrieval models, showing how gradient flow dynamics may favor sharp phase specialization. Evaluating models with chunked KV-cache compression therefore requires measuring across compression phases: high average accuracy can coexist with systematic positional failures.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Compression, Mechanistic Interpretability, Large Language Models, Optimization Dynamics, Failure Mode Analysis

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_mLo1OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies phase sensitivity in chunked KV-cache compression: retrieval accuracy depends on the source token’s position relative to compression windows, with period matching the compression stride. The phenomenon appears in DeepSeek-V4, V4.1, and all compressed pretrained models the authors train, but not in full-attention baselines. Using a Qwen3-derived family where chunked compression is the only substantial architectural change, the authors show that phase sensitivity persists across window/stride choices, kernels, RoPE removal, and uniform averaging. KV-head knockouts and gate analyses reveal phase specialization: some heads contribute disproportionately to retrieval at particular phases, and their learned gate preferences match those phases. Gate cycling shifts within-window weak phases, though not the boundary phase. Theory on an idealized bigram retrieval model shows that optimal compression is concentrated and that gradient flow drives static, head-specific phase selection. Evaluation is mainly synthetic NIAH and code-completion probes, and the paper argues for phase-resolved evaluation rather than average accuracy alone.

Critical strengths

1.The paper identifies phase sensitivity in chunked KV-cache compression: retrieval accuracy depends systematically on the source token's position modulo the compression stride. This pattern appears in DeepSeek-V4, DeepSeek-V4.1, and in Qwen3-derived models pretrained from scratch where chunked compression is the only substantial architectural change, while full-attention baselines remain flat. The controlled pretraining is especially consequential because it moves the claim beyond correlation in one model family and provides strong evidence that the compression stride itself causes the periodic weak spots. This matters for the overall assessment: it exposes a general evaluation blind spot, since averaged long-context metrics can hide large phase-dependent accuracy gaps.

2.The paper goes beyond documenting an anomaly. KV-head knockouts show phase-dependent contributions to retrieval, and some heads develop persistent gate preferences for particular offsets that match the phases where their knockout damage is strongest. Gate-cycling interventions largely shift within-window weak phases as predicted. The idealized induction analysis further explains why optimal compression may concentrate and why gradient flow can produce static, head-specific phase selection. This is consequential because it provides a plausible mechanism and testable predictions, making the phenomenon more than a benchmark curiosity and suggesting concrete directions for mitigation or phase-resolved evaluation.

Critical weaknesses

1.The evaluation relies mainly on synthetic needle-in-a-haystack retrieval and a few code-completion probes. There is no systematic evidence on standard long-context benchmarks such as LongBench, RULER, HELMET, or on downstream QA, summarization, or reasoning tasks. Without this, the paper cannot establish whether phase sensitivity materially affects real applications or is largely confined to synthetic diagnostics. This is consequential because the central claim is that averaged evaluation hides an important failure mode; if the failure does not translate into meaningful downstream degradation, its practical significance remains uncertain.

2.In the DeepSeek NIAH setup, moving the target key's phase also shifts all distractor records and changes absolute positions, because the whole record block moves together with outer fillers. The In-sequence Padding control that decouples distractor phases is only applied to the smaller self-trained models, not to DeepSeek-V4. This is consequential because the DeepSeek-V4 observations are a major part of the paper’s motivation and evidence. If those effects are partly driven by distractor phase or absolute position rather than the target’s phase, the claim that phase sensitivity is a general property of chunked compression in large models is weakened.

Questions for authors
  1. How do the observed phase gaps translate into downstream task accuracy on standard long-context benchmarks such as LongBench, RULER, or GSM8K?

  2. Can the DeepSeek NIAH evaluation be modified to vary the target phase independently of distractor phases, as In-sequence Padding does for the smaller models?

  3. What mechanism causes the boundary phase failure, and why does gate cycling not restore it?

  4. How robust are the theoretical conclusions under relaxed assumptions, such as approximate geometry, nonlinear gates, multi-token values, and noisy retrieval?

  5. Can the authors provide identity or alternative control interventions for the knockouts, and can they ablate compressed memory separately from local memory?

  6. Do the authors recommend a concrete phase-resolved evaluation protocol or a training-time fix that reduces phase sensitivity without sacrificing average performance?

  7. Do the main findings hold at larger model scales, and are there plans to test larger compressed models beyond the 0.6B Qwen3 family?

6 comments
AuthorsOpenReview ↗

Official Response to Reviewer mLo1 (1/3)

Dear Reviewer, thank you for the careful and detailed review. We are glad that you found the controlled pretraining to be strong evidence that the compression stride itself causes the periodic weak spots.

We first summarize our responses below and then address each point in detail.


  • Weakness 1 / Question 1: Standard benchmark scores average over phases, so they hide the periodic oscillation. Following your suggestion, we instead test how robustly a model answers the same problem instance when we shift it by a few task-agnostic prefix tokens, using long-context retrieval problems from five standard benchmarks. We find that DeepSeek-V4-Flash-Base trails V3.1-Base by only 3.2 pp in mean accuracy, but by 8.2 pp in robust accuracy, which requires a correct answer in every shifted version. The log-probability variance of its answers across shifts is also 3.5 times that of V3.1-Base.
  • Weakness 2 / Question 2: We rebuilt the DeepSeek 128K retrieval prompts with In-sequence Padding so that only the target's phase varies. We find that DeepSeek-V4-Flash-Base and DeepSeek-V4-Pro-Base still show phase gaps of 28.9 pp and 20.7 pp, and phases 0 and 3 of the four-token period remain the weakest for both.
  • Question 3: Our mechanistic and theoretical analysis suggests that the models can learn a single-layer induction solution within a window, and this solution needs a key and its value in one compressed entry. At the chunk boundaries, this solution does not exist, so rotating the gates does not transfer to it.
  • Question 4. We will state which theoretical results depend on which assumptions. We will also list approximate geometry, nonlinear gates, multi-token values and noise as open extensions.
  • Question 5. Our knockout experiments already include identity and matched-random controls. In the large models, the target lies far outside the local window. We also find that our controlled runs with minimal local memory remains phase-sensitive.
  • Question 6. We recommend the phase-resolved evaluation protocol from our second general response, which reports robust accuracy next to mean accuracy. We also report shifting the compression boundaries across layers as a partial training-time fix.
  • Question 7. We find that GLM-5.3-Flash, which utilized another type of chunked compression (W=4), shows the same four-token period with a 22.7 pp gap.

In the following threads, we will respond to the mentioned weakness and questions in more details.


Weakness 1 / Question 1: Evidence on standard long-context benchmarks

The evaluation relies mainly on synthetic needle-in-a-haystack retrieval and code-completion probes, with no systematic evidence on standard long-context benchmarks or on downstream tasks.

Thank you for raising this point, which brings us insights toward developing the new phase-resolved evaluation protocol based on existing long context benchmarks.

Standard benchmark scores average over phases, so they hide the periodic oscillation. Following your suggestion, we introduce a phase-resolved evaluation protocol for existing long-context benchmarks by appending a set of task-agnostic prefixes to the same problem instance and evaluate the consistency of the model behavior on the same instance when it is prepended with different prefixes. Please see the detailed explanation in our second general response.

On an augmented evaluation set derived from 366 retrieval problem instances from OpenAI MRCR, BABILong, HELMET, RULER and LongBench v2 (with length over 64k tokens), we find that DeepSeek-V4-Flash-Base trails DeepSeek-V3.1-Base by only 3.2 pp in average accuracy but by 8.2 pp in robust accuracy (the fraction of problem instances answered correctly under all prefixes). The answer log-probability variance of V4-Flash-Base is also 3.5 times that of V3.1-Base, and its flip rate is 27.9% against 16.9%. This shows that on standard long context benchmarks, models with compressed KV can exhibits less robust behavior despite matching on the average metrics.

In the revision, we will add the evaluation protocol and the corresponding results to Section 2 and Appendix C.


Our responses continue in Part 2/3

AuthorsOpenReview ↗

Official Response to Reviewer mLo1 (2/3)

Part 2 of 3


Weakness 2 / Question 2: Varying the target phase independently in DeepSeek

Moving the target key's phase in the DeepSeek setup also moves the distractor records and changes absolute positions, and the In-sequence Padding control is applied only to the smaller self-trained models.

Thank you for pointing this out. Following your suggestion we report the new result in part (a) of our first general response and summarize it here.

(a) Phase sensitivity persists in DeepSeek when only the target's phase varies. Following your suggestion, we rebuilt the 128K NIAH retrieval prompts with the In-sequence Padding schema. In this new evaluation set, each prompt contains 8,001 four-token key-value pairs. The pairs are separated by 8,000 shuffled gaps containing 0 to 7 tokens of “|”, with 1,000 gaps of each length. The target's position has the same distribution in every phase with a mean of 64,000 tokens, and the query position and prompt length are fixed. We evaluate 256 prompts per phase for each model.

We find that the DeepSeek V4 models still show large phase gaps (28.9 pp for V4-Flash-Base and 20.7 pp for V4-Pro-Base), and phases 0 and 3 of the four-token period remain the weak spots:

01234567
DSV4-Flash-Base, Acc (%)62.564.876.647.760.568.073.450.0
DSV4-Flash-Base, Acc diff from best (pp)−14.1−11.70.0−28.9−16.0−8.6−3.1−26.6
DSV4-Pro-Base, Acc (%)83.296.192.685.277.798.494.186.3
DSV4-Pro-Base, Acc diff from best (pp)−15.2−2.3−5.9−13.3−20.70.0−4.3−12.1

We note that the weak phases at 0 and 3 remains consistent as the original evaluation in the current draft.

In the revision, we will add this experiment to Section 2 and give the full protocol in Appendix C.


Question 3: Mechanism of the boundary-phase failure

What mechanism causes the boundary phase failure, and why does gate cycling not restore it?

Thank you for this interesting question. In summary, our mechanistic analysis suggests that the models tend to learn a single-layer retrieval solution for key-value pairs in the same compression chunk, but this solution does not apply when they are cross-chunk, so gate cycling only moves this performance within the window and does not transfer to the cross-chunk scenario.

In our analysis, the heads that matter for retrieval develop phase preferences in which the value gate peaks one position after the key gate (Appendix Figures 15 and 16). As a result, the compression kernel writes a key and the value that follows it into the same compressed entry. Compression thus already mixes each key with its value. A single attention layer then only has to perform the long-range copy. Our theory shows that the model can learn this one-layer induction solution (Section 5). Without compression, an induction head needs at least two layers [1].

Now consider windows that do not overlap, so that W=SW = S. The key at phase S−1S−1 is then the last token of one window, and its value is the first token of the next. No compressed entry contains both, and thus retrieval across windows must rely on a different solution which we believe is more similar to the standard 2-layer induction head. This analysis is consistent with our experiments: boundary-phase accuracy stays at or below 2.9% in all 54 gate-shifted runs, and it recovers once we shift the compression boundaries across layers, which places the key and its value in one window in some layers (part (c) of our first general response).

In the revision, we will add a more comprehensive explanation to Section 4.


Question 4: Robustness of the theory under relaxed assumptions

How robust are the theoretical conclusions under relaxed assumptions such as approximate geometry, nonlinear gates, multi-token values and noisy retrieval?

Thank you for this question. Our optimality result (Theorem 1) allows approximate geometry. In particular, theorem 1 holds for any geometry error ε≤c/(L2HS)\varepsilon \le c/(L^2HS), which permits embedding dimension logarithmic in vocabulary size (Appendix B). Only distinct routes across heads and the gradient-flow result (Theorem 2) need ε=0\varepsilon = 0. In the revision, we will state which results depend on which assumptions, and list exact-geometry dynamics, nonlinear gates, multi-token values and noise as open extensions in Section 5.


References

[1] Olsson et al. In-Context Learning and Induction Heads. Transformer Circuits Thread, 2022.


Our responses continue in Part 3

AuthorsOpenReview ↗

Official Response to Reviewer mLo1 (3/3)

Part 3 of 3


Question 5: Knockout controls and separating compressed from local memory

Can the authors provide identity or alternative control interventions for the knockouts, and can they ablate compressed memory separately from local memory?

Thank you for both suggestions. In short, we believe our experiments already include such controls, and that phase sensitivity persists with minimal local memory.

(a) The knockout sweeps already include identity and matched-random controls. With these controls, we find that the damage is specific to the head and is not a generic effect of perturbation. In the revision, we will move them from the appendix into the main knockout figure in Section 4.

(b) Phase sensitivity remains robust when local memory is reduced to the minimum. We agree that one may be concerned that the retrieval phenomenon could come from the local sliding-window branch instead of compressed attention. In the large models, the target sits 64,000 tokens before the query on average, which is far outside the local SWA windows (with 128 tokens). Therefore the local branch cannot read the target, and that the phase-dependent readout has to run through the compressed path.

For our controlled Qwen3-0.6B models, we have trained variants that shrink local memory as far as the architecture allows (see Figure 13a and 13b). The open-tail variants keep raw tokens only in the current incomplete stride block. At the last token of each block, the raw tail is therefore empty, and every earlier token is visible only through compressed entries. All of these runs use eight-token windows at stride eight, and we average the three evaluated positions that share each phase. We find that phase sensitivity persists with minimal local memory:

01234567
Open tail, 1 KV head, Acc (%)67.391.085.084.370.869.174.158.9
Open tail, 1 KV head, Acc diff from best (pp)−23.70.0−6.0−6.7−20.2−21.8−16.9−32.0
Open tail, 8 KV heads, Acc (%)49.273.631.667.243.460.971.427.0
Open tail, 8 KV heads, Acc diff from best (pp)−24.40.0−42.0−6.4−30.2−12.7−2.2−46.6
In the revision, we will report the open-tail comparison in Section 3.

Question 6: A phase-resolved evaluation protocol and a training-time fix

Do the authors recommend a concrete phase-resolved evaluation protocol or a training-time fix that reduces phase sensitivity without sacrificing average performance?

(a) For evaluation, we recommend the phase-resolved evaluation protocol described in our second general response. One can easily turn existing benchmarks into the prefix-shifted variants, and additionally reports robust accuracy, flip rate and log-probability variance next to mean accuracy. As we can see, one would more easily discover the hidden weak spots in DeepSeek V4-Flash-Base against V3.1-Base

(b) For training, shifting the compression boundaries across layers alleviates the boundary drop. We describe this remedy and its results in part (c) of our first general response. It alleviates the drop at the boundary phase without increasing validation loss, but a significant periodic oscillation remains. In general, we believe that a complete solution to remove phase-sensitivity in models with chunked KV compression remains to be a challenging open question.


Question 7: Phase-Sensitivity on Larger model scales

Do the main findings hold at larger model scales, and are there plans to test larger compressed models beyond the 0.6B Qwen3 family?

Beyond DeepSeek-V4 and V4.1, we now report GLM-5.3-Flash. This compressed model comes from a different developer and uses a different tokenizer and a four-token pooling indexer. We find that GLM-5.3-Flash shows a 22.7 pp phase gap with the same four-token period, and its weakest phases are 3 and 7 (part (b) of our first general response). We have not trained compressed models larger than 0.6B ourselves. In the revision, we will keep the behavioral and mechanistic lines of evidence clearly separated in Sections 2 and 4.


Thank you again for your constructive review. We hope these new results address your concerns, and we would be happy to answer any follow-up questions.

Reviewer_utTmOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, the authors find the phenomenon of phase sensitivity, that in the DeepSeekV4 family, long-context retrieval accuracy can differ by up to 40 percentage points with respect to its position relative to compression-window boundaries, revealing periodic weak spots that average benchmark scores can conceal. Moreover, the authors provide an in-depth ablation study by pretraining a family of transformers from scratch across multiple KV-compression designs, reproducing phase sensitivity across the tested variants, and revealing that different attention components contribute asymmetrically to retrieving information at different source phases.

Critical strengths
  1. The phenomenon reported in this paper is interesting. Compressed attention is one of the less thoroughly studied approaches in the efficient attention family, and the DeepSeek series is an important family of open-source LLMs, so the work is timely.
  2. The paper provides extensive analysis and discussion. The experiments are not limited to DeepSeek-V4. The authors also pretrain models at the 600M scale and run a large set of ablations, attempting to determine whether this phenomenon is a problem of compressed attention in general, or of CSA, namely DeepSeek-style compressed attention.
Critical weaknesses

As a reviewer unfamiliar with CSA, I find the ablations of the potential causes of phase sensitivity in this paper so extensive that it is hard to think of anything still missing. Besides, I still want to ask whether this phenomenon would remain if sparsity were removed. I am not sure whether the authors have run this ablation. They include many ablations in the main text and the appendix, and Table 8 lists a large number of them. I would ask the authors to itemize, in the main text, the variables they have already considered, which would make the paper easier to read. I would also ask them to check whether a similar phenomenon is present in earlier compression work, which would help guide subsequent improvements to compressed attention.

Questions for authors

Beyond the open question in Section 7, did the authors try any concrete remedy, even one that failed, for phase sensitivity in compressed attention? The code-completion examples show the effect outside needle retrieval, but is there more observation on other long-context tasks, especially long-output generation?

2 comments
AuthorsOpenReview ↗

Official Response to Reviewer utTm

Dear Reviewer, thank you for your thoughtful review and for your positive assessment of the phenomenon and the ablations. Our general responses report three additional experiments and a phase-resolved evaluation protocol, which we hope further establish the generality of the phenomenon.

Now we respond to each point in your review as follows:


Weakness 1: Phase sensitivity without sparsity

Would the phenomenon remain if sparsity were removed?

Thank you for this question. We find that the effect does not depend on sparse selection. Top-k selection is a component of DeepSeek's CSA, but our controlled 0.6B models do not use it. In these models, every layer attends to all compressed entries and a short local window through a single softmax. All of these compressed models exhibit phase sensitivity, whereas none of the full-attention baselines do. In the revision, we will state this explicitly in Section 2.


Weakness 2: Itemizing the variables already considered

The ablations are extensive, but the variables already considered should be itemized in the main text.

Thank you for this helpful suggestion. In the revision, we will add the following summary table to the main text:

VariableWhat we varied
Compressioncompressed vs. full attention
Window and strideseveral, including stride not dividing window
Compression operatorlearned gates, uniform averaging, kernel variants
KV heads per layer1, 2, 4, 8
Positional encodingwith and without RoPE
Local windowsize and retention policy
Top-k selectionnot used in any controlled model
Boundaries across layersaligned vs. shifted per layer
Padding protocolPrefix vs. In-sequence Padding
Training seed4 seeds at W8/S8
ModelQwen3-0.6B variants, DeepSeek-V4/V4.1, GLM-5.3-Flash

We find phase sensitivity in every compressed variant, with a period equal to the stride, and in none of the full-attention baselines. Only shifting the boundaries across layers reduces it (Question 1).


Weakness 3: Phase sensitivity in earlier KV-compression architectures

Is a similar phenomenon present in earlier KV-compression architectures?

Thank you for this suggestion. Beyond DeepSeek's compressed sparse attention, we find phase sensitivity in GLM-5.3-Flash, which retains the full KV cache and compresses only the keys of its sparse-attention indexer, in chunks of four tokens. It shows a periodic 22.7 pp accuracy gap with a four-token period (part (b) of our first general response), which suggests that phase sensitivity can arise even when compression only decides which entries a query attends to. Our controlled sweep also includes variants close to earlier designs, and we find that they remain phase sensitive. Mean pooling over non-overlapping windows, as in the Compressive Transformer [1], gives a 19.2 pp gap. Overlapping compressed blocks, as in NSA [2], give gaps of 31.9 to 39.8 pp. In the revision, we will discuss earlier KV-compression architectures in the related work.


Question 1: Concrete remedies

Did the authors try any concrete remedy for phase sensitivity, even one that failed?

Thank you for this question. We tested a simple structural remedy that shifts the compression boundaries across layers, so that a token on a boundary in one layer falls inside a window in other layers (part (c) of our first general response). We find that it raises boundary-phase accuracy from 6.2% to 52.6% at W8/S8 and from 15.9% to 62.1% at W4/S4, and slightly lowers validation loss. However, a periodic oscillation remains, and the spread across phases only narrows. We therefore see this as evidence that the problem is addressable rather than as a complete solution.


Question 2: Other long-context tasks and long-output generation

Is there more evidence on other long-context tasks, especially long-output generation?

Thank you for this question. For the first part, our second general response introduces a phase-resolved evaluation protocol. On retrieval problems from five standard benchmarks, we find that DeepSeek-V4-Flash-Base trails DeepSeek-V3.1-Base by only 3.2 pp in mean accuracy but by 8.2 pp in robust accuracy, the fraction of problem instances answered correctly under all prefixes. However, we have not yet evaluated long-form generation. We will list this as an open question in the revision.


Thank you again for your constructive suggestions. We would be happy to answer any further questions during the discussion period.


References

[1] Rae et al. Compressive Transformers for Long-Range Sequence Modelling. ICLR, 2020.

[2] Yuan et al. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention. arXiv:2502.11089, 2025.

Thank you for your feedback. I hope authors can organize the paper better, and I will maintain my positive assessment.

Reviewer_fMtCOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper shows that chunked KV compression makes information easier or harder to retrieve depending on where it falls within a chunk. It links these repeating weak spots to learned compression preferences: shifting those preferences also shifts where retrieval fails. The finding suggests that average scores can hide systematic memory failures.

Critical strengths
  • Shows that good average recall can hide repeating positions where memory fails, which is a reliability problem that standard evaluations miss.

  • Connects those failures to learned compression preferences and shows that changing the preferences moves the failures. This is something you can give model designers as a specific mechanism to investigate beyond another benchmark improvement.

Critical weaknesses
  • The paper explains the failure convincingly in the controlled models, but have not shown that the same explanation causes DeepSeek’s failures. In the controlled models, they change the compression gates and so the weak positions move. In DeepSeek, they show the failures but cannot isolate their cause as precisely. Since the paper acknowledges this, I am treating it just as a limitation rather than a flaw.
Questions for authors

Can you report how closely the cycled accuracy curve matches the predicted shift directly? Correlating changes that share the same baseline can overstate the match.

1 comment
AuthorsOpenReview ↗

Official Response to Reviewer fMtC

Dear Reviewer, thank you for your thoughtful review and for recognizing that average scores can hide periodic retrieval failures. Our general responses also report phase sensitivity in GLM-5.3-Flash and introduce a phase-resolved evaluation protocol, which shows that on standard long-context benchmarks a 3.2 pp gap in mean accuracy hides an 8.2 pp gap in robust accuracy.

Now we respond to each point in your review as follows:


Weakness 1: Causal link between the controlled explanation and DeepSeek's failures

The controlled models show the mechanism convincingly, but the paper has not shown that the same mechanism causes DeepSeek's failures.

Thank you for understanding that this is indeed a limitation of our work, and for noting that we have already stated it in the Conclusion (Section 7). Our mechanistic results come from the controlled models, where we can intervene on the compression gates directly. In DeepSeek, we cannot vary the training data, training recipe or architecture, so we cannot isolate the cause in the same way.

Following Reviewer mLo1's suggestion, we reran the DeepSeek evaluation so that only the target's phase varies. We find that the four-phase pattern persists, and phases 0 and 3 of the four-token period remain the weakest in both V4 base models (see part (a) of our first general response). However, this is still an observational result, unlike the gate-cycling intervention in our controlled models. In the revision, we will add this experiment to Section 2 and Appendix C, and we will keep framing the DeepSeek experiments in Section 2 as non-causal.


Question 1: Direct match between the cycled and predicted accuracy curves

Can you report how closely the cycled accuracy curve matches the predicted shift directly, since correlating changes that share the same baseline can overstate the match?

Following your suggestion, we computed the mean absolute error between the accuracy observed under gate cycling and the accuracy predicted by shifting the original per-phase scores by δ\delta. For reference, we also report the change in accuracy relative to the original model, both averaged over phases and at the phase with the largest change. We define the fraction explained as 1 − error / change. All values are averaged over δ=±1,±2,±3\delta = \pm 1, \pm 2, \pm 3 and computed over interior phases. Interior phases exclude the two boundary phases per shift.

ModelAvg. prediction error (pp)Avg. change in accuracy (pp)Largest single-phase change (pp)Avg. Fraction explained
W12/S12 1KV2.914.529.6–40.80.79
W12/S12 4KV3.219.734.0–43.90.83
W8/S8 1KV, seed 425.022.930.2–56.10.79
W8/S8 1KV, seed 436.318.332.8–50.70.62
W8/S8 1KV, seed 448.626.337.3–68.20.66
W8/S8 1KV, seed 458.413.519.0–34.40.39

We observe that the agreement is closer for the larger window size. For the W12/S12 models, the prediction error is 2.9 and 3.2 pp, and the predicted shift accounts for 79% and 83% of the change. For the W8/S8 models, the error is 5.0 to 8.6 pp, and the fraction explained ranges from 39% to 79% across training seeds.

In the revision, we will add this table and the per-shift values to Appendix G.


Thank you again for your constructive review. We hope these new results address your concerns on the gate shifting experiments, and we would be happy to answer any follow-up questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response 1: Additional experiments on phase sensitivity

We thank all reviewers for their constructive feedback. Below we report three additional experiments:

(a) phase sensitivity in DeepSeek-V4 when only the target's phase varies.

(b) phase sensitivity in GLM-5.3-Flash, another widely used open-source frontier LLM

(c) a partial remedy that shifts the compression boundaries across layers.

In response to Reviewer mLo1and utTm, we will also introduce a phase-resolved evaluation protocol for standard long-context benchmarks in the next response.


(a) Phase sensitivity persists in DeepSeek when only the target's phase varies.

Following Reviewer mLo1's suggestion, we rebuilt the 128K NIAH retrieval prompts with In-sequence Padding: 8,001 four-token key-value pairs separated by 8,000 shuffled gaps of 0 to 7 “|” tokens, with 1,000 gaps of each length. The target's position has the same distribution in every phase (mean 64,000 tokens), and the query position and prompt length are fixed. We find that the DeepSeek V4 models still show large phase gaps (28.9 pp for V4-Flash-Base and 20.7 pp for V4-Pro-Base), and phases 0 and 3 remain the weak spots:

01234567
DSV4-Flash-Base, Acc (%)62.564.876.647.760.568.073.450.0
DSV4-Flash-Base, Acc diff from best (pp)−14.1−11.70.0−28.9−16.0−8.6−3.1−26.6
DSV4-Pro-Base, Acc (%)83.296.192.685.277.798.494.186.3
DSV4-Pro-Base, Acc diff from best (pp)−15.2−2.3−5.9−13.3−20.70.0−4.3−12.1

(b) Phase sensitivity extends beyond DeepSeek CSA to the sparse attention in GLM-5.3-Flash.

We find that phase sensitivity also appears in GLM-5.3-Flash [1], another popular open-source model served with tens of trillions of tokens. GLM-5.3-Flash retains the full KV cache, but its sparse-attention indexer works on keys compressed in chunks of W=4 tokens. In the setting of (a), adapted to GLM's tokenizer, we find that GLM-5.3-Flash shows a periodic 22.7 pp accuracy gap with a four-token period, with its weakest phases right before the compression boundaries (phases 3):

01234567
GLM-5.3-Flash, Acc (%)70.369.169.950.469.173.069.952.3
GLM-5.3-Flash, Acc diff from best (pp)−2.7−3.9−3.1−22.7−3.90.0−3.1−20.7

(c) A partial remedy for phase-sensitivity: shifting the compression boundaries across layers.

We tried a potential remedy on our controlled 0.6B models. With window size WW, layer ii leaves its first i mod Wi \bmod W tokens uncompressed and starts compressing at token i mod Wi \bmod W, so no token position lies on a boundary in every layer. For example, with W=8W = 8, layer 0 compresses positions 0–7, 8–15 and so on, layer 1 keeps position 0 uncompressed and compresses positions 1–8, 9–16 and so on, and layer 2 keeps positions 0–1 uncompressed and compresses positions 2–9, 10–17 and so on. We compare each model with its non-shifted baseline and mark the baselines' boundary phases in bold.

01234567
W8/S8 baseline, Acc (%)66.771.678.745.047.324.364.76.2
W8/S8 baseline, Acc diff from best (pp)−12.0−7.20.0−33.7−31.5−54.4−14.0−72.5
W8/S8 with per-layer offset, Acc (%)62.361.142.165.350.069.234.352.6
W8/S8 with per-layer offset, Acc diff from best (pp)−6.9−8.1−27.1−3.9−19.20.0−34.9−16.6

At W4/S4, the eight phases cover two periods:

01234567
W4/S4 baseline, Acc (%)42.752.551.115.941.952.452.215.7
W4/S4 baseline, Acc diff from best (pp)−9.80.0−1.4−36.6−10.6−0.1−0.3−36.8
W4/S4 with per-layer offset, Acc (%)49.436.948.362.149.637.047.462.0
W4/S4 with per-layer offset, Acc diff from best (pp)−12.7−25.2−13.80.0−12.5−25.0−14.70.0

We find that the per-layer offset largely alleviates the boundary drop (as boundaries are no longer well-defined with layer-by-layer shifting). However, the oscillation remains periodic, and the spread across phases only narrows from 72.5 to 34.9 pp at W8/S8 and from 36.8 to 25.2 pp at W4/S4. We therefore see it as a partial fix rather than a complete solution.


We hope these additional results can further demonstrate that phase sensitivity is a robust and previously unnoticed failure mode of open frontier LLMs with compressed attention, and that a full solution will require further investigation.

References

[1] Z.ai. GLM-5.3-Flash. Hugging Face model card (zai-org/GLM-5.3-Flash), 2026.

AuthorsOpenReview ↗

General Response 2: A phase-resolved evaluation protocol for standard long-context benchmarks

In response to Reviewer mLo1's request for a phase-resolved evaluation protocol and Reviewer utTm's question on general long-context tasks, we introduce a lightweight protocol that turns existing benchmarks into tests of retrieval stability.

Measuring phase sensitivity on existing benchmarks is not straightforward. In each problem instance, the target information falls at an arbitrary phase, so a benchmark score averages accuracy over all phases and the periodic oscillation is not visible in the mean. We therefore measure the stability of each answer instead. A phase-sensitive model should give different answers to the same problem instance when that problem instance is shifted by a few tokens.

Following the filler sweeps in Section 2, we build eight variants of each problem instance by prepending a natural-language prefix of LL tokens, with L∈L={128,…,135}L \in \mathcal{L} = \lbrace 128, \dots, 135 \rbrace. Every prefix is the last LL tokens of the same fixed passage, so the tokens right before the problem instance are identical across the eight variants, and only the position of the problem instance relative to the compression boundaries changes. With the stride of 4 in DeepSeek-V4, the eight variants cover each phase twice. The protocol changes no benchmark content and needs no access to model internals.

We evaluate every variant by teacher forcing the gold answer. For problem instance ii with prefix length LL, ci(L)∈{0,1}c_i(L) \in \lbrace 0, 1 \rbrace equals 1 if every gold-answer token is the model's top prediction, which means that greedy decoding reproduces the gold answer, and ℓi(L)\ell_i(L) is the log-probability of the gold answer. Over NN problem instances, we report the mean accuracy 18N∑i∑Lci(L)\frac{1}{8N} \sum_i \sum_L c_i(L) together with three stability measures:

  • Robust accuracy: 1N∑imin⁡Lci(L)\frac{1}{N} \sum_i \min_L c_i(L), which reflects the fraction of problem instances answered correctly under every prefix.
  • Log-probability variance: 1N∑i17∑L(ℓi(L)−ℓˉi)2\frac{1}{N} \sum_i \frac{1}{7} \sum_L (\ell_i(L) - \bar\ell_i)^2 with ℓˉi\bar\ell_i the mean over L\mathcal{L}, which reflects the average variance of the gold-answer log-probability across prefixes.
  • Flip rate: 1N∑i[max⁡Lci(L)−min⁡Lci(L)]\frac{1}{N} \sum_i [\max_L c_i(L) - \min_L c_i(L)], which reflects the fraction of problem instances answered correctly under some prefixes and incorrectly under others.

We apply the protocol to 366 retrieval problem instances of 64K to 128K tokens from OpenAI MRCR [1, 2], BABILong [3], HELMET [4], RULER [5] and LongBench v2 [6], and compare DeepSeek-V4-Flash-Base with DeepSeek-V3.1-Base as a reference model without KV chunked compression. We will use V4-FB to denote DeepSeek-V4-Flash-Base and V3.1-B to denote DeepSeek-V3.1-Base in the following table.

Mean Acc.Robust Acc.Log-prob var.Flip rate (%)
BenchmarkV3.1-BV4-FBV3.1-BV4-FBV3.1-BV4-FBV3.1-BV4-FB
MRCR subsets32.028.522.714.70.1990.77920.030.0
BABILong subsets54.150.937.527.50.0540.09932.550.0
HELMET subsets89.087.386.477.30.0390.0886.115.2
RULER subsets99.495.198.493.80.0060.0021.66.2
LongBench v2 subsets60.456.250.033.30.0260.05116.750.0
Average59.456.151.142.90.1020.35816.927.9
Relative diff. (V4-FB vs. V3.1-B)−5.5%−16.0%+251.0%+64.5%

We can see that DeepSeek-V4-Flash-Base reaches a mean accuracy of 56.1%, which is 3.2 pp below the 59.4% of V3.1-Base. However its robust accuracy is 42.9% against 51.1% with a gap of 8.2 pp, which is over 2.5 times the gap in mean accuracy. The log-probability variance of V4-Flash-Base is also 3.5 times that of V3.1-Base (0.358 against 0.102), and the flip rate of V4-Flash-Base is 27.9%, against 16.9% for V3.1-Base.

These additional statistics therefore reveal variation and non-robustness that a standard benchmark score hides.

In the revision, we will add the protocol and the benchmark results to Section 2 and Appendix C.


References

[1] OpenAI. Introducing GPT-4.1 in the API. OpenAI Blog, 2025.

[2] Vodrahalli et al. Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries. arXiv:2409.12640, 2024.

[3] Kuratov et al. BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack. NeurIPS Datasets and Benchmarks Track, 2024.

[4] Yen et al. HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly. ICLR, 2025.

[5] Hsieh et al. RULER: What's the Real Context Size of Your Long-Context Language Models? COLM, 2024.

[6] Bai et al. LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks. ACL, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.