acceptodds
Under review as a conference paper at ICLR 2027

Locate, Then Continue: A Framework for Shared KV Memory in Hybrid Attention Models

12 views

Abstract

Hybrid attention models combine linear attention for efficient long-context modeling with periodic full-attention calls for exact retrieval, yet it is unclear how far those calls, though occurring at different depths, can share computation and state, and what minimal degree of sharing exact retrieval still requires. We answer this question with a descriptive framework that views each full-attention call as writing to and reading from an associative memory. Within this framework, exact retrieval composes two read types: associative lookup, which locates a record by matching its identity, and sequence continuation, which copies the record verbatim by matching predecessor windows against the generated prefix. Reads of one type share a single key function and differ only in their queries, so two read types suffice for direct, chained, and multi-target retrieval regardless of read count, hop count, content length, or model depth. Conversely, under an empirically motivated role-stability premise, that queries alone cannot reliably switch one cache between lookup and continuation, one layer does not suffice, making two layers the minimum, typically ordered as lookup then continuation. Experiments on 0.5B and 1.3B hybrids support the framework's predictions: the shallow group specializes in lookup and the deep group in continuation, merging two continuation groups preserves retrieval, and a single shared layer fails to recover verbatim copying. The resulting design, Group Share, reduces independent full-attention layers from seven to two at 1.3B, saving 10.4% of parameters and 57.4% of total cache at 16K while maintaining general capabilities; it even yields higher retrieval accuracy at context lengths from 2K to 16K.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, Hybrid attention, Cross-layer KV cache sharing, Long-context retrieval, Efficient architectures

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PvasOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2(+1)
Soundness
1
Read review
Paper summary

This paper modified Qwen architecture: sharing attention layers groupwise and claim a better retrieval performance. They also provided a analysis framework for that.

Critical strengths

They find that less unique attention layers may lead to better retrieval. This is not completely new idea, but they did test it on qwen 3.5 and the retrieval improved.

Critical weaknesses

qwen 3.5 is a hybrid architecture. The GDN layers also have context memory and also design for associative memory actually. Though GDN has fixed size memory, it is still not sure its impact on retrieval. So the conclusion on attention is not isolated.

The ppl and downstream tasks degrade. This may be a more severe problem than the improvement in retrieval.

Questions for authors

How can you isolate your study on attention only? Maybe test on a attention-only architecture.

There are plenty of works on KV reuse, like the YOCO[https://www.alphaxiv.org/pdf/2405.05254], deepseek v4.1 flash[https://arxiv.org/abs/2609.19969]. Their empirical results are strong enough that models with KV reuse in attention can do good retrieval. What's the contribution of this work?

7 comments
AuthorsOpenReview ↗

Dear Reviewer Pvas(1/2)

Thank you for the review. New experiments show that exact retrieval in our models is primarily executed by full attention, while Gated DeltaNet plays a limited direct memory role (item 1). We also clarify that downstream general tasks show no significant average decline (mean drop of 0.08 pp), and the perplexity gap stems mainly from FFN parameter reduction (item 2). Relative to YOCO and DeepSeek-V4.1-Flash, our contribution establishes how many independent layers exact retrieval requires and uncovers the underlying mechanism (item 3). We have revised the presentation to highlight these contributions clearly (item 4).

1. Separating attention and Gated DeltaNet memory contributions (W1, Q1) [new]. Bidirectional ablations show that exact retrieval is primarily driven by full attention, whereas Gated DeltaNet contributes modestly as a direct memory store. At 1.3B, we masked the needle in one pathway while keeping the other intact. When masked in full attention, retrieval accuracy drops to 0 across all tasks. Conversely, removing the target tokens from the recurrent state of all 21 Gated DeltaNet layers causes only modest drops:

TaskModelNativeHidden from full attentionControlHidden from Gated DeltaNetControl
single-1G2100.00.0100.0100.099.7
Baseline100.00.0100.0100.0100.0
single-2G2100.00.0100.0100.0100.0
Baseline100.00.0100.0100.0100.0
single-3 (UUID)G294.30.094.376.078.3
Baseline96.30.096.385.393.3
multikeyG256.70.056.751.753.0
Baseline47.00.046.741.350.0
multivalueG238.10.037.426.030.7
Baseline22.80.023.121.526.5
multiqueryG236.70.036.826.727.1
Baseline27.30.027.920.728.5
MeanG270.960.0070.8663.3964.79
Baseline65.580.0065.6761.4766.39

When full attention cannot read the needle, all tasks drop to 0 in both models. When Gated DeltaNet excludes the needle, the mean drops by only 1.4 pp (G2) and 4.9 pp (Baseline) relative to controls. However, Gated DeltaNet plays a specific indirect role: a few heads write preceding-token features into continuation keys, and zeroing this channel drops UUID from 94.3% to 0.3% [new]. Prior work on induction circuits shows that previous-token heads supply this preceding-token feature in pure-attention models (Olsson et al., 2022). We therefore expect a similar read mechanism to hold in pure Transformers. Due to rebuttal time and compute constraints, direct verification on pure-attention backbones remains for future work.

AuthorsOpenReview ↗

Dear Reviewer Pvas(2/2)

2. Perplexity and downstream tasks (W2). Downstream general tasks show no significant average decline. On the seven zero-shot benchmarks in Table 2 (1.3B, lm-evaluation-harness), Group Share trails Baseline by only 0.08 pp on average (46.97% vs. 47.05%), with all individual gaps within 0.68 pp and gains on PIQA, ARC-Easy, and LAMBADA:

Zero-shot accuracy (%)HellaSwagPIQAWinoGrandeARC-EasyARC-ChallengeLAMBADABoolQMean
Baseline (1.308B)35.9470.0253.4348.8621.2541.6758.1747.05
Group Share (1.172B)35.3670.2452.8849.5420.9041.8058.0446.97
Difference (pp)-0.58+0.22-0.55+0.68-0.35+0.13-0.13-0.08

At 1.3B, Group Share reduces parameters by 10.4% and full-attention cache by 71.4%, incurring a 2.09% perplexity increase while improving six-task retrieval by 5.00 pp. Given discussion-period compute limits, we ran 0.5B ablations in place of retraining 1.3B models: two-cache variants with independent FFNs achieve lower PPL than Baseline (3.3135 vs. 3.3168), whereas sharing FFNs matches G2 (3.3252 vs. 3.3258) [new] (Table R1). Moreover, 1.3B training curves show that the PPL gap does not widen over training. Together, the 0.5B ablations trace the perplexity increase to FFN parameter reduction, while our main 1.3B model demonstrates a favorable trade-off across perplexity, parameter count, cache savings, and downstream performance.

Excerpt from Table R1. 0.5B models: caches, sharing, loss, and retrieval.

ModelParams (M)Loss
Baseline501.663.3168
Two caches, K/V [new]500.093.3121
Two caches, K/V + attention [new]493.013.3135
Two caches, K/V + FFN [new]468.243.3252
G2461.153.3258

3. Contribution relative to YOCO and DeepSeek-V4.1-Flash (Q2). YOCO generates one global KV cache via a self-decoder for independent cross-decoder layers, while DeepSeek-V4.1-Flash (released September 17, 2026) assigns fixed layers to reuse earlier KV states or sparse indices. Both propose specific reuse topologies for inference efficiency. In contrast, we characterize the sharing limits and circuit mechanism for exact retrieval, showing that merging distinct read types into a single cache or layer substantially degrades retrieval. On the same backbone, a YOCO-style single cache (KV-only) reaches only 39.00% UUID versus 79.00% for G2 (Table R1). Our query-offset interventions explain why one jointly trained layer fails to serve both lookup and continuation (General Response 1/2) [new]. These findings establish concrete design rules for periodic hybrids on group count, sharing scope, cache-writing depth, and K/V tying. Unlike YOCO, Group Share preserves standard hybrid topologies without altering layer layouts. Moreover, DeepSeek-V4.1-Flash's reuse of KV with recomputed queries aligns with our finding that homogeneous reads can share keys while specializing in queries. We will discuss and cite this work in the revision.

Excerpt from Table R1. 0.5B models: caches, sharing, loss, and retrieval. UUID: exact match (%) on all 300 test items.

ModelCachesParams (M)UUID
Baseline6501.6679.67
KV-only1499.7039.00
Two caches, K/V [new]2500.0962.67
G22461.1579.00

4. Presentation. We have condensed §2 and restated our contributions around group count determination, the query-offset mechanism, and design rules (General Response 1/2).

Thank you for the comprehensive response and new ablations. Your analysis tracing the PPL gap to FFN sharing and clarifying the negligible zero-shot drop (-0.08 pp) resolves my quality concerns. The new mechanistic interventions are also valuable, so I will raise my score.

Before deciding whether to raise it further, two core concerns remain from your new results:

Two-cache lower bound (Table R1): Adding query heads to a single-cache model (More query heads, 1 cache) reaches 76.33% UUID (close to G2's 79.00%) and outperforms G2 on the 6-task average (58.90% vs. 57.76%) with lower loss and fewer parameters. Doesn't this suggest G1's bottleneck is query-head capacity rather than a fundamental need for two caches?

GDN coupling & masking drop (Table R3 & Item 1): Zeroing GDN's lag-1 channel drops UUID to 0.3%, meaning two-group sufficiency relies on GDN providing "free" previous-token features (which pure-attention models would need an extra attention group to compute). Also, in Item 1, masking GDN drops UUID from 94.3% to 76.0% (and Control to 78.3%)—why does the control itself degrade performance so sharply?

Looking forward to your clarification.

Reviewer_5x5MOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies cross-layer sharing of full-attention parameters and KV caches in hybrid attention models. It describes exact retrieval through two operations: associative lookup and sequence continuation. This framework motivates Group Share, which partitions full-attention calls into two contiguous groups, each sharing an entire block and one KV cache. Experiments at 0.5B and 1.3B investigate functional specialization and cross-depth memory reuse. The method reduces parameters and cache memory while achieving competitive benchmark performance.

Critical strengths
  1. Interesting functional perspective. The distinction between locating a record and continuing its content provides an intuitive way to reason about KV sharing in hybrid attention models. Separating the number of reads from the number of independent memories is useful.
  2. Substantial effort toward mechanistic validation. Group ablations, common-parent merging, fixed-KV adaptation, and content substitution go beyond aggregate benchmark comparisons. These experiments provide meaningful evidence that later layers can use shared memory.
  3. Practical memory savings. At the tested 1.3B scale, Group Share reduces parameters by 10.4% and total cache memory at 16K by 57.4%, with broadly competitive general-task accuracy.
Critical weaknesses
  1. The KV-only baseline uses one cache, whereas Group Share uses two. A two-cache baseline with identical grouping but independent Q/O projections and FFNs is missing. The experiments therefore do not isolate the benefit of whole-block sharing over conventional KV sharing.

  2. The lower bound depends on role stability, which is assumed rather than established. Single-cache models have half the KV storage, and the merged G1 inherits lookup-specialized parameters that remain frozen. These controls cannot separate a functional limitation from insufficient capacity or adaptation difficulty.

  3. Group ablations and linear probes show different task sensitivities, but both diagnostic task families require lookup and continuation. They do not distinguish predecessor matching from alternatives such as identity-plus-position addressing. The assumed retrieval-progress variable is also not mechanistically established.

  4. Under role stability, supporting both LC and CL within one forward pass should require three groups. The multi-hop experiment instead externalizes intermediate identities across generation steps. This leaves an important test of the framework’s predictive value unresolved.

Questions for authors
  1. Can you compare against a two-cache KV-sharing baseline with identical group boundaries, call positions, and training budgets? What changes when Q/O and FFN sharing are ablated separately?
  2. Does the two-group advantage persist against a single cache with equal total KV storage, trained jointly from scratch? Does the merged G1 result persist when its full-attention parameters can adapt?
  3. Can selective perturbations of record identity, predecessor windows, and position distinguish the proposed mechanisms? How is retrieval progress represented and updated, particularly when content contains repeated substrings?
  4. Can you test a task family requiring both LC and CL without intermediate generated tokens, and compare two versus three groups under controlled memory budgets?
4 comments
AuthorsOpenReview ↗

Dear Reviewer 5x5M(2/2)

Q3. Selective perturbations and retrieval progress [new]. The three proposed perturbations differentiate the two mechanisms, providing clear empirical support for predecessor matching (General Response 2/2, Tables R3–R4). At 1.3B, we perturbed the predecessor window, record identity, and position independently. Zeroing the channel that writes preceding-token features drops G2 UUID from 94.3% to 0.3%.

Excerpt from Table R3. Source and causal test of continuation keys at 1.3B [new]. Metric: exact-match UUID (%) across 300 items (the first target token is unaffected in all conditions).

ConditionG2Baseline
native94.396.3
lag-1 channel of writer heads zeroed0.34.0
control: lag-2 channel of writer heads zeroed93.796.3
control: lag-1 channel of other same-layer heads zeroed94.795.3

When inserting distractor digit strings matching the initial digits of the target before diverging, copying errors increase systematically with prefix length (G2: 6% with no shared digits, 19% with 2, 33% with 5). Distractors structured as separate records are copied equally often (33%), whereas identity-plus-position predicts 0%.

Excerpt from Table R4. Distractor copying at 1.3B. We report how frequently the model copies the distractor at the divergence step (%). Unless noted, the distractor lies far from the target.

DistractorG2Baseline
shares no window with the target (control)63
shares the first 2 digits194
shares the first 5 digits3324
2 digits, written as another record3320

Position breaks ties only when predecessor windows are identical, favoring the earlier instance (38% vs. 3%).

Excerpt from Table R4. Distractor copying at 1.3B. We report how frequently the model copies the distractor at the divergence step (%). Unless noted, the distractor lies far from the target.

DistractorG2Baseline
shares the first 2 digits194
2 digits, placed just before the target3862
2 digits, placed just after the target31

Regarding retrieval progress tracking, the model maintains no explicit progress index, as progress is implicit in the generated suffix and updates auto-regressively. Substring repetitions trigger copying errors only when the suffix context cannot disambiguate them. Repeating a 5-token block within a single value produces zero errors, showing that the effective predecessor window extends beyond the kernel size (4) of Gated DeltaNet short convolutions. We revise §2.2 and §4.1 accordingly.

Q4. Lookup-then-continuation and reverse order in a single pass. At this scale, pointer-chasing without intermediate scratchpads cannot test the three-group prediction, which we defer to open questions. In this setting, all architectures succeed at 1 hop (98–100%) but drop to chance at ≥2 hops, including the six-layer Baseline (with a single seed of one two-group variant achieving 97.5% at 2 hops). Because even Baseline fails to compose distinct reads in a single forward pass, this benchmark does not separate two groups from three. Moreover, the record syntax allows continuation alone to bridge a hop, preventing a clean test of the reverse read order. Our claims hold for direct, chained, and multi-target retrieval, while single-pass composition is marked as open alongside this benchmark design.

AuthorsOpenReview ↗

Dear Reviewer 5x5M(1/2)

Thank you for the precise questions, which we address with targeted experiments. A second cache provides substantial benefits whether sharing only K/V or entire layers (Q1). The bottleneck of a single cache lies in query specialization rather than storage capacity (Q2). Continuation addresses keys via preceding-token matching rather than identity-plus-position (Q3). We could not verify the single-pass three-group prediction at this scale and designate it as an open question (Q4).

Q1. Two caches with only K/V shared, and attention versus FFN sharing [new]. A second cache helps across both configurations. Sharing either attention projections or FFNs alongside K/V matches G2 performance. The requested models follow G2's grouping, call positions, and training budget (Table R1, General Response 2/2). When sharing only K/V, adding the second cache raises UUID from 39.00% to 62.67%. When sharing the entire layer, it raises UUID from 62.33% to 79.00%. Perplexity divides cleanly into two tiers depending on whether FFNs are shared. The two-cache K/V-only model trails G2 mainly due to lookup: when given the first target token to bypass lookup, it reaches 97.33% UUID. Its learned lookup offset is weak, and freezing all weights while doubling this offset at the target start raises held-out UUID from 57.33% to 73.33% (General Response 2/2).

Excerpt from Table R1. 0.5B models: caches, sharing, loss, and retrieval. UUID: exact match (%) on all 300 test items.

ModelCachesParams (M)LossUUID
Baseline6501.663.316879.67
KV-only1499.703.317939.00
Two caches, K/V [new]2500.093.312162.67
Two caches, K/V + attention [new]2493.013.313581.67
Two caches, K/V + FFN [new]2468.243.325278.00
G22461.153.325879.00

Q2. Equal KV storage in a single cache, and adaptable merged G1 [new]. Equal storage alone does not close the performance gap. At 0.5B, we trained a single-cache ablation model (Wide KV) from scratch with KV storage matching G2's combined two caches and 8 query heads, reaching 73.67% UUID (vs. 79.00% for G2). Its gain over G1 stems from query heads rather than storage: adding 8 query heads to standard G1 storage achieves 76.33% (Table R1).

Excerpt from Table R1. 0.5B models: caches, sharing, loss, and retrieval. UUID: exact match (%) on all 300 test items.

ModelCachesShared among the calls of a groupUUID
G11entire layer62.33
Wide KV [new]1entire layer, 2× KV storage, 8 query heads73.67
More query heads [new]1entire layer, 8 query heads76.33
G22entire layer79.00

This holds when merging a trained Baseline into G1 and fine-tuning: even adapting its query projections (or all full-attention weights) alongside GDN layers, UUID remains 0.00% across three seeds (six-task mean: 46–47%), whereas G2 reaches 99.5–99.7% UUID (mean: 89–90%).

Both findings confirm that the bottleneck resides in query formulation rather than cache capacity. Given the initial target token, G1 smoothly completes continuation, and freezing K/V while training solely a dedicated lookup query elevates G1 from 62.33% to 89.00%. Premise H captures this joint-optimization constraint: while a single cache can store the necessary states, joint training struggles to satisfy both reads simultaneously within one shared layer (General Response 1/2, Table R2).

Excerpt from Table R2. Tolerance to lookup offset at 0.5B. UUID: exact match (%); all weights frozen. The lookup offset is the mean query vector at the first target position minus that of the same token elsewhere. The interventions in Table R2 split items into two equal sets: 150 items to estimate the offset vector and 150 held-out items for evaluation.

ModelNo interventionOffset removed at target start (lookup)Fixed offset added at target start (lookup)
G160.019.385.3
G277.314.086.0

Thank you for the thorough rebuttal and the new experiments, which resolve my concerns about the KV-sharing baseline (Q1) and the continuation mechanism (Q3). I maintain my score of 3. The key remaining issues are:

  1. New results weaken Premise H: single-cache models with more query heads are statistically indistinguishable from G2 on UUID, and a dedicated lookup query on G1’s frozen cache reaches 89.00%. So “two layers are the minimum” is empirical observation at 0.5B.
  2. Merged-G1 result is weak support for H because it uses only lookup group’s parameters, a small adaptation budget, and three repeats from one parent rather than independent seeds.

Thank you again for your review.

As the rebuttal period closes, we would like to check whether our response addresses your concerns.

Our additional experiments suggest that a single group is limited by the conflicting demands of the two reads on one shared layer rather than by storage capacity or weight adaptation. Specifically, a single cache with G2's KV storage still trails G2, and the merged G1 stays at 0.00% even when its full-attention weights adapt (Table R1\textcolor{red}{\text{Table R1}}). With all weights frozen, the lookup offset raises G1 from 60.0% to 85.3% at the target start but drops it to 0.0–8.7% at continuation steps (Table R2\textcolor{red}{\text{Table R2}}). This trade-off indicates that a single shared layer struggles to serve both reads simultaneously. In addition, continuation relies on predecessor matching for addressing (Tables R3–R4\textcolor{red}{\text{Tables R3–R4}}). Zeroing the lag-1 channel of the writer heads drops G2 UUID from 94.3% to 0.3%.

Taken together, these results align with our main hypothesis. In this setting, the number of independent full-attention layers needed for exact retrieval appears to depend on the number of distinct read types rather than the total number of calls. Exact retrieval in our tasks involves only lookup and continuation. Consequently, two groups are sufficient in these scenarios.

We hope these additional experiments and clarifications help resolve any remaining questions. We sincerely appreciate your time and constructive feedback.

Reviewer_yVxzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Group Share, which partitions full-attention calls in a hybrid language model into groups sharing block parameters and KV caches while retaining all calls. It interprets exact retrieval as lookup followed by continuation, motivating two groups, with minimality conditional on role stability. Experiments on 0.5B and 1.3B models investigate this interpretation. At 1.3B, the method reduces parameters by 10.4% and total cache at 16K by 57.4%, while improving average retrieval scores over an independent-layer baseline, with a small language-modeling loss penalty.

Critical strengths
  • The integration of cross-layer parameter and KV-cache sharing achieves meaningful resource savings while largely preserving performance on the reported general benchmarks.

  • The locate/continue framework offers an interesting interpretation of shared-memory retrieval, supported by group ablations, merging experiments, and cache interventions.

  • Fixed-cache adaptation and content substitution provide useful evidence that later readers can access and use shared KV content.

Critical weaknesses

1. Positioning relative to prior work

The paper should discuss Universal YOCO (YOCO-U) [1], which already combines recursive parameter sharing in its self-decoder with cross-layer global KV sharing in its cross-decoder. Although Group Share differs in its backbone and in where parameters and caches are shared, these architectural differences do not, by themselves, establish a substantive obstacle to applying the existing combination of sharing techniques to the setting studied here. The authors should clarify what additional technical challenge Group Share addresses and what is novel beyond this combination. A comparison with a suitably adapted prior sharing architecture would strengthen this positioning.

2. Scope and usefulness of the locate/continue interpretation

The proposed interpretation is interesting, but it primarily characterizes retrieval and copying. Since the architecture is intended for general-purpose language modeling, it remains unclear how this retrieval-specific account justifies the sharing structure when attention must also support other computations. Prior work on retrieval heads [2] identifies a sparse subset of heads with retrieval behavior across layers. The paper should explain how its group-level interpretation relates to these head-level findings. While the group ablations, probes, and cache interventions support functional differentiation, more fine-grained head- and token-level evidence is needed to establish that the observed roles specifically correspond to identity-based lookup and predecessor-based continuation, rather than a broader division of labor.

I am also not yet convinced of the framework's practical guidance for architecture design. Parameter and cache sharing can be motivated without this interpretation, as illustrated by existing architectures. The important question is therefore what additional, discriminating predictions the framework provides and how these guide the choice of group count, boundaries, and cache-writing locations. The untested prediction concerning an additional group for within-pass CL computations is particularly relevant here.

3. Quality-efficiency tradeoff and long-context evaluation

The persistent language-modeling loss gap relative to the baseline warrants further discussion. Although Group Share uses fewer parameters, this alone does not establish that its quality-efficiency tradeoff is competitive with existing sharing approaches. For example, YOCO-U reports competitive or lower loss under matched training-FLOP budgets. Cross-paper results are not directly comparable, but they motivate including stronger sharing baselines under controlled budgets.

Finally, pretraining is primarily conducted at 2K, followed by continued training and evaluation up to 16K. Retrieval performance already degrades substantially within this range for both models. For a method motivated by KV-cache efficiency, I consider evaluation at substantially longer contexts, ideally around 100K tokens or beyond, important for establishing practical relevance. Such experiments should report both retrieval quality and resource savings, rather than memory reduction alone.

References

[1] Universal YOCO for Efficient Depth Scaling. https://arxiv.org/abs/2604.01220v1

[2] Retrieval Head Mechanistically Explains Long-Context Factuality. https://arxiv.org/abs/2404.15574v1

Questions for authors

please refer to the weakness part

4 comments
AuthorsOpenReview ↗

Dear Reviewer yVxz(1/2)

Thank you for directing us to YOCO (YOCO-U) and the retrieval heads literature. YOCO and YOCO-U study how to build an efficient decoder around a single cache, whereas we investigate how many independent full-attention layers exact retrieval fundamentally requires (W1). Our answer, one group per read type, is grounded in a verified mechanism and yields actionable architectural guidelines (W2). The perplexity gap stems from FFN parameter reduction rather than cache sharing, and cache savings scale substantially with context length (W3).

W1. Positioning relative to YOCO and YOCO-U. We apologize for omitting a citation to YOCO-U and only briefly comparing with YOCO; the revision thoroughly discusses both in §1. Structurally, YOCO uses a self-decoder to produce one global KV cache that cross-decoder layers read with independent parameters. YOCO-U extends this by executing the self-decoder recursively, trading 2–3× extra compute for greater effective depth while keeping cache-reading layers independent. Both optimize decoders around a single cache.

Additional technical challenge. Group Share seeks the minimal number of independent full-attention layers needed for exact retrieval in periodic hybrids. Reducing groups saves cache and parameters, yet collapsing to one group drops UUID from 79.67% to 62.33%, and simply adding capacity cannot compensate (widening KV reaches 73.67%; adding query heads reaches 76.33%). The core technical challenge is preventing functional interference when multiple calls across different depths share weights and cache.

Novelty beyond simple combination. Rather than merely combining periodic hybrids with KV sharing, we uncover the underlying circuit mechanism and establish a principled design rule: the minimal group count equals the number of distinct read types. Specifically, lookup requires a query offset that continuation cannot tolerate, prohibiting them from sharing a layer, whereas homogeneous reads can share one layer regardless of call count or depth (detailed in W2).

Excerpt from Table R2. Tolerance to lookup offset at 0.5B. UUID: exact match (%); all weights frozen. The lookup offset is the mean query vector at the first target position minus that of the same token elsewhere. The interventions in Table R2 split items into two equal sets: 150 items to estimate the offset vector and 150 held-out items for evaluation. Parentheses: control using an equal-norm random vector.

ModelNo interventionOffset removed at target start (lookup)Offset ×2 at continuation steps, shallow callsOffset ×2 at continuation steps, deep calls
G160.019.30.0 (57.3)8.7 (59.3)
G277.314.034.7 (70.7)2.7 (42.0)

Comparison with adapted prior architectures. On the exact same hybrid backbone, adapting YOCO so the first full-attention call writes a single global KV cache read by the remaining independent layers yields only 39.00% UUID. Adding a second KV cache raises this to 62.67% [new], whereas tying entire layers in two groups (G2) achieves 79.00% without extra compute (Table R1). Notably, YOCO-U's recursive execution is orthogonal to Group Share and can be combined in principle, though we have not empirically tested this combination (left for future work).

Excerpt from Table R1. 0.5B models: caches, sharing, loss, and retrieval. UUID: exact match (%) on all 300 test items.

ModelCachesParams (M)UUID
Baseline6501.6679.67
KV-only1499.7039.00
Two caches, K/V [new]2500.0962.67
G22461.1579.00
AuthorsOpenReview ↗

Dear Reviewer yVxz(2/2)

W2. Practical guidance and fine-grained mechanisms.

Scope of explanation. Our framework specifically explains exact retrieval rather than all general language modeling computations in full attention. Because Group Share retains every full-attention call, Gated DeltaNet layer, and residual stream, non-retrieval computations remain supported by these preserved pathways. Notably, sharing only attention while keeping FFNs independent yields lower loss than Baseline (3.3135 vs. 3.3168, Table R1).

Relation to retrieval heads. Wu et al. define retrieval heads via copy-paste behavior, identifying which heads execute retrieval. In contrast, we provide a causal explanation for why retrieval succeeds. Section 2 derives what each read type matches against, and our interventions test the corresponding components. Zeroing the preceding-token channel in the keys impairs continuation (Table R3), whereas removing the lookup offset from the query impairs lookup (Table R2). Their copy-paste behavior corresponds to continuation in our framework, while our analysis pinpoints how upstream heads generate the key features that continuation heads match.

Head- and token-level evidence. At the head level, a small subset of Gated DeltaNet heads writes the preceding token into the keys used for continuation via short convolutions. In 1.3B G2, zeroing this channel drops UUID from 94.3% to 0.3%, whereas zeroing the same channel in other heads has no effect (94.7%). Across all conditions, the first target token (lookup) remains unaffected [new] (Table R3).

Excerpt from Table R3. Source and causal test of continuation keys at 1.3B [new]. Metric: exact-match UUID (%) across 300 items (the first target token is unaffected in all conditions).

ConditionG2
native94.3
lag-1 channel of writer heads zeroed0.3
control: lag-2 channel of writer heads zeroed93.7
control: lag-1 channel of other same-layer heads zeroed94.7

At the token level, distractors sharing initial digits with the target are erroneously copied (up to 33%), whereas distractors placed right after the target are rarely copied (3%) (Table R4), confirming predecessor-based addressing.

Excerpt from Table R4. Distractor copying at 1.3B. We report how frequently the model copies the distractor at the divergence step (%). Unless noted, the distractor lies far from the target.

DistractorG2Baseline
shares no window with the target (control)63
shares the first 2 digits194
shares the first 5 digits3324

Design guidance. Grounded in our read-type mechanism, the framework informs three architectural choices before training.

(1) Group count: under our §2 decomposition for direct, chained, and multi-target retrieval, two read types suffice, pointing to two groups. Collapsing to G1 yields 62.33% UUID, which widening KV (73.67%) or adding query heads (76.33%) fails to match against G2 (79.00%). Merging a redundant non-retrieval group from G3 into G2 preserves accuracy (+0.56 pp).

(2) Group assignment: subject to prerequisite feature availability, homogeneous reads across depths can share one group. Cache writing should occur after upstream predecessor features form, and writing Group B's cache at layer 4 at 0.5B still reaches 81.00% UUID (Table 4).

(3) K/V tying: K and V within a group should be written jointly by the same projections. Writing K or V separately drops UUID to 64.67% and 11.67% (Table 4). Similarly, patching K and V jointly in the cache succeeds at 75.12%, versus 19.10% when patching V alone (§5). Exact group boundaries and single-pass three-group cases remain unverified at this scale and are listed as limitations.

W3. Downstream quality and long contexts. New ablations demonstrate that the perplexity gap arises from sharing FFNs rather than sharing caches: at 0.5B, two-cache models with independent FFNs attain lower loss than Baseline [new].

Excerpt from Table R1. 0.5B models: caches, sharing, loss, and retrieval.

ModelParams (M)Loss
Baseline501.663.3168
Two caches, K/V [new]500.093.3121
Two caches, K/V + attention [new]493.013.3135
Two caches, K/V + FFN [new]468.243.3252
G2461.153.3258

Cache savings expand with context length, reducing total cache footprint by 69.3% at 128K [new].

Memory at 1.3B. Group Share reduces total cache (full-attention KV plus recurrent states) by 57.4% at 16K and by 69.3% at 128K (from 923.5 to 283.5 MiB), lowering peak decoding memory at 128K from 3.72 to 2.84 GiB [new].

Within the 16K training window, Group Share outperforms Baseline across all eight evaluated context lengths. When continually trained to 32K, both architectures degrade at the 32K mark, with Group Share showing a steeper decline (19.28% vs. 23.94%) [new].

Thank you for the additional experiments and clarifications. I appreciate the effort, but two concerns remain.

1. Definition and interpretation of the lookup offset

The lookup-offset analysis remains difficult to assess. Please specify how the lookup offset vector is computed: which query positions are compared, what "the same token elsewhere" means, how averaging is performed, and whether the vector is estimated separately for each head, call, or group.

More importantly, I do not find the equal-norm random-vector control persuasive for establishing lookup-specific semantics. Matching the norm does not account for the distribution across dimensions or alignment with the keys, so the stronger effect of the lookup offset vector may reflect sensitivity to a structured direction rather than lookup-continuation interference. Table R2 reports random controls for the continuation interventions, but not for offset removal or addition at the target start. What evidence distinguishes a lookup-specific signal from other differences between answer-start and continuation queries? Clearer definitions and more targeted controls are needed to support the proposed interpretation.

2. Retrieval quality at longer contexts

The new 32K results do not preserve the retrieval advantage reported at 2K-16K: after continued training to 32K, Group Share scores 19.28%, compared with 23.94% for Baseline. Meanwhile, the 128K results establish memory savings but do not report corresponding retrieval quality. Thus, my concern about the practical quality-efficiency tradeoff at longer contexts remains unresolved.

Could the authors explain the steeper degradation of Group Share and provide the 32K training/evaluation details and per-task results? In particular, is there evidence that the larger memory savings at longer contexts can be achieved while retaining competitive retrieval quality?

AuthorsOpenReview ↗

Dear yVxz:

Thank you for acknowledging our additional work and for noting the limits of the equal-norm control and of our long-context evaluation. As suggested, we added key-aligned controls and per-task 32K-stage results, addressing the offset definition (Q1), lookup specificity (Q2), and longer contexts (Q3) below.

Q1. Offset definition. Using post-q_norm, pre-RoPE queries, we compute O=q‾(L)−q‾(same token)O = \overline{q}(L) - \overline{q}(\text{same token}) on 150 UUID items and test it on 146 held-out items. To answer your four questions directly, we first compare the query at position LL, right before the first value token, against the query at "the same token elsewhere." This means every other occurrence of that token ID in the sequence, primarily the matching token before the needle value where no retrieval occurs. Prior to subtraction, each term is averaged across all matching positions and items. We estimate OO separately for each head of every full-attention call, including each call within a shared group.

Q2. Lookup specificity. We agree that matching vector norm alone does not control for key alignment, so we introduce alignment-matched controls [new]. On the 150 estimation items, we decompose OO into a component along the onset key direction uu (the first value token's key minus the haystack mean) and an orthogonal remainder. Each key-aligned control is formed from the mean key contrast of another prompt token class, orthogonalized against uu and scaled to yield the same logit gain, and norm-matched controls scale OO's remainder and a random vector to the same norm. To address your point about missing target-start controls, we evaluate removing and adding both components of OO, as well as adding the random and key-aligned control vectors, at the target start LL in Table RAR_A. Although the onset component holds only 2–19% of OO's energy, removing it degrades lookup far more than removing the remainder, and only adding it clearly improves lookup.

To distinguish lookup from other differences between answer-start and continuation queries, we track which token the edited query reads. When we split D=q‾(L)−q‾(C)D = \overline{q}(L) - \overline{q}(C) the same way and add each part during continuation in Table RBR_B, only the onset component frequently causes the model to re-read the value's first token, triggering restarts in 39.0–55.5% of steps at the rescued LL level (defined in Table RAR_A), whereas the remainder and controls rarely trigger restarts and redirect reads elsewhere. Thus the signal is specific to lookup and that continuation cannot tolerate the level lookup requires.

Table RAR_A. Edits at the target start L (first value token correct, %)

Edit at LG2G1Baseline
None85.665.186.3
Remove O's onset component21.935.647.3
Remove O's remainder67.158.965.1
Add O's onset component93.882.995.2
Add O's remainder (same norm)84.962.385.6
Add random (same norm)84.264.484.9
Add control (same logit, best of 3)87.063.088.4
Remove D's onset component3.43.43.4
Add D's onset component (rescued L level)95.992.595.9

Table RBR_B. Additions at continuation steps (restarts in %, re-reading the value's first token)

Added at CG2G1BaselineReads go to
D's onset component at native L level0.10.10.0— (≥98.5% correct)
D's onset component at rescued L level55.551.339.0value's first token
D's remainder (whole)5.91.13.0other value tokens (19–37%)
Controls (same logit, max of 3)1.21.50.4own class or other value tokens

Q3. Long contexts. Group Share retains its retrieval advantage up to 20K while saving 57–60% of the cache (59.7% at 20K [new]), but not beyond 20K (Table RCR_C). Both 1.3B models continued from 16K for only 3,605 steps of 64K tokens (lr 2.5e-5, cosine) on 5,768 FineWeb documents over 32K tokens (0.19B tokens) and were evaluated on six RULER tasks (300 items each) as in the paper, leaving both under-trained beyond 20K. At 128K, we only estimated the 69.3% KV-cache savings without training at that length. We apologize for these training limitations and will accordingly restrict our long-context claims to 20K.

Table RCR_C. RULER after the 32K stage (1.3B, Baseline / Group Share, %)

Lengthsingle-1single-2UUIDMKMVMQMean
16K100.0/99.043.0/66.325.0/42.332.3/34.313.9/22.29.3/13.837.26/46.32
20K97.7/96.031.3/40.78.3/28.029.7/32.711.1/17.75.6/7.830.61/37.14
24K99.0/88.034.0/20.310.0/8.731.0/24.012.4/8.65.2/3.631.94/25.53
28K84.0/70.06.3/18.72.3/10.016.7/26.76.8/6.03.7/2.819.97/22.36
32K87.0/55.313.0/17.72.3/9.326.3/22.710.8/6.44.2/4.223.94/19.28

Together, the Q1–Q2 lookup-specific conflict and Q3 results up to 20K remain consistent with using one shared group per read type, and we thank you for pointing out the gaps in our earlier evidence.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Dear Reviewers and Area Chair: General Response (2/2), Results and Revisions

This comment collects the empirical results cited in General Response 1/2 and individual replies. At 0.5B, all models share the same backbone (with six full-attention calls), training data order, and compute budget. G1 shares one layer across all six calls, G2 (Group Share) uses two groups of three calls each, and Baseline allocates an independent layer to each call. We designate the first three calls as shallow and the last three as deep. UUID denotes exact match (%) on the UUID retrieval task, which is the only task among the six that distinguishes sharing architectures at 0.5B because its 36-character target requires one lookup followed by sustained continuation reads. Table R1 reports UUID on all 300 test items. The interventions in Table R2 split items into two equal sets: 150 items to estimate the offset vector and 150 held-out items for evaluation.

Table R1. 0.5B models: caches, sharing, loss, and retrieval. Retrieval is the six-task average.

ModelCachesShared among the calls of a groupParams (M)LossUUIDRetrieval
Baseline6none501.663.316879.6756.97
KV-only1K/V499.703.317939.0052.39
G11entire layer451.033.325362.3355.38
Wide KV [new]1entire layer, 2× KV storage, 8 query heads452.013.318973.6758.58
More query heads [new]1entire layer, 8 query heads451.623.323876.3358.90
Two caches, K/V [new]2K/V500.093.312162.6752.67
Two caches, K/V + attention [new]2K/V + attention projections493.013.313581.6760.47
Two caches, K/V + FFN [new]2K/V + FFN468.243.325278.0056.78
G22entire layer461.153.325879.0057.76

Table R2. Tolerance to lookup offset at 0.5B. The lookup offset is the mean query vector at the first target position minus that of the same token elsewhere. With all weights frozen, we remove it at the target start, add a fixed offset there, or double the offset at continuation steps in shallow versus deep calls. Evaluated on 150 held-out items, with parentheses reporting the control using an equal-norm random vector.

ModelCachesParams (M)No interventionOffset removed at target start (lookup)Fixed offset added at target start (lookup)Offset ×2 at continuation steps, shallow callsOffset ×2 at continuation steps, deep calls
G11451.0360.019.385.30.0 (57.3)8.7 (59.3)
Wide KV [new]: 2× KV storage, 8 query heads1452.0170.714.791.31.3 (44.0)36.7 (65.3)
More query heads [new]: 8 query heads1451.6272.725.392.00.7 (68.7)5.3 (66.0)
G22461.1577.314.086.034.7 (70.7)2.7 (42.0)
Baseline6501.6677.312.084.762.0 (76.0)0.0 (18.0)

Table R3. Source and causal test of continuation keys at 1.3B [new]. Linear decomposition of the cached keys shows that the preceding-token feature originates almost entirely from a few Gated DeltaNet heads right before the cache-writing layer (Group B's F16 in G2, and the primary continuation layer F12 in Baseline). We zero the lag-1 channel of their short convolutions, using the lag-2 channel and the lag-1 channel in other same-layer heads as controls. Metric: exact-match UUID (%) across 300 items (the first target token is unaffected in all conditions).

ConditionG2Baseline
native94.396.3
lag-1 channel of writer heads zeroed0.34.0
control: lag-2 channel of writer heads zeroed93.796.3
control: lag-1 channel of other same-layer heads zeroed94.795.3

Table R4. Distractor copying at 1.3B. A distractor digit string is inserted into the context: its first mm digits match the target value before diverging. We report how frequently the model copies the distractor at the divergence step (%). Unless noted, the distractor lies far from the target. Identity-plus-position addressing predicts 0% across all conditions.

DistractorG2Baseline
shares no window with the target (control)63
shares the first 2 digits194
shares the first 5 digits3324
2 digits, written as another record3320
2 digits, placed just before the target3862
2 digits, placed just after the target31

Memory at 1.3B. Group Share reduces total cache (full-attention KV plus recurrent states) by 57.4% at 16K and by 69.3% at 128K (from 923.5 to 283.5 MiB), lowering peak decoding memory at 128K from 3.72 to 2.84 GiB [new].

Revisions. We ground Premise H in the verified mechanism, condense §2, refine §4.2, and frame the single-pass composition as an open question. New material is incorporated into §1 and §4.1 (related work and continuation key circuits), §3 (design guidelines), §4.3 (offset mechanism and Fig. R1), §5 (needle ablation and memory scaling to 128K), §6 (limitations), and Table 4 (the five 0.5B ablation models).

AuthorsOpenReview ↗

Dear Reviewers and Area Chair: General Response (1/2), Claim and Additional Work

We thank the reviewers and Area Chair for their constructive feedback. This paper studies the number of independent full-attention layers required for exact retrieval in periodic hybrid language models, where multiple full-attention calls across depths share physical layers and KV caches. In our design, full-attention calls performing the same read type share a single layer across depths. Our derivation in §2 indicates that exact retrieval across direct, chained, and multi-target settings typically involves two read types: lookup finds where the target starts, and continuation copies it token by token. The reviews examine three core issues: functional specialization versus raw capacity, the circuit mechanism across full attention and Gated DeltaNet, and the source of the perplexity gap. Two reviewers also ask about YOCO-style cache sharing. To address these points, we conducted three sets of new experiments (marked [new]), with detailed results in General Response 2/2.

First, five new 0.5B ablation models separate the second cache from capacity (Table R1).A single-cache model with the same total KV storage still trails the two-cache G2 (two shared layer groups) on UUID retrieval (73.67% vs. 79.00%). In a pair of models with comparable parameters (499.7M and 500.1M) whose groups share only K/V, the second cache raises UUID from 39.00% to 62.67%. Thus, the second cache provides functional specialization rather than raw storage capacity. The perplexity gap comes from the parameters removed by sharing the FFN: two-cache models with independent FFNs achieve lower PPL than Baseline (unshared layers).

Second, query interventions explain the advantage of two groups over one. Lookup and continuation select keys through different mechanisms. Continuation selects the key whose preceding token matches the token just generated. Lookup has no generated target token to match, so its query must carry an additional component pointing to the target start, which we call the lookup offset. At the target start, removing this offset in G2 drops UUID from 77.3% to 14.0%. Conversely, injecting the offset into the continuation steps of deep calls drops UUID to 2.7% (vs. 42.0% for a norm-matched random vector). Shallow calls tolerate this offset substantially better than deep calls (Table R2), allowing two groups to specialize: the deep group handles continuation free from offset interference, while the shallow group retains a strong offset for lookup. In contrast, a single shared layer (G1) performs continuation across all calls, so joint training suppresses the offset to protect continuation. Confirming this, freezing all weights and adding a fixed offset at the target start boosts G1 by +25.3 pp, compared to only +8.7 pp for G2 (Table R2). The bottleneck therefore stems from the conflicting functional demands imposed on a single layer during joint training, which a second group decouples. In the revision, we incorporate these intervention findings as empirical motivation for Premise H (the interference between lookup and continuation during joint training).

Third, causal tests at 1.3B show that exact retrieval is primarily executed by full attention. Masking the needle in full-attention layers drops retrieval accuracy to 0, whereas masking it in Gated DeltaNet layers reduces mean accuracy by only 1.4 pp (G2) and 4.9 pp (Baseline) relative to controls. Furthermore, a few Gated DeltaNet heads supply the prerequisite for continuation: they write the preceding token into the key at each position, and full-attention continuation matches against this feature, mirroring how induction heads leverage previous-token heads (Olsson et al., 2022). Zeroing this channel lowers UUID from 94.3% to 0.3%. With this preceding-token feature written into the keys, every continuation read matches against the same key feature and differs only in its query, enabling continuation reads across depths to share a single layer.

Relation to YOCO and other cache sharing. YOCO, YOCO-U, Zamba, and Hymba propose specific layer- or cache-sharing topologies for inference efficiency. In contrast, we characterize the sharing limits and circuit mechanism for exact retrieval, showing that merging distinct read types into a single cache or layer substantially degrades retrieval. On the same backbone, a YOCO-style single cache (one KV cache read by otherwise independent layers) reaches only 39.00% UUID against 79.00% for G2 (Table R1). Prior evaluations of YOCO and YOCO-U do not reveal this degradation because they omit multi-token UUID tasks and multi-cache variants, focusing instead on single numeric needles where all architectures we test approach 100%. We find that the architectural gap emerges primarily on tasks like UUID that depend heavily on sustained continuation.

Revision plan. Based on the reviewers' feedback, we will tighten the exposition in §2, note the query-offset findings as empirical motivation for Premise H, summarize the requested ablations and causal tests in the main text with full details in the appendix, and clarify the common-parent merge.

Revision checklist:

  1. Tighten the exposition in §2 and note practical design considerations for periodic hybrids (§2, §3).
  2. Summarize the lookup-offset interventions, noting them as empirical motivation for Premise H (§2.3, §4.3).
  3. Clarify auto-regressive retrieval progress and summarize the continuation-key causal tests and selective perturbations (§2.2, §4.1).
  4. Include the five new 0.5B ablation entries in the main ablation table and clarify the common-parent merge procedure (§4.2).
  5. Note single-pass composition as an open question alongside benchmark design considerations in the limitations (§2.3, §6).
  6. Add citations and brief discussions of YOCO-U, DeepSeek-V4.1-Flash, induction heads, and retrieval heads (§1).
  7. Place the full tables and protocols for the bidirectional needle ablation, 128K memory scaling, 32K continued training, and merged-G1 fine-tuning in the appendix (supporting §4 and §5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.