- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2(+1)
- Soundness
- 1
Read review
Paper summary
This paper modified Qwen architecture: sharing attention layers groupwise and claim a better retrieval performance. They also provided a analysis framework for that.
Critical strengths
They find that less unique attention layers may lead to better retrieval. This is not completely new idea, but they did test it on qwen 3.5 and the retrieval improved.
Critical weaknesses
qwen 3.5 is a hybrid architecture. The GDN layers also have context memory and also design for associative memory actually. Though GDN has fixed size memory, it is still not sure its impact on retrieval. So the conclusion on attention is not isolated.
The ppl and downstream tasks degrade. This may be a more severe problem than the improvement in retrieval.
Questions for authors
How can you isolate your study on attention only? Maybe test on a attention-only architecture.
There are plenty of works on KV reuse, like the YOCO[https://www.alphaxiv.org/pdf/2405.05254], deepseek v4.1 flash[https://arxiv.org/abs/2609.19969]. Their empirical results are strong enough that models with KV reuse in attention can do good retrieval. What's the contribution of this work?
7 comments
Dear Reviewer Pvas(2/2)
2. Perplexity and downstream tasks (W2). Downstream general tasks show no significant average decline. On the seven zero-shot benchmarks in Table 2 (1.3B, lm-evaluation-harness), Group Share trails Baseline by only 0.08 pp on average (46.97% vs. 47.05%), with all individual gaps within 0.68 pp and gains on PIQA, ARC-Easy, and LAMBADA:
| Zero-shot accuracy (%) | HellaSwag | PIQA | WinoGrande | ARC-Easy | ARC-Challenge | LAMBADA | BoolQ | Mean |
|---|---|---|---|---|---|---|---|---|
| Baseline (1.308B) | 35.94 | 70.02 | 53.43 | 48.86 | 21.25 | 41.67 | 58.17 | 47.05 |
| Group Share (1.172B) | 35.36 | 70.24 | 52.88 | 49.54 | 20.90 | 41.80 | 58.04 | 46.97 |
| Difference (pp) | -0.58 | +0.22 | -0.55 | +0.68 | -0.35 | +0.13 | -0.13 | -0.08 |
At 1.3B, Group Share reduces parameters by 10.4% and full-attention cache by 71.4%, incurring a 2.09% perplexity increase while improving six-task retrieval by 5.00 pp. Given discussion-period compute limits, we ran 0.5B ablations in place of retraining 1.3B models: two-cache variants with independent FFNs achieve lower PPL than Baseline (3.3135 vs. 3.3168), whereas sharing FFNs matches G2 (3.3252 vs. 3.3258) [new] (Table R1). Moreover, 1.3B training curves show that the PPL gap does not widen over training. Together, the 0.5B ablations trace the perplexity increase to FFN parameter reduction, while our main 1.3B model demonstrates a favorable trade-off across perplexity, parameter count, cache savings, and downstream performance.
Excerpt from Table R1. 0.5B models: caches, sharing, loss, and retrieval.
| Model | Params (M) | Loss |
|---|---|---|
| Baseline | 501.66 | 3.3168 |
| Two caches, K/V [new] | 500.09 | 3.3121 |
| Two caches, K/V + attention [new] | 493.01 | 3.3135 |
| Two caches, K/V + FFN [new] | 468.24 | 3.3252 |
| G2 | 461.15 | 3.3258 |
3. Contribution relative to YOCO and DeepSeek-V4.1-Flash (Q2). YOCO generates one global KV cache via a self-decoder for independent cross-decoder layers, while DeepSeek-V4.1-Flash (released September 17, 2026) assigns fixed layers to reuse earlier KV states or sparse indices. Both propose specific reuse topologies for inference efficiency. In contrast, we characterize the sharing limits and circuit mechanism for exact retrieval, showing that merging distinct read types into a single cache or layer substantially degrades retrieval. On the same backbone, a YOCO-style single cache (KV-only) reaches only 39.00% UUID versus 79.00% for G2 (Table R1). Our query-offset interventions explain why one jointly trained layer fails to serve both lookup and continuation (General Response 1/2) [new]. These findings establish concrete design rules for periodic hybrids on group count, sharing scope, cache-writing depth, and K/V tying. Unlike YOCO, Group Share preserves standard hybrid topologies without altering layer layouts. Moreover, DeepSeek-V4.1-Flash's reuse of KV with recomputed queries aligns with our finding that homogeneous reads can share keys while specializing in queries. We will discuss and cite this work in the revision.
Excerpt from Table R1. 0.5B models: caches, sharing, loss, and retrieval. UUID: exact match (%) on all 300 test items.
| Model | Caches | Params (M) | UUID |
|---|---|---|---|
| Baseline | 6 | 501.66 | 79.67 |
| KV-only | 1 | 499.70 | 39.00 |
| Two caches, K/V [new] | 2 | 500.09 | 62.67 |
| G2 | 2 | 461.15 | 79.00 |
4. Presentation. We have condensed §2 and restated our contributions around group count determination, the query-offset mechanism, and design rules (General Response 1/2).
Thank you for the comprehensive response and new ablations. Your analysis tracing the PPL gap to FFN sharing and clarifying the negligible zero-shot drop (-0.08 pp) resolves my quality concerns. The new mechanistic interventions are also valuable, so I will raise my score.
Before deciding whether to raise it further, two core concerns remain from your new results:
Two-cache lower bound (Table R1): Adding query heads to a single-cache model (More query heads, 1 cache) reaches 76.33% UUID (close to G2's 79.00%) and outperforms G2 on the 6-task average (58.90% vs. 57.76%) with lower loss and fewer parameters. Doesn't this suggest G1's bottleneck is query-head capacity rather than a fundamental need for two caches?
GDN coupling & masking drop (Table R3 & Item 1): Zeroing GDN's lag-1 channel drops UUID to 0.3%, meaning two-group sufficiency relies on GDN providing "free" previous-token features (which pure-attention models would need an extra attention group to compute). Also, in Item 1, masking GDN drops UUID from 94.3% to 76.0% (and Control to 78.3%)—why does the control itself degrade performance so sharply?
Looking forward to your clarification.
Dear Reviewer Pvas(1/2)
Thank you for the review. New experiments show that exact retrieval in our models is primarily executed by full attention, while Gated DeltaNet plays a limited direct memory role (item 1). We also clarify that downstream general tasks show no significant average decline (mean drop of 0.08 pp), and the perplexity gap stems mainly from FFN parameter reduction (item 2). Relative to YOCO and DeepSeek-V4.1-Flash, our contribution establishes how many independent layers exact retrieval requires and uncovers the underlying mechanism (item 3). We have revised the presentation to highlight these contributions clearly (item 4).
1. Separating attention and Gated DeltaNet memory contributions (W1, Q1) [new]. Bidirectional ablations show that exact retrieval is primarily driven by full attention, whereas Gated DeltaNet contributes modestly as a direct memory store. At 1.3B, we masked the needle in one pathway while keeping the other intact. When masked in full attention, retrieval accuracy drops to 0 across all tasks. Conversely, removing the target tokens from the recurrent state of all 21 Gated DeltaNet layers causes only modest drops:
When full attention cannot read the needle, all tasks drop to 0 in both models. When Gated DeltaNet excludes the needle, the mean drops by only 1.4 pp (G2) and 4.9 pp (Baseline) relative to controls. However, Gated DeltaNet plays a specific indirect role: a few heads write preceding-token features into continuation keys, and zeroing this channel drops UUID from 94.3% to 0.3% [new]. Prior work on induction circuits shows that previous-token heads supply this preceding-token feature in pure-attention models (Olsson et al., 2022). We therefore expect a similar read mechanism to hold in pure Transformers. Due to rebuttal time and compute constraints, direct verification on pure-attention backbones remains for future work.