acceptodds
Under review as a conference paper at ICLR 2027

Sharing Memory Addressing Across Depth in Hybrid Gated Delta Networks

Abstract

Does every recurrent memory layer in a language model need its own learned addressing rule? We investigate this question by selectively sharing parameters across the depth of a hybrid Gated DeltaNet language model. Our primary intervention ties query and key projections and their short convolutions across recurrent layers, while preserving independent memory states, value and output transformations, feed-forward networks, and normalization parameters. A value-only sharing control removes exactly the same number of parameters, separating component choice from the amount of parameter reduction. In a 400M-class model trained on FineWeb-Edu, query/key sharing removes 4.40% of unique parameters and incurs a smaller held-out language-modeling penalty than parameter-matched value sharing at a common 10.22B-token checkpoint. Sharing additional update-control parameters has a similarly small penalty, whereas whole-block looping changes the quality–parameter trade-off substantially. However, QK-only tying reduces 4K single-needle retrieval from 80.2% to 37.6%, despite a 0.24% relative held-out perplexity increase; sharing update controls reaches 74.8%. These observations motivate a storage–semantic retrieval separation hypothesis: reusable memory addressing may coexist with depth-specific semantic selection and content transformations. We present this as a mechanistic interpretation to test, not an established functional decomposition. It requires neither identical address semantics across depth nor reduced expanded-layer computation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
linear attention, recurrent memory, associative memory, memory allocation, semantic addressing, routing, fast weights, Gated DeltaNet

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_U31xOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the trade-off between parameter sharing and model quality in hybrid Gated DeltaNet language models by selectively sharing QK projections, update controls, value projections, Full Attention parameters, and entire blocks across depth. Per-layer recurrent memory states remain independent, and the QK-only and V-only configurations are designed to remove exactly the same number of unique parameters, enabling a parameter-matched comparison of address-side and value-side sharing.

In a 400M-class model trained for 10.22B tokens, sharing the GDN QK pathway reduces unique parameters by 4.40% with only a small increase in held-out FWE perplexity, while parameter-matched V sharing incurs a larger perplexity penalty. Extending QK sharing to update controls yields a similar parameter-quality trade-off, whereas more aggressive whole-block sharing achieves substantially larger parameter savings at a greater performance cost. Importantly, QK-only sharing reduces the 4K single-needle retrieval score from 80.2% to 37.6% despite near-preserved perplexity, showing that preserving aggregate language-modeling quality does not necessarily preserve retrieval behavior.

Critical strengths
  1. The control group for comparing the impact of shared parameters is well designed. N1 and N3 each save exactly 17,740,800 parameters by sharing only the QK path and the V path, respectively. Since they preserve the execution depth, width, and independent per-layer states, the difference between the two settings cannot be explained solely by differences in compression rates. This allows for a more fine-grained investigation of the effects of component-level sharing compared to reusing entire blocks, making it the most compelling experimental design in this paper.

  2. It specifically demonstrates the retrieval loss masked by nearly identical PPLs. For N1, while the relative increase in FWE PPL is approximately 0.24% and the zero-shot average decreases slightly from 46.95% to 46.61%, the 4K single-needle score drops from 80.2% to 37.6%. This provides a concrete example showing that near-preservation of average predictive quality does not imply preservation of retrieval behavior. Reporting this adverse result adds empirical value beyond demonstrating parameter savings.

Critical weaknesses

(1) The robustness of the key observations is not sufficiently established, and the retrieval results remain unexplained.

The central evidence is based on a single 400M-class model, a single GDN-to-Full-Attention ratio, and a single training seed. This makes it difficult to determine whether the parameter-matched N1/N3 PPL difference is robust across independent training runs. More importantly, N1 and N2 show a large and unexplained retrieval gap. QK-only sharing reduces the 4K single-needle score from 80.2% to 37.6%, despite only a 0.24% relative increase in FWE PPL. In contrast, N2, which additionally shares update-control parameters, achieves 74.8%. The current experiments do not explain why the stronger sharing constraint substantially improves retrieval or whether this result persists across seeds. The multi-key results are near the floor even for the baseline and therefore provide little additional evidence. Repeated runs and further analysis of this N1–N2 gap are needed to establish the reliability and cause of this key observation.

(2) The mechanistic interpretation and resulting novelty are not sufficiently supported by the current experiments.

The N1/N3 comparison provides useful evidence that GDN QK parameters are more tolerant to cross-depth sharing than parameter-matched V parameters in this setting. However, it does not establish that this difference arises from a reusable storage-addressing mechanism. Differences in optimization sensitivity, regularization from shared parameters, or compensation by untied FFNs, Full-Attention layers, and GDN output pathways remain plausible explanations.

This is important because parameter sharing and QK sharing are not themselves new, so much of the paper's contribution depends on the mechanistic insight obtained from selective sharing. Matched-budget controls such as Full-Attention-QK or FFN sharing would help determine whether the observed tolerance is specific to GDN memory addressing or reflects more general component-dependent sensitivity to parameter tying. Without such controls, the experiments show which components tolerate sharing better in this setting, but provide limited evidence for why or whether this supports the proposed storage-addressing/semantic-retrieval interpretation.

Finally, if parameter/storage efficiency is intended as a practical motivation, reporting actual training or inference memory savings would help clarify the significance of the 4.40% reduction in unique parameters.

Questions for authors

(1) Are the key results reproducible across independent training runs?

Do additional runs with different seeds preserve the main N0/N1/N2/N3 trends under the same training budget and data order? In particular, are the N1-vs.-N3 PPL ordering and the large 4K single-needle gap between N1 and N2 consistent across runs? Confidence intervals over evaluation examples alone do not capture training variability, so repeated training runs would be important for establishing the robustness of these observations.

(2) What explains the discrepancy between PPL and retrieval, and especially the large N1–N2 retrieval gap?

N1 increases FWE PPL by only about 0.24%, yet its 4K single-needle score drops from 80.2% to 37.6%. Do the authors have analyses explaining why QK sharing has little effect on average predictive quality but a much larger effect on retrieval?

Moreover, N2 additionally shares update-control parameters but improves the same retrieval score to 74.8%. Why does the stronger sharing constraint substantially improve retrieval? Analyses of memory retention, update dynamics, or training behavior would help clarify this result.

(3) Can the authors provide additional matched-budget controls to support the storage-addressing interpretation?

For example, would Full-Attention-QK sharing or FFN sharing with a comparable parameter reduction produce similar or different effects? Such controls would help determine whether the observed tolerance of GDN QK sharing is specific to memory addressing or reflects a more general difference in component sensitivity to parameter tying.

(4) What is the practical memory benefit of the reported parameter savings?

If parameter/storage efficiency is intended as a practical motivation, can the authors report actual training or inference memory usage for N0, N1, and N2 under matched hardware, batch size, and context length? This would clarify how the 4.40–4.43% reduction in unique parameters translates into end-to-end memory savings.

Reviewer_RkiYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies which components of a hybrid Gated DeltaNet language model can share parameters across depth. Its principal intervention ties GDN query/key projections and short convolutions while preserving independent recurrent states and untied content/output pathways. An exactly parameter-matched value-sharing control and stronger sharing variants are evaluated on a 400M-class model trained for 10.224B tokens. QK sharing incurs a smaller FineWeb-Edu perplexity penalty than value sharing, but substantially degrades 4K single-needle retrieval; additionally sharing update controls largely recovers that retrieval score.

Critical strengths

1 and N3 each remove exactly 17,740,800 parameters, controlling reduction amount while changing the shared component. All six variants retain the same expanded architecture and are evaluated at a common token budget on the same 16,777,216-token held-out window. The additional downstream and retrieval evaluations reveal a limitation that perplexity alone would miss: N1 changes FWE PPL from 10.974 to 11.000, yet reduces 4K single-needle accuracy from 80.2% to 37.6%. Reporting this failure, rather than treating near-equal perplexity as functional preservation, makes the study informative.

Critical weaknesses

W1. The central component-level finding has not yet been established reproducibly.

The principal evidence comes from one training seed, one 400M backbone, and one GDN:Full ratio. Identical example ordering across variants is not established, and N0/N3's final 500 updates were replayed from 19,000-step checkpoints with a changed physical execution layout (Section 4 and Appendix C). These caveats are disclosed, and replay does not itself demonstrate a biased result. However, they make independent, matched-order replication particularly important for interpreting relatively fine differences between interventions.

N1's FWE penalty over N0 is 0.026 PPL, while its advantage over parameter-matched N3 is 0.142 PPL; its eight-task average exceeds N3 by 0.61 percentage points. Appendix D supports persistent N1/N3 ordering within the available historical trajectory, so the result is not merely an isolated endpoint fluctuation. Nevertheless, neither that repeatedly monitored validation window nor the larger final holdout estimates training-run variability. The 19,500-step comparison is also intermediate within the original 15B-token schedule. Replication of the central comparisons and a later-checkpoint or matched-cooldown check would determine whether the ordering is a stable architectural finding.

W2. The current interventions do not yet provide a sufficiently discriminating explanation or behavioral design conclusion.

The evidence for greater tolerance of QK sharing is metric-dependent. N1 is better than N3 on FWE PPL (11.000 versus 11.142), but their WikiText results are effectively tied, with N3 slightly lower (27.055 versus 27.069), and N1 is substantially worse on 4K single-needle retrieval (37.6% versus 70.2%). N2 shares only 138,375 additional update-control parameters yet reaches 74.8% retrieval. This is an interesting interaction, but its cause remains unresolved.

The parameter-count match excludes unequal reduction amount; it does not separate reusable addressing from gradient aggregation, optimization sensitivity, or compensation by the six untied Full-attention layers, FFNs, and GDN output paths. There is no completed update-control-only control or matched-budget Full-QK/FFN comparison, and no pathway-specific retrieval intervention. The near-floor multi-key results cannot resolve these alternatives.

The authors correctly present the functional interpretation as tentative. However, once these qualifications are respected, the contribution remains a preliminary observation about sharing tolerance in one recipe, rather than a robust explanation of addressing/content specialization or a broadly supported practical sharing rule. The large N1/N2 retrieval difference is a particularly important phenomenon to replicate and isolate.

Questions for authors
  1. Replication and training budget: Do the N1/N3 language-modeling ordering and the N1/N2 retrieval gap persist across independent seeds with matched example order? Does the ordering remain at a later checkpoint or after a common cooldown?

  2. Control-sharing interaction: What happens when update controls alone are shared? Can this control and a targeted pathway intervention distinguish N2's retrieval recovery from optimization variation or compensation by untied modules?

  3. Optimization under tying: Given the reported common learning-rate recipe and global-norm clipping threshold, are gradients for a shared tensor summed or normalized across depth before clipping? How are shared tensors initialized relative to the untied models? Would a gradient-scaling control alter the QK-versus-V comparison?

  4. Behavioral interpretation: Can a matched-budget Full-QK or FFN control provide evidence that discriminates the proposed storage–semantic separation hypothesis from generic differences in component sensitivity to tying?

Reviewer_vNHWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines which parameters of a hybrid Gated DeltaNet language model can be shared across its recurrent layers while each layer still keeps its own independent state. The central comparison contrasts two options: tying the query/key projections together with the short convolutions, versus tying a value pathway with an exactly matched parameter count. For a 400M-parameter model compared at the same 10.22B-token checkpoint, QK sharing reduces the parameter count by 4.40% while incurring a smaller perplexity penalty on FineWeb-Edu than value sharing. However, single-needle retrieval at 4K context drops sharply under QK-only sharing. The authors take these results as motivation for a hypothesis contrasting storage addressing with semantic selection, but explicitly leave that interpretation unproven. Some settings can reduce lots of parameters, but their model performance decreases a lot as well.

Critical strengths

S1. The comparison between QK and V sharing is carefully controlled with respect to parameter count. As Equations (5)–(6) and Table 1 demonstrate, each of the two interventions removes precisely 17,740,800 parameters while leaving the states separate. Such a matched design is more informative than comparing variants at unrelated compression ratios.

S2. The evaluation reveals a substantive behavioral limitation: under N1, FineWeb-Edu perplexity shifts from 10.974 to 11.000, whereas single-needle recall at 4K falls from 80.2% to 37.6% (Tables 2 and 4). By reporting this mismatch, the paper is considerably stronger than a compression study that relies on perplexity alone.

S3. The writing draws careful distinctions between sharing weights and sharing activations or states, and between saving parameters and reducing computation. The discussion and appendices are also transparent about the single training seed, the replayed checkpoints, the intermediate learning-rate schedule, and the multikey results that sit near the floor.

Critical weaknesses

W1. The paper’s central component-specific claim rests on limited replication. The N1-versus-N3 comparison is backed by a single trained seed on a single hybrid architecture, and it has not been established that the variants saw identical data ordering; moreover, N0 and N3 rely on replayed checkpoints (Section 4; Appendix C). The 0.142 PPL gap between N1 and N3 is suggestive, and Figure 3 offers corroborating trajectory evidence, but this does not establish a reproducible architectural preference. Likewise, the large gap in retrieval between N1 and N2 must be separated from training variability before it can be attributed to update-control sharing.

W2. The functional interpretation is considerably less well supported than the intervention result itself. Half of the layers employ global full attention, and the untied FFNs and output paths could compensate for the tying. The orthogonal covariance identity in Equation (7) holds, but it does not pin down where semantic selection takes place. Section 6 concedes these alternative explanations; as a result, what the paper currently contributes is a narrow empirical observation rather than evidence of a functional decomposition.

W3. The practical payoff depends on the behavior being retained. N1 reduces weights by 4.40% yet gives up 42.6 percentage points on the informative 4K retrieval probe. N2 largely restores performance on that probe but falls below N0 on the eight-task average, and multikey performance is near the floor for every model (Tables 3–4). These findings do not refute selective sharing, but they leave the useful quality-versus-storage operating point inadequately characterized.

Questions for authors

Q1. Are there any results from independent seeds already on hand for N0/N1/N2/N3 that you could share, along with paired uncertainty estimates derived from the existing 4K retrieval outputs? Please separate variability due to the training seed from uncertainty due to the evaluation samples.

Q2. Were the initial untied parameters, the data order, and the handling of gradients on shared parameters kept consistent across the main variants? Please specify which of these factors were controlled, and whether the N1/N3 ordering holds at comparable neighboring checkpoints.

Q3. Which observation in the present experiments specifically separates the proposed storage/semantic interpretation from compensation or regularization effects? If no such observation exists, please reframe the conclusion around component-specific tolerance to tying and the perplexity/retrieval mismatch that the paper demonstrates.

Q4. Do you have other results with larger models and mode datasets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.