- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the trade-off between parameter sharing and model quality in hybrid Gated DeltaNet language models by selectively sharing QK projections, update controls, value projections, Full Attention parameters, and entire blocks across depth. Per-layer recurrent memory states remain independent, and the QK-only and V-only configurations are designed to remove exactly the same number of unique parameters, enabling a parameter-matched comparison of address-side and value-side sharing.
In a 400M-class model trained for 10.22B tokens, sharing the GDN QK pathway reduces unique parameters by 4.40% with only a small increase in held-out FWE perplexity, while parameter-matched V sharing incurs a larger perplexity penalty. Extending QK sharing to update controls yields a similar parameter-quality trade-off, whereas more aggressive whole-block sharing achieves substantially larger parameter savings at a greater performance cost. Importantly, QK-only sharing reduces the 4K single-needle retrieval score from 80.2% to 37.6% despite near-preserved perplexity, showing that preserving aggregate language-modeling quality does not necessarily preserve retrieval behavior.
Critical strengths
-
The control group for comparing the impact of shared parameters is well designed. N1 and N3 each save exactly 17,740,800 parameters by sharing only the QK path and the V path, respectively. Since they preserve the execution depth, width, and independent per-layer states, the difference between the two settings cannot be explained solely by differences in compression rates. This allows for a more fine-grained investigation of the effects of component-level sharing compared to reusing entire blocks, making it the most compelling experimental design in this paper.
-
It specifically demonstrates the retrieval loss masked by nearly identical PPLs. For N1, while the relative increase in FWE PPL is approximately 0.24% and the zero-shot average decreases slightly from 46.95% to 46.61%, the 4K single-needle score drops from 80.2% to 37.6%. This provides a concrete example showing that near-preservation of average predictive quality does not imply preservation of retrieval behavior. Reporting this adverse result adds empirical value beyond demonstrating parameter savings.
Critical weaknesses
(1) The robustness of the key observations is not sufficiently established, and the retrieval results remain unexplained.
The central evidence is based on a single 400M-class model, a single GDN-to-Full-Attention ratio, and a single training seed. This makes it difficult to determine whether the parameter-matched N1/N3 PPL difference is robust across independent training runs. More importantly, N1 and N2 show a large and unexplained retrieval gap. QK-only sharing reduces the 4K single-needle score from 80.2% to 37.6%, despite only a 0.24% relative increase in FWE PPL. In contrast, N2, which additionally shares update-control parameters, achieves 74.8%. The current experiments do not explain why the stronger sharing constraint substantially improves retrieval or whether this result persists across seeds. The multi-key results are near the floor even for the baseline and therefore provide little additional evidence. Repeated runs and further analysis of this N1–N2 gap are needed to establish the reliability and cause of this key observation.
(2) The mechanistic interpretation and resulting novelty are not sufficiently supported by the current experiments.
The N1/N3 comparison provides useful evidence that GDN QK parameters are more tolerant to cross-depth sharing than parameter-matched V parameters in this setting. However, it does not establish that this difference arises from a reusable storage-addressing mechanism. Differences in optimization sensitivity, regularization from shared parameters, or compensation by untied FFNs, Full-Attention layers, and GDN output pathways remain plausible explanations.
This is important because parameter sharing and QK sharing are not themselves new, so much of the paper's contribution depends on the mechanistic insight obtained from selective sharing. Matched-budget controls such as Full-Attention-QK or FFN sharing would help determine whether the observed tolerance is specific to GDN memory addressing or reflects more general component-dependent sensitivity to parameter tying. Without such controls, the experiments show which components tolerate sharing better in this setting, but provide limited evidence for why or whether this supports the proposed storage-addressing/semantic-retrieval interpretation.
Finally, if parameter/storage efficiency is intended as a practical motivation, reporting actual training or inference memory savings would help clarify the significance of the 4.40% reduction in unique parameters.
Questions for authors
(1) Are the key results reproducible across independent training runs?
Do additional runs with different seeds preserve the main N0/N1/N2/N3 trends under the same training budget and data order? In particular, are the N1-vs.-N3 PPL ordering and the large 4K single-needle gap between N1 and N2 consistent across runs? Confidence intervals over evaluation examples alone do not capture training variability, so repeated training runs would be important for establishing the robustness of these observations.
(2) What explains the discrepancy between PPL and retrieval, and especially the large N1–N2 retrieval gap?
N1 increases FWE PPL by only about 0.24%, yet its 4K single-needle score drops from 80.2% to 37.6%. Do the authors have analyses explaining why QK sharing has little effect on average predictive quality but a much larger effect on retrieval?
Moreover, N2 additionally shares update-control parameters but improves the same retrieval score to 74.8%. Why does the stronger sharing constraint substantially improve retrieval? Analyses of memory retention, update dynamics, or training behavior would help clarify this result.
(3) Can the authors provide additional matched-budget controls to support the storage-addressing interpretation?
For example, would Full-Attention-QK sharing or FFN sharing with a comparable parameter reduction produce similar or different effects? Such controls would help determine whether the observed tolerance of GDN QK sharing is specific to memory addressing or reflects a more general difference in component sensitivity to parameter tying.
(4) What is the practical memory benefit of the reported parameter savings?
If parameter/storage efficiency is intended as a practical motivation, can the authors report actual training or inference memory usage for N0, N1, and N2 under matched hardware, batch size, and context length? This would clarify how the 4.40–4.43% reduction in unique parameters translates into end-to-end memory savings.