- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies how repeated low-precision storage of recurrent states during decoding affects memory in hybrid models using Gated DeltaNet. The authors establish an exact threshold for permanent stalling in the idealized pure decay setting and find that accumulated rounding errors can both degrade predictions and change generation. Stochastic rounding mitigates these effects, while the authors explicitly note that decay stalling has not been established as the cause of the observed output changes.
Critical strengths
-
The authors provide a clear theoretical explanation of decay stalling under the stated assumptions.
-
The selective precision interventions provide strong evidence that recurrent-state storage errors affect prediction quality.
Critical weaknesses
-
The selective-head likelihood interventions are validated in a single model (Qwen3.5-0.8B). However, it is unclear whether these interventions have similar effects on prediction quality in larger models or other model families.
-
Since the stalling theorem assumes fixed gates and pure decay, it does not directly describe active heads, where writes and erasures keep the state moving.
Questions for authors
-
Could the authors provide a small output-level check of the selective-head intervention on a larger model?
-
Could the authors clarify which conclusions from the pure-decay theorem still hold once writes and erasures are present?