Forgetting to Forget: How Rounding Bias Alters Memory in Gated DeltaNet

Repeatedly rounding a language model's recurrent state can change what it remembers during decoding. Lower-precision caches save memory, but may discard small updates each time the state is stored. We prove that round-to-nearest (RTN) permanently halts fixed-gate pure decay below an exact format-dependent threshold. The measured active Gated DeltaNet heads do not freeze: writes and erasures keep them moving, while rounding error accumulates across the full update. Using Qwen3.5-0.8B as our main example, we connect this arithmetic to state error and next-token prediction. In teacher-forced evaluation on 72 documents, storing selected heads in fp32 removes 75.4% of the excess negative log-likelihood caused by bf16 round-to-nearest relative to an fp32-state control. In a separate pooled comparison, full-update stochastic rounding removes 91.4% of the RTN excess without enlarging the bf16 cache. Across five architectures, a post-hoc gate diagnostic flags at-risk bf16 heads; separate replays on fixed inputs show substantial state errors in selected small-decay heads. Free-running Qwen3.5-4B also shows changes in generation length under a fixed decoding protocol. The results motivate testing recurrent-state precision over long decoding trajectories, not only long-input prefill. They do not establish equal serving cost at equal