acceptodds
Under review as a conference paper at ICLR 2027

Forgetting to Forget: How Rounding Bias Alters Memory in Gated DeltaNet

Abstract

Repeatedly rounding a language model's recurrent state can change what it remembers during decoding. Lower-precision caches save memory, but may discard small updates each time the state is stored. We prove that round-to-nearest (RTN) permanently halts fixed-gate pure decay below an exact format-dependent threshold. The measured active Gated DeltaNet heads do not freeze: writes and erasures keep them moving, while rounding error accumulates across the full update. Using Qwen3.5-0.8B as our main example, we connect this arithmetic to state error and next-token prediction. In teacher-forced evaluation on 72 documents, storing selected heads in fp32 removes 75.4% of the excess negative log-likelihood caused by bf16 round-to-nearest relative to an fp32-state control. In a separate pooled comparison, full-update stochastic rounding removes 91.4% of the RTN excess without enlarging the bf16 cache. Across five architectures, a post-hoc gate diagnostic flags at-risk bf16 heads; separate replays on fixed inputs show substantial state errors in selected small-decay heads. Free-running Qwen3.5-4B also shows changes in generation length under a fixed decoding protocol. The results motivate testing recurrent-state precision over long decoding trajectories, not only long-input prefill. They do not establish equal serving cost at equal

Primary area:
foundation or frontier models, including LLMs
Keywords:
Gated DeltaNet, recurrent language models, low-precision inference, rounding error, stochastic rounding, recurrent-state caches, long-context modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AsaNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how repeated low-precision storage of recurrent states during decoding affects memory in hybrid models using Gated DeltaNet. The authors establish an exact threshold for permanent stalling in the idealized pure decay setting and find that accumulated rounding errors can both degrade predictions and change generation. Stochastic rounding mitigates these effects, while the authors explicitly note that decay stalling has not been established as the cause of the observed output changes.

Critical strengths
  1. The authors provide a clear theoretical explanation of decay stalling under the stated assumptions.

  2. The selective precision interventions provide strong evidence that recurrent-state storage errors affect prediction quality.

Critical weaknesses
  1. The selective-head likelihood interventions are validated in a single model (Qwen3.5-0.8B). However, it is unclear whether these interventions have similar effects on prediction quality in larger models or other model families.

  2. Since the stalling theorem assumes fixed gates and pure decay, it does not directly describe active heads, where writes and erasures keep the state moving.

Questions for authors
  1. Could the authors provide a small output-level check of the selective-head intervention on a larger model?

  2. Could the authors clarify which conclusions from the pure-decay theorem still hold once writes and erasures are present?

Reviewer_o83aOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what happens when hybrid language models round the recurrent state of their Gated DeltaNet and Mamba-2 layers to low precision after every decoding step. It proves that a pure-decay recurrence under round-to-nearest locks permanently above half its starting value whenever the per-step decay is below u/(1+2u). It also shows this threshold is sharp under ties-to-even (1/258 for bf16, 1/2050 for fp16). A gate census over 7,920 heads in five open-weight hybrids uses a post-hoc rule, combining a stall fraction with an isotropic-key error horizon, to flag 2.2% to 16.1% of heads per model as at risk in bf16. Fixed-input replays then show larger state error under RTN than under SR for these heads.

In teacher-forced decoding of Qwen3.5-0.8B on 72 long documents, bf16-RTN adds about 0.023 nats/token of final-quarter NLL over an fp32-state control, compared with about 0.002 for SR. Storing only the gate-selected heads in bf16-RTN reproduces 68.7% of that damage, and keeping just those heads in fp32 removes 75.4%. An attribution analysis shows that real heads do not actually freeze (median realized-to-requested decay of 96%), and that correcting only the decay term gives an inconclusive benefit. Finally, in free-running generation on the 30 AIME 2026 problems, Qwen3.5-4B with bf16-RTN state hits the 32K-token cap more often than with fp32 and solves 8 problems instead of 20.

Critical strengths
  1. The selective storage interventions are the most convincing part of the paper. Storing only the gate-selected heads in bf16-RTN reproduces 68.7% of the full excess NLL, while layer-matched random heads reproduce about 3%, a paired ratio of 22.5 with a 99% interval of [14.1, 44.3]. The census also finds flagged heads in all five hybrids from 0.8B to 9B. The store-period estimate gives 49 flagged heads with per-token storage, 4 with 64-token chunks and none at 256, which is a concrete reason to test recurrent-state precision on long decodes and not only on long prefills.

  2. Theorem 1 is correct as stated, and the sharpness argument handles the cases that are easy to get wrong. These are the midpoint tie at 1+2u under ties-to-even, power-of-two starts that drop into the lower binade, and the smallest normal. I checked the constants (1/258, 1/2050, 1/18 and 1/10) and they follow directly from u = 2^(-p). The paper is also unusually honest about failed preregistered predictions, the post-hoc selection rule and the decoder change, which makes it easy to judge how much weight each number can carry.

Critical weaknesses
  1. The paper is framed around decay stalling, but its own evidence never shows that stalling drives the issue in real heads. Active heads realize a median 96% of requested decay, and the 99% interval for the decay-only correction’s error ratio to RTN is [0.527, 1.233]. In the original replay, the horizon criterion alone catches all 37 damaged streams while the stall criterion catches 33. The data fit the more generic and already known story that deterministic RTN bias piles up in long-memory states and SR removes it (Croci and Giles on stagnation, the Nemotron 3 Super report for recurrent caches), and the exact threshold does little work because 1/258 and 1/256 are indistinguishable in bf16 and the float8 formats where δ⋆ really differs from u are never tested at the model level.

  2. The output-level evidence is narrower than what the paper suggests, and the most obvious equal-size baseline is missing. The selection rule was fixed after inspecting the same Qwen replay, the masks use the legacy cutoff u and are chosen per document, the random controls are not matched on error horizon, and the likelihood numbers come from three separate runs with different reference values. There is no fp16-RTN arm in the likelihood or generation experiments, even though the census flags far fewer fp16 heads (9 compared with 46 in Qwen3.5-0.8B), so it is unclear whether SR is needed at all or whether a 16-bit format with more significand bits would suffice. The generation result rests on one model, 30 problems and a presence penalty adopted after registration, and the three SR seeds alone span 13 to 19 correct answers.

Questions for authors
  1. Table 3 reports exactly 46 flagged heads out of 288 for both Qwen3.5-0.8B and Qwen3.5-2B. Is that a coincidence, or do the two checkpoints share their linear-attention gate parameters? Please confirm the logs come from distinct models.

  2. Can you add an fp16-RTN arm, and ideally a per-head scaled 16-bit format, to the teacher-forced and AIME comparisons? These are the natural equal-size baselines for SR that need no random numbers. The census suggests fp16 alone might remove much of the problem if the state range allows it.

  3. How much of the 68.7% and 75.4% survives with a single static mask computed with δ⋆ on held-out calibration text, rather than 72 per-document masks built with u? A static mask is what a deployment would actually use.

  4. Could you run a corruption control matched on error horizon? One option is heads with τ_err above 10^3 but stall fraction below 0.99, another is the top heads ranked by horizon alone. Without such a control, the intervention cannot separate sub-threshold decay from long memory in general.

  5. Which serving stacks actually store GDN states in bf16 with round-to-nearest during decode? How large is the recurrent state relative to the attention KV cache for Qwen3.5 at 32K tokens? The paper notes that the default recurrent state in your generation setup is fp32, so quantifying the memory or bandwidth at stake would make the motivation much more concrete.

Reviewer_RpAjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper shows how storing the recurrent state of hybrid language models in low precision affects long context decoding. It also proves an exact, format-dependent threshold below which RTN permanently stops pure decay and proposes a gate-based diagnostic for flagging at-risk heads.

Critical strengths

S1. The theorem about permanent decay stall looks exact and depends on the number format and is proved clearly.

S2: The proposed gate-based diagnostic is cheap, training-free, and is applied to multiple heads across different architectures.

S3. The experiments are comprehensive. They include open-loop replays against fp64, teacher-forced head-level interventions, and paired document-level bootstrap intervals.

S4. The authors openly report the pre-registered predictions that failed. They also show a practical message that recurrent-state precision should be evaluated over long decoding, not only on long-input prefill.

Critical weaknesses

W1. My major concern is around novelty of this work. It is already very well known that RTN loses small updates in a recurrent state, and stochastic rounding fixes it [1]. Nemotron 3 Super [2] reported the same effect on its Mamba cache, and DAMP [3] also uses decay rates to decide which channels to keep in higher precision. Other concurrent work and more recent works [4, 5, 6] also covers similar ideas. The paper should discuss these and state clearly what it adds.

W2. Weak link between the theorem and the results. The paper is framed around decay stalling, but its own measurements show that real heads still realize most of their requested decay, and correcting only the decay term gives no clear improvement. The results in the paper shows that bf16 round-to-nearest builds up biased error and stochastic rounding reduces it, but it does not show that stalling is the cause, so either the framing should be narrowed or direct evidence for the decay mechanism should be added.

W3. Head selection and controls. The head-selection rule was chosen after looking at the same data it is evaluated on, and the random heads it is compared against are not matched on how long they retain errors. As a result, the intervention results cannot tell whether the damage comes from stalling or simply from these being long-memory heads.

W4. Limited scope and missing baselines. The likelihood results come from a single small model and the generation results from one model on 30 problems, and seed-to-seed variance is high in the generation test. fp16 is the natural same-size alternative and is never evaluated at the model level, and the cost of stochastic rounding (latency, throughput) is not measured.


References

  1. Croci, Matteo, and Michael B. Giles. "Effects of round-to-nearest and stochastic rounding in the numerical solution of the heat equation in low precision." IMA Journal of Numerical Analysis 43.3 (2023): 1358-1390.

  2. Chandiramani, Aakshita, et al. "Nemotron 3 super: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning." arXiv preprint arXiv:2604.12374 (2026).

  3. Zhang, Tao, et al. “DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization.” arXiv:2608.27513 (2026).

  4. Yao, Bingchen, et al. “STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization.” arXiv:2609.38169 (2026).

  5. Pan, Yi, et al. “LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization.” arXiv:2609.38166 (2026).

  6. Khilar, Snigdha Chandan. “Low-Discrepancy Dither for Quantized Recurrent State Caches.” arXiv:2609.39185 (2026).

Questions for authors

Please see the weaknesses. In particular:

(a) What does the paper add beyond Nemotron 3 Super and DAMP?

(b) Is there evidence that the decay channel specifically, rather than RTN bias in general, drives the NLL loss?

(c) Do the selected-head interventions still hold against horizon-matched random controls, and with the exact delta*?

(d) How does a same-size fp16 state compare to bf16-RTN and bf16-SR at the model level?

(e) LeapQuant reports that SR reduces bf16 state error only about 3x at 64k tokens on Qwen3.5-9B. How does that fit with your 91.4% NLL reduction?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.