acceptodds
Under review as a conference paper at ICLR 2027

Memory by Design: Bayesian Layers for Sequence Modeling

Abstract

To battle softmax attention's linearly growing inference memory cost, current recurrent architectures compress history into a fixed-dimensional state. Their recurrence equations alone, however, leave the underlying assumptions about memory implicit. We introduce design models: auxiliary probabilistic models specified by an initial distribution and input-dependent factors encoding memory dynamics and incoming evidence. Once these assumptions are specified, Bayesian filtering determines the recurrent update under the model. Specifically, a linear-Gaussian design model yields the Bayesian Layer, which maintains a memory estimate and its covariance via a Kalman update. Resetting this covariance recovers Delta-rule updates, while a different design model yields the additive recurrences of linear attention, GLA, and Mamba-2. These connections make explicit the assumptions underlying familiar memory updates. We illustrate how the geometry of the Bayesian Layer's uncertainty shapes subsequent updates and helps preserve earlier associations. In associative recall tasks, the Bayesian Layer improves recall on longer sequences and with greater overlap between keys than seen during training. We further evaluate its performance on multi-query associative recall (MQAR) and language modeling. Distilling Bayesian Layers into a pretrained Gated DeltaNet improves retrieval accuracy on RULER over a continued-training control.

Primary area:
learning on time series and dynamical systems
Keywords:
Recurrent sequence architectures, probabilistic graphical models, time-series modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_339kOpenReview ↗
Rating
2
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces design models, an auxiliary probabilistic framework for deriving recurrent memory updates from explicit assumptions about memory dynamics and incoming evidence. Its linear-Gaussian instantiation, the Bayesian Layer (BL), propagates a memory mean and a dense key-space covariance through Kalman filtering. Covariance resets recover Delta-rule updates, while a separate latent-input model yields additive recurrences. The paper analyzes how covariance geometry reduces interference and evaluates BL on controlled associative recall, MQAR, WikiText-103, and layerwise adaptation of a pretrained 340M Gated DeltaNet. The results demonstrate recall improvements, accompanied by modest increases in overall language-model perplexity.

Critical strengths
  1. The framework distinguishes changes to the underlying probabilistic model from approximations to inference. The derivations connecting BL, covariance-reset Delta rules, and additive updates provide a useful vocabulary for understanding and designing recurrent memory mechanisms.

  2. The confusion analysis explains how accumulated evidence reshapes covariance and redirects writes away from well-supported associations. Experiments with controlled key overlap directly test this mechanism and demonstrate substantial advantages of dense covariance over diagonal and conventional updates under strong interference.

Critical weaknesses
  1. Dense-versus-diagonal comparisons are provided in controlled learned recall experiments, but not on MQAR or WikiText-103. Comparisons with closely related uncertainty-aware models, particularly the recent Kalman Delta Networks, would help establish when full covariance provides benefits beyond cheaper uncertainty approximations.
  2. From-scratch experiments use approximately 14–17M non-embedding parameters, while the 340M experiment replaces only four fixed layers. Both BL variants increase overall WikiText-103 perplexity relative to their corresponding baselines. Moreover, the adaptation comparison matches token exposure per selected mixer but differs in objectives, trainable parameters, and total computation, limiting attribution of the retrieval gains solely to the Bayesian update.
Questions for authors
  1. Can you report training, prefill, and decoding throughput together with peak memory usage against GDN and other strong recurrent baselines, specifying head dimensions, state sizes, sequence lengths, and chunk sizes?

  2. Under a shared backbone and training protocol, how do dense, diagonal, propagated-isotropic, and covariance-reset variants compare on MQAR and WikiText-103? How would these results relate to the uncertainty approximations used by Kalman Delta Networks?

  3. How sensitive are the adaptation results to the number and placement of replaced layers? Could a GDN control trained with the same layerwise distillation objective better isolate the effect of replacing the memory update, while reporting both retrieval gains and perplexity costs?

Reviewer_otaZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces design models: auxiliary probabilistic models whose input-dependent dynamics and evidence factors determine recurrent memory updates through Bayesian filtering. Its main instantiation, the Bayesian Layer (BL), maintains a matrix-valued posterior mean together with a dense covariance shared across value columns. A Kalman update uses this covariance to determine the direction and strength of each write, while a query reads from the posterior mean. Resetting the predicted covariance recovers canonical Delta-rule updates; a separate latent-input model yields additive recurrences.

The paper analyzes how covariance geometry can reduce interference between overlapping keys and provides a chunked implementation. Experiments cover controlled associative recall, random-key recall, MQAR, small WikiText-103 language models, and replacement of four mixers in a pretrained 340M Gated DeltaNet. The results support improved recall in several settings, with additional recurrent state and some deterioration in overall language-model perplexity.

Critical strengths

1. A coherent probabilistic construction with a useful mechanistic explanation.

The paper makes the assumptions behind different write rules explicit and distinguishes exact filtering under different models from covariance-reset approximations. Maintaining dense covariance allows the update direction to depart from the incoming key, and the interference analysis explains when this can protect existing associations. The construction also gives a closed recurrence and an implementable structured-dynamics layer. Although the underlying Kalman identities are classical, the combination of this organization, the covariance geometry, and the architectural implementation is a meaningful contribution.

2. Evidence spans controlled mechanisms and learned representations.

The controlled experiments directly compare dense and diagonal covariance while preventing learned key projections from changing the prescribed overlap. At overlap 0.98 and 64 distractor writes, the reported accuracies are 65.0% and 33.9%, respectively. The random-key experiment extends the investigation beyond the engineered pair structure. MQAR and WikiText-103 then test learned representations, including BL variants paired with GDN and RWKV-7 surrounding blocks. Finally, the adapted model improves mean RULER retrieval from 20.51% to 25.51% at 8k context relative to continued GDN. Reporting three training seeds and explicitly acknowledging the perplexity and task-specific regressions strengthens the evidence.

Critical weaknesses

1. The recall benefit has not been placed on a comparable state and compute budget.

The paper motivates its approach through efficient fixed-size recurrent memory, but the experiments match width and/or training tokens rather than recurrent-state storage or computation. Appendix F explicitly acknowledges that state sizes and parameter counts are not constrained to be equal. For example, in the WikiText setup with key and value dimensions both 128, BL stores a 128×128 mean and a 128×128 covariance per head, compared with the corresponding GDN's 128×128 matrix memory, before other buffers. The pretrained replacement further uses 512 real state coordinates and a 512×512 covariance per head.

Appendix C provides a useful complexity analysis and implementation description, but no measured training throughput, prefill/decode latency, or peak-memory comparison is reported. This matters because the chunk algorithm retains sequential dependencies and adds covariance work. Moreover, the recall improvements come with higher overall perplexity: for example, the RWKV-like BL has WikiText LM perplexity 18.12 versus 17.49 for RWKV-7, and adaptation increases perplexity by approximately 3–4% relative to continued GDN. These trade-offs are acknowledged, but a resource table and a suitably enlarged recurrent baseline are needed to determine the practical value of the extra state and computation. This is a limitation of the efficiency evidence, not a contradiction of constant state size with respect to sequence length.

2. The pretrained-model experiment does not isolate dense covariance as the source of the retrieval improvement.

The continued-GDN control is useful, but Appendix G explicitly states that the objectives, trainable parameters, and total computation differ. BL is fitted layerwise to frozen teacher outputs with normalized MSE and a cosine term, whereas continued GDN optimizes next-token cross-entropy and updates both mixers and MLPs at the selected layers. The replacement also changes the state dimension and uses rotational dynamics without the original short convolution. Matching token exposure per selected mixer therefore does not isolate the Bayesian update from these other changes.

The dense-versus-diagonal control on synthetic recall is valuable, and the paired surrounding blocks in the small language models partially address architectural confounding. However, an otherwise identical diagonal- or reset-covariance BL under the same large-model distillation procedure would more directly test the proposed mechanism in the setting supporting the strongest practical claim. At present, the RULER experiment demonstrates an improvement for the complete adapted system, rather than identifying how much of that improvement requires dense covariance.

Questions for authors

1. Resource trade-off — clarification and additional analysis. Please report recurrent-state storage, parameter counts, training throughput, and prefill/decode cost for representative BL and baseline configurations on the same hardware. Does the recall advantage persist against a GDN or another strong recurrent baseline given a comparable recurrent-state or compute budget? A compact, representative comparison would be sufficient to clarify this issue.

2. Covariance attribution — additional analysis. With the same state dimensions, rotational dynamics, surrounding components, and distillation objective as Appendix G, how do dense, diagonal, and reset covariance compare on retrieval and perplexity? The synthetic ablation already supports the mechanism; the question is whether it remains the main driver of the pretrained-model result.

3. Adaptation control — clarification and additional analysis. Can you separate the effect of the adaptation recipe from the layer replacement, for example by applying a comparable layerwise output-matching objective to a suitably initialized non-Bayesian replacement? Please explain how the different trainable components and data streams affect the interpretation of the current continued-GDN control. A same-recipe control is more informative here than matching token exposure alone.

4. Closest probabilistic alternative — clarification or targeted comparison. Section 5 clearly distinguishes BL from KLA's diagonal covariance and from MesaNet's regression-based readout. Given the particularly close relationship to Kalman Linear Attention, can you provide a direct comparison, or explain which controlled result establishes the incremental value of retaining cross-key covariance relative to that alternative? Palimpsa and MesaNet are already included; I am asking specifically about the closest filtering-based alternative, not for a broad expansion of the baseline list.

2 comments

The discussion period ends on Nov 18. We have responded to the weaknesses and questions in your review: measured cost and state size against GDN, and a comparison of dense BL, covariance-reset BL, and GDN under one distillation objective.

We also ask you to consider the theoretical contribution of the paper and its potential impact. Our design-model framework (the first listed contribution) describes existing recurrent layers as a design model together with an inference rule, so that they can be compared by their assumptions about memory (Props. 2 and 3). One can therefore state an assumption about memory and derive the update in a principled way. Our Bayesian Layer is one instance of this framework, and it lets us analyze how the geometry of the uncertainty governs associative recall (Eq. 15).

We believe that these advances have practical implications for the design of recurrent memory and potential impact beyond the Bayesian Layer, particularly in light of emerging work on uncertainty-aware recurrent memory. To the best of our knowledge, no other concurrent work examines the uncertainty geometry in relation to associative recall in modern sequence architectures. Furthermore, our work not only places contemporary sequence architectures in a common language; it also opens the door to new sequence architectures based on different probabilistic graphical models or readouts. We have explored only one of many practical possibilities.

We hope that we can make the most of the interactive part of the discussion period, and we ask you to reconsider your rating on the basis of the new controls and of this contribution.

Thank you to the authors for the detailed response and additional experiments.

  1. The mixer-level benchmarks in Appendix C.4 clarify the computational trade-offs between BL and GDN. With matched recurrent-state storage and approximately matched parameter counts, BL has lower throughput and higher training and prefill peak memory on the same hardware. In the state-matched WikiText-103 comparison in Table 5, full BL achieves higher mean long-distance recall, while overall recall is similar and language-model perplexity is higher. Could the authors offer an interpretation of this recall–perplexity trade-off?
  2. The matched comparison among full BL, covariance-reset BL, and GDN is informative. The average RULER recall improvement of 0.55 ± 0.16 percentage points over covariance-reset BL supports a modest benefit from retaining covariance history. However, without a diagonal BL control in the pretrained-model experiment, the additional contribution of off-diagonal correlations remains unresolved. I therefore consider my concern about covariance attribution partially addressed.
  3. The comparison with GDN under the same layerwise distillation procedure yields an average RULER improvement of 0.59 ± 0.19 percentage points. By matching the training objective, trainable components, and data streams, this comparison resolves my previous concern about differences in the adaptation procedure.
  4. I accept the clarification regarding KLA and the mechanistic evidence provided by the controlled comparison between full and diagonal BL.

Overall, I appreciate the revisions, which have addressed most of my major concerns. I have one additional question regarding the linear-Gaussian design model. I understand that these assumptions concern the conditional latent-memory dynamics and observation model, while the input-dependent parameterizations can be nonlinear. Beyond enabling exact Kalman updates, could the authors clarify the modeling motivation for this choice and its limitations in representing memory dynamics?

Reviewer_txsHOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose auxiliary probabilistic models for memory, and derives recurrent updates through Bayesian filtering. The Bayesian layer maintains a memory mean, and dense key-space covariance through Kalman update. Isotropic covariance reset recovers Delta-rule updates, and a separate latent model for additive recurrences. The analysis explains how covariance reduces interference between overlapping keys. Experiments use a pretrained GDN after replacing four mixers through layerwise distillation in controlled recall, MQAR, and language modeling.

Critical strengths
  1. Interference mechanism is supported by derivation and controlled experiments.
  • Equation (15) relates interference to covariance. With supplied keys preventing learned projections from changing the overlaps, dense covariance reaches much higher recall compared to diagonal covariance. This provides concrete evidence that retaining correlations can preserve associations under this protocol.
  1. Recall improvements extend beyond the key geometry.
  • On WikiText-103, both BL variants improve mean associative-recall accuracy over their corresponding baselines, including the longest-distance bin (Tab. 5).
Critical weaknesses
  1. Pretrained model experiment does not isolate the contribution of covariance propagation.
  • BL uses distilled replacements with rotational dynamics. The control continues the original GDN and updates both mixers and MLPs. The gap in objectives, trainable parameters and computation mean that the 25.51% versus 20.51% RULER result at 8k establishes a difference between adaptation procedures, but cannot attribute it to dense covariance.
  • A matched replacement trained with the same distillation procedure and dynamics, with diagonal or reset covariance, is the necessary control.
  1. The recall gains are not evaluated against the memory and cost.
  • BL stores an additional dense covariance, giving recurrent storage of O(Dm+D2)O(Dm+D^2), and its implementation retains sequential dependencies within and across chunks (Appendix C).
  • Therefore, the learned-token comparisons do not match recurrent-state budgets. This is especially important as the benefits of BL is selective.
  • Without an equal-memory comparison or measured cost, the evidence does not establish whether the recall improvement offers a worthwhile alternative to allocating more resources to an existing recurrent model.
Questions for authors

See weaknesses

2 comments

The discussion period ends on Nov 18. We have responded to the weaknesses and questions in your review: the matched control and measured cost against GDN.

We also ask you to consider the theoretical contribution of the paper and its potential impact. Our design-model framework (the first listed contribution) describes existing recurrent layers as a design model together with an inference rule, so that they can be compared by their assumptions about memory (Props. 2 and 3). One can therefore state an assumption about memory and derive the update in a principled way. Our Bayesian Layer is one instance of this framework, and it lets us analyze how the geometry of the uncertainty governs associative recall (Eq. 15).

We believe that these advances have practical implications for the design of recurrent memory and potential impact beyond the Bayesian Layer, particularly in light of emerging work on uncertainty-aware recurrent memory. To the best of our knowledge, no other concurrent work examines the uncertainty geometry in relation to associative recall in modern sequence architectures. Furthermore, our work not only places contemporary sequence architectures in a common language; it also opens the door to new sequence architectures based on different probabilistic graphical models or readouts. We have explored only one of many practical possibilities.

We hope that we can make the most of the interactive part of the discussion period, and we ask you to reconsider your rating on the basis of the new controls and of this contribution.

Thank you for the detailed response. I have tried my best to understand the shared response and the response to my reviews. However, it seems that under matched training, the gap shrinks from 5% to 0.42%, and the average gain mostly comes from 2K context. I personally think the cost for this gain is too small.

However, I do acknowledge that this review comes from an empirical viewpoint, and I do not have an expertise in Bayesian networks to best assess this contribution. I do acknowledge that the paper's main contributions are on the theoretical side, and would not want to reject a paper only from the empirical results. I would retain my score for now and would need to hear how other reviewers assess this paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.