acceptodds
Under review as a conference paper at ICLR 2027

Learn to Adapt Doubly-sequential Bayesian Neural Networks with Complex Architectures

Abstract

Neural networks used for control must adapt from a few sequentially arriving samples, often drawn from suboptimal data, and do so efficiently. Bayesian neural networks (BNNs) provide a principled framework for sample-efficient sequential adaptation with quantified uncertainty, but few techniques can use data to optimize that adaptation at scale. In this paper, we first show that adaptation can be learned from data by optimizing its backward update one layer at a time. This stands in stark contrast to existing meta-learning techniques that learn only from input–output matching and become prohibitive for large networks. The key enabler is to view training as a sequence of layerwise Bayesian posterior updates, which decomposes the problem into optimizing the update rule of each layer. Layerwise moment propagation of this kind was previously confined to feedforward networks; we extend it to computation graphs composed of attention, normalization, feed-forward, residual, and branching operations, so that transformers and other architectures can use data-driven techniques to optimize adaptation/fine tuning at scale. In parallel, we generalize BNNs to a hybrid of imitation-based and reinforcement-based learning so that BNNs learn a more optimal mapping than the data distribution. This builds on a new perspective: the uncertainty a BNN maintains can be repurposed to distinguish optimal from suboptimal samples and to weight the update toward the former. Finally, we integrate the two parts into a data-driven framework to optimize both the update rules of individual layers and the balance between imitation and reinforcement. The proposed framework allows the adaptation algorithm to use past adaptation histories to self-evolve toward improved sample efficiency, optimality, and robustness.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8dmMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an online adaptation framework for Bayesian neural networks (BNNs) with complex architectures, such as Transformers. By decomposing the adaptation process into layerwise Bayesian posterior updates and deriving the corresponding moment propagation rules, the framework enables policy improvement by integrating reward signals, even when adapting from suboptimal data.

Critical strengths
  1. Insightful Motivation: Integrating meta-learning with layerwise Bayesian posterior updates, and introducing reinforcement learning (reward-guided) signals to overcome the limitations of suboptimal data, presents a clear and insightful perspective.

  2. Comprehensive Theoretical Formulation:The theoretical analysis appears to be relatively comprehensive and well-developed.

Critical weaknesses
  1. Shallow Experimental Validation and Unrepresentative Model Scale: While the paper emphasizes its applicability to complex architectures, the Decision Transformer evaluated in the experiments is restricted to merely 1 attention layer, 2 heads, and a hidden dimension of 8. Such a toy-scale model fails to demonstrate the method's effectiveness and is insufficient to support the paper's broad claims regarding scalability.

  2. Severe Computational and Memory Bottlenecks: The proposed layerwise moment propagation and backward smoothing require maintaining and inverting large covariance matrices. If applied to standard Transformers (e.g., with a hidden dimension 768), the computational complexity and memory overhead would become prohibitive.

Questions for authors

What specific algorithmic improvements do the authors envision to make this method scalable for more complex, real-world scenarios?

2 comments

We sincerely thank the reviewer for the encouraging assessment of the motivation and the theory, and for the clear questions on model scale and computational cost.

Regarding the computational cost, the reviewer is right that propagating covariance matrices through the network is expensive in both computation and memory. This is a limitation of the whole class of Kalman Bayesian neural networks on which our work builds. Our contributions, learning to adapt and reward-guided adaptation, aim to improve the optimality of the adaptation, and they do not add to this cost: each new sample still needs one forward and one backward moment pass.

As a result, regarding the question on algorithmic improvements, we regard scalability as an open problem for this class of methods, and we see three directions. First, the Bayesian update can be applied to part of the network only, for example the last block of a large pretrained model. Second, structured covariances such as diagonal, low-rank or Kronecker-factored forms reduce memory and computation; for a 768×768768 \times 768 projection matrix, a Kronecker-factored covariance stores about 1.2×1061.2 \times 10^{6} entries. Third, the update can be applied to a small adapter attached to a frozen large model, so that the cost depends on the adapter size and not on the model size.

Regarding the model scale, we have added a larger decision transformer with 2 attention layers and hidden dimension 16, where the submission used 1 layer and hidden dimension 8. The table reports the success rate of this larger model as the pendulum mass drifts from 1 to 2, averaged over ten adaptation streams.

\text{Method} & \text{Mass }1.00 & 1.25 & 1.50 & 1.75 & 2.00 \\\\ \hline \text{Ours with learned gains} & 1.000 & \mathbf{0.994} & \mathbf{0.847} & \mathbf{0.839} & \mathbf{0.732} \\\\ \text{Ours without learned gains} & 1.000 & 0.938 & 0.716 & 0.537 & 0.332 \\\\ \text{LO-FI} & 1.000 & 0.981 & 0.769 & 0.558 & 0.501 \\\\ \text{KBT} & 1.000 & 0.662 & 0.254 & 0.080 & 0.018 \\\\ \text{Laplace} & 1.000 & 0.512 & 0.346 & 0.388 & 0.295 \\\\ \text{SWAG} & 1.000 & 0.516 & 0.281 & 0.259 & 0.232 \\\\ \text{No fine-tuning} & 1.000 & 0.990 & 0.700 & 0.560 & 0.180 \end{array}$$ At mass 2, learning to adapt improves on LO-FI by 23 points, the same margin as on the small model, so the benefit does not shrink as the model grows.

Thank you for your response. Although the authors have added an experiment with a larger Decision Transformer (hidden dimension 16), the model remains relatively small-scale. I therefore still have concerns about the applicability and practical value of the proposed method in complex real-world scenarios.

However, given that this is primarily a theoretical paper and that other reviewers have expressed positive assessments of its theoretical contributions, I will maintain my original rating.

Reviewer_d6RKOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This work extends existing work on Kalman Bayesian neural networks (KBNNs). In this setting, a neural network is represented as a computational graph whose variables are modeled with jointly Gaussian moments. A moment propagation rule and a Kalman smoother are used to make forward inferences and update weights as new data samples arrive. The authors extend existing work in this domain to incorporate new moment propagation rules for nodes like LayerNorm so that modern transformer architectures can be used in this framework. They also extended the canonical Kalman smoother to incorporate a learned gain, where the learned gain is optimized to mimic an expert Bayesian inference procedure. Finally, the authors extend the KBNN framework to incorporate reward guided output distributions rather than assuming that target outputs are provided. They test their proposed methods on a simple pendulum task and show that their methods adapt to novel settings better than baselines.

Critical strengths

KBNNs are an attractive alternative to typical gradient-optimized deep learning models in that they can adapt online to new data and quantify their uncertainty. This work's most substantive contribution, in principle, is novel moment propagation rules that allow the KBNN framework to be extended to end-to-end transformer computational graphs (prior work used a frozen transformer backbone).

Critical weaknesses
  • The paper's biggest flaw is the clarity of writing. Concepts and terms are introduced without warning. Terms from prior work are replaced with new terms (Kalman Bayesian neural networks is replaced by Doubly sequential Bayesian neural networks) without justification, making it hard to know which prior background reading is required to understand the work. Ultimately, this paper took me substantially longer to review due to the communication quality. In my assessment, any contributions are lost in the text, and the paper, in its current form, is of little to no value to the community.
  • One of the main contributions is a learned gain for the Kalman smoother step. The evaluation does not apply this to the full network, only to the final layer.
  • The evaluation is weak, only testing a single model and task. Though I noted in the strengths that end-to-end transformer KBNN is an important contribution, it is worth noting that the core framework and even some of the require moment propagation rules already existed. So, a more thorough evaluation would have supported the idea that applying this framework to complex architectures is useful and important.
Questions for authors
  1. There are missing citations on L182-183 and L203-204. Please fix these.
2 comments

We thank the reviewer for their careful reading and constructive feedback, and for recognizing the potential value of extending KBNNs to complex architectures. We apologize that the original presentation obscured the contributions and made the paper unnecessarily difficult to follow. We have substantially revised the manuscript to explain the contributions, their relationship to prior work, and how the different components fit together. We have also fixed the missing citations.

Briefly, our contribution is a framework that uses analytical moment propagation through complex computation graphs to enable layerwise learning of Bayesian adaptation rules and reward-guided adaptation. The propagation machinery makes internal posterior updates available for direct supervision, while reward-weighted output moments allow the same backward update mechanism to improve upon suboptimal observed behavior.

The revised manuscript develops this contribution in three connected parts:

  1. Analytical propagation through complex architectures (Section 4). We specify composable local moment routines, account for dependencies introduced by branching and merging, and characterize when these routines provide the moments required for forward propagation and backward inference (Section 4.3, Proposition 1). This supplies the foundation for applying the subsequent learning methods to architectures containing attention, normalization, and residual connections.

  2. Layerwise learning of adaptation rules (Section 5). We learn gain corrections from expert posterior moments, directly supervising internal Bayesian updates. Training proceeds from output to input, holding previously optimized groups fixed so that dependencies between groups are accounted for (Section 5.3, Algorithm 2).

  3. Reward-guided adaptation and learning its objective (Section 6). We use reward-tilted output moments to drive the same Bayesian updates (Section 6.1), and learn a task-dependent temperature governing the balance between preserving reference behavior and pursuing higher return (Section 6.2).

To make these connections easier to follow, we have strengthened their explanation in the Abstract and Introduction, reiterated the purpose of each component at the beginning of Sections 4–6, and restated their integration in the Conclusion.

We also appreciate the concern about terminology. “Doubly sequential” describes updates that proceed sequentially over incoming samples and backward through network layers (Section 3.3). It is intended to describe the update structure underlying the KBNN methods on which we build, rather than suggest that this established foundation is itself our contribution. Section 3.4 presents the Kalman-style moment updates, and Appendix A discusses their relationship to prior KBNN work.

Regarding moment propagation, the intended contribution is analytical moment propagation as an enabler of the learning framework, rather than a claim of superior speed or accuracy over other propagation methods. Here, “analytical” refers to propagating approximate moments through specified mathematical routines instead of estimating them by Monte Carlo sampling of the network. These routines use Gaussian approximations and, where needed, deterministic quadrature; they do not imply exact Bayesian inference (Appendix B.1). We agree that Kalman smoothing and some individual propagation rules already exist. Our contribution concerns their extension and composition on complex graphs and the learning mechanisms this makes possible.

The framework is also modular: compatible propagation routines can be substituted provided they supply the required marginal moments and cross-covariances (Section 4.1). Thus, if there exist more advanced moment propagation techniques, they can be incorporated into the same layerwise learning and reward-guided adaptation framework. A comparative benchmark of individual propagation methods is therefore outside the paper’s central scope.

Evaluation of the learned gain beyond the final layer:

We apologize for the confusion, as we made the misstatement that only the gain correction matrices in the final layer is learned. The reason we used only the gain correction matrices in the final layer is that the gain corrections from other layers provide little additional performance improvements. In the updated simulation in Section 7.1, we use a different expert described in Appendix H.2, and demonstrate that learning more gain correction matrices gives additional performance improvements.

Thank you to the authors for their efforts in addressing my concerns. The additional appendix results address my concern that one of the primary contributions (learned gain) isn't sufficiently evaluated. My broader comment about a weak evaluation, even for a non-experimental paper, still stands. More tasks, architectures, and ablations would strengthen this component of the paper.

The authors have tried to address my largest concern which is the clarity of the text. It appears that this revision does improve the organization of the paper. Unfortunately, it is a substantial rewrite, which is appropriate given the prior issues, but in my view, it amounts to a totally different paper. As such, I cannot substantively review its content (and there is new content) in the short timeframe before the author-reviewer discussion period ends. Assuming it does address the initial writing concerns, I am open to raising my score, but I will do so after seeing the other reviewers' responses.

Reviewer_tVUsOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The proposed method for BNNs learning involves a layerwise update using variance propagation, meta-learning using learned gain modifiers, and RL using regularized policy iteration to define target output distributions. For the forward pass, they derive rules for propagation of second moments through various sublayer types beyond matrix-vector products. The backward pass starts with a target output distribution and propagates back to posterior distributions for activations and parameters at earlier layers, each adapted by learned gain modifiers.

Critical strengths

Ambitious and extensive theory, including a novel update rule for BNNs based on variance propagation, a method for meta-learning, and integration of reward- and likelihood-based updates.

Critical weaknesses

The algorithm is presented with very little explanation. Subroutines are referenced by name but not defined or explained. Design principles are mostly not given, so even if the reader had enough information to reproduce it, it would still be less than clear how it works or what it is meant to do.

Questions for authors

I work on BNN optimization including layerwise methods with variance propagation, so I expected to understand this paper well. The forward pass makes sense, but for the update there are numerous places where I cannot work out what the algorithm does, or why.

I expected the backward pass to implement (exact or approximate) Kalman smoothing under the linear-Gaussian approximation of the network computed by the forward pass. Is this correct, at least in the first half of sec 4.3 through (18) and with Ls=Ids\mathbf{L}_s=\mathbf{I}_{d_s}?

What is happening in the second half of sec 4.3? What are the student and expert? Why is (19) defined as written? Why is this loss more effective than, say, KL divergence? How is TargetLaw defined? Where does QtarQ^{\rm tar} come from? How is Λexp\Lambda^{\rm exp} used? What is a task object Te\mathcal{T}_e? What is Cexp\mathcal{C}^{\rm exp}? How does ExpertUpdate work?

What setting is envisioned for meta-learning? Why is Ls\mathbf{L}_s expected to transfer well across tasks?

3 comments

(1/2)

We thank the reviewer for pointing out the issues in writing clarity. We have substantially revised the manuscript to address these points. Below, we provide direct clarifications and indicate where the corresponding explanations appear in the revised manuscript. All section and equation numbers below refer to the revised version unless otherwise specified.

1. Interpretation of the backward pass.

Yes, this is the intended interpretation when Lv=IdvL_v=I_{d_v}. More precisely, the backward pass performs Gaussian conditional updates using the moments computed by the forward pass. Section 4.4 (p. 6) now explicitly describes the Gaussian model on the tree formed by the selected target relationships v→τ(v)v\to\tau(v). The updated output moments are propagated backward through its conditional distributions using Eqs. (17)–(18), in the order specified by B1,B2,…B_1,B_2,\ldots.

Thus, the smoothing interpretation applies to this induced Gaussian model, which approximates the original nonlinear network. Setting Lv=IdvL_v=I_{d_v} in Eqs. (19)–(21) recovers the canonical updates in Eqs. (17)–(18). We have separated this backward pass in Section 4.4 from the learned gain corrections introduced in Section 5.

2. Purpose of the second half of the original Section 4.3, and the roles of the student and expert.

This part learns corrections to the backward update so that the resulting posterior moments better match those supplied by a more accurate inference procedure. We now present it separately in Section 5, with subsections for the learned gains, expert posterior moments, and training procedure.

The student is the proposed BNN, which performs adaptation through forward and backward moment propagation. The expert is a computationally more expensive Bayesian posterior estimator that supplies reference means and covariances at internal nodes. Section 5.2 (p. 7) explains that both start from the same task prior and process the same dataset in the same order. Appendix E gives a concrete expert implementation based on sequential Monte Carlo (SMC).

Section 5.3 and Algorithm 2 (pp. 7–8) explain how these reference moments supervise training. For the current group BrB_r, the student first updates the previously trained groups using their fixed correction matrices, then updates BrB_r using the matrices currently being optimized. Its updated moments are compared with the expert moments after the same observation. Training proceeds from output to input, keeping previously optimized groups fixed.

3. Motivation for the loss in the original Eq. (19), and comparison with KL divergence.

The method does not require this particular loss, and we do not claim that it is generally more effective than KL divergence. Section 5.3 now explicitly allows a differentiable discrepancy between student and expert moments. Appendix F (pp. 24–25) presents the normalized moment discrepancy as one example and explains its components in Eqs. (58)–(59).

The mean term measures errors in units of the expert standard deviations. The covariance term compares the full covariance matrices, including their off-diagonal entries, after the same coordinate scaling. The factors dv−1d_v^{-1} and dv−2d_v^{-2} average the squared errors over vector and matrix entries, respectively, while λexp\lambda^{\mathrm{exp}} controls the relative weight of covariance matching. These choices account for differences in coordinate scales and node dimensions while directly supervising the moments retained by the student. Appendix F also describes a variance floor for zero or very small expert marginal variances.

(2/2)

4. Definition of TargetLaw and the source of QtarQ^{\mathrm{tar}}.

TargetLaw was an abstract interface for constructing the output target distribution. We have removed this interface and now specify the targets directly for the two settings.

For supervised adaptation, Section 4.4 (p. 6) initializes the backward pass with

μo+=yk,Σo+=ϵI,\mu_o^+=y_k,\qquad \Sigma_o^+=\epsilon I,

where ϵ\epsilon accounts for uncertainty in the observed output.

For reward-guided adaptation, Section 6.1 defines the target explicitly in Eq. (24) (p. 8):

\propto \pi_{\mathrm{ref}}(a\mid s) \exp\\!\left(\frac{A^{\pi_{\mathrm{ref}}}(s,a)}{\beta_R}\right).$$ Here, the reference policy and advantage estimates determine the target. The paragraph preceding Algorithm 3 (p. 9) defines $Q_k^{\mathrm{tar}}(da)=q_{\beta_R}^{\mathrm{tar}}(a\mid s_k)\\,da$, and Algorithm 3 uses its mean and covariance to initialize the backward pass. **5. Meaning of $\Lambda^{\mathrm{exp}}$, $\mathcal C^{\mathrm{exp}}$, and ExpertUpdate.** These objects belonged to the original abstract description of the expert: $\Lambda^{\mathrm{exp}}$ was an output-replacement factor, $\mathcal C^{\mathrm{exp}}$ denoted the persistent inference state, and ExpertUpdate advanced that state and returned posterior moments. We have replaced this presentation with an explicit supervised SMC example in Appendix E (pp. 23–24). The expert targets the posterior in Eq. (55), obtained from the task prior and the observation likelihoods. Equation (56) gives the incremental likelihood-based particle-weight update. The subsequent text describes normalization, resampling, and posterior-invariant MCMC transitions. The particles and weights constitute the persistent expert state: they are retained across observations and reinitialized when a new task begins. Evaluating the network with these parameter particles produces the internal-node posterior moments in Eq. (57). Appendix E also clarifies that the expert evolves independently of the student’s gain correction matrices, and its moment estimates are held fixed when differentiating the student loss. This example concerns supervised likelihood-based inference. **6. Meaning of a task and the intended meta-learning setting.** The task object previously denoted by $\mathcal T_e$ is now denoted by $\mathcal T$. Section 5.2 (p. 7) defines it as supplying a parameter prior and a training dataset of size $N_{\mathcal T}$. The intended setting is to learn an adaptation rule over a task distribution $p_{\mathrm{task}}$. Each optimization iteration samples a task and initializes both the student and expert from its prior. They then process its observations sequentially. Within each task, Bayesian updates adapt the network’s parameter moments. Across tasks, Algorithm 2 learns the gain correction matrices by matching the student’s updates to the expert’s posterior moments. **7. Why $L_v$ is expected to transfer across tasks.** We would like to clarify that transferring across tasks refers to transferring across tasks within the task distribution. Since the learning process in Algorithm 2 is performed using data from the task distribution, we expect the learned gain correction matrices to have a good performance across tasks within the task distribution.

Thank you for the explanations and the substantial revision of the paper. The presentation is much better and I believe I understand everything now.

This is an unusual situation because the paper has changed so much and really should undergo a fresh set of reviews. Despite understanding the method, I would prefer to have time to carefully analyze it with my own derivations. Reviewer d6RK seems to express the same dilemma.

A further concern is that section 6 is not well connected to the rest of the paper. It applies the proposed training method to regularized policy improvement without any apparent synergy (i.e., the objective and the update are just stitched together). (Incidentally, the one part I can't work out is where the target covariance Σaktar\Sigma_{\boldsymbol{a}_k}^{\rm tar} comes from. Eq 24 should provide only a point estimate.) The experimental evaluation is very thin, so I would suggest removing the RL portion to another paper to make room for more careful evaluation of the main method.

Despite these concerns, the method looks to be a significant advance. With more time to review it carefully and with more thorough experiments, there's a good chance I would come to a strong evaluation. On balance I'm now at a weak accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.