- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an online adaptation framework for Bayesian neural networks (BNNs) with complex architectures, such as Transformers. By decomposing the adaptation process into layerwise Bayesian posterior updates and deriving the corresponding moment propagation rules, the framework enables policy improvement by integrating reward signals, even when adapting from suboptimal data.
Critical strengths
-
Insightful Motivation: Integrating meta-learning with layerwise Bayesian posterior updates, and introducing reinforcement learning (reward-guided) signals to overcome the limitations of suboptimal data, presents a clear and insightful perspective.
-
Comprehensive Theoretical Formulation:The theoretical analysis appears to be relatively comprehensive and well-developed.
Critical weaknesses
-
Shallow Experimental Validation and Unrepresentative Model Scale: While the paper emphasizes its applicability to complex architectures, the Decision Transformer evaluated in the experiments is restricted to merely 1 attention layer, 2 heads, and a hidden dimension of 8. Such a toy-scale model fails to demonstrate the method's effectiveness and is insufficient to support the paper's broad claims regarding scalability.
-
Severe Computational and Memory Bottlenecks: The proposed layerwise moment propagation and backward smoothing require maintaining and inverting large covariance matrices. If applied to standard Transformers (e.g., with a hidden dimension 768), the computational complexity and memory overhead would become prohibitive.
Questions for authors
What specific algorithmic improvements do the authors envision to make this method scalable for more complex, real-world scenarios?
2 comments
Thank you for your response. Although the authors have added an experiment with a larger Decision Transformer (hidden dimension 16), the model remains relatively small-scale. I therefore still have concerns about the applicability and practical value of the proposed method in complex real-world scenarios.
However, given that this is primarily a theoretical paper and that other reviewers have expressed positive assessments of its theoretical contributions, I will maintain my original rating.
We sincerely thank the reviewer for the encouraging assessment of the motivation and the theory, and for the clear questions on model scale and computational cost.
Regarding the computational cost, the reviewer is right that propagating covariance matrices through the network is expensive in both computation and memory. This is a limitation of the whole class of Kalman Bayesian neural networks on which our work builds. Our contributions, learning to adapt and reward-guided adaptation, aim to improve the optimality of the adaptation, and they do not add to this cost: each new sample still needs one forward and one backward moment pass.
As a result, regarding the question on algorithmic improvements, we regard scalability as an open problem for this class of methods, and we see three directions. First, the Bayesian update can be applied to part of the network only, for example the last block of a large pretrained model. Second, structured covariances such as diagonal, low-rank or Kronecker-factored forms reduce memory and computation; for a 768×768 projection matrix, a Kronecker-factored covariance stores about 1.2×106 entries. Third, the update can be applied to a small adapter attached to a frozen large model, so that the cost depends on the adapter size and not on the model size.
Regarding the model scale, we have added a larger decision transformer with 2 attention layers and hidden dimension 16, where the submission used 1 layer and hidden dimension 8. The table reports the success rate of this larger model as the pendulum mass drifts from 1 to 2, averaged over ten adaptation streams.
\text{Method} & \text{Mass }1.00 & 1.25 & 1.50 & 1.75 & 2.00 \\\\ \hline \text{Ours with learned gains} & 1.000 & \mathbf{0.994} & \mathbf{0.847} & \mathbf{0.839} & \mathbf{0.732} \\\\ \text{Ours without learned gains} & 1.000 & 0.938 & 0.716 & 0.537 & 0.332 \\\\ \text{LO-FI} & 1.000 & 0.981 & 0.769 & 0.558 & 0.501 \\\\ \text{KBT} & 1.000 & 0.662 & 0.254 & 0.080 & 0.018 \\\\ \text{Laplace} & 1.000 & 0.512 & 0.346 & 0.388 & 0.295 \\\\ \text{SWAG} & 1.000 & 0.516 & 0.281 & 0.259 & 0.232 \\\\ \text{No fine-tuning} & 1.000 & 0.990 & 0.700 & 0.560 & 0.180 \end{array}$$ At mass 2, learning to adapt improves on LO-FI by 23 points, the same margin as on the small model, so the benefit does not shrink as the model grows.