- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes to dect drift using model unceratinty. It uses a linearized Laplace approximation to estimate uncertainty and derives a theoretical analysis connecting EU to training data. The approach allows drift detection without requiring labels or access to the training data during deployment. While the framework is interesting, the main contribution is very limited and based on preexisting ideas. The paper spent lot of time trying to justify the method mathematically but much of it follows directly from standard matrix decomposition. The overall novelty of the contribution therefore appears limited.
Critical strengths
The topic is important and relevant.
- The proposed method uses EU to monitor distribution shift without requiring labels during deployment. This is relevant when training data is not available.
- the decomposition into training-supported and unsupported parameter directions is interesting interpretation of how training data coverage affects EU.
Critical weaknesses
The theoretical framing offer an incremental contribution but I have not seen a fundamentally new method or a surprising theoretical result. The potential contribution is its explicit theoretical interpretation of uncertainty through prediction-relevant training coverage but the mathematical results are relatively straightforward.
- Bayesian uncertainty formulation is well established
- Laplace framework is also established etc.
- The theoretical analysis is formulated using the full Laplace posterior covariance, which scales quadratically with the number of model parameters. So its unclear how the propose method scals on medium or large models. The validation experiments is done using relatively small models.
- How is S the number of Monte Carlo samples drawn from the Bayesian posterior to estimate epistemic uncertainty (EU) choosen en pratice and what is its relationship with the batch size?
Questions for authors
-see above
2 comments
Thanks for your reply and the further explanations. I also note the practical relevance of the method and the experimental results which show some advantages over predictive entropy. I still think the main contribution is very much based on existing ideas and that the theoretical analysis is not particularly novel for ICLR. I'll therefore stick with my original scores.
Response to Reviewer WeSJ
We address the concerns regarding novelty, scalability, and uncertainty estimation below.
W1 – Novelty and contribution. We agree that Bayesian uncertainty, the Laplace approximation, and the associated matrix decompositions are established tools, and we do not claim these individually as contributions. As emphasized in Section 2, our objective is not to develop another uncertainty estimator. Rather, our contribution lies in connecting these tools to a distinct problem: detecting distribution shifts that expose insufficient prediction-relevant training coverage, rather than simply detecting changes in the input distribution.
This connection has three aspects. First, we formulate a model-aware monitoring objective in which shifts between different but sufficiently covered operating conditions should not necessarily trigger alarms. Second, Theorem 4.2 characterizes how training curvature attenuates EU along supported prediction-relevant directions, while insufficiently constrained directions retain higher uncertainty. This provides a theoretical interpretation of when EU reflects limited training coverage, beyond treating uncertainty as a heuristic novelty score. Third, we translate this characterization into a practical label-free sequential monitoring framework.
The distinction has meaningful experimental consequences. For example, in the ambiguous label-prior experiment of Appendix E.7, predictive entropy increases and triggers false alarms despite unchanged prediction error, whereas EU remains stable. Across the original experiments, the advantage is particularly pronounced in audio classification, with AUROC of 0.98 versus 0.59 for predictive entropy on ESC-50 (CNN), and 0.98 versus 0.80 on UrbanSound8K (CNN-wide). These results illustrate the practical significance of isolating the uncertainty associated with limited model knowledge.
W2 – Scalability. The full Laplace covariance is used for the theoretical characterization, but is not required by our practical implementation. As detailed in Appendix D, we use a post-hoc last-layer Laplace approximation with diagonal generalized Gauss–Newton curvature (Daxberger et al., 2021). Consequently, posterior storage and the associated diagonal operations scale linearly, rather than quadratically, with the number of last-layer parameters. The backbone remains deterministic, and posterior sampling requires only evaluating the final layer on previously computed features.
Our evaluation already includes a ViT-B/16 backbone with approximately 86 million parameters, demonstrating applicability beyond small networks. Furthermore, the framework is not restricted to Laplace approximations: our experiments also use MC-dropout (Gal and Ghahramani, 2016), which does not require storing a posterior covariance and exhibits similar qualitative behavior.
Q – Number of posterior samples and batch size. The posterior sample count S and deployment batch size B are independent parameters serving different purposes. Specifically, S controls the Monte Carlo approximation of per-sample EU in Eq. (17), whereas B determines how many incoming observations are averaged to obtain the monitoring statistic in Eq. (18). Increasing S improves the accuracy of uncertainty estimation, while increasing B reduces the variability of the batch-level monitoring statistic due to sampling deployment inputs.
In our CIFAR-100-C experiments, for example, we use S=20 posterior samples, independently of the deployment batch size. For last-layer Laplace, the backbone is evaluated only once per input, while additional posterior samples require computations only in the final layer. Thus, moderate posterior sampling can be implemented efficiently without repeatedly evaluating the entire network.
References:
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortless Bayesian deep learning. Advances in Neural Information Processing Systems, 34, 2021.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, pp. 1050–1059. PMLR, 2016.