- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes DiMS, a sampler for the set of loss minima of a neural network. The goal is better uncertainty estimates, mainly for OOD detection.
DiMS starts at the trained model with a random velocity drawn from the Laplace approximation. The particle moves on the loss surface as in the Riemannian Laplace approximation (RLA), with two additions: 1) a pull toward lower loss and 2) friction proportional to speed. The particle loses energy and stops at a minimum, so each sample fits the training data but can predict differently elsewhere. The paper shows that the energy decreases and that the particle ends at stationary points of the loss. It also bounds the time to slow down and the distance travelled. Various experiments indicate the improved performance of the method in comparison to other Laplace-based methods, especially in the OOD setting.
Critical strengths
The authors present a simple yet effective change to fix a known weakness of the Laplacian-family methods. Making every sample to be at a minimum via the pull and friction mechanisms seems to favour OOD detection, while maintaining interpolation. The empirical results support the authors' claims, even under Hessian approximations, at least for the tested small-scaled regime. I found the approach quite interesting.
Critical weaknesses
- One thing I would like to point is that it is not clear to me whether each trajectory converges to a single point. Theorem 1 shows convergence to the set , but its last sentence and its proof say that trajectories converge to a local minimum. The proof of Theorem 3 also refers to “the position to which the trajectory converges”. However, LaSalle’s principle only shows that the velocity and the gradient go to zero. With non-isolated minima, the particle can keep sliding along the set of minima. For example, for the Mexican hat function, gradient flow spirals toward a circle of minima forever and never converges to a point (Absil et al., 2005, Section 2). I do not see this as a major issue. Convergence to a point can usually be shown with a Lojasiewicz inequality (e.g., for analytic losses or near Morse-Bott minima) together with friction that stays positive at rest (Haraux and Jendoubi, 1998; Bégout et al., 2015). DiMS’s friction vanishes at rest, so it would need a floor such as . Could the authors either prove convergence to a point or state the results as convergence to the set of stationary points? I think this matters (at least theoretically) for the sampler as without strong convergence (single limit-point), the sample in Eq. (10) depends on , which then acts as a hyperparameter.
- Absil, P.-A., Mahony, R., and Andrews, B. (2005). Convergence of the iterates of descent methods for analytic cost functions. SIAM Journal on Optimization, 16(2), 531–547.
- Bégout, P., Bolte, J., and Jendoubi, M. A. (2015). On damped second-order gradient systems. Journal of Differential Equations, 259, 3115–3143.
- Haraux, A. and Jendoubi, M. A. (1998). Convergence of solutions of second-order gradient-like systems with analytic nonlinearities. Journal of Differential Equations, 144(2), 313–320.
- The experiments compare DiMS only with Laplace-based methods. I think this is reasonable, since the main argument is the improvement over the Laplace approximation and RLA. Deep ensembles are the standard OOD baseline, but they need several training runs, while DiMS works post hoc from a single trained network, so I see this comparison as future work. However, it would be good to compare numerically with the Laplace diffusion method of Roy et al. (2024), which the authors cite. It is the closest method in aim, and it was evaluated on the same LeNet setup on MNIST and FMNIST. The paper also describes such methods as inherently local, which DiMS aims to overcome. A direct comparison would support this claim.
Questions for authors
-
How sensitive are the samples and metrics to the integration time ? Because the friction vanishes at rest, motion along flat directions slows only like . A plot of NLL, OOD AUROC and sample weight norm against , or a comparison of with , would show whether samples have settled at and 200.
-
Can you report the training loss and gradient norm at the final samples, and add a MAP row to Table 1? DiMS reaches 0.950 accuracy on MNIST, below what LeNet usually achieves, so it would help to confirm the samples sit at minima of the full training loss.
-
In the image experiments, does DiMS use the loss with or without the prior? Do the dynamics use the full training set or the 1,000-sample subset mentioned in App. D.3?
-
Following the second point in the weaknesses, how does DiMS compare with Roy et al. (2024)?
-
What is the wall-clock cost relative to RLA, given that DiMS integrates to rather than ?
4 comments
Response to reviewer hQmlr - 2/2
Q1 ( sensitivity). Tables 3-4 compare two horizons on the banana dataset: the wall-clock time of RLA, and (about 7 times the compute).
| DiMS | NLL | ECE | OOD AUROC | Train loss | Decrement | CPU [s]/sample |
|---|---|---|---|---|---|---|
| RLA wall-clock | 0.341 ± 0.004 | 1.44 ± 0.14 | 0.933 ± 0.009 | 0.126 | 0.0169 | 18.3 |
| 0.346 ± 0.003 | 2.26 ± 0.21 | 0.928 ± 0.010 | 0.118 | 0.0012 | 135 |
NLL and OOD AUROC are unchanged within standard error, the training loss settles at the level of the MAP (0.117), and ECE stays below all other methods. The samples have thus largely settled at the shorter horizon.
Q2. MAP rows are added to Table 2 (0.990 on MNIST). For the banana dataset, Table 4 reports training loss, gradient norm and Gauss-Newton decrement at the final samples; the gradient norms are small but not exactly zero at these horizons, and the decrement shows DiMS ends considerably closer to a minimum than RLA and LA + GD. For MNIST your reading is correct: the dynamics are defined on the 1,000-sample subset (Q3), so the samples minimize the subset loss rather than the full training loss, which explains 0.950 against the MAP.
Q3. The image experiments use the loss with the prior, whose precision is tuned by the marginal log-likelihood (Sec. 5), defined on the 1,000-sample subset (App. D.3). The low-rank Hessian approximation has rank 20.
Q5. DiMS in Table 1 is run at the wall-clock time of RLA (18.3 vs. 16.3 CPU s/sample), so the comparison above is at equal cost.
We would like to thank again for the constructive review, especially the suggestion about the friction floor, . We hope our responses address your points and are glad to follow up on any of them.
I would like to thank the authors for their reply. I'm covered by their response and thus I will maintain my acceptance score. Glad that my recommendation help you improve the paper.
We thank the reviewer for the positive assessment and are glad our response addressed the questions. We appreciate the support and hope it will guide the final decision.
Response to reviewer hQmlr - 1/2
Thank you for the constructive review. Your first point led to one of the most useful changes in the revision, so we start there. Changes are in blue in the revised PDF.
W1 (convergence to a set vs. a point, friction floor). You are right: LaSalle's principle gives convergence to the set I, not to a point, and our wording went beyond that. We did both things you suggested.
Statements. Theorem 1 and Eq. 11 now claim only convergence to the set, dist(α(t1),S∗(D))→0; "the position to which the trajectory converges" is gone.
Friction floor. We adopted your η=η0T+ηmin (Eq. 9), and it did more than we expected:
The new banana experiments use η0=0.2 and ηmin=0.05, and it improves the computational speed of our method.
We do not claim convergence to a single point. With the floor in place, a Łojasiewicz argument along the lines of Haraux & Jendoubi and Bégout et al. is the natural route. Those results are Euclidean, whereas Eq. 8 has an additional velocity-dependent geometric term, so it needs its own proof, which we will consider in future work. On t1 acting as a hyperparameter: for now it is one, and Q1 below along with Fig. 14 in the updated manuscript shows how much the horizon matters in practice. The result of our new Theorem 2 justifies replacing t1 by a stopping criterion on the velocity and gradient norms.
W2 / Q4 (Baselines). We evaluate DiMS on the banana dataset (Table 1, App. E, using 5 seeds and S=50), together with a deep ensemble (DE), gradient descent from LA samples (LA + GD), kernel diffusion by Roy et al., and constant-friction / Euclidean ablations:
Kernel diffusion (KD) is run as described by Roy et al., including their convention for the noise scale. Its Euler-Maruyama iterates drift off the minimum submanifold because the kernel of the Jacobian is tangent to it only to first order, so the drift is governed by the curvature along the trajectory; Fig. 10 shows that it persists for step sizes from 10−3 to 10−6. DiMS has no such constraint to violate: the gravitational pull returns the particle to a minimum level set. On locality, a new Corollary 1 makes precise how far DiMS may leave the initial level set: the loss along the path exceeds L(θ∗) by at most ∥v~∥2/(2κ).