acceptodds
Under review as a conference paper at ICLR 2027

Don't Stop Me Yet: Sampling Loss Minima via Dissipative Riemannian Mechanics

Abstract

The minima of modern neural network loss functions are typically not isolated, rather they form connected components of reparameterization invariant solutions on the training data. Analytically characterizing these solutions is a hard problem, but sampling approaches are feasible. By construction, existing methods either spread over low-loss regions, and thus do not sample reparameterization invariant solutions exactly, or are inherently local, which limits exploration of other minima valleys. We propose sampling such reparameterization invariant models using a dynamical system based on kinetic energy, subject to a gravitational pull and a friction term that dissipates energy from the system. Our proposed sampler, DiMS, is guaranteed to sample exactly from the minimum level sets and depends on physically motivated hyperparameters which allows control over the exploration capabilities of the sampler. We consider out-of-distribution detection through uncertainty quantification as the motivating problem and observe improved performance compared to previously proposed approaches.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Uncertainty quantification, Sampling, Riemannian geometry

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hQmrOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DiMS, a sampler for the set of loss minima of a neural network. The goal is better uncertainty estimates, mainly for OOD detection.

DiMS starts at the trained model with a random velocity drawn from the Laplace approximation. The particle moves on the loss surface as in the Riemannian Laplace approximation (RLA), with two additions: 1) a pull toward lower loss and 2) friction proportional to speed. The particle loses energy and stops at a minimum, so each sample fits the training data but can predict differently elsewhere. The paper shows that the energy decreases and that the particle ends at stationary points of the loss. It also bounds the time to slow down and the distance travelled. Various experiments indicate the improved performance of the method in comparison to other Laplace-based methods, especially in the OOD setting.

Critical strengths

The authors present a simple yet effective change to fix a known weakness of the Laplacian-family methods. Making every sample to be at a minimum via the pull and friction mechanisms seems to favour OOD detection, while maintaining interpolation. The empirical results support the authors' claims, even under Hessian approximations, at least for the tested small-scaled regime. I found the approach quite interesting.

Critical weaknesses
  1. One thing I would like to point is that it is not clear to me whether each trajectory converges to a single point. Theorem 1 shows convergence to the set I\mathcal{I}, but its last sentence and its proof say that trajectories converge to a local minimum. The proof of Theorem 3 also refers to “the position to which the trajectory converges”. However, LaSalle’s principle only shows that the velocity and the gradient go to zero. With non-isolated minima, the particle can keep sliding along the set of minima. For example, for the Mexican hat function, gradient flow spirals toward a circle of minima forever and never converges to a point (Absil et al., 2005, Section 2). I do not see this as a major issue. Convergence to a point can usually be shown with a Lojasiewicz inequality (e.g., for analytic losses or near Morse-Bott minima) together with friction that stays positive at rest (Haraux and Jendoubi, 1998; Bégout et al., 2015). DiMS’s friction vanishes at rest, so it would need a floor such as η0T+ηmin⁡\eta_0\sqrt{T} + \eta_{\min}. Could the authors either prove convergence to a point or state the results as convergence to the set of stationary points? I think this matters (at least theoretically) for the sampler as without strong convergence (single limit-point), the sample α(t1)\alpha(t_1) in Eq. (10) depends on t1t_1, which then acts as a hyperparameter.
  • Absil, P.-A., Mahony, R., and Andrews, B. (2005). Convergence of the iterates of descent methods for analytic cost functions. SIAM Journal on Optimization, 16(2), 531–547.
  • Bégout, P., Bolte, J., and Jendoubi, M. A. (2015). On damped second-order gradient systems. Journal of Differential Equations, 259, 3115–3143.
  • Haraux, A. and Jendoubi, M. A. (1998). Convergence of solutions of second-order gradient-like systems with analytic nonlinearities. Journal of Differential Equations, 144(2), 313–320.
  1. The experiments compare DiMS only with Laplace-based methods. I think this is reasonable, since the main argument is the improvement over the Laplace approximation and RLA. Deep ensembles are the standard OOD baseline, but they need several training runs, while DiMS works post hoc from a single trained network, so I see this comparison as future work. However, it would be good to compare numerically with the Laplace diffusion method of Roy et al. (2024), which the authors cite. It is the closest method in aim, and it was evaluated on the same LeNet setup on MNIST and FMNIST. The paper also describes such methods as inherently local, which DiMS aims to overcome. A direct comparison would support this claim.
Questions for authors
  1. How sensitive are the samples and metrics to the integration time t1t_1? Because the friction η0T\eta_0 \sqrt{T} vanishes at rest, motion along flat directions slows only like 1/t1/t. A plot of NLL, OOD AUROC and sample weight norm against t1t_1, or a comparison of t1t_1 with 2t12t_1, would show whether samples have settled at t1=50t_1 = 50 and 200.

  2. Can you report the training loss and gradient norm at the final samples, and add a MAP row to Table 1? DiMS reaches 0.950 accuracy on MNIST, below what LeNet usually achieves, so it would help to confirm the samples sit at minima of the full training loss.

  3. In the image experiments, does DiMS use the loss with or without the prior? Do the dynamics use the full training set or the 1,000-sample subset mentioned in App. D.3?

  4. Following the second point in the weaknesses, how does DiMS compare with Roy et al. (2024)?

  5. What is the wall-clock cost relative to RLA, given that DiMS integrates to t1=50–200t_1 = 50–200 rather than t1=1t_1 = 1?

4 comments
AuthorsOpenReview ↗

Response to reviewer hQmlr - 1/2

Thank you for the constructive review. Your first point led to one of the most useful changes in the revision, so we start there. Changes are in blue in the revised PDF.

W1 (convergence to a set vs. a point, friction floor). You are right: LaSalle's principle gives convergence to the set I\mathcal I, not to a point, and our wording went beyond that. We did both things you suggested.

  • Statements. Theorem 1 and Eq. 11 now claim only convergence to the set, dist(α(t1),S∗(D))→0\mathrm{dist}(\alpha(t_1),S_*(\mathcal D))\to 0; "the position to which the trajectory converges" is gone.

  • Friction floor. We adopted your η=η0T+ηmin⁡\eta=\eta_0\sqrt{T}+\eta_{\min} (Eq. 9), and it did more than we expected:

    • It gives damping of order ∥α˙∥2\lVert\dot\alpha\rVert^2 at rest, which the speed-dependent term alone does not; this is your 1/t1/t observation. We also used this to obtain a bound on the time until a stopping criterion on velocity and gradient norms is met (new Theorem 2). The idea is as follows: the energy HH stalls at uphill turning points, so we consider a look-ahead energy E=H+δ0⟨α˙,∇θL⟩E=H+\delta_0\langle\dot\alpha,\nabla_\theta\mathcal L\rangle, which the floor lets us show is strictly decreasing away from rest. This allows us to bound the total time spent outside ∥α˙∥2<ϵ, ∥∇θL∥2<δ\\{\lVert\dot\alpha\rVert^2<\epsilon,\ \lVert\nabla_\theta\mathcal L\rVert^2<\delta\\}.
    • It makes the guaranteed traveled distance depend on the initial speed (Proposition 1).
    • It recovers Polyak's heavy-ball in the Euclidean limit.

    The new banana experiments use η0=0.2\eta_0=0.2 and ηmin⁡=0.05\eta_{\min}=0.05, and it improves the computational speed of our method.

We do not claim convergence to a single point. With the floor in place, a Łojasiewicz argument along the lines of Haraux & Jendoubi and Bégout et al. is the natural route. Those results are Euclidean, whereas Eq. 8 has an additional velocity-dependent geometric term, so it needs its own proof, which we will consider in future work. On t1t_1 acting as a hyperparameter: for now it is one, and Q1 below along with Fig. 14 in the updated manuscript shows how much the horizon matters in practice. The result of our new Theorem 2 justifies replacing t1t_1 by a stopping criterion on the velocity and gradient norms.

W2 / Q4 (Baselines). We evaluate DiMS on the banana dataset (Table 1, App. E, using 5 seeds and S=50S=50), together with a deep ensemble (DE), gradient descent from LA samples (LA + GD), kernel diffusion by Roy et al., and constant-friction / Euclidean ablations:

MethodAcc.NLLECEOOD AUROCTrain lossCPU [s]/sample
KD (Roy et al.)57.60.75820.710.5506.7622.9
RLA87.50.3254.380.8150.18316.3
DE86.80.3614.450.8880.11746.0
DiMS87.30.3411.440.9330.12618.3

Kernel diffusion (KD) is run as described by Roy et al., including their convention for the noise scale. Its Euler-Maruyama iterates drift off the minimum submanifold because the kernel of the Jacobian is tangent to it only to first order, so the drift is governed by the curvature along the trajectory; Fig. 10 shows that it persists for step sizes from 10−310^{-3} to 10−610^{-6}. DiMS has no such constraint to violate: the gravitational pull returns the particle to a minimum level set. On locality, a new Corollary 1 makes precise how far DiMS may leave the initial level set: the loss along the path exceeds L(θ∗)\mathcal L(\theta^*) by at most ∥v~∥2/(2κ)\lVert\tilde v\rVert^2/(2\kappa).

AuthorsOpenReview ↗

Response to reviewer hQmlr - 2/2

Q1 (t1t_1 sensitivity). Tables 3-4 compare two horizons on the banana dataset: the wall-clock time of RLA, and t1=1000t_1=1000 (about 7 times the compute).

DiMSNLLECEOOD AUROCTrain lossDecrement δ\deltaCPU [s]/sample
RLA wall-clock0.341 ± 0.0041.44 ± 0.140.933 ± 0.0090.1260.016918.3
t1=1000t_1=10000.346 ± 0.0032.26 ± 0.210.928 ± 0.0100.1180.0012135

NLL and OOD AUROC are unchanged within standard error, the training loss settles at the level of the MAP (0.117), and ECE stays below all other methods. The samples have thus largely settled at the shorter horizon.

Q2. MAP rows are added to Table 2 (0.990 on MNIST). For the banana dataset, Table 4 reports training loss, gradient norm and Gauss-Newton decrement at the final samples; the gradient norms are small but not exactly zero at these horizons, and the decrement shows DiMS ends considerably closer to a minimum than RLA and LA + GD. For MNIST your reading is correct: the dynamics are defined on the 1,000-sample subset (Q3), so the samples minimize the subset loss rather than the full training loss, which explains 0.950 against the MAP.

Q3. The image experiments use the loss with the prior, whose precision is tuned by the marginal log-likelihood (Sec. 5), defined on the 1,000-sample subset (App. D.3). The low-rank Hessian approximation has rank 20.

Q5. DiMS in Table 1 is run at the wall-clock time of RLA (18.3 vs. 16.3 CPU s/sample), so the comparison above is at equal cost.

We would like to thank again for the constructive review, especially the suggestion about the friction floor, ηmin⁡\eta_{\min}. We hope our responses address your points and are glad to follow up on any of them.

I would like to thank the authors for their reply. I'm covered by their response and thus I will maintain my acceptance score. Glad that my recommendation help you improve the paper.

We thank the reviewer for the positive assessment and are glad our response addressed the questions. We appreciate the support and hope it will guide the final decision.

Reviewer_Ry39OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DiMS, a deterministic dynamical system that generates neural-network parameters lying on the minimum-loss set. The loss surface is treated as a graph manifold with the pullback metric G(θ)=I+∇L(θ)∇L(θ)⊤G(\theta) = I + \nabla L(\theta) \nabla L(\theta)^\top. A particle moves on this manifold with kinetic energy T=12α˙⊤G(α)α˙T = \tfrac{1}{2}\dot\alpha^\top G(\alpha)\dot\alpha, potential κL\kappa L, and a friction force −η(t)α˙-\eta(t)\dot\alpha with η(t)=η0T\eta(t) = \eta_0\sqrt{T} (Eqs. 8-9). Each sample is the end point of a trajectory that starts at the MAP estimate with an initial velocity drawn from the Laplace approximation N(0,H−1)\mathcal{N}(0, H^{-1}). Lemma 1 shows that the total energy is non-increasing. Theorem 1 uses LaSalle's invariance principle to show convergence to stationary points, and to local minima under a strict saddle assumption. Theorem 2 bounds the first time the kinetic energy falls below ϵ\epsilon, and a lower bound on the arc length of the path is given in Theorem 3. Experiments are reported on the banana data, the Snelson regression data, and a LeNet with about 44k parameters trained on MNIST and Fashion-MNIST. DiMS is compared with the Laplace approximation (LA), linearized LA and Riemannian LA (RLA), and it has the highest OOD AUROC in Table 1 (for example 0.971 against 0.939 for RLA on MNIST vs. Fashion-MNIST).

Critical strengths
  1. The motivation is concrete. RLA geodesics leave the low-loss set and need an arbitrary stop time. Adding a potential and a friction term removes both problems, as Figures 1 and 4 show.

  2. The derivations in Appendices A-C are self-contained. The closed-form Christoffel symbols of the graph metric let the dynamics be evaluated with Hessian-vector products, and code is provided.

  3. In Figure 8 the accuracy of DiMS stays near 99% on MNIST as the stochastic subnetwork grows, while RLA falls below 85%.

Critical weaknesses
  1. DiMS has no target distribution. A sample is the image of N(0,H−1)\mathcal{N}(0, H^{-1}) under the flow map, and the paper does not say which distribution on S∗(D)S^*(D) this approximates or how η0\eta_0 and q(v~)q(\tilde v) change it. In practice, DiMS is a randomized optimizer started at the MAP. The ablations that would separate the geometry from the random restart are missing, namely Euclidean heavy ball (G=IG = I) with the same friction, and a Laplace sample followed by gradient descent to convergence.

  2. The guarantees are weaker than the text says. Theorem 1 gives convergence to local minima, and Eq. (4) defines S∗(D)S^*(D) as the set of all local minima, so nothing ensures that a sample reaches the global minimum level set. The claims that DiMS is "guaranteed to sample exactly from the minimum level sets" (abstract) and that all samples interpolate the training data (Figure 2) are therefore not supported. A compact sublevel set is also required by Theorem 1, and Remark 1 obtains it from a Gaussian prior. However, the banana experiment and Appendix A.3 drop the prior. On the other hand, with the prior the loss is no longer invariant to rescaling symmetries, so its minima are not the invariant sets the paper targets. Theorem 2 bounds the first time T<ϵT < \epsilon, which can occur at an uphill turning point, so it does not show convergence "in finite time" (Conclusion). For a particle that starts at a global minimum and stops at the same loss level, Theorem 3 reduces to Length(γ)≥1/(2 η0)\mathrm{Length}(\gamma) \ge 1/(\sqrt{2}\,\eta_0), independent of the initial velocity.

  3. The baselines look misconfigured. In Table 1, LA reaches 0.564 accuracy on MNIST and 0.147 on Fashion-MNIST, yet in Figure 8 LA with a diagonal Hessian keeps about 97% on MNIST at full network size. MAP accuracy is not reported, but Figure 8 suggests about 99% on MNIST, so DiMS (0.950) also loses four points, which does not fit samples on the minimum-loss set. In the banana experiment DiMS uses a different loss (no prior) and velocity distribution from RLA, and on Snelson it runs to t1=50t_1 = 50 while RLA stops at t1=1t_1 = 1. MAP, deep ensembles (Lakshminarayanan et al., 2017), MC dropout (Gal & Ghahramani, 2016) and the samplers the paper cites (Roy et al., 2024; Fadel et al., 2025; Reichlin et al., 2025) are not compared.

  4. Scale and cost. The largest model has 44k parameters, the image results use 3 repetitions, and no run time is reported. It is also unclear whether the loss in Eq. (8) uses all MNIST training points or the 1,000-point subset of Appendix D.3. In the latter case the samples are minima of a different loss.

  5. Minor issues:

  • The caption of Table 1 says "ID and OOD AUROC", but the top block reports accuracy, NLL, Brier, ECE and MCE.

  • The proof of Theorem 3 refers to "the convergence properties in Theorem 3" before Eq. (C.15); this should be Theorem 1. The normal vector in Eq. (B.18) is not normalized.

  • The minima are not isolated, so convergence of α(t)\alpha(t) to a single point in Theorem 1 needs a Łojasiewicz-type argument (e.g. Haraux & Jendoubi, 1998) in addition to LaSalle's principle.

  • The caption of Table 2 says standard errors are reported, but the table has none. RLA also has lower NLL and Brier score than DiMS there (0.312 vs. 0.355 and 0.0967 vs. 0.1040).

  • There are typos such as "precoditioning", "Sherman-Morisson", "regulartization", "the the".

Questions for authors
  1. Which distribution do DiMS samples approximate, and how do η0\eta_0, κ\kappa and q(v~)q(\tilde v) affect it?

  2. Can you add the ablations in point 1, i.e. G=IG = I, constant friction, and a Laplace sample followed by gradient descent?

  3. Please report the training loss and gradient norm at the returned samples. What fraction reaches the global minimum level set rather than a worse local minimum?

  4. Why do LA and linearized LA reach 0.147 and 0.385 accuracy on Fashion-MNIST in Table 1? What is the rank of the low-rank Hessian, and what is the MAP accuracy?

  5. Which loss defines the dynamics in the MNIST experiments (full training set, the 1,000-point subset or minibatches, with or without the prior)?

  6. What is the wall-clock cost per sample of DiMS compared with RLA and LA, and how many solver steps does a trajectory take?

10 comments
AuthorsOpenReview ↗

Response to reviewer Ry39 - 1/2

Thank you for the detailed and precise review, that shaped much of the revised manuscript (changes are marked in blue in the PDF).

W2 (guarantees). These points were correct and we changed the claims accordingly.

  • Local vs. global. The abstract now reads "converges to the union of minimum level sets", i.e. S∗(D)S_*(\mathcal D) of Eq. 4 (local minima) and we clarified that the interpolation phrasing only relates to the example of Fig. 2.
  • Finite time. The old Theorem 2 and all finite-time claims are removed; small kinetic energy at an uphill turning point is not convergence. We provided a replacement based on the following idea: a look-ahead energy E=H+δ0⟨α˙,∇θL⟩E=H+\delta_0\langle\dot\alpha,\nabla_\theta\mathcal L\rangle that still decreases at turning points, to bound the total time spent outside ∥α˙∥2<ϵ, ∥∇θL∥2<δ\\{\lVert\dot\alpha\rVert^2<\epsilon,\ \lVert\nabla_\theta\mathcal L\rVert^2<\delta\\}.
  • Length bound. The friction floor ηmin⁡\eta_{\min} ensures that damping does not vanish as the particle slows down; this recovers heavy-ball friction in the Euclidean limit, lowers the guaranteed length only by a bounded factor (Corollary 2), and is what guarantees that the stopping tolerance is reached in bounded time.
  • Compactness vs. invariance. Remark 1 now obtains coercivity without requiring a Gaussian prior, via a term that is flat on a ball and grows outside it, thus better preserving the reparameterization invariant solutions within this ball. It is inactive along paths inside the ball, where the dynamics coincide with the no-prior ones of the banana experiment.

W1 / Q1. We agree that the samples are the push-forward of q(v~)q(\tilde v) under the dynamics, not from the Bayesian posterior, and the limitations now say so. What we can quantify: κ\kappa and qq bound the loss along the path by L(θ∗)+∥v~∥2/(2κ)\mathcal L(\theta^\ast)+\lVert\tilde v\rVert^2/(2\kappa), and by L(θ∗)+trΣ/(2κ)\mathcal L(\theta^\ast)+\mathrm{tr}\Sigma/(2\kappa) in expectation (Corollary 1), while η0\eta_0 sets the guaranteed exploration, of order 1/η01/\eta_0, and a lower bound of the same order on the time to realize it (Corollary 2).

Q2 (ablations). Added on the banana dataset (Table 1 and App. E, using 5 seeds, S=50S=50, same MAP and initial velocities).

MethodAcc.NLLECEOOD AUROCTrain lossCPU [s]
LA + GD88.10.3234.270.9270.2022.0
HB (G=IG=I, const. friction)87.30.3323.590.8420.1501.7
RHB (const. friction)86.60.3554.730.7970.1354.4
E-DiMS (G=IG=I)87.30.3452.340.9360.12116.2
DiMS87.30.3411.440.9330.12618.3
RLA87.50.3254.380.8150.18316.3

LA+GD is a strong, cheap baseline that obtains similar accuracy, NLL and OOD AUROC as DiMS, but worse calibration and a higher final training loss. Note that for LA+GD it is important to tune the learning rate and schedule; the other settings we tested either diverge or do not improve performance, which is a clear limitation of this approach. The state-dependent dissipation drives OOD detection, and the Riemannian metric improves calibration (ECE 2.34 to 1.44). Table 1 also adds MAP, a deep ensemble and kernel diffusion (Roy et al.), run as formulated in their paper, and its iterates drift off the minimum submanifold, governed by the curvature along the trajectory (Fig. 10).

Q3. Table 4: at t1=1000t_1=1000 the training loss is 0.118 ± 0.000 against 0.117 for the MAP. Gradient norms are small but not zero at these horizons, as they scale with sharpness. We also report the Gauss-Newton decrement (0.0012 for DiMS, 0.113 for RLA, 2.8 for LA + GD). S∗(D)S_*(\mathcal D) contains all local minima, so we do not separate out global ones.

Q4. The rank is 20, and MAP rows are added to Table 2. All methods share the same Σ\mathbf{\Sigma}. Outside the 20 retained directions the variance equals the prior variance, which is large as the prior precision tuned by the marginal log-likelihood is small. LA adds such a sample to the MAP and leaves the low-loss region. LinLA linearizes the network, but the same samples can inflate the variance of its logits. The diagonal approximation of Fig. 8 gives every parameter its own curvature, so LA samples stay close to the MAP, and with the full Hessian LA already degrades at 40% of the weights. RLA and DiMS use these samples only as initial velocities and corrects for the geometry through the dynamics, which use exact Hessian-vector products.

AuthorsOpenReview ↗

Response to reviewer Ry39 - 2/2

Q5. The 1,000-sample subset (App. D.3), with the tuned prior. Your inference is correct: the samples minimize the subset loss, which explains 0.950 against 0.990 for the MAP.

Q6 / W3-W4. CPU time per sample is above, with DiMS run at the wall-clock time of RLA, whose t1=1t_1=1 is the definition of the exponential map rather than a budget. RLA is unstable on the no-prior loss (App. D.2), so it keeps its own loss as it was originally formulated. Larger models and further image baselines are future work; cost at scale is a stated limitation.

W5. Fixed. We now claim only convergence to the set (Theorem 1, Eq. 11); a Łojasiewicz argument for point convergence, as in Haraux & Jendoubi, is an interesting idea for future work.

We believe the claims now match what is proven and that the ablations separate geometry, friction and restart as you asked. We would be grateful if you would reconsider your assessment.

The main concerns left are experiments.

I just provide some observations. The MAP rows that were added to Table 2 reveal that DiMS is worse than the MAP on all in-distribution metrics on both image datasets.

The ablations are done only for the banana dataset, and there LA+GD is competitive to DiMS.

On the baselines, the explanation for LA at 0.147 accuracy on Fashion-MNIST (rank 20, with very large prior variance in all other directions) confirms that this configuration of LA is degenerate.

The revision fixes some overclaiming in the theory.

Reviewer_SiyjOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper considers the task of sampling from a manifold of minima for exploring the solution space of the empirical loss. It presents a method with a physical interpretation involving a moving particle in a gravitational field with friction. The authors claim some theoretical guarantees for this method. They test their method in out-of-distribution detection, showing improved performance over existing methods.

Critical strengths
  • A coherent and interpretable sampling mechanism
  • A useful analytical structure underlying the method that enables theoretical study of this method
  • Promising empirical results
Critical weaknesses
  • It seems there is a problem with Theorem 1. Consider the smooth, coercive loss L(x)=x2/2L(x)=x^2/2, set κ=1\kappa=1, and choose the strictly positive damping η(t)=e−t\eta(t)=e^{-t} for t≥0t\geq 0. The loss-graph metric is G(x)=1+x2G(x)=1+x^2, so the total energy is
H(x,v)=T(x,v)+V(x)=12(1+x2)v2+12x2,v=x˙.H(x,v)=T(x,v)+V(x) = \frac{1}{2}(1+x^2)v^2+\frac{1}{2}x^2, \qquad v=\dot{x}.

Choose any initial state with H0:=H(x(0),v(0))>0H_0:=H(x(0),v(0))>0. The energy sublevel set

Ω=(x,v):H(x,v)≤H0\Omega=\\{(x,v):H(x,v)\leq H_0\\}

is compact, since it is closed and H(x,v)≥(x2+v2)/2H(x,v)\geq (x^2+v^2)/2. Along the dynamics in Eq. (8), it is easy to show that the energy identity gives

H˙(t)=−2e−tT(t)≥−2e−tH(t),\dot H(t)=-2e^{-t}T(t)\geq -2e^{-t}H(t),

where the inequality follows from 0≤T(t)≤H(t)0\leq T(t)\leq H(t). Multiplying by an integrating factor yields

ddt[e,2(1−e−t)H(t)]≥0.\frac{d}{dt} \left[e^{\\,2(1-e^{-t})}H(t)\right]\geq 0.

Consequently,

H(t)≥H0e−2(1−e−t)≥H0e−2>0for all t≥0.H(t)\geq H_0 e^{-2(1-e^{-t})} \geq H_0e^{-2}>0 \qquad\text{for all }t\geq 0.

The unique stationaty point is (x,v)=(0,0)(x,v)=(0,0), where H=0H=0. Thus no trajectory with H0>0H_0>0 can converge to that point, despite satisfying the positivity and compactness assumptions of Theorem~1. The issue is that strictly positive damping can have finite total integral: ∫0∞η(t),dt=const\int_0^\infty\eta(t)\\,dt=const.

  • There is some overclaiming. For example, Theorem 2 bounds the first attainment of small kinetic energy; it does not establish the paper’s claimed finite-time convergence to the submanifold of minima. So the theoretical guarantees are not as presented in the paper.

  • The experiments do not present the computational cost of the proposed method against other methods.

Questions for authors

Please respond to my list of weaknesses.

3 comments
AuthorsOpenReview ↗

Response to reviewer Siyj - 1/1

Thank you for the careful reading, and in particular for the explicit counterexample, which highlighted an issue in how Theorem 1 was stated. We address your three points below and mark changes in blue in the revised PDF.

W1 (Theorem 1). You are right. The submitted theorem only assumed a dissipation function η(t)>0\eta(t)>0, which admits η(t)=e−t\eta(t)=e^{-t}, and for that choice convergence fails exactly as you show. The hypothesis was wrong, however the sampler was not affected as it in practice was mainly a notational issue. The dissipation function of DiMS (Eq. 9) has always depended on time only through the state, η(α,α˙)\eta(\alpha,\dot\alpha), so the system is autonomous and your example is excluded. The revised Theorem 1 states this explicitly, and the proof verifies each hypothesis of LaSalle's invariance principle:

  1. Autonomous and well-posed. The right-hand side of Eq. 8 has no explicit time dependence and is locally Lipschitz (App. C, Preliminaries), so solutions are unique.
  2. Compact invariant set. H˙=−2ηT≤0\dot H=-2\eta T\le 0 (Lemma 1) and a coercive VV make the sublevel set Ω={H≤Ht0}\Omega=\{H\le H_{t_0}\} compact and positively invariant.
  3. LaSalle (Khalil & Grizzle, Thm. 4.4). Trajectories approach the largest invariant subset of {H˙=0}\{\dot H=0\}. Since η>0\eta>0 whenever α˙≠0\dot\alpha\neq 0, this is {α˙=0}\{\dot\alpha=0\}, and remaining there forces α¨=−κ grad L=0\ddot\alpha=-\kappa\,\mathrm{grad}\,\mathcal L=0.

Step 1 is what your example violates: a friction that decays independently of the motion can run out before the particle has come to rest, whereas state-dependent dissipation removes energy for as long as the particle moves. We found the example instructive enough to include it as Remark 2, as the explanation of why the dissipation must be state-dependent.

W2 (overclaiming). Agreed, and we modified these claims:

  • "Guaranteed to sample exactly from the minimum level sets" (abstract) is now "converges to the union of minimum level sets".
  • All finite-time convergence statements are gone (contributions, Sec. 3, conclusion), and the old Theorem 2, which only bounded the first time the kinetic energy becomes small, is removed. We included an alternative theorem in the revised manuscript (new Theorem 2) based on the following idea: the energy HH alone cannot bound the time to come to rest, since H˙=−2ηT\dot H=-2\eta T vanishes at an uphill turning point however steep the slope is. We therefore consider a look-ahead energy E=H+δ0⟨α˙,∇θL⟩E=H+\delta_0\langle\dot\alpha,\nabla_\theta\mathcal L\rangle, i.e. HH plus the rate at which potential energy is about to change, which keeps decreasing at a turning point. We obtain a bound of the form C/min⁡(ϵ,δ)C/\min(\epsilon,\delta) on the total time the trajectory spends outside ∥α˙∥2<ϵ, ∥∇θL∥2<δ\\{\lVert\dot\alpha\rVert^2<\epsilon,\ \lVert\nabla_\theta\mathcal L\rVert^2<\delta\\}, i.e. with a small velocity and a small gradient.
  • The paper now states what the theory does give (Sec. 3.1, "What does the theory tell us?"): convergence to stationary points of the loss (Theorem 1), low-loss confinement along the entire sampling path (Corollary 1, new), and a guaranteed traveled distance together with a lower bound on its cost (Proposition 1 and Corollary 2, instead of previous Theorem 3), as well as bounded time to reach a state satisfying a stopping criterion based on the velocity and gradient norms (new Theorem 2).

W3 (computational cost). Table 1 now reports CPU seconds per sample on the banana dataset (5 seeds, S=50S=50), together with new baselines:

MethodECE ↓OOD AUROC ↑CPU [s]/sample
LA + GD4.270.9272.0
RLA4.380.81516.3
KD (Roy et al.)20.710.55022.9
DE4.450.88846.0
DiMS1.440.93318.3

DiMS is run at the wall-clock time of RLA, the closest geometry-aware method, and is cheaper per sample than a deep ensemble. It is more expensive than gradient descent from LA samples (LA + GD), which is a strong baseline on accuracy and NLL; what the additional computational cost buys is calibration. The O(SNK)\mathcal O(SNK) cost remains a stated limitation (Sec. 6).

Since your soundness score rested on Theorem 1, we hope the corrected statement and proof resolve that concern, and we would be grateful if you would reconsider your rating. We are happy to clarify any step.

Dear Reviewer Siyj, thank you again for your review. Your concern about Theorem 1 has been addressed: it was a relatively easy fix and the theorem is now stated for the state-dependent dissipation that has always been part of the definition of DiMS, under which your counterexample does not apply, and the proof verifies each hypothesis of LaSalle’s principle (Appendix C). The sampler itself was unaffected.

With the discussion period ending soon, we would be grateful if you could confirm whether this resolves the concern and, if so, reconsider the rating. We are happy to clarify any remaining point.

AuthorsOpenReview ↗

Final note from the authors before the discussion ends

Dear Reviewer Siyj,

As the author-reviewer discussion period is about to close and we have not received any response after our rebuttal and follow-up comment, we would like to leave a final note. We believe your concerns have been addressed in the revised paper, and we would be grateful if you could take this into account when the reviewers and the area chair discuss the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.