acceptodds
Withdrawn from ICLR 2027

Bayesian Deep Equilibrium Models with Sequential Inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Deep Equilibrium Models (DEQs) have drawn considerable attention due to their unique advantages. However, their uncertainty estimation, crucial for prediction-sensitive applications, remains unexplored. In this paper, we propose Bayesian Deep Equilibrium Models to address this gap for the first time. Our study highlights the substantial computational cost associated with uncertainty estimation in Bayesian DEQs. To mitigate this challenge, we introduce a novel sequential inference approach that captures the similarities in the parameters and reduces computational redundancy in the inference, offering a promising method to accelerate uncertainty quantification in DEQs. We also provide theoretical justification for the motivation behind our approach. Comprehensive experiments on Darcy Flow, Navier-Strokes equations, and computer vision datasets demonstrate that our method can speed up uncertainty estimation with Bayesian DEQs by up to 3 times without any sacrifice in performance.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Uncertainty Estimation, Bayesian Nerual Networks, Deep Equilibrium Models, Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_m6EDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tries to solve the high computation cost problem of bayesian inference of DEQs. The paper propose a sequential inference method by combining similar successive parameter samples to reduce computation cost. Experiments have shown up to 3x speedup while maintaining comparable predictive accuracy.

Critical strengths
  1. This paper studies a practical computation bottleneck problem of uncertainty estimation for DEQs.
  2. The paper proposed method appears intuitively simple and effective as a quick add on to existing bayesian methods.
Critical weaknesses
  1. The novelty is limited. The proposed method appears incremental that directly builds on existing bayesian approximations and simply use sampling and warm starts to speed up, with no theoretical guarantee.
  2. Limited empirical evidence. The experiment only evaluate on SWAG and SGLD, I wonder if their method would expand to more methods. Furthermore, the task are generally toy regression problems that are too simple. This limited coverage is insufficient evidence for claimed generality and scalability.
Questions for authors

Please try to address my weakness.

Reviewer_yM6ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors propose a Bayesian Deep Equilibrium (DEQ) model for uncertainty estimation. A naive implementation of Bayesian DEQs involves additional computational overhead compared to conventional Bayesian neural networks, since each posterior weight sample requires solving a fixed-point problem through multiple numerical iterations to obtain the equilibrium representation. To address this issue, the authors exploit the observation that consecutive parameter samples obtained through MCMC tend to be close to each other. Based on this observation, they propose a sequential inference approach in which the equilibrium representation obtained from one posterior sample is used to initialize the fixed-point solver for the next sample, instead of restarting from zero. This reduces the number of numerical iterations required to approximate the equilibrium. Through empirical experiments, the authors demonstrate that their approach achieves comparable predictive performance and uncertainty estimation to standard Bayesian DEQs while reducing computational cost. They also provide a theoretical justification, under certain smoothness and parameter-proximity assumptions, showing that nearby parameter samples produce nearby equilibrium representations.

Critical strengths

The solution presented in the paper is interesting. It addresses a major bottleneck while using a naive bayesian DEQ model.

Critical weaknesses
  1. In Figure 3(a), the authors compute the distance between the initial weight values and the weight values over subsequent epochs. However, they conclude that nearby weight samples are close to each other. Could the authors please explain what exactly they want to convey through this figure? Perhaps a plot showing the distance between weights from consecutive epochs, rather than comparing them with the initial weights, would be more appropriate.

  2. It is not clear what Figure 3(b) intends to convey. I could not find a detailed explanation in the main text. The authors should provide a clearer description of the figure, including how the distances are computed and what the distributions represent.

  3. I understand that the primary motivation of this work is to reduce the computational cost of Bayesian DEQs. However, I believe the experiments are insufficient to establish whether Bayesian DEQs provide reliable uncertainty estimates compared to conventional Bayesian models. It would be meaningful to compare against baselines such as conventional Bayesian neural networks or neural networks using Laplace approximation, rather than only their DEQ counterparts.

  4. Most of the experiments comparing the proposed method with other Bayesian baselines are presented in the appendix. I suggest moving some of the important comparisons to the main paper to better demonstrate the effectiveness of the proposed approach.

  5. In the OOD detection experiments, the authors report only the predictive entropy of OOD samples. I believe this is insufficient to evaluate OOD detection performance. The authors should also report the entropy values of in-distribution (ID) samples and include standard OOD detection metrics such as AUROC and FPR@95.

Questions for authors

Please see the weaknesses section.

Reviewer_CunfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper applies standard approximate Bayesian inference (mainly SGLD and SWAG; MC-Dropout, ensembles, HMC and BBB in the appendix) to Deep Equilibrium Models, and argues that the main obstacle is cost, since every posterior sample needs its own fixed-point solve. The proposed fix, "sequential inference", has two parts: order the posterior samples so that consecutive parameters are close (for SGLD, the chain order; for SWAG, a Langevin-style chain on the Gaussian noise, Eq. 6), and warm-start the solver for sample ii from the fixed point of sample i−1i−1. Theorem 1 bounds the distance between consecutive fixed points via the implicit function theorem. Experiments show that at a fixed small per-sample NFE the warm-started solver is far more accurate than zero initialisation, and the paper reports up to 3× wall-clock speed-up "without any sacrifice in performance".

Critical strengths
  1. The question is relevant. DEQs are increasingly used as scientific surrogates (FNO-DEQ), where calibrated uncertainty matters, and the cost of sample-based inference for implicit models is a real bottleneck.
  2. The idea is simple and plug-and-play: no retraining, negligible extra memory, and it works with any sample-based posterior approximation, which the authors show across six of them.
  3. The empirical signal is consistent. At fixed per-sample NFE, warm-starting is much better than zero initialisation on every task and the instance-level alignment check (Fig. 7c) is a sensible sanity test.
  4. The experiments are broad (PDE regression, classification up to ImageNet, ablations over burn-in, sampling frequency, dropout rate and step size), and the PDE tables give error bars over 5 runs.
Critical weaknesses
  1. The SWAG sampler does not do what the paper says it does, and that matters for the main claim. The idea in Sec. 4.1 is to replace independent Gaussian draws by a chain in which each draw is a small perturbation of the previous one, so that consecutive parameters stay close and the warm start works. The chain in Eq. (6) is ξt+1=(1−τ)ξt+τη\xi_{t+1} = (1−\tau)\xi_t + \sqrt{\tau}\eta. Iterating this does not produce standard normal samples: the variance settles at 1/(2−τ)1/(2−\tau), which for small τ\tau is about one half. So unless something corrects it, the method is sampling from a SWAG posterior whose covariance has been cut in half. The text says a Metropolis–Hastings step removes the discretisation bias, but Algorithm 1 has no accept/reject step and no acceptance rates are reported anywhere, so I cannot tell whether the correction was actually used in the experiments. (It is also unnecessary: for a Gaussian target, ξt+1=ρξt+(1−ρ2)η\xi_{t+1} = \rho\xi_t + \sqrt{(1−\rho^2)}\eta keeps the standard normal exactly, for any ρ\rho, with no correction at all.)

  2. The bigger issue is what the chain does to the ensemble. Remark 1 says that making consecutive samples similar is "different from autocorrelation". I do not believe it is. With τ=0.01\tau = 0.01, consecutive samples have correlation 0.99, which means the 30 samples used in the experiments are worth roughly 0.15 independent samples, nowhere near the 0.86⋅n0.86\cdot n that App. L reports. That number cannot be right for this chain. So the "SI" ensemble is far less diverse than ordinary SWAG, and part of the speed-up simply comes from evaluating an ensemble that barely moves. Table 11, where larger τ\tau gives worse calibration at NFE=1NFE = 1, is exactly what that trade-off would look like.

  3. The core idea is a standard warm start, and the paper does not say so. Starting an iterative solver from the solution of a nearby problem is what continuation methods do, and it is routine inside MCMC for PDE-constrained inverse problems. Within the DEQ literature specifically, DEQ-Flow (Bai et al., 2022) already reuses the previous frame's equilibrium, and the neural DEQ solvers of Bai, Koltun & Kolter learn an initialiser to cut the same cost. The paper cites DEQ-Flow only as an application and never positions "sequential computation" against any of this. .

  4. Theorem 1 says consecutive fixed points are close when consecutive parameters are close, which is the implicit function theorem with an unnecessary d\sqrt{d} factor. It says nothing about how many solver iterations a warm start saves, which is the quantity the paper is actually about.

Questions for authors
  1. Was the Metropolis–Hastings correction actually used in the SWAG experiments, and with what acceptance rates? Is τ\sqrt{\tau} in Eq. (6) a typo for 2τ\sqrt{2\tau}? Why not use the exact AR(1)/pCN update?
  2. How do you reconcile neff/n=0.86n_eff/n = 0.86 (App. L) with a lag-1 correlation of 0.990.99 at τ=0.01\tau = 0.01? What is the effective sample size of the 30 SWAG samples actually used, and how does their predictive variance compare with i.i.d. SWAG draws?
  3. What exactly is the "warm-up strategy" of App. I, and is its cost included in the NFE and timing plots?
  4. At matched total compute, how does SI compare against initialising every sample from the fixed point of θSWAG\theta_{SWAG} and evaluating the samples in parallel?

Comments on the paper

1 comment
Area_Chair_4R2WOpenReview ↗

Please engage in the discussion

Authors: you have not responded to any review. Please reply to the comments and engage in the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.