acceptodds
Under review as a conference paper at ICLR 2027

Efficient Message Passing for Partial Differential Equation Priors

Abstract

Prior information for real-world physical quantities is most elegantly expressed via partial differential equations (PDEs). In this paper, we propose a novel way to solve PDEs using probabilistic inference on a factor graph. In general, factor graphs provide a natural way to encode prior knowledge into a model as explicit factors; here, this knowledge is provided by a governing PDE, which narrows the solution space, while observed data further shape the posterior over the parameters. The approximate parameter posterior is inferred using message passing based on moment matching, without posterior sampling or global gradient-based optimization. We demonstrate our approach on the first-order advection and the second-order semi-linear Fisher-KPP equations, where it achieves predictive accuracy comparable to a standard baseline while providing structured predictive uncertainty. Moreover, the inferred posterior marginal means and uncertainty structure match more closely those obtained using Hamiltonian Monte Carlo than the evaluated mean-field variational inference baseline, while requiring up to 10× less training time in our experiments, with inference speed comparable to variational inference.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
direct message approximation, factor graphs, approximate message passing, probabilistic inference, uncertainty quantification, PDE, physics-informed learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5QYJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tries to train a PINN from a Bayesian inference perspective. It proposes to use a factor graph to represent this problem, and runs a message-passing iteration to conduct sampling. The proposed method outperforms conventional Bayesian PINN methods in efficiency compared to HMC, and outperforms in accuracy compared to variational inference.

Critical strengths
  1. The paper proposes a well-motivated application of message passing to approximate posterior inference in B-PINNs. It is natural to represent the observation model and PDE constraints as two coupled subgraphs, and the moment matching + local linearization enable tractable (Gaussian-approximated) message updates. The experiments also suggest potential benefits of the proposed method.
Critical weaknesses
  1. The empirical gain can be solidified by covering more baselines that use accelerated MC/VI techniques in Bayesian inference. For example, existing works have used maximum a posteriori (MAP) + local second-order Gaussian approximation [1] or tailored MCMC sampling [2] to improve the trade-off between sampling efficiency and accuracy in Bayesian PINNs. Additionally, a more direct SGD baseline would be to use SGD for MAP estimation in the same Bayesian PINN rather than using SGD to train a deterministic PINN.

[1] MacKinlay et al., Gaussian Ensemble Belief Propagation for Efficient Inference in High-Dimensional, Black-box Systems, ICLR 2025. [2] Lin et al., Multi-variance replica exchange SGMCMC for inverse and forward Bayesian physics-informed neural network, Journal of Computational Physics 2022.

Questions for authors
  1. Can you clarify the main sources of the computational benefits of message passing over well-optimized MC or VI methods? How does the proposed method compare to more advanced B-PINN methods?

Curious question:

  1. When the problem setting is amortized inference (e.g., the model needs to condition on PDE boundary condition / paratemers) across a family of PDEs, conditional PINNs can directly have additional inputs. For Bayesian PINNs, this amortization does not seem straightforward. Particularly, can your methods be made amortized? Seems that learning a GNN on the factor graph in this case is a natural solution.
1 comment

Thank you for taking the time to review our work! We appreciate your assessments of our submissions strengths as well as the questions you raised.

Any tables mentioned below are in our general comment.

Baselines: MAP + local Gaussian [1], SGD-MAP, tailored MCMC [2]. We added the MAP of the same Bayesian PINN (Adam + Gauss–Newton, your SGD-MAP) and Laplace at that MAP on all 130 runs (Table 1). Laplace matches HMC, so we claim no edge over it when the MAP is good. At ρ = 10 the MAP search ends on a worse mode in 13 of 20 runs, and 30 random restarts per run do not fix this. Message passing from the prior reaches HMC's mode in all 20 (Table 2; 19 with the code fixes in our reply to F6Fs). On the Gaussian initial condition at ρ = 10 the reverse holds (Table 5). We did not run GEnBP [1], which estimates Gaussian messages from ensembles of a black-box model, or replica exchange [2]; we discuss both. We see our approach's value where a good MAP is hard to find, as at ρ = 10.

Q1. Sources of the computational benefit. We need no MAP search, which HMC and Laplace start from, and we stop after at most 300 closed-form sweeps instead of up to 30,000 Adam steps plus Gauss–Newton. Table 2 gives the measured ratios: HMC at the smallest budget that passes its convergence gate takes 2.6–3.1× our time, Laplace 1.0–1.4×.

Amortization. Not covered here. With a fixed basis and collocation set, the factor graph is the same for every PDE parameter or boundary condition; only the factor potentials change. A GNN that takes those parameters as inputs could then learn the updates. We add it as future work. We think the fixed factor graph makes this a natural next step.

Reviewer_F6FsOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes an interesting perspective on Bayesian PINNs, where the PDE acts as a prior on physical knowledge and the PINN is a factor graph, whose variables and messages are approximated by Gaussian distributions. The authors consider a shallow Bayesian "neural network" with fixed Fourier features and Gaussian-distributed weights, and derive approximate Gaussian messages using moment matching + a linearization.

The experiments show that the method achieves performance competitive with a deterministic PINN on linear advection and Fisher-KPP equations. The model can match posterior marginals obtained via HMC at much lower cost.

Critical strengths
  • The paper presents a novel perspective on Bayesian PINN, and the method achieves posterior inference comparable to HMC at a fraction of the cost, which is quite remarkable.
Critical weaknesses
  • Insufficient evaluation: among the three baselines considered in the paper (deterministic PINN, HMC, VI), VI is clearly collapsed / not converged (as stated in Appendix A.7.2). This makes the comparison harder to interpret, and it's particularly consequential given that VI is the most closely related method. The authors should explain why the method fails to converge, or try a broader hyperparameter search.
  • The "Bayesian NN" is really just a linear layer on top of Fourier features, which is quite far from what one would use in practice.

Errors

GPT Astra found the following mistakes, and I verified both:

  1. A wrong application of Stein's lemma in the forward message for the advection equation in A.5.1, due to the discontinuity of the activation function.

First, by the Gaussian conditionals

\mu_{W_k} + \frac{\operatorname{Cov}(W_k,Z)}{\sigma^2}(Z-\mu),$$ hence $$\operatorname{Cov}(W_k,g'(Z)) =\frac{\operatorname{Cov}(W_k,Z)}{\sigma^2} E[(Z-\mu)g'(Z)].$$ Let $p(z) = N(z; \mu,\sigma^2)$. Integrating by parts $E_{p}[(Z-\mu) g'(Z)]$ we get $$\sigma^2 E[g''(Z)] + \sigma^2[ g'(0^+) + g'(0^-)] p(0).$$ Because activation is so that $g'(0^+)=\beta$, $g'(0^-) = \alpha\beta$, $\operatorname{Cov}(W_k,g'(Z))$ should be $$\phi_k(x) \sigma_k^2 [E[g''(Z)] + \beta (1-\alpha) p_Z(0) ] .$$ The term \$\beta (1-\alpha) p_Z(0) \$ is instead absent from the forward message. A similar omission also appears in $E[c(t,s)]$ in page 20. 2) The message passing as written in Algorithm 1 is not properly Bayesian: the observation-conditioned belief in Equation (6) is used to update the backward message, and then it is multiplied again by the incoming weight belief (see line 16 of Algorithm 1). This is essentially equivalent to multiplying by the prior twice. ### Minor - In the problem formulation the authors claim to focus on linear advection equations, but right after they say "We derive the required factor messages for the first-order linear advection equation and the second-order semi-linear Fisher-KPP". Can this be clarified?
Questions for authors
  • Are α,β\alpha,\beta fixed or learned during training?
  • Are the errors identified above only typos in the text or were the wrong equations used in the experiments?
1 comment

Answer

We thank Reviewer F6Fs for their careful assessment and for acknowledging the novelty of our work, while raising concerns about the limited evaluation.

To briefly address the main concerns, we would like to make the following clarifications:

1) Evaluation and VI baseline. Our goal is to demonstrate the feasibility of message passing for Bayesian PINNs, rather than outperform all existing baselines. We evaluate multiple trajectories across two PDEs under identical experimental settings (Table 1). We acknowledge that the paper's VI had not converged. A broader search (32 settings, chosen by VI's own ELBO on a held-out trajectory) lowers its error, but it still converges in only 6 of 130 runs (Table 1). Our method matches HMC's error (Table 1) in about a third of HMC's time and under a quarter of tuned VI's (Table 2). Laplace at the MAP also matches HMC (Table 1), so we claim no edge over it.

2) Network architecture. We intentionally use a shallow architecture to isolate the inference method and ensure a fair comparison. Appendix A.7.1 also investigates wider networks with up to four neurons and demonstrates that our approach and HMC share similar accuracy and the uncertainty patterns, while VI degrades for high wave speeds. Extending the approach to deeper architectures is outside of the scope of this paper and remains future work.

3) Stein's lemma. We thank the reviewer for pointing out the issue. The term was missing in both the derivation and the code. We reran all 130 runs with the term, together with a second fix (the PDE factors now read cavities instead of marginals). The median rel-L2 changes by under 0.1 % and the widths are unchanged; one Fisher-KPP run at ρ = 10 then ends on a worse mode.

4) Bayesian message passing. Our algorithm follows the sum-product algorithm defined in Equation (7), which avoids double counting incoming information by definition. We agree that Algorithm 1 and Equation (6) need clarification, but the implementation follows this principle correctly.

5) Minor clarifications. We clarify that the primary targets were first-order linear PDEs, while Fisher-KPP demonstrates its extension to a second-order semilinear PDE. The parameters of α and β were fixed during training, and their choice was justified by approximating the ReLU behaviour.

Reviewer_wGMBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The goal of this paper is to construct a probabilistic forward solver of Partial Differential Equations (PDE) problems using Bayesian inference on a factor graph. The architecture is very simple: a nonlinear output unit is applied to a linear combination of Fourier basis elements; the weights are assumed independent Gaussians. The PDE discussed are time dependent with one spatial dimension. Data is considered at t=0 (initial condition) and at collocation points where a PDE residual is trained. The variational inferences method used is sum-product message passing with Direct moment approximation, log-normal activation moments, Stein's lemma, and a randomized factor schedule. The method is tested on two PDEs, 1D advection-diffusion equation and 1D Fischer--KPP equation.

The method is compared with a standard Physics Informed Neural Network (PINN) trained using the standard ADAM optimizer, mean-field Variational Inference (VI), and Hamiltonian Monte Carlo (HMC) reference. The authors report

  • faster training than HMC/VI
  • accuracy comparable with PINN and HMC
  • uncertainty closer to HMC than VI
Critical strengths
  1. The deterministic PDE constraints are encoded as Dirac factors with moment-matched messages. This is a novel contribution.
  2. The residual-factor interface is modular.
  3. The experiments are carefully and fairly crafted. The bases and seeds are shared across methods, double precision is used, and limitations are clearly acknowledged.
  4. The posterior means are very similar to those from HMC. This shows that the approximation is reasonable for the situations discussed, though it is not clear how significant it is.
Critical weaknesses
  1. There is no reason to expect an edge over standard probabilistic methods and there is no indication that it is in fact the case. Removing the PDE gives a model of Bayesian regression with fixed Fourier features. In that case, linear regression, GP regression, Laplace approximation are all mature and reliable methods; the paper gives no evidence that message passing would lead to any improvement in that case. I see no reason why incorporating a PDE would change the situation here.
  2. In the case of the advection-difusion equation, there are other state-of-the-art GP methods which can produce even physics constrained approximations, such as [Harkonen, M., Lange-Hegermann, M. and Raita, B., 2023. Gaussian process priors for systems of linear partial differential equations with constant coefficients.]. It would be interesting to see how the method compares with those results, which were proved clearly superior to standard PINNs (which are on par with the contribution under review).
  3. The results for Fisher--KPP are quite underwhelming; see, in particular, the 4th column of Figure 12. I wonder what is the relevance of uncertainty quantification when the approximations are visibly so far from exact solutions. Moreover, it would be relevant to compare this solver with a classic numerical method. Those are the (deterministic) workhorse for such problems and would work very very well for this low dimensional problem. Designing fair benchmarking between probabilistic and deterministic methods is is a challenge, but it is certainly an important point to cover.
  4. The comparison with PINN is not convincing either. PINNs are known to be difficult to train and unstable, but the results presented by the authors are only slightly better, if at all. It would be interesting to compare the training times as well, as I am sure PINNs will take much longer to train.
  5. The speed in training the current method is also a bit misleading. The method is very fast because no hidden layer/features are trained. This certainly limits the expressivity of the network. It would be interesting to see how the method would compare to the case where the Fourier basis has variable frequencies.
  6. The PDE class on which the method is tested is very narrow. In fact, PDEs in 1+1 dimensions are often easy to solve with most methods. It is completely unclear to which other PDE this method could apply to or if results would be good in higher dimensions; for instance, Laplace's equation in 3D with Dirichlet boundary conditions is also a simple test case, but it displays a completely different behaviour than the equations the authors study.
Questions for authors

Reject: the paper suggests a new method for inference but (i) does not give clear evidence that this method would improve upon standard GP/Laplace methods, with or without PDE; (ii) the test cases are narrow and arguably easy, yet the Fischer--KPP example does not work very well; (iii) the model does not allow for learning features and scalability is unclear; (iv) neither accuracy nor uncertainty quantification are convincing.

  1. Without any PDE, for 2D data (x,y) with y\approx f(x), would the present approach improved on Bayesian linear regression/GP regression/Laplace approximation?
  2. For the linear equation, can you compare with the method from the reference above?
  3. For the 1D Fisher--KPP, can you compare with standard numerical methods?
  4. Does it make sense to include trainable features in a hidden layer, f(x)=g(W^T\phi(Vx+b))?
1 comment

Thank you for the time and effort you took to review our work!

Any tables mentioned below are in our general comment.

Q1. Edge over Bayesian linear regression, GP or Laplace? No. Without the PDE, Bayesian linear regression and GPs are exact on fixed features, and Laplace is the standard for our output nonlinearity; we expect no gain. With the PDE, Laplace at the MAP matches HMC (Table 1), and Bayesian linear regression comes close to our model on 3D Laplace (Table 6). The one difference we measured: at ρ = 10 the MAP search ends on a worse mode in 13 of 20 runs and we do not (Table 2). On the Gaussian initial condition at ρ = 10 the reverse holds (Table 5). We will claim no general edge over Laplace. We still value that it needs no MAP search, which is what fails at ρ = 10. We also think the BP direction is promising as it better supports distributed computation and non-linear factors. The effects of this advantage are not shown here, but a general consideration.

Q2. EPGP. On our advection runs, EPGP is 10× more accurate, better calibrated, and fits in 0.04 s (Table 3). For linear constant-coefficient PDEs it is the better method. It does not cover semi-linear PDEs such as Fisher-KPP. Our approach also covers semi-linear PDEs, where it matches HMC's error (Table 1).

Fisher-KPP, Fig. 12, column 4. That case (Gaussian initial condition, ρ = 10) is our worst: rel-L2 0.4224 and 0.6 % coverage, so our uncertainty is not useful there. HMC on the same model reaches 0.0149 (Table 5), so the failure is our inference, not the model. On the 80 PDEBench Fisher-KPP runs, it does match HMC's error (Table 1).

Q3. Classical solvers. A pseudo-spectral solver started from our 512 initial-condition samples is far more accurate and faster (Table 4). For a forward problem with a known initial condition it is the better tool. What our approach adds is a posterior over the solution, which a classical solver does not give.

PINN comparison and training times. Our PINN is the same fixed-feature model trained on the PINN loss. On advection it matches ours and HMC (median rel-L2 0.068–0.082 per wave speed). On Fisher-KPP its RMSE is 0.137 against our 0.027. Training times are similar: 7–773 s against our 5–496 s on advection, and 24–28 s against 17–23 s on Fisher-KPP. At similar cost, ours also returns uncertainty, which the PINN does not.

Variable frequencies. This is not covered here. All methods here share one fixed basis. We add this as a limitation.

Narrow PDE class; 3D Laplace. We ran your example (Table 6). Every method is limited in accuracy and overconfident, ours more so (widths 0.69–0.90 of HMC's). At 716 weights we take 151–339 s against Laplace's 6–8 s. Each new PDE needs its own residual factor. We add this as a limitation. Our means there are still as accurate as HMC's. See Q1 for our perspective on Laplace.

Q4. Trainable hidden layer. For accuracy, yes: with one hidden layer of 8 units, Laplace is about 60× more accurate on 3D Laplace. For our inference, no: our closed-form messages do not extend past one layer, and a linearised variant started from random weights converged in none of 9 runs. Started at the MAP, it reproduces Laplace. Our approach is aimed at single-layer models for now, as a proof of concept.

Uncertainty not convincing. We agree, however HMC on the same model also undercovers (27 % and 68 % at 90 %, Table 1), so much of this comes from the model. Our means still match HMC's error in about a third of HMC's time (Tables 1 and 2).

Comments on the paper

1 comment

We thank all three reviewers for their careful assessment of our work and their constructive feedback. Their comments helped us find errors, add baselines and state the limits of our approach more clearly. We will apply this feedback in future versions of the paper, including the corrected derivations, the new baselines and the stated limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.