acceptodds
Withdrawn from ICLR 2027

Generative Priors Conditioned on Natural Language for Bayesian Inversion in PDEs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Inferring quantities of interest (QoI) from data is a central task in Science and Engineering. In such contexts, we often have access to both quantitative data and qualitative data. Quantitative data may be represented by noisy sensor measurements, simulation data, re-analysis data; qualitative data may be in the form of text descriptions of experimental setups, expected experiment outcomes, and human-perceived system behaviours. The task we address in this paper is the following. Given a training set of paired qualitative text and quantitative QoI data, we learn to exploit the inherent correlation between the two modalities to learn a highly informative data-driven natural-language-conditional Bayesian prior, such that when presented with a new physical system, we can coherently combine (i) the training dataset, (ii) qualitative text describing the new system, and (iii) a small number of noisy sensor readings from that new system, to perform inference and uncertainty quantification (UQ) over the QoI. To achieve this task, we develop two parallel approaches, one uses conditional diffusion and the other conditional autoencoders, and compare both against classical Bayesian methodology, unconditional generative models and deterministic supervised methods. Each approach has specific strengths and tradeoffs; conditional autoencoder offers theoretical tractability, allows for fast posterior sampling, and provides better-calibrated UQ, whereas conditional diffusion is explored for greater expressiveness and capturing complex posteriors with irregular QoI fields. The approach is tested on the steady-state heat equation, damped Helmholtz equation, and UK weather reanalysis data.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian Inference, Generative Models, Physics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_54UEOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the problem of learning generative Bayesian priors over physical field conditioned on text. Two approaches are considered for the learning and are compared on differing physical datasets. Incorporation of text into the priors is observed to improve the reconstruction accuracy.

Critical strengths
  1. The paper soundly implements two standard approaches: i) autoenconder and ii) score-based diffusion models.
  2. Rigorous testing and reporting of inference on three differing datasets.
Critical weaknesses
  1. The theorems, corollary, and lemma, are not new results and do constitute a theoretical advancement. Rather these are standard results applied to a block-triangular map.
  2. It it not clear what the real novelty is here. Introducing text at inference time is not a new idea and has been applied in other datasets as the authors note (e.g. imaging).
Questions for authors
  1. Theorems 2.3 and 2.3 bound the joint W_2, but it seems the claims of the paper should really be about the conditionals. A small joint error does not necessarily imply a small conditional error. What statements and bounds can be made on the conditional error?
  2. Does the application of this problem of learning text-conditioned Bayesian priors require differing considerations when applied to physical fields, or can it be applied to other datasets? Another way of saying this is, what is special about the physical fields for the methodology described here.
  3. On a related note, it seems that PDEs are used a proxy for physical systems, and these terms are seemingly used interchangeably in the text. What are the considerations for non-PDE physical systems?
2 comments

We thank the reviewer for their time and for the questions raised, and we will use some of the points to improve our work in future revisions.

Q1: We thank the reviewer for raising this question, and we will clarify this point in future revisions to our work.

Q2: This framework can be applied to other fields with paired data and text. However, our focus is on inverse problems in physical systems, which differ from inverse problems in imaging: observations are often a handful of point sensors rather than dense measurements, and calibrated UQ is essential for downstream decision-making.

Q3: Our method does not use the governing equations, and the prior is learned purely from data, so it applies to any physical system, including systems described by ODEs and PDEs, or no known governing equations at all (eg: reanalysis data in UK weather experiment).

Thank you for the response to the raised questions. Unfortunately, the technical weakness and question have not been addressed by the authors. I will keep the score as is.

Reviewer_xwmSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The question tackled in this paper is that of reconstructing a physical field from a few noisy sensor readings together with a natural-language expert description. A text-conditional generative prior is learned from field-text pairs of data by using a sentence encoder that is fined tuned for embedding similarity to track field similarity. Two reconstruction algorithms are built, conditional autoencoder and conditional diffusion model. Both are tested on the steady-state heat equation (a.k.a. Laplace's equation), a damped Helmholtz equation, and UK weather reanalysis data. The authors chose as baselines: GPs, supervised regression, and unconditional versions of the same generative models. They report several metrics. The authors claim that text conditioning consistently improves accuracy and that the autoencoder improves efficiency and calibration of posteriors.

Critical strengths
  1. The problem under consideration is a very difficult and interesting challenge: incorporating expert knowledge on top of sparse measurements with proper uncertainty quantification. The stated main goal of the paper is to improve upon the fact that the state-of-the-art can only cover few scalar parameters.
  2. The authors report several metrics: coverage, calibration curves, NMSE, CPRS, which shows the depth of their effort.
  3. Benchmarking against the unconditional variants of the algorithms highlights the effect of incorporating the text data.
  4. The reporting is honest, admitting that supervised baselines are superior on the UK weather data.
  5. The theoretical results are elegant and their proofs seem correct, though the relevance of the results for the experiments is not so clear.
Critical weaknesses
  1. The biggest weakness by far is that the aim is to incorporate nuanced human knowledge in describing physical systems when too few measurements are available. This requires the systems to have a certain complexity and noisiness, which is simply not true of either PDE example. This is best seen for the Poisson example: The sample space is 4 dimensional (one dimension for the constant value on each edge of the square). In fact, calculating the true functions which serve as data for the learned operator can be done as follows: Calculate the solution of \Delta u=0 with u=0 on 3 edges of the square [0,1]^2 and u=1 on the fourth edge. Rotate the independent variables of the solution by 90, 180, 270 degrees. The four functions thus obtained serve as a basis for the space on which training should be done. Because of this, 4 exact measurements at distinct points are enough to determine the solution. Any more than 4 measurements, allowing for some noise, can be used to perform linear regression/least squares, which will give excellent results. In fact, if I remember correctly, even the authors admit that the results with or without text are similar for their first example. This example is thus way too simple to share any light on whether the method is good or not.
  2. The next biggest issue in my opinion is that the authors have the stated goal to increase the number of scalar parameters that can be included. For instance for Helmholtz the allowed parameters are damping, forcing magnitude, position, each having 3-4 range values; a total of 10 words are meant to encode the expert knowledge. It feels to me that many more characteristics and nuances should be allowed for the incorporation of the text to have a notable impact alongside the sparse measurements.
  3. The theoretical contributions are interesting but there are a few points worth mentioning. First, they clarify why the construction is sound and particularly the estimate in Theorem 2.2 can be viewed as a theoretical guarantee. One issue with it though is that it does not seem that the error on the right-hand-side can easily be driven arbitrarily close to zero (see e.g. Table 1). This also reduces the impact of Theorem 2.1 which holds only at exactly zero training error. Two more minor comments: (i) Corollary 2.4 requires compact support, which is not satisfied by Gaussian priors; this is of course no problem in practice as negative squared exponentials become really small really fast, but since this covers the theoretical results, it is worth pointing out. Maybe the statement works for distributions which decay rapidly? (ii) The Lipschitz constant of D is nowhere defined, at least I could not find the definition.
  4. The experimental metrics are not really conclusive. Perhaps if the authors apply their method on a bigger model (substantially more expressivity for the text, substantially larger dimensional training data), then they can show clearly what the impact of encoding the text is.
Questions for authors

The paper tackles a new and well-motivated problem, the work is technically sound, theoretically supported, and careful about UQ evaluation and empirical metrics. The main reason for suggesting rejection is that the claim is not clearly substantiated by the experiments, as explained in the Weaknesses section. It is also the case that the technical novelty is moderate, combining a sentence encoder, a score model, a pushforward posterior and DPS. My score can rise if the authors show robustness to granular or noisy/degraded text and add oracle baselines.

  1. How does performance degrade when text can be noisy or wrong? Is calibration preserved then?
  2. How do all results compare against an oracle baseline?
  3. Can the authors construct an example in which the data set is high dimensional and the text can be allowed to be an order of magnitude more granular? (compare with Weaknesses 1 and 2 above)
  4. Can the model be trained to achieve really small NMSE? This would connect really well with the theoretical results.
2 comments

We thank the reviewer for the thorough review, and we will use some of the points to improve our work in future revisions.

Weakness 1: We agree that the heat equation experiment is low-dimensional. However, our setting assumes no prior knowledge of the PDE and only learns generative priors from the data. In Table 5, the gain from text at nobs=2,4,6n_{obs} = 2,4,6 is noticeable (eg: mean NMSE of 9.759.75 vs 3.14×10−33.14 \times 10 ^{-3} for Uncon AE and Cond AE, and 7.367.36 vs 3.11×10−33.11 \times 10 ^{-3} for Uncon Diffusion and Cond Diffusion at nobs=6n_{obs}=6).

Weakness 2: We believe there is some misunderstanding regarding our stated goal. The reviewer pointed out that “the main goal of the paper is to improve upon the fact that the state-of-the-art can only cover few scalar parameters”. However, the reference to “a few scalar parameters” in our introduction is related to existing prior elicitation methods, which outputs hyperparameters of a fixed parametric prior (eg: the length scale or variance in a GP prior) and therefore cannot represent complex spatial structure. Our contribution is to learn a generative prior over the entire field. This is independent of how many physical parameters that can define a single system. Even when a description pertains to only a few physical parameters, its effect on the prior over the field is highly non-trivial, and this is precisely what our method learns. Regarding the nuances of texts, we agree that Helmholtz descriptions encode only a few facts, but our UK weather experiments provide an example of higher dimensional settings, where text still improves inference accuracy (Tables 3, 12 and Figure 27).

Weakness 3: Table 1 reports NMSE of the posterior mean, which will remain non-zero even with an exact prior, because the inverse problem is ill-posed. This is not the loss in Eq5 and the right-hand side of Theorem 2.2. The reconstruction loss during training is small, which is of order 10−410^{-4} on heat and 10−310^{-3} on Helmholtz and UK weather. (i) We agree that Gaussian distributions do not have compact support, this corollary is to qualitatively connect MMD used in training with the Wasserstein distance in Theorem 2.2. (ii) ∣∣D∣∣Lip||D||_{Lip} is defined as the Lipschitz constant of DD on Z×T\mathcal{Z} \times \mathcal{T} with respect to the product norm.

Weakness 4: Among our experiments, UK weather is the closest to a larger-scale setting, with real reanalysis fields and more variable descriptions.

Thank you for the response and the clarifications, in particular on the “few scalar parameters” and on the training loss.

The main points of my review are not addressed: there is no robustness study for noisy or wrong text, no oracle baseline, and no example with higher-dimensional data and more granular text. Furthermore, the question of how the bound on the joint distribution relates to the conditional posterior is still open.

I will keep my score. I wish the authors all the best with a revised submission of this work.

Reviewer_NcquOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper uses natural-language generative priors for Bayesian inverse problems in PDEs. The authors finetune a sentence transformer and train two parallel priors (a conditional AE with NUTS and a conditional diffusion model with DPS) to reconstruct physical fields and quantify uncertainty from sparse observations. Experiments cover the steady-state heat equation, damped Helmholtz equation, and UK weather reanalysis data.

Critical strengths

Strength 1: Incorporating text descriptions into ill-posed physical inverse problems is a practical and interesting way to search for good reconstruction solutions with sparse sensors.

Strength 2: The authors test their approach on multiple physical systems and provide clean comparisons against unconditional ablated variants and standard baselines.

Critical weaknesses

Weakness 1: Although not comprehensively discussed, prior works have already explored semantic priors for physical inverse problems, as well as system-conditioned generative priors for Bayesian inversion (e.g., Semantic Regularization of Electromagnetic Inverse Problems).

Weakness 2: Empirical results do not convincingly justify the proposed method, as detailed in questions.

I am really open to raise my score if the authors can address my concerns.

Questions for authors

Q1. Beyond replacing physical or geometric parameters with a text embedding, what is fundamentally new here compared to GABI or CoCoGen? Please clarify what problems or values are introduced by using natural language.

Q2. How does your Conditional Diffusion differ from taking an existing text-conditioned PDE diffusion model (like Text2PDE) and running DPS on it? If they are essentially the same, could this weaken your novelty? If not, what is the key difference, and can you provide a Text2PDE + DPS baseline?

Q3. Why do we need both AE and Diffusion? AE calibrates better, samples much faster, and matches Diffusion in accuracy almost everywhere and even better. You claim Diffusion is better suited for irregular fields, but I didn't find it enough to serve as a second method. Seems Diffusion is not necessary here.

Q4. The contribution of text as condition to the UK weather task seems to be marginal. The improvement over the unconditional baseline drops sharply compared to the two synthetic PDEs. Since natural language conditioning is the core of the paper, how much value does text actually add?

Q5. In Table 3, Conditional Supervised is just as accurate as or even better than your proposed generative models, yet runs much faster. Why would a practitioner use your heavy pipeline in practice? Only for UQ?

Q6. Why does conditioning on text sometimes improve NMSE while degrading posterior coverage? If UQ is a core selling point, shouldn't a better-informed prior help calibration rather than making the posterior overconfident?

Q7. How does Theorem 2.2 connect to actual posterior inference? It bounds the Wasserstein distance on the joint prior distribution Pu,τP_{u,\tau}, whereas we care about the conditional posterior given text and observations Pu∣y,τP_{u\vert{}y,\tau}. What guarantee does this bound actually provide for the inverse solution?

Q8. You may want to trim down your five contributions in Intro. Some read just like implementation steps rather than distinct contributions.

2 comments

We thank the reviewer for the detailed and constructive questions, and we will use some of the points to improve our work in future revisions.

Weakness 1: We thank the reviewer for pointing us to “Semantic Regularization of Electromagnetic Inverse Problems”, which we will cite in our paper. This method belongs to the same class as the supervised baseline in our paper, which learns a mapping directly from measurements to deterministic reconstructions, so it provides no UQ and is tied to the observation process seen during training. This will be discussed further in our response to Q5 about limitations of supervised methods.

Q1: GABI and CoCoGen condition on exact parameters (geometry and physical parameters), whereas we target settings where such parameters are unavailable, and only vague descriptions in natural language are there to provide system knowledge. Our contribution is learning a generative prior conditioned on this imprecise information and combining it with sparse noisy measurements for posterior inference.

Q2: We agree that our Cond Diffusion is architecturally close to Text2PDE + DPS, and we do not claim novelty for this component, our contribution is the whole pipeline for Bayesian inversion.

Q3: Diffusion achieves lower NMSE than AE on weather experiment (Tables 3, 12, 13), and lower quartiles of NMSE on heat equation at all nobsn_{obs} and on Helmholtz for nobs≥8n_{obs} \ge 8 (Tables 6 and 10), showing its greater expressiveness, whereas AE offers better calibration and faster sampling. Keeping both shows that the pipeline is suitable to the two most common classes of generative models.

Q4: We agree that the gain from text on UK weather experiment is smaller, which is expected since the text there carries least information, where each description covers only 8 of 18 regions and reports coarse regional averages of temperature and wind speed. The gain is more visible from Figures 27 and 28, showing that the gain is greatest when nobsn_{obs} is small and is concentrated inside the regions the text describes, indicating that the model uses information the text provides.

Q5: Beyond UQ, the drawback of supervised methods is that the supervised map is tied to the observation process it was trained on. Generative priors are plug-and-play methods: once trained, they can be combined with any observation operator (eg: different number of sensors, noice levels) at inference time without retraining. An ablation study was provided in Table 14 in the Appendix, where in our weather experiments, temperature and wind are observed at different locations, which is a layout that was not seen during training of the supervised model. In this case, both conditional generative models outperform the supervised baselines (NMSE 4.414.41 and 4.204.20 vs 5.11×10−25.11 \times 10^{-2} for Cond Supervised).

Q6: Posterior coverage alone does not fully determine the quality of UQ, an overly wide posterior can achieve high coverage while being uninformative. From Figures 1 and 3, it can be seen that unconditional priors tend to produce diffuse posterior with high standard deviations. We report CRPS in Tables 5, 9 and 12, which is a proper scoring rule that jointly penalises mis-calibration and over-dispersion, where conditional models constantly achieve lower CRPS than unconditional models across all experiments.

Q7: We thank the reviewer for raising this question, and we will clarify this point in future revisions to our work.

Reviewer_NcquOpenReview ↗

Official Comment by Reviewer Ncqu

Thanks for the response. However, my main concerns remain unresolved, so I will keep my original score. I wish the authors all the best.

Comments on the paper

1 comment
Area_Chair_HNk8OpenReview ↗

Please engage in the discussion

Reviewers: if you have not replied to the authors' response, please read it and engage in the discussion. Thanks!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.