acceptodds
Under review as a conference paper at ICLR 2027

From Learning Operators to Inferring Operators: ORIA for Bayesian Reasoning in Function Space

13 views

Abstract

Scientific inference requires reasoning about unknown systems from limited observations and using uncertainty to guide further discovery. We introduce ORIA, a continuum framework for Bayesian operator inference in function space. Trained under priors over operators, ORIA conditions on a few input–output function pairs to produce joint posterior predictive distributions for unseen systems, without test-time optimization. Its continuum architecture supports prediction across discretizations, while its joint predictive distribution captures uncertainty and dependencies across entire response fields. This enables uncertainty-aware prediction of global field-structure-dependent physical quantities and guides the selection of informative experiments. Comparisons with exact Bayesian references assess posterior fidelity, and experiments on diverse physical systems demonstrate up to an order of magnitude lower relative error than baselines. Posterior-guided measurement selection reduces average prediction error by 13–18% relative to random selection. We further characterize how reliability depends on the training prior and how prior revision can improve transfer. ORIA paves a promising path toward efficient foundation models for scientific computing that know what they do not know.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Operator learning, Nerual operator, in-context learning, uncertainty quantification, scientific machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_b2GROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ORIA, an in-context model for inferring an unknown operator from a few input–output function pairs. Given these demonstrations and a new input function, ORIA predicts the response as a joint Gaussian over many locations, in a single forward pass and without any test-time training. It uses quadrature-weighted attention so that the same model works across different grids and resolutions. ORIA is trained on synthetic priors over operators, following the prior-fitted network idea, so that its output approximates the Bayesian posterior predictive under that prior. The experiments have four parts. The first compares ORIA with the exact posterior on a linear-Gaussian operator family and shows that good pointwise calibration can hide errors in the joint distribution. The second tests few-shot prediction on three finite-element (FEM) problem families and on public Navier–Stokes (NS) data, against neural-process, FNO-based and classical baselines. The third uses ORIA's uncertainty to choose which measurements to take next. The fourth studies what happens when the test data drift away from the training prior, including a diagnostic for detecting the drift and an attempt to fix it by revising the prior.

Critical strengths
  • The paper's strongest point is about joint uncertainty. In the exact-Bayes experiment, ORIA models with covariance rank 8, 9 and 16 all reach about 91% pointwise coverage. Yet their joint KL to the true posterior differs by about two orders of magnitude (Figure 1e). The FEM experiments show why this matters in practice. If you keep ORIA's predictions but drop the off-diagonal covariance entries, coverage of spatial averages falls from about 0.90 to about 0.63, while pointwise coverage stays the same (Table 10). This is a clean, convincing demonstration. Anyone who reports only pointwise coverage for function-valued predictions should take note.
  • Another strength is how openly the authors report their results. The appendices spell out which cohorts are paired and which are not. They also say the public NS block was reused while revising the prior, and they report results that hurt their own story. Examples are the 71% coverage on NS and the intervals that get narrower under an unseen nonlinearity. This made the paper much easier to judge fairly, and I think the joint-uncertainty finding alone is worth publishing. My concerns are about how far the rest of the evidence supports the paper's larger claims.
Critical weaknesses
  • My main concern is that the "Bayesian operator inference" claim has only been checked in the easiest possible case. The only comparison against an exact posterior uses a linear-Gaussian prior with nine modes. In that case the true posterior predictive is itself a low-rank Gaussian, which is exactly the shape ORIA's output head produces. For every other experiment, the true posterior is non-Gaussian, and the paper never checks how close ORIA gets to it. The fidelity results therefore can't be extended to the physical systems.
  • Even in the easy case, the model stops contracting its posterior beyond the number of demonstrations it was trained on. Trained with up to 8 demonstrations, its KL at 16 demonstrations is about 50 times its KL at 1, and its intervals are about 35% wider than the exact ones (Tables 1 and 2). Appendix F suggests a likely reason. The inter-function attention normalizes over the demonstrations, so copying every demonstration leaves the prediction unchanged. A true posterior would contract if those copies were new, independent measurements. The architecture therefore can't represent the Bayesian posterior in general. The paper mentions this but doesn't connect it to the contraction failure or try to fix it. On the technical side, the method mostly combines existing parts: PFN-style training, continuum attention and the low-rank Gaussian output of TNP-ND and related models. That is fine, but it means the evidence has to carry the paper, and the Bayesian part of that evidence is thin.
  • The accuracy comparisons are weaker than the headline numbers suggest. The "order of magnitude" gains mostly come from baselines that never see the training prior, such as per-episode FNOs fit to 8 pairs, spectral fitting and the context mean. That shows the prior helps, which isn't surprising. The baselines that do use the prior look under-tuned. Adapted TNP-ND gets worse on cooling as the number of demonstrations goes from 4 to 8 (0.32 to 0.46). It also barely improves on the cavity problem, where its error is still around 0.81 at 8 demonstrations. Pretrained FNO with fine-tuning gets worse with more demonstrations on conduction (0.73 to 1.29). Behavior like this usually points to a bug or a poorly tuned recipe, not a fair reference point. TNP-ND also got only four 10k-step pilot runs to select its configuration.
  • Several baselines were evaluated on separate archived cohorts with a different ORIA checkpoint, some with only 12 to 21 episodes. The closest methods are also missing entirely. ICON, GenICON and Zebra are discussed in related work, and GenICON produces probabilistic predictions too, but none of them appears in the experiments.
  • The NS test is narrower than it looks as well. All episodes come from one fixed viscosity and forcing, so this tests transfer to a single operator, not inference of unseen operators. At one demonstration, FNO (0.300) and Poseidon-B (0.313) both beat ORIA (0.367). The prior revision that fixes ORIA's NS coverage was tuned on the same reused block it is evaluated on.
  • The measurement-selection result doesn't use the joint covariance on the physical problems. The FEM experiments pick points by marginal variance, and against a simple diversity rule the conduction gain is 2.6%, with an interval that includes zero.
  • I'd tone down the claim that ORIA points toward models "that know what they do not know." The 71% coverage on NS and the narrower intervals under an unseen nonlinearity suggest otherwise. Running the joint selection rule on the FEM problems would make the measurement-selection section match its framing. The paper uses five separately trained models (linear-Gaussian, 1D grammar, FEM, 2D physics-mixture and NS), and the main text switches between them quickly. A small table listing each model, its prior and where it is used would help a lot.
Questions for authors
  • Can you check posterior fidelity on a nonlinear prior where a reference posterior can still be computed, for example with MCMC or importance sampling on a low-dimensional version of the 1D grammar prior? This would show whether the Bayesian claim holds when the true posterior is not Gaussian.
  • Does a fix for the replication invariance, such as a context-size feature or unnormalized aggregation over demonstrations, stop the KL from growing between 8 and 16 demonstrations?
  • Why are ICON, GenICON and Zebra missing from the experiments? GenICON in particular seems like the most direct comparison for probabilistic in-context operator prediction.
  • Can you explain why TNP-ND and the pretrained FNO get worse with more demonstrations? Could you rerun all baselines on the shared test bank with the same ORIA checkpoints and a tuning budget closer to ORIA's?
  • Can you test on data where the operator actually varies across episodes, for example NS with different viscosity or forcing? Can you also evaluate the revised priors on a block of data that hasn't been used before?
Reviewer_jqCEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed a foundation-model type transformer operator for solution operators of partial differential equations (PDEs). Using in-context learning, the model takes grids, a function query, and a prompt consisting of input-output pairs that implicitly determine the PDE, and outputs a Gaussian distribution around the approximate output associated to the PDE solution. The Gaussian uncertainty is induced from a prior distribution over PDEs and noise in the context data. The authors use the framework for a variety of problems and tasks, such as measurement selection and robustness to prior shifts. The numerical experiments further explore the quality of the uncertainty produced by the ORIA framework.

Critical strengths

Building foundation models for PDEs that accept demonstrations to identify the system operator is highly timely. The authors build on this recent idea by endowing the output space of solutions with uncertainty induced by noisy data and a prior over the forward operators/PDE. The numerical experiments are comprehensive and well-ablated.

Critical weaknesses
  • The Gaussian ansatz for the output of the ORIA is a central weakness; this is not highlighted anywhere in the abstract or introduction, which could mislead the reader. The authors should deeply explain why they choose the Gaussian joint density parametrization for the output of Φ⋆\Phi^\star, and why this is justified for PDE problems. The training loss is nice, and the mean and covariance outputs are easy to implement. Are there other reasons?

  • I find most of sec 3 hard to understand, and wonder if it is even "Bayesian". For example, what precisely is the probability measure q(⋅∣C)q(\cdot| \mathcal{C}), how is it defined? And how does the so-called Bayesian posterior predictive P⋆P^\star depend on q⋆q^\star? What spaces are they defined on? The true Bayesian posterior would be some non-Gaussian probability measure over a space of operators (since the prior is over operators). But the ORIA approximation is misspecified.

  • The definition of q⋆q^\star is especially confusing, given line 161 at the bottom. Since it belongs to P(RN)\mathcal{P}(\mathbb{R}^N), how can different discretizations N′≠NN'\neq N yield consistent laws if they cannot be compared on the same metric space?

  • I also notice a lot of AI generated writing in the paper. Also the AI use statement is missing the required parts of the disclosure under ICLR policy.

Questions for authors
  • In the numerical comparisons, what does it mean to show FNO with m>1m>1 demonstrations? FNO is not an in-context learning model, so it cannot accept context inputs like C\mathcal{C}.

  • The authors should clarify what the "measurements" are in Sec 5.3: XX, ff, or both?

  • The related work in Sec 1 and Sec 2 is repetitive, likely from an old draft that was not merged.

  • Can the method handle correlation not only across space and time for a single output function, but (since the parameter is an operator), correlation across multiple output functions? In the Gaussian case of the paper, this would involve block covariance operators with non-zero off-diagonal blocks.

  • Can you explain how ORIA can have better instance-wise performance for a fixed operator than task-specific neural operator (like FNO) trained for that operator?

  • The Task in Sec 3 is very interesting, ambitious, and timely. However, if mm is very small (like in the experiments with m=O(10)m=O(10)), why is the performance still so "good"? As the authors mentioned, from such a small context, many operators can explain the prompt. Is there some trivial low-dimensional structure in the prior? Certainly for the linear-Gaussian example, what what about beyond that?

  • A limitation is that in Eqn 1, f⋆f_\star and DmXD_m^X are queried on the same grid XX. Is this necessary? It is much more natural to allow different grids/representations between the query and the prompt.

  • What is μ\mu in Line 203-204

Reviewer_sbbWOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces ORIA, a pretrained model that uses a few input–output function pairs to infer the behaviour of an unknown physical operator without updating its weights. It predicts a joint Gaussian distribution over the output field, using a low-rank covariance to capture uncertainty correlations across locations and quadrature-weighted attention to handle different sampling grids. Experiments show strong few-shot prediction and demonstrate that accurate uncertainty at individual points does not guarantee accurate uncertainty in spatial averages. However, its uncertainty can become overconfident outside the training prior, and the external validation is limited. Its novelty lies mainly in combining in-context operator learning, prior-based training, and correlated uncertainty prediction, rather than introducing a fundamentally new inference principle.

Critical strengths

On the technical side the paper builds on top of existing work with novelty being adapting these pieces to a particular structure: multiple input–output functions share an unknown operator, and the model returns an explicit, correlated predictive distribution for a new input function across discretizations. Its alternating within-function and between-function attention is a concrete architectural design for that problem. However, the paper needs to establish why this is more than an operator-conditioned neural process with quadrature weights.

To me the strong contribution is in empirical validation; the exact-Bayes audit, the demonstration that pointwise calibration hides joint errors, and the consequences for uncertainty in spatial averages. That way the experiments have been performed rigorously.

Critical weaknesses

The main weakness is limited methodological novelty: ORIA combines established ideas without clearly isolating what is fundamentally new, and direct comparisons with closely related methods such as ICON, GenICON, and Zebra are missing. External validation is narrow, and reusing evaluation data to guide prior revisions weakens the evidence for generalization. Finally, the Gaussian, low-rank predictive distribution restricts the uncertainty it can represent, while exact Bayesian validation is limited to a favourable linear-Gaussian setting.

Questions for authors

See the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.