- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces a novel operator-theoretic framework for probabilistic conditioning, formalizing the conditioning operation as a nonlinear operator that maps joint probability densities to their conditional densities or kernels. The authors establish rigorous theoretical foundations by proving continuity, quantitative Hölder stability, and universal approximation theorems using neural operators (such as Fourier Neural Operators and Transformer Neural Operators) over both compact subsets of continuous densities and unbounded domains. Empirically, the work provides a proof-of-concept demonstration on 2D Gaussian mixture models and kernel density estimation (KDE) settings, showing that trained neural operators can accurately emulate conditioning and outperform traditional plug-in numerical integration estimators.
Critical strengths
- Framing probabilistic conditioning as an infinite-dimensional mapping offers a powerful paradigm for amortizing conditional inference across diverse joint distributions without retraining.
- The theoretical analysis is exceptionally thorough, establishing local/global Lipschitz stability, maximal continuous extensions, and FullSpace Neural Operator (FullNO) universal approximation guarantees on unbounded domains.
- The empirical evaluation successfully demonstrates that FNOs and TNOs can approximate conditional kernels on synthetic Gaussian mixtures and achieve lower median errors than standard KDE plug-in baselines.
Critical weaknesses
- Theoretical stability bounds rely heavily on a strict uniform lower bound on data marginals (), with constants that blow up as , limiting robustness in realistic settings where marginals approach zero.
- The reliance on grid-based function representations ( grids in 2D) incurs the curse of dimensionality (), rendering grid discretizations intractable for higher-dimensional problems.
- The framework requires explicit evaluation access to joint probability densities, whereas modern machine learning applications almost exclusively provide empirical point clouds of samples.
- The evaluation lacks comparisons against modern amortized conditional generative models (e.g., conditional normalizing flows or conditional diffusion models), and the architecture lacks explicit enforcement of probability density constraints (non-negativity and integration to one).
Questions for authors
- How does the required training sample size and model parameter scale as the dimensions of the state space and query space increase beyond 2, given the exponential scaling cost of grid-based discretizations?
- How robust are trained neural operators when input joint densities violate the strict uniform lower bound condition (), such as near support boundaries or deterministic relationships where marginals vanish?
- How do you ensure during training and inference that predicted conditional kernels satisfy fundamental probability axioms (non-negativity and integrating to 1 for each fixed ) without explicit architectural normalization layers?
- Can the proposed operator-learning framework be extended to take empirical point clouds (samples) directly as input—thereby bypassing intermediate grid estimation—and what theoretical adjustments would be required?
4 comments
Response to Reviewer azX9 (Part 2)
Sample-based inputs and the formulation of conditioning (W3/Q4).
We agree that building an architecture, with approximation guarantees, that directly accepts samples is a key objective. However, such an extension raises fundamental mathematical questions. Indeed, if one attempts to formulate conditioning as an operator acting on probability measures, including discrete measures, endowed with the topology of weak convergence, one quickly runs into problems. Indeed, the conditioning operator is not continuous in such a space. This is discussed in Example E.3 in the Appendix, briefly recounted below: Write for the point mass at and consider the sequence
\quad , \quad P=\tfrac12\delta_{(0,0)}+\tfrac12\delta_{(1,0)}.$$ Note that $P_n \to P$ in a weak sense. However, their conditional distributions at $y=0$ satisfy $$P_n(\cdot\mid 0)=\delta_0, \quad , \quad P(\cdot\mid 0)=\tfrac12\delta_0+\tfrac12\delta_1$$ and it clear that $P_n(\cdot\mid 0) \nrightarrow P(\cdot\mid 0)$. This shows that extending our framework requires identifying appropriate distribution classes, topologies, and approximation criteria where conditioning is continuous and approximable by Neural Operators. Section 6 identifies the sample-based extension as an open direction. **Probability axiom satisfaction (Q3).** The architectures used in our experiments do not impose non-negativity and unit integral as hard constraints. Our results concern approximation of conditional densities as functions. Density estimators that are not bona fide probability densities have a long history in statistics . For example, higher-order kernel density estimators can take negative values. Tsybakov [1, Section 1.2.1, p. 10, "Positivity constraint"] explicitly discusses this issue and shows that taking the positive part cannot increase pointwise squared error. Orthogonal series density estimators provide another classical example [1, Section 1.7]. More generally, correcting estimators that are not bona fide densities has a long history; see, for example, Gajek [2]. For our $L^1$ setting, validity can be enforced by a simple post-processing step. For a fixed query $y$, let $f_y$ be an integrable raw prediction, define $f_y^+(x)=\max\\{f_y(x),0\\}$ and $Z_y=\int f_y^+(x)\,dx$, and set $$\widetilde{f}_y(x)=\frac{f_y^+(x)}{Z_y}$$ when $Z_y>0$, using a fixed probability density otherwise. For any true conditional density $\kappa_y$, $$\|\widetilde{f}_y-\kappa_y\|_{L^1} \le 2\|f_y-\kappa_y\|_{L^1}.$$ Indeed, the change caused by normalization is bounded by $\|f^+_y - \kappa_y \|_{L^1}$ and clipping cannot increase the $L^1$ error, namely $\|f^+_y - \kappa_y \|_{L^1} \leq \|f_y - \kappa_y \|_{L^1}$. Thus, exact density constraints can be enforced while preserving the $L^1$ approximation guarantee up to a factor of two. In the final manuscript we will clarify the fact that we are not enforcing hard probability distribution constraints and we will discuss ways to amend this. **References** [1] Alexandre B. Tsybakov. *Introduction to Nonparametric Estimation*. Springer, 2009. Section 1.2.1, p. 10, and Section 1.7, pp. 48–49. [DOI](https://doi.org/10.1007/b13794). [2] Lesław Gajek. "On Improving Density Estimators which are not Bona Fide Functions." *The Annals of Statistics*, 14(4):1612–1618, 1986. [DOI](https://doi.org/10.1214/aos/1176350182). **Comparisons with conditional generative models (W4).** Amortization in conditional flows and diffusion models is relevant to the broader goal of conditional inference. However, such methods typically do not provide amortization across problem instances but rather across query points $y \in E$. Indeed, in their standard fixed-joint formulation, such methods learn $y\mapsto\rho(\cdot\mid y)$ for a given data-generating joint $\rho$ which at inference can be queried at any $y \in E$, hence the amortization. For us, amortization is a stronger notion: we propose a conditional operator that maps any joint $\rho$ to the collection of its conditions, e.g. to the conditional kernel $\kappa_\rho$ that contains all conditions $\rho(\cdot \mid y) = \kappa_\rho(\cdot \mid y)$ for any $y \in E$. In other words, we amortize over both the problem instance $\rho$ and all query points $y \in E$. Therefore, a fair comparison would account for the cost of re-training a conditional flow model for every new joint $\rho$ whereas a trained Neural Operator would just condition $\rho$ with no extra training cost. An accuracy / cost trade-off would likely arise, whereby problem instance specific methods might achieve better accuracy but at a much greater compute cost. Such comparisons would be useful and will be certainly pursued in future work. We will discuss these considerations further in both the related work and open questions sections of the paper.I thank the authors for taking their time for the detailed and mathematically thoughtful responses to all four of the questions.
Given that the paper's primary contribution is a rigorous operator-theoretic foundation for amortized conditioning and that the theoretical questions have been satisfactorily addressed, I am raising my score from 2 to 3.
We thank the reviewer for their kind words and constructive feedback.
Response to Reviewer azX9
We thank the reviewer for the thoughtful and valuable feedback. Below we address their comments.
Marginal lower bounds and robustness (W1/Q2).
The dependence on the marginal lower bound reflects the stability of conditioning in the chosen error metric. In particular, uniform approximation of the entire conditional kernel requires accuracy even at arbitrarily rare queries y∈E.
The quantitative stability bounds do, indeed, deteriorate as δ→0. However, we view this as a fundamental aspect of the problem of conditioning densities in the supremum norm. Indeed, in Theorem 4.6 we extend conditioning to all joints with positive marginals without requiring a uniform lower bound, a set we name X+. Then, in Theorem 4.7 we show that this set is maximal meaning that no continuous extension of the conditioning operator exists on a larger set. As the limit δ→0 effectively exposes densities in the boundary of this maximal set X+ it is necessary that the stability estimates deteriorate with this limit. That being said, we wish to note that class of densities admitting such a uniform lower bound is considerably large. For a parametric family pθθ∈Θ of continuous densities with positive marginals mpθ>0 defined on a compact domain D×E and with compact parameter space Θ, if the parametrization θ↦pθ is also continuous then δ:=minθ∈Θminy∈Empθ(y)>0 and thus pθθ∈Θ⊆Xδ in the notation of the paper. Many standard parametric families, on compact parameter ranges bounded away from degeneracy, satisfy these assumptions.
Lastly, we point the reviewer to Figures 1 and 2: Neural Operators (NOs) exhibit low error even in regions of low probability mass, although there does appear to be a mild deterioration in performance as one approaches the boundary of the domain, see right-most plot in Figure 1. Moreover, as can be in the left-most plot of Figure 2, one actually observes a widening of the performance gap between the plug-in KDE to the Neural Operator (NO) as the uniform lower bound δ decreases. One might say that the NO relies on the joint-conditional example pairs it has seen to do inference on regions of low marginal probability 0<mρ(y)≪1 that result in few observations.
Dimensional scaling (W2/Q1).
We agree that a grid based approach used in the proof-of-concept experiments scales exponentially with dimension. However, neither the conditioning formulation nor the neural-operator architectures underlying our approximation theorems require a tensor-grid representation. Indeed, a closer look at Section 3.1 reveals that all components of the considered Neural Operator architectures are agnostic to the representation of the underlying function. For example, the lifting and activation operations act pointwise and the nonlocal operations in Definition 3.2 are integral features $ a_j(v)=\int_\Omega (T_jv(z))\cdot\psi_j(z),dz . $ In general, all of these operations could be carried out on an input function represented in terms of arbitrary bases, for example polynomials, wavelets, kernel functions or some other sparse basis expansions or low-rank representation. Ultimately, the effectiveness of such alternate function representations would depend on appropriate regularity or compressibility properties of the underlying family of probability densities. Section 6 identifies scalable function representation as an open problem. In the final manuscript, we will discuss these possibilities and the additional analysis needed to establish their efficiency.
Lastly, we wish to note that we view our work as foundational rather than algorithmic. As such, we view the experiments as demonstrating consistency with the theory we have developed and not as proposing a practical scheme for conditioning in high dimensions. We view this paper as the first step in a longer research program that aims to build amortized conditional operators from samples rather than densities. Providing such an architecture with approximation guarantees is a formidable challenge; please see our answer below for further discussion of this.