acceptodds
Under review as a conference paper at ICLR 2027

Universal Approximation of Measure-to-Measure Operators by Pushforwards

Abstract

Many learning tasks map an input distribution to an output distribution. A natural way to model such an operator is to transform each input sample using a continuous function that may depend on the entire input distribution, and then take the distribution of the transformed samples. This defines a measure-dependent pushforward model and includes measure-theoretic formulations of transformers. We ask when such models can approximate arbitrary continuous operators between spaces of probability measures. We first show that universal approximation fails when atomic inputs are allowed: some continuous measure-to-measure operators that split or redistribute atomic mass cannot be approximated arbitrarily well by deterministic pushforward models. We then introduce the uniform level set condition, which requires a continuous measure-dependent scalarization whose shrinking level set neighborhoods carry uniformly vanishing mass over the input family. This condition is satisfied, in particular, by compact families of absolutely continuous measures. On every compact family satisfying this condition, we prove that any continuous measure-to-measure operator with outputs of finite -th moment can be uniformly approximated, in the -Wasserstein distance, by continuous measure-dependent pushforwards. Combining our theorem with existing approximation results for measure-dependent in-context maps yields universal approximation by measure-theoretic transformers. We also extend the framework to continuously-varying source measures, yielding a corresponding universality result for a class of pushforward models that are closely aligned with cross-attention architectures.

Primary area:
learning theory
Keywords:
Measure-to-measure operators, pushforwards, universal approximation, Wasserstein distance, operator learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_JTEDOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper is discussing approximation of push-forward operators projecting one probability measure to another. In my understanding, the goal is to know, when I can approximate continuous push-forward operators.

Critical strengths
  • I am generally interested in approximations, because it clarifies under which assumptions I can expect my class of models to work. With this sense, I consider the work genuinely interesting.
Critical weaknesses
  • The paper is very hard to follow, as it is full of notation the reader needs to be familiar with. The reading is supported with examples, but the chosen examples are kind of trivial. For example Example 2.3 and 2.4 are almost the same.
  • Despite honest tries, I have not understood much the consequence for transformers and which properties the application of derived theorem tells me about their functionality.
Questions for authors
  • I do not know, if the result is kind of trivial. The approximation of measurable functions by neural network is well known. Applying this theorem to pushforward operator trivially extends to probability measures as follows:

Let f,f′:X→Rdf, f' : X \to \mathbb R^d be measurable, ν/ν′\nu / \nu' is probability measure by projecting measure μ\mu by $f / f'. $ Define coupling π\pi by joint projection of measure μ\mu with (f,f′,)(f,f',) which defined distribution on Rd×Rd\mathbb R^d\times\mathbb R^d with marginals are ν\nu and fν′.f\nu'. Then it holds that Wpp(ν,ν′)≤∫∣y−y′∣p dπ=∫∣f(x)−f′(x)∣p dμ(x)W_p^p(\nu,\nu') \le \int |y-y'|^p\,d\pi = \int |f(x)-f'(x)|^p\,d\mu(x) and the approximation bound on ff trivially propagates to error on measures.

I confess that I might not understood the main claim of the paper.

Reviewer_pAbyOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This is a theory paper that studies the approximation of an arbitrary continuous operator between spaces of probability measures. It first shows that such approximation fails in general when atomic input measures are allowed. It then introduces a mild condition on the family of input measures, termed the uniform level set condition, which excludes atoms. It establishes that under this condition, every continuous measure-to-measure operator with outputs of finite pp-th moment can be uniformly approximated in the WpW_p distance by continuous measure-dependent pushforward maps. Combined with existing universality results, this yields universal approximation of such operators by measure-theoretic transformers.

Critical strengths
  • The paper is well written and addresses a well-motivated theoretical question.
  • The results are clearly presented with helpful intuition, and the proofs are carefully developed.
Critical weaknesses
  • The cited paper of Lavenant and Savaré [LS] studies a closely related problem under different assumptions, some of its results parallel those presented here. In particular, Theorem 3.6 is comparable to Theorem 1.5 of [LS]. The present paper allows arbitrary continuous target operators by restricting the input family, whereas [LS] impose Lipschitz continuity and a non-splitting condition on the operator. The current literature review describes [LS] only in terms of the non-splitting condition and does not mention their approximation result. The paper would benefit from a direct comparison of the respective assumptions, conclusions, and limitations.
  • Proposition 2.2 is proved through counterexamples. Examples 2.3 and 2.4, and the obstructions they demonstrate, are well known in the optimal transport literature. While presenting them explicitly in the given context and deriving quantitative lower bounds are useful, I do not think these observations are major independent contributions, as suggested by (C1).
  • Some minor mistakes. For example, line 270 says “By taking ρ\rho to be the Lebesgue measure…”, but ρ\rho in Example 3.4 is defined to be a probability measure.
Questions for authors
  • Could the author add a direct comparison to Lavenant and Savaré, clarifying the differences in assumptions, conclusions, and scope?
Reviewer_WTpWOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper gives a new general condition in which the universal approximation ability of push-forward maps holds for measure-to-measure maps, widening existing work's specific conditions such as support-preserving transformations and non-splitting condition. The new condition generalizes such existing assumptions and can be stated as one simple condition called uniform level set condition. Under this condition, the non-splitting condition is naturally derived, and combining with existing universal approximation ability of transformers for continuous push-forward map, the universal approximation ability of measure-to-measure maps.

Critical strengths

Novelty of the contribution: The proposed new condition "uniform level set condition" is indeed concise but captures the essence of universal approximation ability of push-forward maps. It is instructive to point out atomic component is essential to prevent approximation ability, and that argument naturally connects to the uniform level set condition. I think this is a nice theoretical contribution.

Soundness: As far as I checked the proof, it looks correct (even if there would be a mistake, it should be minor and fixable). The proof is basically using the standard space notion. They showed an existence of push-forward to the uniform distribution on [0,1] based on uniform level set condition. And, they also showed that there is an approximated push forward map to F(μ)F(\mu) from the uniform distribution using the standard-space-like argument. This argument should be correct. I think the construction of push-forward from the uniform level set condition is elegant.

Writing: Overall, the writing is good. They describe the difference from existing work well and the theoretical contribution is clearly stated.

Critical weaknesses
  • The universal approximation ability is already known, although it is shown under more specific conditions. In that sense, showing the universal approximation ability is not very surprising. Of course, the main contribution is to provide a new useful notion of assumptions (which is informative for theoreticians), but the indicated outcome itself does not provide much new insight to the ML community.

  • Although interpreting transformers as a measure-to-measure map is a well-known perspective, this view point discards the position encoding and the order of tokens is completely neglected. In that sense, this kind of analysis does not give very precise understanding on LLMs intellectual functionality that heavily depends on the token order.

Questions for authors
  • Please look at the weakness part.
  • I am not very much convinced by the reason why the universal approximability should be provided specifically to the push-forward construction. The atomic measure issue could be resolved by once taking convolution with a Gaussian distribution (with small variance) and then apply a push-forward. The Gaussian convolution can be realized by a few layers of transformers and thus it can be easily resolved. If we take this slight modification into account, then the uniform level set condition would not be very essential. Could you let me know any strong motivation about why we need to insist on push-forward construction?

Comments on the paper

1 comment
Area_Chair_PmzGOpenReview ↗

Reminder: Follow-up Feedback Before Discussion Closes

Dear reviewers,

With the discussion period nearing its end, I would appreciate it if you could review the authors’ responses and revised manuscript and share their follow-up feedback.

Please indicate which concerns have been addressed and which remain, and update your assessment and score where appropriate.

Thank you for your time and contributions to the discussion.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.