acceptodds
Under review as a conference paper at ICLR 2027

All roads lead to Markov kernel networks: A representation theorem for discretization-stable learning

Abstract

A central goal of operator learning is to design networks that are stable to changes in the discretization of the input function, a property sometimes called "discretization invariance". We consider a quantitative refinement of this idea called *uniform discretization stability* that provides a Hölder-stability bound controlling changes in a network's output in terms of the Wasserstein distance between input measures. Our main result is a representation theorem showing all uniformly discretization stable operators can be written exactly as a *stable Markov kernel network*—an abstract architecture that is a joint generalization of transformers, Fourier neural operators (FNO), Flowers, and other successful learned operators. Our representation theorem characterizes a design space for discretization-stable operator learning; designing new architectures becomes a question of which parts of a Markov kernel network to parameterize, and how. We further show that the read kernels of these networks can be realized as solutions of regularized variational problems, with particular choices of costs and regularizers recovering the kernels of transformers, FNO, Flowers, their hybrids, and beyond. This variational formulation provides a systematic approach to architecture design, together with sufficient conditions for uniform discretization stability.

Primary area:
learning theory
Keywords:
operator learning, discretization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RHDTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In the paper "All Roads Lead to Markov Kernel Networks: A Representation Theorem for Discretization-Stable Learning," the authors establish a representation theorem showing that uniformly discretization-stable operators can be represented exactly as stable Markov kernel networks. They further develop a variational formulation that connects several existing neural operator architectures and provides sufficient conditions for discretization stability.

The paper presents an interesting theoretical result addressing a relevant problem in operator learning. While I have not verified the proofs in detail, the mathematical approach appears reasonable. My primary concern is the practical significance of the results. The representation theorem establishes an interesting connection, but it is unclear what this characterization provides for the design of practical operator architectures. The absence of numerical experiments makes it difficult to assess the utility of the proposed framework.

Critical strengths

The paper addresses an important and timely problem in neural operator learning. The representation theorem is an interesting theoretical contribution, and the variational formulation provides a unified perspective on several existing architectures. The mathematical development appears thorough, with explicit assumptions, proofs, and counterexamples.

Critical weaknesses

The primary weakness is that the practical implications of the theoretical results remain unclear. While the theorem establishes that a representation exists, it does not establish whether that representation can be learned or implemented efficiently, or whether it provides useful guidance for designing new architectures. The paper also provides no numerical experiments, even on a simple example, to demonstrate that the proposed framework offers practical benefits.

Questions for authors
  1. What does the representation theorem provide beyond establishing the existence of a stable Markov kernel network? In particular, how does it inform the design or parameterization of practical neural operator architectures?

  2. Can the authors demonstrate the utility of the proposed framework on a simple numerical example, particularly one where the variational formulation leads to improved discretization stability or another measurable advantage over existing architectures?

1 comment

We thank the reviewer for the thoughtful comments and questions. Our general reply synthesizes our response to several inter-related concerns; (A) and (D) are particularly relevant to the points raised here.

Reviewer_mDjAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors construct a unified framework to analyze the discretization stability of . They focus on Markov kernel networks to describe the models, and show a representation theorem that guarantees all uniformly discretization stable operators can be written exactly as a stable Markov kernel network. They also investigate the connection between the Markov kernel and an optimization problem.

Critical strengths

The authors construct a unified framework with Markov kernels to analyze the discretization stability. The results give us a guideline for constructing discretization stable models. The framework enables us to analyze many kinds of recently important architectures, including transformers, neural operators, and Flowers, in a unified manner. The manuscript is mathematically well-written, and the statements are clear.

Critical weaknesses

Although the topic is very interesting and the results provide some meaningful observations, there are some concerns.

  • As the authors also mention in Section 5, the deep structure of the network is not reflected in the proof of the representation theorem. I understand that the results in this paper are fundamental theoretical results, but providing some ideas how the depth is related to the discretization stability would lead to further development of this framework.

  • The authors should clearly explain which kinds of operators satisfy the condition (16). Since the condition is measured in the L∞L^{\infty} norm of the output, this seems to be a fairly strong condition.

Questions for authors
  • In Definition 3.4, the authors impose a Hölder stability condition on the Markov kernels of each layer. As seen in the proof of Theorem 3.5, the resulting exponent of the whole network is βp∏ℓβℓ\beta_p \prod_{\ell} \beta_\ell, so it decreases as the number of layers grows unless all components are Lipschitz, and the constants accumulate multiplicatively. This makes the stability bound weaker for deep networks. Do practical models, such as transformers or FNOs, satisfy Definition 3.4, and if so, do you have any examples of how the constants and exponents behave across layers?

  • Could you provide some examples of operators that satisfy condition (16)? For instance, do standard PDE solution operators (e.g., for elliptic or heat equations) satisfy it?

1 comment

We thank the reviewer for taking the time to carefully consider our submission. We have addressed several inter-related points in our general reply. Points (C) and (B), respectively, are intended to be answers for your two questions.

Reviewer_ccibOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses discretization stability in operator learning problems. The authors define uniform discretization stability and establish guarantees for network architectures satisfying this property. They introduce an abstract class of architectures called Markov Kernel Networks, which generalizes well-known architectures such as Transformers and Flowers. The authors prove that any uniformly discretization stable operator on an admissible class of inputs admits an exact representation as a stable Markov Kernel Network. Additionally, they connect the design of network architectures to solutions of variational optimization problems.

Critical strengths
  1. Originality and mathematical sundness. The authors address the important problem of characterizing and constructing discretization stable architectures for operator learning. The paper develops interesting and apparently original theoretical results, supported by careful and rigorous mathematical arguments. In particular, I find the representation theorem and the stability results for variationally constructed Markov kernels to be meaningful contributions.
  2. Exposition quality. The manuscript is generally well written and fairly easy to follow. The authors carefully introduce the mathematical objects they work with and provide several useful examples. Although Markov Kernel Networks are defined at a fairly abstract level, the connections to practically relevant architectures, such as Transformers and Flowers, help motivate the framework and clarify its generality. I particularly appreciate Table 1, which provides an exceptionally clear and useful summary of the constructions in Proposition 4.1.
Critical weaknesses
  1. Uncertain practical significance. Although the theoretical results are interesting and meaningful, I am not fully convinced that they provide actionable insights for designing practical neural operator architectures. In particular, the constructive proof of Theorem 3.6 explicitly incorporates the target operator into the Markov kernels themselves. While this establishes the exact representation result, it remains unclear how such a construction could translate into a tractable, parameterized architecture or guide practical architectural choices. The authors acknowledge related limitations in Section 5, which I appreciate, but the practical implications of the theory remain uncertain.
Questions for authors
  1. Could the authors elaborate on how their theoretical results might inform the design of practical neural operator architectures? In particular, could the authors specify concrete ways in which the representation theorem or the variational kernel constructions could guide architectural choices or lead to improvements in accuracy, efficiency, or stability beyond existing approaches?
  2. Could the authors provide examples of practically important operator learning problems for which the stability condition in Corollary 3.8 holds, preferably with quantitative stability estimates? This would help clarify the practical applicability of the main representation theorem.
2 comments

We thank the reviewer for the thoughtful comments. We have combined our responses into the general reply above, to give a unified presentation of several closely-related points.

In the general reply, (A)--(D) are all relevant to the points raised here. For the questions in particular, (D) gives a small numerical illustration for the first question, and (B) highlights examples of operators for the second question.

Thank you for your response and for addressing my comments. I maintain my original positive score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to feedback and questions

We thank all reviewers for the thoughtful questions and comments, which have led to improvements in the paper. We have revised and expanded our preprint in an attempt to increase clarity and address the points raised in the discussion. Revised sections of the main text that we wish to highlight in particular have been colored blue in the newly uploaded preprint.

Since many of the questions and concerns are closely related, we will attempt to address them in a unified fashion here to avoid repretition across the individual responses.

(A) The practical significance of the representation theorem

We acknowledge the focus of our paper is more on the theoretical side. Nonetheless, we believe one important practical implication of the representation theorem is that it defines a restricted design space for neural network architecture satisfying an important property, namely uniform discretization stability. Human or agentic network designers can search within this space by specifying concrete forms of a small number of components defined in Definition 3.5. The representation theorem tells us that searching in this way will not a priori exclude important architectures they may wish to consider.

(B) Examples of operators satisfying the stability condition characterizing Markov kernel networks

We have included a new paragraph under Corollary 3.8 (now Corollary 3.9 in our revision), giving some examples of operators satisfying the condition. Further details including quantitative bounds are in the new Appendix F. We further clarify that the exact operator characterization in Corollary 3.9 does not limit the approximation capabilities of Markov kernel networks, since operators of this type can be used to approximate other operators that do not themselves satisfy the stability condition; appendix C contains a discussion and proof of universal approximation capabilities of Markov kernel networks.

(C) Discretization stability through network depth

It is indeed true that each additional layer with Hölder exponent βℓ<1\beta_\ell<1 reduces the exponent βp∏ℓ=0L−1βℓ\beta_p\prod_{\ell=0}^{L-1}\beta_\ell guaranteed by the proof of Theorem 3.5 (now Theorem 3.6), where βp\beta_p is the decoder exponent. This is a reason to prefer Lipschitz components for deep networks. Both softmax attention layers and Fourier neural operator layers induce W1W_1-Lipschitz updates of the hidden context. We have highlighted these estimates in blue in Appendix D.2 and Appendix D.3.1, respectively.

Genuinely Hölder estimates enter in an important way when we consider point reads, such as those in the common output decoders and the SelfWarp operators in Flowers networks. For example, the FNO decoder reads the final hidden field at a single query point. This point read has sharp exponent β=1/(dx+1)\beta=1/(d_x+1). We have highlighted this example and its proof in Appendix D.3.1. Since output decoders appear only once in a network, the multiplicatively-compounding stability deterioration is less of a concern. However, for Flowers, the theory predicts that increasing depth may indeed decrease the discretization stability.

(D) Example of Section 4 variational principles to guide network design

The variational formulation in Section 4 is a tool that can be used to design network components favoring or discouraging certain behaviors, by encoding those behaviors in the objective function to be minimized. We have added a numerical illustration of this idea in a new appendix H.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.