acceptodds
Under review as a conference paper at ICLR 2027

Neural Scaling Laws of Transformer Operator Network

Abstract

Transformers have emerged as powerful architectures for learning solution operators of physical systems. Empirically the prediction error has been observed to decrease when the data size and model size increase, suggesting neural scaling behavior. Yet a theoretical understanding of such scaling laws for transformer-based operator learning remains limited. In this work, we develop a theoretical framework for characterizing the approximation and generalization errors of transformer-based operator learning. Our analysis builds on a local-to-global approximation principle that is naturally aligned with the softmax attention mechanism and yields discretization-invariant output functions. On approximation theory, we derive a universal approximation error of transformer-based operator learning for H\"older-regular operators. On generalization theory, we establish a power scaling law between the prediction error and the training data size. The rate of convergence represented by the scaling exponent explicitly reflects the dimensions of the input and output domains, the regularity of the underlying functions and operators, and crucially, the intrinsic dimension of the input function class. By exploiting this intrinsic low-dimensional structure, our analysis yields a power-law generalization rate for operator learning, in contrast to the logarithmic-type power-law rates appearing in existing analyses of operator learning with feedforward neural networks. Numerical experiments validate the predicted power-law scaling and confirm that the convergence rate varies systematically with the intrinsic dimension of the input function class.

Primary area:
learning theory
Keywords:
neural scaling laws, operator learning, transformers, neural operators, approximation theory, generalization theory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wskWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors establish constructive approximation and generalization bounds for Transformer-based operator learning. A two-level softmax partition of unity localizes the input function and the output query, and a two-block Transformer realizes the resulting interpolant. Polynomial metric covering complexity of the input family yields power bounds in model size and training-function count, with explicit dependence on regularity and discretization. Burgers and KdV experiments illustrate how fitted data-scaling exponents vary between parameter families of different dimensions.

I find the direct treatment of metric input families interesting. The explicit attention realization and the accounting for quadrature, routing leakage, and weight magnitudes are useful contributions. The main proof arguments appear consistent under the sequence-length condition and fixed preprocessing used in the appendices. My principal concerns are the asymptotic scope of that condition, the precise statistical hypothesis class, and the strength of the numerical interpretation. These appear addressable through focused clarification and proof completion.

Critical strengths
  • The input-family assumption uses metric covering numbers and does not require a fixed finite linear basis or explicit latent coordinates.
  • The construction tracks dense parameter counts and polynomial weight bounds, rather than proving qualitative universal approximation alone.
  • The statistical analysis correctly distinguishes independent training functions from their output queries. The experiments use repeated runs, held-out functions, and validation-based checkpoint selection.
Critical weaknesses
  1. Asymptotic scope of the compatibility condition (Theorems 1–2; Appendix B.3).

    The theorems explicitly assume P=CUCV+1≥nx+2P=C_UC_V+1\ge n_x+2, so the following concern is about applicability. Set

    b=\frac{d_U}{\gamma}+\frac{d_2}{\beta}.$$ With $r_u\asymp\varepsilon^{1/\gamma}$, (32) gives $M_u\gtrsim\varepsilon^{-2/\gamma}\log(1/\varepsilon)$. The prescribed grid therefore satisfies $n_x\gtrsim\varepsilon^{-a}[\log(1/\varepsilon)]^{d_1/\alpha}$, whereas $P\lesssim\varepsilon^{-b}$. Compatibility eventually fails when $b\le a$, including equality because of the logarithm. For example, $d_1=d_2=d_U=\alpha=\beta=\gamma=1$ satisfies the displayed dimension inequality but gives $a=3$, $b=2$. Please explain the role of the weaker inequality $b\ge d_1/\alpha$ and give a sufficient, realizable regime in which compatibility holds along $\varepsilon\to0$, including the tolerance selected in Theorem 2. An upper covering bound alone does not ensure enough tokens. Alternatively, provide an explicit padding construction and verify its routing, aggregation, and complexity bounds. Padding may preserve the exponents, but that requires an argument. The scaling implications should retain the scope of the conditional theorems.
  2. Hypothesis class in the generalization theorem (Definition 1 and (7); Lemma 10; Appendix D.3).

    Lemma 10 fixes preprocessing, and its proof uses Z0=Z~0Z_0=\widetilde Z_0 when network parameters vary. Definition 1 and Section 3.3 do not expressly fix the affine embedding WE,bEW_E,b_E. Please state that the ERM uses the fixed preprocessing of Lemma 6, or extend the perturbation argument to trainable embeddings. The supplied proof establishes the bound for the restricted class; it should match the class in Theorem 2. This appears repairable and does not by itself invalidate the displayed statistical exponent.

  3. Specific contribution and comparison of assumptions (Sections 1 and 4).

    The introduction already acknowledges power rates for feedforward operator learning under low-dimensional structure. Please make the comparison more explicit in the abstract and main discussion. In the cited Liu et al. (2026), Section 4.4, the input structure is a finite linear basis with exact coefficient recovery; the present metric-covering formulation and discretized observations differ. The two-level POU and routing also build on the cited Shi et al. and Shi–Liao constructions. A short comparison of assumptions, error norms, parameter counts, and rates would identify the additional operator-learning contribution. The power-versus-logarithmic contrast should not suggest an architectural advantage obtained under matched assumptions.

  4. Interpretation and uncertainty of the numerical slopes (Figure 3; Appendix E).

    The slopes decrease from 2.83 to 2.51 for Burgers and from 2.10 to 1.47 for KdV. Section 5 appropriately describes qualitative consistency and disclaims quantitative estimation of the theoretical rate. Please reconcile the conclusion's stronger “predicted quantitative data scaling law” wording with this statement. The experiments use a fixed four-head architecture, while Theorem 2 selects a growing architecture and grid. Faster empirical decay is compatible with a conservative upper bound.

    The families also change beyond dimension. For Burgers, the normalized bumps give E∥u0∥L22=dU/48\mathbb E\|u_0\|_{L^2}^2=d_U/48, so energy increases fourfold and bump density changes. KdV changes soliton count and nominal configurations. A small comparison varying active parameters in a fixed, suitably normalized family would help isolate dimension. Please report per-repeat slope uncertainty and sensitivity to the fitted sample-size range. Pointwise MSE error bars do not establish uncertainty in slope differences, especially with nested Burgers training sets. These are requests to strengthen the existing illustration, rather than to match the theoretical exponent or add extensive benchmarks.

Questions for authors
  • What explicit construction ensures compatibility along the asymptotic sequence used in the scaling statements?
  • Is the embedding fixed in the ERM class, and how robust are the empirical slope differences to fitting uncertainty and family controls?
7 comments
AuthorsOpenReview ↗

Response to Reviewer wskW (Part 1/2)

Response to Reviewer wskW

We thank the reviewer for the careful evaluation of our paper and for recognizing the metric-covering formulation, explicit Transformer construction, and quantitative complexity analysis.

Weaknesses 1: The sequence-length compatibility condition P≥nx+2P\ge n_x+2 may fail as the approximation tolerance decreases. The stated dimension inequality does not guarantee this condition, potentially restricting the asymptotic scope of Theorems 1 and 2.

Response: We thank the reviewer for pointing out this issue. We agree that the stated dimension inequality does not, by itself, guarantee P≥nx+2P\ge n_x+2. In the updated manuscript, we explicitly retain P=CUCV+1≥nx+2P=C_UC_V+1\ge n_x+2 as a separate compatibility condition in Theorems 1 and 2.

This condition arises from our particular two-block Transformer construction, which uses the sequence dimension to accommodate both the discretized input observations and the input-output anchor pairs. The resulting approximation and generalization guarantees hold whenever this compatibility condition is satisfied.

However, we are confident that the compatibility condition can be removed through an alternative three-block Transformer construction. The key idea is to distribute the quadrature features across sequence tokens rather than storing all nxn_x contributions within each anchor-pair token. Specifically, we choose

D=d1+d2+7,P=max⁡nxCU,CUCV+1.D=d_1+d_2+7,\qquad P=\max\\{n_xC_U,C_UC_V\\}+1.

The first block extracts the quadrature features, the second block aggregates them into input-space localization scores and assembles the joint input-output logits, and the third block performs Softmax aggregation over the anchor pairs. This separates quadrature aggregation from the final Softmax POU aggregation, avoiding the original sequence-length bottleneck independently of the relative growth rates of nxn_x, CUC_U, and CVC_V.

We are confident that this three-block construction works and can remove the original compatibility restriction. We plan to incorporate the complete construction and corresponding approximation and complexity analysis in a subsequent revision. The theorems in the current manuscript retain the stated compatibility condition.

Weaknesses 2: The Transformer hypothesis class in Theorem 2 is not fully consistent with the covering-number proof, which assumes fixed preprocessing.

Response: We have clarified immediately before Theorem 2 that the clipped ERM hypothesis class uses the fixed preprocessing map constructed in Lemma 6, while the attention, feedforward, and readout parameters are variable. This is the same class treated in the Transformer covering-number proof (Lemma 10), resolving the inconsistency.

Weaknesses 3: The novelty and theoretical assumptions should be compared more explicitly with existing feedforward operator learning results and previous Softmax POU constructions.

Response: We have added Table 1 in Section 3.2 to compare the assumptions and scaling guarantees of the feedforward DeepONet analysis of Liu et al. (2026) with our Transformer results. We also explicitly acknowledge the earlier Softmax POU constructions of Shi & Liao (2026) and Shi et al. (2026). The distinction is our function-space-level localization under an L2L^2 metric-covering condition, discretization of functional inner products, and explicit Transformer realization and complexity analysis; we do not attribute polynomial rates solely to attention.

AuthorsOpenReview ↗

Response to Reviewer wskW (Part 2/2)

Weaknesses 4: The numerical scaling results require stronger interpretation, uncertainty analysis, and controlled input families to isolate intrinsic-dimensional effects.

Response: We have replaced the original dimension comparison by controlled input families with dU∈{2,4,6,8}d_{\mathcal U}\in\{2,4,6,8\} across Burgers', KdV, and Navier--Stokes, matching expected input L2L^2 energy within each PDE (Figure 3 and Appendix E.2--E.4). The paired exponent differences p(2)−p(8)p(2)-p(8) are positive for all five seeds in each PDE, with 95%95\% seed-bootstrap confidence intervals excluding zero. Appendix E.5 now includes per-seed exponent estimates, paired differences, and sensitivity to fitting windows of 6, 5, and 4 points. The fitting-window trend is clearest for Navier--Stokes; Burgers' and KdV show some sensitivity. We interpret the results as finite-sample qualitative evidence, not as measurement of the theoretical upper-bound exponent.

Questions:

1: What explicit construction ensures compatibility along the asymptotic sequence used in the scaling statements?

Response: As discussed in our response to Weakness 1, the current two-block construction assumes P=CUCV+1≥nx+2P=C_UC_V+1\ge n_x+2; the dimension inequality alone does not ensure this condition along the asymptotic sequence. We have therefore retained it explicitly in Theorems 1 and 2, including for the tolerance selected in the data-scaling statement.

To address the underlying architectural bottleneck, we propose a three-block construction with D=d1+d2+7D=d_1+d_2+7 and P=max⁡nxCU,CUCV+1P=\max\\{n_xC_U,C_UC_V\\}+1. The first block distributes the nxCUn_xC_U quadrature features across sequence tokens, the second aggregates them into input-space localization scores and forms the CUCVC_UC_V joint logits, and the third applies Softmax attention to aggregate the corresponding anchor values. By construction, P−1P-1 accommodates both sets of features regardless of their relative growth as ϵ→0\epsilon\to0. We are confident this approach removes the original compatibility bottleneck. A complete realization and complexity analysis will be included in a subsequent revision; the guarantees in the current manuscript remain conditional on the stated two-block compatibility assumption.

2: Is the embedding fixed in the ERM class, and how robust are the empirical slope differences to fitting uncertainty and family controls?

Response: Yes. The embedding is fixed as part of the preprocessing map specified in the clipped ERM class immediately before Theorem 2, consistently with the covering-number argument in Lemma 10. For the empirical slope differences, the paired p(2)−p(8)p(2)-p(8) comparisons are positive across all five seeds for Burgers', KdV, and Navier--Stokes, with 95%95\% bootstrap confidence intervals excluding zero (Figure 3, Appendix E.5). We also added fitting-range sensitivity over n=32n=32--10241024, 6464--10241024, and 128128--10241024 (Appendix E.5). The trend is stable for Navier--Stokes, while Burgers' and KdV exhibit greater fitting-window variability.

  • W1: Yes, an explicit compatibility condition is acceptable as a temporary solution. I also believe a three-block construction could address this condition. Please give a concrete regime where compatibility holds as ε→0\varepsilon\to0, including the tolerance selected in Theorem 2. The three-block extension still requires a complete construction and complexity analysis.

  • W2: OK, the fixed-preprocessing clarification resolves my concern.

  • W3: The new Table 1 looks OK.

Reviewer_mTw9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study approximation and generalization rates of trasnformer-based operator learning models. They construct trasnformers that appeoximate well the operators of interest and derive power-law rates, both in model sizes and sample sizes. They ran Burgers' and KdV experiments and empirically show the power-law decaying errors.

Critical strengths
  • The paper studies relevant questions about scaling laws of transformer-based operator learning models.
  • The rates the authors obrained are explicit and related to the intrinsic dimension and smoothnes assumption
  • The proofs are well-written and not difficult to follow
  • The construction is clean
  • The experiments are well-motivated
Critical weaknesses

-It is unclear to me what role attention plays in the derived rates. In my understanding, the network computes the distance between the inputs and the anchors, and it returns softmax weighted average of the anchor values. The number of anchors is controlled by the dimensionality of the input space (i.e. intrinsic dimension of the input functions). I believe this is where the power law comes from, rathert than the attention mechanism. I would like the authors to elaborate on this point, and whether a simple MLP with similar construction has the same properties.

-It is unclear to me how the experiments and the theory are related. The rate in theorem 2 is obtained by choosing the tolerancethat depends on nn. This implies varying model size. The experiments use one model size for all nn. If I read the bound correctly, the obtained exponents seem to be larger than the exponents in the bound (this should be noted explicitly). The section 5 mentions that the expontents do not reflect the theory, while the abstract says that the experiments valdiate the scaling laws.

-The paper tests the effect of the intrinsic dimension with only 2 values per PDE. I believe that one cannot draw a clear conclussion on this matter with only 2 data points per equation. Moreover, it is unclear to me whether the difference in exponents comes only from the intrinsic dimension alone, or there are other factors influencing the exponents with increasing intrinsic dimension (for example, the KdV family with the higher intrinsic dimension also has more and faster solitons).

Questions for authors
  • Could the authors comment on how their exponents compare with simple MLP results (under the same low-dimensional assumption)?
  • How do fitted exponents change when the model sizes are varied?
  • How do fitted exponents change when the noise is added?
  • Is there an empricial, functional relationship between the intrinsic dimension and the fitted exponent, in the 2 experiments?
4 comments
AuthorsOpenReview ↗

Response to Reviewer mTw9 (Part 1/2)

Weaknesses 1: It is unclear to me what role attention plays in the derived rates. In my understanding, the network computes the distance between the inputs and the anchors, and it returns softmax weighted average of the anchor values. The number of anchors is controlled by the dimensionality of the input space (i.e. intrinsic dimension of the input functions). I believe this is where the power law comes from, rather than the attention mechanism. I would like the authors to elaborate on this point, and whether a simple MLP with similar construction has the same properties.

Response: We thank the reviewer for this observation. The polynomial rate is enabled by the low-dimensional covering structure of the input function class, while Softmax attention provides a natural mechanism for realizing the corresponding POU approximation. In particular, dot-product attention constructs the localization logits, and the Softmax operation directly produces normalized weights for aggregating local operator values (Section 4.1 and Appendix B).

For a ReLU-based MLP, implementing the same Softmax POU would generally require a different approximation construction and complexity analysis. Whether an MLP can achieve the same scaling exponents under our general L2L^2 metric-covering assumptions is not established by our results. Our contribution is to exploit the structural alignment between Softmax attention and POU to establish quantitative approximation and generalization guarantees for Transformer operator learning.

Weaknesses 2: It is unclear to me how the experiments and the theory are related. The rate in theorem 2 is obtained by choosing the tolerancethat depends on n. This implies varying model size. The experiments use one model size for all n. If I read the bound correctly, the obtained exponents seem to be larger than the exponents in the bound (this should be noted explicitly). The section 5 mentions that the expontents do not reflect the theory, while the abstract says that the experiments valdiate the scaling laws.

Response: Theorem 2 gives an upper bound for the clipped ERM in a Transformer class whose architecture is chosen as a function of the approximation tolerance, and hence of nn. It does not predict the fitted slope for a fixed architecture on a particular PDE; empirical exponents larger than the theoretical upper-bound exponent are therefore not contradictory. Our data-scaling experiments keep the architecture fixed to isolate the effect of training-set size. The revised experiments cover four intrinsic dimensions across Burgers', KdV, and Navier--Stokes (Figure 3), and separate width and noise ablations are reported in Appendix E.6 (Figures 10 and 11). We have clarified in Section 5 and the abstract that the empirical observations provide qualitative evidence for the intrinsic-dimension dependence, not a quantitative verification of the theoretical exponent.

Weaknesses 3: The paper tests the effect of the intrinsic dimension with only 2 values per PDE. I believe that one cannot draw a clear conclussion on this matter with only 2 data points per equation. Moreover, it is unclear to me whether the difference in exponents comes only from the intrinsic dimension alone, or there are other factors influencing the exponents with increasing intrinsic dimension (for example, the KdV family with the higher intrinsic dimension also has more and faster solitons).

Response: We have expanded the experiments to four intrinsic dimensions, dU∈{2,4,6,8}d_{\mathcal U}\in\{2,4,6,8\}, for each of Burgers', KdV, and Navier--Stokes (Figure 3). Within each PDE, we fix the eight spatial basis functions, PDE parameters, and expected input L2L^2 energy, while varying the number of independent input parameters. The mean fitted exponents decrease from 2.132.13 to 1.631.63 (Burgers'), 1.791.79 to 1.151.15 (KdV), and 1.581.58 to 0.290.29 (Navier--Stokes) between dimensions 2 and 8. The paired p(2)−p(8)p(2)-p(8) differences are positive in all five seeds for each PDE (Appendix E.5). These controls reduce the confounding identified in the original experiments, although changing the input family can also affect the resulting solution dynamics.

AuthorsOpenReview ↗

Response to Reviewer mTw9 (Part 2/2)

Questions:

1: Could the authors comment on how their exponents compare with simple MLP results (under the same low-dimensional assumption)?

Response: The polynomial exponent is driven by low-dimensional input complexity and is not exclusive to Transformers. Our Table 1 (Section 3.2) compares our theoretical scaling guarantees with the feedforward DeepONet results of Liu et al. (2026), including their respective assumptions. In the corresponding band-limited Lipschitz setting, the model-scaling exponents agree up to logarithmic factors. We therefore do not claim a strictly better exponent than feedforward operator-learning methods; our contribution is a constructive Transformer-specific approximation and generalization guarantee.

2: How do fitted exponents change when the model sizes are varied?

Response: We have added a width ablation for Burgers' with dU=4d_{\mathcal U}=4. Transformer widths 9696, 192192, and 384384 (approximately 0.150.15, 0.600.60, and 2.382.38 million parameters) yield mean fitted exponents p=1.81p=1.81, 1.871.87, and 1.971.97, respectively. The largest model also achieves lower test MSE at larger sample sizes. The scaling curves appear in Figure 10(a), with seed-level exponents and paired test-MSE comparisons in Appendix E.6.

3: How do fitted exponents change when the noise is added?

Response: We have added independent Gaussian noise to Burgers' training labels at 0%0\%, 1%1\%, and 5%5\% of the pooled clean-output RMS, while retaining clean validation and test targets. The corresponding mean fitted exponents are p=1.87p=1.87, 1.701.70, and 1.361.36 over the investigated sample-size range. These finite-range effects are shown in Figure 11(a), with uncertainty and test-MSE ratios in Appendix E.6. We do not interpret this change as a noise-dependent asymptotic exponent predicted by Theorem 2.

4: Is there an empricial, functional relationship between the intrinsic dimension and the fitted exponent, in the 2 experiments?

Response: Across the four tested dimensions dU∈{2,4,6,8}d_{\mathcal U}\in\{2,4,6,8\}, the mean fitted exponent generally decreases as intrinsic input dimension increases (Figure 3). The complete exponent-versus-dimension results, five-seed bootstrap confidence intervals, and paired exponent differences are reported in Appendix E.5. This establishes an empirical trend over the measured dimensions; the available data do not identify a universal functional form p(dU)p(d_{\mathcal U}).

I thank the authors for their responses and the additional experiments. The extended experiments with four intrinsic dimensions address my concern about the dimension comparison. I also appreciate the new comparison with DeepONet results in Table 1. I am willing to raise my score to 3.

We sincerely thank the reviewer for their constructive feedback and for recognizing the additional experiments and theoretical comparisons in our revised manuscript.

Reviewer_fUpxOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper develops approximation and generalization theory for Transformer-based operator learning under three assumptions: a H"older-continuous operator, H:older-regular input and output functions, and an input function class with covering dimension d_U. The approximation result (Theorem 1) builds a two-block softmax Transformer that realizes a two-level softmax partition of unity, one over input-function anchors and one over output-domain anchors. The generalization result combines this construction with a covering-number bound for the clipped Transformer class and shows that the mean squared error of the empirical risk minimizer decays up to logarithmic factors.

Critical strengths

The most important strength is the exact factorization in the attention construction. The explicit size count in Theorem 1, the parameter-magnitude bound, and the covering number that drives Theorem 2 all rely on this factorization.

Critical weaknesses

The most critical weakness is that the theory does not describe the Transformers it claims to explain. As a result, the “neural scaling laws” in the title are not really scaling laws of Transformer operator networks. The constructed network is a lookup table over a fixed cover, built into the weights:

  1. The anchor values u_k(x_r) and G(u_k)(z_l) are hard-coded into the value vectors (Lemma 7).
  2. The sequence length P = C_U·C_V + 1 is set by the cover size, not by the number of input samples (Theorem 1). The input tokens are padded to reach this length.
  3. The number of heads, H₁ = (n_x+3)P + 2, grows like \epsilon^(−s). Almost all of N_total therefore sits in heads, not in the depth or width that practitioners scale.
  4. Each forward pass handles one query point. Practical Transformer operators such as GNOT, OFormer, and Transolver work differently. Their token count equals the input discretization. They have a few heads and process many queries at once.
Questions for authors
  1. The work and the framework should have a direct link to the practical architectures used in the field.
  2. The band-limited corollary gives s = d_U/\gamma + d₂/\beta. This is the rate any piecewise-constant approximator achieves on a set of covering dimension d_U. Under the same Assumption 2-2, what rate do the existing feedforward results achieve (Liu et al. 2024, 2025, 2026)? Please give a side-by-side comparison. Without it, the contrast in the abstract with “logarithmic-type rates” for feedforward networks credits the Transformer for an improvement that comes from the assumption.
  3. The single-level softmax partition of unity comes from Shi & Liao (2026), and the two-level version comes from Shi et al. (2026). Which steps of the proof of Theorem 1 are new beyond the transfer to operators and the quadrature step? Similarly, Theorem 2 reuses the decomposition and Lemmas 3–4 of Liu et al. (2026). Which part of the generalization argument is specific to Transformers beyond Lemma 10?
3 comments
AuthorsOpenReview ↗

Response to Reviewer fUpx (Part 1/2)

We thank the reviewer for the valuable comments and for recognizing the exact attention factorization and the explicit parameter-complexity analysis in our theoretical construction.

Weaknesses General: "The most critical weakness is that the theory does not describe the Transformers it claims to explain. As a result, the “neural scaling laws” in the title are not really scaling laws of Transformer operator networks."

Response: We thank the reviewer for this comment. Our work does not claim to characterize a particular existing Transformer operator architecture. Instead, we consider a general encoder-only Transformer architecture with native Softmax attention and point-wise feedforward layers in Section 2.2 (Definition 1). For this Transformer architecture, Theorem 1 establishes a quantitative approximation bound in terms of the network parameter count, while Theorem 2 derives a power-law generalization bound in terms of the training data size. These results characterize the model and data scaling laws for the specified Transformer operator network class.

Weaknesses: 1: The constructed network is a lookup table over a fixed cover, built into the weights. The anchor values (u_k(x_r)) and (G(u_k)(z_l)) are hard-coded into the value vectors (Lemma 7).

Response: We thank the reviewer for this comment. Our approximation proof consists of two steps: constructing an oracle approximator using a two-level Softmax partition of unity over the input function space and output domain, and explicitly realizing this oracle approximator with a Transformer architecture (Section 4.1 and Appendix B). The anchor values (uk(xr))(u_k(x_r)) and (G(uk)(zl))(G(u_k)(z_l)) naturally arise in the oracle construction and are encoded into the Transformer parameters to establish the constructive approximation guarantee in Theorem 1.

Importantly, this construction is only used to control the approximation (bias) term in our generalization analysis. Theorem 2 establishes a generalization bound for an empirical risk minimizer over the entire specified Transformer hypothesis class, rather than only for the explicitly constructed network. The estimation (variance) term is controlled through the covering number of this Transformer class (Lemma 10). Thus, the resulting generalization scaling law is not restricted to a lookup-table-based predictor.

In our numerical experiments, the Transformer is trained directly from input-output data without explicitly constructing the anchors or hard-coding their values into the network.

Weaknesses 2: The sequence length P=CUCV+1P=C_UC_V+1 is set by the cover size, not by the number of input samples (Theorem 1). The input tokens are padded to reach this length.

Response: We thank the reviewer for this comment. The sequence length P=CUCV+1P=C_UC_V+1 is a specific choice in our constructive approximation proof, where the CUCVC_UC_V tokens represent input-output anchor pairs to realize the two-level Softmax POU (Section 4.1 and Appendix B). This construction establishes a sufficient approximation guarantee rather than prescribing a particular architecture for practical training.

Importantly, our construction uses native Softmax attention, point-wise ReLU feedforward layers, and sinusoidal positional encodings, which are standard Transformer components. Our numerical experiments use a two-block Transformer with four attention heads, without requiring the proof-specific sequence length or padding construction (Section 5 and Appendix E). The sequence length in the theoretical construction is explicitly accounted for in the total parameter count NtotalN_{\rm total} and the resulting model scaling law in Theorem 1.

Weaknesses 3: The number of heads, H1=(nx+3)P+2H^1=(n_x+3)P+2, grows like ϵ−s\epsilon^{-s}. Almost all of NtotalN_{\rm total} therefore sits in heads, not in the depth or width that practitioners scale.

Response: We thank the reviewer for this observation. Our construction uses a shallow Transformer with two blocks, where multiple attention heads perform feature extraction in parallel without increasing the sequential depth (Section 4.1 and Appendix B). Shallow architectures are also common in practical neural operators, such as GNOT and OFormer. Moreover, our numerical experiments use only two Transformer blocks with four attention heads, without requiring the large head count in the constructive proof. The growing number of heads is therefore a sufficient theoretical construction, rather than a requirement for practical training.

AuthorsOpenReview ↗

Response to Reviewer fUpx (Part 2/2)

Weaknesses 4: Each forward pass handles one query point. Practical Transformer operators such as GNOT, OFormer, and Transolver work differently. Their token count equals the input discretization. They have a few heads and process many queries at once.

Response: We thank the reviewer for this observation. Our pointwise formulation allows predictions at arbitrary output locations (Section 2.2), and multiple queries can be evaluated in parallel through batching. In fact, our numerical experiments use the same formulation, processing multiple output queries in parallel as independent input-query pairs.

Moreover, our approximation guarantee in Theorem 1 controls the error of the entire predicted output function in L2(ρy)L^2(\rho_y), rather than the error at a single fixed query point. The formulation therefore extends directly to evaluating multiple output queries by applying the same predictor to each query. Thus, the single-query formulation does not restrict the model to predicting only one output location.

Questions:

1: The work and the framework should have a direct link to the practical architectures used in the field.

Response: We thank the reviewer for this suggestion. Our framework is based on standard Transformer components, including native Softmax attention and point-wise feedforward layers (Section 2.2). The constructive proof demonstrates how these components realize the localization and aggregation required for operator approximation (Section 4.1). Moreover, our numerical experiments use a practical two-block Transformer with four attention heads, without imposing the theoretical anchor-based construction (Appendix E). Thus, while our explicit construction is designed for theoretical analysis, the underlying architecture and attention mechanism are directly relevant to practical Transformer operator learning.

2: The band-limited corollary gives s=dU/γ+d2/βs=d_{\mathcal U}/\gamma+d_2/\beta. This is the rate any piecewise-constant approximator achieves on a set of covering dimension dUd_{\mathcal U}. Under the same Assumption 2-2, what rate do the existing feedforward results achieve (Liu et al. 2024, 2025, 2026)? Please give a side-by-side comparison. Without it, the contrast in the abstract with “logarithmic-type rates” for feedforward networks credits the Transformer for an improvement that comes from the assumption.

Response: We agree that the polynomial rate is enabled by the low-dimensional input structure, rather than by a unique advantage of the Transformer architecture. We have added a side-by-side comparison of the assumptions and scaling guarantees for DeepONet and our Transformer in Table 1 (Section 3.2) of the updated manuscript. In the corresponding band-limited Lipschitz setting with dU=bUd_{\mathcal U}=b_U and γ=β=1\gamma=\beta=1, the model-scaling exponents coincide up to logarithmic factors (Appendix A.3). Our architecture-specific contribution is the explicit realization of the operator approximation by native Transformer attention, together with a Transformer parameter-complexity and generalization analysis.

3: The single-level softmax partition of unity comes from Shi & Liao (2026), and the two-level version comes from Shi et al. (2026). Which steps of the proof of Theorem 1 are new beyond the transfer to operators and the quadrature step? Similarly, Theorem 2 reuses the decomposition and Lemmas 3–4 of Liu et al. (2026). Which part of the generalization argument is specific to Transformers beyond Lemma 10?

Response: We thank the reviewer for this question. We acknowledge that our analysis builds on the Softmax POU framework of Shi & Liao (2026) and Shi et al. (2026), as well as the ERM analysis of Liu et al. (2026).

For Theorem 1, the key distinction is the extension of Softmax POU to operator learning, where the input and output are functions and the approximation must account for the regularity of the target operator. Our construction directly localizes input functions in a potentially infinite-dimensional function space through L2L^2 covering numbers and functional inner products, without requiring explicit finite-dimensional coordinates. We then approximate these inner products from discrete observations and explicitly realize the resulting two-level Softmax POU using native Transformer attention, establishing quantitative approximation and parameter-complexity bounds (Section 4.1 and Appendix B).

For Theorem 2, we adopt the general ERM decomposition and bias-variance analysis of Liu et al. (2026). However, the statistical complexity of our Transformer hypothesis class differs from that of their DeepONet class. We therefore establish a Transformer-specific covering-number bound (Lemma 10) and combine it with our approximation result to derive the data scaling law.

We would appreciate the reviewer’s feedback on our detailed rebuttal and revised manuscript, which directly address the concerns raised in the original review. As the discussion period is concluding, we kindly ask the reviewer to reconsider the assessment in light of these substantial revisions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions

We thank the reviewers for their valuable feedback. We have uploaded a revised manuscript, with changes highlighted in blue. The main updates include clarifications of the theoretical construction and generalization analysis, a comparison with existing operator-learning scaling results, and new experiments on Burgers', KdV, and Navier--Stokes equations. We also provide additional scaling analyses, ablation studies, and representative solution predictions in Section 5 and Appendix E. We address the reviewers' individual concerns in our responses below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.