- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors establish constructive approximation and generalization bounds for Transformer-based operator learning. A two-level softmax partition of unity localizes the input function and the output query, and a two-block Transformer realizes the resulting interpolant. Polynomial metric covering complexity of the input family yields power bounds in model size and training-function count, with explicit dependence on regularity and discretization. Burgers and KdV experiments illustrate how fitted data-scaling exponents vary between parameter families of different dimensions.
I find the direct treatment of metric input families interesting. The explicit attention realization and the accounting for quadrature, routing leakage, and weight magnitudes are useful contributions. The main proof arguments appear consistent under the sequence-length condition and fixed preprocessing used in the appendices. My principal concerns are the asymptotic scope of that condition, the precise statistical hypothesis class, and the strength of the numerical interpretation. These appear addressable through focused clarification and proof completion.
Critical strengths
- The input-family assumption uses metric covering numbers and does not require a fixed finite linear basis or explicit latent coordinates.
- The construction tracks dense parameter counts and polynomial weight bounds, rather than proving qualitative universal approximation alone.
- The statistical analysis correctly distinguishes independent training functions from their output queries. The experiments use repeated runs, held-out functions, and validation-based checkpoint selection.
Critical weaknesses
-
Asymptotic scope of the compatibility condition (Theorems 1–2; Appendix B.3).
The theorems explicitly assume , so the following concern is about applicability. Set
b=\frac{d_U}{\gamma}+\frac{d_2}{\beta}.$$ With $r_u\asymp\varepsilon^{1/\gamma}$, (32) gives $M_u\gtrsim\varepsilon^{-2/\gamma}\log(1/\varepsilon)$. The prescribed grid therefore satisfies $n_x\gtrsim\varepsilon^{-a}[\log(1/\varepsilon)]^{d_1/\alpha}$, whereas $P\lesssim\varepsilon^{-b}$. Compatibility eventually fails when $b\le a$, including equality because of the logarithm. For example, $d_1=d_2=d_U=\alpha=\beta=\gamma=1$ satisfies the displayed dimension inequality but gives $a=3$, $b=2$. Please explain the role of the weaker inequality $b\ge d_1/\alpha$ and give a sufficient, realizable regime in which compatibility holds along $\varepsilon\to0$, including the tolerance selected in Theorem 2. An upper covering bound alone does not ensure enough tokens. Alternatively, provide an explicit padding construction and verify its routing, aggregation, and complexity bounds. Padding may preserve the exponents, but that requires an argument. The scaling implications should retain the scope of the conditional theorems. -
Hypothesis class in the generalization theorem (Definition 1 and (7); Lemma 10; Appendix D.3).
Lemma 10 fixes preprocessing, and its proof uses when network parameters vary. Definition 1 and Section 3.3 do not expressly fix the affine embedding . Please state that the ERM uses the fixed preprocessing of Lemma 6, or extend the perturbation argument to trainable embeddings. The supplied proof establishes the bound for the restricted class; it should match the class in Theorem 2. This appears repairable and does not by itself invalidate the displayed statistical exponent.
-
Specific contribution and comparison of assumptions (Sections 1 and 4).
The introduction already acknowledges power rates for feedforward operator learning under low-dimensional structure. Please make the comparison more explicit in the abstract and main discussion. In the cited Liu et al. (2026), Section 4.4, the input structure is a finite linear basis with exact coefficient recovery; the present metric-covering formulation and discretized observations differ. The two-level POU and routing also build on the cited Shi et al. and Shi–Liao constructions. A short comparison of assumptions, error norms, parameter counts, and rates would identify the additional operator-learning contribution. The power-versus-logarithmic contrast should not suggest an architectural advantage obtained under matched assumptions.
-
Interpretation and uncertainty of the numerical slopes (Figure 3; Appendix E).
The slopes decrease from 2.83 to 2.51 for Burgers and from 2.10 to 1.47 for KdV. Section 5 appropriately describes qualitative consistency and disclaims quantitative estimation of the theoretical rate. Please reconcile the conclusion's stronger “predicted quantitative data scaling law” wording with this statement. The experiments use a fixed four-head architecture, while Theorem 2 selects a growing architecture and grid. Faster empirical decay is compatible with a conservative upper bound.
The families also change beyond dimension. For Burgers, the normalized bumps give , so energy increases fourfold and bump density changes. KdV changes soliton count and nominal configurations. A small comparison varying active parameters in a fixed, suitably normalized family would help isolate dimension. Please report per-repeat slope uncertainty and sensitivity to the fitted sample-size range. Pointwise MSE error bars do not establish uncertainty in slope differences, especially with nested Burgers training sets. These are requests to strengthen the existing illustration, rather than to match the theoretical exponent or add extensive benchmarks.
Questions for authors
- What explicit construction ensures compatibility along the asymptotic sequence used in the scaling statements?
- Is the embedding fixed in the ERM class, and how robust are the empirical slope differences to fitting uncertainty and family controls?
7 comments
Response to Reviewer wskW (Part 2/2)
Weaknesses 4: The numerical scaling results require stronger interpretation, uncertainty analysis, and controlled input families to isolate intrinsic-dimensional effects.
Response: We have replaced the original dimension comparison by controlled input families with across Burgers', KdV, and Navier--Stokes, matching expected input energy within each PDE (Figure 3 and Appendix E.2--E.4). The paired exponent differences are positive for all five seeds in each PDE, with seed-bootstrap confidence intervals excluding zero. Appendix E.5 now includes per-seed exponent estimates, paired differences, and sensitivity to fitting windows of 6, 5, and 4 points. The fitting-window trend is clearest for Navier--Stokes; Burgers' and KdV show some sensitivity. We interpret the results as finite-sample qualitative evidence, not as measurement of the theoretical upper-bound exponent.
Questions:
1: What explicit construction ensures compatibility along the asymptotic sequence used in the scaling statements?
Response: As discussed in our response to Weakness 1, the current two-block construction assumes ; the dimension inequality alone does not ensure this condition along the asymptotic sequence. We have therefore retained it explicitly in Theorems 1 and 2, including for the tolerance selected in the data-scaling statement.
To address the underlying architectural bottleneck, we propose a three-block construction with and . The first block distributes the quadrature features across sequence tokens, the second aggregates them into input-space localization scores and forms the joint logits, and the third applies Softmax attention to aggregate the corresponding anchor values. By construction, accommodates both sets of features regardless of their relative growth as . We are confident this approach removes the original compatibility bottleneck. A complete realization and complexity analysis will be included in a subsequent revision; the guarantees in the current manuscript remain conditional on the stated two-block compatibility assumption.
2: Is the embedding fixed in the ERM class, and how robust are the empirical slope differences to fitting uncertainty and family controls?
Response: Yes. The embedding is fixed as part of the preprocessing map specified in the clipped ERM class immediately before Theorem 2, consistently with the covering-number argument in Lemma 10. For the empirical slope differences, the paired comparisons are positive across all five seeds for Burgers', KdV, and Navier--Stokes, with bootstrap confidence intervals excluding zero (Figure 3, Appendix E.5). We also added fitting-range sensitivity over --, --, and -- (Appendix E.5). The trend is stable for Navier--Stokes, while Burgers' and KdV exhibit greater fitting-window variability.
-
W1: Yes, an explicit compatibility condition is acceptable as a temporary solution. I also believe a three-block construction could address this condition. Please give a concrete regime where compatibility holds as , including the tolerance selected in Theorem 2. The three-block extension still requires a complete construction and complexity analysis.
-
W2: OK, the fixed-preprocessing clarification resolves my concern.
-
W3: The new Table 1 looks OK.
Response to Reviewer wskW (Part 1/2)
Response to Reviewer wskW
We thank the reviewer for the careful evaluation of our paper and for recognizing the metric-covering formulation, explicit Transformer construction, and quantitative complexity analysis.
Response: We thank the reviewer for pointing out this issue. We agree that the stated dimension inequality does not, by itself, guarantee P≥nx+2. In the updated manuscript, we explicitly retain P=CUCV+1≥nx+2 as a separate compatibility condition in Theorems 1 and 2.
This condition arises from our particular two-block Transformer construction, which uses the sequence dimension to accommodate both the discretized input observations and the input-output anchor pairs. The resulting approximation and generalization guarantees hold whenever this compatibility condition is satisfied.
However, we are confident that the compatibility condition can be removed through an alternative three-block Transformer construction. The key idea is to distribute the quadrature features across sequence tokens rather than storing all nx contributions within each anchor-pair token. Specifically, we choose
D=d1+d2+7,P=maxnxCU,CUCV+1.
The first block extracts the quadrature features, the second block aggregates them into input-space localization scores and assembles the joint input-output logits, and the third block performs Softmax aggregation over the anchor pairs. This separates quadrature aggregation from the final Softmax POU aggregation, avoiding the original sequence-length bottleneck independently of the relative growth rates of nx, CU, and CV.
We are confident that this three-block construction works and can remove the original compatibility restriction. We plan to incorporate the complete construction and corresponding approximation and complexity analysis in a subsequent revision. The theorems in the current manuscript retain the stated compatibility condition.
Response: We have clarified immediately before Theorem 2 that the clipped ERM hypothesis class uses the fixed preprocessing map constructed in Lemma 6, while the attention, feedforward, and readout parameters are variable. This is the same class treated in the Transformer covering-number proof (Lemma 10), resolving the inconsistency.
Response: We have added Table 1 in Section 3.2 to compare the assumptions and scaling guarantees of the feedforward DeepONet analysis of Liu et al. (2026) with our Transformer results. We also explicitly acknowledge the earlier Softmax POU constructions of Shi & Liao (2026) and Shi et al. (2026). The distinction is our function-space-level localization under an L2 metric-covering condition, discretization of functional inner products, and explicit Transformer realization and complexity analysis; we do not attribute polynomial rates solely to attention.