- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper characterizes causal map families that admit uniform approximation across all sampling resolutions by a single transformer with length-independent parameters, under common input regularity. With additional target regularity, it also establishes approximation and generalization upper bounds and finite-bit approximation lower bounds.
Critical strengths
-
The paper provides a unified treatment of sequence causality, uniform approximation across resolutions with shared parameters, and the continuous-time limit.
-
Beyond universality, it provides quantitative approximation and generalization upper bounds together with finite-bit approximation lower bounds, examining the complexity required for approximation.
Critical weaknesses
The dependence on token dimension is an important limitation of the quantitative results. Both the approximation and prediction RMSE upper bounds have convergence exponent , which deteriorates with dimension for fixed regularity . This yields weak quantitative guarantees for high-dimensional token representations. Moreover, the finite-bit approximation lower bound has no corresponding dimension penalty in its exponent, leaving unclear whether this deterioration reflects intrinsic difficulty of the target class or losses in the construction and analysis.
Questions for authors
To what extent can the current proofs explain the dimension-dependent gap between the approximation upper and finite-bit lower bounds? In relation to Kovachki, Lanthaler, and Mhaskar (2024), please clarify whether this gap can be traced to losses in the present construction and analysis, or to obstacles in transferring their lower-bound arguments to the present assumptions and complexity measure.
2 comments
Thank you for clarifying the sources of the dimension dependence and the differences from prior lower bounds. The gap remains a limitation, but the explanation addresses my question and does not change my positive assessment of the unified causal approximation framework. I maintain my weak-accept recommendation.
Thank you for highlighting the unified theory and the quantitative gap.
Weakness
The factor d+2 comes from our construction; we have not shown it to be intrinsic. Roughly, we resolve distributions in the d+1 token-time coordinates using polynomial moments, recover those moments from attention, and approximate the resulting feature-to-output map by a shallow ReLU network. The number of moments grows with dimension; their unstable recovery requires finer feature precision, adding the extra power in the complexity estimate. The learning bound inherits this exponent from the approximation theorem.
The proof may be pessimistic because it resolves ambient measures and approximates the readout on a full feature cube, rather than exploiting the smaller set of realizable histories. But improving the exponent, especially removing its dimension dependence, appears beyond the present moment-recovery/readout argument: it would require a more stable representation or a quantitatively efficient approximation directly on that history set. We will explain these potential losses without claiming that they are removable or that the upper rate is optimal.
Question
Both issues matter: our upper construction has identifiable costs, while the available lower bounds do not yet match the target class and complexity measure needed to certify those costs. Proposition B.7 already proves logarithmic finite-bit hardness, but uses paths varying in only one token coordinate and therefore does not establish a dimension-dependent exponent.
Kovachki, Lanthaler, and Mhaskar (2024), Theorem 2.10, prove logarithmic lower bounds for Fréchet-smooth functionals on rich compact input sets. Their argument embeds high-dimensional smooth-function problems into the operator class. This supports the relevance of logarithmic rates, but does not directly settle our gap: our smoothness is stability under regular tests of history measures, not Fréchet differentiability; their bounds concern function-value samples or continuous real-valued encodings, whereas ours concerns finite-bit descriptions. A transfer must build a hard subclass satisfying our regularity condition and compare these information models. We will clarify this distinction and identify matching approximation and statistical lower bounds as open problems.