acceptodds
Under review as a conference paper at ICLR 2027

Universal Consistency of Transformers as Functional Regressors

Abstract

Transformers are an indispensable component of modern foundation models, yet their theoretical understanding for functional regression on non-Euclidean domains remains underexplored. While the universal approximation properties of Euclidean Transformers have been extensively studied approx_tran, their statistical consistency as functional regressors remains less understood, particularly beyond Euclidean settings. Motivated by recent advances in hyperbolic neural networks and the intrinsic hierarchies of data such as text, images, and graphs, we develop a statistical framework for analyzing Transformers in hyperbolic spaces. We establish universal approximation results for broad classes of sequence-to-sequence functional maps and prove universal statistical consistency for least-squares functional regression without imposing a parametric model specification. We further derive stochastic error bounds for the corresponding empirical estimators, yielding an asymptotic rate of , where denotes the number of input tokens and the input embedding dimension. Our analysis explicitly characterizes the curse of dimensionality induced by the ambient embedding dimension, providing a quantitative account of the statistical complexity of nonlinear attention-based Transformer architectures. The Euclidean setting is recovered as a special case of the proposed framework. Experiments on real datasets with both continuous and categorical responses empirically support our theoretical results and demonstrate the practical relevance of hyperbolic representations for functional sequence-to-sequence regression.

Primary area:
learning theory
Keywords:
Transformers, Self-attention, Statistical Consistency, Hyperbolic space

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2HEXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors proved a universal (strong) consistency theorem for functional regressors in hyperbolic setting. They also provided an estimation on stochastic error bounds and provide the corresponding asymptotic bound. Justifications and implications of the theoretical results are discussed through numerical experiments.

Critical strengths

The theoretical analysis provided in the paper is sound and impactful.

Critical weaknesses

Some of the results and statements are not completely clear. See the Questions.

Questions for authors
  1. While it is implicated and referred, it would be helpful to have a formal definition of the consistency provided, and the meaning of terms such as universal and strong explained.
  2. In related work, while works on attention and hyperbolic networks are provided, the consistency theorem, which is a critical topic in statistics and has been extensively studied, has not been adequately discussed. It would be great if the relationship between the consistency theorem presented in this paper and those in the literature can be clarified.
  3. is the lower bound (2,1,4) on the parameter based on the limit of universal approximation results discussed below equation (8), what do we know about situation when it is violated?
  4. The discussion below Table 2 on asymptotic behavior enabling an upper bound on the covering number does not provide enough details for the statement to be understood and verified.
  5. The proof of Lemma 16 is not very clearly written, first of all, there seems to be no Lemma 8 in the paper, then, it is quite difficult to reach equation (36) from the statements provided in the lemma, and the constant C∗C^* is never specified.
Reviewer_MedvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies universal statistical consistency of transformers for sequence-to-sequence regression in hyperbolic spaces. The authors propose a hyperbolic transformer formulation and claim to establish universal approximation, strong universal consistency, and a convergence rate of O(t−1/(2d))O(t^{-1/(2d)}).

Critical strengths
  1. The authors establish a statistical consistency beyond universal approximation, which is a meaningful research direction. Extending such guarantees to hyperbolic transformers could contribute to understanding learning on non-Euclidean domains.

  2. The authors provide an extension of transformer operations to hyperbolic spaces, with Euclidean transformers recovered in the limit, offering a useful framework for theoretical analysis.

Critical weaknesses

I find that this paper contains several serious errors in its mathematical results and proofs, as shown below.

  1. Lemma 4 is false: Eq. (18) claims that hyperbolic geodesic distance equals Euclidean distance between logarithmic embeddings, but this is incorrect in general. Specifically, for c=1c=1, p=(0.5,0)p=(0.5,0), and q=(0,0.5)q=(0,0.5), the two distances are approximately 0.8400.840 and 0.7770.777, respectively. Consequently, the equivalence of regression objectives in Eq. (9) is invalid, and Lemma 5 does not generally characterize the Bayes predictor for squared hyperbolic geodesic loss.

  2. Lemma 9 is also false as stated: its proof incorrectly assumes that LpL^p-separated functions have a per-sample probability of distinguishable sign patterns at least (ϵ/2M)K(\epsilon/2M)^K.

  3. The consistency and convergence-rate proofs contain additional substantial gaps. In particular, Lemma 10 invokes pseudo-dimension bounds without justifying their application to softmax and hyperbolic operations. Moreover, the covering-number bound in Corollary 12 does not support the vanishing probability conclusion required in Lemma 14. The claimed O(t−1/(2d))O(t^{-1/(2d)}) rate is likewise not established, particularly because no quantitative rate of the approximation error is provided.

Furthermore, I think the writing is also a bit confusing. For example, there are many bold texts that I don't know why they have to be emphasized, and the format of the "Contributions" also looks strange.

Questions for authors

Please see my concerns in the Weaknesses.

Reviewer_bQNYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies Transformers as regression models when the input and output tokens lie in hyperbolic space (the Poincaré ball). The main contributions are theoretical: a universal approximation result for this model, a proof that the least-squares estimator is universally consistent (Theorem 1), and a convergence rate of O(t−1/2d)O(t^{-1/2d}) in the number of tokens tt (Lemma 16). The Euclidean case is recovered as c→0c\to0. Their experiments train language, vision, and graph Transformers with an RMSE loss on six datasets at several curvatures, and compare test RMSE curves against t−1/2dt^{-1/2d}.

Critical strengths

S1. The paper asks whether Transformers trained by least squares are universally consistent for any data distribution. This goes beyond the usual universal approximation results, and a result of this kind would connect expressivity and generalization for Transformers. The motivation is strong and clearly explained in the introduction.

Critical weaknesses

W1. The paper states that it measures error through the hyperbolic squared distance (Line 255), but it then works only with the Euclidean error between log-mapped points, citing Lemma 4 to claim the two are equal (Eq. 9, Line 256). Lemma 4 doesn't seem to hold. The hyperbolic distance equals the distance between log-mapped points only when both points lie on the same line through the origin. For example, with c=1c=1, p=(0.5,0)p=(0.5,0) and q=(0,0.5)q=(0,0.5), Eq. 12 gives 0.840 while the tangent distance is 0.777. Since consistency in hyperbolic space is the paper's main contribution, this gap is critical.

W2. Table 2 also contains values that are pretty strange. SQuAD F1 is reported on a 0-1 scale but with standard deviations above 1 (e.g., 0.8533±1.4290.8533\pm1.429). Moreover, TweetQA is exactly 100 for every curvature without explanation, which may point to a degenerate target or metric. As a result, the experiments give little support for the stated rate or for the benefit of curvature.

Questions for authors

Clarification:

  1. For SQuAD, TweetQA, and the classification datasets, what is the regression target, and how are the model's continuous outputs converted into predictions for computing F1 and accuracy?
  2. How many seeds are used in Table 2?
  3. The MNIST test RMSE is on very different scales in Fig. 4(c) and Fig. 5(a). Is there a difference in their setup or normalization?

Minor Comments:

  1. Eq. 12 is half of the distance induced by the conformal factor λxc=2/(1−c∥x∥2)\lambda_x^c=2/(1-c\|x\|^2); please state this convention.
  2. The proof of Lemma 16 cites "Lemma 8", which is never stated.
  3. Figs. 5-6: the text is too small to read.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.