Preserving Regularity in Finite-Precision Transformer Approximation

Transformers are the dominant architecture for large language models and are widely used in multimodal foundation models. Their substantial storage requirements have made low-bit quantization an important tool for practical deployment. Yet quantitative Transformer approximation theory predominantly treats weights as unconstrained real numbers, leaving a gap between theoretical guarantees and finite-precision representations. This question goes beyond approximation accuracy: quantization can increase a model's sensitivity to small input perturbations even when the approximation error remains small. It is therefore important to understand whether finite-precision Transformers can simultaneously achieve accurate approximation, controlled storage complexity, and preservation of a prescribed regularity bound. In this paper, we construct a finite-precision Transformer that approximates matrix-valued $\lambda$-Hölder functions on $[0,1{]}^{d\times n}$ while preserving a prescribed componentwise ${\ell }^1$-Hölder bound $L$. Its hidden feature weights are discrete at $\{0,\pm 1,\pm \frac{1}{2}\}$ and target-independent, while target dependence is confined to the output coefficients. This separation allows us to directly control the effects of quantization on both approximation accuracy and regularity. For $D=dn$, the resulting Transformer achieves uniform error $\epsilon$ with $\mathcal{O}({\epsilon }^{-D/\lambda })$ nonzero parameters and a common $\mathcal{O}(\log (1/\epsilon ))$-bit fixed-point format for all parameters. These guarantees concern parameter storage under exact forward evaluation. As an application, empirical risk minimization over a structured finite-precision Transformer class achieves expected squared prediction risk $\mathcal{O}((\log (em)/m{)}^{2\lambda /(2\lambda +D)})$ under sub-Gaussian noise, matching the classical Hölder minimax exponent up to a logarithmic factor.