On the Exact Minimum Width for Universal Approximation with Deep ReLU Networks

How narrow can a deep network be while still approximating every continuous map from $[0,1{]}^d$ to ${\mathbb{R}}^k$? For ReLU networks, the exact minimum width has been known only in a few cases. We prove that it is exactly $k$ whenever $k\ge 2d+2$, improving the previous upper bound by one, and show that the same holds for GELU, SiLU, Softplus, and other activations that approximate ReLU. Our proof builds on a recent diffeomorphism approach that separates the minimum width into a geometric quantity $w(d,k)$ and the additional width needed to approximate diffeomorphisms. We show that one extra neuron suffices for every continuous nonaffine activation with a nonzero derivative at some point, giving the general upper bound $w(d,k)+1$. For ReLU, this extra neuron is necessary when approximating diffeomorphisms on full-dimensional regions, but universal approximation only requires approximation on the lower-dimensional image of $[0,1{]}^d$. We exploit this distinction to remove the extra neuron when $k\ge 2d+2$, where $w(d,k)=k$. For $k=2d+1$, we construct embeddings for which the geometric argument fails even after small perturbations. More generally, for all $k\le 2d+1$, the minimum width is either $w(d,k)$ or $w(d,k)+1$, and we conjecture that it is always $w(d,k)$.