Structural Correspondence and Universal Approximation in Diagonal plus Low-Rank Neural Networks

Structured low-rank and sparse-plus-low-rank parameterizations, motivated by methods such as LoRA and low-rank factorization, impose strong structural constraints on the linear transformations within neural networks. We study the expressive consequences of these constrained structures from the perspective of universal approximation. We first characterize that purely low-rank neural networks, despite being able to exactly interpolate arbitrary finite scalar datasets, can fail to approximate continuous functions that vary along directions ignored by the low-rank hidden mapping. We show that augmenting the low-rank structure with a one-parameter diagonal component suffices to overcome this geometric bottleneck, yielding a Diagonal-plus-Low-Rank (DLoR) structure. We show that a full-rank linear transformations can be reconstructed using DLoR components either by expanding network width through additive decomposition or by extending network depth through multiplicative decomposition. We term this correspondence between additive width expansion and multiplicative depth extension the Structural Correspondence framework. By combining these constructions with local linearization of the activation function, we prove that DLoR neural networks restore universal approximation for general activation functions. These results characterize how architectural width and depth can compensate for severe rank constraints without requiring unrestricted dense hidden transformations.