Nonasymptotic $\mu$P Theory for Neural Networks with Applications to Small-to-Large Generalization

Maximal Update Parameterization ($\mu$P) provides a principled framework for scaling neural networks while preserving non-degenerate feature-learning dynamics in the width limit. However, existing theory for $\mu$P is largely asymptotic, relying on the infinite-width limit and leaving its behavior in finite-width regimes poorly understood. In this paper, we use the Tensor Program (TP) framework to provide a non-asymptotic characterization of gradient descent dynamics in canonical scalar-input, width-$m$ shallow settings: two-layer linear and quadratic networks, and three-layer linear networks. Our analysis provides high-probability guarantees for the finite-width approximation over each fixed training horizon. As an application, we analyze small-to-large generalization for KL-regularized domain reweighting under two-layer square-loss dynamics, showing that the learned domain weights along the width-$m$ training trajectory remain close to their infinite-width counterparts with discrepancy $\tilde{O}(1/\sqrt{m})$. Finally, synthetic and real-world experiments provide qualitative evidence of cross-width domain-weight stability.