When Does Scale Trade for Time?

Can a larger model trained briefly behave like a smaller model trained for longer? We study this question, which we call scale-time equivalence, in a deliberately small and controlled setting where it can be derived, measured, and falsified. Theoretically, we analyze a random subspace model, in which the trained network occupies a random $p$-dimensional subspace of a larger nonlinear model. This model contains Gaussian linear random-feature models, a workhorse of scaling theory, as the special case of a quadratic loss, but, unlike them, allows the network output to depend nonlinearly on the trainable parameters. We prove that under fixed-learning-rate gradient flow, the function learned by a model of scale $p$ after time $t$ is approximated by a $p$-independent trajectory evaluated at $pt$, with an error that decays as $p^{-1/2}$ for any fixed value of $pt$. The bound grows exponentially only in the effective training progress $pt$, not along the scale-time tradeoff itself, so it guarantees equivalence between models of very different scales as long as both are large and their shared value of $pt$ is moderate. Empirically, in MLPs and CNNs trained with fixed-learning-rate SGD on MNIST, SVHN, and CIFAR-10, the number of epochs needed to generalize follows an approximate power law in model scale, with a common exponent across datasets, architectures, and training-set sizes, and small models trained for long predict large models trained briefly. With Adam, which lies outside the theorem's assumptions, the equivalence visibly breaks. Finally, we conjecture, and give preliminary evidence, that where scale-time equivalence holds, parameter-wise and time-wise double descent share a mechanism: the relative rates at which signal and noise are acquired.