Infinite From Finite: Recursive Basis Networks

Depth is a fundamental axis of neural scaling, yet deep neural networks typically obtain deeper computation by stacking more parameterized layers, tightly coupling functional depth to parameter depth. In this paper, we explore a new route to neural scaling, where functional depth grows through synthesis from a finite parameter reservoir. We propose Recursive Basis Networks (RBNs), which achieve dynamic functional synthesis by recursively recomposing a shared basis dictionary through input-conditioned states. This enables RBNs to synthesize heterogeneous functional transformations without layer-specific parameters, turning depth into a runtime-controllable dimension while keeping the transformation parameter budget fixed. Therefore, RBNs transform parameter accumulation into functional composition, yielding a large compositional space from finite primitives. Across diverse approximation and representation-learning settings, RBNs demonstrate the effectiveness of dynamic functional synthesis, with strong expressivity, parameter efficiency, and flexible depth scaling. These results establish it as a general principle for deriving expandable functional complexity from finite parameterization. Our code will be released on our website.