WINR: Neural Implicit Representations for Weight Space

Training large neural networks from scratch for every architectural variant remains a major computational bottleneck. Model growth methods reduce this cost by warm-starting larger target networks from smaller pretrained checkpoints, but they typically rely on hand-crafted rules such as weight tiling or layer duplication. These discrete heuristics restrict expansion to rigid scaling patterns and initialize large portions of the target network as direct copies of the source. In this paper we ask whether model growth can instead be formulated as a continuous field, and propose WINR, a method that replaces discrete tensor expansion with a learned continuous weight field. Rather than treating pretrained parameters as arrays to be copied, WINR views them as samples of an underlying function that maps structured geometric and semantic coordinates to weights. A single field, fitted to one or more pretrained networks, can be queried to synthesize weights for target architectures with new shapes, depths, or layer compositions. This continuous formulation naturally supports fractional expansion and cross-architecture synthesis beyond standard copy-based growth rules. Experiments on transformer model zoos show that WINR improves convergence over random initialization and achieves competitive performance against existing model growth methods.