Geometry-Aware First-Layer Initialization for ReLU Implicit Neural Representations

Implicit neural representations (INRs) provide a flexible framework for continuous signal modeling, yet ReLU MLPs under default initialization often struggle to recover fine-grained details despite their substantial representational capacity. Analyses based on representation rank and the neural tangent kernel (NTK) explain important aspects of this difficulty, but do not by themselves establish where the dominant limitation arises within the network. We investigate this question through Inlet Rank Collapse: despite the substantial dimensional expansion from input coordinates to first-layer features, default initialization of ReLU MLP fails to produce a corresponding increase in effective representation dimensionality. We examine this mismatch as a bottleneck in the effective use of network capacity and use it to develop a common perspective on existing remedies, including positional encoding, sinusoidal activations, and batch normalization. This diagnosis motivates Geometric Initialization, a coordinate-only construction of first-layer weights and biases that enriches the initial feature representation while preserving the ReLU architecture and subsequent training procedure. Experiments show improved reconstruction and greater benefits from increasing network width, demonstrating that first-layer initialization alone can help ordinary ReLU networks make better use of their existing representational capacity.