acceptodds
Under review as a conference paper at ICLR 2027

Geometry-Aware First-Layer Initialization for ReLU Implicit Neural Representations

11 views

Abstract

Implicit neural representations (INRs) provide a flexible framework for continuous signal modeling, yet ReLU MLPs under default initialization often struggle to recover fine-grained details despite their substantial representational capacity. Analyses based on representation rank and the neural tangent kernel (NTK) explain important aspects of this difficulty, but do not by themselves establish where the dominant limitation arises within the network. We investigate this question through Inlet Rank Collapse: despite the substantial dimensional expansion from input coordinates to first-layer features, default initialization of ReLU MLP fails to produce a corresponding increase in effective representation dimensionality. We examine this mismatch as a bottleneck in the effective use of network capacity and use it to develop a common perspective on existing remedies, including positional encoding, sinusoidal activations, and batch normalization. This diagnosis motivates Geometric Initialization, a coordinate-only construction of first-layer weights and biases that enriches the initial feature representation while preserving the ReLU architecture and subsequent training procedure. Experiments show improved reconstruction and greater benefits from increasing network width, demonstrating that first-layer initialization alone can help ordinary ReLU networks make better use of their existing representational capacity.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
implicit neural representations, neural initialization, coordinate-based networks, ReLU networks, feature geometry, image reconstruction›

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_v1k7OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors address the well known problem of spectral bias in ReLU-MLP INRs, i.e. the tendency of fitting low frequency components of the fitted image, while being unable to represent more detail, i.e. higher spatial frequencies. ReLU-MLPs (as opposed to SIREN and similar architectures) have had this reputation for a while now, which prompted many new types of activations as well as special positional encodings. The authors show now now that with an appropriate initialization, this spectral bias can largely be eliminated in ReLU-MLPs, which refutes this commonly held belief.

To explain this, the authors develop a framework to investigate the spectral properties of the NTK with respect to weights and activations, and show that what they call the “inlet-rank” plays a crucial role in the representation capability of an INR, and use this to motivate their proposed first-layer initialization for INRs built with pure ReLU-MLPs. They also apply the same framework to draw comparisons across the different popular architectures and achitectural components of INRs and compare them with respect to the notion of rank developed in this framework.

Critical strengths

The paper refutes a commonly held belief of the mentioned strong spectral bias of the ReLU MLPs. This belief has prompted the creation of many other types of INR architectures, starting with SIREN - but it turns out with the right initialization, ReLU-MLPs can perform a lot better than believed up to now. The authors have managed to explain this through their framework, that also allows us to analyze other architectures.
In my opinion however this first insight is breaking fresh ground and hopefully motivates the community to look more into this issue, and potentially unearth more performance in existing architectures by improving the initializations.

Critical weaknesses

There are no critical weaknesses.

Questions for authors

There are just a few details that I’d like to ask the authors to consider addressing:

  • In the paper a fixed value for epsilon was proposed. Depending on how we choose the grid size m = p^d, would it make sens to make it a function of p? (E.g. epsilon = 0.1* 1/p.)

  • In my experience, for example with random fourier features as the first layer, it doesn’t really make a difference whether we make them fixed or trainable, as even when they are trainable, they practically do not change. Is this the the same case for the proposed geometric initialization?

  • As proposed, the geometric initialization depends on m = p^d for d dimensions. It seems like we could generalize this by just drawing from an uniform random distribution on [-1, 1]^d instead, and applying the same normalization as proposed for the grid. Have you done any experiments along this line?

Reviewer_xF5COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper study the inlet rank collapse in ReLU INRs. A insufficient rank diminishes the benefits of widening the network. Under this finding, the authors propose Geometric Initialization for the first layer of ReLU INRs. And the authors show that Geometric Initialization can improve the performance of ReLU INRs comparing to the default setting.

Critical strengths
  1. This paper is easy to fellow and understand.
  2. The insights of this paper is clear and substantial.
  3. Based on the finding, the authors propose a straightforward and easy method to improve the performance of the ReLU INRs.
Critical weaknesses
  1. Network width utilization constitutes one of the authors’ core claims. However, the experiments are run with a fixed budget of 5000 steps. To validate this claim, the performance after convergence should be reported. Otherwise, the observed results may be merely benefited from a faster convergence rate.
  2. A quantitative or causal connection between inlet rank collapse and performance is missing. Stable rank does not serve as a sufficient statistic of performance, which is illustrated by the bias-only observation. Providing the rank–PSNR correlation or perform interventions where rank is modified with all other variables held constant is preferred. Otherwise, the logic from diagnosis to method is merely motivational and lacks conclusive evidence.
  3. In wide networks, Shuffled and Bias-only perform near Geometric, while Weights-only is noticeably inferior, which raises questions about whether geometric-direction–threshold pairing is necessary. Providing the performance of a simple baseline using random directions with matched bias and threshold distributions is preferred, where bias scale, threshold distribution, and activation RMS are independently controlled.
  4. Under Geometric, all first-layer features in the initial central region are zero, covering around 2.18% of pixels for width 64. Many zero features remain in this region post-training. The repair pilot achieves local improvement yet introduces global mixing. It remains unclear how to address the spatial coverage problem.
  5. The impact is limited. This method only targets ReLU INRs and is not applicable to advanced networks like SIREN. And even with the proposed method, ReLU INRs remain inferior to PE and SIREN in performance.
Questions for authors

Please refer to weakness.

Reviewer_cvHsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors note that ReLU-based INRs struggle to generate fine details and say the issue is related to limited diversity in the first-layer features. They propose a geometry-aware initialization method that uses grid anchors to set the first-layer weights and ReLU activation boundaries. Their method allows leaveing the network architecture and training procedure unchanged. The method is evaluated through rank and NTK analysis and also through image reconstruction experiments across varying network widths and 3D occupancy fitting tasks.

Critical strengths
  1. I appreciate that the proposed method only changes the initialization of the first layer and does not introduce additional network parameters or architectural components. The results in Table 3 (lines 388–413) show that this relatively simple modification can improve reconstruction, especially for wider networks. At width 1024, the proposed initialization improves PSNR from 19.61 dB to 21.99 dB compared with the default initialization. This is a meaningful improvement given that the architecture and subsequent training procedure remain unchanged.
  2. The authors evaluate six initialization variants across five network widths, using 32 images and three random seeds. They also report paired confidence intervals, which makes the improvement over the default initialization more convincing.
Critical weaknesses
  1. I am not fully convinced about the novelty of the proposed initialization or the claim that inlet rank collapse is the main limitation of ReLU INRs.

The proposed method assigns each neuron a direction based on an anchor point and chooses its bias so that the corresponding ReLU activation boundary passes near that point (Section 4, lines 264–301). However, this general idea has already been explored in the literature. For example, Shin and Karniadakis [Shin, Yeonjong, and George Em Karniadakis. "Trainability of relu networks and data-dependent initialization." Journal of Machine Learning for Modeling and Computing 1.1 (2020).]  proposed a data-dependent initialization that positions ReLU activation boundaries near selected training inputs. Similarly, Cyr et al. (Cyr, Eric C., et al. "Robust training and initialization of deep neural networks: An adaptive basis viewpoint." Mathematical and Scientific Machine Learning. PMLR, 2020.) introduced Box initialization, which distributes ReLU activation boundaries across the input domain to improve the initial approximation basis. Although the proposed regular-grid and radial-direction construction differs from these methods, I do not think the paper sufficiently explains why this particular arrangement should be better. More importantly, neither of these closely related initialization approaches is included in the experiments. I think these comparisons are necessary to establish what the proposed method contributes beyond existing geometry-based initialization strategies.

I also have concerns about the explanation based on Inlet Rank Collapse. In lines 253–259, the authors suggest that rank deficiency introduced at the first layer persists throughout the network and limits its effective capacity. However, nonlinear layers can increase the rank of their representations, which the authors themselves acknowledge in Appendix A.6 (lines 777–791). Therefore, I do not think the analysis establishes that the first layer is necessarily the primary bottleneck.

Furthermore, the experimental results do not show a consistent relationship between initial feature rank and reconstruction performance. For example, Appendix E.5 (lines 1051–1060) reports that the Weights-only variant has a higher initial stable rank than the default initialization but achieves lower PSNR at width 1024. Similarly, Bias-only achieves nearly the same reconstruction quality as Geometric despite having a lower stable rank. These observations suggest that feature rank alone cannot explain the improvements. I think the authors need to clarify whether their method primarily improves feature diversity, activation coverage, initialization scale, or some combination of these factors.

  1. The ablations are limited in terms of which part of the proposed geometric initialization is responsible for the improvement, and the evaluation remains limited. My concern is that the proposed geometric arrangement does not appear to account for most of the improvement. In Table 3 (lines 388–398), the complete Geometric initialization achieves 21.99 dB at width 1024, while Shuffled achieves 21.93 dB and Bias-only achieves 21.88 dB. This means that shuffling the association between weight directions and biases reduces performance by only about 0.054 dB. Although this difference is statistically positive in the reported paired analysis, it is quite small compared with the overall 2.38 dB improvement over the default initialization. This makes me question how important the specific radial arrangement and anchor–direction pairing actually are. It seems possible that most of the improvement comes from the bias distribution or the resulting activation patterns rather than the proposed geometric organization.

As shown in Figure 6 and Appendix F, all first-layer features are initially zero in a central region covering approximately 21.15% of the image at width 4 and 2.18% at width 64. The authors show that modifying a few biases can improve reconstruction in this region, even though it reduces initial stable rank. This is another indication that maximizing feature diversity globally may not be sufficient and that spatial coverage should be considered more carefully in the method design.

The 3D occupancy experiment is a useful addition, but the paper provides limited details about the evaluated shapes and how the proposed 2D grid-based initialization is extended to 3D.

Questions for authors
  • How does the proposed method compare with existing geometry-aware ReLU initialization approaches, particularly Shin and Karniadakis (2020) and Box initialization (Cyr et al., 2020)? Since these methods also position activation boundaries using input geometry, I would be interested in understanding what advantage the proposed radial-grid construction provides under the same training settings.

  • Could the authors provide additional ablations to better separate the effects of the geometric arrangement, bias initialization, and activation scale? In particular, Shuffled and Bias-only achieve almost the same performance as the complete method at width 1024. Could the authors please explain why the specific geometric pairing remains necessary, and compare these variants while matching initial activation statistics and bias magnitudes?

  • Do the improvements remain consistent with longer training and different network architectures?Additional details on the 3D initialization and evaluation protocol would also be helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.