- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors address the well known problem of spectral bias in ReLU-MLP INRs, i.e. the tendency of fitting low frequency components of the fitted image, while being unable to represent more detail, i.e. higher spatial frequencies. ReLU-MLPs (as opposed to SIREN and similar architectures) have had this reputation for a while now, which prompted many new types of activations as well as special positional encodings. The authors show now now that with an appropriate initialization, this spectral bias can largely be eliminated in ReLU-MLPs, which refutes this commonly held belief.
To explain this, the authors develop a framework to investigate the spectral properties of the NTK with respect to weights and activations, and show that what they call the “inlet-rank” plays a crucial role in the representation capability of an INR, and use this to motivate their proposed first-layer initialization for INRs built with pure ReLU-MLPs. They also apply the same framework to draw comparisons across the different popular architectures and achitectural components of INRs and compare them with respect to the notion of rank developed in this framework.
Critical strengths
The paper refutes a commonly held belief of the mentioned strong spectral bias of the ReLU MLPs. This belief has prompted the creation of many other types of INR architectures, starting with SIREN - but it turns out with the right initialization, ReLU-MLPs can perform a lot better than believed up to now. The authors have managed to explain this through their framework, that also allows us to analyze other architectures.
In my opinion however this first insight is breaking fresh ground and hopefully motivates the community to look more into this issue, and potentially unearth more performance in existing architectures by improving the initializations.
Critical weaknesses
There are no critical weaknesses.
Questions for authors
There are just a few details that I’d like to ask the authors to consider addressing:
-
In the paper a fixed value for epsilon was proposed. Depending on how we choose the grid size m = p^d, would it make sens to make it a function of p? (E.g. epsilon = 0.1* 1/p.)
-
In my experience, for example with random fourier features as the first layer, it doesn’t really make a difference whether we make them fixed or trainable, as even when they are trainable, they practically do not change. Is this the the same case for the proposed geometric initialization?
-
As proposed, the geometric initialization depends on m = p^d for d dimensions. It seems like we could generalize this by just drawing from an uniform random distribution on [-1, 1]^d instead, and applying the same normalization as proposed for the grid. Have you done any experiments along this line?