- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates a training failure in sinusoidal neural fields constructed with a sum of learnable sinusoids: depending on the random draw of initial frequencies, optimization either fits the target signal or collapses to outputting the target's constant mean color. The authors predict this collapse prior to training using a single target-independent scalar from one forward pass.
Critical strengths
S1. Predicting network behavior is important in machine learning. S2. Empirically establishing lower and upper cutoffs of across 300 runs on three CelebA images and demonstrating via input-domain scaling and per-layer weight rescaling that the failure resides in the hidden layers rather than the coordinate-input layer; (2) motivating the functional form of via the small-correlation slope of a random sinusoidal layer's NNGP correlation map and showing how the empirical threshold shifts with bank family, fast-frequency terms, and network depth.
Critical weaknesses
W1: The theoretical derivation via the NNGP characterizes signal propagation at initialization near , but fails to explain why networks actually collapse or depart during gradient-based training.
Questions for authors
Can you show us why the initialization theoretically influences the training dynamics of the sinusoidal network?
2 comments
Follow-up before the discussion closes
Dear Reviewer eRo9, thank you again for your review. The discussion period closes on November 18 AoE. We would be glad to know whether our response answers your question on how the initialization shapes the training dynamics. We are happy to clarify any point before the deadline.
Response to Reviewer eRo9
We thank the reviewer for noting that "predicting network behavior is important in machine learning," and that input-domain scaling and per-layer weight rescaling demonstrate "that the failure resides in the hidden layers." The weakness and the question both ask how the state at initialization shapes training, and we answer them together. All changes are shown in blue.
The state at initialization decides what the gradient of the output head does in the first steps, and this starts the collapse. The revised Section 6 follows this path. It writes the output as y(x) = W h(x) + b, where h(x) holds the features of the last hidden layer. The argument has three steps.
This account makes three predictions about training, and all three hold.
A collapsed run can still depart later. In the ten recorded runs that depart from above the band of random banks, κ rises by at most 16% before departure, while the hidden weights change direction (Section 6.2 and Appendix G). The account does not predict the epoch of this late departure, and a theory of this phase remains open.
We hope this answers your question, and we would be glad to discuss further.