A Hidden-Layer Phase Scale Predicts When Sinusoidal Neural Fields Collapse

Sinusoidal activations remove the low-frequency bias of ReLU networks and are now standard in neural fields. Recent work replaces the single sinusoid of SIREN with a sum of sinusoids whose frequencies are drawn at random when the network is built. Such a network either fits its target or outputs the mean colour of that target, and we found nothing in between. One quantity predicts which of the two, from a single forward pass before training. It is the largest phase scale over the hidden sinusoidal layers. The phase scale of a layer is the slowest frequency of its sinusoids multiplied by the standard deviation of that layer's input. Computing it does not need the target. Over 300 runs on three images, 181 of 181 runs train below a phase scale of 1.81 and none of 51 trains above 4.0. The failure does not sit in the first layer, the one that sees the coordinates. Rescaling that layer by a factor of 20 leaves every run at the mean colour, while rescaling the hidden layers changes the outcome. The correlation map of a random sinusoidal layer gives the phase scale its form. It also accounts for the boundary moving down as the network deepens. A failed network reaches the mean colour by shrinking the weights of its output head. Between those cutoffs the bank of frequencies does not decide the outcome by itself. On identical initial weights, moving only the starting value of the output bias turns runs that train into runs that collapse, and runs that collapse into runs that train. Across 1859 runs the phase scale also orders the risk of collapse on Kodak-24, and it carries to audio, to shapes and to a PDE.