acceptodds
Under review as a conference paper at ICLR 2027

A Hidden-Layer Phase Scale Predicts When Sinusoidal Neural Fields Collapse

16 views

Abstract

Sinusoidal activations remove the low-frequency bias of ReLU networks and are now standard in neural fields. Recent work replaces the single sinusoid of SIREN with a sum of sinusoids whose frequencies are drawn at random when the network is built. Such a network either fits its target or outputs the mean colour of that target, and we found nothing in between. One quantity predicts which of the two, from a single forward pass before training. It is the largest phase scale over the hidden sinusoidal layers. The phase scale of a layer is the slowest frequency of its sinusoids multiplied by the standard deviation of that layer's input. Computing it does not need the target. Over 300 runs on three images, 181 of 181 runs train below a phase scale of 1.81 and none of 51 trains above 4.0. The failure does not sit in the first layer, the one that sees the coordinates. Rescaling that layer by a factor of 20 leaves every run at the mean colour, while rescaling the hidden layers changes the outcome. The correlation map of a random sinusoidal layer gives the phase scale its form. It also accounts for the boundary moving down as the network deepens. A failed network reaches the mean colour by shrinking the weights of its output head. Between those cutoffs the bank of frequencies does not decide the outcome by itself. On identical initial weights, moving only the starting value of the output bias turns runs that train into runs that collapse, and runs that collapse into runs that train. Across 1859 runs the phase scale also orders the risk of collapse on Kodak-24, and it carries to audio, to shapes and to a PDE.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
sinusoidal neural fields, implicit neural representations, initialization, trainability, signal propagation, spectral bias

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_eRo9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates a training failure in sinusoidal neural fields constructed with a sum of learnable sinusoids: depending on the random draw of initial frequencies, optimization either fits the target signal or collapses to outputting the target's constant mean color. The authors predict this collapse prior to training using a single target-independent scalar from one forward pass.

Critical strengths

S1. Predicting network behavior is important in machine learning. S2. Empirically establishing lower and upper cutoffs of κ\kappa across 300 runs on three CelebA images and demonstrating via input-domain scaling and per-layer weight rescaling that the failure resides in the hidden layers rather than the coordinate-input layer; (2) motivating the functional form of κ\kappa via the small-correlation slope of a random sinusoidal layer's NNGP correlation map and showing how the empirical threshold shifts with bank family, fast-frequency terms, and network depth.

Critical weaknesses

W1: The theoretical derivation via the NNGP characterizes signal propagation at initialization near c=0c=0, but fails to explain why networks actually collapse or depart during gradient-based training.

Questions for authors

Can you show us why the initialization theoretically influences the training dynamics of the sinusoidal network?

2 comments
AuthorsOpenReview ↗

Response to Reviewer eRo9

We thank the reviewer for noting that "predicting network behavior is important in machine learning," and that input-domain scaling and per-layer weight rescaling demonstrate "that the failure resides in the hidden layers." The weakness and the question both ask how the state at initialization shapes training, and we answer them together. All changes are shown in blue.

Weakness and Question. The theoretical derivation via the NNGP characterizes signal propagation at initialization near c = 0, but fails to explain why networks actually collapse or depart during gradient-based training. Can you show us why the initialization theoretically influences the training dynamics of the sinusoidal network?

The state at initialization decides what the gradient of the output head does in the first steps, and this starts the collapse. The revised Section 6 follows this path. It writes the output as y(x) = W h(x) + b, where h(x) holds the features of the last hidden layer. The argument has three steps.

  1. The head can learn almost nothing from the initial features. At a large κ, the correlation map sends the correlation between distinct coordinates toward zero, and nearby coordinates receive nearly unrelated features (Section 4.4). This is the regime near c = 0 that the map describes. For features with zero correlation, Proposition 2 in Appendix C.4 bounds the gain of the least-squares head at a fraction n/(N−1) of the energy, which is 0.017 dB on a 256×256 image. On six banks above κ = 4.0 the measured gain is under 0.03 dB.
  2. The gradient therefore shrinks W and moves b to the mean colour. For fixed features the loss is quadratic in W and b, and its minimum is the least-squares head, whose output is almost the mean colour. The initial W h(x) is a random pattern that makes the output worse than the mean colour, at 6.9 to 7.8 dB against 14.2 dB. Training removes this pattern by shrinking W, while b moves to the mean colour. For [8]×8 in Figure 3, W falls to 27% of its initial norm by epoch 25, and by epoch 75 the output equals the mean colour (Section 6.1).
  3. A small head slows the learning of the hidden layers. The hidden layers receive their gradient only through W, since the gradient with respect to h(x) is Wᵀ times the error. A small W passes back a small gradient. [8]×8 stays at the mean colour for 2000 epochs, and six runs of three such banks stay there for 10,000 epochs (Section 6.2).

This account makes three predictions about training, and all three hold.

  • Depth. The map multiplies one factor per hidden layer and predicts a lower cutoff for a deeper network. All five predictions of the depth study held. A random bank at κ = 2.98 trains in all 15 runs at depths 3 and 4 and in none at depths 5 and 6 (Section 5.3).
  • Residual path. The path adds the identity to the derivative of each layer and keeps a route for the gradient when the mean derivative of the sinusoid approaches zero. With the path, [30]×8 collapses in none of 9 runs, against 9 of 9 without it. The default STAF network has this path and collapses in none of 240 runs, although 104 of them start above 4.0 (Section 7.2).
  • Output bias. A collapse ends with b at the value that gives the mean colour, and the distance b must travel can then matter. In 18 pairs of runs with identical initial weights, the run whose b starts near this value collapses and the run whose b starts at least 1.0 away trains (Section 6.3).

A collapsed run can still depart later. In the ten recorded runs that depart from above the band of random banks, κ rises by at most 16% before departure, while the hidden weights change direction (Section 6.2 and Appendix G). The account does not predict the epoch of this late departure, and a theory of this phase remains open.

We hope this answers your question, and we would be glad to discuss further.

AuthorsOpenReview ↗

Follow-up before the discussion closes

Dear Reviewer eRo9, thank you again for your review. The discussion period closes on November 18 AoE. We would be glad to know whether our response answers your question on how the initialization shapes the training dynamics. We are happy to clarify any point before the deadline.

Reviewer_nsPrOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents the hidden layer phase scale, a statistic over the hidden sinusoidal layers of sinusoidal neural field. It shows that this scale is indicative of when the training of neural fields will collapse or succeed. The work shows that the largest phase scale is indicative of successful training. The phase scale comes from a single forward before training and without the target. Large phase scale over the hidden layers tends to lead to the collapse of the sinusoidal neural fields.

This work defines two threshold values for the phase scale using the Celeb-A dataset. It combines Monte Carlo over random banks of the sinusoidal layer parameters to define the thresholds. Training succeeds when the phase scale is lower than the lower threshold, but it collapses when the phase scale exceeds the upper threshold. The phase scales that land between the two thresholds produce mixed results between collapsed and successful training. This work establishes that the trainability of sinusoidal neural fields depends on the phase geometry at initialization.

Critical strengths

(1) The relevance of training collapse in sinusoidal neural field underscores the significance of this work. It tries to solve one of the main problems associated with using neural fields

(2) The paper is also well organized.

Critical weaknesses

(1). The abstract appears to be misleading. Here is an excerpt from the abstract: "Over 300 runs on three images, 181 of 181 runs train below a phase 1.81 and none of the 51 trains above 4.0". "181 of 181 runs" seems to suggest 181 runs instead of the 300 runs. Also, following sentence starts with "The failure does not ..." It is not clear what failure means. Does it mean outputting the mean color instead of the target?

(2). It is not clear how other factors (such as dataset, task, network depth) affect the values of the upper and lower threshold for the phase scale.

Questions for authors

(1) Are the thresholds dataset independent of dataset, task, and neural network depth (and not bank depth)?

(2) The summarized text under the abstract shows 181 out of 181 runs after stating 300 runs. Please check this.

(3). Is it possible to design a weight initialization method for sinusoidal neural fields such that their largest hidden phase scales are less than the lower thresholds?

3 comments
AuthorsOpenReview ↗

Response to Reviewer nsPr

We thank the reviewer for the positive assessment, and for noting that "the relevance of training collapse in sinusoidal neural field underscores the significance of this work" and that "the paper is also well organized." We answer each point below, and all changes are shown in blue.

Weakness 1 and Question 2. "181 of 181 runs" seems to suggest 181 runs instead of the 300 runs. ... It is not clear what failure means.

The abstract now defines a collapse and states how all 300 runs split. It says that a network "either fits its target or collapses to the mean colour of the target", and that "on three CelebA images, all 181 runs below κ = 1.81 train, none of the 51 runs above 4.0 trains, and the other 68 of the 300 runs have mixed outcomes." The conclusion uses the same term.

Weakness 2 and Question 1. Are the thresholds independent of dataset, task, and neural network depth (and not bank depth)?

The cutoffs held on a second dataset, while the depth, the kind of bank and the step size move them. Table 1 on page 6 collects every factor we varied.

  • Dataset. The cutoffs were set on CelebA and then fixed. On three Kodak-24 images, all 60 networks below 1.81 train and none of the 60 networks above 4.0 trains. Section 4.2 reports this new study.
  • Depth. κ itself does not change with depth, but a deeper network has a lower cutoff, as the correlation map predicts. A random bank at κ = 2.98 trains in all 15 runs at depths 3 and 4 and in none at depths 5 and 6 (Section 5.3).
  • Task. In an exploratory study, audio switches between κ = 1.14 and 2.44. The Burgers equation has no constant solution to collapse into, and there κ shows up in the error (Section 7.3).

κ gives the same axis in each of these settings, and the position of the cutoffs on this axis is measured for each kind of network.

Question 3. Is it possible to design a weight initialization method for sinusoidal neural fields such that their largest hidden phase scales are less than the lower thresholds?

Yes. Scaling down the hidden weights before training is such an initialization, and in our test it removed the collapse in all 60 networks. The confirmation study draws random banks for five-layer networks and trains each network once on one of three Kodak-24 images, with the training setting of the CelebA study. It also gives every network above 4.0 a rescaled copy. Starting at the first hidden layer, the copy multiplies the weight matrix of each hidden layer by one factor until the phase scale of that layer is 1.5. This takes one forward pass per layer and leaves the frequencies, the first layer and the head unchanged.

Table R1. The 60 networks above κ = 4.0 in the confirmation study.

NetworksRuns that trainMedian gain over the mean colour
Original0 of 600.0 dB
Rescaled to κ = 1.560 of 6035.5 dB
Drawn below κ = 1.81, for reference60 of 6027.0 dB

The rescaled copies train as reliably as the networks drawn below the lower cutoff. During training, κ rises above 1.81 in 27 of the 60 copies, mostly after epoch 200, and all 27 still train (Appendix B and Figure 5). SIREN gives a second example. Its initialization holds κ near 1, and it trains in all 72 runs under the STAF setting (Section 7.2). A deeper network or a larger step lowers the cutoff (Table 1), and the target value of κ should then be lower as well.

We hope these changes answer your questions, and we would be glad to discuss further.

Reviewer_nsPrOpenReview ↗

Authors have addressed my concerns.

Thanks for your response.

You have addressed my concerns under Q1 and Q2. For Q3, Table R1 strengthens the contribution of your work from just characterizing failed cases to suggesting how to mitigate it.

I am satisfied with your response and will upgrade my rating accordingly.

Dear Reviewer nsPr, thank you for checking our response and the revision, and for raising your rating. Your questions led us to add Table 1 and the rescaled networks in Table R1, and both made the paper clearer and more useful.

Reviewer_zNbuOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

In this paper the authors investigate why sinusoidal neural fields built on a randomly drawn “bank” of frequencies sometimes fail to train and instead output the mean of their target. They propose a quantity computable at initialization, the phase scale (the slowest frequency of a layer’s bank times the standard deviation of that layer’s input, maximized over hidden layers), and show that it separates runs that train from runs that collapse at the extremes. They then study how the boundary depends on bank family and depth, describe what failed runs do during training, and show that the initial output bias can flip outcomes in the mixed region.

Critical strengths

Strong Points: A few interventions are clean, e.g. rescaling the first layer versus hidden layers and the paired bias experiment on identical initial weights.

Critical weaknesses

Weak Points: The experimental design feels very ad hoc. The number of images, seeds and runs varies from experiment to experiment without justification. Many of the 30 experiments were added in response to failed predictions. The paper reads more as a record of an exploration than as a designed study supporting a general claim.

The theoretical component is limited. By the authors’ own account, the analysis in Appendices E and F explains the form of the phase scale but not where the threshold sits or why runs collapse, and the threshold itself has to be measured separately for each bank family.

I found the paper is extremely difficult and unwieldy to read. It is written as a dense sequence of short, declarative statements that make it hard to follow a line of reasoning or a core argument the authors wish to convey.

Questions for authors

My main question for improving the paper would be what general lesson the authors want readers to take away, and whether it holds beyond this specific setting. It would also help to restructure the experiments around a small number of pre-planned, adequately powered studies.

5 comments
AuthorsOpenReview ↗

Response to Reviewer zNbu

We thank the reviewer for the detailed review, and for noting that "a few interventions are clean, e.g. rescaling the first layer versus hidden layers and the paired bias experiment on identical initial weights." The comments led us to rebuild the evidence around pre-planned studies and to rewrite the whole paper. All changes are shown in blue. We begin with the main question.

Question 1. What general lesson the authors want readers to take away, and whether it holds beyond this specific setting.

The lesson is that the hidden layers at initialization largely decide whether a sinusoidal network trains, and that one forward pass before training measures this through κ. The revised introduction states this lesson and the three results that support it. The lesson holds beyond the setting where the cutoffs were set.

  • The cutoffs set on CelebA held on three new Kodak-24 images, with no error in 60 runs at either end (Section 4.2).
  • STAF draws a new bank in every layer. Under the initialization of its released code, none of the 32 runs below 1.81 collapses (Section 7.1).
  • In an exploratory study, κ also separates the outcomes on audio (Section 7.3).

The position of the cutoffs moves with the depth, the kind of bank and the step size. Table 1 collects these effects.

Weakness 1 and Question 2. The experimental design feels very ad hoc. ... It would also help to restructure the experiments around a small number of pre-planned, adequately powered studies.

We followed this suggestion. The main text now rests on four studies, each designed with its predictions before any run started (Section 3).

  • Calibration. 100 random banks on three CelebA images set the two cutoffs (Section 4.1).
  • Confirmation, new. 240 new runs on three Kodak-24 images test the fixed cutoffs (Section 4.2).
  • Fast terms and depth, rerun. Every cell has 15 runs, five seeds on each of three images (Section 5).
  • STAF, rerun. Five seeds on each of the 24 Kodak images for each initialization (Section 7.1).

The sizes follow from the design. With no error in 60 runs at each end, the error rate of each cutoff is below 4.9% at 95% confidence. The four predictions of the confirmation study and the nine of the two grid studies all held.

Table R1. The confirmation study, with both cutoffs fixed in advance.

Range of κRuns that train
Below 1.8160 of 60
1.81 to 2.3616 of 20
2.36 to 3.006 of 20
3.00 to 4.000 of 20
Above 4.00 of 60
Above 4.0, hidden layers rescaled to κ = 1.560 of 60

Both cutoffs hold on images that played no part in setting them, and lowering κ in the hidden layers removes every collapse. The earlier experiments, including those added after a failed prediction, now sit in Appendix L with every prediction and its outcome.

Weakness 2. The theoretical component is limited. ... the threshold itself has to be measured separately for each bank family.

The theory explains why the hidden layers decide the outcome and predicts how the cutoff moves with depth, and Section 5 measures where the cutoff sits. The original Appendices E and F are now Appendices D and C. Section 4.4 draws three consequences from the analysis.

  • κ is a frequency times a standard deviation over the hidden layers, and the first layer does not set the outcome. Two interventions confirm this (Section 4.3).
  • A deeper network has a lower cutoff. All five predictions of the depth study held (Section 5.3).
  • At a large κ, a linear head can fit almost nothing at initialization (Proposition 2), and Section 6 shows that this starts the collapse.

For random banks, one pair of cutoffs held on both datasets. Banks with one fixed frequency switch at a higher κ, and Section 5.2 traces this shift to the fast terms.

Weakness 3. I found the paper is extremely difficult and unwieldy to read. It is written as a dense sequence of short, declarative statements ...

We rewrote the whole paper with this comment in mind.

  • Introduction. Before the lesson, it explains κ as the angle that the slowest sinusoid turns when the input of a layer moves by one standard deviation.
  • Sections 4 to 7. Each section opens with its conclusion, and each subsection title states its result. Each sentence now follows from the one before, and the steps of each experiment moved to the appendix.
  • Section 6. It is rebuilt around the output of the head, y(x) = W h(x) + b, and shows step by step how a run collapses.
  • Appendix. It follows the order of the main text, and Table 2 maps each appendix to its section.

We hope the revision addresses your concerns. If it does, we would be grateful if you would consider raising your score. We would be glad to answer any further question.

AuthorsOpenReview ↗

Follow-up before the discussion closes

Dear Reviewer zNbu, thank you again for your review. The discussion period closes on November 18 AoE. We would like to know whether our response and the second revision address your concerns. The revision adds a confirmation study with both thresholds fixed before training and rewrites the text around a single argument. We are glad to clarify any remaining point before the deadline.

Design of study is improved. Writing quality is improved, although I have to say its difficult to exactly parse the soundness of the new manuscript given that the authors have added multiple thousands of new words, and complete redone many of the analyses, in some cases with what appear to be different architectures/initialisations? Can the authors comments on why there are such large changes in results for the STAF architecture?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the second revision

We thank the reviewers again for their comments. We have uploaded a second revision. Every change against the submitted version is shown in blue, and the submitted version remains available under Revisions. Text that was deleted, or moved without any change, cannot be shown in colour. The main text still fits in ten pages.

The reviewers found the problem important (nsPr, eRo9) and the paper well organized (nsPr). They noted that the interventions on the first layer and on the hidden layers place the failure in the hidden layers (eRo9, zNbu), that the correlation map motivates the form of κ (eRo9), and that the paired bias experiment on identical initial weights is clean (zNbu). The revision keeps these results and adds the following.

  1. A confirmation study in Section 4.2 and Appendix B. The two cutoffs set on CelebA were fixed in advance and tested in 240 new runs on three Kodak-24 images. All 60 networks below 1.81 train, and none of the 60 networks above 4.0 trains. When the hidden weights of these 60 networks are scaled down to κ = 1.5, all 60 train. (zNbu, nsPr)
  2. Uniform grids in Sections 5.2 and 5.3. The studies of the fast terms and of the depth were rerun with 15 runs in every cell, five seeds on each of three images. All nine predictions written before these runs held. (zNbu, nsPr)
  3. Training dynamics in Section 6. The section now follows the output head through a collapse, from the features at initialization to the mean colour. (eRo9, zNbu)
  4. A rewritten text. The introduction states the main lesson and the three results that support it. Section 3 lists the four studies behind the main text, each designed before its runs. Each section opens with its conclusion, and each subsection title states its result. The history of the experiments and every prediction moved to Appendix L. (zNbu)
  5. STAF under its own initialization in Section 7.1 and Appendix I. Our original STAF runs used Glorot weights, which the released code of STAF does not use. The new runs use the initialization of the released code and the amplitudes of the published paper, and a third network separates κ from the step size of the weights. Appendix I keeps the earlier runs, marked as the Glorot reproduction.
  6. Layout. The appendices now follow the order of the main text, and Table 2 on page 14 maps each appendix to its section. The correlation map of the original Appendix F is now Appendix C, and the original Figure 4 is now Figure 16 in Appendix J.

We reply to each review below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.