acceptodds
Under review as a conference paper at ICLR 2027

Stable initialization without the CLT

Abstract

Successful training of deep neural networks is highly dependent on the distribution of the initial weights. If the weights are too large, network training blows up; if they are too small, the model fails to learn features. Stable initialization is the optimal moderation between these two extremes. The conventional theory of random networks uses the Central Limit Theorem to control inter-neuron dependencies, which introduces distributional approximation error and coupling between layers. For networks with sine activations, we derive the uniform-phase initialization, which obviates distributional approximation and fully decouples the layers. Ours is the first work to use the sine function's periodic symmetry. Models trained with the uniform-phase initialization outperform the state of the art in neural representation tasks like image and audio fitting. We find that our untuned models are competitive with the best-tuned baselines from previous work and support P width scaling.

Primary area:
learning theory
Keywords:
neural network initialization, sinusoidal neural networks, implicit neural representations, signal propagation, maximal update parameterization, stable initialization, moment matching

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eA2JOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed to use a uniformly sampled bias distribution to replace the initialization in the sine-activated function. The authors claimed that the previous initialization methods might ignore the periodicity of the sine network. Instead, they argue for uniformly sampling bias on [0, 2pi], which will let the network have independent inputs to the weights or activations. In addition, the authors also tried to match the output distribution of the initialized network to the input distribution by tuning the initialized network weights. The authors have applied this strategy on 1d audio and 2d image fitting tasks, and the proposed method achieved good results.

Critical strengths
  1. This paper raised an important and interesting question: how to initialize the sine-activated network to make it effective for image/audio fitting? Generally, this is an important topic for a general sine-activated network.
  2. The authors have provided both theoretical analysis and quantitative results.
  3. The proposed method is simple and should be analyzed thoroughly to see if it has the potential to be generally used in sine-activated networks or other periodic-activated networks.
Critical weaknesses
  1. I feel like this paper has made relatively more general arguments than it actually proved. The authors have done experiments on 2d image and 1d audio fitting, which are lower-dimensional applications. It is sure that a paper could do experiments on these tasks, but the authors should narrow down the main claim to limit the discussion within these fields. The theoretical and quantitative results are not convincing to back up the authors’ claim.
  2. In addition, the entire paper should clearly state that it is based on the sine-activated network, not general to all architectures. The current version exaggerates the arguments and confuses the idea with a wider range of network structures.
  3. This paper’s analysis is based on the assumptions made in Lemma 1 that separate the activation and the bias from the input distribution. However, this is not valid for many vision tasks except for fitting. The authors did not provide any discussion, analysis, or experimental comparisons on this.
  4. This paper lacks a clear ablation study. Some of the recent methods are only mentioned in the related work section, but not compared in the main results.
  5. The paper writing needs substantial improvements. The entire paper is written in a dialog tone but with some strange words. The logic link between sentences needs to be reorganized, especially in the introduction section.
  6. The reference format is inconsistent. Papers are not correctly referenced.
Questions for authors

Please see my detailed comments above. My main concern is that this paper only focused on the sine-activated network for image/audio fitting tasks. However, the authors have exaggerated the proposed method’s performance and overclaimed for general use. The experiments are insufficient to back up the arguments. And the writing of the paper is unclear and needs a big improvement. I encourage the authors to rethink the motivations and decide what should be included in the paper and narrow down the argument to a specific setting.

5 comments
AuthorsOpenReview ↗

Preliminary clarifications

In this early phase of the rebuttal period, we seek clarifications so that we have concrete ideas to strengthen the pending revision.

I feel like this paper has made relatively more general arguments than it actually proved.

Can you please clarify which "relatively more general arguments"? We are happy to refine the paper's claims or to strengthen the theory where it falls short of the claims.

The current version exaggerates the arguments

Can you please clarify what claims are exaggerated? In our view, the primary contribution is Theorem 1, which is followed by a full proof in the main text.

The entire paper is written in a dialog tone but with some strange words. The logic link between sentences needs to be reorganized, especially in the introduction section.

Can you please identify the "strange words"? We prepared the main text of this paper entirely by hand, without AI, and are eager to improve the writing during rebuttal.

Can you also clarify which sentences lack a "logic link"? We take responsibility for the clarity of our writing and would be pleased to strengthen the Introduction in this way.

The reference format is inconsistent. Papers are not correctly referenced.

Can you give examples of unclear references?

the authors have ... overclaimed for general use.

Can you clarify which claims? We would like to strengthen our paper by aligning the claims better with evidence.

AuthorsOpenReview ↗

reminder #1

Dear reviewer eA2J, we are following up for your answers to these questions, as we look forward a constructive rebuttal discussion.

AuthorsOpenReview ↗

reminder #2

Dear reviewer eA2J, we are following up for your answers to these questions, as we look forward a constructive rebuttal discussion.

Reviewer_C9zVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper initializes sine MLPs with uniform-phase biases and Gaussian hidden weights. It proves that the expected Jacobian Gram matrices between hidden layers equal the identity for any fixed input, width, and depth. Data-based moment matching sets the input and output scales without tuning the input frequency; image and audio fitting are used to evaluate the method.

Critical strengths
  1. The periodicity argument gives a clean, exact finite-width result, with a short and convincing proof of the required independence.
  2. The zero-shot input/output calibration is practical. It matches or beats frequency-tuned baselines on most grayscale and RGB images (Tables 3-4), including a clear gain at width 16 (Table 2).
Critical weaknesses
  1. The central guarantee is an ensemble mean-square identity, not stability of a typical network or of actual loss gradients. Figure 3 shows spectral broadening with depth, and the loss-gradient claims in Appendix A.3 use an independence heuristic. The abstract and conclusion overstate what is proved; the claims should be narrowed or supported with typical gradient measurements.
  2. The experiments do not isolate the source of the gains: phase biases, input scaling, and output mean/covariance all change together. The baselines have no output bias (Appendix C), which particularly confounds the large CIELAB gains. Matched output calibration and component ablations are needed before attributing those gains to the proposed initialization.
  3. Several RGB wins are only about 1-3% (Table 4), but no across-seed variation is reported. More seriously, Section 4.2 reports 29.33 dB for the proposed audio model and claims it beats CVP26 by nearly 20 dB; Figure 5 labels them at 25.1 dB and 29.4 dB, respectively, reversing that comparison. The audio result needs correction.
Questions for authors
  1. How much does each component contribute? Please compare the phase-only, input-matching-only, and output-matching-only variants, and apply the same output bias and scale to the CIELAB baselines.
  2. Are the image and width-16 gains robust across independent seeds? Please report the mean and variation for the main comparisons.
  3. Which audio SNR values are correct, and what caused the Section 4.2/Figure 5 disagreement?
  4. Can you report median or quantile Jacobian singular values and early-training gradient norms versus depth to support the stated stability claim beyond the ensemble mean?
6 comments
AuthorsOpenReview ↗

Rebuttal 1/3

Thank you for your review. It raises a couple of minor corrections and is helpfully attentive to details, but we respectfully submit that it has altogether missed the point of the paper. We invite a constructive discussion during rebuttal in case there is something that we have missed in this response.

The point of this paper is not to achieve a substantive improvement over CVP26 in empirical performance. In the submitted manuscript, we have deliberately hedged such claims. Our method is "competitive with the best-tuned baselines from previous work" (L19–20, emphasis added) and "on par with a grid search over prior hyperparameterizations" (L480, emphasis added). This paper should not be read as a work which finds an empirical weakness, iterates architectural adjustments, and reports empirical improvement. Such a work would rightly face a high bar for empirical validation. But our contribution is not empirical.

Like CVP26, this is a theoretical paper that targets the root cause of poor generalization in SIRENs, namely, gradient explosion. With the exception of narrow networks (which vindicate our theory in a contrived setting), we expect a null result compared to the CVP26 baseline. Aside from the input scale, our remaining guarantee is gradient stability in the hidden layers. (This was attempted heuristically in SM20, which devotes 4 pages of the appendix to initialization, but, as CVP26 points out, ultimately fails to ensure gradient stability.) CVP26 has already achieved this goal by other means.

The derivation in CVP26 takes 10 pages in order to guarantee the stability of the mere preactivations. Its gradient stability argument is non-rigorous and treats the hidden layer Jacobians as independent even though they are not (Appendix A.4). While independence is a relatively benign step to a plausible conclusion, still it is an intuitive step, not a mathematical one.

Our stable initialization result marks a step change compared to the theoretical state of the art for sine networks, in three ways. It is more correct, it is simpler, and it is stronger.

  • more correct The derivation found in SM20 repeatedly treats approximation ≈\approx as equality, which is not mathematically rigorous. While CVP26 has a rigorous result on preactivations, the gradient derivation—core to the stability claim—is not rigorous. Until our work, the problem is not yet solved at the standard of mathematical proof.
  • simpler Our proof is substantially shorter and more elementary than the theoretical contributions of both SM20 and CVP26. This reflects the novelty of uniform phase initialization. At least as early as "Understanding the difficulty of training deep feedforward neural networks" (Xavier Glorot and Yoshua Bengio, AISTATS 2010), it has been assumed that hidden layers have an intractable distribution and that second-moment control would require some kind of approximation. We turn the clock back to 2010 and take a different path: we make no approximation.
  • stronger Our gradient stability claims that EDhDh⊺=EDh⊺Dh=I\mathbb E Dh Dh^\intercal = \mathbb E Dh^\intercal Dh = I are, to our understanding, without precedent in their exact equality. Our result implies the second-moment results of CVP26 (Appendix A.4), but not the other way around; therefore it is a strictly stronger theory.
AuthorsOpenReview ↗

Rebuttal 2/3

The central guarantee is an ensemble mean-square identity, not stability of a typical network or of actual loss gradients. Figure 3 shows spectral broadening with depth, and the loss-gradient claims in Appendix A.3 use an independence heuristic.

We clarify that by stable initialization, we are referring to second moment stability of the gradients, as pursued by Glorot and Bengio (2010). Our solution is exact, and exactness is the strongest possible guarantee that a mathematical theory can offer.

Figure 3 is a diagnostic offered for comparison to CVP26, where the counterpart result appears in neither the hypothesis nor the conclusion of any theoretical statement. It is a noteworthy observation, but is not dispositive as to the usefulness of our theory and at this point only raises questions without providing answers.

The abstract and conclusion overstate what is proved; the claims should be narrowed or supported with typical gradient measurements.

Can you please clarify which claims are overstated?

The experiments do not isolate the source of the gains: phase biases, input scaling, and output mean/covariance all change together. The baselines have no output bias (Appendix C), which particularly confounds the large CIELAB gains. Matched output calibration and component ablations are needed before attributing those gains to the proposed initialization.

Respecting CIELAB, we maintain that zero bias is the right choice for baselines, because it aligns with the official reproduction code provided by the authors of CVP26 (github, archived). Keeping this convention is crucial to the faithfulness of our Figure 1, which replicates CVP26's Cameraman figure (including CVP26, SM20, and non-SIRENs).

However, in the interest of abundant thoroughness, we have carried out a second set of ablations on the CIELAB image fields. In these tests, we divide the baseline CIELAB targets by 100, putting the lightness channel in [0,1][0,1] and the signed chromatic channels on a comparable scale. As a result, we find just as in RGB that our method is comparable to the best-tuned baseline SIRENs.

ImageUP-MM (no frequency tuning)SM20 (best frequency)CVP26, σa=0\sigma_a=0 (best frequency)CVP26, σa=1\sigma_a=1 (best frequency)
Hubble18.1619.0518.4919.21
Skin45.2047.1346.9546.76
Astronaut20.0420.4821.2021.50
Coffee15.7416.3416.6017.05

More seriously, Section 4.2 reports 29.33 dB for the proposed audio model and claims it beats CVP26 by nearly 20 dB; Figure 5 labels them at 25.1 dB and 29.4 dB, respectively, reversing that comparison. The audio result needs correction.

This was an oversight partly due to a bug that we fixed during rebuttal. The correct results are SNRs of 29.32dB (ours), 32.22dB (CVP26), and 26.31dB (SM20).

Several RGB wins are only about 1-3% (Table 4), but no across-seed variation is reported. 2. Are the image and width-16 gains robust across independent seeds?

We note that most of these experiments were run on the commercial GPU cloud, and to run multiple-seed trials would exceed the computational budget for this experiment. Likewise, our primary baseline, CVP26, only reports single-seed results for image fitting.

We argue that our image fitting experiments already capture a kind of variability by virtue of the fact that we use many more images (four grayscale and four color) than prior works such as CVP26. Our narrative holds that across all images, our method, without tuning, is competitive with baseline methods, with tuning.

We appreciate the suggestion and will consider a limited version during rebuttal.

AuthorsOpenReview ↗

Rebuttal 3/3

How much does each component contribute? Please compare the phase-only, input-matching-only, and output-matching-only variants, and apply the same output bias and scale to the CIELAB baselines.

Can you report median or quantile Jacobian singular values and early-training gradient norms versus depth to support the stated stability claim beyond the ensemble mean?

Thank you for the suggestions. We will test them as budget allows during the rebuttal.

Reviewer_oEvWOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1(−1)
Read review
Paper summary

The paper proposes uniform-phase initialization for sinusoidal MLPs. Uniformly distributed biases decouple activation derivatives from the weight matrices at each fixed input, yielding exact preservation of hidden-layer Jacobian second moments at finite width and depth. The method is combined with data-dependent calibration of the input and output layers through output moments and a normalized structure tensor. Experiments evaluate the combined method on image and audio fitting, compare it with input-frequency sweeps for earlier SIREN initializations, and explore width-dependent learning-rate scaling.

Critical strengths

The uniform phase decoupling argument is clean, and Theorem 1 appears correct under the stated assumptions. Obtaining exact finite-width identities without a Gaussian approximation is a useful analytical observation. The resulting initialization is also straightforward to implement..

Critical weaknesses
  1. The demonstrated contribution beyond prior work is too limited. Uniform random phases are established in random Fourier features (Rahimi and Recht, “Random Features for Large-Scale Kernel Machines,” NeurIPS 2007), while sinusoidal initialization, spectral control, and data-dependent initialization have been studied by Combette et al. (2026), Novello et al. (2025), Yeom et al. (2025), and Alsakabi et al. (2026). These works do not establish the same multilayer independence result, so I recognize the distinction. However, the main additional guarantee here is exact preservation of ensemble Jacobian second moments. The paper does not establish why making this particular statistic exact constitutes a substantial advance in initialization quality. Appendix A.4 shows spectral broadening with depth, and Appendix A.3 still relies on an independence approximation for the loss cotangent. Thus, removing the CLT from the moment calculation does not resolve the broader training-stability problem. The claims of an exact solution to stable initialization and the strongest possible guarantee substantially exceed the demonstrated scope.

  2. The experiments do not establish the practical value of the central contribution. Uniform-phase hidden initialization is evaluated together with changes to input frequencies, output covariance, and output mean. Crucially, Appendix C states that the baseline output layers have no bias, whereas the proposed method sets its output bias to the empirical target mean. This is consequential for the CIELAB experiments: for example, the Skin image has a first-channel mean of 72.99, which the proposed method incorporates at initialization. Without equivalent target centering and output scaling for the baselines, the large CIELAB gains cannot establish an advantage of uniform-phase initialization over simpler data calibration. The narrow-network experiment also retains the baseline frequency from the wider-network setup, so it does not isolate the benefit of avoiding a finite-width approximation. The paper reports no trained-model ablation separating these effects and no uncertainty across seeds for the fitting comparisons. Relevant recent methods cited in Section 1.3, such as weight-scaling initialization and TUNER, are not evaluated. Finally, the audio results are inconsistent: Section 4.2 reports 29.33 dB for the proposed method and an approximately 20 dB advantage over CVP26, whereas Figure 5 reports 25.1 dB and 29.4 dB, respectively.

References cited in this review

  • Rahimi and Recht (2007): Random Features for Large-Scale Kernel Machines. Advances in Neural Information Processing Systems 20 (NIPS 2007).

  • Combette et al. (2026): A New Initialization to Control Gradients in Sinusoidal Neural Networks. International Conference on Learning Representations (ICLR 2026).

  • Novello et al. (2025): Tuning the Frequencies: Robust Training for Sinusoidal Neural Networks. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2025), pp. 3071–3080. This is the TUNER method.

  • Yeom et al. (2025): Fast Training of Sinusoidal Neural Fields via Scaling Initialization. International Conference on Learning Representations (ICLR 2025). This is the weight-scaling initialization method.

  • Alsakabi et al. (2026): JA-SIREN: Deterministic Initialization for Sinusoidal Networks via Spectral Matching. arXiv preprint arXiv:2606.06671v1.

Questions for authors
  1. Can you separate the effects of uniform-phase hidden initialization and data-dependent input/output calibration in trained-model experiments? In particular, does the advantage persist when the baselines receive equivalent output centering, covariance scaling, and output-bias parametrization?

  2. What evidence shows that finite-width exactness itself improves learning? For the narrow-network claim, can you compare against baselines with frequency and learning rate selected for that width, while controlling input/output calibration and reporting variation across seeds?

  3. How does the method compare with applicable recent sinusoidal initialization methods cited in Section 1.3, particularly Yeom et al. (2025) and Novello et al. (2025), under comparable training and tuning budgets? What practical advantage remains over these approaches?

  4. Which audio results are correct: Section 4.2 or Figure 5? Please reconcile the numerical discrepancy and provide the corresponding configurations, since the two presentations imply opposite rankings between the proposed method and CVP26.

7 comments
AuthorsOpenReview ↗

Rebuttal 1/3

Thank you for your review. It raises a couple of minor corrections and is helpfully attentive to details, but we respectfully submit that it has altogether missed the point of the paper. We invite a constructive discussion during rebuttal in case there is something that we have missed in this response.

The point of this paper is not to achieve a substantive improvement over CVP26 in empirical performance. In the submitted manuscript, we have deliberately hedged such claims. Our method is "competitive with the best-tuned baselines from previous work" (L19–20, emphasis added) and "on par with a grid search over prior hyperparameterizations" (L480, emphasis added). This paper should not be read as a work which finds an empirical weakness, iterates architectural adjustments, and reports empirical improvement. Such a work would rightly face a high bar for empirical validation. But our contribution is not empirical.

Like CVP26, this is a theoretical paper that targets the root cause of poor generalization in SIRENs, namely, gradient explosion. With the exception of narrow networks (which vindicate our theory in a contrived setting), we expect a null result compared to the CVP26 baseline. Aside from the input scale, our remaining guarantee is gradient stability in the hidden layers. (This was attempted heuristically in SM20, which devotes 4 pages of the appendix to initialization, but, as CVP26 points out, ultimately fails to ensure gradient stability.) CVP26 has already achieved this goal by other means.

The derivation in CVP26 takes 10 pages in order to guarantee the stability of the mere preactivations. Its gradient stability argument is non-rigorous and treats the hidden layer Jacobians as independent even though they are not (Appendix A.4). While independence is a relatively benign step to a plausible conclusion, still it is an intuitive step, not a mathematical one.

Our stable initialization result marks a step change compared to the theoretical state of the art for sine networks, in three ways. It is more correct, it is simpler, and it is stronger.

  • more correct The derivation found in SM20 repeatedly treats approximation ≈\approx as equality, which is not mathematically rigorous. While CVP26 has a rigorous result on preactivations, the gradient derivation—core to the stability claim—is not rigorous. Until our work, the problem is not yet solved at the standard of mathematical proof.
  • simpler Our proof is substantially shorter and more elementary than the theoretical contributions of both SM20 and CVP26. This reflects the novelty of uniform phase initialization. At least as early as "Understanding the difficulty of training deep feedforward neural networks" (Xavier Glorot and Yoshua Bengio, AISTATS 2010), it has been assumed that hidden layers have an intractable distribution and that second-moment control would require some kind of approximation. We turn the clock back to 2010 and take a different path: we make no approximation.
  • stronger Our gradient stability claims that EDhDh⊺=EDh⊺Dh=I\mathbb E Dh Dh^\intercal = \mathbb E Dh^\intercal Dh = I are, to our understanding, without precedent in their exact equality. Our result implies the second-moment results of CVP26 (Appendix A.4), but not the other way around; therefore it is a strictly stronger theory.
AuthorsOpenReview ↗

Rebuttal 2/3

Corrections

Appendix C states that the baseline output layers have no bias

Thank you for pointing this out. Actually, we were wondering why the baselines perform worse than expected on CIELAB, and your explanation seems spot-on. We maintain that zero bias is the right choice for baselines, because it aligns with the official reproduction code provided by the authors of CVP26 (github, archived). Keeping this convention is crucial to the faithfulness of our Figure 1, which replicates CVP26's Cameraman figure (including CVP26, SM20, and non-SIRENs).

However, in the interest of abundant thoroughness, we have carried out a second set of ablations on the CIELAB image fields. In these tests, we divide the baseline CIELAB targets by 100, putting the lightness channel in [0,1][0,1] and the signed chromatic channels on a comparable scale. As a result, we find just as in RGB that our method is comparable to the best-tuned baseline SIRENs.

ImageUP-MM (no frequency tuning)SM20 (best frequency)CVP26, σa=0\sigma_a=0 (best frequency)CVP26, σa=1\sigma_a=1 (best frequency)
Hubble18.1619.0518.4919.21
Skin45.2047.1346.9546.76
Astronaut20.0420.4821.2021.50
Coffee15.7416.3416.6017.05
  1. Which audio results are correct: Section 4.2 or Figure 5? Please reconcile the numerical discrepancy and provide the corresponding configurations, since the two presentations imply opposite rankings between the proposed method and CVP26.

This discrepancy is an honest mistake in between revisions of our manuscript. We suspect the technical reason is that JIT compilation in JAX is non-deterministic, and this led to a glitch in training on the run which produced the audio figures. When set to deterministic JIT compilation, the results are SNRs of 29.32dB (ours), 32.22dB (CVP26), and 26.31dB (SM20).

AuthorsOpenReview ↗

Rebuttal 3/3

Clarifications

We will address question 1 in a later response pending further experiments.

Appendix A.3 still relies on an independence approximation for the loss cotangent

We clarify for the sake of comparison that while this is an approximation found in CVP26, it is not the only approximation.

Thus, removing the CLT from the moment calculation does not resolve the broader training-stability problem.

How to study training dynamics accounting for the correlation between loss gradients and parameters remains an open problem in machine learning. We provide a heuristic calculation for comparison to CVP26 but do not claim to make theoretical progress in this area.

The claims of an exact solution to stable initialization and the strongest possible guarantee substantially exceed the demonstrated scope.

We clarify that by stable initialization, we are referring to second moment stability of the gradients, as pursued by Glorot and Bengio (2010). Our solution is exact, and exactness is the strongest possible guarantee that a mathematical theory can offer.

What evidence shows that finite-width exactness itself improves learning?

We nowhere claimed that finite-width exactness improves learning in general. Our narrow-network example stresses the "scaling law" of asymptotic approximations by exploring the leftmost region of the curve. You raise an interesting ablation which we may pursue during rebuttal.

How does the method compare with applicable recent sinusoidal initialization methods cited in Section 1.3, particularly Yeom et al. (2025) and Novello et al. (2025), under comparable training and tuning budgets? What practical advantage remains over these approaches?

Theoretically, these works lack the theoretical guarantees of our exact gradient moments.

Yeom et al. (2025) solve a hyperparameter optimization problem related to the characteristic frequency, which incurs a runtime cost. Our moment matching initialization efficiently finds good hyperparameters using a plug-in estimator.

Novello et al. (2025) has an interesting heuristic for initialization, but it is significantly more conceptually complicated than the uniform-phase initialization.

Our main counterpart is the state of the art Combette et al. (2026), which we replicate and benchmark against.

We appreciate the suggestions and will consider them during rebuttal.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Why this paper should be accepted

Contrary to the tendency of the current reviews, this paper should be accepted to ICLR.

  1. The machine learning community almost abandoned gradient descent in the 90s due to the belief that it would get stuck in local minima. See reviews by Lecun, Bengio, and Hinton (2015) and Schmidhuber (2015).

  2. Researchers blamed exploding and/or vanishing gradients, e.g. in the context of autoencoders, see Hinton and Salakhutdinov (2006).

  3. Thus began a quest for initial weights with the property that early gradients are well-conditioned.

  4. The fundamental difficulty is that the initial gradient is intractable, because neuron values are correlated.

  5. In the past two decades, researchers have sought to control the initial gradient by approximating the initial preactivation distribution, e.g. by linearization (Glorot and Bengio 2010) or CLT-based mean-field theory (He et al. 2015).

  6. For ReLU in particular, the initial distributions have been worked out in higher fidelity by Hanin (2018), Burkholz and Dubatokova (2019), and Zavatone-Veth and Pehlevan (2021). These analyses are technically demanding, using sophisticated combinatorial tools from theoretical physics and giving their answers in terms of special functions.

  7. Sine as an activation function for deep networks was proposed by Parascandolo, Huttunen, and Virtanen (submitted to ICLR 2017) with optimistic preliminary results. Sitzmann et al. SM20 (2020) find sine strikingly effective for the neural representation task.

  8. But not for free. The main methodological challenge in the above two works is how to initialize the sine network so that training is well posed. Our primary baseline, Combette, Pustelnik, and Venaille CVP26 (ICLR 2026) addresses sinusoidal network initialization using the conventional CLT-based theory.

  9. Recall that CLT-based theory is not absolute. It is a solution to a problem, namely, the intractable initial distributions, (4), which is true for general MLPs.

  10. It is generally assumed that (4) holds for sine networks as well.

  11. But we point out it does not. The initial distribution of a sine network can be made tractable by initializing uniform biases. Our insight supersedes SM20 and CVP26 by eliminating the problem that they solve.

  12. (4) is no longer true. Not only are neurons uncorrelated, they are independent (between and within layers) at initialization.

  13. As a result, we are able to guarantee exact second moments of Jacobians and gradients that are intractable for general activation functions. Our result is stronger, and our proof is elementary, without limits, special functions, or fixed points.

  14. Our initialization achieves the objectives of (5) exactly in 1 page, where prior work achieves them approximately in 10 pages. This reverses the prevalent intuition that sine networks are theoretically difficult to analyze, requiring e.g. frequency-domain expansion around 0.

  15. Moreover, our exact moment results allow us to prescribe characteristic input and output scales with no hidden constant or need for tuning. This is a one-line solution to a hard problem that has been previously addressed by hand-tuning (SM20), meta-learning, optimization, and discrete sine transform 1, 2.

  16. Our empirical results support the machine learning community's consensus (1–2) that stable gradients make for effective learning—a null hypothesis we do not reject. Our initialization, without tuning, is competitive with CVP26 and baselines, with tuning.

  17. We indicate a new paradigm for thinking about random neural nets. Following the rich tradition of case study, toy model, and Gedankenexperiment in theoretical physics and rational mechanics, we complement current universal theories by analyzing a specific case with surprising, explicit properties, in order to shed light on a 20-year-old practical problem.

AuthorsOpenReview ↗

Frequently Asked Questions

Is there any precedent for the claim that neurons are independent at initialization?

Across layers: no.

Within layers: yes—as a simplifying assumption. See Section 2, "Gaussian Pre-Activations in Neural Networks: Myth or Reality?" by Wolinksi and Arbel (2025).

We prove what others assume. (Lemma 1)

What is original about uniform phase, when it has been used before in random Fourier features?

It is true that other works have observed translation invariance of the uniform-phase sinusoid process. However, our work targets a different application, namely, a deep neural network where not only the coefficients but also the frequencies are being learned. Our key observation is that translation invariance of the sine vector implies independence of components.

Moreover, we also make the connection that uniform phase arises as the limit of CVP26 with infinite bias variance. Accordingly, their theory replicates our theoretical result, albeit with a semi-rigorous independence heuristic (see answer above).

Do the experiments prove the claim that uniform-phase initialization gives a better loss than baselines?

No, and that is not our claim. We claim that:

  1. our hidden layer stability is at least as good as CLT-based initialization (this is a theoretical contribution), and
  2. our input/output calibration eliminates the need for hyperparameter tuning.

Both of these claims are backed up by experiments that replicate CVP26, with an emphasis on the headline application to neural representation (SM20) that made SIRENs so successful.

We have also gone above and beyond by trying multiple pseudorandom seeds as well as ablating output location/scale calibration.

AuthorsOpenReview ↗

Conclusion

While we regret that we have not been able to answer review concerns comprehensively during the author-reviewer discussion period, we appreciate the attention to detail, especially to the correctness of our theoretical appendices and the soundness of our empirical investigation. This interaction has also strengthened the presentation of the significance of our theoretical result.

We entreat a fair consideration of this paper's originality and contribution, and we thank the reviewers and AC for their consideration of this paper.

AI disclosure: all rebuttal responses were drafted by hand, but we used AI to assist with literature survey.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.