acceptodds
Under review as a conference paper at ICLR 2027

Depth Calibrates the Effective Bandwidth of Sinusoidal Neural Fields

Abstract

The frequency-scale hyperparameter of a sinusoidal neural field, or , is commonly treated as the main control on what the network represents. The spectrum a trained network realizes departs from that nominal scale through weight rescaling and through the harmonics that layer composition generates, and how the two relate as the network deepens has not been characterized. We measure the realized spectrum directly, through an effective bandwidth defined as the 95% spectral-energy radius of the trained output on a dense grid, over roughly 7,000 fits across two datasets, two architectures and four depths. Near-optimal nominal values span 4-10.7x across depths while the realized spans only 1.5-2.3x. Depth amplifies spectral expansion. At a fixed knob the realized bandwidth grows from 18 to 727, and the operational expansion boundary shifts down with depth, [32, 16, 12, 12] on both datasets under a fixed 2x criterion. A refined sweep resolves that boundary into a steep, localized crossover, and near-optimal nominal scales lie no farther than its near edge. The geometry supports a target-spectrum-conditioned selector. It uses one ratio, fit on two shallow depths of one dataset, with the signal bandwidth estimated from training coordinates only. Over the completed candidate grid its mean regret is 0.20 dB on held-out depths, 0.59 dB on a held-out dataset without fitting a target-side bandwidth map, 0.52 dB transferring to an architecture-specific SIREN map and 0.12 dB on a 32-image held-out cohort. Its worst case is 2.35 dB, at one frequency just past the expansion boundary. Across depth the effective bandwidth is the more stable quantity, and it is the one to calibrate. All experiments concern controlled 2D image regression with sinusoidal-activation networks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
implicit neural representations, sinusoidal neural fields, SIREN, effective bandwidth, spectral bias, network depth, frequency initialization, hyperparameter calibration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mMnmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the relationship between nominal frequency hyperparameters and the spectrum realized by trained sinusoidal neural fields. The authors define effective bandwidth and systematically examine its behavior across frequency settings, network depths, architectures, and datasets. The paper presents three main findings: (1) the effective bandwidths of near-optimal configurations remain relatively stable across depths; (2) increasing network depth can amplify spectral expansion; and (3) a target-conditioned frequency rule achieves low regret across several held-out settings.

Critical strengths
  1. The paper provides a systematic empirical analysis showing that the effective bandwidths of near-optimal configurations remain relatively stable across network depths. The depth-amplified spectral expansion results and layerwise analyses provide useful intuition for understanding this phenomenon.

  2. The experimental setup is well-designed and clearly described.

Critical weaknesses
  1. While the authors acknowledge relevant prior work on harmonic generation and spectral control, its findings remain somewhat incremental. A clearer distinction from existing spectral analyses and a stronger discussion of the conceptual contributions would strengthen the paper.

  2. The study is primarily empirical, with limited explanation of the mechanisms behind the stability of near-optimal effective bandwidths and depth-amplified spectral expansion. Stronger connections to existing theoretical results would help explain these observations.

Questions for authors
  1. Given that the paper aims to study sinusoidal neural fields in general, what is the motivation for focusing on the frequency-bank architecture in the primary analyses? In particular, the sharp spectral expansion crossover appears more pronounced in the frequency-bank network than in standard SIREN, raising questions about whether this behavior is a general property of sinusoidal neural fields.

  2. Several key components of the paper rely on heuristic or empirically determined choices, including the 95% spectral-energy threshold used to define effective bandwidth, the 2$\times$ criterion for identifying the expansion boundary, and the calibrated ratio (ρ=0.78\rho=0.78). Could the authors clarify how sensitive the main conclusions are to these choices? In particular, what motivates the 95% threshold, and how well is the fixed bandwidth ratio expected to generalize beyond the evaluated settings?

  3. The paper focuses exclusively on sinusoidal neural fields, and it remains unclear whether the findings generalize to other non-sinusoidal neural field architectures, such as Gaussian, or WIRE. Do the observed spectral behaviors and calibration rule extend to non-sinusoidal neural fields? Although the authors acknowledge this limitation, experiments with additional neural field architectures would help establish the applicability of the observed spectral behavior and calibration rule.

  4. The experiments are limited to 2D image regression at a fixed resolution. How well do the findings transfer beyond fixed-resolution 2D image regression, particularly to 3D NeRF or PDE solving? Are the observations consistent across different image resolutions and training-coordinate sampling ratios?

2 comments
AuthorsOpenReview ↗

Response to Reviewer mNnm

e thank the reviewer for describing the analysis as systematic, the layerwise analyses as providing "useful intuition", and the setup as "well-designed and clearly described". We answer the questions in order and then the two weaknesses.

Question 1. "What is the motivation for focusing on the frequency-bank architecture in the primary analyses? ... raising questions about whether this behavior is a general property of sinusoidal neural fields."

We used the bank network as the main case because the knob of this network is an explicit frequency set that can be frozen, and SIREN reproduces the general findings. In the bank network we compare a frozen arm with a free arm. Their test PSNR differs by at most 0.30 dB, and the free frequencies move by 0.34% on average (Section 3.5). The measured bandwidth therefore comes from training with fixed frequencies. SIREN shows the same depth trend. Its optimal ω₀ moves from 16 to 12, 8 and 4 for L = 2 to 5, its near-optimal values span 4.0× in ω₀ against 2.15× in B_eff, and the selector transfers to it with a mean regret of 0.52 dB. The sharpness does not carry over. The largest adjacent B_eff ratio exceeds 20× for the bank and stays between 2.0 and 3.0 for SIREN (Section 5.2). In our data the depth shift and the concentration in B_eff hold in both architectures, and the localized crossover is specific to the bank.

Question 2. Sensitivity to the 95% threshold, the 2× rule and ρ = 0.78.

The selections change little under these choices, and Appendix D reports each case.

  • The 95% threshold. We use 95% as a common energy radius that keeps the bulk of the spectrum and leaves out the far tail. At 90%, every selection stays the same except the Kodak own-map selection at L = 4, which moves to a near-optimal value and lowers the regret there from 1.97 dB to zero. At 99%, the radius follows the expanded tail and the regret rises to 1.5 to 2.7 dB, which Section 7 lists as a limitation.
  • The 2× rule. Under 2× the boundaries are 32, 16, 12 and 12 on both datasets. Under 3× the L = 3 boundary disappears, because the added f_max = 20 splits one jump into two steps of 2.58 and 2.06 on CelebA. A refined sweep at L = 3 shows this jump as a steep crossover, with the peak log slope of 5.66 on CelebA and 4.25 on Kodak on the same interval [18, 20]. The rule detects this crossover on the grid, and the exact position depends on the threshold and the grid.
  • The ratio ρ. It was fit once on CelebA at L = 2 and 3 and then frozen. It gives mean regrets of 0.20 dB on held-out depths, 0.49 and 0.59 dB on Kodak, 0.52 dB on SIREN, and 0.06 to 0.12 dB in three runs made after all definitions were fixed (Table 1). Beyond these settings we have no evidence, and we do not treat ρ as a constant.

Questions 3 and 4. Non-sinusoidal fields, and tasks beyond 2D image regression at a fixed resolution.

Both are outside the present study, and Section 7 lists them as open. B_eff is defined on the output of any network sampled on a dense grid, and it applies unchanged to Gaussian or WIRE activations and to 3D fields. We have not run these cases. All results use 2D images at 128 × 128 with 25% training coordinates. Five different 25% splits keep the L = 3 regret at 0.00 dB and give 0.27 to 0.54 dB at L = 5 (Appendix D). Other sampling ratios and resolutions remain untested.

Weaknesses 1 and 2. The findings appear incremental, and the mechanism is mostly empirical.

Prior theory explains why composition expands the spectrum, and the paper measures what that theory leaves open. Yüce et al. (2022) and TUNER show that depth generates harmonics and bound the spectrum. They do not give the realized band of the good models or the depth at which the expansion turns steep in trained networks. The layerwise measurements support a cascade-like amplification account, with late-layer amplification in 32 of 32 past-boundary pairs and in 0 of 32 before the boundary (Section 5.3). They do not establish the mechanism, and the next version will connect them to the expansion of TUNER.

We hope these answers help, and we would be glad to discuss further.

Thank you to the authors for the detailed rebuttal. I appreciate the clarifications regarding the architectural differences and sensitivity analyses, which address some of my concerns.

However, my main concerns still remain. While the empirical findings are well presented, the conceptual novelty over existing spectral analyses is still limited. Moreover, the underlying mechanisms remain insufficiently explained, and the generalizability beyond the evaluated settings is not clear.

Overall, the rebuttal clarifies several points but does not substantially change my assessment of the paper's contributions.

Reviewer_UrgfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how frequency parameters (e.g., ω0\omega_0) affect the final spectral content of sinusoidal neural networks.

This paper is very confusing, from the title and abstract to the introduction and the rest of the paper. It is very hard to understand what it is really proposing. In general, it does not follow the common format of an INR paper and uses complicated vocabulary.

Critical strengths

Understanding and controlling the spectral content of sinusoidal networks remains an interesting problem.

Critical weaknesses

It is unclear what the near-optimal nominal values mean. Specifically, there are no clear units or reference values for the numbers given when referring to “nominal values”.

The partition of the training and test sets is hard to explain: with a 25% - 75% split, the network would have no problem learning the signal at the samples (42.9 dB in L238), but would struggle to generalize (21.6 dB in L239).

Recent works have approached the problem of controlling the bandlimits of SIRENs using hash grids (SASNet [1]) or different initialization schemes for deep networks (Combette et al. [2]). Recent work should be acknowledged and proper comparisons added to establish the paper's contribution.

[1] Feng et al. "SASNet: Spatially-Adaptive Sinusoidal Networks for INRs." CVPR. 2026.

[2] Combette et al. "A New Initialization to Control Gradients in Sinusoidal Neural Networks." ICLR. 2026.

Questions for authors

Could you improve the presentation of the method, clearly explaining what is proposed and how it works?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Urgf

We thank the reviewer for the review and agree that "understanding and controlling the spectral content of sinusoidal networks remains an interesting problem". We answer the question first and then each weakness.

Question. "Could you improve the presentation of the method, clearly explaining what is proposed and how it works?"

The paper proposes a measurement and a selection rule built on it, and it proposes no new architecture.

  1. The measurement. After training, we sample the network on a dense 512 × 512 grid and compute B_eff, the radius that holds 95% of the energy of the output spectrum (Equation 1).
  2. The finding. Across depths, the best test PSNR is reached at very different nominal settings, while the best models share a narrow band of B_eff (Section 4.2).
  3. The rule. To set the knob for a new depth or dataset, we pick the candidate whose B_eff is closest to ρ times the signal bandwidth B_sig of the target, with ρ = 0.78 fit once on two shallow CelebA depths (Section 6.1).

The next version will open the method section with these three steps.

Weakness 1. "It is unclear what the near-optimal nominal values mean. Specifically, there are no clear units or reference values."

A nominal value is the frequency multiplier set at initialization, and it has no fixed unit in the output. In the bank network f_max is the top frequency of the set geomspace(2, f_max, 8), and in SIREN ω₀ multiplies the input of each sine. Both act on the output of a dense layer. The frequency they produce in the output therefore depends on the trained weights, and this is why we call them nominal. B_eff is measured on the output, in radians per unit of the input coordinate, which spans [−1, 1]. The two scales should not be compared directly (Section 3.3). For reference, the SIREN default is ω₀ = 30, and our grids span f_max from 3 to 64 and ω₀ from 4 to 128. A near-optimal value is a setting whose mean test PSNR lies within 0.5 dB of the best one at that depth.

Weakness 2. "The partition of the training and test sets is hard to explain ... 42.9 dB in L238 ... 21.6 dB in L239."

The two numbers come from different networks, and the gap between them is the effect the paper studies. The 42.9 dB is the training PSNR at f_max = 48. The 21.6 dB is the best test PSNR, reached at f_max = 16. At f_max = 48 the test PSNR falls to 7.8 dB. We train on 25% of the pixels and test on the other 75% to measure how well the network fills in pixels it never saw. With all pixels in training, a higher f_max keeps improving the fit, and there is no optimum to locate. The split makes this optimum visible, and the paper asks how it moves with depth. The next version will state this purpose at the start of Section 3.1.

Weakness 3. SASNet and Combette et al.

We thank the reviewer for both references and will cite and discuss them in the next version. As the reviewer notes, SASNet controls the bandlimit with hash grids, and Combette et al. propose a new initialization that controls the gradients. Our paper changes neither the architecture nor the initialization. It measures the spectrum that standard trained networks realize and uses it to set the existing knob. B_eff can be measured on networks built with either method, and we will add this comparison.

Reviewer_ziTSOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies empirically the spectrum created by SIREN-type networks, i.e., networks that use sinusoidal nonlinearities. The focus is on measuring available bandwidth, culminating in an empirical approach to matching the target bandwidth of a signal to fit (the paper studies 2D image fitting problems as experimental proxy, i.e., results apply to low-dimensional function spaces).

The paper relates hyperparameters to spectral bounds, and emphasizes that depth is an important factor in creating high-frequency signal components (which is theoretically plausible and probably not even very surprising from a harmonic distortion perspective, where non-linear maps of spectra create high-frequency harmonics iteratively with repeated concatenation / depth). Finally, this can be, to some extend, helpful in tuning hyperparameters automatically to input signals.

Critical strengths

The paper follows a clear concept, which is interesting, also from a practical point of view: How do spectral properties of the considered class of DNNs change with parameters and depth? Experiments are clear and systematic, and a lot of background information is given. I would also appreciate the general idea of studying spectral profiles to understand function space priors of these architectures, representational capabilities and exploring mechanisms for automatic parameter / architecture tuning.

Critical weaknesses

I see two major weaknesses: Firstly, I did not see any very surprising new findings that would not be qualitatively clear from a naive theoretical prediction. Obviously, depth will create distortions that induce higher-frequency harmonics (in general, running a signal with a certain spectrum through a non-linearity creates higher-order distortions). However, I should stress that the detailed measurements of bandwidth profiles are beyond such qualitative predictions (I am not familiar with the literature that aims at making more quantitative predictions, as cited in the paper).

Secondly, I found the experimental design somewhat limiting: As we are just dealing with 2D input signals, I would have expected to see actual 1D or 2D power spectra plots (for example, measuring multiple random initializations of the network and assessing spectral profiles, or looking at residuals of fits over frequency bands). The notion of bandwidth is certainly a relevant factor to applications, but other effects such as spectral sparsity or induced frequency-space priors (sparsity, decays, gaps) might be equally important.

More generally speaking, the contribution / novel insight / potential impact of the paper seems limited, and in my perception remains below the bar for a top-level venue such as ICLR. Some additional work might be able to create a more interesting/potentially impactful submission; the subject of study as such does seem to have potential.

Questions for authors

What is the biggest / most important / most surprising new insight? How would it potentially impact theoretical understanding and/or practice in this area? I am happy to be convinced here that I am too pessimistic in my outlook, and I would thus consider addressing this question a central issue.

1 comment
AuthorsOpenReview ↗

Response to Reviewer ziTS

We thank the reviewer for finding the concept "interesting, also from a practical point of view" and the experiments "clear and systematic". Weakness 1 and the question ask the same thing, and we answer them together first.

Weakness 1 and Question. "I did not see any very surprising new findings that would not be qualitatively clear from a naive theoretical prediction." "What is the biggest / most surprising new insight? How would it potentially impact theoretical understanding and/or practice in this area?"

The most surprising result is that across depth the good models share a realized bandwidth and not a nominal setting. Harmonic generation predicts that a deeper network realizes more high frequencies at the same knob. It does not predict where the generalization optimum sits. On CelebA the optimal f_max moves from 32 at L = 2 to 3 at L = 5, and the near-optimal nominal values span 10.7× across depths. The measured B_eff of the same models spans only 1.52×. Kodak and SIREN show the same pattern, at 2.32× and 2.15× in B_eff against 10.7× and 4.0× in the knob (Section 4.2). Two further results go beyond the qualitative picture. The expansion boundary falls at f_max = 32, 16, 12 and 12 for L = 2 to 5 on both datasets, and a refined sweep shows a steep crossover on the same interval [18, 20] on both. A parameter-matched random Fourier feature network, which places the frequencies in the input encoding, shows no such shift with depth (Section 4.3).

For practice, the result gives a way to set the knob when the depth or the dataset changes. One ratio, ρ = 0.78, fit on two shallow CelebA depths, selects f_max from a bandwidth map. The mean regret is 0.20 dB on held-out depths, 0.59 dB on Kodak with the CelebA map and 0.12 dB on 32 fresh CelebA images (Table 1). An absolute target bandwidth taken from CelebA fails on Kodak at every depth, with a mean regret of 2.69 dB. For theory, the maps give a measured target. Yüce et al. (2022) and TUNER explain why composition expands the spectrum. Our measurements show which realized band the good models occupy and where the expansion turns steep, which a theory of trained networks would need to reproduce.

Weakness 2. "I would have expected to see actual 1D or 2D power spectra plots ... but other effects such as spectral sparsity or induced frequency-space priors (sparsity, decays, gaps) might be equally important."

We agree that a single radius summarizes the spectrum and does not show the shape. B_eff is the 95% energy radius of the output spectrum, and the paper uses it as one fixed measurement. The next version will add radial power spectra of trained outputs and of their residuals over several initializations, below and past the boundary. Sparsity, decay and gaps may also matter, and the paper does not test them. We will name them as open factors in Section 7.

We hope this answers the central question, and we would be glad to discuss further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.