acceptodds
Withdrawn from ICLR 2027

NASA-INR: Non-linear Adaptive Spectrally-spanning Activation for Implicit Neural Representation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Implicit neural representations (INRs) have emerged as a powerful approach to model complex data offering continuous, resolution independent, and differentiable representations. However, existing methods do not explicitly address symmetry induced spectral attenuation without relying on complex modulation. To counteract this effect, we introduce a non-linear adaptive spectral spanning activation(NASA). We also introduce a non-linear phase expansion (NOPE) for activations in INR. Using harmonic analysis and Chebyshev polynomials, we showcase that the resulting activations contain both odd and even polynomial components, leading to a reduced spectral attenuation in the activation. We test where NASA stands among other existing activations. Furthermore, we investigate how the addition of NOPE to existing activations and adaptive forms of existing activations with tunable parameters compare with their non-modified forms. Finally, the resulting models are further evaluated on image reconstruction, inpainting, super-resolution, and denoising tasks against their classic counterparts. Through our work, nearest SOTA methods were surpassed by a mean increment of 2.57 dB in image reconstruction, 0.61 dB in super resolution, 0.26 dB in image inpainting 0.26. Overall, the study investigates phase asymmetry as a possible design dimension for improving the spectral representation capabilities of implicit neural networks.

Primary area:
learning theory
Keywords:
Implicit Neural Representations, Spectral Bias, Activation Functions, Chirp Functions, Harmonic Analysis, Image Representation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_4M9qOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Building on the observation from COSMO-INR that odd/even symmetric activation functions have alternating zero Chebyshev coefficients, which may lead to spectral attenuation, the authors propose a real-valued nonlinear phase expansion, NOPE:u↦u+ku2+1+η3k2u3,u \mapsto u + ku^2 + \frac{1+\eta}{3}k^2u^3, where the quadratic term breaks the symmetry and the cubic term ensures monotonicity of the phase map. Based on this formulation, the authors further introduce the NASA activation. The method is evaluated mainly on image-related INR tasks, including image fitting on nine selected Kodak images, as well as single-image inpainting, super-resolution, and denoising. The paper also provides an NTK analysis and visualization.

Critical strengths
  1. The experimental results on image-related tasks are strong.
  2. The derivation showing how the cubic term prevents phase folding is clear.
Critical weaknesses
  1. I have concerns about the main motivation based on activation symmetry. SIREN/FINER activations are indeed odd as scalar functions of their pre-activation input. However, an actual neuron takes w⊤x+bw^\top x+b as input to the activation, and with a nonzero bias, sin⁡(w⊤x+b)\sin(w^\top x+b) is no longer an odd function of xx. And the oddness of the activation itself does not imply a lack of completeness or insufficient representation capacity. Therefore, it is unclear to me whether the alternating-zero Chebyshev coefficients of the scalar activation constitute a meaningful limitation at the network level.

  2. The experiments are currently focused mainly on images. At least, I believe the full results on all 24 Kodak images should be reported in the supplementary material, since this evaluation is inexpensive. More broadly, INR methods can naturally be evaluated on more modalities, as the authors themselves state: “This approach has been applied to audio, images, 3D objects, and high-dimensional signals.” Moreover, compared with the baselines, the proposed design introduces several changes simultaneously, including the tanh window, the asymmetric phase, and the learnable T/cT/c parameters. A causal ablation study is needed to see which component is responsible for the improvement.

Questions for authors
  1. What are the individual contributions of the window function, the asymmetric phase, and the learnable activation parameters?

  2. How does NASA compare with COSMO-INR under the same experimental setting?

  3. What are the mean and variance over all 24 Kodak images using at least three random seeds?

Reviewer_7ySMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

NASA-INR proposes two modifications to activation functions for INRs. NOPE introduces a quadratic-cubic transformation of the pre-activation to break its odd/even symmetry and obtain both parity classes of Chebyshev coefficients. NASA combines this transformation with a tanh window and two learnable parameters, T and c, to adapt the activation at each layer. They provide Chebyshev and NTK analyzes and evaluate their approach on image reconstruction, inpainting, super-resolution, and denoising.

Critical strengths

NASA provides a simple alternative to complex-valued modulation, with two learnable activation parameters per layer. The analysis in Appendix A.1 shows that the quadratic–cubic phase remains monotonic under the conditions c ≥\geq 0 and η\eta > 0. This is a useful property because it prevents phase folding while retaining the asymmetric quadratic component.

Critical weaknesses
  1. I am not fully convinced by the motivation and the explanation for why the proposed activation improves reconstruction. First, the authors state in lines 57–60 that layer-wise adaptive activations are a comparatively underexplored direction. However, several existing works have already investigated this problem. For example, MIRE explores choosing different activation functions for individual layers, while STAF learns the frequency, amplitude, and phase parameters of sinusoidal activations. Earlier work on locally adaptive activation functions also introduced trainable parameters at different layers. The authors acknowledge some of these methods in their related work (lines 93–110), but I think they should explain more clearly what is missing from these approaches and what is fundamentally different about the proposed adaptation.

My main concern is with the motivation for breaking activation symmetry (section 3.1). The authors argue that introducing both odd and even Chebyshev coefficients improves spectral representation. While the analysis shows that NOPE introduces these components, I am not convinced that this necessarily explains better reconstruction in a complete INR. For example, even a standard sinusoidal activation with a nonzero bias contains both sine and cosine components when expressed in terms of the input coordinates. I think they need to better explain why activation symmetry remains a significant limitation when the network already has learnable biases.

Another issue is that NASA introduces several modifications simultaneously: a nonlinear phase transformation, a tanh window, and learnable T and c parameters. There are no sufficient ablations to understand which component is responsible for the improvement.

I also have concerns about the NTK interpretation. The authors associate stronger diagonal values and larger eigenvalues with improved high-frequency learning. However, the eigenvalue distribution alone does not tell us which spatial frequencies the network learns faster. Similarly, a stronger diagonal does not by itself establish shift invariance. I think these claims need either a more careful explanation or additional experiments examining the frequency content of the NTK eigenfunctions.

  1. The experiments are limited to support the claimed improvements over existing methods. First, some essential implementation details are missing, including network depth and width, learning rates, training iterations, and initialization settings. The authors mention using Adam and an RTX 6000 Ada GPU (lines 290–297), but this is not enough to reproduce the experiments or assess whether the baselines are fairly configured. Activation-based INRs can be particularly sensitive to initialization and frequency-related hyperparameters.

For image reconstruction, the authors state that they evaluate all 24 Kodak images (line 290), but Table 2 only reports nine selected images using a single seed. I would like to see the results on the complete dataset or on the DIV2K dataset, as is done in the literature. The results of the proposed modifications are also not consistent. Based on Table 2, NOPE improves SIREN by approximately 0.58 dB on average, but decreases FINER by 0.62 dB and WIRE by 2.66 dB. Similarly, introducing adaptive parameters does not meaningfully improve SIREN or FINER on average, and substantially reduces WIRE's performance. This raises an important question about the claimed benefits of symmetry breaking and layer-wise adaptation.

The inverse-problem experiments are also limited. Inpainting is evaluated on a single Celtic-knot image with 20% observed pixels (lines 424–431). Super-resolution uses one image at three downsampling scales (lines 435–445), and denoising is tested on one image with a single noise setting (lines 469–476). These experiments need to cover more images and degradation settings to support the generality of the method.

I would also like to see stronger comparisons with closely related work. COSMO-INR is particularly relevant because the paper directly builds on its argument about activation symmetry, yet it is not included in the experiments.

Questions for authors
  • I would like to see ablations that separate the contributions of nonlinear phase expansion, the tanh window, and the learned parameters. For example, how does NASA perform when the quadratic phase term is removed but still has the cubic term, or when the window is removed, or when T and c are fixed?

  • Does the NTK analysis include gradients with respect to the learned activation parameters T and c? How do the eigenvalue results support the claims about faster high-frequency learning?

Reviewer_aCG3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The manuscript introduces a modifies INR approach aimed at addressing "the issue of spectral bias arising from spectral attenuation due to the symmetry of activation functions". The contributions of the manuscript include (i) a new activation function, called {\it non-linear spectrally-spanning activation} function, shortened as NASA (?), and (ii) a layer-wise adaptive version of the activation function implementation. The activation relies on some locally adaptive variables. The manuscript presents a list of standard numerical demonstrations on 2D image reconstruction, inpainting, super-resolution and denoising.

Critical strengths

The numerical experiments cover multiple image processing tasks with some results being potentially competitive, especially for the 2D image reconstruction task using NASA.

Critical weaknesses

The manuscript lacks a clear logical structure. It introduces sequentially 3 ideas, the "non-linear phase expansion" (NOPE), the "non-linear spectrally-spanning activation function" (NASA) and the "layer-wise adaptive activation". None of these ideas is completely new and they are related to other results from the literature, as correctly stated. The major weakness, in my opinion, is that these ideas are not developed in a logical form and their interplay is not explained theoretically or practically. In particular, the numerical experiments are listed without clarifying the impact of each idea and without a critical discussion of the numerical results. For examples, the impact of NOPE in Table 2 and 3 is rather inconsistent but this is not explained or discussed; NASA appears very competitive on image reconstruction, comparable to SOTA for inpainting and super-resultion and less competitive for denoising but no explanation is provided. This is a missed opportunity to explain what the potential advantages of NASA could be compared to alternative methods. The conceptual justification for NASA is primarily heuristic so it is unclear to me what the method really offers compared to existing solutions. While the appendix contains some theoretical observations, these observations are not properly related to the rest of the manuscript.

Another weakness is the lack of comparison for the computing time.

Questions for authors

The author should include a table comparing the computing time and number of iterations of the different methods, for a proper comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.