acceptodds
Under review as a conference paper at ICLR 2027

Activation-Aware Initialization for Odd-Sigmoid Networks

Abstract

Activation functions and weight initialization are tightly coupled, especially in deep networks with bounded sigmoidal nonlinearities. Standard zero-mean Gaussian initializations, including Xavier, He, and edge-of-chaos variants, often become brittle in deep and narrow fully connected networks, where forward activations collapse and backpropagated gradients decay. We study this problem for a class of bounded, odd, monotone activations whose slopes decrease away from the origin, which we call odd-sigmoid activations. For an activation , our initialization sets a diagonal gain using the critical slope and adds Gaussian noise whose scale is calibrated by a terminal sign-flip surrogate. The scalar analysis gives a closed-form relation between the surrogate sign-flip target, depth, and noise scale, while an empirical depth schedule transfers this calibration to finite-width networks. Across deep fully connected networks on image benchmarks and PINN benchmarks, the proposed diagonal-plus-noise initialization is substantially less sensitive to depth, width, activation scaling, and learning-rate choice than Gaussian i.i.d. baselines. The results suggest that sign-flip calibration is a practical tool for training deep odd-sigmoid networks, while also highlighting the need for structured initialization beyond variance-only Gaussian schemes.

Primary area:
optimization
Keywords:
Weight initialization, Activation Function, Signal Propagation, Physics Informed Neural Networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SzEwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates how to design reliable weight initialization for deep neural networks with bounded odd-sigmoid activations, aiming to prevent signal attenuation, vanishing gradients, and training instability. Specifically, the paper extends identity-plus-noise initialization from tanh networks to a broader class of odd-sigmoid activations and proposes a depth-dependent noise calibration strategy based on the terminal sign-flip rate. Experimental results demonstrate that the proposed method enables more stable training of deep sigmoid networks and physics-informed neural networks (PINNs) compared with conventional Gaussian initialization methods, such as Xavier, He, and EOC.

Critical strengths

First, it focuses on a well-motivated application setting. Although modern deep learning architectures predominantly use ReLU, GELU, and residual connections, this does not diminish the importance of initialization for sigmoid-based networks, particularly in applications such as PINNs. Second, the paper provides a comprehensive theoretical analysis. The authors first define the odd-sigmoid function class and investigate the fixed-point structure of an iterative mapping based on the activation function and a scaling parameter. They identify a critical value determined by the reciprocal of the activation function's derivative at the origin, at which the fixed-point structure changes. This observation naturally motivates the activation-aware diagonal gain. The authors further derive the relationship between the terminal sign-flip probability, noise scale, and network depth using a scalar surrogate model. Third, the experimental evaluation is extensive, covering multiple odd-sigmoid activations, different network depths and widths, four classification datasets (MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100), various activation scaling factors, batch normalization, learning-rate sensitivity, and four PDE benchmarks. Overall, the experimental results provide convincing evidence that the proposed structured initialization is more stable than conventional Gaussian initialization methods in the tested deep odd-sigmoid networks.

Critical weaknesses

My primary concern is that the closest prior work already contains the main architecture-level insight, while the paper does not sufficiently demonstrate the practical value of its additional contributions. As acknowledged in the Related Work section, Lee et al. (2025) proposed an identity-plus-noise initialization for tanh networks. Their method similarly combines a deterministic diagonal-like matrix with Gaussian noise and uses fixed-point analysis to explain why this structure improves signal propagation in deep tanh networks. Consequently, a plausible alternative explanation is that most of the performance improvements reported in this paper arise from the previously established diagonal/identity structure rather than the newly introduced sign-flip calibration. Although sign-flip calibration is arguably the most distinctive component of the proposed method, the current experiments do not sufficiently demonstrate that it is responsible for the observed performance improvements.

Questions for authors

1.Since Lee et al. (2025) have already proposed an identity-plus-noise initialization for tanh networks, could the authors clarify the practical advantages of the proposed method over this closely related approach? In particular, is there a direct comparison under the same network architectures, activations, and training settings? Such a comparison would help determine whether the reported improvements primarily arise from the existing diagonal-plus-noise structure or from the additional contributions of this paper.

2.The proposed terminal sign-flip calibration appears to be the most distinctive component of the method. However, Figure 6 suggests that the deterministic initialization with zero noise already achieves performance comparable to the calibrated variants in several settings. Could the authors provide further evidence that sign-flip calibration offers meaningful improvements over the deterministic initialization or a simpler depth-dependent noise schedule? This would help establish whether calibration is essential to the observed training stability.

4 comments
AuthorsOpenReview ↗

Response to Reviewer SzEw (1/2)

We sincerely thank you for your time and thoughtful review. We have carefully addressed each of your concerns, and all changes are marked in red in the revised manuscript.

W1. The closest prior work, Lee et al. (2025), already contains the main architecture-level insight. The improvements may come from the diagonal/identity structure rather than from sign-flip calibration, and the experiments do not show that the calibration is responsible.

Q1. What are the practical advantages over Lee et al. (2025)? Is there a direct comparison under the same network architectures, activations, and training settings?

A. Thank you for this valuable suggestion. The original submission gave partial evidence. In Figures 17 and 18 (tanh, depths 50 and 100), the calibrated noise scale gives higher accuracy than σz=0\sigma_z = 0, but the differences there are small. We agree that clearer evidence is needed to separate the effect of the diagonal structure from the effect of the calibrated noise.

New experiment. Our initialization has two parts, W=D+ZW = D + Z. The backbone DD is a deterministic diagonal matrix with entries ω=1/f′(0)\omega = 1/f'(0), and ZZ is Gaussian noise with scale σz\sigma_z. Lee et al. (2025) use the identity as the backbone and a fixed noise scale of 0.085. We added a direct comparison that changes these two parts separately, under identical architectures and training settings (Table 1 in Section 5.2 and Table 9 in Appendix C.6).

The goal of our paper is an initialization that trains across the odd-sigmoid class and not only for tanh, so we test two activations outside the setting of Lee et al. (2025), 100tanh⁡(x)100\tanh(x) and 1000erf(x)1000 \mathrm{erf}(x). For tanh itself ω=1\omega = 1, and the two backbones coincide. All networks have width 64 and are trained for ten epochs. Entries are test accuracy in % (mean ± std over three seeds) at depths 50, 100, and 150.

Table 1. 100tanh⁡(x)100\tanh(x)

Backbone, noise σz\sigma_zMNIST, 50MNIST, 100MNIST, 150F-MNIST, 50F-MNIST, 100F-MNIST, 150
1. Lee et al. (2025): II, fixed 0.08567.56 ± 1.8767.55 ± 1.8767.56 ± 1.8864.87 ± 1.7064.87 ± 1.6564.73 ± 1.66
2. ωI\omega I, fixed 0.08510.91 ± 0.76failedfailed9.93 ± 0.31failedfailed
3. ωI\omega I, fixed 0.085ω0.085\omega10.83 ± 0.8977.30 ± 3.0963.09 ± 2.5854.47 ± 6.7553.35 ± 10.4753.98 ± 11.64
4. ωI\omega I, no noise66.91 ± 2.6264.65 ± 0.1938.37 ± 31.9538.30 ± 0.5119.53 ± 0.7912.14 ± 3.04
5. Ours: ωI\omega I, sign-flip calibrated93.25 ± 0.0793.51 ± 0.4791.81 ± 1.0484.08 ± 0.6483.80 ± 1.2682.62 ± 0.85

Table 9. 1000erf(x)1000 \mathrm{erf}(x)

Backbone, noise σz\sigma_zMNIST, 50MNIST, 100MNIST, 150F-MNIST, 50F-MNIST, 100F-MNIST, 150
1. Lee et al. (2025): II, fixed 0.08561.45 ± 1.9161.98 ± 2.0861.94 ± 1.3060.49 ± 0.2461.48 ± 0.1961.12 ± 3.19
2. ωI\omega I, fixed 0.08511.26 ± 0.4310.01 ± 0.459.90 ± 0.34failed12.48 ± 4.73failed
3. ωI\omega I, fixed 0.085ω0.085\omega12.76 ± 3.5970.06 ± 6.5613.93 ± 5.4042.44 ± 0.2644.67 ± 0.7843.91 ± 2.16
4. ωI\omega I, no noise66.27 ± 2.7065.65 ± 3.7149.29 ± 25.6238.63 ± 0.5638.67 ± 0.2013.06 ± 5.30
5. Ours: ωI\omega I, sign-flip calibrated88.86 ± 0.7491.63 ± 0.7091.55 ± 0.6282.67 ± 0.9582.95 ± 0.7582.47 ± 0.79

The gain needs both parts, the matched backbone and the calibrated noise. Our method is 18 to 30 points above Lee et al. (2025) in every column. The diagonal structure alone does not explain this gain. A diagonal backbone without calibrated noise stays far below our method, whether it is the identity (row 1) or matched to the activation (rows 2 to 4). Rows 3 to 5 share the backbone ωI\omega I and differ only in the noise, and replacing no noise or a fixed noise level by the sign-flip calibrated scale raises the accuracy by 16 to 82 points.

Practical advantages over Lee et al. (2025). Lee et al. (2025) is built for tanh, with the identity backbone and one empirically chosen noise constant for all depths, and on the two activations above it reaches 60 to 68%. Our method computes the backbone from the activation as ω=1/f′(0)\omega = 1/f'(0) and sets the noise scale for each activation and depth from a single target (p=0.4p = 0.4 in all our classification experiments). The same rule therefore applies to every odd-sigmoid activation without manual tuning, and it reaches 82 to 94% in the same setting.

AuthorsOpenReview ↗

Response to Reviewer SzEw (2/2)

Q2. Figure 6 suggests that the deterministic initialization with zero noise already performs comparably in several settings. Is there further evidence that sign-flip calibration offers meaningful improvements over the deterministic initialization or a simpler depth-dependent noise schedule?

A. Thank you for this helpful suggestion. We agree with your reading of Figure 6. In that figure the activation is tanh, for which ω=1\omega = 1. In this case, the initialization without noise performs about as well as the calibrated ones.

The noise becomes essential for the rescaled activations in Tables 1 and 9, where ω\omega is far from 1. Row 4 of the two tables in our answer to W1 and Q1 is the initialization without noise, with the same backbone as ours. With 100tanh⁡(x)100\tanh(x) it reaches 66.91%, 64.65%, and 38.37% on MNIST at depths 50, 100, and 150, while our method reaches 93.25%, 93.51%, and 91.81%. Across both activations and both datasets, the calibrated noise adds 23 to 70 points, and the initialization without noise degrades with depth while ours stays stable.

Regarding a simpler depth-dependent schedule, we also compared with the rule σz=ωs/L\sigma_z = \omega \sqrt{s/L}, where ss is selected by a validation grid search over five values (0.875, 1.75, 3.5, 7, 14). The setting is that of Table 1 with 100tanh⁡(x)100\tanh(x), and the result is the last row of Table 1 in the revision (Section 5.2).

Noise scale chosen bySelection timeMNIST, 50MNIST, 100MNIST, 150F-MNIST, 50F-MNIST, 100F-MNIST, 150
Sign-flip target p=0.4p = 0.4 (ours)0.6 s93.25 ± 0.0793.51 ± 0.4791.81 ± 1.0484.08 ± 0.6483.80 ± 1.2682.62 ± 0.85
Grid search over ss43.0 s94.41 ± 0.1493.41 ± 0.2791.62 ± 0.9884.58 ± 0.3683.58 ± 0.5482.50 ± 0.63

The grid search reaches comparable accuracy, within 1.2 points, but selecting the noise scale takes about 70 times longer because five candidate networks must be trained, and this cost grows with the model and the data. The sign-flip target needs only forward passes at initialization. It also tells us where ss should be. The target p=0.4p = 0.4 corresponds to ss between 3.30 and 3.69 (Appendix B.4), which is the center of the grid above.

Thank you again for your time and thoughtful review. We hope that our responses and the revised manuscript address your concerns, and we would be glad to discuss any remaining questions. If our responses address your concerns, we would greatly appreciate your reconsidering the rating.

AuthorsOpenReview ↗

Reminder for Reviewer SzEw

Dear Reviewer SzEw,

As the discussion deadline approaches, we would like to kindly ask whether our responses and the revised manuscript have addressed your concerns. We would be happy to clarify any remaining points. If the issues you raised are resolved, we kindly ask you to consider updating your evaluation.

Thank you for your time and consideration.

Best regards,
Authors of Submission 55800

AuthorsOpenReview ↗

[Reminder] Could you kindly verify if the provided clarification addresses your concerns?

Dear Reviewer SzEw,

We have carefully considered your questions in preparing our responses and revising the manuscript. As the discussion period is set to close in less than 30 hours, we would be grateful for any additional suggestions or specific points to further enhance our manuscript.

If our responses have resolved the concerns you raised, please consider revisiting your evaluation.

Thank you once again for your support and thoughtful guidance throughout this process. We deeply value your time and effort in reviewing our work.

Best regards,

Authors of Submission 55800

Reviewer_XWWQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies weight-initialization for a class of odd-sigmoid activation functions. The authors define a general class of such functions and establish theoretical results concerning the scalar dynamics of corresponding neural networks at initialization. These findings motivate a simple diagonal-plus-noise initialization scheme. The noise component is designed through empirical calibration, using a (also empirically motivated) desired terminal sign-flip rate. The method is evaluated on several benchmarks, including physics-informed neural networks, where it generally outperforms standard initialization schemes.

Critical strengths
  • The proposed initialization method is well motivated and supported by a solid theoretical foundation.
  • The authors develop a unified framework for initializing networks with a broad class of activation functions. The relevance of the odd-sigmoid function class is well justified.
  • The paper presents extensive empirical evaluations across a variety of settings, including different datasets, network depths, widths, and activation functions, providing strong experimental support for the proposed method.
Critical weaknesses
  • The authors mention Lee et al. (2025) as a close predecessor, but do not include their method as a baseline in their experiments.
  • While the authors provide motivation for sign-flip calibration, its advantage over simpler alternatives is unclear to me. In particular, it is unclear why a simpler depth-dependent noise scale, such as σ∝1/L\sigma \propto 1/\sqrt{L}, could not be calibrated directly instead.
Questions for authors
  • How does the proposed method compare to other diagonal-plus-noise initialization schemes, like the mentioned Lee et al. (2025) work?
  • The authors show that their calibrated noise scale satisfies σz2≈sω2L\sigma_z^2 \approx \frac{s \omega^2}{L}, where ss is a constant. How does the proposed sign-flip calibration compare to directly tuning ss for this simpler depth-dependent noise rule? This could help clarify whether sign-flip calibration itself offers benefits beyond selecting an appropriate noise magnitude.
3 comments
AuthorsOpenReview ↗

Response to Reviewer XWWQ (1/2)

We sincerely appreciate the time you took to read and review our paper. We have carefully addressed each of your concerns, and all changes are marked in red in the revised manuscript.

W1. The authors mention Lee et al. (2025) as a close predecessor, but do not include their method as a baseline in their experiments.

Q1. How does the proposed method compare to other diagonal-plus-noise initialization schemes, like Lee et al. (2025)?

A. Thank you for this valuable suggestion. We have added Lee et al. (2025) as a baseline in Table 1 (Section 5.2) and Table 9 (Appendix C.6) of the revision. Both tables train width-64 networks for ten epochs at depths 50, 100, and 150 and report test accuracy in % (mean ± std over three seeds). Our method outperforms Lee et al. (2025) and all other baselines at every depth on both datasets.

The tables also cover the other comparisons you asked about. Rows 2 to 4 are further diagonal-plus-noise variants (the backbone rescaled by ω\omega, the backbone and the noise rescaled by ω\omega, and the ω\omega backbone without noise). Rows 5 to 8 are scaled Xavier, scaled orthogonal, LSUV, and EOC. Rescaling the backbone alone (rows 2 and 4), or the backbone together with a fixed noise level (row 3), does not recover the accuracy. It is recovered only when the ω\omega backbone is combined with the depth-dependent noise scale given by sign-flip calibration (row 9).

Table 1. 100tanh⁡(x)100\tanh(x)

InitializationMNIST, 50MNIST, 100MNIST, 150F-MNIST, 50F-MNIST, 100F-MNIST, 150
1. Lee et al. (2025)67.56 ± 1.8767.55 ± 1.8767.56 ± 1.8864.87 ± 1.7064.87 ± 1.6564.73 ± 1.66
2. Backbone rescaling only10.91 ± 0.76failedfailed9.93 ± 0.31failedfailed
3. Backbone + noise rescaling10.83 ± 0.8977.30 ± 3.0963.09 ± 2.5854.47 ± 6.7553.35 ± 10.4753.98 ± 11.64
4. No noise66.91 ± 2.6264.65 ± 0.1938.37 ± 31.9538.30 ± 0.5119.53 ± 0.7912.14 ± 3.04
5. Scaled Xavier29.39 ± 26.1669.44 ± 11.1450.93 ± 28.6710.00 ± 0.0071.43 ± 1.2929.47 ± 33.73
6. Scaled orthogonal14.28 ± 4.9189.54 ± 0.7086.13 ± 4.3613.92 ± 6.7779.52 ± 1.4877.73 ± 2.55
7. LSUV27.80 ± 7.8010.95 ± 0.7510.34 ± 1.1013.33 ± 5.7712.50 ± 4.3210.01 ± 0.01
8. EOC88.43 ± 0.6768.46 ± 9.4914.43 ± 5.3478.81 ± 1.9852.28 ± 36.6269.76 ± 3.09
9. Proposed (p=0.4p = 0.4)93.25 ± 0.0793.51 ± 0.4791.81 ± 1.0484.08 ± 0.6483.80 ± 1.2682.62 ± 0.85

Table 9. 1000erf(x)1000 \mathrm{erf}(x)

InitializationMNIST, 50MNIST, 100MNIST, 150F-MNIST, 50F-MNIST, 100F-MNIST, 150
1. Lee et al. (2025)61.45 ± 1.9161.98 ± 2.0861.94 ± 1.3060.49 ± 0.2461.48 ± 0.1961.12 ± 3.19
2. Backbone rescaling only11.26 ± 0.4310.01 ± 0.459.90 ± 0.34failed12.48 ± 4.73failed
3. Backbone + noise rescaling12.76 ± 3.5970.06 ± 6.5613.93 ± 5.4042.44 ± 0.2644.67 ± 0.7843.91 ± 2.16
4. No noise66.27 ± 2.7065.65 ± 3.7149.29 ± 25.6238.63 ± 0.5638.67 ± 0.2013.06 ± 5.30
5. Scaled Xavier77.11 ± 4.3868.02 ± 1.2146.39 ± 4.8174.36 ± 0.7172.43 ± 1.2258.42 ± 8.20
6. Scaled orthogonal13.56 ± 3.8381.72 ± 0.4074.92 ± 8.6510.01 ± 0.0168.85 ± 0.7669.50 ± 1.57
7. LSUV30.40 ± 23.4411.35 ± 0.0210.95 ± 0.7610.01 ± 0.0110.00 ± 0.0010.01 ± 0.01
8. EOC29.99 ± 32.2746.15 ± 30.1513.38 ± 3.4910.00 ± 0.0171.71 ± 2.5657.34 ± 0.99
9. Proposed (p=0.4p = 0.4)88.86 ± 0.7491.63 ± 0.7091.55 ± 0.6282.67 ± 0.9582.95 ± 0.7582.47 ± 0.79
AuthorsOpenReview ↗

Response to Reviewer XWWQ (2/2)

W2. The advantage of sign-flip calibration over simpler alternatives is unclear. Why could a simpler depth-dependent noise scale, such as σ∝1/L\sigma \propto 1/\sqrt{L}, not be calibrated directly?

Q2. How does sign-flip calibration compare to directly tuning ss in σz2≈sω2/L\sigma_z^2 \approx s \omega^2 / L? Does it offer benefits beyond selecting an appropriate noise magnitude?

A. Thank you for this question. We ran this comparison and added it to the revision (Section 5.2 and the last row of Table 1).

Directly tuning ss reaches comparable accuracy, at about 70 times the selection cost. In the setting of Table 1, we set σz=ωs/L\sigma_z = \omega \sqrt{s/L} and selected ss by a validation grid search over five values (0.875, 1.75, 3.5, 7, 14). Each candidate is trained for three epochs on 12,000 examples and scored on 9,000 held-out examples. The accuracies differ by at most 1.2 points, in either direction.

Noise scale chosen bySelection timeMNIST, 50MNIST, 100MNIST, 150F-MNIST, 50F-MNIST, 100F-MNIST, 150
Sign-flip target p=0.4p = 0.4 (ours)0.6 s93.25 ± 0.0793.51 ± 0.4791.81 ± 1.0484.08 ± 0.6483.80 ± 1.2682.62 ± 0.85
Grid search over ss43.0 s94.41 ± 0.1493.41 ± 0.2791.62 ± 0.9884.58 ± 0.3683.58 ± 0.5482.50 ± 0.63

The benefits of sign-flip calibration are as follows.

  • It needs no training search. Selecting the noise scale takes 0.6 s with the sign-flip target, which needs only forward passes at initialization. The grid search takes 43.0 s, about 70 times longer, because it trains five candidate networks. This gap grows with the complexity of the model and the data, since every candidate must be trained.
  • It tells us where ss should be. The target p=0.4p = 0.4 corresponds to ss between 3.30 and 3.69 (Appendix B.4), and the search grid above is centered on this value. Without the sign-flip analysis, neither the 1/L1/L form (Figure 2, Proposition B.1) nor this center is known in advance. The constant matters. In Table 1, no noise and a fixed noise level reach 64.65% and 77.30% on MNIST at depth 100, against 93.51% for our method.
  • Its target is fixed and interpretable. pp is the fraction of coordinates whose sign differs from the input, so it has the same meaning for every activation and depth. In our classification experiments, the single value p=0.4p = 0.4 worked across all tested activations and depths without retuning.

Thank you again for your time and thoughtful review. We hope that our responses and the revised manuscript address your concerns, and we would be glad to discuss any remaining questions.

AuthorsOpenReview ↗

Reminder for Reviewer XWWQ

Dear Reviewer XWWQ,

As the discussion deadline approaches, we would like to kindly ask. whether our responses and the revised manuscript have addressed your points. We would be happy to clarify any remaining points

Thank you for the time and appreciation that you have dedicated to our work.

Best regards,
Authors of Submission 59255

Reviewer_43ZxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a method to initialize neural networks based on a specific decomposition of the operation made by a layer, where a layer is regarded as a vector of gains applied element-wise to the input. By measuring the propagation of the variance of the gains, one can infer a good initialization scheme for the parameters. To design it, the paper proposes to aim for a given "sign-flip rate" of the output of the network.

Critical strengths

None

Critical weaknesses

Clarity

Section 4.2. This section, crucial in the paper, contains many notations that are not clearly defined:

  • what is the "sign-flip rate"?
  • what are prealp_{\mathrm{real}} and psurp_{\mathrm{sur}}?
  • what is the "target" π\pi?
  • what is ss?

Also, some notations are defined after being used, such as the sign-flip rate.

The paragraph l. 214-244 is an inline explanation of the initialization/fitting strategy. It carries so much information and notation that it is very difficult to read. In particular, either the fitting phases should be put in Appendix, either the entire process should be decomposed into simpler elements and take more space in the paper.

L. 202: why mod Nl−1\mathrm{mod} \, N_{l-1}? According to the previous section, we have Nl=nN_l = n for all ll. Besides, it does not seem to be used in Lemma 4.5 or anywhere else. That being said, there is indeed an issue in the case where the layer sizes are heterogeneous, how is this case handled in this paper?

Novelty

This paper is extremely close to [A]. The notation and the initial ideas are the same:

  • formulation of a neuron's computation as a gain when propagating the input;
  • same type of activation functions (Def. 4.1 instead of tanh only), Prop. 4.2 and Th. 4.3 are trivial, proofs are essentially the same;
  • the distribution of the gains formula (2) is almost the same as formula (2) in [A], the only difference is using ω\omega instead of 11, which is a trivial extension of [A].

However, the paper cites [A].

Relevance of the approach

The approach is the same as in [A]. Despite [A] has been peer-reviewed and published, its approach is questionable. The idea consists in studying: xil+1=f(ail+1xil), ail+1=wiil+1+∑j≠iwijl+1xjlxil.x_i^{l+1} = f(a_i^{l+1} x_i^l), \, a_i^{l+1} = w_{ii}^{l+1} + \sum_{j \neq i} \frac{w_{ij}^{l+1} x_j^l}{x_i^l}. In particular, the layers are regarded as vectors of gains applied element-wise to the input. Some questions arise inevitably:

  • this choice of decomposition is arbitrary; one can as well write: xil+1=f(ail+1x1l), ail+1=wi1l+1+∑j≠1wijl+1xjlx1l,x_i^{l+1} = f(a_i^{l+1} x_1^l), \, a_i^{l+1} = w_{i1}^{l+1} + \sum_{j \neq 1} \frac{w_{ij}^{l+1} x_j^l}{x_1^l},so does it mean that propagation of information is only about propagating feature #1? Maybe there is a good theoretical argument to dismiss this decomposition, but it should be provided somewhere;
  • it is possible to write this cleanly only if the network is rectangular (same width for every layer); naturally, one can deal with layers of different sizes, but it must be at the cost of denying this "gain" interpretation of information propagation; for instance, in the case where the number of neurons decreases from layer ll to layer l+1l+1, some information should simply disappear, which is obviously not the case (it is hidden in the "gain");
  • besides, this kind of difficulty is hidden behind the index "j≠ij \neq i" of the sum, which does not tell us the number of elements in it.

Empirical validation

The comparison of experimental results obtained with the proposed method and concurrent methods is debatable:

  • Figure 7:
    • this is the only figure in the main part of the paper comparing methods, which is likely to be insufficient;
    • the comparison contains: the proposed method ×5\times 5 hyperparameters against Xavier and EOC; one should have selected only one hyperparameter and tested the method against more concurrent methods ([A] itself, Deep Kernel Shaping [B] and/or TAT [C], LSUV [D], orthogonal initialization [E], etc.).
    • the setup is narrow and dubious: 1) why highlighting PDE benchmarks instead of the elementary vision benchmarks (such as in [A] with MNIST and CIFAR)? 2) the choice of optimization schedule (300 epochs of Adam + 100 epochs of L-BFGS) looks fine-tuned, why not using plain SGD?
    • for Helmholtz2D and Poisson2D, the dispersion of the method (inter-quartile) looks huge...
  • Appendix C.4:
    • the proposed initialization does not outperform the EOC;
    • the choice of arctan(0.001x) is highly debatable: no one would argue that we can change the scaling without changing the initialization distribution; to be fair, using a 0.001 factor in the activation function should be associated with a multiplication of the variance of the Xavier initialization by a factor 1/0.001.

References

[A] Robust weight initialization for tanh neural networks with fixed point analysis, Lee et al., ICLR 2025.

Missing references

Deep Kernel Shaping [B] Rapid training of deep neural networks without skip connections or normalization layers using deep kernel shaping, Martens et al., arXiv preprint arXiv:2110.01765.

TAT [C] Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers, Zhang et al., ICLR 2022.

LSUV [D] All you need is a good init, Mishkin & Matas, ICLR 2016.

Orthogonal initialization [E] Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice, Pennington et al., NeurIPS 2017.

Questions for authors

How do you handle heterogeneous layer sizes?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 43Zx (1/3)

Thank you for taking the time to read our paper and for your detailed feedback.

The ICLR 2027 review form asks reviewers to identify the one or two critical weaknesses that support rejection. Your review raises concerns in four areas (clarity, novelty, relevance of the approach, and empirical validation), with several points under each, so we could not be certain which of them weighed most in your assessment. In line with this ICLR policy and to keep the discussion focused, we have organized our response in two parts.

  • Part 1: the concerns that we understand to be most critical to your recommendation.
  • Part 2: the remaining points.

We have tried to respond to every point you raised, and we welcome further discussion. All revisions and additions are marked in red in the revised manuscript.

Part 1

W1. The paper is close to [A].

A. Thank you for raising this point. The main contribution of this paper, sign-flip calibration of the noise scale, does not appear in [A]. We agree that the paper builds on [A]: Sections 2 and 3 say so, and Eq. (1) and Lemma 4.5 follow that framework. What is new is how the noise is chosen.

  • [A]: tanh only, an identity backbone, and one empirically chosen noise scale (σz=0.085\sigma_z = 0.085 in our notation) used at every depth.
  • This paper: The suitable noise scale depends on the activation and the depth (Figure 5), so a single constant cannot fit all cases. We therefore use (i) the backbone gain ω=1/f′(0)\omega = 1/f'(0) for the class F\mathcal{F}, and (ii) a noise scale set for each activation and depth through a target terminal sign-flip rate. For (ii) we give a closed form in a scalar surrogate (Theorem 4.7), a depth schedule that transfers it to the network (Figures 2 and 13), and the resulting scaling law σz2≈sω2/L\sigma_z^2 \approx s \omega^2 / L (Appendix B.4). To our knowledge, the sign-flip rate is first proposed in this paper, where it is used to design the initialization.

The table below (Table 1 of the revision) trains width-64 networks with 100tanh⁡(x)100\tanh(x), for which ω=0.01\omega = 0.01, for ten epochs at depths 50, 100, and 150. Entries are test accuracy in % (mean ± std over three seeds). Rows 2 and 3 are the direct extension of [A]: 11 replaced by ω\omega in the backbone D, without and with the same rescaling of the noise.

InitializationMNIST, 50MNIST, 100MNIST, 150F-MNIST, 50F-MNIST, 100F-MNIST, 150
1. [A]67.56 ± 1.8767.55 ± 1.8767.56 ± 1.8864.87 ± 1.7064.87 ± 1.6564.73 ± 1.66
2. [A], backbone rescaled by ω\omega10.91 ± 0.76failedfailed9.93 ± 0.31failedfailed
3. [A], backbone and noise rescaled by ω\omega10.83 ± 0.8977.30 ± 3.0963.09 ± 2.5854.47 ± 6.7553.35 ± 10.4753.98 ± 11.64
4. ω\omega backbone, no noise66.91 ± 2.6264.65 ± 0.1938.37 ± 31.9538.30 ± 0.5119.53 ± 0.7912.14 ± 3.04
5. Ours (ω\omega backbone, sign-flip target p=0.4p = 0.4)93.25 ± 0.0793.51 ± 0.4791.81 ± 1.0484.08 ± 0.6483.80 ± 1.2682.62 ± 0.85

[A] stays near 67.6% on MNIST and 64.8% on Fashion-MNIST at every depth. Replacing 11 by ω\omega fails, and also rescaling the noise by ω\omega remains 16 to 82 points below our method. Removing the noise degrades with depth. Only the ω\omega backbone combined with sign-flip calibrated noise trains at all three depths. Table 9 (Appendix C.6) shows the same pattern for 1000erf(x)1000 \mathrm{erf}(x).

On Proposition 4.2, Theorem 4.3, and Eq. (2). We do not present them as the main contribution. Their role is to identify the conditions (Definition 4.1) under which the tanh argument carries over, and to make the critical gain explicit as ω=1/f′(0)\omega = 1/f'(0). This gives the gain of scaled and composite activations without a new derivation (Corollary 4.4). Rows 2 to 4 of the table above show that this gain alone is not sufficient.

AuthorsOpenReview ↗

Response to Reviewer 43Zx (2/3)

W2. The gain decomposition is arbitrary (one could equally take x1x_1 as the reference), and it can be written cleanly only for rectangular networks.

Q1. How do you handle heterogeneous layer sizes?

A. Thank you for these questions. We answer them together, since both concern the gain formulation.

With respect, our method does not have either of these two limitations. The reference coordinate is not an arbitrary choice, and unequal widths were already covered by the definition of the initialization in the original submission. Both facts follow directly from the construction. However, we recognize that the original text left these steps implicit, so the revision states them explicitly (marked in red).

1. The reference coordinate follows from the diagonal backbone. Our initialization is W=ωI+ZW = \omega I + Z, so the deterministic part of neuron ii acts on the input with the same index, xix_i. This is why we take xix_i as the reference. For xi≠0x_i \neq 0 the gain then has conditional mean ω\omega for every neuron and every state (Lemma 4.5), so each coordinate follows the scalar map x↦f(ax)x \mapsto f(ax) of Section 4.1 with a gain centered at ω\omega. With another reference such as x1x_1, the rewriting is still exact, but the conditional mean becomes ωxi/x1\omega x_i / x_1, which changes with the state, so the scalar analysis cannot be applied directly. As for your question whether propagation is then only about feature #1, it is not. With either reference, every coordinate enters through the gain. This reasoning was implicit in Lemma 4.5, and we now state it in lines 257 to 261.

2. Unequal widths were already part of the method. The initialization is defined for arbitrary widths: the backbone places ω\omega at entries with i≡j(modNℓ−1)i \equiv j \pmod{N_{\ell-1}}, and the noise variance uses the fan-in Nℓ−1N_{\ell-1} (lines 203 to 206). All classification networks in the paper already contain unequal widths at the input and output layers and are initialized with this rule. We set Nℓ=nN_\ell = n only to simplify the notation of the analysis, and we had omitted the unequal-width derivation because it is the same computation. In a narrowing layer, inputs without a successor neuron are neither discarded nor hidden: they enter every gain through the sum over all Nℓ−1N_{\ell-1} inputs.

Following your concern, we added the derivation as the paragraph "Unequal layer widths" in Appendix A.2 (lines 864 to 878), with pointers in lines 115 to 117 and 206 to 207. There, output neuron ii takes the input with index r(i)=1+((i−1) mod Nℓ−1)r(i) = 1 + ((i-1) \bmod N_{\ell-1}) as its reference, and its gain has the same Gaussian law as in Lemma 4.5, with fan-in Nℓ−1N_{\ell-1}.

Part 2

W3. Clarity of Section 4.2.

A. Thank you for this suggestion. The quantities in question (the sign-flip rate, etc.) were defined in the original submission, and we have rewritten this part in the revision to make them easier to read (lines 209 to 215 and 235 to 248, in red).

W4. The index "j≠ij \neq i" of the sum does not tell us the number of elements in it.

A. In the main text, the sums are now written with explicit limits, from j=1j = 1 to NℓN_\ell with j≠ij \neq i.

W5. The comparison uses five hyperparameter settings of the proposed method against only Xavier and EOC. One setting should be compared with more methods ([A], DKS [B], TAT [C], LSUV [D], orthogonal [E]).

A. Thank you for this suggestion. We have added Table 1 (Section 5.2) and Table 9 (Appendix C.6), which compare a single setting of our method, p=0.4p = 0.4, with [A], LSUV [D], orthogonal initialization [E], Xavier, and EOC. Xavier and orthogonal use the gain ω\omega, and EOC is computed for the given activation, so every baseline is adapted to the activation scale.

Table 1: test accuracy (%) with 100tanh⁡(x)100\tanh(x) in width-64 networks after ten epochs (mean ± std over three seeds).

InitializationMNIST, 50MNIST, 100MNIST, 150F-MNIST, 50F-MNIST, 100F-MNIST, 150
[A]67.56 ± 1.8767.55 ± 1.8767.56 ± 1.8864.87 ± 1.7064.87 ± 1.6564.73 ± 1.66
Scaled Xavier29.39 ± 26.1669.44 ± 11.1450.93 ± 28.6710.00 ± 0.0071.43 ± 1.2929.47 ± 33.73
Scaled orthogonal [E]14.28 ± 4.9189.54 ± 0.7086.13 ± 4.3613.92 ± 6.7779.52 ± 1.4877.73 ± 2.55
LSUV [D]27.80 ± 7.8010.95 ± 0.7510.34 ± 1.1013.33 ± 5.7712.50 ± 4.3210.01 ± 0.01
EOC88.43 ± 0.6768.46 ± 9.4914.43 ± 5.3478.81 ± 1.9852.28 ± 36.6269.76 ± 3.09
Ours (p=0.4p = 0.4)93.25 ± 0.0793.51 ± 0.4791.81 ± 1.0484.08 ± 0.6483.80 ± 1.2682.62 ± 0.85

Our method has the highest mean accuracy at every depth on both datasets, and Table 9 shows the same for 1000erf(x)1000 \mathrm{erf}(x). DKS[B] and TAT[C] reshape the activation itself, whereas we keep it fixed and change only the initialization, so they are not directly comparable.

AuthorsOpenReview ↗

Response to Reviewer 43Zx (3/3)

W6. The setup is narrow: why highlight PDE benchmarks instead of elementary vision benchmarks such as MNIST and CIFAR, and why Adam followed by L-BFGS instead of plain SGD?

A. Vision benchmarks are already in the paper. Specifically, results on MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100 are in Figures 5 and 6 in the main text and Appendices C.1 to C.5. In addition, we report PDE benchmarks because, as explained in the Introduction, odd-sigmoid activations are widely used in PINNs, which makes them the natural application setting for this activation class. As for the optimizer, Adam followed by L-BFGS is the standard training procedure for PINNs rather than a choice we made for this paper. [1] compares Adam, L-BFGS, and their combination, and finds the combination best. We followed this standard procedure and applied the same schedule to every initialization.

W7. For Helmholtz2D and Poisson2D, the dispersion of the method (inter-quartile range) looks huge.

A. Each box in Figure 7 aggregates 15 runs, five odd-sigmoid activations with three seeds each, so the spread comes mainly from the different activations. Figure 40 in Appendix C.7 shows each activation separately at p=0.4p = 0.4. On these two problems the error depends strongly on the activation and is largest for 0.2tanh⁡(x)0.2\tanh(x). At small-to-moderate pp, our method still attains a lower median error than Xavier and EOC on every PDE.

W8. Appendix C.4: the proposed initialization does not outperform EOC.

A. Appendix C.4 was designed to test learning-rate sensitivity rather than to compare final accuracy. As stated in Section 5.2 (Learning-Rate Sensitivity), its purpose is to study optimization stability. Each run uses only 200 iterations, and the figures show the range of learning rates in which training is feasible. In this test our method trains over a typically wider range of learning rates, and in three settings only our method reaches high accuracy (Figures 36(f), 37(e), and 37(f)).

The accuracy comparisons with EOC are in other parts of the paper, and our method is better there.

  • Figure 5 (MNIST). Our method reaches high accuracy over a broader region of depths and noise scales.
  • Figure 7 (PINNs). The median error is 14 to 45 times lower on the four PDE benchmarks.
  • Table 2, Appendix C.1 (small data). The accuracy is higher in all 24 settings.
  • Figures 26 to 30 and 34, Appendix C.2 (depth and width). As depth or width grows, EOC loses accuracy earlier than our method.
  • Figures 31 to 33, Appendix C.3 (training without BN). In the 800-layer network only our method trains.
  • Tables 4, 6, and 8, Appendix C.5 (activation scales). For scales from 10410^4 to 10910^9 only our method trains.
  • New Tables 1 and 9 (large activation scales). Our method is above EOC in every column. In Table 1, EOC drops from 88.43% to 14.43% on MNIST as the depth grows from 50 to 150, while our method stays between 91.81% and 93.51%.

W9. For a rescaled activation such as arctan⁡(0.001x)\arctan(0.001x), the comparison is fair only if the baselines are also adapted to the activation scale.

A. The original submission already compares against baselines adapted to the activation scale: EOC is computed for each rescaled activation in Appendix C.4, and Xavier is rescaled by ω\omega in Appendix C.7. The new Tables 1 and 9 add Xavier and orthogonal initialization rescaled by ω=1/f′(0)\omega = 1/f'(0), and our method remains the best. Part of Table 1 is shown in our answer to W5.

Thank you again for your time. We hope that our responses and the revision address your concerns, and we welcome further discussion. If they do, we would be grateful if you would consider revisiting your rating.

[1] Rathore et al., Challenges in training PINNs: A loss landscape perspective, ICML 2024.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Dear Reviewers and AC,

We thank all reviewers for taking the time to review our work and for their valuable feedback. We appreciate the recognition of the well-motivated setting, the theoretical analysis, and the extensive experiments (Reviewers XWWQ and SzEw), the unified framework for a broad class of activations (Reviewer XWWQ), and the clear presentation (Reviewer SzEw).

The latest revision has been uploaded and addresses all comments raised by the reviewers. Edits in the PDF are highlighted in red. Below we summarize the changes.

Main concerns

1. Relation to and comparison with Lee et al. (2025) (Reviewers XWWQ, SzEw, and 43Zx). We added two tables, Table 1 (Section 5.2) and Table 9 (Appendix C.6), which compare our method with Lee et al. (2025) and with other initializations (scaled Xavier, scaled orthogonal, LSUV, and EOC) for two activations, 100tanh⁡(x)100\tanh(x) and 1000erf(x)1000 \mathrm{erf}(x). Our method is 18 to 30 points above Lee et al. (2025) and 4 to 17 points above the best of the other baselines in every column. The tables also show that the diagonal structure shared with Lee et al. (2025) is not enough by itself. With this structure but without the calibrated noise, the accuracy is 16 to 82 points lower.

2. Sign-flip calibration versus a simpler depth-dependent noise rule (Reviewers XWWQ and SzEw). We added a comparison with a direct grid search over ss in σz=ωs/L\sigma_z = \omega \sqrt{s/L} (last row of Table 1). The search reaches comparable accuracy, within 1.2 points, but selecting the noise scale takes about 70 times longer, and the initialization without noise is 23 to 70 points below ours.


Further points raised by Reviewer 43Zx

For eight of the ten points below, the relevant content was already in the original submission, explicitly or implicitly. For these we point to its location and, where helpful, state it more explicitly.

PointResponseStatus
W1. Novelty relative to Lee et al. (2025)The review compares only the backbone. Our main contribution, sign-flip calibration (Section 4.2), is new to our knowledge and was in the original submission.Already in the original
W2. The gain decomposition is arbitraryThe reference coordinate is fixed by the diagonal backbone, which gives the conditional mean ω\omega (Lemma 4.5). Now stated in lines 257 to 261.Implicit in the original
Q1. Heterogeneous layer sizesThe initialization was already defined for arbitrary widths (lines 203 to 206). The derivation for unequal widths is added in Appendix A.2.Already in the original
W3. Notation of Section 4.2All quantities were defined. The passage is rewritten for readability (lines 209 to 215 and 235 to 248).Already in the original
W4. Limits of the sumsWritten explicitly in the main text.Notation fixed
W5. More baselinesScaled Xavier, scaled orthogonal, LSUV, EOC, and Lee et al. (2025) are compared in Tables 1 and 9.New tables
W6. The optimizer schedule looks fine-tunedAdam followed by L-BFGS is the standard PINN training setup [1], not one tuned for our method.Already in the original
W7. The dispersion in Figure 7 looks hugeThe caption of Figure 7 already states the reason: each box aggregates five different activations.Already in the original
W8. EOC is not outperformedEOC fails where our method trains in Figures 36(f), 37(e), 37(f) and Tables 4, 6, 8, and it is 14 to 45 times worse in the PINN experiments.Already in the original
W9. Baselines are not adapted to the activation scaleThey already are. EOC is by definition computed for the given activation, and Xavier is rescaled by ω\omega in the PINN experiments (Appendix C.7). Tables 1 and 9 add further scaled baselines.Already in the original

Thank you again for the time and care you have given our paper.

Best regards,

Authors of Submission 55800

[1] Rathore et al., Challenges in training PINNs: A loss landscape perspective, ICML 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.