- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper investigates how to design reliable weight initialization for deep neural networks with bounded odd-sigmoid activations, aiming to prevent signal attenuation, vanishing gradients, and training instability. Specifically, the paper extends identity-plus-noise initialization from tanh networks to a broader class of odd-sigmoid activations and proposes a depth-dependent noise calibration strategy based on the terminal sign-flip rate. Experimental results demonstrate that the proposed method enables more stable training of deep sigmoid networks and physics-informed neural networks (PINNs) compared with conventional Gaussian initialization methods, such as Xavier, He, and EOC.
Critical strengths
First, it focuses on a well-motivated application setting. Although modern deep learning architectures predominantly use ReLU, GELU, and residual connections, this does not diminish the importance of initialization for sigmoid-based networks, particularly in applications such as PINNs. Second, the paper provides a comprehensive theoretical analysis. The authors first define the odd-sigmoid function class and investigate the fixed-point structure of an iterative mapping based on the activation function and a scaling parameter. They identify a critical value determined by the reciprocal of the activation function's derivative at the origin, at which the fixed-point structure changes. This observation naturally motivates the activation-aware diagonal gain. The authors further derive the relationship between the terminal sign-flip probability, noise scale, and network depth using a scalar surrogate model. Third, the experimental evaluation is extensive, covering multiple odd-sigmoid activations, different network depths and widths, four classification datasets (MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100), various activation scaling factors, batch normalization, learning-rate sensitivity, and four PDE benchmarks. Overall, the experimental results provide convincing evidence that the proposed structured initialization is more stable than conventional Gaussian initialization methods in the tested deep odd-sigmoid networks.
Critical weaknesses
My primary concern is that the closest prior work already contains the main architecture-level insight, while the paper does not sufficiently demonstrate the practical value of its additional contributions. As acknowledged in the Related Work section, Lee et al. (2025) proposed an identity-plus-noise initialization for tanh networks. Their method similarly combines a deterministic diagonal-like matrix with Gaussian noise and uses fixed-point analysis to explain why this structure improves signal propagation in deep tanh networks. Consequently, a plausible alternative explanation is that most of the performance improvements reported in this paper arise from the previously established diagonal/identity structure rather than the newly introduced sign-flip calibration. Although sign-flip calibration is arguably the most distinctive component of the proposed method, the current experiments do not sufficiently demonstrate that it is responsible for the observed performance improvements.
Questions for authors
1.Since Lee et al. (2025) have already proposed an identity-plus-noise initialization for tanh networks, could the authors clarify the practical advantages of the proposed method over this closely related approach? In particular, is there a direct comparison under the same network architectures, activations, and training settings? Such a comparison would help determine whether the reported improvements primarily arise from the existing diagonal-plus-noise structure or from the additional contributions of this paper.
2.The proposed terminal sign-flip calibration appears to be the most distinctive component of the method. However, Figure 6 suggests that the deterministic initialization with zero noise already achieves performance comparable to the calibrated variants in several settings. Could the authors provide further evidence that sign-flip calibration offers meaningful improvements over the deterministic initialization or a simpler depth-dependent noise schedule? This would help establish whether calibration is essential to the observed training stability.
4 comments
Response to Reviewer SzEw (2/2)
Q2. Figure 6 suggests that the deterministic initialization with zero noise already performs comparably in several settings. Is there further evidence that sign-flip calibration offers meaningful improvements over the deterministic initialization or a simpler depth-dependent noise schedule?
A. Thank you for this helpful suggestion. We agree with your reading of Figure 6. In that figure the activation is tanh, for which . In this case, the initialization without noise performs about as well as the calibrated ones.
The noise becomes essential for the rescaled activations in Tables 1 and 9, where is far from 1. Row 4 of the two tables in our answer to W1 and Q1 is the initialization without noise, with the same backbone as ours. With it reaches 66.91%, 64.65%, and 38.37% on MNIST at depths 50, 100, and 150, while our method reaches 93.25%, 93.51%, and 91.81%. Across both activations and both datasets, the calibrated noise adds 23 to 70 points, and the initialization without noise degrades with depth while ours stays stable.
Regarding a simpler depth-dependent schedule, we also compared with the rule , where is selected by a validation grid search over five values (0.875, 1.75, 3.5, 7, 14). The setting is that of Table 1 with , and the result is the last row of Table 1 in the revision (Section 5.2).
| Noise scale chosen by | Selection time | MNIST, 50 | MNIST, 100 | MNIST, 150 | F-MNIST, 50 | F-MNIST, 100 | F-MNIST, 150 |
|---|---|---|---|---|---|---|---|
| Sign-flip target (ours) | 0.6 s | 93.25 ± 0.07 | 93.51 ± 0.47 | 91.81 ± 1.04 | 84.08 ± 0.64 | 83.80 ± 1.26 | 82.62 ± 0.85 |
| Grid search over | 43.0 s | 94.41 ± 0.14 | 93.41 ± 0.27 | 91.62 ± 0.98 | 84.58 ± 0.36 | 83.58 ± 0.54 | 82.50 ± 0.63 |
The grid search reaches comparable accuracy, within 1.2 points, but selecting the noise scale takes about 70 times longer because five candidate networks must be trained, and this cost grows with the model and the data. The sign-flip target needs only forward passes at initialization. It also tells us where should be. The target corresponds to between 3.30 and 3.69 (Appendix B.4), which is the center of the grid above.
Thank you again for your time and thoughtful review. We hope that our responses and the revised manuscript address your concerns, and we would be glad to discuss any remaining questions. If our responses address your concerns, we would greatly appreciate your reconsidering the rating.
Reminder for Reviewer SzEw
Dear Reviewer SzEw,
As the discussion deadline approaches, we would like to kindly ask whether our responses and the revised manuscript have addressed your concerns. We would be happy to clarify any remaining points. If the issues you raised are resolved, we kindly ask you to consider updating your evaluation.
Thank you for your time and consideration.
Best regards,
Authors of Submission 55800
[Reminder] Could you kindly verify if the provided clarification addresses your concerns?
Dear Reviewer SzEw,
We have carefully considered your questions in preparing our responses and revising the manuscript. As the discussion period is set to close in less than 30 hours, we would be grateful for any additional suggestions or specific points to further enhance our manuscript.
If our responses have resolved the concerns you raised, please consider revisiting your evaluation.
Thank you once again for your support and thoughtful guidance throughout this process. We deeply value your time and effort in reviewing our work.
Best regards,
Authors of Submission 55800
Response to Reviewer SzEw (1/2)
We sincerely thank you for your time and thoughtful review. We have carefully addressed each of your concerns, and all changes are marked in red in the revised manuscript.
A. Thank you for this valuable suggestion. The original submission gave partial evidence. In Figures 17 and 18 (tanh, depths 50 and 100), the calibrated noise scale gives higher accuracy than σz=0, but the differences there are small. We agree that clearer evidence is needed to separate the effect of the diagonal structure from the effect of the calibrated noise.
New experiment. Our initialization has two parts, W=D+Z. The backbone D is a deterministic diagonal matrix with entries ω=1/f′(0), and Z is Gaussian noise with scale σz. Lee et al. (2025) use the identity as the backbone and a fixed noise scale of 0.085. We added a direct comparison that changes these two parts separately, under identical architectures and training settings (Table 1 in Section 5.2 and Table 9 in Appendix C.6).
The goal of our paper is an initialization that trains across the odd-sigmoid class and not only for tanh, so we test two activations outside the setting of Lee et al. (2025), 100tanh(x) and 1000erf(x). For tanh itself ω=1, and the two backbones coincide. All networks have width 64 and are trained for ten epochs. Entries are test accuracy in % (mean ± std over three seeds) at depths 50, 100, and 150.
Table 1. 100tanh(x)
Table 9. 1000erf(x)
The gain needs both parts, the matched backbone and the calibrated noise. Our method is 18 to 30 points above Lee et al. (2025) in every column. The diagonal structure alone does not explain this gain. A diagonal backbone without calibrated noise stays far below our method, whether it is the identity (row 1) or matched to the activation (rows 2 to 4). Rows 3 to 5 share the backbone ωI and differ only in the noise, and replacing no noise or a fixed noise level by the sign-flip calibrated scale raises the accuracy by 16 to 82 points.
Practical advantages over Lee et al. (2025). Lee et al. (2025) is built for tanh, with the identity backbone and one empirically chosen noise constant for all depths, and on the two activations above it reaches 60 to 68%. Our method computes the backbone from the activation as ω=1/f′(0) and sets the noise scale for each activation and depth from a single target (p=0.4 in all our classification experiments). The same rule therefore applies to every odd-sigmoid activation without manual tuning, and it reaches 82 to 94% in the same setting.