- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes NeuroPlastic, an optimizer inspired by multi-factor synaptic plasticity in neurobiology, which attempts to introduce a multi-signal modulation mechanism into backpropagation to achieve adaptive gradient scaling. The paper uses gradient magnitude, moving averages, and a memory-like state to compute a modulation factor, thereby stabilizing network parameters. On small-scale datasets and in data-scarce settings, NeuroPlastic achieves slight improvements over established optimizers such as SGD and Adam, but under more complex experimental conditions (CIFAR-10 + ResNet-18), NeuroPlastic performs worse than the tuned baseline optimizers.
Critical strengths
-
Translating the metaphor of multifactor interactions in synaptic plasticity into adaptive modulation factors for gradient flow is an interesting design idea that provides a heuristic approach to dynamic gradient scaling.
-
The motivation is presented clearly, the structure is generally complete, and the mapping from neurobiological concepts to optimizer components is explained intuitively.
Critical weaknesses
1.The synaptic plasticity described in the paper is essentially a nonlinear combination of gradient moving averages and second moments, rather than actual local or global neuronal activity and neuromodulatory behavior involved in synaptic plasticity. Compared with existing baseline optimizers, it lacks a breakthrough at the mechanistic level.
2.The experimental results presented in the paper indicate that the proposed optimizer lacks competitiveness. Furthermore, compared with the widespread application of classical optimizers, NeuroPlastic lacks experiments on large-scale tasks.
3.NeuroPlastic maintains three parameter-sized state tensors, requiring approximately 50% more optimizer-state memory than AdamW at equal precision, as acknowledged in Appendix A.1. Consequently, it remains unclear whether the modest improvements over the gradient-magnitude-only ablation justify the additional computational and memory costs.
4.NeuroPlastic introduces multi-factor computations and bounded stabilization steps, involving several new decay rates, smoothing constants, and clipping thresholds, but the paper provides no analysis or experiments concerning these parameters.
Questions for authors
I recommend conducting benchmark evaluations on at least ImageNet-1k image classification and small-scale language models. If the method only provides benefits on small-scale tasks, its practical value for modern machine learning would be considerably limited.
Reviewer's responses to authors
Dear reviewers,
If not done already (thank you - rev. x2Xh), could you please take a look and reply briefly to the "Official Comment(s) by Authors" / Rebuttals, possibly by an acknowledgement that you have seen it.
Feel free to respond in more detail, in particular, if you have further questions or opinions to the replies from the authors.
With best regards, an AC