- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies the sensitivity of the training dynamics for ReLU with perturbation of the initialization to be the same order as the learning rate. The authors characterize the sensitivity error of the finite-step iteration, provide a 1D example to support the theorem, and show experimental verification.
Critical strengths
- Both theoretical and experimental results are presented to support the authors’ claim.
- The problem studied in this paper is practical; the sensitivity analysis is an important characterization of the training dynamics.
Critical weaknesses
- The overall paper is technical in the sense that there is quite some jargon that is not explained or interpreted in the machine learning setting. The paper needs a significant rewrite.
- The theoretical contribution seems to be a standard extension of classical results, instead of being specifically derived under the machine learning setting.
Questions for authors
- What is the “event”?
- I was expecting there would be some nontrivial concrete results on whether, when, or how the neuron is activated for the critical response since it is the most nontrivial resolution based on my understanding. But it seems only standard error analysis is presented. Maybe this is already included in Theorem 2, but I’m not sure what the positive indices refer to.
- Notations not explained: in line 136, in line 142, and more.
Thank you for the time and care you put into this discussion
As the discussion period comes to a close, we’d like to thank the Area Chair for overseeing the review and all the reviewers for reading our paper closely, working through our responses, and taking the time to ask follow-up questions. We appreciate the care you took in pointing out both what was clear and where we still had work to do.
Your feedback changed how we present the paper. It led us to revisit the experiments and explain more clearly when a change in activation timing affects a decision, how large the observed gains are, and which comparisons we can certify. The follow-up questions were especially helpful in showing us where a distinction that seemed clear to us as authors still needed a better explanation.
This was my first time taking part in ICLR as a student author, and it was an honor to be part of the discussion. I learned a lot from seeing how careful questions can make a paper clearer and its claims more precise. That’s something I’ll carry into my future work. I hope the ICLR community continues to grow as a place where people can ask candid questions, learn from one another, and help each other improve their work.
Thank you again for your time and for helping us make the paper better.