acceptodds
Under review as a conference paper at ICLR 2027

Branch Geometry and Finite-Radius Sensitivity of Hard-ReLU Training

Abstract

Differentiating a training run gives a local sensitivity, whereas an outer update makes a finite change to its initialization or training parameters. For hard-ReLU gradient descent (GD), we characterize the response when the perturbation radius is proportional to the GD step size. In this regime, integer changes in activation-crossing indices contribute at leading order. Smooth Euler bias shifts each crossing phase, and earlier rounding decisions move later branch boundaries. For piecewise- dynamics with finitely many separated, same-direction transverse events, we derive the discrete crossing indices and an endpoint expansion with uniform error away from recursive phase boundaries. Contractive affine regions admit an explicit finite remainder that, together with complete branch verification, certifies candidate ordering. A strongly convex scalar example reverses both infinitesimal predictions on of 9,801 tested meshes, close to the proved asymptotic frequency of . Coupled-event experiments isolate downstream rounding feedback; a nonlinear-network study shows how prediction accuracy changes with radius and branch validity. Together with local automatic differentiation and uniform flow consistency, the theory gives three sufficient response regimes. It characterizes finite objective comparisons under stated conditions; it does not guarantee optimizer gains.

Primary area:
learning theory
Keywords:
differentiating through optimization, finite-radius sensitivity, automatic differentiation, ReLU networks, nonsmooth optimization, hybrid dynamical systems, bilevel optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wNNaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the sensitivity of the training dynamics for ReLU with perturbation of the initialization to be the same order as the learning rate. The authors characterize the sensitivity error of the finite-step iteration, provide a 1D example to support the theorem, and show experimental verification.

Critical strengths
  • Both theoretical and experimental results are presented to support the authors’ claim.
  • The problem studied in this paper is practical; the sensitivity analysis is an important characterization of the training dynamics.
Critical weaknesses
  • The overall paper is technical in the sense that there is quite some jargon that is not explained or interpreted in the machine learning setting. The paper needs a significant rewrite.
  • The theoretical contribution seems to be a standard extension of classical results, instead of being specifically derived under the machine learning setting.
Questions for authors
  • What is the “event”?
  • I was expecting there would be some nontrivial concrete results on whether, when, or how the neuron is activated for the critical response since it is the most nontrivial resolution based on my understanding. But it seems only standard error analysis is presented. Maybe this is already included in Theorem 2, but I’m not sure what the positive indices refer to.
  • Notations not explained: f+,f−f^+,f^- in line 136, φT\varphi_T in line 142, and more.
Reviewer_qCREOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

The paper investigates how to predict the differing training outcome resulting from a small change to initialization in training hard-ReLU-based functions, as focusing only on the standard AD derivative does not properly take into account the discontinuity in the ReLU derivative that changes the gradient function while training (between discrete steps). The authors indicate that this depends on the size of the change made, in particular when this is approximately on the scale of the learning rate, a new predictor is needed.

Critical strengths

I believe it is an original/innovative insight to couple the radius epsilon to the learning rate, and I think it is a well-found realization that there is a critical level of epsilon ~ learning rate.

The worked out analysis seems very sound, properly borrowing ideas from numerical analysis in a new domain.

Critical weaknesses

I am very curious to what extent such results scale either to bigger/deeper networks, as I can imagine things quickly become chaotic/messy, and some of the assumptions may become tricky, as I can imagine with many more parameters there can be multiple thresholds crossed in the same step.

Questions for authors

I wonder if in general not a whole different approach is needed when scaling up, like mean-field-theory ideas from physics. Any insights or suggested ideas in this direction would be welcome, just to put it in a broader context, or maybe point me at what parts of the paper already make such a statistical view possible.

I can also understand that the authors focus on smaller/easier settings in such a theoretical paper, and I appreciate that things are properly worked out at this level first, however I would appreciate it if the authors could keep some attention on the (presumably) goal of applying this to larger scales, or point to/explain more the parts of the paper that make this plausibly feasible in the future in general.

Reviewer_vLbUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies finite-perturbation sensitivity in hard-ReLU training, focusing on the regime where the perturbation radius is proportional to the GD step size. The authors derive a recursive branch-response formula with error bounds to capture discrete activation-switching effects that conventional sensitivities may miss. Experiments on simple systems and small ReLU networks support the theory, but provide limited evidence of practical training benefits.

Critical strengths
  1. The theoretical results are carefully developed, with detailed proofs and clearly stated assumptions, and appear mathematically sound.

  2. The paper provides a clear characterization of finite-radius sensitivity, particularly when the perturbation radius is comparable to the GD step size.

  3. The numerical experiments and ablation studies effectively illustrate the proposed branch-switching mechanisms and support the theoretical findings.

  4. The authors explicitly acknowledge the restrictive theoretical assumptions and the limitations of the method's practical applicability.

Critical weaknesses
  1. The main ideas are closely related to existing work on nonsmooth sensitivity and finite perturbation analysis (e.g., Stewart & Anitescu, 2010; Ho & Cao, 1991; Wardi et al., 2017). While the analysis of the specific perturbation regime is new, its additional significance beyond these established frameworks remains unclear.

  2. The analysis requires a fixed number of well-separated switching events, which may not hold in larger neural networks.

  3. It remains unclear how the theoretical results can guide actual optimization or improve neural network training.

  4. The experiments mainly focus on simple systems and very small ReLU networks, without demonstrating meaningful improvements in practical optimization performance.

Questions for authors

In addition to the concerns mentioned above, I have a few questions:

  1. Could the authors clarify the main new insights provided by the theoretical analysis beyond existing nonsmooth sensitivity and finite perturbation analysis?

  2. How can the theoretical results provide practical guidance for choosing perturbation radii or improving optimization?

  3. Could the authors include larger-scale experiments or realistic bilevel optimization tasks to demonstrate practical benefits?

  4. How relevant is the regime ϵ=Θ(η)\epsilon=\Theta(\eta) in practical neural network training? Is this regime commonly encountered?

  5. How applicable are the theoretical results when the number of activation-switching events increases with network size or training duration?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you for the time and care you put into this discussion

As the discussion period comes to a close, we’d like to thank the Area Chair for overseeing the review and all the reviewers for reading our paper closely, working through our responses, and taking the time to ask follow-up questions. We appreciate the care you took in pointing out both what was clear and where we still had work to do.

Your feedback changed how we present the paper. It led us to revisit the experiments and explain more clearly when a change in activation timing affects a decision, how large the observed gains are, and which comparisons we can certify. The follow-up questions were especially helpful in showing us where a distinction that seemed clear to us as authors still needed a better explanation.

This was my first time taking part in ICLR as a student author, and it was an honor to be part of the discussion. I learned a lot from seeing how careful questions can make a paper clearer and its claims more precise. That’s something I’ll carry into my future work. I hope the ICLR community continues to grow as a place where people can ask candid questions, learn from one another, and help each other improve their work.

Thank you again for your time and for helping us make the paper better.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.