acceptodds
Under review as a conference paper at ICLR 2027

WHAT EMPIRICAL TRAINING DYNAMICS DO NOT DETERMINE: RELU HISTORIES AND ROBUST PREDICTION SEPARATION

Abstract

We study whether a ReLU network's empirical training record determines its breakpoint history, the order in which its breakpoints cross during training. The record includes training predictions, loss, the empirical neural tangent kernel, parameter and gradient norms, and the Hessian spectrum. For shallow one-dimensional networks with two training samples, we find an open set of initial parameters. From each point in this set, every stretchable complete reversal can occur from another initialization with the same record on a common finite gradient-flow interval. We also give a common gradient-descent schedule. A three-sample construction gives a prediction gap above 0.04 on a fixed interval after 16 steps at learning rate 0.005. This gap survives prescribed data perturbations and independent initialization and update errors. Further examples use two-dimensional data and two hidden layers, including a training activation change. These are selected constructions, not claims about typical initialization or generalization.

Primary area:
learning theory
Keywords:
ReLU networks, training dynamics, learning theory, combinatorial geometry, gradient flow

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_1NxKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies toy-sized ReLU networks and their training artefacts when tasked with learning simple piecewise functions. It is hard to follow along and to locate the research question it tries to tackle. The empirical data fails to add value to the story, the theorem comes without a proof and formal definitions of the vocabulary used, and the appendix remains largely unrelated.

Critical strengths

None

Critical weaknesses

The goal of the paper is not clear, and the execution is unclean. The learnability of piecewise linear functions with ReLU networks is unsurprising. The empirical examples and data do not follow a clear, research-question-guided setup and remain meaningless to me.

Questions for authors

What are the main goals of the work and what are you trying to show with the experiments? What does Theorem 3.1 add to our knowledge about universal approximation/convergence results for ReLU networks and gradient descent? How does it link to stochastic gradient descent?

1 comment

Thank you for your comments and questions. We appreciate your feedback on the research question and on what the experiments are intended to show. We will address these points in future revisions and improve the presentation of the definitions and theoretical arguments.

Your suggestions will also inform our future work, where we plan to study more general settings with more training samples and deeper networks.

Reviewer_cfQ9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the question of whether the observed external history can determine the training process. The authors show in Theorem 3.1 that, even if the external history recordings coincide, a learning procedure might instantiate any break-point crossing ordering. Then, the paper constructs a few examples in a relaxed setup to validate their claim more broadly.

Critical strengths

(1) I consider Theorem 3.1 an interesting result. It constructs cases where the external appearance of training coincides, but the learning process actually differs.

(2) The authors provided extensive experiments to verify their conclusion.

Critical weaknesses

(1) Overall, I find the substantive contribution insufficient. Theorem 3.1 is a novel, but nevertheless niche, contribution in a narrow setting. Section 4 can be shortened or partly moved to the appendix. The main point of that section is that the conclusion goes beyond the strict setup of the theorem, which can be presented in a more concise way. Section 6, in my view, serves little to support the main point of the paper, and most of it can be moved to the appendix. Overall, I would suggest the author to deepen their theoretical discussion and extract more comprehensive insight into the problem. Perhaps there could be other results in other setups?

(2) The practical and theoretical value of the result is insufficient. From a practical perspective, normally, people train a network to reach a certain validation loss. We rarely care whether two same-looking loss curves imply the same learning process. From a theoretical perspective, this paper refutes the idea that the same training diagnostic history implies the same learning process. Nevertheless, I don't think anyone really expected or cared about that (Perhaps the author can provide a few more papers in the related work part to better contextualize their results?). I agree that this paper might be understood as a "pioneer" in the discussion of this problem, but I cannot see the significance of such discussions.

(3) The writing of this paper needs improvement. Section 6 is not understandable, with too much experimental minutiae and few takeaway points.

Questions for authors

(1) The motivation for E2 is unclear. Independent perturbations of θA\theta_A and θB\theta_B would generally destroy the exact equality of their empirical training records. Although E2 shows that the different crossing histories and off-sample prediction gap persist, how does this support the paper’s central claim about ambiguity under identical records?

(2) Related work on function- or loss-preserving transformations and network embeddings includes [1,2] in differentiable settings and [3,4] for ReLU-like networks, with [4] also treating smooth activations. The similar symmetry framework of Zhao et al. (2023) appears closely related to the mechanism explored in E5. Could the authors discuss whether [1–4], particularly the ReLU results in [3,4], provide further insight into the construction underlying E5?

(3) The appendix is rather messy. There seems to be a lot of AI-generated development/verification logs. You need to clean it up.

[1] Şimşek et al. (2021). Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances. ICML.

[2] Zhang et al. (2021). Embedding Principle of Loss Landscape of Deep Neural Networks. NeurIPS.

[3] Wu et al. (2025). Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding. ICLR.

[4] Fukumizu et al. (2019). Semi-flat Minima and Saddle Points by Embedding Neural Networks to Overparameterization. NeurIPS.

1 comment

Thank you for the detailed feedback. We are glad that you found Theorem 3.1 interesting, and we appreciate the references you suggested.

In future revisions, we will clarify the role of E2, discuss how E5 relates to the work on network symmetries and embeddings, and simplify the main text and appendix. We also plan to explore extensions to larger training sets and deeper networks. Thank you for the time you spent on the paper and for these helpful suggestions.

Reviewer_bnF3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the the dynamics of learning of ReLU networks focusing on the times when the ReLU function changes its slope when evaluated on training points in addition to looking at objects like the loss, the NTK, parameter and gradient norms, and the NTK. Examples are done on very small toy networks (1 or 2 inputs, 1 output, 2,3,4 hidden units) etc...

Critical strengths

This paper works out a specific question on specific small toy examples.

Critical weaknesses

The questions addressed by this paper are esoteric. They are addressed in extremely toy examples. Any theoretical results are again esoteric - I can't imagine a wide audience who would be interested in these questions. It does not have any apparent impact on the practitioners of deep learning. Not at all clear what generalizable lessons are learned.

Questions for authors

Why would any general audience, including practioners of deep learning care about this result? What design principles can they take away from this? (I am willing to learn).

1 comment

Thank you for reading our paper and for your comments on the scope and relevance of the results. We appreciate your point about the need to explain why these findings matter more broadly. In future work, we plan to explore larger training sets and deeper networks, and to examine what these constructions can tell us about neural network training more generally. Thank you for your time and suggestions.

Comments on the paper

2 comments

Dear Reviewers,

I'm writing to gently remind you that the rebuttal discussion phase deadline is approaching. Please take a moment to read the authors' rebuttal, and feel free to respond if any concerns remain that you'd like them to address. Thank you very much for your time and effort.

Reviewer_1NxKOpenReview ↗

no responses so far

Dear AC,

I kindly note that there has been no response from the authors so far.

Best, a reviewer

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.