acceptodds
Under review as a conference paper at ICLR 2027

Alignment and Convergence Analyses of Gradient Flow in Two-Layer Convolutional Neural Networks

Abstract

Understanding why gradient-based training of neural networks learns to extract signal and ignore noise remains a central open problem in deep learning theory. We take a step toward this goal by analyzing the gradient-flow dynamics of two-layer ReLU convolutional networks with sum-pooling, trained on the exponential or logistic loss for binary classification from a small, balanced initialization. Each input consists of one signal patch and some noise patches; the signal patches of the two classes lie in two well-separated, nearly antipodal cones, so that the label-signed signal patches share a common axis , and the noise patches are orthogonal to the signal subspace. Our main result indicates that during the early phase of the training, every filter that activates a signal patch of its own class at initialization aligns with either or , depending on the sign of its second-layer weight, to a prescribed cosine and within an explicit time determined by the signal geometry; such a filter thereafter activates every signal patch of its class and none of the other class. The alignment is driven by a self-reinforcing feedback mechanism. Since same-class signal patches lie in a narrow cone, every activated signal patch pushes the filter toward all the others, thus toward the axis. The noise patches, being orthogonal to the signal subspace, enter the analysis only through a single scalar that bounds their aggregate effect. In the second phase of the training ,the projection of the aligned filter onto grows monotonically, and the training loss decays at rate . Experiments on synthetic and MNIST data confirm the two phases.

Primary area:
optimization
Keywords:
Neural Alignment, Convergence Analysis, Convolutional Neural Networks, Gradient Flow

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_v5WAOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper provides a non-asymptotic analysis of gradient flow for a two-layer ReLU convolutional network with shared filters and sum-pooling, trained for binary classification from a sufficiently small, balanced initialization. Its setting builds on the signal–noise patch models studied by Cao et al. and Kou et al.: each input contains one signal patch and additional noise patches. The signal patches lie in two sufficiently narrow, nearly antipodal cones, so that the label-signed signal patches are concentrated around a common axis u⋆u_\star. The noise patches are assumed to be orthogonal to the signal subspace. Under these assumptions, the paper shows quantitative early alignment for filters that initially activate a signal patch of their corresponding class. When both classes have such filters, it also proves convergence of the training loss at rate (O(1/t)).

The paper presents a clearly explained and technically useful extension of early-alignment theory to convolutional networks. Its conceptual novelty is moderate, but the quantitative analysis is valuable.

Critical strengths

Understanding how gradient-based training learns features in simple neural architectures under structured data assumptions is an important problem in deep learning theory. This paper extends the early-alignment analysis developed for fully connected two-layer ReLU networks to convolutional filters acting on signal and noise patches. Although earlier work has already analyzed two-layer CNNs, the explicit description of filter directions and alignment times provides a useful complementary result.

The manuscript is clearly written and well organized. The setting, assumptions, and main results are stated carefully, and the proof sketch explains the central mechanism effectively. The comparison with prior work also helps identify which parts of the analysis are inherited and which require additional arguments.

The technical contribution is strong. In particular, the analysis bounds the combined contribution of the active noise patches, uniformly over all filter directions and uses a potential function to obtain an explicit alignment-time bound. These arguments address obstacles that prevent a direct application of the earlier fully connected analysis. The main proof arguments appear sound for the globally defined gradient-flow trajectories assumed in the paper.

Critical weaknesses

The qualitative mechanism (early directional alignment followed by fitting) is already familiar from previous work on two-layer ReLU networks. The contribution is therefore primarily a quantitative extension to a structured convolutional setting, rather than a new learning mechanism. I nevertheless consider the extension technically useful.

The scope of the guarantee is limited by exact signal–noise orthogonality and a restrictive cone-width condition (θ∗\theta_* small). Moreover, the explicit transition-time bound can be very large when Γ>1\Gamma>1. The authors acknowledge these limitations. A discussion of what the theorem explains about convolutional architectures (in particular, what the analysis reveals specifically about weight sharing and sum-pooling) and what remains outside its scope would strengthen the paper. For example, do the results imply that CNNs learn faster than MLPs, or that weight sharing improves sample complexity or leads to a different implicit bias?

The results are conditional on the existence of a global Carathéodory solution to the specified gradient-flow equation. Appendix B explicitly leaves this question open. Since the ReLU update is discontinuous, this is a substantive modeling issue that deserves clarification in the main presentation.

Minor correction:

On p. 28, in the proof of Lemma 17, the intermediate inequality ℓi(t1)≤e1/2\ell_i(t_1)\le e^{1/2} does not follow from ∣fi(t1)∣≤1/2|f_i(t_1)|\le1/2 for the normalized logistic loss. At margin (-1/2), that loss equals (2\log(1+e^{1/2})>e^{1/2}). The proof can instead use ℓi(t1)≤2log⁡(1+e1/2)\ell_i(t_1)\le2\log(1+e^{1/2}) directly. This appears to leave the lemma and all subsequent constants unchanged.

Questions for authors
  1. On p. 15, lines 789–790, you state that existence is not addressed. Do the current assumptions ensure global existence for the convention σ′(z)=1z>0\sigma'(z)=\mathbf 1_{{z>0}}? If the assumptions do not exclude this, could you identify sufficient additional conditions, or explain whether a differential-inclusion formulation would preserve the results?

  2. For the proposed data model, what do weight sharing and patchwise activation contribute beyond the corresponding fully connected analysis? A conceptual comparison with an MLP trained on the concatenated patches would help clarify the architectural significance of the result.

  3. Which step of the current analysis relies most critically on the two-layer structure? What would be the main obstacle to extending the alignment argument to a network with an additional trainable layer?

5 comments
AuthorsOpenReview ↗

Reply to Question 1

We sincerely thank the reviewer for their careful reading of our manuscript and constructive feedback. We particularly appreciate the reviewer's insightful comments on the strength of assumptions and the existence of solutions. It has directly helped us improve the readability of our manuscript and the soundness of both theory and the experimental design. We have uploaded a revised PDF. All changes are marked in blue, and the numbering of the existing equations, lemmas and propositions is unchanged; new material is appended (Appendices A.3, A.4, F and G).

Question 1

With the convention σ′(z)=1{z>0}\sigma'(z)=\mathbf{1}_{\lbrace z>0\rbrace } fixed in the paper (in particular σ′(0)=0\sigma'(0)=0), an absolutely continuous solution satisfying (2) almost everywhere does not always exist under Assumptions 1–4. We found an example satisfying all assumptions in which no absolutely continuous trajectory satisfying (2) exists beyond a finite time, which is smaller than t1t_1 (Proposition 5, Appendix G).

Following your advice, we now write (2) as the differential inclusion θ˙∈−∂L(θ)\dot\theta\in-\partial L(\theta) with the Clarke subdifferential, which is the standard notion of gradient flow for nonsmooth losses such as those of ReLU networks (Lyu & Li, 2020; Ji & Telgarsky, 2020). For this formulation, global solutions always exist, and Theorem 1 holds for every solution with the same assumptions and the same constants. The changes to our proofs are local. In the original proofs a Carathéodory solution is used in only two ways: the trajectory is absolutely continuous, so that differential inequalities can be integrated, and equation (2) bounds the time derivatives of the filter norms, the cosine with u⋆u_\star, the signal coordinates and the loss. Solutions of the inclusion are also absolutely continuous and satisfy the same bounds, so the same arguments prove Theorem 1.

The solutions themselves are concrete objects. For almost every tt the velocity is minus the minimal-norm element of ∂L(θ(t))\partial L(\theta(t)) (Davis et al., 2020, Lemma 5.2; Ji & Telgarsky, 2020, Lemma B.9); Proposition 4 (Appendix G) gives its explicit form for our network.

Section 2.1 now states the solution concept and the existence result. The new Appendix G contains the definition (Definition 2), global existence and the energy identity (Proposition 3), the minimal-norm property (Proposition 4) and the counterexample (Proposition 5). The proofs in Appendices B, D and E are adapted accordingly; the statement of Theorem 1 is unchanged.

AuthorsOpenReview ↗

Reply to Question 2

Question 2

The CNN and the MLP on concatenated patches differ in two independent choices: whether the filter is shared across positions, and whether the ReLU acts on each patch or on their sum. We discuss them separately and then report an experiment that crosses them.

Weight sharing is where the CNN gains. A shared filter is pulled by every signal patch wherever it sits, so the drive is the full class mass R±R_\pm, and Theorem 1 does not depend on the position of the signal patch. For the concatenated MLP the answer depends on the positions. If the signal always sits at the same position and the noise is small, the signed inputs are pairwise positively correlated, Min et al. (2024) apply directly, and our result adds nothing. If the position varies, each weight block is pulled only by the inputs whose signal sits at that position, and the drive toward (u⋆,…,u⋆)/P(u_\star,\dots,u_\star)/\sqrt P is only of order R±/PR_\pm/\sqrt P.

Patchwise activation is a convenience for the proof and a cost, not an empirical gain. The gate of a signal patch depends on ⟨wj,si⟩\langle w_j,s_i\rangle alone, and by orthogonality the derivative of ⟨wj,gi⟩\langle w_j,g_i\rangle contains no noise term, so one-way switching holds at any noise level. The cost is that each noise patch is gated separately: the noise patches of an input cannot cancel before the ReLU, and an aligned filter keeps activating noise patches of both classes. This is where GG and our weak bound on t2t_2 come from.

Experiment (new; Appendix A.4, Table 2). We train the four two-layer networks obtained by crossing the two choices, on the same data (θ⋆=4∘\theta_\star=4^\circ, n=40n=40, P=5P=5) with the signal patch at a random position in each input. The table gives the mean alignment of the hidden units when the loss starts to decrease (noise patch norm 0.1ρ0.1 \rho):

shared filterone filter per position
ReLU per patch0.999 (our CNN)0.83
ReLU after the sum0.9980.946 (concatenated MLP)

The two networks with a shared filter are indistinguishable. The two without it align about three times more slowly and begin to fit twice as late; with noise patches as large as the signal patches, the gap widens (0.93 and 0.92 against 0.60 and 0.69). When the signal patch is kept at a fixed position, the concatenated MLP begins to fit as early as the CNN. In this data model the architectural gain is therefore weight sharing, and the patchwise ReLU makes no measurable difference.

We added Appendix A.4 to report the comparison of the four architectures and a new paragraph, “Role of the convolutional structure”, to Section 3.1.

AuthorsOpenReview ↗

Replies to Question 3 and Weakness; References

Question 3

The step that relies most on it is the balancedness in Lemma 3 together with the directional dynamics (8). With two layers, the coefficient that back-propagation attaches to filter jj is the scalar vjv_j: it is the same for every input, its sign is fixed, and by balancedness its size is $\lVert w_j\rVert $. The direction of each filter then obeys its own equation on the sphere, with unit speed and independent of the other filters. Three things rest on this: the fixed split into V±\mathcal V_\pm; the positivity of the drive on the signal coordinates, ⟨zA,gi⟩≥μρiRA\langle z_{\mathcal A},g_i\rangle\ge\mu\rho_iR_{\mathcal A}, which gives one-way switching (Lemmas 12–13); and the control of the noise by a single quantity GG, since all activated noise patches enter with the same weight.

With an additional trainable layer, f=a⊤σ(VΦ(X))f=a^\top\sigma(V\Phi(X)) with Φj\Phi_j the pooled response of filter jj, the coefficient becomes cij=∑kakσ′(hik)Vkjc_{ij}=\sum_k a_k\sigma'(h_{ik})V_{kj}. (i) It depends on the input and has no fixed sign, so the drive on a signal coordinate, ∑k∈Ackjλk⟨gk,gi⟩\sum_{k\in\mathcal A}c_{kj}\lambda_k\langle g_k,g_i\rangle, can be negative, and one-way switching fails. (ii) The noise enters with input-dependent weights, so one level GG no longer controls it; this is the same difficulty that limits our bound on t2t_2, where the loss weights become unequal, now present from the start. (iii) The speed of a filter direction is proportional to the size of the upper layers, of order ϵ\epsilon, so alignment has to be compared with the window in a rescaled time (cf. Kumar & Haupt (2025) for homogeneous networks with locally Lipschitz gradients).

Overall if the added layer is linear, cij=(V⊤a)jc_{ij}=(V^\top a)_j does not depend on ii, and with a balanced initialization across layers we expect its sign to stay fixed, so that the filters follow the two-layer dynamics in the clock dτ=∥a∥dtd\tau=\lVert a\rVert dt and Part 1 extends. With a ReLU in the added layer, the question becomes whether the upper layers become approximately rank one, giving the filters an approximately fixed sign, before the filters have to align. We regard this as the main open problem; the conclusion now states the obstacle.

Weakness: Exact orthogonality and t2t_2 bound

Exact orthogonality is used in two places, and the answer is different for each.

(1) Early alignment (Theorem 1, Part 1) tolerates a deviation. Here orthogonality is used only to ensure that the noise has no component along u⋆u_\star or along the signal patches (Lemma 7). We now prove (Proposition 2, Appendix F) that Part 1 still holds for noise that is not exactly orthogonal to the signal subspace, provided the signal-subspace component of the activated noise is at most half the tolerance ϱ1\varrho_1 that the proof already allows for the small-initialization approximation error. In the proof, this component plays exactly the role of that approximation error.

(2) Final convergence (Theorem 1, Part 2) needs exact orthogonality with our proof. Noise–signal orthogonality removes direct noise terms from the signal-projected dynamics, so positive signal correlations keep filters inside the signal cones once they enter (Lemma 15). After the early phase, the loss weights differ across input classes. The signal term acting on a filter involves only the inputs of its own class, whereas a noise component inside the signal subspace involves all inputs, and we have no control on the ratio of loss weights across classes. This is the same obstacle that makes our bound on t2t_2 weak. We now state it as a limitation.

The new Appendix F proves that Part 1 holds without exact orthogonality (Proposition 2) and explains why our proof of Part 2 needs it. Section 2.2 mentions this after Assumption 1, and the conclusion lists the restriction on Part 2 as a limitation.

Minor correction (Lemma 17)

Thank you for this correction. The proof now bounds the logistic loss by 2log⁡(1+e1/2)2\log(1+e^{1/2}) directly, as you suggest; the statement of the lemma and all constants are unchanged.

References:

Davis, D., Drusvyatskiy, D., Kakade, S., & Lee, J. D. (2020). Stochastic subgradient method converges on tame functions. Foundations of Computational Mathematics, 20(1), 119–154.

Ji, Z., & Telgarsky, M. (2020). Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems (NeurIPS) 33.

Kumar, A., & Haupt, J. (2025). Early directional convergence in deep homogeneous neural networks for small initializations. Transactions on Machine Learning Research.

Lyu, K., & Li, J. (2020). Gradient descent maximizes the margin of homogeneous neural networks. International Conference on Learning Representations (ICLR).

Min, H., Mallada, E., & Vidal, R. (2024). Early neuron alignment in two-layer ReLU networks with small initialization. International Conference on Learning Representations (ICLR).

Reviewer_YUMNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the training dynamics of a two-layer ReLU CNN under small balanced initialization. It shows that suitably initialized filters align with the signal direction and that the training loss subsequently converges to zero.

Critical strengths

The paper gives explicit bounds on filter alignment and subsequent loss convergence in the presence of noise, with both layers trained. Controlling the full filter direction, rather than only its signal component, provides useful information about how early training develops a preference for the signal.

Critical weaknesses

The assumptions on the data are strong. The signal patches must lie in very narrow cones, and the noise must be exactly orthogonal to the entire signal subspace. It is unclear how much the conclusions depend on this exact separation. I would like to see more explanation of what happens when the noise has a small component along the signal directions.

The noise condition in Assumption 4 also seems restrictive. It requires controlling the noise relative to a single signal patch, even when there are many training samples. The authors acknowledge this limitation, but it is difficult to tell how much it limits the settings covered by the theorem.

Questions for authors

1.For the data models considered in the paper, what ranges of sample size, cone angle, and noise level satisfy the second condition in Assumption 4? This would help assess how restrictive the condition is.

2.Which steps require exact orthogonality between noise and the signal subspace, and can they tolerate a small deviation from orthogonality?

5 comments
AuthorsOpenReview ↗

Reply to Question 1

We sincerely thank the reviewer for their careful reading of our manuscript and constructive feedback. We particularly appreciate the reviewer's insightful questions on the validity of assumptions. It has directly helped us improve the readability of our manuscript and the soundness of our experimental design. We have uploaded a revised PDF. All changes are marked in blue, and the numbering of the existing equations, lemmas and propositions is unchanged; new material is appended (Appendices A.3, A.4, F and G).

Question 1 and Weakness: Assumption 4

When the condition is void. If all signal patches of a class are parallel (θ⋆=0\theta_\star=0), which is the data model of Cao et al. (2022) and Kou et al. (2023), the condition holds for every sample size and every noise level. It constrains only our extension from a fixed signal direction to cones of signal patches.

Explicit form for θ⋆>0\theta_\star>0. The condition is Gsin⁡θ⋆≤ρmin⁡/48G\sin\theta_\star\le\rho_{\min}/48. For Gaussian noise with per-coordinate standard deviation σξ\sigma_\xi, we measure G≈cσξn(P−1)DG\approx c \sigma_\xi\sqrt{n(P-1)D} with cc between 1.1 and 1.3 for $P\in\lbrace 2,3,5\rbrace $ and nn from 40 to 400, so the condition reads sin⁡θ⋆≲ρmin⁡48cσξn(P−1)D.\sin\theta_\star \lesssim \frac{\rho_{\min}}{48 c \sigma_\xi\sqrt{n(P-1)D}}. The admissible angle is inversely proportional to the noise level and shrinks like 1/n(P−1)1/\sqrt{n(P-1)}.

Ranges. The table gives the largest admissible value of 2θ⋆2\theta_\star, the maximal angle between two label-signed signal patches. The noise level is expressed by the norm of a noise patch (about σξD\sigma_\xi\sqrt D) relative to that of a signal patch. We show P=2P=2 (one signal and one noise patch, as in Cao et al. 2022) and P=5P=5 (our experiments).

nnP=2P=2, noise 5%P=2P=2, noise 1%P=5P=5, noise 5%P=5P=5, noise 1%
406.3∘6.3^\circ14∘14^\circ †3.1∘3.1^\circ14∘14^\circ †
1004.0∘4.0^\circ14∘14^\circ †2.0∘2.0^\circ10∘10^\circ
2002.8∘2.8^\circ14∘14^\circ1.4∘1.4^\circ7.0∘7.0^\circ
4002.0∘2.0^\circ10∘10^\circ1.0∘1.0^\circ5.0∘5.0^\circ

† Limited by the cone-width condition, Assumption 3(b), which gives 2θ⋆≤14∘2\theta_\star\le14^\circ, and not by this condition. The values use c=1.2c=1.2.

Also, for Theorem 1, the constant 1/481/48 is not optimized. It is needed only for parts (b) and (c) of the auxiliary Proposition 1. For Theorem 1 itself, the proof of Lemma 14(ii) goes through with Gsin⁡θ⋆≤ρmin⁡/2G\sin\theta_\star\le\rho_{\min}/2 and the same t1t_1; with this constant every entry of the table equals the 14∘14^\circ limit of Assumption 3(b).

Why the condition has this form. It compares the aggregate noise GG, which grows like n\sqrt n, with the mass of a single signal patch. A filter nearly orthogonal to the common axis may activate only one signal patch, and our drift bound requires that patch alone to dominate the noise at every instant. It is a sufficient condition from a worst-case argument.

Whether it plays a crucial role empirically. Our experiments indicate that it does not. We added an experiment designed for this question. The reference run satisfies Assumptions 1–4 for the realized data (θ⋆=4∘\theta_\star=4^\circ, noise patch norm ρmin⁡/150\rho_{\min}/150, n=40n=40, P=5P=5, D=100D=100, balanced initialization; Assumption 4 is certified with a rigorous upper bound on GG). We then violate one condition at a time and record the mean alignment cˉ\bar c at the moment the loss starts to decrease. We violate this condition alone (n=400n=400, θ⋆=4∘\theta_\star=4^\circ, noise patch norm 0.04ρmin⁡0.04 \rho_{\min}): it is exceeded by a factor between 6 and 27 (relative to the stated constant 1/481/48), and the alignment at the onset of fitting is unchanged (0.999 in both runs; Table 1, Appendix A.3). We therefore regard the condition as a cost of our worst-case, instantaneous drift bound rather than as a property of the dynamics. Whether an assumption on how the patches are spread inside their cones can remove it is open, and we state this condition as a limitation.

We have modified the current manuscript. Section 2.2 (“How restrictive the assumptions are”) now gives the scaling of the admissible cone angle with the sample size and the noise level. Table 1 (Appendix A.3) reports an experiment in which this condition alone is violated. The conclusion lists the condition as a limitation.

AuthorsOpenReview ↗

Reply to Question 2; References

Question 2 and Weakness: Exact orthogonality

Exact orthogonality is used in two places, and the answer is different for each.

(1) Early alignment (Theorem 1, Part 1) tolerates a deviation. Here orthogonality is used only to ensure that the noise has no component along u⋆u_\star or along the signal patches (Lemma 7). We now prove (Proposition 2, Appendix F) that Part 1 still holds for noise that is not exactly orthogonal to the signal subspace, provided the signal-subspace component of the activated noise is at most half the tolerance ϱ1\varrho_1 that the proof already allows for the small-initialization approximation error. In the proof, this component plays exactly the role of that approximation error.

(2) Final convergence (Theorem 1, Part 2) needs exact orthogonality with our proof. Noise–signal orthogonality removes direct noise terms from the signal-projected dynamics, so positive signal correlations keep filters inside the signal cones once they enter (Lemma 15). After the early phase, the loss weights differ across input classes. The signal term acting on a filter involves only the inputs of its own class, whereas a noise component inside the signal subspace involves all inputs, and we have no control on the ratio of loss weights across classes. We now state it as a limitation.

We also tested this empirically, in the experiment described in our reply to Question 1. With isotropic Gaussian noise, the signal-subspace component of the noise exceeds the tolerance of Proposition 2 at least fourfold, and the alignment at the onset of fitting is 0.997, against 0.999 in the reference run (Table 1, Appendix A.3); the two phases are unchanged.

The new Appendix F proves that Part 1 holds without exact orthogonality (Proposition 2) and explains why our proof of Part 2 needs it. Section 2.2 mentions this after Assumption 1, and the conclusion lists the restriction on Part 2 as a limitation.

References:

Cao, Y., Chen, Z., Belkin, M., & Gu, Q. (2022). Benign overfitting in two-layer convolutional neural networks. Advances in Neural Information Processing Systems (NeurIPS) 35.

Kou, Y., Chen, Z., Chen, Y., & Gu, Q. (2023). Benign overfitting in two-layer ReLU convolutional neural networks. Proceedings of the 40th International Conference on Machine Learning (ICML), PMLR 202, 17615–17659.

Dear Reviewer,

We would like to kindly follow up on our previous response. We hope that our explanations have addressed your concerns. Please let us know if you have any remaining questions or concerns. We would be happy to provide further clarification.

Thanks.

Reviewer_kT47OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the training dynamic of a two-layer ReLU convolutional network with sum pooling under exponential and logistic losses. Under some assumptions, the authors establish two distinct phases: an early alignment phase and a second phase where the projection of each wjw_j onto the associated axis ±u⋆\pm u_\star grows and where the convergence rate is O(1/t). Experiments on MNIST dataset confirm these two phases.

Critical strengths

S.1 This paper is clearly written, and well-motivated.

S.2. The proofs I've read are rigorous.

The code is provided.

Critical weaknesses

W.1 The proofs of the technical results seem particularly designed for viv_i scalars; and also for balanced init with non zero columns init.

W.2 It is not clear how strong are the assumptions 1--4. Could the authors detail that?

W.3 About Section 4, the comparison with the results Theorem 1 is not clear. In particular, the discussion about \hat{t}_1 and \hat{t}_2 and theorem 1 is not clear. Q_{\pm} is defined by summing only on \hat{\mathcal{V}}_j^{\pm} (which is not done in the code provided), and it seems normal that it should not be normal not to use Q±Q_{\pm} as balanced initializations give a constant sign, which not the case anymore with unbalanced init. (minor in appendix A, Q±Q_{\pm} is not defined. It is only defined in appendix E equation 43)

Questions for authors

Q.1 + Q.2 + Q.3: could the authors comment W1 & W2?

Q.4 The experiments are not done with all assumptions of theorem 1 (which seems partly a good thing, as it supports the two phases beyond these assumptions if the assumptions are strong). However, it is not clear which assumption plays a crucial role or not. It is not clear either which assumption is strong or not.

Q.5 I would like to have more details about the solutions of (2). Do we always have the existence of AC ae solutions?

Q.6 is lemma 3 a known result?

minor: P is not yet defined at line 098

5 comments
AuthorsOpenReview ↗

Replies to Question 1 and Question 2

We sincerely thank the reviewer for their careful reading of our manuscript and constructive feedback. We particularly appreciate the reviewer's insightful questions on the strength of assumtions and the existence of solutions. It has directly helped us improve the readability of our manuscript and the soundness of our experimental design. We have uploaded a revised PDF. All changes are marked in blue, and the numbering of the existing equations, lemmas and propositions is unchanged; new material is appended (Appendices A.3, A.4, F and G).

Question 1 or Weakness 1

As to nonzero initialization, it is without loss of generality. A neuron with wj(0)=0w_j(0)=0 has vj(0)=0v_j(0)=0 under (3) and stays at zero for all times, so it never contributes to the network.

Balanced initialization is the standard example in the literature (Arora et al., 2018; Arora et al., 2019; Phuong & Lampert, 2021; Boursier et al., 2022; Chistikov et al., 2023; Dominé et al., 2025; Min et al., 2025). It is a device of the analysis, not a condition for the phenomenon. And we do not rely on balancedness in the experiments. But relaxing balanced initialization could be a relevant future direction of our research.

Balanced initialization is used in three places: (i) the sign of vjv_j is constant (Lemma 3), which splits the neurons once and for all into V+\mathcal V_+ and V−\mathcal V_-; (ii) ∣vj∣=∥wj∥|v_j|=\|w_j\|, so in (8) the direction of every filter moves with unit speed factor; (iii) the network outputs are controlled by the filter norms alone (Lemma 4). If ∣vj(0)∣≥∥wj(0)∥|v_j(0)|\ge\|w_j(0)\|, the conserved quantity vj2−∥wj∥2≥0v_j^2-\|w_j\|^2\ge0 still fixes the sign of vjv_j, and (ii) only multiplies the directional dynamics by ∣vj∣/∥wj∥≥1|v_j|/\|w_j\|\ge1; we expect Part 1 of Theorem 1 to extend to this case. If ∣vj(0)∣<∥wj(0)∥|v_j(0)|<\|w_j(0)\|, vjv_j can change sign and a filter can switch from one class to the other. This can happen under the Gaussian initialization of our experiments, which the theorem does not cover.

Question 2 or Weakness 2:

For each assumption we state how the proof relies on it and how restrictive it is. Below, we call a condition mild if it holds generically, moderate if it excludes part of the parameter range for a structural reason and does not tighten as nn grows, and strong if it confines a parameter to a narrow range or tightens as nn grows.

Assumption 1 (noise orthogonal to the signal subspace): strong but commonly used. It is an exact structural condition, standard for this patch model (Cao et al., 2022; Kou et al., 2023). We can prove that Part 1 of Theorem 1 still holds for noise that is not exactly orthogonal to the signal subspace, provided it is bounded by a constant. Part 2 of Theorem 1 needs exact orthogonality with our proof s. We will add a new section in the appendix to discuss it.

Assumption 2 (initialization margin): mild. It keeps filters away from the direction exactly opposite to the axis. It holds with probability one for a random initialization.

Assumption 3(a) (two cones, θ⋆<45∘\theta_\star<45^\circ): moderate. It makes all label-signed signal patches pairwise positively correlated, which gives the one-way switching of the signal activations. It is the analogue of the correlation assumption used for fully connected networks, imposed on the signal patches only.

Assumption 3(b) (cone width): strong. It keeps the sideways pull of the signal patches small enough for the cosine with the common axis u⋆u_\star to reach c∗c^*. Much of this comes from measuring alignment with u⋆u_\star.

Assumption 4, first condition (noise versus class mass): moderate. A filter that leans toward ±u⋆\pm u_\star is pulled by all signal patches of a class, and the noise must be a fraction of that pull. Since Rmin⁡R_{\min} grows linearly in nn and GG like n\sqrt n, the condition relaxes as nn grows.

Assumption 4, second condition (noise versus one patch): strong when θ⋆>0\theta_\star>0. A filter nearly orthogonal to u⋆u_\star may activate a single signal patch, and our bound compares the noise with that one patch. The condition tightens as nn grows.

We added a new paragraph to Section 2.2, “How restrictive the assumptions are”, which rates each assumption.

AuthorsOpenReview ↗

Replies to Question 3 and Question 4

Question 3 or Weakness 3

Thank you for pointing this out. In the revised draft, Q±Q_\pm is defined in the main text (Section 3.1) before it is used, and Section 4 and Appendix A.1 give precise definitions of t^1\hat t_1 and t^2\hat t_2. As you noted, under the unbalanced initialization of the experiments the sign of vjv_j is not fixed, so Q±Q_\pm is summed over the filters with the current sign of vjv_j, as in the code; Appendix A.1 now says so.

Relation to Theorem 1. t^1\hat t_1 and t^2\hat t_2 are landmarks of the two phases, not estimates of the bounds. In Theorem 1, t1t_1 in (9) is an upper bound on the time at which every filter in V~±\widetilde{\mathcal V}_\pm satisfies cos⁡(wj,±u⋆)≥c∗\cos(w_j,\pm u_\star)\ge c^\ast. t^1\hat t_1 is the first iteration at which every live filter lies, in the signal plane, within the cone half-angle α\alpha of the axis of its class, hence within θ⋆\theta_\star of ±u⋆\pm u_\star: it is then aligned with the common axis at least as well as the signed signal patches themselves. So t^1\hat t_1 is a measured alignment time and t1t_1 bounds such a time from above. For the base run, the value of t1t_1 in (9) corresponds to about 7.2×1047.2\times10^4 iterations (iteration kk corresponds to time kη/nk\eta/n), against t^1=5780\hat t_1=5780. The proof of Part 2 shows that the loss decays like O(1/t)O(1/t) as soon as the aligned filters have grown along ±u⋆\pm u_\star. Precisely, once Q±=∑j∈V~±⟨±u⋆,wj⟩2Q_\pm=\sum_{j\in\widetilde{\mathcal V}_\pm}\langle\pm u_\star,w_j\rangle^2 both exceed a threshold b0>0b_0>0, the gradient of these filters alone gives ∥∇L∥2≥cL2\lVert \nabla L\rVert ^2\ge c L^2 with a constant cc proportional to b0b_0, hence L˙≤−cL2\dot L\le -c L^2. Before this time the filters are small and the loss stays near its initial value. The time t2t_2 of Theorem 1 is the first time at which Q±≥b0Q_\pm\ge b_0, and t^2\hat t_2 is this time measured in the experiment.

Question 4

The current ablation studies in our paper start from a base run that already violates Assumption 3(b) and both conditions of Assumption 4, and uses an unbalanced initialization, so they cannot isolate the effect of a single assumption. We added an experiment designed for this question: which assumptions play a crucial role.

The reference run satisfies Assumptions 1–4 for the realized data (θ⋆=4∘\theta_\star=4^\circ, noise patch norm ρmin⁡/150\rho_{\min}/150, n=40n=40, P=5P=5, D=100D=100, balanced initialization; Assumption 4 is certified with a rigorous upper bound on GG). We then violate one condition at a time and record the alignment cˉ\bar c at the moment the loss starts to decrease.

Reference run: cˉ=0.999\bar c=0.999.

Violations with no effect on the two phases: Assumption 1, with isotropic Gaussian noise: cˉ=0.997\bar c=0.997. Assumption 3(b) exceeded tenfold (θ⋆=24∘\theta_\star=24^\circ): cˉ=0.976\bar c=0.976. The second condition of Assumption 4 exceeded by a factor between 66 and 2727: cˉ=0.999\bar c=0.999. Unbalanced Gaussian initialization: cˉ=0.999\bar c=0.999. Beyond Assumption 3(a), with θ⋆=50∘\theta_\star=50^\circ and 60∘60^\circ, the two phases persist and cˉ=0.898\bar c=0.898 and 0.8600.860, which follows cos⁡(θ⋆/2)\cos(\theta_\star/2): each filter aligns with the signal patches of its own class.

Violations that destroy the phenomenon: First condition of Assumption 4, tested with θ⋆=0\theta_\star=0 so that it is the only condition violated: cˉ=0.999, 0.991, 0.93, 0.60\bar c=0.999,\ 0.991,\ 0.93,\ 0.60 when the threshold is exceeded by factors of about 2, 7, 20, 702,\ 7,\ 20,\ 70. So the phenomenon needs a noise level below the class signal mass, which is what this condition expresses up to its constant. Initialization scale: cˉ=0.990\bar c=0.990, 0.4590.459 and 0.0480.048 for ϵ=10−3\epsilon=10^{-3}, 10−210^{-2} and 10−110^{-1}; for large ϵ\epsilon the fitting begins before the directions have converged.

We added Appendix A.3, which describes this experiment; Section 4 summarizes its conclusion.

AuthorsOpenReview ↗

Replies to Question 5 and Question 6; References

Question 5

No. With the convention σ′(z)=1{z>0}\sigma'(z)=\mathbf{1}_{\lbrace z>0\rbrace } fixed in the paper (in particular σ′(0)=0\sigma'(0)=0), an absolutely continuous solution satisfying (2) almost everywhere does not always exist under Assumptions 1–4. We found an example satisfying all assumptions in which no absolutely continuous trajectory satisfying (2) exists beyond a finite time, which is smaller than t1t_1 (Proposition 5, Appendix G).

We now write (2) as the differential inclusion θ˙∈−∂L(θ)\dot\theta\in-\partial L(\theta) with the Clarke subdifferential, which is the standard notion of gradient flow for nonsmooth losses such as those of ReLU networks (Lyu & Li, 2020; Ji & Telgarsky, 2020). For this formulation, global solutions always exist, and Theorem 1 holds for every solution with the same assumptions and the same constants. The changes to our proofs are local. In the original proofs a Carathéodory solution is used in only two ways: the trajectory is absolutely continuous, so that differential inequalities can be integrated, and equation (2) bounds the time derivatives of the filter norms, the cosine with u⋆u_\star, the signal coordinates and the loss. Solutions of the inclusion are also absolutely continuous and satisfy the same bounds, so the same arguments prove Theorem 1. The solutions themselves are concrete objects. For almost every tt the velocity is minus the minimal-norm element of ∂L(θ(t))\partial L(\theta(t)) (Davis et al., 2020, Lemma 5.2; Ji & Telgarsky, 2020, Lemma B.9); Proposition 4 (Appendix G) gives its explicit form for our network.

We have updated the main proof and added Appendix G to discuss the solutions.

Question 6

The full statement is established for two-layer fully connected ReLU networks under balanced, nonzero initialization by Boursier et al. (2022, Lemma 1). We are not aware of an explicit statement of the full lemma for our CNN architecture. Du et al. (2018, Theorems 2.1 and 2.3) establish neuron-wise balancedness for fully connected networks and layer-wise balancedness for CNNs. For our sum-pooling CNN, the same positive-homogeneity argument yields conservation of vj2−∥wj∥2v_j^2-\lVert w_j\rVert ^2 for each filter, while a Grönwall argument rules out finite-time vanishing and thus ensures sign preservation. Our lemma is therefore a direct adaptation of these established properties. A sentence before Lemma 3 (Appendix B) now gives these attributions.

References:

Arora, S., Cohen, N., & Hazan, E. (2018). On the optimization of deep networks: Implicit acceleration by overparameterization. International Conference on Machine Learning (ICML).

Arora, S., Cohen, N., Golowich, N., & Hu, W. (2019). A convergence analysis of gradient descent for deep linear neural networks. International Conference on Learning Representations (ICLR).

Boursier, E., Pillaud-Vivien, L., & Flammarion, N. (2022). Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs. Advances in Neural Information Processing Systems (NeurIPS) 35.

Cao, Y., Chen, Z., Belkin, M., & Gu, Q. (2022). Benign overfitting in two-layer convolutional neural networks. Advances in Neural Information Processing Systems (NeurIPS) 35.

Chistikov, D., Englert, M., & Lazić, R. (2023). Learning a neuron by a shallow ReLU network: Dynamics and implicit bias for correlated inputs. Advances in Neural Information Processing Systems (NeurIPS), 36, 23748–23760.

Davis, D., Drusvyatskiy, D., Kakade, S., & Lee, J. D. (2020). Stochastic subgradient method converges on tame functions. Foundations of Computational Mathematics, 20(1), 119–154.

Dominé, C. C. J., Anguita, N., Proca, A. M., Braun, L., Kunin, D., Mediano, P. A. M., & Saxe, A. M. (2025). From lazy to rich: Exact learning dynamics in deep linear networks. International Conference on Learning Representations (ICLR).

Du, S. S., Hu, W., & Lee, J. D. (2018). Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in Neural Information Processing Systems (NeurIPS), 31, 384–395.

Ji, Z., & Telgarsky, M. (2020). Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems (NeurIPS) 33.

Kou, Y., Chen, Z., Chen, Y., & Gu, Q. (2023). Benign overfitting in two-layer ReLU convolutional neural networks. Proceedings of the 40th International Conference on Machine Learning (ICML), PMLR 202, 17615–17659.

Lyu, K., & Li, J. (2020). Gradient descent maximizes the margin of homogeneous neural networks. International Conference on Learning Representations (ICLR).

Min, H., Zhu, Z., & Vidal, R. (2025). Neural collapse under gradient flow on shallow ReLU networks for orthogonally separable data. arXiv preprint arXiv:2510.21078.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.