- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper provides a non-asymptotic analysis of gradient flow for a two-layer ReLU convolutional network with shared filters and sum-pooling, trained for binary classification from a sufficiently small, balanced initialization. Its setting builds on the signal–noise patch models studied by Cao et al. and Kou et al.: each input contains one signal patch and additional noise patches. The signal patches lie in two sufficiently narrow, nearly antipodal cones, so that the label-signed signal patches are concentrated around a common axis . The noise patches are assumed to be orthogonal to the signal subspace. Under these assumptions, the paper shows quantitative early alignment for filters that initially activate a signal patch of their corresponding class. When both classes have such filters, it also proves convergence of the training loss at rate (O(1/t)).
The paper presents a clearly explained and technically useful extension of early-alignment theory to convolutional networks. Its conceptual novelty is moderate, but the quantitative analysis is valuable.
Critical strengths
Understanding how gradient-based training learns features in simple neural architectures under structured data assumptions is an important problem in deep learning theory. This paper extends the early-alignment analysis developed for fully connected two-layer ReLU networks to convolutional filters acting on signal and noise patches. Although earlier work has already analyzed two-layer CNNs, the explicit description of filter directions and alignment times provides a useful complementary result.
The manuscript is clearly written and well organized. The setting, assumptions, and main results are stated carefully, and the proof sketch explains the central mechanism effectively. The comparison with prior work also helps identify which parts of the analysis are inherited and which require additional arguments.
The technical contribution is strong. In particular, the analysis bounds the combined contribution of the active noise patches, uniformly over all filter directions and uses a potential function to obtain an explicit alignment-time bound. These arguments address obstacles that prevent a direct application of the earlier fully connected analysis. The main proof arguments appear sound for the globally defined gradient-flow trajectories assumed in the paper.
Critical weaknesses
The qualitative mechanism (early directional alignment followed by fitting) is already familiar from previous work on two-layer ReLU networks. The contribution is therefore primarily a quantitative extension to a structured convolutional setting, rather than a new learning mechanism. I nevertheless consider the extension technically useful.
The scope of the guarantee is limited by exact signal–noise orthogonality and a restrictive cone-width condition ( small). Moreover, the explicit transition-time bound can be very large when . The authors acknowledge these limitations. A discussion of what the theorem explains about convolutional architectures (in particular, what the analysis reveals specifically about weight sharing and sum-pooling) and what remains outside its scope would strengthen the paper. For example, do the results imply that CNNs learn faster than MLPs, or that weight sharing improves sample complexity or leads to a different implicit bias?
The results are conditional on the existence of a global Carathéodory solution to the specified gradient-flow equation. Appendix B explicitly leaves this question open. Since the ReLU update is discontinuous, this is a substantive modeling issue that deserves clarification in the main presentation.
Minor correction:
On p. 28, in the proof of Lemma 17, the intermediate inequality does not follow from for the normalized logistic loss. At margin (-1/2), that loss equals (2\log(1+e^{1/2})>e^{1/2}). The proof can instead use directly. This appears to leave the lemma and all subsequent constants unchanged.
Questions for authors
-
On p. 15, lines 789–790, you state that existence is not addressed. Do the current assumptions ensure global existence for the convention ? If the assumptions do not exclude this, could you identify sufficient additional conditions, or explain whether a differential-inclusion formulation would preserve the results?
-
For the proposed data model, what do weight sharing and patchwise activation contribute beyond the corresponding fully connected analysis? A conceptual comparison with an MLP trained on the concatenated patches would help clarify the architectural significance of the result.
-
Which step of the current analysis relies most critically on the two-layer structure? What would be the main obstacle to extending the alignment argument to a network with an additional trainable layer?
5 comments
Reply to Question 2
Question 2
The CNN and the MLP on concatenated patches differ in two independent choices: whether the filter is shared across positions, and whether the ReLU acts on each patch or on their sum. We discuss them separately and then report an experiment that crosses them.
Weight sharing is where the CNN gains. A shared filter is pulled by every signal patch wherever it sits, so the drive is the full class mass , and Theorem 1 does not depend on the position of the signal patch. For the concatenated MLP the answer depends on the positions. If the signal always sits at the same position and the noise is small, the signed inputs are pairwise positively correlated, Min et al. (2024) apply directly, and our result adds nothing. If the position varies, each weight block is pulled only by the inputs whose signal sits at that position, and the drive toward is only of order .
Patchwise activation is a convenience for the proof and a cost, not an empirical gain. The gate of a signal patch depends on alone, and by orthogonality the derivative of contains no noise term, so one-way switching holds at any noise level. The cost is that each noise patch is gated separately: the noise patches of an input cannot cancel before the ReLU, and an aligned filter keeps activating noise patches of both classes. This is where and our weak bound on come from.
Experiment (new; Appendix A.4, Table 2). We train the four two-layer networks obtained by crossing the two choices, on the same data (, , ) with the signal patch at a random position in each input. The table gives the mean alignment of the hidden units when the loss starts to decrease (noise patch norm ):
| shared filter | one filter per position | |
|---|---|---|
| ReLU per patch | 0.999 (our CNN) | 0.83 |
| ReLU after the sum | 0.998 | 0.946 (concatenated MLP) |
The two networks with a shared filter are indistinguishable. The two without it align about three times more slowly and begin to fit twice as late; with noise patches as large as the signal patches, the gap widens (0.93 and 0.92 against 0.60 and 0.69). When the signal patch is kept at a fixed position, the concatenated MLP begins to fit as early as the CNN. In this data model the architectural gain is therefore weight sharing, and the patchwise ReLU makes no measurable difference.
We added Appendix A.4 to report the comparison of the four architectures and a new paragraph, “Role of the convolutional structure”, to Section 3.1.
Replies to Question 3 and Weakness; References
Question 3
The step that relies most on it is the balancedness in Lemma 3 together with the directional dynamics (8). With two layers, the coefficient that back-propagation attaches to filter is the scalar : it is the same for every input, its sign is fixed, and by balancedness its size is $\lVert w_j\rVert $. The direction of each filter then obeys its own equation on the sphere, with unit speed and independent of the other filters. Three things rest on this: the fixed split into ; the positivity of the drive on the signal coordinates, , which gives one-way switching (Lemmas 12–13); and the control of the noise by a single quantity , since all activated noise patches enter with the same weight.
With an additional trainable layer, with the pooled response of filter , the coefficient becomes . (i) It depends on the input and has no fixed sign, so the drive on a signal coordinate, , can be negative, and one-way switching fails. (ii) The noise enters with input-dependent weights, so one level no longer controls it; this is the same difficulty that limits our bound on , where the loss weights become unequal, now present from the start. (iii) The speed of a filter direction is proportional to the size of the upper layers, of order , so alignment has to be compared with the window in a rescaled time (cf. Kumar & Haupt (2025) for homogeneous networks with locally Lipschitz gradients).
Overall if the added layer is linear, does not depend on , and with a balanced initialization across layers we expect its sign to stay fixed, so that the filters follow the two-layer dynamics in the clock and Part 1 extends. With a ReLU in the added layer, the question becomes whether the upper layers become approximately rank one, giving the filters an approximately fixed sign, before the filters have to align. We regard this as the main open problem; the conclusion now states the obstacle.
Weakness: Exact orthogonality and bound
Exact orthogonality is used in two places, and the answer is different for each.
(1) Early alignment (Theorem 1, Part 1) tolerates a deviation. Here orthogonality is used only to ensure that the noise has no component along or along the signal patches (Lemma 7). We now prove (Proposition 2, Appendix F) that Part 1 still holds for noise that is not exactly orthogonal to the signal subspace, provided the signal-subspace component of the activated noise is at most half the tolerance that the proof already allows for the small-initialization approximation error. In the proof, this component plays exactly the role of that approximation error.
(2) Final convergence (Theorem 1, Part 2) needs exact orthogonality with our proof. Noise–signal orthogonality removes direct noise terms from the signal-projected dynamics, so positive signal correlations keep filters inside the signal cones once they enter (Lemma 15). After the early phase, the loss weights differ across input classes. The signal term acting on a filter involves only the inputs of its own class, whereas a noise component inside the signal subspace involves all inputs, and we have no control on the ratio of loss weights across classes. This is the same obstacle that makes our bound on weak. We now state it as a limitation.
The new Appendix F proves that Part 1 holds without exact orthogonality (Proposition 2) and explains why our proof of Part 2 needs it. Section 2.2 mentions this after Assumption 1, and the conclusion lists the restriction on Part 2 as a limitation.
Minor correction (Lemma 17)
Thank you for this correction. The proof now bounds the logistic loss by directly, as you suggest; the statement of the lemma and all constants are unchanged.
References:
Davis, D., Drusvyatskiy, D., Kakade, S., & Lee, J. D. (2020). Stochastic subgradient method converges on tame functions. Foundations of Computational Mathematics, 20(1), 119–154.
Ji, Z., & Telgarsky, M. (2020). Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems (NeurIPS) 33.
Kumar, A., & Haupt, J. (2025). Early directional convergence in deep homogeneous neural networks for small initializations. Transactions on Machine Learning Research.
Lyu, K., & Li, J. (2020). Gradient descent maximizes the margin of homogeneous neural networks. International Conference on Learning Representations (ICLR).
Min, H., Mallada, E., & Vidal, R. (2024). Early neuron alignment in two-layer ReLU networks with small initialization. International Conference on Learning Representations (ICLR).
Reply to Question 1
We sincerely thank the reviewer for their careful reading of our manuscript and constructive feedback. We particularly appreciate the reviewer's insightful comments on the strength of assumptions and the existence of solutions. It has directly helped us improve the readability of our manuscript and the soundness of both theory and the experimental design. We have uploaded a revised PDF. All changes are marked in blue, and the numbering of the existing equations, lemmas and propositions is unchanged; new material is appended (Appendices A.3, A.4, F and G).
Question 1
With the convention σ′(z)=1{z>0} fixed in the paper (in particular σ′(0)=0), an absolutely continuous solution satisfying (2) almost everywhere does not always exist under Assumptions 1–4. We found an example satisfying all assumptions in which no absolutely continuous trajectory satisfying (2) exists beyond a finite time, which is smaller than t1 (Proposition 5, Appendix G).
Following your advice, we now write (2) as the differential inclusion θ˙∈−∂L(θ) with the Clarke subdifferential, which is the standard notion of gradient flow for nonsmooth losses such as those of ReLU networks (Lyu & Li, 2020; Ji & Telgarsky, 2020). For this formulation, global solutions always exist, and Theorem 1 holds for every solution with the same assumptions and the same constants. The changes to our proofs are local. In the original proofs a Carathéodory solution is used in only two ways: the trajectory is absolutely continuous, so that differential inequalities can be integrated, and equation (2) bounds the time derivatives of the filter norms, the cosine with u⋆, the signal coordinates and the loss. Solutions of the inclusion are also absolutely continuous and satisfy the same bounds, so the same arguments prove Theorem 1.
The solutions themselves are concrete objects. For almost every t the velocity is minus the minimal-norm element of ∂L(θ(t)) (Davis et al., 2020, Lemma 5.2; Ji & Telgarsky, 2020, Lemma B.9); Proposition 4 (Appendix G) gives its explicit form for our network.
Section 2.1 now states the solution concept and the existence result. The new Appendix G contains the definition (Definition 2), global existence and the energy identity (Proposition 3), the minimal-norm property (Proposition 4) and the counterexample (Proposition 5). The proofs in Appendices B, D and E are adapted accordingly; the statement of Theorem 1 is unchanged.