- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies online spherical SGD with correlation loss on the Gaussian target . Here is the cyclic shift by coordinates, with , and is uniform on the sphere. Three procedures are compared, each with a prescribed step size (Eq. 6): a tied student (T), an untied student of width (U), and the same untied student trained on the full-group-averaged loss (A).
- (Thm 3.2). For polynomial links with information exponent , weak directional recovery takes samples for T and A and for U. With , U and A cover the whole orbit at the same scales (Cor. 3.5).
- (Thm 3.7). Weak subspace recovery takes samples for T and A, against for U.
The upper bounds rest on a two-phase argument: the largest initial correlation is selected, then grows. The lower bounds are non-escape estimates (Thm B.4). One experiment (, , ) illustrates the scalings.
Critical strengths
- For each procedure, at the prescribed step sizes, upper and lower bounds match up to polylogarithmic factors. The lower bound holds uniformly over polynomially many weights.
- The analysis handles a number of interacting teacher directions that grows with . Interaction terms are controlled through the near-orthogonality (Lemma C.1) and an initial relative gap of order (Lemma C.3). This makes rigorous the mechanism by which the initial leader remains the leader, and extends greedy-selection arguments (Ben Arous et al., 2026) to growing rank.
Critical weaknesses
- The separation is between three algorithms at prescribed step sizes. Thm B.4 is proved for with the rescaling (9), and App. A (l. 873–875) states that the lower bounds hold "for the specified spherical SGD updates and learning rates". Neither the abstract ("a polynomial sample complexity separation between symmetry-aware and symmetry-agnostic feature learning") nor Thm 3.2 mentions this restriction. The lower bound for U does not exclude a shorter hitting time under another step size or schedule. Moreover, the rate is itself specific to vanilla online SGD: loss smoothing (Damian et al., 2023) and batch reuse (Dandi et al., 2024b) change it, and the paper does not discuss whether the factor persists under them.
- Interpretation of the factor . Per fresh sample, an augmented weight takes the average of the untied single-weight gradients at the transformed copies , with a step times larger than in U (Eqs. 65–66). This is U on a minibatch of the copies, with the step scaled linearly in the batch size. Since the orbit is nearly orthogonal, these copies act on the correlations almost as independent samples. The factor then equals the number of processed examples per fresh sample, and also the per-iteration overhead of A (l. 228–229): counted in gradient evaluations, A and U both cost . This reading determines what "symmetry-aware" buys and should be discussed.
- The setting is narrow. The group enters the proofs only through the near-orthogonality of a random orbit (Lemma C.1) and through the tied identity. The condition excludes (stride-one convolution) and local filters with overlapping patches, which are the cases behind the convolutional interpretation. The correlation loss decouples the student neurons (l. 441–444), so U and A are independent single-neuron runs on shared data. The second layer is not trained.
Questions for authors
- Does the lower bound of Thm B.4 for U extend to all step sizes, constant or time-varying? If not, can Fig. 1 be complemented by a step-size sweep for each procedure?
- Under the square loss, Eq. (66) no longer holds. Do A and T still have the same sample complexity, and does the factor persist?
- For (the conjecture in Sec. 5), which step fails: the initialization gap (Lemma C.3) or the control of the interaction terms? Is the agreement of Fig. 1 at evidence for the conjecture?