acceptodds
Under review as a conference paper at ICLR 2027

Population Onset and Finite-Batch Resolution in One-Step Feature Learning

Abstract

A population gradient can point toward a task-relevant direction even when a finite empirical gradient cannot resolve it. We study this gap for one-step feature learning in two-layer networks with spiked Gaussian inputs. We identify two target-weighted covariance quantities with distinct roles. The population amplification sets the learning-rate scale at which task-directed overlap becomes order one, whereas the ratio gives a sufficient independent-batch scale for recovering that movement empirically. These scales can vary independently, so anisotropy can amplify feature movement without making it easier to estimate from finite data. Under a narrower smooth activation class and explicit triangular conditions, we derive at the critical batch scale a joint random law separating coherent task alignment from total representation movement. For rank-one spikes, an exact alignment–anisotropy map characterizes where population amplification, finite-batch difficulty, and the critical law arise; the population and critical results also extend to fixed-rank spikes. Direct-onset, factorial, and geometry-wide experiments support the predicted normalizations and expose finite-dimensional deviations. The results concern one-step representation dynamics, not downstream risk, and the recovery scale is sufficient rather than necessary.

Primary area:
learning theory
Keywords:
High-dimensional learning, one-step feature learning, spiked covariance, task alignment, representation learning, finite-batch resolution, feature-gradient recovery, random features

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UsbmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies one gradient step in a two-layer network with Gaussian inputs and a spiked covariance. It identifies two quantities with different roles: A controls the learning-rate scale for an order-one change in task-related overlap, while H/(A^2) gives a sufficient batch-size scale for reproducing the population overlap. At the critical batch scale, the authors derive a joint fluctuation law for coherent alignment and squared overlap. Synthetic experiments test these predictions.

Critical strengths
  1. I think separating the strength of the expected update from the batch size needed to estimate it is a useful question. The experiments that hold one geometric quantity approximately fixed while varying the other help make this distinction pretty clear.

  2. The joint critical law seems to be an interesting part of the analysis. It distinguishes coherent alignment from squared overlap, which can also increase because of noise. I also appreciate that the authors report finite-dimensional deviations and do not claim that these results establish better downstream prediction.

Critical weaknesses
  1. I found the introduction difficult to follow. Terms such as “population onset,” “resolution,” and “coherent alignment” appear before I had a clear understanding of the experiment or what counts as learning. I would have preferred the motivation, model, and observables to come first. In particular, it took me some time to understand that “resolution” concerns fluctuations around an unbiased gradient estimate.

  2. I'm not yet sure how much recovering the chosen projection (i.e. q_i) tells us about feature learning. The observables measure projection onto v, but do not control movement orthogonal to v or normalized directional alignment. A batch could reproduce the population projection while producing a large update elsewhere. The authors acknowledge that this is not a prediction-risk result, which is fair, but I would like a clearer explanation of why this observable captures an important learning phenomenon.

Questions for authors
  1. Could the authors clarify the relationship between recovering Q_a or Q_2 and recovering a useful feature direction? In particular, can their recovery condition hold while the empirical update has poor directional alignment because of orthogonal noise?
2 comments

We thank the reviewer for the careful assessment and for isolating the distinction between projected recovery and full-gradient directional recovery.

Yes. The new experiment shows that projected task recovery and full-gradient directional recovery are genuinely different notions. This is consistent with the theorem's intended scope: Theorem 4.1 controls the population-selected task projection, not the angle of the complete high-dimensional update.

Direct full-matrix evidence

Coherent projected recovery is already near its population value while most empirical-update energy remains orthogonal. Near (χFL≈0.05)(\chi_{\rm FL}\approx0.05), Figure 3(a) gives projected-recovery ratios of 0.994−1.0230.994-1.023 with every bootstrap interval containing one. Yet at �0��0�, the clean and intermediate geometries have

\qquad R_\parallel=0.124,\ 0.119.$$ **Most of the empirical update energy—about $88\%$ in these $n=800$ cells—remains outside the task-selected direction.** Equivalently, the representative full-update cosine is about $0.35$, while the task-directed energy fraction is about $0.12$. Here $�5�$ compares the complete empirical and population update matrices, and $�6�$ is the exact rowwise energy fraction in $�7�$. Appendix M.1 verifies $�8�$ to numerical precision. Figure 3(a) uses 500 independent initializations per realized cell, and the full-matrix panels use 100. The comparison uses matched $\text{tanh}, �9�$, three predeclared geometries, bootstrap 95% intervals, and the complete population update including self-prediction. > Why the projection onto $v=\frac{\Sigma\xi}{\sqrt B}$ is meaningful? **The projection is not arbitrary:** $v$ is selected analytically by the population label gradient. The theorem therefore asks whether the empirical batch resolves the task-relevant component selected by the population dynamics themselves. The leading row update inherits the auxiliary sign\$$a_i$\$. Weighting the row overlap by $�12�$ removes that sign, so $�13�$ measures coherent task-directed movement across rows. By contrast, $�14�$ measures total projected energy and may include incoherent movement. Recovery of $�15�$ is therefore more specific than an increase in projected energy alone. > Scope and presentation The Introduction now distinguishes population onset, finite-batch resolution, and coherent alignment before presenting the results. Sections 2.4--2.5 define the population-selected direction and observables, while Section 5.3 and Figure 3 expose the projected/full-gradient distinction directly. The revised paper therefore makes **projected statistical recovery and full-gradient directional recovery explicitly distinct notions**. **Revision:** Introduction; Sections 2.4--2.5 and 5.3; Figure 3; Appendix M.1; Discussion. --- We hope this clarification of the projected scope, together with the full-matrix evidence, addresses the reviewer's concern about orthogonal noise and directional recovery.

Thank you again for the thoughtful review. We have added the full-matrix experiment directly addressing the projected-versus-full-gradient recovery question and clarified the role of the task-selected projection. We value the discussion during the rebuttal period and would be very happy to hear any remaining concerns or questions.

Reviewer_14vWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies the one-step gradient of the first layer of a student in a teacher-student scenario For a known teacher direction and a 2-layer student model with frozen readout weights and data drawn from a spiked Gaussian (multiple orthogonal spikes are allowed). they expand on previous works by having anisotropic data, and by showing the onset learning rate (controlled by A) can change separately from the resolution batch size (unlike in the unspiked scenario)

Critical strengths

The paper builds on the existing models for one-step learning on spiked Gaussian data for two-layer models and shows that the critical batch size is controlled by the signal-to-noise ratio of the feature gradient along the covariance-weighted task direction. It also shows that the onset learning rate is controlled by how strongly the input covariance amplifies the task direction in general and, in particular, in the unaligned case (see critical weaknesses).

Critical weaknesses

The paper “Learning in the Presence of Low-dimensional Structure: A Spiked Random Matrix Perspective” (Ba, Erdogdu, Suzuki, Wang, and Wu; NeurIPS 2023) deals with a spiked model for one-step updates of a two-layer network with anisotropic data, and they identify the same A term (Section 4.2 and Appendix B.1 Eq. B.3). You do cite a previous paper of theirs (from 2022), but not this one.

Questions for authors

Figure 3.c shows that for β=0.6,δ=0.75\beta=0.6,\delta=0.75 the variance does not agree even though n was doubled, does increasing n further produce a trend that goes towards 1? additionally could you plot the variance ratio against the error C/(sA2)C/ (sA^2) and mFL/deffm_{FL} / d_{eff} if the difference does not shrink with these two ratios, that would make me think the reason is not finite size, but a possible theory gap

In Figure 7, both QaQ_a and Q2Q_2 sit above 1 for every k/nk/n ratio, but since the widths share randomness, is this offset a single high draw or a true bias not captured by the theory? Could you check with independent seeds per width, and in other cells and values of nn?

2 comments

We thank the reviewer for the careful assessment and for pointing us to the relevant Ba et al. (2023) comparison and the finite-size questions.

Relation to Ba et al. (2023)

Ba et al. (2023) should indeed have been cited, and we have added this comparison. Importantly, the exact specialization sharpens the distinction between their fully aligned setting and the phenomenon studied here.

In the fully aligned specialization,

\quad\Longrightarrow\quad B=A=1+\theta,\qquad H=(1+\theta)^2,\qquad \frac{H}{A^2}=1.$$ **Thus, Ba et al. contain the aligned specialization of the covariance-amplification factor, while the two-scale separation studied here is absent in that specialization.** Their setting cannot separate population onset from finite-batch resolution because the second coordinate is fixed at one. Our contribution is the partial-alignment regime in which these quantities decouple, together with the alignment-dependent regime map and critical fluctuation law. We do not compare raw learning-rate exponents because the network and normalization conventions differ. > Larger-$�0�$ and theorem-margin diagnostics **The larger-$�1�$ study shows that the difficult higher-$�2�$ geometry $�3�$, corresponding to the variance discrepancy highlighted in the review, does not exhibit a simple monotone finite-dimensional convergence trend.** We extended this geometry through $�4�$; its variance ratios across increasing dimensions are \$ 1.750,\quad 1.715,\quad 1.484,\quad 1.340,\quad 1.794. \$ We therefore do not infer convergence to one from this finite-dimensional sequence. More generally, across 26 predeclared geometry--dimension cells, with 200 independent trials per cell, the variance discrepancies become larger as the finite-dimensional theorem margins deteriorate. Before inspecting the outcomes, we fixed a proxy combining effective-dimension load, width load, projected noise, and primitive self-prediction ratios. **Across the predeclared grid, finite-dimensional variance error is systematically larger when the proof margins are poorer.** Absolute variance-ratio error has descriptive Spearman correlations \$ $0.739, 0.767, 0.765$ \$ with $�6�, �7�$, and the combined proxy, respectively. We do not infer a convergence rate from these descriptive correlations. Figure 10 displays all three prespecified comparisons rather than selecting a favorable subset. > Width and shared-randomness control **The independent-seed rerun, which also includes additional geometries and a second dimension, rules out shared randomness as the sole explanation for the earlier offset.** At the same time, no uniform width trend appears and the difficult $�8�$ offset does not uniformly decay with dimension. We therefore interpret the residual as geometry-dependent finite-size behavior rather than a systematic width effect. Figure 2 also isolates the two coordinates using exact finite matrices: at matched $�9�, �10�$ differs by a factor of three; at matched $�11�, �12�$ differs by a factor of five. These are coordinate-isolation diagnostics rather than critical-law realizations. The added diagnostics sharpen the distinction from the aligned prior setting while making the finite-dimensional limitations explicit. **Revision:** Introduction; Related Work; Sections 2.4, 5.2, and 5.4; Figures 2 and 10; Appendices L.4, M.2, and M.4. --- We hope these clarifications and additional diagnostics address the literature-positioning and finite-size questions raised in the review.

Thank you again for the detailed review. We have incorporated the Ba et al. (2023) comparison and added the requested finite-size and width diagnostics. We value the discussion during the rebuttal period and would be very happy to hear any remaining concerns or questions about these revisions.

Reviewer_eCghOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
3
Read review
Paper summary

The paper analyzes one-step feature learning and how the gradient overlaps with a covariance-weighted task direction and whether this overlap can be recovered with finite batch sizes. The paper provides results on when the learning rate and batch size result in significant overlap and recovery and analyzes how alignment between the task direction and a covariance spike can impact these results.

Critical strengths
  • The paper presents novel theorems about feature learning and how covariance and the task direction affect learning.
Critical weaknesses
  • The main weakness is that the paper is difficult to read with notation used without being defined such as F1popF_1^{pop} or kk and mm and generally references to undefined ideas such as "the independently fitted ridge readouts used in the experiments" being mentioned in the definitions section. The main results also have relatively little explanations of the implications with Theorem 3.2 simply demonstrating the overlap and onset learning rate scale and no further explanation.

  • The overlap with the covariance-weighted task direction is an indirect measure of both how well the gradient picks up the task direction and of how the loss is impacted. Furthermore, it is focused only on the first layer and the first step, providing limited results and intuition about training in practice. While this could still be informative, implications are not further explored in the experiments or discussion.

Questions for authors
  • Would you be able to provide some explanation for how the results could inform training or further theoretical developments?
7 comments

We thank the reviewer for the careful assessment and for highlighting the importance of making the geometric interpretation and practical implications more explicit.

The paper's central distinction is that anisotropic geometry controls population movement and finite-batch resolvability through two different target-weighted quantities. The revision now makes this structure explicit before the theorem statements and connects it directly to geometry-aware one-step calibration.

Geometric interpretation

Section 2.4 expresses the geometry through the task-weighted spectral measure �0�:�0�:

\qquad \frac{H}{A^2} =1+\frac{\operatorname{Var}_p(\lambda)}{\mathbb E_p[\lambda]^2}.$$ This gives the two scales distinct geometric meanings. **The quantity $�1�$ is task-weighted mean covariance amplification, while $�2�$ is normalized task-weighted spectral heterogeneity and the inverse projected-SNR scale.** The direction $�3�$ is selected analytically by the population task gradient, rather than chosen after observing the experiment. Because the leading row update inherits the auxiliary $\text{sign} �4�, �5�$ removes that sign and measures coherent task-directed movement across rows. In contrast, $�6�$ measures total projected overlap energy, including incoherent movement; neither observable is prediction risk. Theorem 3.2 identifies the first learning-rate order, $�7�$, at which population movement in this task-selected direction becomes order one. > Training and theoretical implications The normalized coordinates are \$c=\frac{\eta A}{\sqrt{k}}, \chi_{\rm FL} =\frac{H}{m_{\rm FL}A^2}.\$ **These coordinates substantially reduce cross-geometry variability in one-step behavior.** Quantitatively, the reduction is roughly a factor of four to six across update energy, coherent alignment, and total projected energy. At $�8�$, the median coefficients of variation fall to $23.5\%$, $16.0\%$, and $20.6\%$ of their raw-coordinate values for update energy, $�12�$, and $�13�$, respectively. This provides a **geometry-aware normalization for one-step calibration**; we do not claim optimizer optimality. The separation also identifies concrete directions for multi-step dynamics, preconditioning, and determining when coherent representation movement improves prediction. > Presentation The revised setup defines the task geometry and observables before using them. Section 2.2 distinguishes the fixed auxiliary coefficients from the independently fitted experimental ridge readout. Figure 3 makes concrete that projected recovery and full-update alignment are distinct objects. The revision thus makes both the geometric interpretation and the operational one-step consequence of the two-scale theory explicit. **Revision:** Sections 2.2, 2.4--2.5, and 5.3; Theorem 3.2; Figure 3; Discussion; Appendix M.3. --- We hope these clarifications make the geometric meaning, scope, and practical one-step implications of the results substantially clearer.

Thank you for the response.

The main concerns remain as the provided clarifications are primarily a restatement of results and do not further the discussion around connections and implications for training in practice or a more general understanding of training dynamics. It remains unclear how the objects being studied relate to the underlying function being learned or what it tells us about the relationships in the data.

The paper still has many instances where notation or terminology is used without definition and more generally, the paper continues to use confusing or imprecise phrasing, resulting in the paper being very difficult to read.

As a result, I will keep my score.

Thank you for the follow-up. We respectfully disagree with two aspects of the assessment and would appreciate clarification on the specific instances the reviewer has in mind.

1. Definition and use of notation

Regarding the statement that the paper still contains “many instances where notation or terminology is used without definition,” we have carefully rechecked the revised manuscript and do not believe this characterization is accurate.

The principal quantities are introduced explicitly before use:

  • B,S,yB,S,y in the data/teacher model;
  • v,A,H,Cv,A,H,C in Section 2.4;
  • qi,Q2,Qaq_i,Q_2,Q_a in Section 2.5.

The associated terminology is also introduced together with these definitions: population amplification, finite-batch resolution, coherent task-directed movement, and total projected overlap energy are explained where the corresponding quantities first appear. We also reorganized the setup so that the geometry and observables precede the theorem statements that use them.

If the reviewer can point to specific remaining instances of undefined notation, terminology, or imprecise phrasing, we would be very happy to correct them. Without concrete examples, it is difficult for us to determine what aspects of the revised presentation remain unclear.

2. Connection to the underlying function and data geometry

We would also like to clarify the connection to the underlying function and to the statistical relationships in the data. This connection is not merely interpretive; it follows directly from the population-gradient calculation.

The teacher is y=σ⋆(S)y=\sigma^\star(S) with S=ξ⊤x/ξ⊤ΣξS=\xi^\top x/\sqrt{\xi^\top\Sigma\xi}, so the target function depends only on the one-dimensional task coordinate SS. The direction v=E[xS]=Σξ/ξ⊤Σξv=\mathbb{E}[xS]=\Sigma\xi/\sqrt{\xi^\top\Sigma\xi} is therefore not an arbitrary projection: it is the covariance-weighted task direction selected by the population label gradient. Since SS is the only input coordinate on which the target depends, movement along this direction measures how the first-layer representation initially adapts to task-relevant input variation.

This connection becomes explicit in the exact projected-gradient identity Li=A E[σ⋆′(S)σ′(Ui)]+ri E[σ⋆(S)σ′′(Ui)]L_i=A\,\mathbb{E}[\sigma^{\star\prime}(S)\sigma'(U_i)] + r_i\,\mathbb{E}[\sigma^\star(S)\sigma''(U_i)], where A=ξ⊤Σ2ξ/(ξ⊤Σξ)A=\xi^\top\Sigma^2\xi/(\xi^\top\Sigma\xi).

Thus, AA is not an auxiliary geometric statistic: it is the coefficient multiplying the leading first-channel teacher-dependent term in the projected population gradient. It quantifies how the covariance structure amplifies the task-driven component of the representation update.

The second quantity, H/A2H/A^2, governs how difficult that same teacher-relevant component is to resolve from a finite empirical batch. In spectral form, A=Ep[λ]A=\mathbb{E}_p[\lambda] and H/A2=1+Var⁡p(λ)/Ep[λ]2H/A^2=1+\operatorname{Var}_p(\lambda)/\mathbb{E}_p[\lambda]^2, where pj∝(uj⊤ξ)2λjp_j\propto(u_j^\top\xi)^2\lambda_j.

These weights depend jointly on the task direction and covariance eigenspectrum. Hence the theory identifies a concrete task--data interaction: the same anisotropic covariance can amplify the population movement induced by the teacher while simultaneously making that movement statistically harder to estimate, depending on task--covariance alignment. Under partial alignment these two effects decouple; in the fully aligned specialization they collapse.

The central contribution is therefore not a generic one-step statistic, but the identification of two distinct task-dependent geometric scales governing the creation and finite-sample estimation of the same teacher-relevant representation component.

3. Scope of the claim

We agree that the paper does not prove a multi-step downstream prediction theorem. That is a limitation of scope, but it is different from saying that the quantities studied are disconnected from the function being learned.

Our result characterizes how the teacher function and anisotropic data geometry jointly determine:

  1. the first task-directed representation movement, and
  2. the finite-sample difficulty of estimating that movement.

The first is obtained directly from the teacher-dependent population gradient; the second quantifies the statistical resolvability of that same component. What remains open is how this initial task-directed representation change propagates through subsequent optimization into downstream prediction.

We therefore believe the revised paper provides a concrete statement about the interaction between the function being learned and the statistical structure of the data, while clearly delimiting the multi-step and prediction-level questions that are beyond its present scope.

We would appreciate it if the reviewer could identify the specific remaining instances of undefined notation or imprecise terminology so that we can address any concrete presentation issues directly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.