- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how preconditioning changes the features learned by a network, focusing on which input covariance directions are emphasized. It combines a characterization through preconditioned Gram matrices, a feature-selection theorem for a single ReLU neuron, and experiments on noisy labels, distribution shifts, and transfer.
Critical strengths
- I think the paper is clearly written. I appreciate the clear intuition and the effort to make the effect of preconditioning on feature learning explicit.
- The controlled experiments give a nice intuition for why optimizer rankings can change across tasks, depending on whether informative features lie in high- or low-variance directions. Despite the simplified setting, I find this perspective valuable.
- The comparisons at matched training loss help distinguish differences in generalization from differences in optimization progress.
Critical weaknesses
-
The theory is quite limited and in the synthetic experiments the teacher vector is aligned with a single covariance eigenvector. This makes the spectral interpretation particularly clean, but I would like to understand how much of the picture survives when the target depends on a combination of high- and low-variance directions.
-
The connection with practical optimizers needs further justification. The information-theoretic result assumes that the first-layer preconditioner depends only on training inputs, whereas adaptive optimizers generally depend on gradients and the evolving model. Also, interpreting Adam as , or SAM as seems rather approximate. For example, the SAM expansion gives which is not generally equivalent to covariance preconditioning with . Could the authors clarify when these interpretations are justified?
-
The transfer conclusion seems stronger than the evidence. The experiments support (p=-1) when the second task uses the opposite spectral subspace and the first layer is frozen. This is interesting, but does not establish that whitening generally produces the most transferable representations. How does the conclusion change when the tasks share informative directions, or when the first layer is also fine-tuned?
Questions for authors
See above.
5 comments
Response to reviewer ZVvY
W2
Thank you for this comment.
We agree that the theoretical guarantees and the interpretations of practical optimizers require a clear distinction. Theorems 2.1–2.2 require Assumption 1.P and generally do not cover preconditioners that depend on gradients and the evolving model. We have clarified this scope in the main text. Our experiments with actual optimizers examine whether the qualitative spectral trends persist beyond these assumptions.
For Adam, the correspondence to is a conditional scaling interpretation. Adam uses the elementwise factor , where estimates the second moment of the gradient. If the input covariance is approximately diagonal in the optimizer's coordinate system and these estimates satisfy , where is the input variance and is independent of the input direction, then, for negligible ,
\approx a_j^{-1/2}\lambda_i^{-1/2}.$$ Thus, under these approximations, the input-side scaling corresponds to the covariance power $p=-1/2$. Regarding SAM, We agree that the connection between $I+\frac{\rho}{\|\nabla\mathcal{L}\|}H$ and the $p=1$ interpretation required clarification. Let $g=\nabla\mathcal{L}$ and $c=\rho/\|g\|$. The first-order approximation to the SAM gradient is $$g_{\mathrm{SAM}}\approx(I+cH)g.$$ Dividing by the positive scalar $1+c$ preserves the update direction and gives $$\frac{g_{\mathrm{SAM}}}{1+c} \approx \frac{1}{1+c}H^0g+\frac{c}{1+c}H^1g =(1-\alpha)H^0g+\alpha H^1g, \qquad \alpha=\frac{c}{1+c}.$$ **Thus, within this approximation, the rescaled SAM update is a convex combination of the updates associated with the zeroth and first powers of the Hessian, shifting its spectral weighting toward the $p=1$ regime relative to GD $(p=0)$.** Additionally, for fixed $\rho$, a decreasing gradient norm increases $c$ and hence the coefficient $\alpha$ of the $H^1g$ term. **If $\|g\|\to0$, then $\alpha\to1$, suggesting a shift toward the $p=1$ regime.** We have revised the main text and Appendix F to explain these connections above. # W3 Regarding the choice of $p=-1$, we agree with your point. If the spectral components that will be important for downstream tasks are known during pretraining, it would indeed be preferable to choose a value of $p$ that preserves or emphasizes those components. **However, in general, information about downstream tasks is not available during pretraining.** In such cases, $p=-1$ treats all spectral components equally, allowing the learned representations to adapt to a wide range of downstream tasks. We have added a more detailed discussion of this point at the end of Section 3.3 and highlighted the additions in red. Regarding the decision to freeze the first layer, our intention was to experimentally isolate and evaluate how well the representations learned during pretraining can be utilized for a different task while preserving most of the learned features. We therefore froze the first layer. At the scale of our experiments, allowing the first layer to be updated would enable further dynamic feature learning, making it difficult to isolate and measure the influence of the spectral bias induced during pretraining. We therefore considered this setting beyond the scope of the present study.Response to reviewer ZVvY
We hope that the above explanations address your concerns. If any concerns remain, please let us know.
Otherwise, we would greatly appreciate it if you could reconsider your score in light of our responses and revisions. Thank you for your time and consideration.
Response to reviewer ZVvY
Thank you for pointing out this inconsistency. We have clarified that independence applies to the underlying Gaussian randomness, rather than to the initial first-layer weights themselves. Specifically, we initialize
W1
We agree that this question is important for practical applications and believe it connects naturally to our findings. Our conclusion that generalization benefits from alignment between the spectral bias of learned features and the teacher signal suggests that, for mixed teachers, effective preconditioning should reflect the teacher’s relative reliance on high- and low-variance directions.
To investigate this, we have added an experiment in Appendix G.7 in which the teacher depends on a mixture of high- and low-variance directions. We generate inputs as
\beta\sim\mathcal N(0,I_{10}),\qquad \Sigma_X=\operatorname{diag}(10I_5,0.1I_5),$$ and define the teacher by $$y=h^\star\left(\sqrt{1-a}\,\beta_1+\sqrt{a}\,\beta_{10}\right)+\epsilon,$$ where $a\in\{0,0.25,0.5,0.75,1\}$ and independent Gaussian noise $\epsilon$ is added at SNR $=1$. Here, $\beta_1$ and $\beta_{10}$ are standardized coordinates corresponding to high- and low-variance input directions, respectively. Increasing $a$ shifts the teacher’s dependence toward the low-variance direction. The other experimental settings were made consistent with Section 3.1 of the main text. The table reports test MSE on normalized targets as mean ± standard deviation over 10 seeds. The lowest mean in each row is shown in bold. | Low-direction mixture ratio $a$ | $p=0$ | $p=-0.5$ | $p=-1$ | $p=-1.5$ | $p=-2$ | |---:|---:|---:|---:|---:|---:| | 0 | **0.507 ± 0.053** | 0.508 ± 0.056 | 0.525 ± 0.058 | 0.552 ± 0.058 | 0.678 ± 0.095 | | 0.25 | 0.551 ± 0.083 | 0.521 ± 0.079 | **0.519 ± 0.071** | 0.544 ± 0.069 | 0.618 ± 0.090 | | 0.5 | 0.588 ± 0.133 | 0.543 ± 0.107 | **0.529 ± 0.088** | 0.548 ± 0.080 | 0.597 ± 0.093 | | 0.75 | 0.647 ± 0.203 | 0.569 ± 0.133 | **0.545 ± 0.113** | 0.555 ± 0.100 | 0.590 ± 0.098 | | 1 | 0.771 ± 0.230 | 0.635 ± 0.174 | 0.587 ± 0.138 | 0.571 ± 0.133 | **0.568 ± 0.129** | Among the tested exponents, the lowest mean test MSE shifts from $p=0$ for the purely high-variance teacher to $p=-1$ for intermediate mixtures and $p=-2$ for the purely low-variance teacher. These results support the dependence of generalization on teacher–spectrum alignment beyond single-eigenvector targets. In particular, they suggest that whitening-like preconditioning with $p=-1$ can be useful when both spectral bands carry predictive information. These results provide empirical evidence that the dependence on teacher–spectrum alignment persists beyond single-eigenvector targets. In particular, they suggest that whitening-like preconditioning with $p=-1$ can be useful when both spectral bands carry predictive information.