acceptodds
Under review as a conference paper at ICLR 2027

How Does Preconditioning Guide Feature Learning in Deep Neural Networks?

Abstract

Preconditioning is widely used in machine learning to accelerate convergence on the empirical risk, yet its role on the expected risk remains underexplored. In this work, we investigate how preconditioning affects feature learning and generalization performance. We first show that the input information available to the model is conveyed solely through the Gram matrix defined by the preconditioner’s metric, thereby inducing a controllable spectral bias on feature learning. Concretely, instantiating the preconditioner as the -th power of the input covariance matrix, we prove that, for a single ReLU neuron, increasing monotonically shifts the learned feature's sensitivity toward higher-variance eigenspaces, even though all choices of attain the same limiting population loss. We then empirically investigate how this spectral bias interacts with task-relevant features in three settings: robustness to noise, out-of-distribution generalization, and forward knowledge transfer. Our experiments show that learned representations favor the spectral components emphasized by preconditioning, and that generalization improves when this bias aligns with task-relevant features.

Primary area:
optimization
Keywords:
Preconditioning, Optimization, Generalization, Feature Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZVvYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how preconditioning changes the features learned by a network, focusing on which input covariance directions are emphasized. It combines a characterization through preconditioned Gram matrices, a feature-selection theorem for a single ReLU neuron, and experiments on noisy labels, distribution shifts, and transfer.

Critical strengths
  1. I think the paper is clearly written. I appreciate the clear intuition and the effort to make the effect of preconditioning on feature learning explicit.
  2. The controlled experiments give a nice intuition for why optimizer rankings can change across tasks, depending on whether informative features lie in high- or low-variance directions. Despite the simplified setting, I find this perspective valuable.
  3. The comparisons at matched training loss help distinguish differences in generalization from differences in optimization progress.
Critical weaknesses
  1. The theory is quite limited and in the synthetic experiments the teacher vector is aligned with a single covariance eigenvector. This makes the spectral interpretation particularly clean, but I would like to understand how much of the picture survives when the target depends on a combination of high- and low-variance directions.

  2. The connection with practical optimizers needs further justification. The information-theoretic result assumes that the first-layer preconditioner depends only on training inputs, whereas adaptive optimizers generally depend on gradients and the evolving model. Also, interpreting Adam as p=−1/2p=-1/2, or SAM as p=1p=1 seems rather approximate. For example, the SAM expansion gives I+cHI+cH which is not generally equivalent to covariance preconditioning with p=1p=1. Could the authors clarify when these interpretations are justified?

  3. The transfer conclusion seems stronger than the evidence. The experiments support (p=-1) when the second task uses the opposite spectral subspace and the first layer is frozen. This is interesting, but does not establish that whitening generally produces the most transferable representations. How does the conclusion change when the tasks share informative directions, or when the first layer is also fine-tuned?

Questions for authors

See above.

5 comments
AuthorsOpenReview ↗

Response to reviewer ZVvY

Thank you for pointing out this inconsistency. We have clarified that independence applies to the underlying Gaussian randomness, rather than to the initial first-layer weights themselves. Specifically, we initialize

W1

We agree that this question is important for practical applications and believe it connects naturally to our findings. Our conclusion that generalization benefits from alignment between the spectral bias of learned features and the teacher signal suggests that, for mixed teachers, effective preconditioning should reflect the teacher’s relative reliance on high- and low-variance directions.

To investigate this, we have added an experiment in Appendix G.7 in which the teacher depends on a mixture of high- and low-variance directions. We generate inputs as

\beta\sim\mathcal N(0,I_{10}),\qquad \Sigma_X=\operatorname{diag}(10I_5,0.1I_5),$$ and define the teacher by $$y=h^\star\left(\sqrt{1-a}\,\beta_1+\sqrt{a}\,\beta_{10}\right)+\epsilon,$$ where $a\in\{0,0.25,0.5,0.75,1\}$ and independent Gaussian noise $\epsilon$ is added at SNR $=1$. Here, $\beta_1$ and $\beta_{10}$ are standardized coordinates corresponding to high- and low-variance input directions, respectively. Increasing $a$ shifts the teacher’s dependence toward the low-variance direction. The other experimental settings were made consistent with Section 3.1 of the main text. The table reports test MSE on normalized targets as mean ± standard deviation over 10 seeds. The lowest mean in each row is shown in bold. | Low-direction mixture ratio $a$ | $p=0$ | $p=-0.5$ | $p=-1$ | $p=-1.5$ | $p=-2$ | |---:|---:|---:|---:|---:|---:| | 0 | **0.507 ± 0.053** | 0.508 ± 0.056 | 0.525 ± 0.058 | 0.552 ± 0.058 | 0.678 ± 0.095 | | 0.25 | 0.551 ± 0.083 | 0.521 ± 0.079 | **0.519 ± 0.071** | 0.544 ± 0.069 | 0.618 ± 0.090 | | 0.5 | 0.588 ± 0.133 | 0.543 ± 0.107 | **0.529 ± 0.088** | 0.548 ± 0.080 | 0.597 ± 0.093 | | 0.75 | 0.647 ± 0.203 | 0.569 ± 0.133 | **0.545 ± 0.113** | 0.555 ± 0.100 | 0.590 ± 0.098 | | 1 | 0.771 ± 0.230 | 0.635 ± 0.174 | 0.587 ± 0.138 | 0.571 ± 0.133 | **0.568 ± 0.129** | Among the tested exponents, the lowest mean test MSE shifts from $p=0$ for the purely high-variance teacher to $p=-1$ for intermediate mixtures and $p=-2$ for the purely low-variance teacher. These results support the dependence of generalization on teacher–spectrum alignment beyond single-eigenvector targets. In particular, they suggest that whitening-like preconditioning with $p=-1$ can be useful when both spectral bands carry predictive information. These results provide empirical evidence that the dependence on teacher–spectrum alignment persists beyond single-eigenvector targets. In particular, they suggest that whitening-like preconditioning with $p=-1$ can be useful when both spectral bands carry predictive information.
AuthorsOpenReview ↗

Response to reviewer ZVvY

W2

Thank you for this comment.

We agree that the theoretical guarantees and the interpretations of practical optimizers require a clear distinction. Theorems 2.1–2.2 require Assumption 1.P and generally do not cover preconditioners that depend on gradients and the evolving model. We have clarified this scope in the main text. Our experiments with actual optimizers examine whether the qualitative spectral trends persist beyond these assumptions.

For Adam, the correspondence to p=−1/2p=-1/2 is a conditional scaling interpretation. Adam uses the elementwise factor (v^ij+ϵ)−1(\sqrt{\hat v_{ij}}+\epsilon)^{-1}, where v^ij\hat v_{ij} estimates the second moment of the gradient. If the input covariance is approximately diagonal in the optimizer's coordinate system and these estimates satisfy v^ij≈ajλi\hat v_{ij}\approx a_j\lambda_i, where λi\lambda_i is the input variance and aja_j is independent of the input direction, then, for negligible ϵ\epsilon,

\approx a_j^{-1/2}\lambda_i^{-1/2}.$$ Thus, under these approximations, the input-side scaling corresponds to the covariance power $p=-1/2$. Regarding SAM, We agree that the connection between $I+\frac{\rho}{\|\nabla\mathcal{L}\|}H$ and the $p=1$ interpretation required clarification. Let $g=\nabla\mathcal{L}$ and $c=\rho/\|g\|$. The first-order approximation to the SAM gradient is $$g_{\mathrm{SAM}}\approx(I+cH)g.$$ Dividing by the positive scalar $1+c$ preserves the update direction and gives $$\frac{g_{\mathrm{SAM}}}{1+c} \approx \frac{1}{1+c}H^0g+\frac{c}{1+c}H^1g =(1-\alpha)H^0g+\alpha H^1g, \qquad \alpha=\frac{c}{1+c}.$$ **Thus, within this approximation, the rescaled SAM update is a convex combination of the updates associated with the zeroth and first powers of the Hessian, shifting its spectral weighting toward the $p=1$ regime relative to GD $(p=0)$.** Additionally, for fixed $\rho$, a decreasing gradient norm increases $c$ and hence the coefficient $\alpha$ of the $H^1g$ term. **If $\|g\|\to0$, then $\alpha\to1$, suggesting a shift toward the $p=1$ regime.** We have revised the main text and Appendix F to explain these connections above. # W3 Regarding the choice of $p=-1$, we agree with your point. If the spectral components that will be important for downstream tasks are known during pretraining, it would indeed be preferable to choose a value of $p$ that preserves or emphasizes those components. **However, in general, information about downstream tasks is not available during pretraining.** In such cases, $p=-1$ treats all spectral components equally, allowing the learned representations to adapt to a wide range of downstream tasks. We have added a more detailed discussion of this point at the end of Section 3.3 and highlighted the additions in red. Regarding the decision to freeze the first layer, our intention was to experimentally isolate and evaluate how well the representations learned during pretraining can be utilized for a different task while preserving most of the learned features. We therefore froze the first layer. At the scale of our experiments, allowing the first layer to be updated would enable further dynamic feature learning, making it difficult to isolate and measure the influence of the spectral bias induced during pretraining. We therefore considered this setting beyond the scope of the present study.
AuthorsOpenReview ↗

Response to reviewer ZVvY

We hope that the above explanations address your concerns. If any concerns remain, please let us know.

Otherwise, we would greatly appreciate it if you could reconsider your score in light of our responses and revisions. Thank you for your time and consideration.

Reviewer_RjpCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how preconditioning influences feature learning and generalization beyond its effect on optimization speed. It characterizes the input information available to training and prediction through preconditioner-induced Gram matrices, then considers covariance-power preconditioning, P = Σ^p. A single-ReLU population model establishes that increasing p shifts feature sensitivity toward higher-variance eigenspaces despite identical limiting population loss. Experiment shows how this preference interact with informative directions under label noise, invariant and spurious features under distribution shift, and feature reuse in forward transfer.

Critical strengths
  1. The covariance-power framework provides an interpretable connection between the precondition and spectral feature preference, which is clear and very helpful to explain why optimizer effectiveness depends on task structure.

  2. The label-noise, OOD, and transfer experiments offer complementary evidence with repeated runs, matched-loss comparisons, and the additional CNN experiment that strengthen the empirical investigation.

Critical weaknesses
  1. Initialization assumptions (lines 138–140, 682–684). In Theorem 2.1, P(0)P^{(0)} is assumed independent of W1(0)W_1^{(0)}, while Proposition B.1 specifies Gaussian initialization with covariance σ2P(0)\sigma^2P^{(0)} and independence from XtrainX_{\mathrm{train}}. These requirements appear to be inconsistent when P(0)P^{(0)} varies with the training data. Please clarify the initialization of P(0)P^{(0)} and reconcile the independence assumptions.

  2. Test-time characterization (lines 759–791). If I understanded correctly, it seems that the proof of Theorem 2.2 assumes independence between xx and activations computed from xx, while omits the training activations required for the subsequent update. Please specify whether xx is fixed or random, and revise the proof to track the full joint state with appropriate conditioning to see whether your conclusion is still true.

  3. Evidence for the spectral explanation (Sections 3.1 and 3.2.2, Tables 2–3). The experiments show that generalization varies with the optimizer and preconditioning exponent. However, it did not directly show whether these differences arise from changes in reliance on particular spectral components, and Theorem 2.3 demonstrates this mechanism only in a restricted single-neuron setting. I am wondering whether it can be generalized to more general architectures.

Questions for authors
  1. Assumptions and optimizer mapping (lines 993–995). In the part of mapping optimizers to the spectral bias pp, for the SAM, it only shows that the precondition is governed by I+ρ∥∇L(θ)∥HI + \frac{\rho}{\| \nabla \mathcal L(\theta) \|} \mathbf H. I am not quite sure how it can be interpreted as p=1p=1? Can you clarify this?

  2. Theorem 2.2 (lines 759–791). How is independence from xx justified when the state includes activations computed from xx? How is the next gradient determined without retaining the training activations? Please clarify whether the test point is fixed or random and provide a corrected proof tracking the full joint state, including any necessary conditioning on x⊤P(0)xx^\top P^{(0)}x.

  3. Generalization claims (Sections 3.1 and 3.2.2, Tables 2–3). What direct evidence supports the proposed spectral mechanism in the evaluated MLP and CNN architectures? Please also specify the number of OOD runs.

  4. Minor comments

    • Lines 403–404: “based on accuracy on ID validation (Val) accuracy” repeats “accuracy.” Please simplify it to “based on ID validation accuracy.”

    • Lines 907–911: The definition of bi,jb_{i,j} uses an undefined δi,j\delta_{i,j}, whereas the preceding coefficient is ai,ja_{i,j}. Please clarify or correct the notation.

    • Lines 1001–1002: Replace “SAM corresponds the” with “SAM corresponds to,” and qualify the claimed correspondence as discussed above.

4 comments
AuthorsOpenReview ↗

Response to reviewer RjpC

We sincerely thank you for taking the time to review our work and for your insightful feedback. We address your concerns below and hope that our responses resolve them.

AuthorsOpenReview ↗

Response to reviewer RjpC

W1

Thank you for pointing out this inconsistency. We have clarified that independence applies to the underlying Gaussian randomness, rather than to the initial first-layer weights themselves. Specifically, we initialize W1(0)=(P(0))1/2R,W_1^{(0)}=(P^{(0)})^{1/2}R, where RR has i.i.d. N(0,σ2)\mathcal N(0,\sigma^2) entries and is independent of the training data and the remaining initial parameters.

Thus, W1(0)W_1^(0) may depend on the training inputs through P(0)P^(0). Accordingly, Proposition B.1 now explicitly states the Gaussian initialization conditionally on XtrainX_{\text{train}}, with covariance σ2P(0)\sigma^2P^(0). This reconciles the initialization assumptions in Theorem 2.1 and Proposition B.1.

W2&Q2

Thank you for pointing out these issues. We have clarified in Theorem 2.2 that (x,y)∼μ(x,y)\sim\mu is a random test sample independent of the training data and initialization. We do not assume independence between xx and the activations computed from it. We have revised Appendix C to explicitly track (Ztrain(t),z(t),θ2:L(t))(Z_{\mathrm{train}}^{(t)},z^{(t)},\theta_{2:L}^{(t)}), including the training representations required for subsequent updates. We have also added the full-row-rank assumption rank⁡(Xtrain)=dx\operatorname{rank}(X_{\mathrm{train}})=d_x almost surely. This ensures that every test input lies in the span of the training inputs, allowing the initial test representation to be reconstructed as z(0)=Ztrain(0)(GP(0))†cP(0).z^{(0)}=Z_{\mathrm{train}}^{(0)}(G_P^{(0)})^\dagger c_P^{(0)}. Conditioned on the Gram history, cross-Gram history, and training labels, the initial training state (Ztrain(0),θ2:L(0))(Z_{\mathrm{train}}^{(0)},\theta_{2:L}^{(0)}) is independent of XtrainX_{\mathrm{train}}. (Ztrain(t),z(t),θ2:L(t))(Z_{\mathrm{train}}^{(t)},z^{(t)},\theta_{2:L}^{(t)}) is a deterministic function of this initial state and the conditioning variables, so the conditional independence is preserved. This establishes Eq. (6) under the revised assumptions without assuming independence between the test input and its activations.

W3&Q3

Thank you for this suggestion. We have added a spectral-shuffling experiment in Appendix G.4 to directly examine spectral reliance in trained MNIST networks. We compute the eigendecomposition of the training-input covariance and shuffle either high- or low-variance spectral components across ID test samples, keeping the models and labels fixed. The shuffled bands cover target cumulative explained variance ratios of 0%, 10%, 20%, 30%, and 40%.

As shown in the table below, robustness to high-variance shuffling follows AdaHessian > Adam > SAM, whereas robustness to low-variance shuffling follows SAM > Adam > AdaHessian. These opposite rankings indicate stronger relative reliance on high-variance components for SAM and on low-variance components for AdaHessian, with Adam exhibiting intermediate behavior. (We believe that Figure 10 in the manuscript presents these results more clearly.)

Target cumulative explained varianceSAMAdamAdaHessian
0%98.46 ± 0.0999.27 ± 0.2299.95 ± 0.04
10%87.18 ± 0.3088.96 ± 1.5289.46 ± 2.55
20%58.43 ± 4.2361.93 ± 6.4468.80 ± 0.35
30%46.96 ± 1.5953.34 ± 2.7659.13 ± 2.86
40%27.02 ± 0.5435.27 ± 4.4639.74 ± 1.96

・Low-variance shuffling (smallest eigenvalues first)

Target cumulative explained varianceSAMAdamAdaHessian
0%98.46 ± 0.0999.27 ± 0.2299.95 ± 0.04
10%98.18 ± 0.0997.97 ± 0.3993.82 ± 3.86
20%96.06 ± 0.0291.24 ± 1.1886.32 ± 5.01
30%92.70 ± 0.3582.45 ± 1.6376.96 ± 4.32
40%87.20 ± 0.9774.18 ± 1.5568.01 ± 3.75

These interventions directly support our spectral interpretation in a multi-neuron network, extending the empirical evidence beyond the single-neuron setting of Theorem 2.3.

Q1

Thank you for the question. We agree that the connection between I+ρ∥∇L∥HI+\frac{\rho}{\|\nabla\mathcal{L}\|}H and the p=1p=1 interpretation required clarification.

Let g=∇Lg=\nabla\mathcal{L} and c=ρ/∥g∥c=\rho/\|g\|. The first-order approximation to the SAM gradient is gSAM≈(I+cH)g.g_{\mathrm{SAM}}\approx(I+cH)g. Dividing by the positive scalar 1+c1+c preserves the update direction and gives

\approx \frac{1}{1+c}H^0g+\frac{c}{1+c}H^1g =(1-\alpha)H^0g+\alpha H^1g, \qquad \alpha=\frac{c}{1+c}.$$ **Thus, within this approximation, the rescaled SAM update is a convex combination of the updates associated with the zeroth and first powers of the Hessian, shifting its spectral weighting toward the $p=1$ regime relative to GD $(p=0)$.** Additionally, for fixed $\rho$, a decreasing gradient norm increases $c$ and hence the coefficient $\alpha$ of the $H^1g$ term. **If $\|g\|\to0$, then $\alpha\to1$, suggesting a shift toward the $p=1$ regime.** We have revised the main text and Appendix F to explain this connection and replace the previous direct identification of SAM with $p=1$. # Q4 Thank you for your suggestion and have revised the manuscript accordingly.
AuthorsOpenReview ↗

Response to reviewer RjpC

We hope that the above explanations address your concerns. If any concerns remain, please let us know.

Otherwise, we would greatly appreciate it if you could reconsider your score in light of our responses and revisions. Thank you for your time and consideration.

As the discussion period is approaching the deadline, we would like to kindly follow up and ask whether you might be able to share any further comments or considerations regarding our paper.

Thank you again for all your time and effort in reviewing our work.

Reviewer_VRVpOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper focus on studying how preconditioner in optimization affects the feature learning and generalization ability of neural networks. The authors first proved that information-theoretically, the features learned from training relies on the input similarity geometry defined by the preconditioner. Then, the paper studies the special case when P=ΣXpP=\Sigma_X^p and considers a single ReLU neuron with Gaussian inputs. The authors concludes that increasing pp monotonically shifts sensitivity toward higher-variance features, although all choices of pp attain the same limiting population loss. Additionally, the paper empirically studies the effect of alignment between the teacher and the input spectrum and power pp on the generalization from three different points of view: robustness to noise, out-of-distribution (OOD) generalization, and forward knowledge transfer.

Critical strengths

None.

Critical weaknesses

Theoretical analysis in the paper is hard to follow and has a gap with practice and the interpretation. Theorem 2.2 does not give precise assumptions on the data distribution (x,y)∼μ(x,y)\sim\mu. It is possible that the input space has finite samples, then we cannot conclude the independence result as (6). Assumption 1.P P(t)P^{(t)} only depends on XtrainX_{\text{train}} and does not depend on labels, model parameters, gradient history, or initialization, which is not consistent with practice and the empirical results. The empirical Hessian-based preconditioners depend is not satisfying Assumption 1.P.

Questions for authors
  1. What are the g2:Lg_{2:L} and θ2:L\theta_{2:L}? Do we have nonlinear activation functions like MLP here?

  2. Is there any assumption of the per-sample loss ℓ\ell?

  3. Implication 2 claims that feature learning and generalization are determined solely by Gram matrix induced by the p–dependent similarity geometry together with the training labels. But what is the feature learning here? I do not understand how this conclusion is obtained from section 2.2. You only compute the preconditioned Gram matrix in this special case but there is not analysis of feature learning or generalization error.

3 comments
AuthorsOpenReview ↗

Response to reviewer VRVp

We sincerely thank you for taking the time to review our work and for your insightful feedback. We address your concerns below and hope that our responses resolve them.

Regarding the missing assumptions on the data distribution

Thank you for pointing out that Theorem 2.2 did not specify sufficient assumptions on the input distribution. We agree that a finite-support distribution may yield a rank-deficient training matrix, in which case the independence claim in Eq. (6) is not guaranteed by the original argument. To address this, we have added the full-row-rank assumption rank⁡(Xtrain)=dx\operatorname{rank}(X_{\mathrm{train}})=d_x almost surely to Theorem 2.2 and revised the proof in Appendix C accordingly. This condition allows the initial test representation to be reconstructed from the initial training representation using the training Gram matrix and the cross-Gram vector. Together with the other stated assumptions and the update equations, this establishes the conditional independence in Eq. (6). Finite-support distributions that violate the rank condition are outside the theorem’s scope.

Regarding the limitation of Assumption 1.P

Thank you for pointing this out. We agree that general adaptive Hessian preconditioners are not covered by Theorems 2.1–2.2. Assumption 1.P is used to establish the conditional-independence guarantees in these theorems. Nevertheless, within the input-side preconditioned update form considered in our analysis, the representation-update identities in Eqs. (3) and (5) remain valid without Assumption 1.P. Specifically, given the current state and loss gradients, the first-layer updates are expressed through the Gram matrix and cross-Gram vector, even when the preconditioner is state-dependent. This preserves an algebraic connection between representation learning and input geometry, although it does not establish the conditional-independence guarantees of Theorems 2.1–2.2. Our experiments with actual optimizers provide empirical evidence for the qualitative spectral trends beyond these theoretical guarantees.

We have clarified in the main text that Theorems 2.1–2.2 require Assumption 1.P, which generally excludes state-dependent Hessian preconditioners. We have also clarified that the experiments with actual optimizers test whether the qualitative trends observed with covariance-based preconditioning persist beyond this assumption.

Regarding Q1

As shown in the setup of Section 2 of the manuscript, g2:Lg_{2:L} denotes the remaining network after the first linear transformation, including any nonlinear activation functions, and θ2:L\theta_{2:L} denotes its trainable parameters. Our framework therefore includes nonlinear networks such as MLPs. We have clarified these definitions in the Setup section.

Regarding Q2

Theorems 2.1 and 2.2 do not require a specific form of the per-sample loss, such as squared error or cross-entropy, nor do they require convexity. We only require that the loss and the corresponding gradients used in the updates are well-defined and measurable. We have explicitly clarified these regularity assumptions in the revised manuscript.

Regarding Q3

Thank you for the question. By feature learning, we mean the acquisition of internal representations through W1W_1 and θ2:L\theta_{2:L}. Under their assumptions and for a fixed model and training procedure, Theorems 2.1–2.2 show that the joint distribution of the first-layer training representations ZtrainZ_{\text{train}} and downstream parameters θ2:L\theta_{2:L} depends on the training inputs only through the Gram and training labels. The first-layer test representation zz and θ2:L\theta_{2:L} additionally depends on the cross-Gram. Since the model output f(x)=g2:L(z(x);θ2:L)f(x)=g_{2:L}(z(x);\theta_{2:L}) is a deterministic function of these quantities, its predictive distribution inherits the same dependence. Consequently, for a fixed test distribution and loss, the generalization error averaged over initialization also depends on the input data only through the Gram and cross-Gram and the training labels. This is the conclusion summarized in Implication 1.

Implication 2 specializes this argument to P=ΣXpP=\Sigma^p_X. For fixed input data, each spectral component of the Gram matrix and cross-Gram vector is weighted by sj2(p+1)s_j^{2(p+1)}, so the scalar pp controls the relative weighting of high- and low-variance directions. Combined with Implication 1, this shows that learned representations and test predictions, and hence the expected generalization error for a fixed test distribution and loss, depend on the inputs only through this pp-dependent similarity geometry and the training labels.

We have revised Implications 1 and 2 to clarify this reasoning, with the changes highlighted in red.

AuthorsOpenReview ↗

Response to reviewer VRVp

We hope that the above explanations address your concerns. If any concerns remain, please let us know.

Otherwise, we would greatly appreciate it if you could reconsider your score in light of our responses and revisions. Thank you for your time and consideration.

As the discussion period is approaching the deadline, we would like to kindly follow up and ask whether you might be able to share any further comments or considerations regarding our paper.

Thank you again for all your time and effort in reviewing our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

Dear reviewers,

Thank you all for your insightful and constructive feedback on our paper.

We believe that we have addressed all of your concerns in our detailed responses below. We would greatly appreciate it if you could revisit our paper in light of these clarifications and revisions and consider updating your scores accordingly. As the discussion period is approaching its deadline, we would also be grateful if you could share any comments or questions to facilitate continued discussion.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.