acceptodds
Under review as a conference paper at ICLR 2027

Correlations Are Ruining Your Gradient Descent

Abstract

Natural gradient descent, and the preconditioned optimizers which have followed from it, correct gradient descent by estimating curvature and applying its inverse to the gradient. For any single layer of a deep neural network, this correction can be shown to approximately separate into two factors: one which accounts for correlations in the gradient signal, and one which accounts for correlations in the data arriving at that layer. Herein we show that this second factor can be expressed as a change of basis rather than of curvature and, considered thus, it can be removed from the optimizer entirely and placed within the network itself. Specifically, we show that, once forward weights are corrected, right-sided natural gradient descent can be implemented in a form which has three exactly equivalent forms: a gradient preconditioner, a decorrelating linear transform in the forward pass, or a set of recurrent dynamics in which all updates are Hebbian and rely only upon locally available signals. The (recurrent) decorrelation method we discover aligns closely with proposed methods for inhibitory learning and homeostatic plasticity in the brain. Notably, our transform requires no matrix inverse, no matrix decomposition, and no information beyond that which is available at an individual synapse. It can therefore be applied in distributed and physically constrained systems, including neuromorphic hardware and nervous systems, where existing preconditioners cannot. We demonstrate its effectiveness empirically in convolutional and residual network models while significantly improving a range of approximate biologically plausible methods for credit assignment.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
natural gradient descent, decorrelation, biologically plausible learning, preconditioning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rcG6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The article proposes to use the replacement of gradient descent in the standard backprop algorithm, with a version of natural gradient algorithm. In this version, the decorrelation of the input vector emerges naturally, therefore, the authors propose the use of decorrelating network architectures as the solution to improve convergence behavior. They demonstrate the impact of this approach on CIFAR10 and CIFAR100 datasets, for different networks (e.g. LeNet) and different algorithms (BP, Feedback Alginment, sign concordance, error predictive coding) . The authors' experiments show notable performance improvement when decorrelation is introduced.

Critical strengths
  • Idea of using natural gradient and observing its decomposition in terms of the input correlations is noteworthy.

  • The proposed whitening methods, preserving output an having recurrent structure are new to my knowledge.

  • The performance improvements relative to not using whitening and relative to the other preconditioning methods are remarkable (especially for FA.)

Critical weaknesses
  • Preconditioning/whitening idea for NN training were already introduced. The main contribution of this article is somewhat restricted to alternative and successful implementation of the whitening (especially relative to the PRONG method in the cited article (Desjardins et al. (2015)).

  • The presentation requires improvement. See my notes below.

SECTION 1

This section is missing disscussion about basic references on backprop alternatives including originating articles on FA,DFA and Node Perturbation.

Section 1.1 is also missing discussion of Newton which introduces loss curvature based alignment for the gradient and its potential relation with/deviation from the natural gradient approach. The authors could also provide meaning and motivation for the natural gradient, especially how the use of metric change can help faster/robust convergence to global optima. It is not clear from the last statement whether the authors perform mean centering in the algorithm or not.

Section 1.3: Neither decorrelation nor whitening is properly defined here. They are not defined earlier also. Section 1.1 has the inverse of the data correlation matrix, but this is not whitening, the multiplication with an inverse square root of the covariance is. I think the authors should link this section to Equation (2) to clarify what is intended.

CONTRIBUTIONS

  1. the authors call the correlation inverse is decorrelating, but multiplying with the inverse of correlation inverse is not decorrelation process. (but with a square root of the covariance is)

SECTION 2

Section 2.1: The title of this subsection is not clear to me at the first read (what is the considered basis and what is the assumed inner product, what does it mean to place parameters in a basis.) The last sentence of the first paragraph is also unclear too.

Section 2.2: Natural gradient based derivation in Appendix D is not clear. Which Riemannian metric is used? It is also a bit confusing. M_i is essentially a square root of covariance inverse matrix that we are trying to learn, however, its update requires the inverse of the covariance, and the natural gradient approach nicely eliminates this requirement?

Section 2.3 The equivalence in unnumbered equation (above (4)) is for xi−1x_{i-1} input only. We can not guarantee that the system preserved equivalence for other inputs right?

Appendix E This appendix is about the derivation of the M update equation, which I found confusing. In this there is a sequence of equations. it is not clear if they are meant to imply each other. For example, the first equality does not imply second equality.

Section 2.4 Given that the result is offered in Theorem form, it better be followed by a formal proof. Furthermore, a discussion before Theorem about why we need such alternative forms or their significance would provide some clarity.

Regarding, R_i and M_i, they better be clearly defined as square root and inverse square root of covariance(correlation matrix) for inputs. Using R_i notation might give the wrong impression that it is a correlation (covariance) matrix for x_i's.

Section 3

I don't think the acronym CRGD for the proposed method is provided in the article. That makes the discussion in Section 3 a bit confusing. It is not clear, which of the three whitening operations of Theorem is applied.

Questions for authors
  • Could you clarify decorrelation vs whitening within the context of the article, clarifying the existence of mean centering (especially considering the layer activations with ReLU nonlineariities.), correlation vs. covariance, covariance inverse vs. square root of covariance inverse?

  • Regarding Contributions 1. if we have a cnn network, for which it is easy to compute output without recurrence, would not adding recurrent whitening network components increase inference computation complexity due to the required recurrence iterations (per input)?

  • Regarding "these implementations are exactly equivalent in the updates which they produce." in Section 2.4, How does equations (3) and (4) updates are equivalent? (4) is deliberately set not to affect forward wegiht updates.

  • How does the performance and complexity compare to the PRONG in (Desjardins et al. (2015)).

  • Is there a formal proof about why the whitening applied to each layer increases convergence speed? We also observe that it also increases the performance (accuracy) level for example for FA. Do we expect/justfiy this improvement?

  • Since this approach is proposed for mainly improving bio-plausible models, can we consider updates in (4) and (7) bio-plausible?

  • It would bin interesting to see covariance eigenvalues at the output of M filters for each layer, to evaluate the success of whitening. could you provide these (may be in appendix) as plots and also max/min eigenvalue disparity?

6 comments
AuthorsOpenReview ↗

Response: Weaknesses

We thank the reviewer for their detailed feedback, which has made the paper considerably clearer. Changes are marked in purple. Note that a new Appendix D has shifted some later letters (old D→E, E→F). This comment addresses the weaknesses; the next addresses the remaining questions.

Preconditioning/whitening idea for NN training were already introduced.

We agree that whitening for training is not new, and that PRONG is conceptually closest. As the reviewer notes in the strengths, what is new is that our whitening preserves the network's function and can be implemented by recurrent dynamics. This is what makes it fully local: every update uses only quantities available at a single synapse, which PRONG's eigendecompositions cannot offer. We compare performance and complexity directly in our answer to the PRONG question (Response 2/2).

Section 1: missing references on FA, DFA and Node Perturbation

We agree, and have added these in Section 1.3 (lines 137–142), including originating articles for node perturbation (Flower & Jabri, 1992; Fiete & Seung, 2006).

Section 1.1: Newton, motivation for natural gradient, mean centering

We agree that Newton's method belongs here, and have added it in lines 66–68. Our Appendix A, which motivates natural gradient descent, was never referenced in the introduction; we now refer to it in line 66. We have also clarified the last sentences of Section 1.1 to state that we do not mean-center, and link to Appendix D (see also our answer to Question 1).

Section 1.3: neither decorrelation nor whitening is properly defined

You are correct. We removed "whitening" from the title of Section 1.3, added a footnote defining decorrelation, and the first sentence now states that these methods address the right factor of Equation 2. Full definitions are given in the new Appendix D.

Contributions: the correlation inverse is called decorrelating

In the contributions, "decorrelating" only describes the transform MM applied to layer inputs (point 1) and existing methods that diagonalize input correlations (point 3). The inverse correlation only appears as a preconditioner, which we do not call decorrelating. If the reviewer had another sentence in mind, we would gladly correct it.

Section 2.1: basis and inner product

Gradient descent implicitly measures parameter changes with the Euclidean inner product, which is only appropriate if the parameters are orthonormal under the loss-distance metric (Section 1.1). Writing z=Wx=(WΣ1/2)xˉz = Wx = (W\Sigma^{1/2})\bar x shows that the coordinates in which this holds are WΣ1/2W\Sigma^{1/2}, not WW. Gradient descent on WW therefore steps in a basis skewed by Σ1/2\Sigma^{1/2}, which is the detour shown in Figure 1 and what the last sentence of that paragraph states.

Section 2.2 / Appendix D (now E): which metric?

We now link this derivation to the loss-distance metric of Section 1.1. Plain gradient descent on MM requires ⟨xx⊤⟩\langle xx^\top\rangle (the rightmost term of its gradient). Right-sided natural gradient multiplies by its inverse, cancelling that term, so the update needs neither the correlation matrix nor its inverse.

Section 2.3 and Appendix E (now F): equivalence for a single input

Thank you for catching this. The equation should have been the matrix identity W(u)M(u)=WMW^{(u)}M^{(u)} = WM, which holds for all inputs: Equations 3 and 4 give M(u)=AMM^{(u)} = AM and W(u)=WA−1W^{(u)} = WA^{-1}, with A=(1+ηM)I−ηMxˉxˉ⊤A = (1+\eta_M)I - \eta_M\bar x\bar x^\top. Appendix F, which derives the WW and RR updates, started from the same mis-statement. Both are fixed, and Appendix F now makes explicit how each step follows.

Section 2.4: formal proof and significance of the forms

We agree, and have added a formal proof of Theorem 1 in Appendix G. As to significance: the preconditioning form connects our method to existing optimizers, the linear form is efficient on conventional hardware, and the recurrent form is the only fully local one.

Notation of R_i and M_i

The strength of MM and RR is that they are learned online and without any eigendecomposition or matrix inversion. However, that also means that they only approximately whiten during training, so we cannot define them as exact (inverse) square roots. We agree that RR may suggest a correlation matrix, and are happy to rename it if the reviewer prefers.

Section 3: CRGD acronym; which form is applied

Thank you, we now define CRGD at the start of Section 3, and state there that experiments use the preconditioning form, with the recurrent form tested separately in Appendix M (Fig. 7b).

AuthorsOpenReview ↗

Response: Questions

Response 2/2: Questions

Q1. Decorrelation vs whitening, mean centering, correlation vs covariance, inverse vs inverse square root?

We agree these needed defining, and have added Appendix D. Please consult this section for a clearer description of decorrelation and whitening in this work. In short, decorrelation is used to communicate when M⟨xx⊤⟩M⊤M\langle xx^\top\rangle M^\top is diagonal, and whitening is the special case in which it is the identity.

We do not mean-center, even after ReLUs, since Equation 2 uses the uncentered ⟨xx⊤⟩\langle xx^\top\rangle, which we call "correlation" in the signal-processing sense. It is important to note that this was not an arbitrary choice: an uncentered second moment falls out of our natural gradient derivation (see Appendix C).

You are correct that the inverse itself does not decorrelate. Instead, the whitening transform MM (an inverse square root) acts on the data, and the full inverse then appears in the effective update, ΔV=ΔWM=δx⊤M⊤M=δx⊤⟨xx⊤⟩−1\Delta V = \Delta W M = \delta x^\top M^\top M = \delta x^\top\langle xx^\top\rangle^{-1}, exactly as Equation 2 requires.

Q2. Would recurrent whitening increase inference cost in a CNN?

The reviewer is correct that the recurrent form needs extra iterations per input. However, the recurrent form is only needed where computation must be local, e.g. on analogue or neuromorphic hardware; on GPUs, the preconditioner or linear forms are efficient and can be used instead. Moreover, at inference no extra cost arises in any form: since the whitening is linear, MM (or R−1R^{-1}) can be folded into the weights, V=WMV = WM, giving exactly the original CNN's architecture and cost.

Q3. How are the updates of (3) and (4) equivalent?

We see how this sentence could be misread. It is the three implementations (preconditioner, linear transform, recurrent dynamics) that are equivalent. Equations 3 and 4 are not equivalent to each other, but together define the mechanism that all three share. Theorem 1 states this, and Appendix G now proves it.

Q4. Performance and complexity compared to PRONG?

Appendix M (Fig. 7a) now includes this comparison. PRONG outperforms CRGD by about 5%. In complexity, PRONG stores samples, centres them, and computes an O(n3)O(n^3) eigendecomposition at checkpoints. Ours is a rank-one Hebbian update costing O(n2)O(n^2) per sample, without centering, which our derivation shows is not required. PRONG thus requires a global coordinator and cannot run in distributed or neuromorphic systems, nor serve as a model of the brain.

Q5. Is there a formal proof that whitening increases convergence speed? Is the FA improvement expected?

We know of no formal proof for deep networks. However, for a linear layer under squared loss, the Hessian is ⟨xx⊤⟩⊗I\langle xx^\top\rangle\otimes I, and whitening its inputs reduces its condition number to one, which is the standard argument for faster gradient descent.

We did expect FA to benefit. Approximate methods can be seen as semi-random guesses of the gradient. Figure 2 (right) shows that directions within 90° of the natural gradient are brought into closer alignment with it by our method. Since FA's updates are the least accurate, it could be said to have the most to gain.

Q6. Are the updates in (4) and (7) bio-plausible?

Both numerators are local Hebbian outer products: each synapse WjkW_{jk} needs only hjh_j and xˉk\bar x_k, and each lateral synapse RjkR_{jk} needs only xjx_j and xˉk\bar x_k. You are correct that the squared norm in the denominator is not local. Since it only scales the learning rate, we add a demonstration that it can be replaced with a constant (Appendix N.3, Fig. 8b). In a fully bio-plausible run (batch size 1, recurrent form, no momentum, constant norm), CRGD still clearly beats SGD (Fig. 7b). This was also raised by reviewer rqq4; see our response there for more details.

Q7. Covariance eigenvalues at the output of M, and max/min eigenvalue disparity?

Thank you for this suggestion. Appendix O now reports the whitening loss (Fig. 9), the max/min eigenvalue ratio, i.e. condition number (Fig. 10), and the eigenspectra of each layer (Fig. 11) during training. The whitening loss of the whitened inputs ends orders of magnitude below that of the raw inputs, the condition number drops by two to five orders of magnitude, and the spectra flatten.

I would like to thank the authors for their detailed point responses. With the changes implemented, the article's presentation has improved.

The authors confirm that the essential contribution of the article (relative to PRONG) is the use of recurrent dynamics for whitening. As a consequence, the article should be framed accordingly, computationally efficient/biologically plausible whitening to replace PRONG's whitening approach. However, that requires careful exploration and comparison with the existing computationally efficient and biologically plausible methods to position the article's contribution in this respect. There are various whitening approaches involving recurrent dynamics connected with PCA/efficient coding related work. For example, Eq (6) for recurrent dynamics is not really novel. For example, equations (42) and (43) of Cruces&Cichocki 2003 [a] are essentially the same implementation as defined by weight updates (7) and recurrent dynamics (6), where (43) can be considered as Euler step for (6). There are also

[a] Cruces, Cichocki "GLOBALLY CONVERGENT NEWTON ALGORITHMS FOR BLIND DECORRELATION", ICA 2003.

There are also various other neural recurrent whitening implementations especially in computational neuroscience field (starting from Godal and Foldiak's work on anti-hebbian decorrelation to more recent whitening approaches). I think an article positioned to offer better bio-plausible whitening should reflect this literature and clarify its contributions.

Reviewer_r4hTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes online whitening with compensating weight updates to implement input-side natural-gradient preconditioning, reporting improved performance for backpropagation and biologically inspired learning methods on CIFAR-10/100.

Critical strengths
  1. The function-preserving weight correction addresses a meaningful problem

  2. The reported improvements are reasonably consistent across settings

Critical weaknesses

The experiments do not directly validate the most distinctive proposed mechanism. Appendix J states that the experiments use the preconditioning implementation with a mini-batch size of 128. These results support the method’s empirical utility, but do not directly validate the single-sample recurrent implementation or establish the practical benefit of the compensating weight update.

Questions for authors
  1. How is the layer-wide squared norm in Equations (4) and (7) obtained using only locally available signals? Does the implementation require a shared or broadcast signal?

  2. What conditions ensure that the correction denominator stays away from zero and that the recurrent dynamics in Equation (6) remain stable?

3 comments

The experiments do not directly validate the most distinctive proposed mechanism. Appendix J states that the experiments use the preconditioning implementation with a mini-batch size of 128. These results support the method's empirical utility, but do not directly validate the single-sample recurrent implementation or establish the practical benefit of the compensating weight update.

We agree that this was missing. In response to this comment and the request by reviewer rqq4, we have added a fully biologically plausible run. This version uses a batch size of 1, the recurrent formulation, a fixed constant in place of the non-local norm, and neither weight decay nor momentum. Figure 7b of Appendix M presents these results: CRGD clearly outperforms SGD trained under the same constraints, and loses only ~3% relative to the batched, preconditioned formulation.

Note that the learning rates for these batch-1 simulations were chosen in a sub-optimal manner (based upon dividing the best learning rate from a sweep at batch size 128). This is not necessarily optimal, so one might even expect further performance improvements from tuning at batch size 1, and from tuning the constant used in place of the norm.

To establish the practical benefit of the compensating weight update, Figure 7a now includes an ablation against decorrelated backpropagation (DBP; Dalm et al., 2024), which learns the whitening transform in the same way but without the function-preserving correction. In like-for-like simulations with individually tuned hyperparameters, our function-preserving correction improves performance by 5%.

How is the layer-wide squared norm in Equations (4) and (7) obtained using only locally available signals? Does the implementation require a shared or broadcast signal?

The reviewer is correct that, as written, the exact squared norm requires a shared or broadcast signal within a layer, and so is not strictly local. This concern was also raised by reviewer rqq4. However, the denominator only acts as a per-sample scaling of the learning rate, and so long as Equations 4 and 7 use identical values, the function-preserving property of our learning rules holds exactly. At the minimum of the whitening loss, the norm tends to the layer's width in expectation. We therefore replace the norm with a fixed scalar, removing the need for any shared signal. Appendix N.3 (Figure 8b) shows that the whitening loss is reduced equivalently when a sufficiently small constant is used in place of the norm. In training simulations (Appendix M, Figure 7b), CRGD with a constant of 1 still clearly outperforms SGD.

What conditions ensure that the correction denominator stays away from zero and that the recurrent dynamics in Equation (6) remain stable?

We have added Appendix N to address exactly this in terms of the two key requirements: invertibility and stability. For invertibility: with γ=1+ηM\gamma = 1 + \eta_M, the denominator is positive whenever ηMxˉ⊤xˉ<γ\eta_M \bar x^\top \bar x < \gamma. With the fixed constant cc this becomes ηMc<1+ηM\eta_M c < 1 + \eta_M, which holds for any learning rate when c=1c = 1 (the denominator then equals γ\gamma), and for c<1001c < 1001 at our ηM=10−3\eta_M = 10^{-3}. Under this condition, each update multiplies RR by a positive definite matrix, so RR remains invertible. For stability: Appendix N.1 solves the dynamics of Equation 6 analytically, showing convergence to xˉ=R−1x\bar x = R^{-1}x whenever all eigenvalues of RR have positive real parts. N.2 proves that this holds throughout learning with full-data statistics for a sufficiently small learning rate (RR stays symmetric positive definite). For single-sample updates, RR can accumulate small rotations; we show these grow only as ηMt\eta_M\sqrt{t}, and can be bounded by a decaying learning rate or a local decay of RR towards II. Appendix O confirms empirically that the smallest real part of the eigenvalues remains well above zero throughout training, including for the bio-plausible runs.

I would like to thank the authors for for their detailed responses.

My concerns have been addressed. And I would like to raise my score to 3.

Also, some parts of your reply is in latex format, while it doesnt show up correctly.

Reviewer rcG6 pointed out existing equations on whitening that seems similar to this work, it would be nice if authors could clarify that part as well.

We thank the reviewer for their positive response and also for the upgraded score. This is much appreciated.

We have now further updated the manuscript to address reviewer rcG6's concerns and hope to have made even clearer the relation of our work to existing equations for whitening.

Reviewer_rqq4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper considers the input-side factor of the natural gradient, arguing that this can be handled by whitening activations rather than preconditioning gradients. The authors derive an online whitening rule and a correction to the forward weights that preserves the network function. They show equivalence between gradient preconditioning, an explicit linear transform, and recurrent dynamics with Hebbian updates. Experiments on CIFAR-10/100 show improvements, particularly for approximate credit assignment methods.

Critical strengths

Interesting connection between natural gradients, whitening, and biologically plausible learning. The function-preserving correction is a nice result.

Strong improvements for approximate credit assignment, particularly predictive coding. Results are fairly consistent across the architectures tested.

The recurrent interpretation is interesting and potentially useful beyond the specific optimization setting.

Critical weaknesses

The locality claim seems overstated. Equations 4 and 7 depend on the squared norm of the full activity vector. This requires information beyond the pre- and postsynaptic activities of an individual connection since to compute the norm we need to sum the entire population of neurons. It is thus unclear how it could work without a separate population-wide normalization circuit

The experiments do not validate the central equivalence. The theoretical equivalence holds for single-sample updates, but experiments use mini-batches of 128 and appear to use the direct preconditioning formulation. There is no experimental demonstration of the recurrent implementation, which is arguably the most interesting contribution. Additionally, the core theorems concern ordinary gradient updates, but the experiments use momentum and weight decay in the optimizer. A changing parameterization generally also requires transforming optimizer state. The function-preserving weight correction alone does not address this.

There are some apparent issues with the derivation and implementation. Algorithm 2 seems inconsistent with Theorem 1: the task-gradient update uses the unwhitened input, and the update ordering does not appear to preserve the claimed equivalence. Also, the recurrent dynamics require stability conditions beyond invertibility of the transform. These conditions are not established.

Limited ablations. Hard to tell whether the gains come from the proposed mechanism or simply from input normalization. A diagonal preconditioner would be an obvious baseline. More importantly, there is no direct ablation showing the benefit of the function-preserving correction, arguably one of the main contributions. The actual degree of whitening achieved during training is also not reported.

Novelty relative to prior work needs clarification. The connection between whitening and natural gradients, as well as function-preserving reparameterization, is already present in prior work (e.g., Desjardins et al., 2015). Online decorrelation and recurrent whitening are also well studied. The precise additional contribution here should be made clearer, ideally with direct comparisons to existing whitening methods.

Practical comparisons are somewhat unconvincing. The method is over 4x slower than SGD and roughly 3x slower than Muon in the reported timing experiment. Comparisons are mostly in terms of epochs rather than wall-clock time. Experiments are also limited to CIFAR-scale tasks and relatively short training runs. This makes the broader optimizer claims difficult to assess.

Questions for authors

How is the normalization in Equations 4 and 7 computed locally? Does this require a global signal?

Can the authors verify numerically that the three implementations give identical updates? Algorithm 2 in particular seems inconsistent with the theorem.

Have the recurrent dynamics been implemented and tested? What ensures their stability during training?

How does the method compare to diagonal input normalization, and how much of the gain requires full decorrelation?

2 comments

We thank the reviewer for their constructive feedback. We genuinely think that it helped us improve the paper considerably. See the updated PDF's purple sections for changes to our manuscript.

The locality claim seems overstated.

The norm in the denominators of Equations 4 and 7 could indeed be considered non-local, and this criticism is well placed. However, since it is strictly a positive scalar, it impact is as a variable scaling of the learning rate. As long as its value is equivalent for Equations 4 and 7, function preservation holds, and at the minimum of the whitening loss in expectation it tends to the layer's width. To address this fully, we replace this term with a static scalar and explore a wide range of values (Figure 7b, Appendix M; Figure 8b, Appendix N.3). We show that a sufficiently small constant delivers equivalent performance.

With this addition we can thus ensure that our approach can be proposed as fully local.

The experiments do not validate the central equivalence. ... momentum and weight decay

We agree, and have added simulations demonstrating our equivalences empirically: 1) a 'bio-plausible simulation' with vanilla SGD (no momentum or weight decay), batch-size 1, the recurrent formulation, and xˉ⊤xˉ←1\bar x^\top \bar x \leftarrow 1 (Figure 7b), in which CRGD still clearly improves over SGD, and 2) a demonstration that the recurrent dynamics converge to the fixed point of the linear method if integrated far enough in time (Figure 8a).

The reviewer is also correct that with momentum, function-preserving updates must also be applied to the momentum states. This uses only quantities already available at the synapse, and only applies to the recurrent and linear formulations, so our preconditioner-based experiments are unaffected. We add an acknowledgement of this in Section 2.4 (Lines 307-313).

Algorithm 2 seems inconsistent with Theorem 1

Thank you for your keen eye. This was mis-written on our part. We have added the missing bar on x in the forward update, and it is correct that the update order must be preserved, so we have also fixed the ordering in Algorithm 2.

The recurrent dynamics require stability conditions beyond invertibility

We agree. Appendix N now provides an analytic solution to the recurrent dynamics (N.1), which converge if R is invertible and its eigenvalues have only positive real parts. N.2 shows that small learning rates are sufficient to meet these conditions, including when a constant replaces the norm, and that a regularizer can further aid with this. Appendix O demonstrates that these conditions hold in practice and that whitening is achieved stably.

Limited ablations

Appendix M now includes these ablations (Figure 7a): a diagonal-only preconditioner, and decorrelated backpropagation (DBP, Dalm et al., 2024), which whitens without any function-preserving correction. A diagonal preconditioner does not improve over plain SGD, and without function preservation, final performance falls behind our approach by about 5%.

The actual degree of whitening achieved during training is also not reported.

This was indeed missing. In response, and to the question of reviewer rcG6 on eigenspectra, Appendix O reports the whitening loss, eigenspectra and condition number across training. The whitened inputs have a whitening loss orders of magnitude below the raw inputs, and the eigenspectrum flattens late in training, as we would expect.

Novelty relative to prior work needs clarification.

Figure 7a also now compares against PRONG (Desjardins et al., 2015). We are transparent that PRONG outperforms CRGD here; however, it relies on periodic eigendecompositions of stored samples, which cannot be performed locally or on alternative hardware. Our contribution is showing that function-preserving whitening can be achieved by local, recurrent dynamics that are exactly equivalent to right-sided natural-gradient preconditioning. Prior online decorrelation and whitening methods are discussed further in Sections 1.3 and 4. We hope this helps to clarify the novelty.

The method is over 4x slower than SGD ...

On typical machine learning hardware, our methods are indeed rather slow. However, the point of the paper is not only to speed up training. Instead, while preconditioners perform well on traditional hardware, they cannot be applied on neuromorphic hardware, nor serve as a model of the brain. We show that they are equivalent to recurrent whitening dynamics that are performant. On alternative hardware (neuromorphic, biological, analogue) the computational tradeoff could be very different and this paper hopes to bridge across such spaces.

Experiments are also limited to CIFAR-scale tasks

We hope that Table 1 provides some evidence and confidence for the scaling of these methods to deeper architectures. We acknowledge that investigating these approaches at larger scale and in custom hardware would be natural next steps.

Dear Reviewer rqq4,

We hope that our response above addressed the concerns that you held in your review. Additionally, please see our responses to reviewer rcG6 in which we provided a further update to our paper in order to provide a more clear embedding of our work within the range of existing methods for whitening and decorrelation.

If you have any remaining concerns or questions, approximately 24 hours remain in this discussion phase and we would deeply appreciate the opportunity to address them. Otherwise if you are satisfied we would also be glad to hear.

Regardless, on behalf of the authors: thank you for engaging in this review process with us.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.