- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The article proposes to use the replacement of gradient descent in the standard backprop algorithm, with a version of natural gradient algorithm. In this version, the decorrelation of the input vector emerges naturally, therefore, the authors propose the use of decorrelating network architectures as the solution to improve convergence behavior. They demonstrate the impact of this approach on CIFAR10 and CIFAR100 datasets, for different networks (e.g. LeNet) and different algorithms (BP, Feedback Alginment, sign concordance, error predictive coding) . The authors' experiments show notable performance improvement when decorrelation is introduced.
Critical strengths
-
Idea of using natural gradient and observing its decomposition in terms of the input correlations is noteworthy.
-
The proposed whitening methods, preserving output an having recurrent structure are new to my knowledge.
-
The performance improvements relative to not using whitening and relative to the other preconditioning methods are remarkable (especially for FA.)
Critical weaknesses
-
Preconditioning/whitening idea for NN training were already introduced. The main contribution of this article is somewhat restricted to alternative and successful implementation of the whitening (especially relative to the PRONG method in the cited article (Desjardins et al. (2015)).
-
The presentation requires improvement. See my notes below.
SECTION 1
This section is missing disscussion about basic references on backprop alternatives including originating articles on FA,DFA and Node Perturbation.
Section 1.1 is also missing discussion of Newton which introduces loss curvature based alignment for the gradient and its potential relation with/deviation from the natural gradient approach. The authors could also provide meaning and motivation for the natural gradient, especially how the use of metric change can help faster/robust convergence to global optima. It is not clear from the last statement whether the authors perform mean centering in the algorithm or not.
Section 1.3: Neither decorrelation nor whitening is properly defined here. They are not defined earlier also. Section 1.1 has the inverse of the data correlation matrix, but this is not whitening, the multiplication with an inverse square root of the covariance is. I think the authors should link this section to Equation (2) to clarify what is intended.
CONTRIBUTIONS
- the authors call the correlation inverse is decorrelating, but multiplying with the inverse of correlation inverse is not decorrelation process. (but with a square root of the covariance is)
SECTION 2
Section 2.1: The title of this subsection is not clear to me at the first read (what is the considered basis and what is the assumed inner product, what does it mean to place parameters in a basis.) The last sentence of the first paragraph is also unclear too.
Section 2.2: Natural gradient based derivation in Appendix D is not clear. Which Riemannian metric is used? It is also a bit confusing. M_i is essentially a square root of covariance inverse matrix that we are trying to learn, however, its update requires the inverse of the covariance, and the natural gradient approach nicely eliminates this requirement?
Section 2.3 The equivalence in unnumbered equation (above (4)) is for input only. We can not guarantee that the system preserved equivalence for other inputs right?
Appendix E This appendix is about the derivation of the M update equation, which I found confusing. In this there is a sequence of equations. it is not clear if they are meant to imply each other. For example, the first equality does not imply second equality.
Section 2.4 Given that the result is offered in Theorem form, it better be followed by a formal proof. Furthermore, a discussion before Theorem about why we need such alternative forms or their significance would provide some clarity.
Regarding, R_i and M_i, they better be clearly defined as square root and inverse square root of covariance(correlation matrix) for inputs. Using R_i notation might give the wrong impression that it is a correlation (covariance) matrix for x_i's.
Section 3
I don't think the acronym CRGD for the proposed method is provided in the article. That makes the discussion in Section 3 a bit confusing. It is not clear, which of the three whitening operations of Theorem is applied.
Questions for authors
-
Could you clarify decorrelation vs whitening within the context of the article, clarifying the existence of mean centering (especially considering the layer activations with ReLU nonlineariities.), correlation vs. covariance, covariance inverse vs. square root of covariance inverse?
-
Regarding Contributions 1. if we have a cnn network, for which it is easy to compute output without recurrence, would not adding recurrent whitening network components increase inference computation complexity due to the required recurrence iterations (per input)?
-
Regarding "these implementations are exactly equivalent in the updates which they produce." in Section 2.4, How does equations (3) and (4) updates are equivalent? (4) is deliberately set not to affect forward wegiht updates.
-
How does the performance and complexity compare to the PRONG in (Desjardins et al. (2015)).
-
Is there a formal proof about why the whitening applied to each layer increases convergence speed? We also observe that it also increases the performance (accuracy) level for example for FA. Do we expect/justfiy this improvement?
-
Since this approach is proposed for mainly improving bio-plausible models, can we consider updates in (4) and (7) bio-plausible?
-
It would bin interesting to see covariance eigenvalues at the output of M filters for each layer, to evaluate the success of whitening. could you provide these (may be in appendix) as plots and also max/min eigenvalue disparity?
6 comments
Response: Questions
Response 2/2: Questions
Q1. Decorrelation vs whitening, mean centering, correlation vs covariance, inverse vs inverse square root?
We agree these needed defining, and have added Appendix D. Please consult this section for a clearer description of decorrelation and whitening in this work. In short, decorrelation is used to communicate when is diagonal, and whitening is the special case in which it is the identity.
We do not mean-center, even after ReLUs, since Equation 2 uses the uncentered , which we call "correlation" in the signal-processing sense. It is important to note that this was not an arbitrary choice: an uncentered second moment falls out of our natural gradient derivation (see Appendix C).
You are correct that the inverse itself does not decorrelate. Instead, the whitening transform (an inverse square root) acts on the data, and the full inverse then appears in the effective update, , exactly as Equation 2 requires.
Q2. Would recurrent whitening increase inference cost in a CNN?
The reviewer is correct that the recurrent form needs extra iterations per input. However, the recurrent form is only needed where computation must be local, e.g. on analogue or neuromorphic hardware; on GPUs, the preconditioner or linear forms are efficient and can be used instead. Moreover, at inference no extra cost arises in any form: since the whitening is linear, (or ) can be folded into the weights, , giving exactly the original CNN's architecture and cost.
Q3. How are the updates of (3) and (4) equivalent?
We see how this sentence could be misread. It is the three implementations (preconditioner, linear transform, recurrent dynamics) that are equivalent. Equations 3 and 4 are not equivalent to each other, but together define the mechanism that all three share. Theorem 1 states this, and Appendix G now proves it.
Q4. Performance and complexity compared to PRONG?
Appendix M (Fig. 7a) now includes this comparison. PRONG outperforms CRGD by about 5%. In complexity, PRONG stores samples, centres them, and computes an eigendecomposition at checkpoints. Ours is a rank-one Hebbian update costing per sample, without centering, which our derivation shows is not required. PRONG thus requires a global coordinator and cannot run in distributed or neuromorphic systems, nor serve as a model of the brain.
Q5. Is there a formal proof that whitening increases convergence speed? Is the FA improvement expected?
We know of no formal proof for deep networks. However, for a linear layer under squared loss, the Hessian is , and whitening its inputs reduces its condition number to one, which is the standard argument for faster gradient descent.
We did expect FA to benefit. Approximate methods can be seen as semi-random guesses of the gradient. Figure 2 (right) shows that directions within 90° of the natural gradient are brought into closer alignment with it by our method. Since FA's updates are the least accurate, it could be said to have the most to gain.
Q6. Are the updates in (4) and (7) bio-plausible?
Both numerators are local Hebbian outer products: each synapse needs only and , and each lateral synapse needs only and . You are correct that the squared norm in the denominator is not local. Since it only scales the learning rate, we add a demonstration that it can be replaced with a constant (Appendix N.3, Fig. 8b). In a fully bio-plausible run (batch size 1, recurrent form, no momentum, constant norm), CRGD still clearly beats SGD (Fig. 7b). This was also raised by reviewer rqq4; see our response there for more details.
Q7. Covariance eigenvalues at the output of M, and max/min eigenvalue disparity?
Thank you for this suggestion. Appendix O now reports the whitening loss (Fig. 9), the max/min eigenvalue ratio, i.e. condition number (Fig. 10), and the eigenspectra of each layer (Fig. 11) during training. The whitening loss of the whitened inputs ends orders of magnitude below that of the raw inputs, the condition number drops by two to five orders of magnitude, and the spectra flatten.
I would like to thank the authors for their detailed point responses. With the changes implemented, the article's presentation has improved.
The authors confirm that the essential contribution of the article (relative to PRONG) is the use of recurrent dynamics for whitening. As a consequence, the article should be framed accordingly, computationally efficient/biologically plausible whitening to replace PRONG's whitening approach. However, that requires careful exploration and comparison with the existing computationally efficient and biologically plausible methods to position the article's contribution in this respect. There are various whitening approaches involving recurrent dynamics connected with PCA/efficient coding related work. For example, Eq (6) for recurrent dynamics is not really novel. For example, equations (42) and (43) of Cruces&Cichocki 2003 [a] are essentially the same implementation as defined by weight updates (7) and recurrent dynamics (6), where (43) can be considered as Euler step for (6). There are also
[a] Cruces, Cichocki "GLOBALLY CONVERGENT NEWTON ALGORITHMS FOR BLIND DECORRELATION", ICA 2003.
There are also various other neural recurrent whitening implementations especially in computational neuroscience field (starting from Godal and Foldiak's work on anti-hebbian decorrelation to more recent whitening approaches). I think an article positioned to offer better bio-plausible whitening should reflect this literature and clarify its contributions.
Response: Weaknesses
We thank the reviewer for their detailed feedback, which has made the paper considerably clearer. Changes are marked in purple. Note that a new Appendix D has shifted some later letters (old D→E, E→F). This comment addresses the weaknesses; the next addresses the remaining questions.
Preconditioning/whitening idea for NN training were already introduced.We agree that whitening for training is not new, and that PRONG is conceptually closest. As the reviewer notes in the strengths, what is new is that our whitening preserves the network's function and can be implemented by recurrent dynamics. This is what makes it fully local: every update uses only quantities available at a single synapse, which PRONG's eigendecompositions cannot offer. We compare performance and complexity directly in our answer to the PRONG question (Response 2/2).
Section 1: missing references on FA, DFA and Node PerturbationWe agree, and have added these in Section 1.3 (lines 137–142), including originating articles for node perturbation (Flower & Jabri, 1992; Fiete & Seung, 2006).
Section 1.1: Newton, motivation for natural gradient, mean centeringWe agree that Newton's method belongs here, and have added it in lines 66–68. Our Appendix A, which motivates natural gradient descent, was never referenced in the introduction; we now refer to it in line 66. We have also clarified the last sentences of Section 1.1 to state that we do not mean-center, and link to Appendix D (see also our answer to Question 1).
Section 1.3: neither decorrelation nor whitening is properly definedYou are correct. We removed "whitening" from the title of Section 1.3, added a footnote defining decorrelation, and the first sentence now states that these methods address the right factor of Equation 2. Full definitions are given in the new Appendix D.
Contributions: the correlation inverse is called decorrelatingIn the contributions, "decorrelating" only describes the transform M applied to layer inputs (point 1) and existing methods that diagonalize input correlations (point 3). The inverse correlation only appears as a preconditioner, which we do not call decorrelating. If the reviewer had another sentence in mind, we would gladly correct it.
Section 2.1: basis and inner productGradient descent implicitly measures parameter changes with the Euclidean inner product, which is only appropriate if the parameters are orthonormal under the loss-distance metric (Section 1.1). Writing z=Wx=(WΣ1/2)xˉ shows that the coordinates in which this holds are WΣ1/2, not W. Gradient descent on W therefore steps in a basis skewed by Σ1/2, which is the detour shown in Figure 1 and what the last sentence of that paragraph states.
Section 2.2 / Appendix D (now E): which metric?We now link this derivation to the loss-distance metric of Section 1.1. Plain gradient descent on M requires ⟨xx⊤⟩ (the rightmost term of its gradient). Right-sided natural gradient multiplies by its inverse, cancelling that term, so the update needs neither the correlation matrix nor its inverse.
Section 2.3 and Appendix E (now F): equivalence for a single inputThank you for catching this. The equation should have been the matrix identity W(u)M(u)=WM, which holds for all inputs: Equations 3 and 4 give M(u)=AM and W(u)=WA−1, with A=(1+ηM)I−ηMxˉxˉ⊤. Appendix F, which derives the W and R updates, started from the same mis-statement. Both are fixed, and Appendix F now makes explicit how each step follows.
Section 2.4: formal proof and significance of the formsWe agree, and have added a formal proof of Theorem 1 in Appendix G. As to significance: the preconditioning form connects our method to existing optimizers, the linear form is efficient on conventional hardware, and the recurrent form is the only fully local one.
Notation of R_i and M_iThe strength of M and R is that they are learned online and without any eigendecomposition or matrix inversion. However, that also means that they only approximately whiten during training, so we cannot define them as exact (inverse) square roots. We agree that R may suggest a correlation matrix, and are happy to rename it if the reviewer prefers.
Section 3: CRGD acronym; which form is appliedThank you, we now define CRGD at the start of Section 3, and state there that experiments use the preconditioning form, with the recurrent form tested separately in Appendix M (Fig. 7b).