- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies random feedback alignment (RFA) dynamics. The authors show that the classical conserved quantities under GF dynamics for a 2 layer linear NN are not conserved under RFA dynamics. The authors give experimental evidences that balanced initializations are beneficial for RFA training for a non linear NN.
Critical strengths
S1. The proof of Theorem 1 is rigorous
S2. It is interesting that balanced init could be beneficial for RFA training
Critical weaknesses
W1. The introduction is not well written
W2. The motivation of the paper is not clear. Why do we care about Theorem 1? In particular, while it is well known that balancedness is an invariant of gradient flow (which the authors refer to as backpropagation), the RFA dynamics that the present paper studies has no reason to share this invariant.
W3.There is a confusion between gradient flow and gradient descent in this paper. Line 139-146: it is GF that preserves -balancedness, not GD. For GD, the conserved quantities remain approximatively conserved, not exactly conserved.
W4. The link between conservation and balancedness initialization is not well motivated. The authors give experimental evidence that balanced init is beneficial for RFA training for non linear networks. However, for non linear networks the quantity is not preserved.
Questions for authors
please comment the weaknesses above