- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors introduce an on-off circuit motif for "biologically plausible" credit assignment, which accomodates nonnegative firing rates, Dale's law, and layer-local learning. For the base models they consider, they get comparable performance on MNIST, Fashion MNIST, CIFAR-10, and Tiny ImageNet.
Critical strengths
For the base models the authors consider, they get comparable performance on MNIST, Fashion MNIST, CIFAR-10, and Tiny ImageNet.
Critical weaknesses
-
Dale's law is not necessarily biologically plausible, so it is unclear to me why this is a principal motivation for the authors (and seems awfully restrictive on what the learning algorithm can do, but see also my next point below)
-
The base models used are not great and are quite below SOTA top-1 performance on these tasks, even with the backprop baseline in Tables 1-2. For example, with modern CNNs, SOTA performance on MNIST is 99.9% (not 97-98% as in Table 1), SOTA performance on Fashion MNIST is 99.65% (not 87-88% as in Table 1), SOTA performance on CIFAR-10 is around 99% (not 54-56% as in Table 1), and finally SOTA performance on TinyImageNet is 66-67% not 42-43% as in Table 2.
Questions for authors
-
Related to the base model weakness, how many layers are the networks used?
-
Can you try ImageNet-1k with 224x224 images (rather than 64x64 images), and compare with other methods like Information Alignment (IA) of Kunin et al. ICML 2020 (https://arxiv.org/abs/2003.01513) which maintains positive firing rates via relu and scales to backprop with hundreds of layers on ImageNet?