- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors propose a recurrent architecture, termed Gating on Weights with Feedback (GaWF), and find that relative to parameter-matched RNNs, LSTMs, GRUs, and SSMs it can outperform them on a cluttered target-tracking MNIST task.
Critical strengths
I enjoyed the parameter-matched controls (Fig. 2) and detailed analyses of the internals of the GaWF (Fig. 11), including the topographic matching in Fig. 5.
Critical weaknesses
-
The differences between the GaWF and RNN on test accuracy in Fig. 2a appear to be quite small.
-
There is only one task as far as I can tell, and it is quite toy and synthetic (based on MNIST images).
More details on these weaknesses with my questions, in the "Questions" section below.
Questions for authors
- The task is synthetic (and there really is only one, CM-MNIST), what about on real images? Say you take ImageNet images and add clutter? Could you compare to the multi-layer Reciprocal Gated Cells (RGC) with feedback (Nayebi et al. NeurIPS 2018: https://arxiv.org/abs/1807.00053) and CORNet-S ConvRNNs (Kubilius et al. NeurIPS 2019: https://arxiv.org/abs/1909.06161) on these images?
I suggest this since both of these models have top-down feedback and have been applied to much larger scale datasets such as ImageNet. In particular, if you happen to be short on time during the rebuttal stage, the Reciprocal Gated Cell (RGC) of Nayebi et al. 2018 does segregate its feedback signal in the cell architecture itself, so it may be the most relevant of the two comparisons, though comparing to both would be ideal.
-
The models used are quite small, what happens if it is increased while keeping parameter count across models the same -- do the model differences remain the same? Or is this not possible on the current MNIST task because it will overfit otherwise given the smallness of the task? This overfitting issue can be alleviated by training on larger naturalistic datasets like ImageNet. The reason I ask this is because I worry any benefit from GaWF may be due to a "law of small numbers" effect, and may not hold on larger models and larger, more naturalistic, datasets (which also end-up producing brain-like responses in the primate ventral visual pathway, as ImageNet does, cf. Yamins et al. PNAS 2014).
-
In Appendix A.2 pg. 13, the initializations of the encoders are said to be the same, but I am not sure if this means the RNNs/SSMs are initialized the exact same. If that's the case, this is quite suboptimal for various choices of these models -- as their initialization schemes (especially for SSMs) really matter (e.g. HiPPO). What happens if you do the optimal initialization schemes, along with optimal parameters, for each model family?
2 comments
Thank you for running this. I will raise my score from 1 to 2, and I encourage the authors to continue to develop their study with other biologically-motivated RNN architectures (the RGC of Nayebi et al. 2018 and CORNet-S of Schrimpf et al. 2019, both of whose PyTorch implementation can be found in, e.g. https://github.com/neuroagents-lab/PyTorchTNN and https://github.com/dicarlolab/CORnet/tree/master, respectively) and realistic tasks (e.g. ImageNet) as I have highlighted in my initial review, which weren't addressed.
Thank you for the constructive review and for the positive comments on the parameter-matched controls and the analyses of the GaWF internals.
Effect size, model scale, and a second task family. We also examined whether the benefits of the GaWF design extend to a different, harder task and are retained in a multi-layer architecture. In preliminary experiments, we trained a three-layer GaWF with DQN on Atari Breakout from pixels for 3M agent steps, with four-frame action repeat and frame stacking and the minimal four-action set. In this multi-layer GaWF, the first two layers receive feedback from the previous hidden state of the adjacent higher layer, and the third receives the previous step's four action values from the readout. GaWF, RNN, LSTM, and GRU each used three recurrent layers, approximately 7.55M parameters (about 13 times the CM-MNIST budget), and 10 seeds. GaWF has the highest mean of the four models in final 100-episode training return (mean ± SEM, ε = 0.05):
Initialization. Appendix A.2 (p. 13) states that all architectures instantiate the same encoder class under the same PyTorch initialization rules; this refers only to the CNN encoder. Each sequence model keeps the default initialization of its reference implementation: our S5 and Mamba wrappers instantiate the library modules without overriding their internal parameters. S5 therefore retains its diagonalized HiPPO-based state-matrix initialization with log-uniform step sizes, and Mamba retains the default initialization of the mamba_ssm module.