- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes a meta-learned plasticity rule for a feedforward neural network classifier, parameterized by an MLP that is a function of the pre- and post-synaptic activity, the current synaptic weight, reward (whether the current step's class output is correct), and a per-neuron "plasticity state" that is updated at every step (but not used in the forward computation). The rule is meta-learned and evaluated on a suite of online learning tasks (permuted-MNIST, RHM, permuted-Fashion-MNIST, and Omniglot), demonstrating performance superior to classical local plasticity rules and in some cases comparable to online backprop.
Critical strengths
- The work towards mechanistic understanding of the rule through fitting it with an interpretable (polynomial) family, as well as perturbation of learning signals (plasticity state, reward), and the step-index control providing insight to the function of the plasticity state.
A minor note, although the paper was easy to follow without it, I appreciated the thorough glossary in the appendix.
Critical weaknesses
- Lacking comparable baselines with learned plasticity, e.g. meta-learning (rather than grid-search) the parameters of the classical rules, meta-learned polynomial fits (with or without feedback; see Questions 1-2), as well as including the version without plasticity states in Fig 3A-D.
- The complexity of the rule appears to be only partially justified. From the authors' analysis, the per-neuron plasticity state appears to contribute relatively little unique information beyond a timing signal, and even causes deterioration for learning beyond the trained episode length unless clamped/reset; and the reward is essentially unused.
Questions for authors
- What is the rationale for quadratic polynomial fit for the stateless rules, but cubic for stateful, and moreover using L1 for stateless but not for stateful? Even in the cubic expansion there are sufficiently few terms that make fitting straightforward, and would obviate the need for an ad-hoc appending of an Oja term to the quadratic approximation.
- Relatedly, why fit the polynomial to the learned rule instead of parameterizing the rule with a polynomial during meta-learning? Including the meta-learned polynomial as a baseline could strengthen the performance comparison in Fig 4.
- How much of a role does randomizing P per-episode play? Is this really necessary, or can the task network operate directly on inputs, or with a fixed P (and how would that affect generalization)?
- How important is it to train on the combination of pMNIST and RHM for generalization vs within-family performance?
3 comments
Response to reviewer P9PV: Question 4
Question 4. To answer this question, we meta-trained rules on pMNIST alone and on RHM alone. The results can be found in Appendix G.5 (Figure 27). The combination makes little difference within a task family, but does appear to matter for generalization across task families. For example, training only on RHM appears to prevent the rule from generalizing to Fashion-MNIST and Omniglot.
Thank you to the authors for the additional experiments and clarifications. I have raised my soundness score from 2 to 3.
Response to reviewer P9PV: Weaknesses and questions 1 to 3
Thank you for the careful review and helpful suggestions.
Weakness 1. Thank you, we agree that these are useful comparisons. We meta-learned the learning rates (and additional parameters for BCM) of the classical rules.
We did this in two ways, i) initializing the hidden rates at a small value, 10−6, and ii) initializing at the optimum of the grid search. Please see Appendix G.1 (Figure 19) for the performance of each rule.
Meta-learning the learning rates did not substantially improve baseline performance found by grid search.
For both initialization cases, we find that the comparisons agree with the grid-searched versions. One exception is feedback alignment on RHM, which performed worse when initialized at small hidden rates, but performed nearly as well when initialized at the grid search optimum.
We agree that rules without plasticity states should be added to Figure 3A–D. We have done this in the updated manuscript.
Weakness 2. We agree with both observations. It was an outcome we did not expect. The plasticity states appear to mostly keep time within the episode, the rule barely uses the reward, and the learned rule is largely unsupervised in the hidden layers. We have performed additional studies on the effect of the reward. In line with MLSP's relative reward insensitivity we find that a polynomial rule meta-learned without the reward performs, relative to the readout-only baseline, about as well as a polynomial rule meta-learned with the reward (Appendix G.3, Figure 24).
We agree that learning beyond the trained episode length is a real limitation of this work, as mentioned in the main paper Discussion. For future work, we would be interested to use much longer training windows or truncated BPTT to extend the learning window beyond the rapid online learning setting.
Question 1. Our initial motivation was to use the simplest interpretable fit that retained much of the rule's performance. In the case of the stateful rule, we found that a quadratic-plus-Oja and a cubic fit were insufficient, which is why we considered a modulated cubic fit (Appendix D.4).
We agree that for consistency the same cubic fit should be applied to both. We fitted a full cubic polynomial (all 20 terms of degree at most three in presynaptic activation, postsynaptic activation and weight, Oja's term among them) to each rule (i.e. random seed initialization) of both the stateful and stateless MLSP by least squares, with no penalty and no added term.
For rules without plasticity states, the cubic explains 73% and 76% of the held-out variance of the weight updates of W(1) and W(2) (mean of ten rules), against 68% and 70% for the quadratic-plus-Oja fit. We have added the details of this fit to Appendix G.2 (Figures 20 and 21).
Crucially for the conclusions of the paper, in neither case do we see the fitted polynomial reduce to any known classical rule.
Following questions from other reviewers, we investigate a task setting similar to Confavreux et al. (2020), where the task objective is extracting the first principal component. We find that the rule meta-trained with an MLP basis is close to Oja's rule when we fit a polynomial to it (Appendix G.10, Figure 32).
Question 2. This is a very good suggestion, and we have attempted to fulfill it within the rebuttal timeframe.
We meta-learned with a quadratic and a cubic polynomial basis and have included the results in Appendix G.3 (Figure 23).
After 100,000 meta-training episodes, both gain less over readout-only learning than MLSP using an MLP basis (with or without plasticity states) on all four task families.
Question 3. This is a good point. Because of the choice of task families, randomizing P is not necessary. If we were to meta-train with a task whose input distribution the rule could exploit, a fixed P may hurt generalization, but in this case it was additional randomization that we don’t expect to make much difference.
We test this by running additional seeds with rules meta-trained with one fixed P, and find they are as accurate as rules meta-trained with a new P every episode, whether tested with their own P or a new one. The results have been added to Appendix G.4 (Figure 25). As suspected, for the task families initially tested, redrawing P each episode did not make a difference (regardless of whether P is redrawn during evaluation). The same holds for the XOR task, which was added at the suggestion of another reviewer. Rules meta-trained for a network without P, in which W(1) receives the 784 inputs directly, gain less over readout-only learning (Appendix G.4, Figure 26).