- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposed a particle filter-based method, WPF, to tackle the forgetting and loss of plasticity caused by order-dependence in continual learning and reinforcement learning. The paper shows theoretical proof that particle-filtering, under some assumptions, achieve approximately permutation invariant result. The developed method is more practical than conventional particle filtering as it combines with gradient descent and able to perform learning in high-dimension. Empirical result show WPF effectiveness in CL and Lifelong RL setting.
Critical strengths
-
Diverse experiment settings, not only cover normal SplitMNIST/CIFAR100, it also covers Lifelong RL with ProcGen. To the best of my knowledge, not much paper cover both CL and Lifelong-RL together and good to see a method that works for both field.
-
Technically solid and well motivated derivation that address the key weakness of particle filter in high-dimensional learning setting. In bayesian update, we sacrifice tractability for a solid belief, where we have a distribution instead of point estimate. Since the posterior is intractable, this method provide a theoretically motivated hybrid to make it computationally feasible.
Critical weaknesses
-
Strong assumption made in Line 167-178 for Eq. 9, where . However, this is unstable when the bound is divergence (for infinite/large ) and there is no ablation that is reasonable for some small, finite (at least matching experiment setup ). Also assumming is small is not always true and lack empirical proof.
-
One classic and easy way to reduce the forgetting due to order of training in both CL and RL is Experience Replay (ER) [1]. ER with small amount of replay sample is known to have very strong performance. It is unclear whether under same additional memory space, whether storing small amount of samples already outperform the need of have N copies of model parameters (N=100 basd on Line 363).
-
Intuitively, under the assumptions made, I still do not see that how WPF helps in breaking the order-dependence in the learning algorithm. Each independently intialized weight are trained with same data sequence and updated with gradient descent and combined in a non-trivial, bayesian motivated weights. Thus, intuitively, I think what happenning is that with a larger N and diverse starting point, different trajectory will learn slightly differently and forget differently, each of them cover the blindspot of one another and the WPF essentially provide an mathematically grounded framework to combine all learned weights and more blindspots are covered. In my opinion, WPF is not practical in CL unless N is very small but I think it has better usefulness in other settings as it provide a grounded and effective way in combining knowledge.
-
No ablation studies on key hyperparameter, especially the number of particles.
Minor:
, and . after each Eq
Nice to see great improvement, with great stability. But it is well known that increasing parameter space reduce forgetting a lot, no direct ablation isolating that factor [2][3].
Baselines compared are rather old (2017 or earlier), but practically, these methods are still extremely strong
Slight notation confusion of using for weight, but not a big deal
[1] Robins, Anthony. "Catastrophic forgetting, rehearsal and pseudorehearsal." Connection Science 7.2 (1995): 123-146.
[2] Dyer, Ethan, Aitor Lewkowycz, and Vinay Ramasesh. "Effect of scale on catastrophic forgetting in neural networks." (2022).
[3] Davari, MohammadReza, et al. "Probing representation forgetting in supervised and unsupervised continual learning." 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022.
Questions for authors
- What's the model utilized in the SplitMNIST/CIFAR100 experiment (apologies if I overlook, but I couldn't find it in the main paper nor the appendix)? Based on supplementary material code, it is two different custom defined CNN for SplitMNIST and CIFAR-100, I believe testing on standard ResNet or ViT will be a better practice.
- Is it possible to provide some preliminary, simple result to show that the assumptions made in Line 167-178 and 198-207 are reasonable?
- I am sorry for asking more experiments but will it be reasonable to compare against base model with same parameter / VRAM budget (e.g. Let N=10 in WPF, then let's say each has 10 parameter, then maybe you should compare against a model train with SGD but with 10$\times$10 number of parameters, i.e. 1M to 10M param) to rule out the possibility that the reduce in forgetting and increase in plasticity is due to increase in parameter space.
- What is the VRAM consumption and computation speed of WPF compared with EWC (that requires roughly N=3)?