- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes IV-EWC a method to improve curriculum learning. The aim of IV-EWC is to prevent forgetting when learning on subsequent tasks in the curriculum. IV-EWC works by applying EWC with a specific regularization coefficient schedule and a particular way of selecting points to construct the estimate of the fisher information. The authors evaluate performance on 3 datasets and compare to 5 baselines. IV-EWC achieves a marginal accuracy performance improvement.
Critical strengths
-
Looking at problems like forgetting due to the distribution shifting between different parts of the curriculum is an interesting perspective. Though has been explored before.
-
I think the case study is a nice analysis.
Critical weaknesses
-
No mention of previous work on forgetting in curriculum learning is given. There have been many works on this, including theoretical work on convergence speedup , for example Saglietti, Mannelli & Saxe, An Analytical Theory of Curriculum Learning in Teacher-Student Networks NeurIPS 2022
-
Paper claimed main novelty of IV-EWC is using influence functions in the estimation of the fisher information in EWC. But, nowhere in the main paper is this presented.
-
The theory section has many problems:
A) The main claim is that curriculum learning improves convergence speed. This is not tested in the experiments. In fact, as the experiments use the same learning rate per curriculum step it renders the theory empty, in the sense the only reason you would expect an improved convergence speed is if you could use larger learning rates earlier on. This makes the theory and method/experiments seem quite unrelated.
B) The theory section is standard analysis and no references or indication of this is given in the paper. For example, Lemmas 1 and 2 and Theorem 3 are not novel. This means that the theory section could be shortened to a paragraph or two.
C) Theorem 2 proof looks to have an error where eq. 4 should be for a single gradient step where in the proof this is assumed to be over learning the whole curriculum task t.
-
Performance improvement is very marginal, baselines are outdated and no comparison to doing no curriculum learning IV-EWC not tested over different curriculum methods and the curriculum method used in the paper is unclear
Questions for authors
-
Why does figure 1. c) not seem to link up with the results in table 1, am I missing something?
-
Why was the main claim of the paper, using influence functions, not described in the main text nor fully in the appendix?