- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper empirically examines various continual-learning metrics (e.g., forgetting) in time series analysis, in 3 datasets, 5 objectives, and 5 models.
Critical strengths
The paper is asking interesting questions, e.g., how does the objective affect forgetting? How much of the damage of forgetting can be fixed?
Critical weaknesses
-
I found the paper very hard to read, starting from the introduction. The main issue for me is that many sentences are difficult to interpret unless you understand the context for them, which appears only later (this context awareness is something the authors can do easily, but not a new reader). For example, Q2 on line 048 reads: "Is event forgetting a single failure mode or do diagnostics split (Q2)?" I do not understand what "failure mode" or "diagnostic split" mean here. As another example, it is hard to understand the setting because the notation is not clear; e.g., in the first displayed equation on line 118, k,K, and t_k are not defined.
-
The results are purely empirical; this is fine, but the authors need to convince the reader they are representative, at least in some specific practical domain. However, the authors do not establish this: the models are rather small, and it is not clear that they have optimized hyperparameters to be competitive within their domain. Therefore, it is hard for me to understand whether the findings reflect properties of the objective or architecture, as claimed, or are specific to the particular models with their hyperparameter configurations.
Questions for authors
Please address the above weaknesses.