- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the problem under the setting of lifelong and continual learning, and raises a conclusion that does perfect memory can not guarantee reliable prediction. The authors formalize taskwise honest prediction, and prove sharp minimax lower bounds to support their claim. In addition, authors propose a scalable stochastic top-B dual Hedge algorithm to optimize the policy. The proposed methods are compared with other two prediction rules on digit vision benchmark, and various ablation studies are complemented to show its optimal performance.
Critical strengths
-
The authors provide a theoretical investigation into an important but underexplored question in lifelong or continual learning, i.e., whether perfect memory guarantee reliable prediction when the catastrophic forgetting is avoided totally. Existing studies primarily focus on mitigating the CF phenomenon and ignore the prediction performance without the storage limitation.
-
The experiments are comprehensive and provide strong empirical evidence for the theoretical results, demonstrating the effectiveness of the proposed method.
Critical weaknesses
-
The paper relies heavily on mathematical notation, but many of them are not clearly defined or explained when they are first introduced, making some parts of the paper difficult to follow and understand.
-
The theoretical results are derived under the ideal assumptions of perfect memory, which eliminates catastrophic forgetting but may be difficult to achieve in real-world scenarios. Since the number of tasks increases, retaining all historical samples can introduce substantial storage overhead and raise privacy concerns. Therefore, the practical applicability of the theoretical findings is unclear.
-
Each individual sample is regarded as a task in the experiments, which is against the conventional continual learning setting. In CL setting, a group of samples which follow the same data distribution would be regarded as a task. Since the paper focuses on lifelong learning and continual learning, the conventional CL experimental setting should also be considered or added to demonstrate the effectiveness of the proposed method.
Questions for authors
-
Why is there a relationship between perfect memory and the reliable prediction of unobserved tasks? In class-incremental learning, even though the knowledge or samples of previous tasks are retained perfectly, the prediction of unobserved tasks is impossible to be achieved. For example, the class labels of task 1 samples are ranged from 1 to 20, and class labels of task 2 samples are ranged from 21 to 40. How can a model trained solely on task 1 generate correct predictions for task 2 samples?
-
Can the experimental findings be generalized to more complex problems, such as NLP or vison-based benchmark beyond the digit recognition? Can similar results be obtained in models more complex than MLPs? If not evaluated, could you comment on potential challenges in these new experiment settings?