- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem that provides the long-time wrong supervision through persistent replay in contaminated continual learning. The core idea is to split the current-task participation and persistent memory admission. The proposed method VAULT routes the sample by calculating the hierarchical evidence and stacked zentropy, and certificate the sample that will be saved into persistent memory via classifier and validation-prototype agreement. Then, it preserves the knowledge of certified memory from feature/logit/relation preservation and gradient projection.
Critical strengths
S1. Nice Problem setting to split the participation and persistent admission. The sample that is worth to train the current task and the sample that is reliable to replay iteratively in the future works are not equivalent. VAULT addresses explicitly those difference.
S2. Empirical results are promising in small-memory regime. VAULT shows the FAA 59.28% and memory purity 98.66% under 5 tasks, 50% contamination, and cap 500 constraint on CIFAR-100.
S3. Evaluation of memory quality is well-aligned with research purpose. The paper analyzes not only nominal capacity but also actual occupancy, purity, and effective clean capacity. It validates the motivation that acquiring the reliable replay supervision is more important than naively filling in the memory.
Critical weaknesses
W1. Contribution of validation-prototype supervision. VAULT leverages labeled validation prototype on not only checkpoint selection, but also memory certification. Hence, it is uncertain that how much portion of the entire improvement can get the admissions mechanism.
W2. Empty-memory case of HE/SZ ablation hides the effectiveness of core mechanism. Empty replay memory (i.e., Acutal=0) can be occurred under w/o HE, w/o SZ, or etc, thereby those FAA drop is caused from two things; (1) Hierarchical evidence or the effect of changing stacked zentropy. (2) The diminished effect of persistent replay. The current results show that the corresponding component needs to generate the memory under the previous threshold setup, but it is hard to validate the better trust estimator under the identical replay budget.
W3. Lack of contribution attribution compared to framework complexity. VAULT combines with lots of things, but those might be reasonable. The paper should split the effectiveness of each component to show the necessity, but current ablation studies are not enough. In particular, comparison by changing the admission criterion is needed while keeping the identical backbone, bootstrap, validation access, and replay objectiveness.
W4. Threshold selection and sensitivity. In routing and certification, there are several thresholds, and those can affect to memory occupancy and purity directly. However, those analyses are not included in this manuscript.
W5. Clarification of scope. All the experiments are conducted on the Task incremental learning setup, but in the initial part of the paper including abstract and introduction, there is no mention about task incremental learning. Hence, I expect that this paper can cover the class incremental learning while reading introduction section. Moreover, I'm curious that why the proposed method did not experiment under the class incremental learning. To the best of my understanding, the proposed method can compare with the other baselines (in fact those baselines comes from class incremental learning setup) under the CIL setup.
W6. Buffer size. The benefit of VAULT is focused on cap 500. As the cap increases, the performance of VAULT is saturated, while the performance of EARL beats VAULT. I think purity-occupancy trade-off make the limits to potential improvement as increasing the buffer. It would be nice approach when the authors discuss more deeply those kind of topics in the manuscript.
W7. Real noise scenario. There is a paper in terms of real noise CIFAR-100 that humans annotated [1]. Those result will validate the effectiveness of the proposed method can be worked in real scenario.
[1] Jiaheng Wei, et al. "Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations", ICLR 2022.
W8. Statistics. Table 1--3 shows only average performance, but in noisy environment, standard deviation is also very important metric to show the robustness of the method. Please report the standard deviation, and if the number is negligibly small, it validates the robustness.
W9. Fairness between the proposed method and baselines. Most of the baselines use the single backbone (RN50), but the proposed method uses two: semantic encoder and plastic encoder. Therefore, for fair comparison, the authors should match the parameter size.
Questions for authors
See the weaknesses. Here are the minor comments below.
Q1. Why are most of the figures and tables located in the middle of the page? Typically, most of the figures and tables in the paper should be located at the top of the paper.
Q2. In contamination procedure, there are two types to make noisy labels: (1) Symmetric noise (2) Asymmetric noise (Please see [1]). Could you show the results of your method under those environments?
Q3. I think Table 6 should be located in the main script, not Appendix. It would be good evidence to show the effectiveness of VAULT in various datasets (especially large-scale dataset). Moreover, please discuss the reason of low performance in WebVision.
[1] ihwan Bang, Hyunseo Koh, Seulki Park, Hwanjun Song, Jung-Woo Ha, and Jonghyun Choi. Online continual learning on a contaminated data stream with blurry task boundaries. CVPR 2022.
I thank the authors for their detailed responses and clarifications. I think that most of my main concerns are resolved. Therefore, I will increase my overall evaluation.