- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Master-and-Stop Training, a data-centric training strategy designed to mitigate the deterioration of OOD detection performance caused by overfitting during the later stages of training. The key idea is to identify samples that have been sufficiently learned and temporarily exclude them from backpropagation. The proposed mastery criterion combines sample-level prediction uncertainty with its second-order temporal difference.
Critical strengths
- This method is direct. It modifies sample contributions during training without requiring additional OOD samples or changing the inference-time scoring function.
- Rather than designing another post-hoc scoring function, this paper investigates how sample-level training dynamics influence the intrinsic OOD detection capability of a classifier.
Critical weaknesses
- Although the experimental results suggest that MST improves OOD detection, it remains unclear whether the improvement specifically comes from identifying mastered samples or from reducing the effective amount of training on easy examples. There should be more ablation study to demonstrate it.
- The most important issue: The benchmark and evaluation paradigm are quite old-school. Please consider more modern setting beyond the classification task, which is extensively studied several years ago. Moreover, the selected classification tasks in your main exp are also quite naive!
- The paper is low-formatting. i. Contribution is short but unclear; ii. the methodology part don't have a clear overview to present your holistic achitecture; iii. The presentation is verbose in introduction.
- The performance looks incremental and trivial, where the performance gain with the help of your method even is negative, c.f., Table 10.
Questions for authors
See weaknesses.