- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MIST, a patch-token-level unlearning method for Vision Transformers that selectively forgets the knowledge of a specific disease from a multi-label chest X-ray classifier. Starting from the observation that existing methods sit at the two extremes of the forgetting–retention trade-off, the method aims to limit collateral degradation by confining the intervention to a sparse part of the token space. In experiments on five diseases from NIH ChestX-ray14, the method succeeds in forgetting the target disease while limiting the accuracy drop on the diseases to be protected.
Critical strengths
The problem formulation, as a machine-learning problem, is clear. The main strength is that the paper addresses a largely unexplored setting: concept-level selective unlearning in medical imaging. In particular, the setting of forgetting only one disease in multi-label chest X-rays, where diseases co-occur, differs from existing work, and the observation that label co-occurrence places forgetting and retention in direct conflict is very reasonable. Rather than preparing a single training/test split, the evaluation is conducted by cross-validation, which makes the experiments highly reliable.
Critical weaknesses
- I was able to understand the technical challenge. However, I cannot understand the clinical significance of wanting to unlearn, in a pinpoint manner, a disease that co-occurs with multiple other diseases. Since the paper builds its argument on medical data, the necessity of solving this problem from a medical standpoint should be stated. Otherwise, the structure of this paper, whose introduction is specialized to the medical setting, has to be regarded as insufficient.
- In unlearning research, a model retrained from scratch without the forget data serves as the reference for forgetting, but this paper does not report it. Because this basic evaluation is not conducted, the effectiveness cannot be measured from multiple perspectives.
- Although the objective function is designed to suppress the representations associated with d_f, the evaluation is conducted on classification results, and standard baseline verifications such as relearning attacks with a small amount of data and membership inference are not performed.
- The parameter settings of the compared methods are fixed to a single configuration, and no comparison over a comprehensive range of parameters is conducted. This means that the discussion is very likely insufficient with respect to compared methods that may be able to adjust the trade-off.
Questions for authors
- Can the authors explain the problem setting from the standpoint of clinical practice? If the problem is not one whose necessity can be sufficiently appreciated, there is a possibility that the dataset used is not appropriate.
- Can the evaluation of forgetting be conducted not only at the classification level but also at the representation level?
- Regarding the evaluation metrics and evaluation methods, is it possible to conduct the evaluation under the standard settings commonly used in ordinary unlearning studies?
- Can experiments be conducted over a broad range of settings for the model parameters? Depending on the responses to these points, the score may go up.