- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents a novel method for federated domain-specific class unlearning, called DeRe. The paper appears to address a non-standard unlearning setting: it assumes that clients belong to different domains while sharing the same set of classification classes. When a class-unlearning request is issued by a client, the global model should be updated so that it no longer assigns samples from that client’s domain to the requested class, while samples of the same class from other clients’ domains should still be correctly classified. Differently from a “traditional” retrained model, in the proposed setting, a forced misclassification for all the samples to forget is expected.
The paper proposes a three-stages mechanism and needs a recovery phase. The authors assume that the clients and server are trusted; part of the unlearning phase requires the requesting client to actively participate.
The proposed method is evaluated against six state-of-the-art baselines which, however, are not designed to solve the same problem described in the paper.
Critical strengths
- The paper defines and addresses what seems a novel federated domain-specific class unlearning setting.
Critical weaknesses
-
In experiments, authors compare to baselines that are designed to approximate the performance or behavior of a "retrained model"; however, if such "traditional" retrained model is not the gold standard baseline, most of the results may appear inflated by optimizing for a different objective (state-of-the-art baselines try to match retrained-model performances which do not represent the best possible within the unlearning setting of the paper, while DeRe is built to optimize what is expected from assumptions). As a result, most of the baselines in main tables (Table 1 and Table 2) report "bad" performances for the metric used as principal axis.
-
Deliberately forcing the model to fail on domain-specific samples may create a very obvious behavioral signature that could itself reveal information about the deletion request or about the forgotten domain. Such forced over-forgetting can become a membership/unlearning signal.
Questions for authors
Clarification requests:
-
The paper, in light of its non-standard unlearning setting, introduces a new metric, ; authors state that they follow standard practice of previous work, but I did not find evidence of this metric being used. Authors then proceed to evaluate the best performing method based on this metric. Can the authors clarify what is standard within existing FU literature and what is newly introduced in their paper?
-
Deliberately forcing the model to fail on domain-specific samples may create a very obvious behavioral signature that could itself reveal information about the deletion request or about the forgotten domain. Such forced over-forgetting can become a membership/unlearning signal. Can the authors discuss on this?
-
Can the authors provide an analysis of the forced missclassification: is this misclassification inflating one specific class or a specific subset of classes, or it redistributes the classification across all the others?
-
From Figure 5, the method seems insensitive to its hyperparameter; I think that this analysis should cover ranges of tuning where it is detrimental for performances. Can the authors show a larger landscape of hyperparameters including tuning that makes the method underperform?
-
No space is dedicated to the assumption of honest clients and server (defined as "trusted server and clients"): the clients have to actively participate to the unlearning process (and have to run a specific local algorithm). This assumption may be defensible, but limits the scope of the proposed solution (e.g., can it be used in regular cross-device FL settings?). Can the authors detail on this or explicitly acknowledge what limitations such assumptions imply?
-
Can the authors report variance for their experiments?
-
Can the authors report the complete detail for reproduction of other baselines results? How have they been tuned?
-
In Section 4.4, the paper defines as the unlearned model after “recovery rounds.” However, the recovery protocol used to obtain the results in Tables 1 and 2 remains unclear from Section 5.2. In particular: (a) is fixed and identical across all baselines, or is a common total communication/computation budget enforced, considering that different methods have different unlearning costs? (b) What criterion is used to stop the recovery phase?
Suggestions for future work:
- Indicate the model architectures used in experiments in the main text and not only in appendix.