- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper investigates whether machine unlearning can reveal which individuals have requested to be removed from a model's training data. The authors introduce Fresh-Identity Indistinguishability (FII), a privacy criterion that measures whether forgotten identities can be distinguished from never-seen identities more easily than in a model retrained from scratch. They develop a detection attack based on embedding similarity statistics and evaluate it under two settings, where the attacker either has access to some training images or only uses previously unseen images. Experiments on Market-1501, CUHK03-NP, and Tiny-ImageNet, using different model architectures, show that forgotten identities can leave detectable traces in the embedding space. The authors also evaluate five unlearning methods across 39 configurations and find that none simultaneously achieves strong forgetting, preserves model utility, and eliminates the detection signal under their evaluation criteria. Overall, the paper highlights an additional privacy risk in machine unlearning that is not captured by conventional forgetting metrics.
Critical strengths
-
Interesting research question: The paper raises an important privacy concern that is often overlooked in machine unlearning: whether the act of requesting data deletion can itself be detected from a released model.
-
New evaluation perspective: The proposed Fresh-Identity Indistinguishability (FII) criterion provides a useful perspective on evaluating unlearning methods beyond conventional forgetting effectiveness and model utility.
-
Well-motivated evaluation baseline: Comparing detection performance against a retrained model is a reasonable choice, as it helps distinguish residual training signals from differences that may exist even without exposure to the forgotten identities.
Critical weaknesses
-
Limited evaluation of unlearning methods: The paper evaluates only four existing unlearning methods and one newly introduced diagnostic baseline. This relatively narrow selection makes it difficult to determine whether the observed privacy leakage is a general limitation of machine unlearning or specific to the methods studied.
-
Unclear interpretation of FII: The paper defines FII using the detection-AUC gap relative to a retrained model, but the retrained model itself can distinguish target identities from never-seen identities. This raises questions about whether the proposed metric fully captures the intended notion of deletion-request privacy.
-
No effective mitigation strategy: While the paper demonstrates that existing unlearning methods can leave detectable traces, it does not propose a practical solution to eliminate these signals while maintaining model utility. As a result, the contribution is primarily diagnostic, with limited guidance on how to improve existing methods.
Questions for authors
Generalizability of the findings: The experiments focus primarily on person re-identification models. What evidence supports the applicability of the proposed privacy concern to other model types or unlearning settings?
4 comments
(2/2)
Q1, generalisability. The released-against-oracle gap replicates in every setting we have: Tiny-ImageNet as a class-level proxy with no cameras and an oracle at chance; ResNet-18 and ConvNeXt-T beside the ResNet-50 (3/3 seeds each under both adversaries); a ViT-B/16 (3/3 under the mixed adversary by every test, 3/3 under the fresh one by the permutation and bootstrap tests, Appendix R); and CUHK03-NP (3/3 under the fresh adversary; its mixed row is saturated for oracle and released model alike, as Table 2 notes). A detector fitted on one architecture or dataset reads the others above their oracle (Table 13; Table 28 for the ViT). The statistics it uses are cosine similarities among the adversary's own images, and the same battery, less its camera statistic, runs on Tiny-ImageNet (Section 4); from that study Appendix D concludes that the signal is not a property of person re-identification, of metric learning with a BNNeck head, or of Market-1501's camera structure. On the unlearning setting, the gap persists when the 100 identities are deleted one or ten at a time instead of in one batch (Appendix S: +0.19 to +0.27 under the mixed adversary at the 100th request, interval excluding zero in 20/20 runs).
Ethics flag. The ethics statement covers the data (public benchmarks, no face recognition, no real erasure request, services simulated over our own galleries) and the dual-use question: the detector needs a labelled opt-out set and is released for reproducibility and defensive auditing. We will answer any further question the AC or the ethics reviewer raises.
Thank you again. If a particular section still reads badly, tell us which and we will fix it while the discussion is open.
Thank you for the detailed rebuttal and clarifications. The responses have adequately addressed my main concerns regarding the evaluation coverage, the interpretation of FII, and the scope of the paper's contributions. I'd like to increase my score.
Thank you for reading the rebuttal and for raising your score. Your comments on the evaluation coverage and on the interpretation of FII shaped the revision, and your note on presentation led to the rewritten abstract and introduction. Any further suggestion is welcome, and we will carry it into the final version.
(1/2)
Thank you for the review, and for being direct about presentation. We took it seriously: the abstract now reports four AUCs rather than a dozen figures, the introduction states the claim in a single paragraph on its first page, and the per-seed detail of the abstract and introduction has moved into the tables. Point by point:
W1, the range of methods. The submission had four published methods and our own diagnostic baseline (SCRUB, one of the four, collapsed at 100 identities and is in Table 3 as such); the revision widens that to ten. It adds SSD (Fisher dampening, no gradient descent), Bad-Teacher, SalUn, random-label fine-tuning, SCRUB at its published lower rates, and a re-implementation of De-ReID, all under the identical protocol on the submitted checkpoints (Appendix Q): ten methods and 72 operating points. Every added method, at the operating point the paper's rule selects, leaks on 5/5 seeds under the mixed adversary, with gaps from +0.09 to +0.23 against the original's +0.25, and so do finetune and NegGrad+. Appendix R repeats the exercise for eight methods on a ViT-B/16: every operating point that keeps retain mAP above 0.95 leaks under the mixed adversary on 3/3 seeds. The leakage is not specific to the original methods.
W2, the interpretation of FII. You are right that the retrained model itself separates the targets from never-seen identities (0.72 against a 0.48 null floor), and that is exactly why FII is defined relative to the oracle rather than to chance. Retraining is the exact-unlearning reference, the thing a deletion request is honoured against, so a criterion that retraining fails cannot be asked of approximate methods; the paired gap charges a method only for what it leaves beyond retraining. Appendix N adds the complementary fact: against retained training identities the released original sits at chance (0.49) while the oracle separates them at 0.96, so what the detector reads is "was trained on", and exact retraining is the one procedure that removes it. Both facts are now on the first page: the FII paragraph of the introduction says why the oracle, not chance, is the reference, and the claim paragraph says what the mark is.
W3, mitigation. The paper tests the zero-cost post-hoc defences on the released embedding (nulling the direction that separates forgotten from never-seen identities, Table 7; Gaussian noise and PCA truncation, Table 16) and one unlearning baseline of our own, artifact_aware. That baseline is the one method whose selected operating point sits at the criterion's margin: its gap is within 0.02 of zero under both adversaries, with 1/5 seeds rejecting under the paired bootstrap (the mixed rejection above the oracle, the fresh one below), and it pays for that in retrieval on never-seen identities, test mAP 0.66 against the original's 0.74 and the oracle's 0.72 (Appendix P). In a single-seed control at 20 identities (Section 9, Table 18), the same procedure applied to never-seen identities instead moves the real requesters' gap by -0.33 without seeing a forget image, so its non-leak is not specific to the opt-out identities. Across the ten methods the gap shrinks only where retrieval has been paid for: the methods that leave forget mAP at the original's level (SSD at alpha >= 5, SCRUB at its published rates, NegGrad+, finetune: forget mAP 0.96 to 0.97 in Appendix Q) leak at +0.22 to +0.25, and the smaller gaps belong to operating points that pay in never-seen retrieval (standard-test mAP 0.63 to 0.68 against the original's 0.74, Appendix Q). The contribution is the criterion and the audit, and the audit carries the guidance the data support: a deletion procedure has to move the opt-out identities to where never-trained ones sit, and FII is the test it should be scored against. Appendix Q is what that scoring looks like for ten methods.