- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies identity unlearning in open-set face recognition. Since face recognition models are expected to generalize to unseen identities, forgotten identities may remain verifiable. Verification accuracy therefore cannot reliably distinguish successful unlearning from residual training information. The authors call this the Forgetting-or-Generalization (FOG) ambiguity. The authors introduce Semantic Residual (SR), which uses the noise prediction error of a conditional diffusion model to assess identity information that remains usable in embeddings after unlearning. They also propose FUSE, which combines surrogate representations derived from facial regions, identity cone modeling, and geometric constraints to modify target identity representations while preserving the structure of retained identities. Experiments on four face datasets compare FUSE with several existing methods and suggest a favorable balance between unlearning and retaining model utility.
Critical strengths
- The paper identifies a meaningful evaluation problem in open-set identity unlearning. Face recognition models need to generalize to unseen identities, so continued verification of a forgotten identity does not necessarily indicate failed unlearning. Likewise, a decrease in verification accuracy does not establish that its training influence has been removed. The discussion of FOG clearly explains this limitation and motivates SR as a complementary evaluation measure.
- FUSE shows competitive empirical performance. The reported results indicate that FUSE can reduce semantic residuals for target identities while retaining reasonably strong verification performance on retained and unseen identities.
Critical weaknesses
- SR may be sensitive to changes in the feature representation. The primary auditing generator is adapted to the original feature distribution, so a decrease in SR may also reflect a mismatch between the generator and the modified embeddings. The existing experiments do not fully rule out this explanation, and the inversion attack results are also evaluated primarily using SR.
- The correlation analysis across generators needs clarification. Table 7 does not specify the correlation measure, unit of analysis, or range of data included. It therefore remains unclear whether these correlations support the claim of consistent method rankings.
Questions for authors
- Could you provide a control experiment to distinguish reduced identity information from the generator’s inability to use the modified embeddings? For example, align the new embeddings to the original space using only retained identities, then reevaluate SR and identity recovery.
- What SR data, correlation measure, and unit of analysis were used to compute the coefficients in Table 7?
- Does “classifier-level changes” in Appendix D mean updating only the classifier head, or updating the encoder as well? If the encoder and the conditioning embeddings used for SR remain unchanged, SR would also be expected to remain unchanged under the same auditing settings.
4 comments
Part 2/4
W1 (continued): inversion attack evaluated without SR
We ran a new inversion attack and evaluated it without SR.
Attack. We use an unconditional diffusion model trained on WebFace4M. At each denoising step, we guide it to match the mean embedding of the forgotten identity under the unlearned model. This attack is different from the optimization-based attack in Appendix E.6.1, which we keep.
Evaluation. We embed the generated images with an independent ResNet-101 trained on WebFace4M. This model is not used in training, unlearning, or SR computation. We use 50 forget identities and generate 10 images per identity (500 per method). The gallery has 5,284 CASIA-WebFace identities, each represented by the mean embedding of its real images. A generated image counts as recovered if its nearest gallery identity is the forgotten one and their cosine similarity is at least 0.4. Chance is about 0.02%. The 50 identities are unlearned together in one model per method. This differs from Table 1 and Part 1/4, where a single identity is unlearned.
We also verified that the forgotten identities are not part of WebFace4M. We searched WebFace4M with ArcFace similarity for each of them and manually checked the top-10 matches.
| Method | Identity recovery (%) |
|---|---|
| Real forget images (upper bound) | 93.2 |
| Unguided samples (lower bound) | 0.8 |
| OG | 85.4 |
| Retrain | 32.8 |
| FT | 47.4 |
| GA | 55.0 |
| L1-sparse | 48.6 |
| RL | 52.8 |
| BS | 49.8 |
| BE | 57.2 |
| SalUn | 60.0 |
| SG | 46.2 |
| FUSE (ours) | 40.6 |
- The attack works. On the original model it recovers the identity almost as often as real images do (85.4% vs. 93.2%). Unguided samples stay near zero (0.8%), so the recovery comes from the attacked model and not from the diffusion prior.
- FUSE has the lowest identity recovery among the approximate methods (40.6%). The baselines are between 46.2% and 60.0%. The difference to SG, the closest baseline, is small (5.6 points).
- Retrain is at 32.8%. FUSE covers 85% of the distance from the original model (85.4%) to Retrain.
- Retrain itself is far above the lower bound. This is again the FOG ambiguity: an identity that was never used in training can still be partly recovered.
- The attack agrees with SR on the main result, although the two settings differ (50 identities here, one identity in Table 1): FUSE has the lowest independently judged recovery among the approximate methods here, and the aligned SR closest to Retrain in Part 1/4. The order of the baselines is not identical, so we do not claim a ranking among the baselines.
Revision. This experiment is added to Appendix E.6.2, with a summary in Section 4. We also corrected the text of Appendix E.6.1 to match its table.
Part 3/4
W2 and Q2: the correlation analysis in Table 7
Thank you for pointing this out. The full specification is below. We first clarify the setting. Table 7 was not computed from the values in Table 3. Table 3 uses a single forget identity, as in Table 1. This reflects a typical deletion request. Table 7 comes from a separate D-LORD experiment in which five identities are unlearned together. The revised captions now state this.
- Unit of analysis: the unlearning method (9 methods).
- Data: one and one value per method and generator, listed in the first table below.
- Measure: Pearson correlation between two generators, with and pooled (n = 18).
| Method | IDiff-Face | IDiff-Face | Arc2Face | Arc2Face | Guided Diff. | Guided Diff. |
|---|---|---|---|---|---|---|
| FT | 0.408 | 0.420 | 0.426 | 0.387 | 0.376 | 0.433 |
| GA | 0.428 | 0.358 | 0.428 | 0.351 | 0.376 | 0.410 |
| L1-sparse | 0.501 | 0.457 | 0.489 | 0.394 | 0.455 | 0.381 |
| RL | 0.487 | 0.459 | 0.476 | 0.403 | 0.502 | 0.460 |
| BS | 0.441 | 0.389 | 0.459 | 0.362 | 0.428 | 0.412 |
| BE | 0.522 | 0.375 | 0.510 | 0.402 | 0.590 | 0.370 |
| SalUn | 0.542 | 0.456 | 0.517 | 0.422 | 0.512 | 0.448 |
| SG | 0.499 | 0.334 | 0.510 | 0.358 | 0.478 | 0.384 |
| FUSE (ours) | 0.592 | 0.310 | 0.603 | 0.317 | 0.588 | 0.322 |
Pooling and can inflate the correlation, so we now also report them separately (n = 9 each). Spearman rank correlation is in parentheses.
| Generator pair | Pooled (n = 18) | only | only |
|---|---|---|---|
| IDiff-Face vs. Arc2Face | 0.94 | 0.96 (0.97) | 0.85 (0.82) |
| IDiff-Face vs. Guided Diffusion | 0.87 | 0.90 (0.88) | 0.74 (0.67) |
| Arc2Face vs. Guided Diffusion | 0.84 | 0.87 (0.87) | 0.67 (0.55) |
Agreement is high on the retain set and moderate on the forget set. With a single forget identity (Table 3), the order of the baselines on the forget set varies more between generators (Spearman 0.14 to 0.58), while FUSE remains the lowest under all three.
Revision. In the revised PDF, Table 3 is now Table 4 and Table 7 is now Table 8. Their captions state the setting and the measure, Appendix E.4.4 includes both tables above (Tables 8 and 9), and the claim in Section 4 is reworded.
Part 4/4
Q3: the meaning of "classifier-level changes"
Thank you for the question. The encoder is updated in all baselines, and we have clarified this in Appendix D. For the classification-based baselines (GA, L1-sparse, RL, BS, BE, SalUn, SG), the loss is computed through the classifier head, as in the original methods. The gradients flow through the whole network, so both the head and the encoder are updated. FT also fine-tunes the whole network. For evaluation we drop the head and use only the encoder's embeddings. These are also the embeddings used for SR. You are right that if only the head were updated, SR would stay at 1. This is not what we see. Baseline SR values in Table 1 are between 0.34 and 0.65, and forget-set verification also changes. For example, GA reduces Vf from 0.836 to 0.518 on CASIA-WebFace.
Revision. Appendix D now reads: "For classification-based baselines (GA, -sparse, RL, BS, BE, SalUn, SG), the unlearning objective is computed through the classifier head, while all network parameters, including the encoder, are updated; FT also fine-tunes the full network. Evaluation uses only the updated encoder's embeddings, which are also the embeddings used for SR."
We hope these responses address the reviewer's concerns. We are happy to clarify any remaining points.
Part 1/4
We thank the reviewer for the detailed review and for suggesting a concrete control experiment. We respond to each point below.
W1 and Q1: control for generator mismatch
We agree with the concern. Table 1 already hints at it: Retrain has seen all retained identities, yet its retain residual is 0.650 and not 1. So part of any drop in SR comes from the change in the embedding space. We ran the control you proposed. For each unlearned model, we learn an orthogonal (Procrustes) map from its embedding space to the original space. The map is fitted only on retained identities and never sees forget data. We fit it on 60% of the retained identities and compute the aligned residuals on the remaining 40%. We then recompute SR on the mapped embeddings. We use an orthogonal map because it only rotates the space, so it cannot add or remove identity information.
For identity recovery, we report Rank-1 identification of the aligned forget embeddings against the original prototypes of all 10,000 training identities. This check does not use any generator. As in Table 1, a single identity is unlearned at a time. Rank-1 is averaged over 10 such identities.
Revision. This experiment is added to Section 4 and Appendix E.7. The paper now reads SR relative to the retain residual and to Retrain, and lists the dependence on the generator, and the fact that the control is so far on CASIA-WebFace, as limitations in Appendix F.2.