- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Lethe, a selective machine unlearning method for Vision Transformers that aims to forget specific samples while preserving knowledge of their classes. The work is based on the observation that masking highly attended image patches can reduce memorization signals without significantly affecting classification accuracy. Building on this, they propose a contrastive learning objective that pushes the model’s predictions on forgotten images toward those obtained from attention-masked versions of the original version and away from the complete original predictions, followed by fine-tuning on retained data. They show across multiple datasets and ViT architectures that Lethe generally achieves a closer approximation to a retrained model than existing approximate unlearning methods.
Critical strengths
-
Simple and intuitive approach: The use of attention for unlearning is interesting, and the proposed method is conceptually easy to follow (provided all the experiments are clarified and complemented -- see weaknesses).
-
Strong empirical performance: The method outperforms the compared baselines in most experimental settings.
-
Extensive evaluation: The authors evaluate the method across five datasets, eight ViT architectures, and multiple forgetting ratios.
-
Useful ablations: I appreciate the ablation studies on masking strategies, masking ratios, contrastive targets, and training epochs.
Critical weaknesses
- MIA evaluation
The MIA evaluation is quite limited and relies on the relatively simple protocol used in SalUn. I would like to see stronger attacks, such as LiRA or RMIA, to better assess whether the forgotten samples can still be identified as training members. I am also not fully convinced by the interpretation of the MIA results. The reduction in MIA is observed on an already retrained model after applying input masking. Could this simply reflect a distribution shift in the attack's input features rather than reduced memorization? I would appreciate some discussion of this.
More generally, the method provides no formal guarantees of forgetting. The authors should be clearer that the evidence for removing memorized information is empirical, rather than suggesting that memorization has actually been erased.
- Swin-like architectures
The attention computation for Swin-T and Swin-S is unclear, the proposed solution relies on CLS-token attention, but standard Swin Transformers do not have a CLS token. How are the attention scores and masked positive targets computed for these architectures?
- Theoretical analysis
The theorem is essentially a standard gradient convergence result. It says nothing about whether minimizing the proposed loss actually leads to successful unlearning, nor does it cover the complete two-stage training procedure. I think the authors should frame this result more carefully.
In the text, mutual information and KL divergence are treated as equivalent, although they measure different quantities. This should be corrected.
- Contrastive unlearning and motivation
I find the results in Appendix A.14, Table 11, particularly concerning. The contrastive stage alone reduces classification performance to almost chance level, despite the claim that it preserves class-level knowledge while suppressing instance-specific information. This seems to contradict the main motivation and deserves a better explanation.
I would like to see an additional ablation comparing Lethe against a simple perturbation + fine-tuning or gradient ascent + fine-tuning baseline. This would help determine whether the contrastive objective is actually responsible for selective forgetting, or whether the method mainly disrupts the model before recovering performance through fine-tuning.
Several important baselines are missing. In particular, Contrastive Unlearning is cited but not compared against, even though it includes experiments on ViTs. CoUn (Khalil, Y., et al. (2025). Coun: Empowering machine unlearning via contrastive learning. In NeurIPS) is also missing. Given the proposed contribution, I find the absence of contrastive-unlearning baselines a significant weakness.
Other relevant approaches, such as the distillation-based SCRUB and adversarial unlearning method AMUN (Ebrahimpour-Boroojeny, A., et al. (2025). Not all wrong is bad: Using adversarial examples for unlearning. In ICML.), are also missing from the comparisons. I would additionally appreciate a discussion of more recent work such as MUNKEY (Laguna, S. et al. (2026) Models Designed to Forget: Machine Unlearning via Key Deletion. In Neurips), which also explores separating generalizable knowledge from sample-specific memorization.
- General concerns
Overall, I am not fully convinced that the experiments support all the claims made in the paper. Lethe performs well in many settings, but it is not always the best method. I think the authors should be more careful when claiming consistent superiority.
The paper assumes that highly attended patches contain instance-specific memorized information. However, the experiments only show that masking these patches works better than random masking. I would frame the proposed connection between attention and memorization as a hypothesis rather than an established mechanism.
I would appreciate a brief discussion of whether the approach could extend to non-Transformer architectures, such as ResNets.
It would also be useful to understand whether the approach works for class-wise unlearning. The current evaluation only considers random sample forgetting, which limits the conclusions about its applicability to other unlearning settings.
- Numerical inconsistencies
I noticed several inconsistencies in Table 2 (page 7). For example, the FA and MIA gaps for ViT-S are incorrectly calculated, affecting the reported Average Gap. There is also an incorrect RA gap for ViT-T. I encourage the authors to double-check all tables, including those in the appendix, as these errors make it difficult to trust the reported comparisons.
- Reproducibility and presentation
The reproducibility statement incorrectly refers to Appendix A.4 for implementation details and hyperparameters, whereas these are provided in Appendix A.2. Please check the appendix references throughout the manuscript.
I could not find a link to the code. Releasing the implementation would help reproducibility.
Minor: The figures would look more consistent with the rest of the paper if their fonts matched the manuscript (e.g., LaTeX or Times New Roman).
Questions for authors
Please tackle every bullet point in the weaknesses section: Stronger MIA attack results and clarifying the interpretation, explain how the Swin-like architectures are handled, reframe the theoretical statements, explanation and ablation of the effect of the contrastive stage, comparing against the two mentioned contrastive unlearning baselines, extending the related work to the mention missing references, clarify the conclusions supported by the results without "overclaiming", fixing numerical inconsistencies or references,
10 comments
Response and Clarification for Weakness 1(2/2):
Regarding distribution changes induced by masking, we distinguish two related factors: the train–test mismatch arising when an attacker is trained on clean inputs but evaluated on masked inputs, and the effect of masking itself on model outputs and attack-feature distributions. To examine the former, we have added controlled experiments under both Original and Retrain on DeiT-T / CIFAR-100 with a 10% forgetting ratio, using the same 5% attention-guided masking strategy across all masked conditions.
Membership is determined by each model’s actual downstream training history: for Original, both retain and forget samples are members, while independent held-out samples are non-members; for Retrain, retain samples are members, while forget and independent held-out samples are non-members. We evaluate Clean → Clean, Clean → Masked, and Masked → Masked conditions. In the Masked → Masked condition, the SVM attacker is retrained and evaluated on masked data.
| Model | Attack Training → Evaluation | TA (%) | Retain-set MIA (%) | Forget-set MIA (%) | Holdout MIA (%) |
|---|---|---|---|---|---|
| Original | Clean → Clean | 82.10 | 28.00 | 27.19 | 18.16 |
| Original | Clean → Masked | 82.02 | 11.50 | 12.36 | 18.25 |
| Original | Masked → Masked | 82.02 | 15.20 | 14.01 | 18.81 |
| Retrain | Clean → Clean | 81.24 | 24.49 | 16.80 | 17.47 |
| Retrain | Clean → Masked | 81.25 | 10.16 | 16.10 | 17.91 |
| Retrain | Masked → Masked | 81.25 | 13.20 | 15.62 | 17.69 |
Comparing matched attack-training and evaluation conditions, the MIA values for Original’s retain and forget sets, both consisting of training members, decrease from 28.00% to 15.20% and from 27.19% to 14.01%, respectively. For Retrain, the retain-set MIA decreases from 24.49% to 13.20%. By comparison, the corresponding metrics for non-members change relatively little, and classification accuracy remains nearly unchanged. These results show that the decline in training-member detection persists after the attacker adapts to masked inputs and is not confined to the train–test mismatch in the Clean → Masked condition.
At the same time, masking itself changes the inputs and prediction features derived from model outputs, and retraining the attacker does not eliminate these changes. Accordingly, these experiments with fixed model parameters characterize the effect of input masking on membership inference. To further examine whether forgotten samples remain identifiable as training members after Lethe updates the model parameters, we evaluated the resulting unlearned models using RMIA. At the reported operating points with FPR ≤ 1%, Lethe’s member-detection rate is 2.24%, close to Retrain’s 1.61% and lower than those of the other evaluated approximate unlearning methods. This indicates that, under the evaluated RMIA attack, membership exposure for Lethe’s forgotten samples is close to that of Retrain, which did not use these samples during downstream training.
Regarding the nature of the evidence for forgetting, we will explicitly state that our support for unlearning effectiveness is empirical, based on membership inference performance under the evaluated attacks and behavioral proximity to the Retrain reference trained only on the retain set. We will revise statements about memorization removal to reflect the measured results and evaluation conditions, avoiding claims that improved resistance to membership inference establishes complete erasure of memorized information. We will also explicitly describe the attack protocols and define all reported metrics.
Response and Clarification for Weakness 2:
Thank you for this comment. The CLS-token-based formulation in Section 4.1 applies to ViT/DeiT, whereas our Swin-T/S implementation uses an attention aggregation strategy that does not require a CLS token.
Specifically, we extract the attention weights from the last block of the final stage. We compute each spatial token’s importance by summing the attention it receives from all query tokens and averaging across attention heads:
where denotes the attention from query token to key token in head . For example, in our Tiny-ImageNet setup, images are resized to 224×224, and the final-stage attention operates on a 7×7 spatial grid within a single unshifted window. We select the highest-scoring positions and zero out the corresponding 32×32 regions in the input tensor.
The subsequent construction of positive targets follows the original method. We will explicitly describe this Swin-specific attention aggregation and spatial masking procedure in the revised manuscript.
Response and Clarification for Weakness 1(1/2):
Thank you for the suggestion. To strengthen the MIA evaluation, we have added RMIA evaluation on DeiT-T / Tiny-ImageNet with a 10% forgetting ratio, further examining whether forgotten samples can still be identified as training members after unlearning.
As shown below, Lethe achieves an RMIA AUC of 0.5145, close to the chance-level value of 0.5. At the reported operating points with FPR ≤ 1%, Lethe achieves a TPR of 2.24%, the lowest among the evaluated approximate unlearning methods and the closest to Retrain’s 1.61%, with a gap of 0.63 percentage points. These results provide additional empirical support for unlearning effectiveness under the evaluated setting using a stronger membership inference attack.
Parentheses in the FA, RA, and TA columns indicate absolute differences from Retrain.