acceptodds
Under review as a conference paper at ICLR 2027

Selective Machine Unlearning for Vision Transformers via Attention-Guided Contrastive Learning

Abstract

Machine unlearning enables pre-trained models to forget specific data in compliance with privacy regulations. While approximate unlearning methods work well for class-wise forgetting, random data forgetting, which involves removing individual samples while retaining others in the same class, remains highly challenging, especially for Vision Transformers (ViTs). Through selective masking experiments, we discover that masking a small proportion of the highest-attention patches preserves ViT's recognition capability while significantly degrading its memorization. Building on this insight, we propose Lethe, a contrastive unlearning method tailored for ViTs that uses masked images as positive logits and original images as negative logits, guiding the model to forget sample-specific details while retaining category-level outlines. Experimental results on multiple benchmarks and ViT architectures show that Lethe surpasses existing approximate unlearning methods, achieving the closest gap to the retain-only Retrain reference.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Machine Unlearning, Vision Transformers

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_K77LOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Lethe, a selective machine unlearning method for Vision Transformers that aims to forget specific samples while preserving knowledge of their classes. The work is based on the observation that masking highly attended image patches can reduce memorization signals without significantly affecting classification accuracy. Building on this, they propose a contrastive learning objective that pushes the model’s predictions on forgotten images toward those obtained from attention-masked versions of the original version and away from the complete original predictions, followed by fine-tuning on retained data. They show across multiple datasets and ViT architectures that Lethe generally achieves a closer approximation to a retrained model than existing approximate unlearning methods.

Critical strengths
  • Simple and intuitive approach: The use of attention for unlearning is interesting, and the proposed method is conceptually easy to follow (provided all the experiments are clarified and complemented -- see weaknesses).

  • Strong empirical performance: The method outperforms the compared baselines in most experimental settings.

  • Extensive evaluation: The authors evaluate the method across five datasets, eight ViT architectures, and multiple forgetting ratios.

  • Useful ablations: I appreciate the ablation studies on masking strategies, masking ratios, contrastive targets, and training epochs.

Critical weaknesses
  • MIA evaluation

The MIA evaluation is quite limited and relies on the relatively simple protocol used in SalUn. I would like to see stronger attacks, such as LiRA or RMIA, to better assess whether the forgotten samples can still be identified as training members. I am also not fully convinced by the interpretation of the MIA results. The reduction in MIA is observed on an already retrained model after applying input masking. Could this simply reflect a distribution shift in the attack's input features rather than reduced memorization? I would appreciate some discussion of this.

More generally, the method provides no formal guarantees of forgetting. The authors should be clearer that the evidence for removing memorized information is empirical, rather than suggesting that memorization has actually been erased.

  • Swin-like architectures

The attention computation for Swin-T and Swin-S is unclear, the proposed solution relies on CLS-token attention, but standard Swin Transformers do not have a CLS token. How are the attention scores and masked positive targets computed for these architectures?

  • Theoretical analysis

The theorem is essentially a standard gradient convergence result. It says nothing about whether minimizing the proposed loss actually leads to successful unlearning, nor does it cover the complete two-stage training procedure. I think the authors should frame this result more carefully.

In the text, mutual information and KL divergence are treated as equivalent, although they measure different quantities. This should be corrected.

  • Contrastive unlearning and motivation

I find the results in Appendix A.14, Table 11, particularly concerning. The contrastive stage alone reduces classification performance to almost chance level, despite the claim that it preserves class-level knowledge while suppressing instance-specific information. This seems to contradict the main motivation and deserves a better explanation.

I would like to see an additional ablation comparing Lethe against a simple perturbation + fine-tuning or gradient ascent + fine-tuning baseline. This would help determine whether the contrastive objective is actually responsible for selective forgetting, or whether the method mainly disrupts the model before recovering performance through fine-tuning.

Several important baselines are missing. In particular, Contrastive Unlearning is cited but not compared against, even though it includes experiments on ViTs. CoUn (Khalil, Y., et al. (2025). Coun: Empowering machine unlearning via contrastive learning. In NeurIPS) is also missing. Given the proposed contribution, I find the absence of contrastive-unlearning baselines a significant weakness.

Other relevant approaches, such as the distillation-based SCRUB and adversarial unlearning method AMUN (Ebrahimpour-Boroojeny, A., et al. (2025). Not all wrong is bad: Using adversarial examples for unlearning. In ICML.), are also missing from the comparisons. I would additionally appreciate a discussion of more recent work such as MUNKEY (Laguna, S. et al. (2026) Models Designed to Forget: Machine Unlearning via Key Deletion. In Neurips), which also explores separating generalizable knowledge from sample-specific memorization.

  • General concerns

Overall, I am not fully convinced that the experiments support all the claims made in the paper. Lethe performs well in many settings, but it is not always the best method. I think the authors should be more careful when claiming consistent superiority.

The paper assumes that highly attended patches contain instance-specific memorized information. However, the experiments only show that masking these patches works better than random masking. I would frame the proposed connection between attention and memorization as a hypothesis rather than an established mechanism.

I would appreciate a brief discussion of whether the approach could extend to non-Transformer architectures, such as ResNets.

It would also be useful to understand whether the approach works for class-wise unlearning. The current evaluation only considers random sample forgetting, which limits the conclusions about its applicability to other unlearning settings.

  • Numerical inconsistencies

I noticed several inconsistencies in Table 2 (page 7). For example, the FA and MIA gaps for ViT-S are incorrectly calculated, affecting the reported Average Gap. There is also an incorrect RA gap for ViT-T. I encourage the authors to double-check all tables, including those in the appendix, as these errors make it difficult to trust the reported comparisons.

  • Reproducibility and presentation

The reproducibility statement incorrectly refers to Appendix A.4 for implementation details and hyperparameters, whereas these are provided in Appendix A.2. Please check the appendix references throughout the manuscript.

I could not find a link to the code. Releasing the implementation would help reproducibility.

Minor: The figures would look more consistent with the rest of the paper if their fonts matched the manuscript (e.g., LaTeX or Times New Roman).

Questions for authors

Please tackle every bullet point in the weaknesses section: Stronger MIA attack results and clarifying the interpretation, explain how the Swin-like architectures are handled, reframe the theoretical statements, explanation and ablation of the effect of the contrastive stage, comparing against the two mentioned contrastive unlearning baselines, extending the related work to the mention missing references, clarify the conclusions supported by the results without "overclaiming", fixing numerical inconsistencies or references,

10 comments

Response and Clarification for Weakness 1(1/2):

Thank you for the suggestion. To strengthen the MIA evaluation, we have added RMIA evaluation on DeiT-T / Tiny-ImageNet with a 10% forgetting ratio, further examining whether forgotten samples can still be identified as training members after unlearning.

As shown below, Lethe achieves an RMIA AUC of 0.5145, close to the chance-level value of 0.5. At the reported operating points with FPR ≤ 1%, Lethe achieves a TPR of 2.24%, the lowest among the evaluated approximate unlearning methods and the closest to Retrain’s 1.61%, with a gap of 0.63 percentage points. These results provide additional empirical support for unlearning effectiveness under the evaluated setting using a stronger membership inference attack.

MethodFA (%)RA (%)TA (%)RMIA AUCTPR at FPR ≤ 1%Achieved FPR
Retrain76.5391.6576.920.49261.61%1.00%
FT86.55 (10.02)96.48 (4.83)75.76 (1.16)0.61184.18%1.00%
GA93.40 (16.87)93.27 (1.62)77.46 (0.54)0.65536.68%1.00%
IU90.64 (14.11)91.58 (0.07)75.70 (1.22)0.64685.90%0.95%
RL85.26 (8.73)95.79 (4.14)76.24 (0.68)0.47623.68%0.95%
ℓ₁-Sparse86.55 (10.02)96.50 (4.85)75.90 (1.02)0.61304.17%0.95%
SalUn84.05 (7.52)95.18 (3.53)75.66 (1.26)0.48033.36%0.95%
Q-MUL83.17 (6.64)94.78 (3.13)76.36 (0.56)0.47873.01%0.95%
Lethe80.90 (4.37)94.09 (2.44)75.04 (1.88)0.51452.24%1.00%

Parentheses in the FA, RA, and TA columns indicate absolute differences from Retrain.

Response and Clarification for Weakness 1(2/2):

Regarding distribution changes induced by masking, we distinguish two related factors: the train–test mismatch arising when an attacker is trained on clean inputs but evaluated on masked inputs, and the effect of masking itself on model outputs and attack-feature distributions. To examine the former, we have added controlled experiments under both Original and Retrain on DeiT-T / CIFAR-100 with a 10% forgetting ratio, using the same 5% attention-guided masking strategy across all masked conditions.

Membership is determined by each model’s actual downstream training history: for Original, both retain and forget samples are members, while independent held-out samples are non-members; for Retrain, retain samples are members, while forget and independent held-out samples are non-members. We evaluate Clean → Clean, Clean → Masked, and Masked → Masked conditions. In the Masked → Masked condition, the SVM attacker is retrained and evaluated on masked data.

ModelAttack Training → EvaluationTA (%)Retain-set MIA (%)Forget-set MIA (%)Holdout MIA (%)
OriginalClean → Clean82.1028.0027.1918.16
OriginalClean → Masked82.0211.5012.3618.25
OriginalMasked → Masked82.0215.2014.0118.81
RetrainClean → Clean81.2424.4916.8017.47
RetrainClean → Masked81.2510.1616.1017.91
RetrainMasked → Masked81.2513.2015.6217.69

Comparing matched attack-training and evaluation conditions, the MIA values for Original’s retain and forget sets, both consisting of training members, decrease from 28.00% to 15.20% and from 27.19% to 14.01%, respectively. For Retrain, the retain-set MIA decreases from 24.49% to 13.20%. By comparison, the corresponding metrics for non-members change relatively little, and classification accuracy remains nearly unchanged. These results show that the decline in training-member detection persists after the attacker adapts to masked inputs and is not confined to the train–test mismatch in the Clean → Masked condition.

At the same time, masking itself changes the inputs and prediction features derived from model outputs, and retraining the attacker does not eliminate these changes. Accordingly, these experiments with fixed model parameters characterize the effect of input masking on membership inference. To further examine whether forgotten samples remain identifiable as training members after Lethe updates the model parameters, we evaluated the resulting unlearned models using RMIA. At the reported operating points with FPR ≤ 1%, Lethe’s member-detection rate is 2.24%, close to Retrain’s 1.61% and lower than those of the other evaluated approximate unlearning methods. This indicates that, under the evaluated RMIA attack, membership exposure for Lethe’s forgotten samples is close to that of Retrain, which did not use these samples during downstream training.

Regarding the nature of the evidence for forgetting, we will explicitly state that our support for unlearning effectiveness is empirical, based on membership inference performance under the evaluated attacks and behavioral proximity to the Retrain reference trained only on the retain set. We will revise statements about memorization removal to reflect the measured results and evaluation conditions, avoiding claims that improved resistance to membership inference establishes complete erasure of memorized information. We will also explicitly describe the attack protocols and define all reported metrics.

Response and Clarification for Weakness 2:

Thank you for this comment. The CLS-token-based formulation in Section 4.1 applies to ViT/DeiT, whereas our Swin-T/S implementation uses an attention aggregation strategy that does not require a CLS token.

Specifically, we extract the attention weights from the last block of the final stage. We compute each spatial token’s importance by summing the attention it receives from all query tokens and averaging across attention heads:

sj=1H∑h=1H∑i=1NAij(h),s_j = \frac{1}{H}\sum_{h=1}^{H}\sum_{i=1}^{N}A^{(h)}_{ij},

where Aij(h)A^{(h)}_{ij} denotes the attention from query token ii to key token jj in head hh. For example, in our Tiny-ImageNet setup, images are resized to 224×224, and the final-stage attention operates on a 7×7 spatial grid within a single unshifted window. We select the ⌊ρN⌋\lfloor\rho N\rfloor highest-scoring positions and zero out the corresponding 32×32 regions in the input tensor.

The subsequent construction of positive targets follows the original method. We will explicitly describe this Swin-specific attention aggregation and spatial masking procedure in the revised manuscript.

Reviewer_8aSvOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the question: "can we decouple a ViT’s recognition capability from its memorization of specific training samples?" This kind of question and statements in the Introduction is very significant in the unlearning field, as balancing generalization and memorization on the specific training samples remain challenging but essential. Specifically, authors focus on the ViT architecture and its attention mechanism. With the attention patch masking experiments, they discover only with the minimal highest-attention patch masking can help the differentiate recognition and memorization on the samples. They propose the "Lethe" unlearning method, consisting of guided masking and contrastive loss.

Critical strengths
  • The motivation and explanation of the main method of this paper is very intuitive and clear.
  • There's no error on the formula, theorem and its prove under the stated assumptions.
  • Although Lethe's framework is somewhat incremental and simple, the authors provided various controlled experiments that support the validity and effectiveness of their approach.
  • The experiments were well structured for various datasets and different models.
Critical weaknesses

Major

  • This paper very focuses on the ViT architecture and attention mechanism, but this makes a curiosity on the generalizability and expansion of this work. Based on the analyses on the attention mechanism, this paper would be better if focusing on the attention mechanism based architecture, not only ViT family.
  • I understand that there's a page limitation of ICLR, but to better emphasize the motivation and novelty of this paper, authors should consider moving the related works section forward from the later part of the paper and appendix.
  • For the larger model, [1] argue that measuring the memorization capability of the model using the MIA score may not reliably reflect memorization in large-scale settings. In this flow, some recent works focus on the relearning [2-4] or its speed. So, how about additionally evaluating it from a relearning perspective?
  • Theorem and proof are neat, but this does not appear to imply memorization removal for a specific sample. It appears to correspond to a mathematical discussion of convergent logic.

Minor

  • As authors argued, class-wise unlearning is sufficiently achieve strong performance, while the sample-wise unlearning is more challenging, but the readers may still wonder like: "Then Lethe is also working on class-wise unlearning not only sample-wise?"
  • For the title of Sec. 4, it might be better to use a more descriptive title rather than only the method name.
  • In Sec. 2 and 5, authors define the Average Gap (AG) to compare the performance of Lethe to other methods, but how about using ToW or ToW-MIA [5]?

[1] Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettlemoyer, L., ... & Hajishirzi, H. (2024). Do membership inference attacks work on large language models?. Conference on Language Modeling (COLM).

[2] Tran, K., & Woo, S. S. (2026). Efficient Unlearning through Maximizing Relearning Convergence Delay. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Findings, 7968–7977.

[3] Ha, S., Park, S. & Yoon, S.W.. (2026). Unlearning’s Blind Spots: Over-Unlearning and Prototypical Relearning Attack. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:39108-39126

[4] Xu, X., Yue, X., Liu, Y., Ye, Q., Zheng, H., Hu, P., Du, M. & Hu, H.. (2026). Unlearning Isn’t Deletion: Investigating Reversibility of Machine Unlearning in LLMs. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:142657-142704

[5] Zhao, K., Kurmanji, M., Barbulescu, G. O., Triantafillou, E., & Triantafillou, P. (2024). What makes unlearning hard and what to do about it. Advances in Neural Information Processing Systems, 37, 12293-12333.

Questions for authors

See the weaknesses.

13 comments

Response and Clarify for Major Weakness 1:

Thank you for the suggestion. We agree that it is important to distinguish Lethe’s architecture-specific motivation from the broader applicability of its unlearning framework. We initially focused on ViTs because their explicit spatial attention maps provide a natural basis for our masking analysis. However, the contrastive unlearning framework can also be adapted to use other spatial importance signals.

To investigate this possibility, we conducted additional sample-wise unlearning experiments with ResNet-18 on CIFAR 100, with a forgetting ratio of 10%. Specifically, we implemented Lethe-CAM, which uses class activation maps instead of ViT attention maps to guide region masking while retaining the original contrastive objective. This experiment examines whether the framework can extend beyond attention-based vision models to a CNN.As shown in the table below, Lethe-CAM achieves the lowest AG of 2.72 among the compared approximate unlearning methods

MethodFARATAMIAAG
Retrain75.6799.9875.3756.600
l1-sparse82.93 (7.26)93.98 (6.00)75.16 (0.21)39.93 (16.67)7.54
SalUn81.02 (5.35)95.94 (4.04)69.96 (5.41)44.18 (12.42)6.81
QMUL78.73 (3.06)97.12 (2.86)72.41 (2.96)47.85 (8.75)4.41
Lethe-CAM79.06 (3.39)97.94 (2.04)74.64 (0.73)51.90 (4.70)2.72

These results provide evidence that Lethe’s guided masking and contrastive unlearning framework can be adapted beyond ViTs. We will include this experiment and its implementation details in the revision and clarify the distinction between our ViT-focused analysis and the framework’s potential applicability to other architectures.

Response for Major Weakness 2:

Thank you for this comment. We will move the Related Work section earlier in the main text and incorporate the most relevant discussions from the appendix to better contextualize our motivation and clarify the novelty of Lethe.

Response and Clarify for Major Weakness 3:

Thank you for this suggestion. We have added relearning experiments on DeiT-T / Tiny-ImageNet with 10% forgetting. All methods undergo supervised relearning on the same 10,000 unmasked forget samples using AdamW, a batch size of 128, and 400 optimization steps. Starting checkpoints are fixed for each method; the reported standard deviations reflect randomness during relearning.

Methodinitial FA50 steps FA100 steps FA400 steps FA
Retrain76.5378.09±0.1879.64±0.2485.44±0.29
RL85.2685.96±0.1787.46±0.3692.89±0.26
GA93.4093.53±0.1494.03±0.0996.27±0.25
FT86.5587.94±0.1988.87±0.1493.66±0.13
IU90.6493.23±0.1493.79±0.0596.14±0.24
l1-Sparse86.5587.90±0.1988.87±0.0993.64±0.13
SalUn84.0586.71±0.0788.01±0.2793.39±0.25
Lethe80.9081.34±0.1882.68±0.2588.56±0.15

FA after 400 relearning steps:

Method1e-53e-51e-4
Retrain81.73±0.1985.44±0.2988.29±0.26
RL89.89±0.2292.89±0.2694.20±0.35
GA95.42±0.1696.27±0.2595.98±0.53
FT90.81±0.3493.66±0.1395.02±0.38
IU95.23±0.1196.14±0.2495.60±0.76
l1-Sparse90.80±0.3393.64±0.1395.06±0.39
SalUn90.33±0.1893.39±0.2594.41±0.77
Lethe84.00±0.0888.56±0.1592.96±0.42

At a learning rate of 3×10⁻⁵, Lethe’s FA remains the closest to Retrain among the compared approximate methods at all reported checkpoints. Lethe also achieves the closest final FA to Retrain across all three evaluated learning rates (1×10⁻⁵, 3×10⁻⁵, and 1×10⁻⁴).

Reviewer_UZbVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Lethe, an attention-guided contrastive method for random-sample unlearning in Vision Transformers. The authors first observe that masking a small fraction of high-attention image patches can preserve classification accuracy while changing membership-inference outcomes. Building on this observation, Lethe aligns the unlearned model’s logits on a forget image with the original model’s logits on its masked counterpart, while repelling the original-image logits. A subsequent retain-set fine-tuning stage is used to recover utility. Experiments across ViT, DeiT, and Swin architectures on several benchmarks report lower average gaps to a retain-only retraining reference than the compared approximate-unlearning methods.

Critical strengths
  1. Relevant problem setting. Random-sample unlearning is more demanding than class-wise forgetting because the model must suppress the influence of selected examples while preserving other instances from the same classes.
  2. Simple and practical framework. The method uses attention-derived masks and a logit-level objective without introducing a large auxiliary network, making the overall pipeline relatively straightforward to implement.
  3. Broad empirical coverage. The paper evaluates multiple Transformer families, datasets, and forgetting ratios, and includes useful comparisons of masking strategies, masking ratios, runtime, and training-stage allocation.
Critical weaknesses
  1. The key motivation does not establish recognition–memorization decoupling. In Section 3, Table 1, the masking experiment is performed using a retain-only Retrain model, which was not trained on the forget set during downstream training. The reported decrease in MIA from 24.49% to 10.16% after 5% masking therefore cannot by itself demonstrate removal of previously learned forget-sample information. It could instead reflect a distribution shift in the queried inputs or changed behavior of the membership attacker. The paper should compare masked and unmasked member/non-member examples under both the original and retrained models and clearly specify the MIA protocol.
  2. The contribution of the contrastive mechanism is not adequately isolated. Lethe combines masked-positive alignment, original-logit repulsion, and extensive retain-set fine-tuning, but lacks positive-only, negative-only, and perturbation-plus-fine-tuning controls. More importantly, Appendix A.14, Table 11 shows that the contrastive-only model reaches just 0.56% test accuracy, whereas 2CL + 8FT recovers 75.04%. This makes it difficult to determine whether Lethe performs selective forgetting or largely disrupts the model before recovering utility through retain-set training. Stage-wise metrics and matched-budget controls are essential.
  3. The attention definition is not consistent with all evaluated architectures. Equation (1) computes patch importance from class-token attention in the last attention layer. Standard Swin Transformers use window-based attention and do not provide the corresponding ViT-style class token, yet Table 2 includes Swin-T and Swin-S results. The paper should explain exactly how importance scores and image-level masks are constructed for Swin, including window-to-image aggregation and hierarchical feature resolutions.
  4. The forgetting evaluation is too dependent on aggregate proximity to Retrain. Average Gap equally averages FA, RA, TA, and MIA, but closeness in these summary statistics does not establish the removal of information from individual forgotten samples. In addition, the meaning of the reported MIA success rate is insufficiently specified, especially when values fall below 50%. Stronger membership-inference evaluation, attacker calibration, and sample-level forget/retain analyses would substantiate the claimed privacy benefit. The five-run analysis in Appendix A.12 is useful but covers only one principal setting.
  5. The methodological contribution and theoretical support remain limited. High-attention masking, logit similarity objectives, and retain-set fine-tuning are individually familiar ingredients; the paper has not yet demonstrated that their specific combination yields a distinct unlearning mechanism rather than a favorable engineering recipe. Theorem 4.2 establishes a standard smooth nonconvex optimization bound for the contrastive loss, but does not imply forgetting or closeness to the retrained model. The claimed ViT-specific insight should be positioned more cautiously against attention-based masking and Transformer unlearning literature.
Questions for authors

See Weakness

7 comments

Response and Clarify for Weakness 1:

Thank you for this comment. We would like to clarify a typo in the dataset name in Section 3: “forget set” should read “retain set” in the description of the MIA evaluation. The reported decrease from 24.49% to 10.16% concerns retained samples subjected to masking, which participated in Retrain’s downstream training. The purpose of this MIA masking experiment is to examine whether masking makes it more difficult for the attacker to determine, from the model’s outputs, whether a sample was used during training. Meanwhile, we use test accuracy to assess whether category recognition performance remains stable, thereby comparing the effects of masking on membership inference and category recognition.

Nevertheless, to investigate changes in the input distribution and attacker behavior, we have added controlled experiments under both Original and Retrain on DeiT-T / CIFAR-100 with a 10% forgetting ratio, using the same 5% attention-guided masking strategy across all masked conditions. Membership is determined according to each model’s actual downstream training history: for Original, both the retain and forget sets are members, while the independent holdout set consists of non-members; for Retrain, the retain set consists of members, while both the forget and independent holdout sets consist of non-members. We evaluate three conditions: Clean → Clean, Clean → Masked, and Masked → Masked. In the Masked → Masked condition, the SVM attacker is retrained on masked data and evaluated under matching masking conditions to examine whether the same trend persists after the attacker adapts to masking.

ModelAttack Training → EvaluationTA (%)Retain MIA (%)Forget-set MIA (%)Holdout MIA (%)
OriginalClean → Clean82.1028.0027.1918.16
OriginalClean → Masked82.0211.5012.3618.25
OriginalMasked → Masked82.0215.2014.0118.81
RetrainClean → Clean81.2424.4916.8017.47
RetrainClean → Masked81.2510.1616.1017.91
RetrainMasked → Masked81.2513.2015.6217.69

As shown in the table, when attack training and evaluation conditions are matched—comparing Clean → Clean with Masked → Masked—the MIA values for Original’s retain and forget sets, both of which contain training members, decrease from 28.00% to 15.20% and from 27.19% to 14.01%, respectively. The non-member holdout result remains relatively stable (18.16% → 18.81%). For Retrain, the MIA value for the member retain set decreases from 24.49% to 13.20%, while the non-member forget and holdout results change comparatively little: 16.80% → 15.62% and 17.47% → 17.69%, respectively. Meanwhile, classification accuracy remains nearly unchanged for both Original (82.10% → 82.02%) and Retrain (81.24% → 81.25%).

These results directly address two concerns. First, the phenomenon occurs not only in Retrain but also in Original, on both retain and forget samples that participated in its training. Second, the decrease in member MIA persists after the attacker adapts to masked inputs, while the non-member results remain relatively stable. Thus, the observations do not reflect a uniform decrease in MIA across all samples and are not confined to mismatched attack-training and evaluation inputs. The specific meaning of “recognition–memorization decoupling” in this experiment is that under the evaluated attack, masking reduces the proportion of training members correctly identified as members while preserving classification accuracy. The experiment holds model parameters fixed and examines the different responses of classification performance and membership-inference performance to input masking.

We will correct the dataset reference in Section 3, incorporate the member/non-member comparisons above, and explicitly clarify the meaning of “recognition–memorization decoupling.” We will also specify the SVM attacker’s input features, training and evaluation splits, membership labels, and the definitions of the reported MIA metrics.

Response and Clarify for Weakness 2:

We would like to clarify the roles of Lethe’s two stages. The CL objective encourages alignment with masked-image logits and separation from original-image logits, but does not explicitly enforce correct class predictions. Without a supervised classification loss, updates to shared parameters can substantially alter classification decisions. The accuracy drop after CL alone is therefore consistent with this intermediate objective. Retain-set fine-tuning introduces explicit classification constraints and is an indispensable component of the complete method.

Table 10 provides complementary evidence for choosing logits as the contrastive target: within the complete pipeline, logit-level CL achieves a lower AG than feature-level CL (2.79 vs. 4.95). This supports the choice of optimization target, while the CL-only results in Table 11 describe the intermediate model before retain-set fine-tuning.

To isolate the components’ contributions, we have added positive-only, negative-only, perturbation + FT, GA + FT, and FT-only controls on DeiT-T / Tiny-ImageNet with a 10% forgetting ratio. Lethe achieves the lowest AG (2.79) among all evaluated variants, supporting the contribution of the complete contrastive objective. We will include these results in the revised manuscript.

DeiT-T / Tiny-ImageNet/10% forgetting:

FARATAMIAAG
retrain76.5391.6576.9240.060
FT-only(10 epochs)86.55(10.02)96.48(4.83)75.76(1.16)30.93(9.13)6.29
FT-only(8epochs)86.91(10.38)95.77(4.12)77.32(0.40)27.78(12.28)6.80
2 Positive-only + 8 FT87.33(10.80)95.50(3.85)77.24(0.32)32.29(7.77)5.69
2 Negative-only + 8 FT83.13(6.60)94.75(3.10)76.38(0.54)35.64(4.42)3.67
Random perturbation + 8FT84.16(7.63)95.42(3.77)76.82(0.10)33.46(6.60)4.53
2GA+8FT88.23(11.70)97.17(5.52)78.06(1.14)42.56(2.50)5.22
Lethe(2CL+8FT)80.90(4.37)94.09(2.44)75.04(1.88)37.60(2.46)2.79

Response for Weakness 3:

Thank you for this comment. Eq. (1) applies to ViT/DeiT, whereas our Swin-T/S implementation uses an attention aggregation strategy that does not require a CLS token.

Specifically, we extract the attention weights from the last block of the final stage. Each spatial token’s importance is computed by summing the attention it receives from all query tokens and averaging across attention heads:

sj=1H∑h=1H∑i=1NAij(h),s_j = \frac{1}{H}\sum_{h=1}^{H}\sum_{i=1}^{N}A^{(h)}_{ij},

where Aij(h)A^{(h)}_{ij} denotes the attention from query token ii to key token jj in head hh.

For example, in our Tiny-ImageNet setup, images are resized to 224×224. With an initial patch size of 4 and successive patch-merging operations, the four stages have spatial resolutions of 56×56, 28×28, 14×14, and 7×7. The final-stage grid lies within a single unshifted window, so no cross-window aggregation or inverse shift is needed in this configuration.

We select the ⌊ρN⌋\lfloor\rho N\rfloor highest-scoring positions from the final 7×7 grid and map them to the corresponding 32×32 regions of the input tensor, reflecting the final stage’s effective spatial stride. These regions are set to zero to construct the image-level mask. Positive targets are then generated following the original method.

We will revise Section 4.1 to distinguish these architecture-specific definitions and explicitly describe the hierarchical resolutions and spatial mask mapping for Swin.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions and Rebuttal

We thank the AC and all reviewers for their constructive feedback. Our work tackles selective machine unlearning in Vision Transformers, aiming to forget individual training samples while preserving performance on other samples from the same classes. We observe that masking a small fraction of high-attention patches preserves classification accuracy while reducing training-member detection under the evaluated attack. This motivates Lethe, an attention-guided contrastive unlearning framework that achieves the smallest Average Gap to retraining among compared methods in most evaluated settings.

During the discussion, we addressed concerns about evaluation reliability, the contribution of the contrastive objective, and generalizability through additional experiments and clarifications. We also refined the theoretical claims to accurately reflect their scope. Following these responses and revisions, all reviewers ultimately provided positive overall assessments of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.