- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces UNISON, a privacy-oriented machine unlearning technique that targets models that have been trained on poisoned data and removes the global influence of the poisoning attack while keeping performance on the retain set. The authors study this problem in the contained setting of full-information deletion of identified poisoned examples with a known-trigger and restored labels. Their method, UNISON, approaches this problem through balancing objectives of forgetting the trigger and preserving retain set performance. It does this through identifying objective-based salient weights in each layer of the network based on gradient directionality under different loss functions. UNISON is evaluated on a suite of image classification tasks using models from the ResNet family and shows a good balance of performance across retention of performance on the clean data and on privacy metrics.
This paper should be rejected because 1) the setting is not well motivated or realistic, 2) the method has not been well situated within the literature and lacks substantial reference to the broader study of unlearning from poisoned models, 3) I have substantial concerns about the evaluation of this method and do not support a significant contribution.
Critical strengths
- The UNISON mechanism used to balance the objectives is intuitive and well motivated
- The authors have provided experimental code for their implementation, enabling open verification of their results.
- Each claim was well scoped within the text and sentences are easy to read.
Critical weaknesses
- The work is not well situated within the larger literature. This problem greatly coincides with that of Corrective Unlearning (Goel et al. 2024) and broader discussions of poisoning attacks in machine unlearning (Pawelczyk et al. "Machine unlearning fails to remove data poisoning attacks.").
- The work is missing key, and more recent, baselines and discussions of unlearning methods in comparison to UNISON which look at subsets of weights when tackling unlearning (eg. SSD - Foster et al. 2023, RELOAD - Newatia et al. 2026). A discussion setting up UNISON in comparison to other methods which use salient weights would be very helpful to identify the strengths of UNISON.
- The experimental setup covers multiple datasets and does good ablations of the components of the algorithm - but all experiments are performed on ResNet family models which leads me to ask if this method generalises beyond these models
- On reading through the released code, and running some experiments, I observed that in the main UNISON code the model is left in
trainmode - which leads me to question whether the authors performed their metric evaluations in eval mode or not as this can cause BN statistics to drift. - It is not clear to me which evaluation procedure tests/measures how much impact the poisoned data has on the final model. ASR is undefined, but seems to be the main metric for this scenario and it is not clear why MIA is the right way to evaluate model poisoning.
- A relatively milder criticism is that I do not see the justification/motivation for this setting. Corrective unlearning defines a substantially harder problem, whereas the setting outlined by the authors in this work assumes access to full identification of poisoned samples (not necessarily the case), a subset of the retain data, knows the mechanism through which the poisoning attack was conducted, and knows the corrected labels for the poisoned data. Furthermore, access to further held-out sets is assumed. Could the authors please provide a setting in which this is realistic?
Questions for authors
- Could the authors please define the ASR metric? It is not clear whether this comes from prior work or not and how it is formulated
- Could the authors report results with BN statistics frozen during unlearning or confirm whether this was done, to identify whether the main effect on the ASR is from the UNISON updates or from BN drift?
8 comments
Thank you for the updated experiments, implementation investigation, and further details on literature and motivation. This raises a few more questions from me.
Firstly, and I apologise if this is already present in the paper and I have missed it, but have you studied multiple different attacks in conjunction?
Our setting differs from partial-information corrective unlearning because we focus on identified poisoned examples with a known trigger and corrected labels. This allows us to study a specific question: whether membership leakage and learned backdoor behavior can be removed together, rather than treating them as the same forgetting objective.
Could you further elaborate on "whether membership leakage and learned backdoor behavior can be removed together, rather than treating them as the same forgetting objective."?
This ties in to my question about the strong assumptions in the setting you have chosen to study, as I am not convinced by the modifications to Section 2 and 5 about the practicality of the setting. Your updates to Section 5 have stated model maintenance pipelines "in which suspicious training records are identified through data auditing or incident investigation, corrected labels are obtained through verification, and a small trusted dataset is retained for calibration". To me this is unrealistic and does not explain how/when a trigger transformation can be reliably identified.
I think it is important to further evaluate UNISON under partial-information settings closer to corrective unlearning, with stronger ablations to each component of UNISON's setting. In Table 15 the spatial mismatch is not characterised, explained, or quantified. Furthermore an ablation on not identifying all of the poisoned/corrupted data.
Attack Success Rate (ASR)
Thank you for the clarification on this metric. ASR as I understand it is reported as the success rate of attacks from the poisoned data. This raises a few questions, 1) How many attacks were conducted to calculate the percentage? 2) When is an attack considered a success?
Poisoning attacks are not typically a binary measure, and I would encourage the authors to quantify this in a more appropriate manner, unless I have misunderstood how ASR is calculated.
Further clarifications
Thank you for your follow-up questions and the opportunity to improve the work. We have revised the manuscript to clarify our evaluation, assumptions, and ASR calculation.
1. Evaluation with multiple poisoning attacks
We evaluate UNISON against four attacks: BadNets, WaNet, ISSBA, and Narcissus, tested separately across four datasets (Table 2, Table 5). We evaluated them separately to isolate their effects and understand whether UNISON can jointly reduce membership leakage and learned backdoor behaviour across different poisoning attacks. As mentioned in the manuscript, our contribution is to address this limitation of existing methods. For example, AMUN can reduce membership leakage while keeping the backdoor behaviour, and RNP can suppress attack success without removing membership information. Although we have not tested simultaneous attacks, results across four families support UNISON's effectiveness in different settings (Section 4.1). We will include this direction in future work.
2. Why privacy forgetting and backdoor forgetting are different
This distinction is a main motivation for UNISON. Privacy forgetting aims to remove information that could reveal whether a sample was used during training, while backdoor forgetting aims to remove the malicious behaviour learned from poisoned samples.
Our experiments show that improving one does not necessarily improve the other. For example, AMUN achieves a diagnostic membership-inference deviation of 0.025 but retains 74.6% ASR. In contrast, RNP reduces ASR to 4.8% but retains a diagnostic membership deviation of 0.163 (Section 4.2, Table 1). We also use independent likelihood-ratio and learned black-box membership attacks to evaluate privacy forgetting (Section 4.1, Appendix C), rather than relying only on the diagnostic loss-threshold metric.
3. Practicality of the setting and trigger information
Our work focuses on post-incident corrective deletion, where an investigation or data audit has already identified suspicious samples, verified their labels, and provided trigger information. The remaining challenge is removing their influence without retraining the entire model. We understand that obtaining this information may be difficult in practice, especially for unknown triggers. Our goal is not to solve poison detection or trigger reconstruction, but to study joint forgetting once the required information is available (Section 2). We have clarified this setting in Sections 2 and 5. We also evaluate imperfect identification, noisy labels, limited retain data, and inaccurate triggers in Section 4.4 and Appendices B and G.
4. Partial-information experiments and trigger mismatch
We have now clarified this in Section 4.4 and Appendix G.2, Table 15. For the 4 × 4 BadNets trigger on 32 × 32 images, shifts of 1, 2, and 4 pixels correspond to IoUs of 0.60, 0.33, and 0.00. ASR increases from 3.2% to 6.5%, 11.8%, and 22.4%, respectively, while clean accuracy remains between 91.2% and 91.9%. This shows that UNISON can still reduce backdoor success with small trigger inaccuracies.
We also evaluate incomplete poison identification (Appendix B.1, Table 6). At 75%, 50%, and 25% recall, ASR increases to 8.9%, 17.6%, and 39.2%, respectively. Additionally, combining 75% recall with a one-pixel trigger shift gives 91.1% clean accuracy and 13.6% ASR, with LR and BB deviations of 0.036 and 0.048 (Section 4.4, Appendix G.3, Table 16).
5. Clarification of ASR and attack success
We have added the mathematical definition and evaluation procedure in Section 4.1. For each targeted backdoor experiment, we apply the trigger to held-out test images. An attack is counted as successful when the model predicts the attacker's target class. Images already belonging to that class are excluded.
For CIFAR-10, we evaluate 9,000 eligible images per model across five independently trained models, giving 45,000 triggered predictions per attack configuration. We report the mean and standard deviation of ASR across the five models. Thus, ASR measures the percentage of successful triggered predictions, not whether an entire poisoning experiment succeeds.
We also include additional behavioural measures (Section 4.4, Appendix G.7, Table 20). UNISON achieves target-class confidence of 0.13 versus 0.08 for retraining and triggered correct-label accuracy of 87.2% versus 90.5%. Together, these results provide a broader understanding of the remaining backdoor behaviour beyond ASR alone.
We truly appreciate the opportunity to improve the work based on your expert and careful comments. These experiments and clarifications clarify our motivation, evaluation, and scope. Our main contribution is showing why privacy and backdoor forgetting, among others, should be considered together. We hope the revised manuscript addresses your key concerns. We welcome any further suggestions and commit to addressing them.
Additional Baselines, BatchNorm Controls, and Corrective Unlearning Evaluation
Thank you for taking the time to review our paper and inspect the released code. We appreciate the specific concerns you raised. We have revised the paper and added several experiments to address them. We have provided the PDF with the changes and a diff file too.
We agree that the connection to earlier corrective unlearning work needed to be explained more clearly.
We have expanded Section 6 (Related Work) to discuss Corrective Machine Unlearning (Goel et al., 2024), the work of Pawelczyk et al. on residual poisoning effects after unlearning, and parameter-selective approaches such as SSD and RELOAD.
Our setting differs from partial-information corrective unlearning because we focus on identified poisoned examples with a known trigger and corrected labels. This allows us to study a specific question: whether membership leakage and learned backdoor behavior can be removed together, rather than treating them as the same forgetting objective.
Following your suggestion, we have added SSD and RELOAD to the comparison (Section 4.2 and Appendix G).
SSD and RELOAD achieve ASRs of 48.0% and 32.0%, respectively, compared with 3.2% for UNISON. Their diagnostic membership deviations are 0.050 and 0.038, compared with 0.017 for UNISON. Clean accuracy remains above 91% for all three methods.
These results provide further support for jointly addressing privacy and behavioral forgetting. We have also clarified the different information assumptions of the compared methods.
Thank you for pointing out the possible effect of BatchNorm running statistics. We have added two controls to examine this directly, as you suggested.
With BatchNorm statistics frozen during unlearning, ASR changes only slightly from 3.2% to 3.9%, while clean accuracy changes from 91.9% to 91.7%. The LR and BB membership deviations also remain close, changing from 0.021/0.024 to 0.023/0.026.
In comparison, the BN-only control retains 91.3% ASR. This suggests that the main reduction in attack success comes from UNISON's parameter updates rather than BatchNorm statistic changes alone.
We have included these results in Section 4.4 and Appendix G and clarified the evaluation-mode protocol in Appendix F.
We have added a ViT-Tiny experiment on CIFAR-10 with BadNets (Section 4.4 and Appendix G). UNISON achieves 83.9% clean accuracy and 6.2% ASR, with LR and BB membership deviations of 0.032 and 0.038.
Although retraining performs better, these results show that the joint forgetting approach also works with a transformer architecture.
We have clarified the ASR definition and evaluation procedure in Section 4.1, following standard backdoor evaluation practice.
We have also explained why both ASR and MIA are needed. ASR measures whether the learned malicious behavior remains active, while MIA measures whether the forgotten training examples still leave detectable membership information. Since these are different effects of poisoned data, measuring only one would not provide a complete evaluation.
We have expanded Sections 2 and 5 to explain the intended use case more clearly. UNISON targets post-incident model maintenance, where poisoned samples have already been identified and the goal is to remove their privacy and behavioral influence without retraining the model from scratch.
We agree that this does not solve the separate problem of discovering unknown poisoned samples or triggers, and have made this limitation explicit.
Thank you again for the detailed feedback. We believe the additional baselines, BatchNorm controls, transformer evaluation, and clearer positioning have strengthened the paper and addressed the main questions raised by you. We are happy to work on any further suggestions you may have.