- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes conformal machine unlearning, a framework that defines successful unlearning through the behavior of conformal prediction sets rather than similarity to a model retrained from scratch. The authors introduce EFACE (Excluding Forgotten data by Adjusting CoveragE), an optimization-based method designed to increase miscoverage on the forget set while preserving coverage on retained samples. The method is evaluated on ImageNet100, CIFAR100, and 20NewsGroups, considering both class-wise and cluster-wise forgetting scenarios.
Critical strengths
The paper's motivation that a drop in top-1 accuracy may not be sufficient evidence of forgetting, and that a model can predict the wrong label while still including the true label in a conformal prediction set, seems interesting. The paper derives a connection between and in the -conformal unlearning algorithm through Lemma 2 under the mentioned assumption. The evaluation includes several established unlearning baselines, in-sample and out-of-sample unlearning, class-wise and cluster-wise forgetting, and multiple datasets.
Critical weaknesses
(1) Due to the strong assumption noted in Line 149, the proposed algorithm cannot handle random forgetting scenarios and relies heavily on access to the forget set. (2) The cluster-wise forgetting setting lacks sufficient justification as to why it serves as a practical primary evaluation scenario. (3) Although the paper introduces novel unlearning metrics, the authors must still demonstrate how conformal machine unlearning performs under standard metrics, including forget accuracy (FA), retain accuracy (RA), membership inference attack (MIA), and test accuracy (TA). The limited MIA results presented in Table 7 fail to establish the superiority of the proposed approach. Furthermore, the newly defined metrics in the paper rely on maximum prediction-set sizes (the thresholds c for retained data and d for forgotten data). Thus, results obtained using these new metrics may appear strong without demonstrating that the desired behavior holds across most samples. (4) The authors should clarify the sensitivity of their theoretical guarantees and empirical results when distribution shift occurs between the calibration set and the forget cohort. Additionally, the authors should explicitly quantify the coverage gap across all evaluated settings. (5) Deferring the entire Related Work section to the appendix is bad practice and makes it difficult to assess the paper's novelty relative to existing work.
Questions for authors
Please see the weaknesses.
4 comments
Official Comment by Reviewer dAFY
I thank the authors for the provided responses. Some of my concerns still remain.
- I understand the motivation behind unlearning and untraining, and I believe this is an interesting area of research. However, the paper is written in a way that makes its motivation and novelty relative to existing work unclear. The Related Work section (Section 6) does not read like a proper literature review. Additionally, several recent works, such as [1], have not been discussed in the paper.
[1] E. Triantafillou, A. I. Humayun, M. Ribero, A. M. Turner, M. C. Mozer, G. Kaissis, "Is your algorithm unlearning or untraining?" arXiv preprint arXiv:2604.07962, Apr. 2026.
-
I thank the authors for adding the accuracy and MIA results to some of the tables. However, unlike the metrics you have defined, accuracy and MIA are meaningful evaluation metrics when their results are compared against those of the gold-standard method, namely, retraining from scratch. However, the corresponding values for the RT method are missing from the tables.
-
My question regarding the newly defined metrics in the paper, which rely on maximum prediction-set sizes (the thresholds (c) for retained data and (d) for forgotten data), remains unanswered. In particular, the results obtained using these metrics may appear strong without demonstrating that the desired behavior holds across most samples.
Follow-up discussion
We thank the reviewer for their engagement with our rebuttal and their further constructive comments.
1. The related work section (Section 6) and discussing Triantafillou et al. (2026).
We respectfully note that the related work section (now titled Comparison with Closely Related Methods to avoid confusion) is dedicated to the most relevant work of Shi, Liu, and Wang (2025), which discusses conformal objectives in unlearning. We contrast our work with theirs and present a theoretical, algorithmic, and empirical critique of theirs. We further refer to residual knowledge effects and to the use of external data to induce unlearning, which bear similarities to EFACE. The literature survey, on the other hand, is part of our introduction where we categorise unlearning methods and explain that they target the untraining objective. For that matter, the naming 'untraining' vs 'unlearning' is due to Triantafillou et al. (2026), which we cite twice in our introduction. Following the reviewer's comment, we now contrast our work with Triantafillou et al. (2026) in the introduction (highlighted in blue, lines 93-98): they also separate unlearning from untraining, but they sketch both notions relative to a retrained model and leave the formal definition and the evaluation of unlearning open, whereas we define the unlearning of a population at the output of the model, with no retrained reference. We are happy to discuss and include any particular relevant work that the reviewer believes would improve the positioning of our work.
2. RT in accuracy and MIA.
Kindly note that we already include the accuracy of RT as well as the certified method PABI and now also report MIA corresponding to all image experiments in Tables 3 (main text), 15, 16, 17, and 18 (all in Appendix H.2.6), which separate our comparison with RT from the comparison with the other baselines. We comment on these results in Section 5.1.2. Nonetheless, we agree with the reviewer that it is useful to consolidate the results with RT along with the other baselines, and we now add those results to the tables of accuracy and MIA. We appreciate the opportunity to clarify that, as explained in our introduction, the case of unlearning populations of data with shared characteristics requires expanding the evaluation approaches, and hence the aim of EFACE is not to get as close as possible to RT in terms of accuracy. This argument is at the core of the discussion our work presents.
3. Reliance on maximum prediction-set sizes in the proposed metrics
This is an important point, and we would like to clarify it. The reviewer is right that CF and mCF are conditional on the set size, so they could appear strong if many samples had sets larger than c or d. Our work addresses this in three ways. First, at c = d = , CF and mCF reduce to the standard coverage and miscoverage over all samples (Section 4.1). Figures 5, 6, and 7 (Appendix H.2.4) report for all methods as c and d vary up to , where EFACE remains stable. Second, Corollary 1 (Section 4.1) makes this dependence explicit: the bounds on CF and mCF degrade exactly by the probability that the set size exceeds c or d, so the guarantees account for the excluded samples. Third, smaller values of c and d expose failure modes rather than hide them, such as the collapse of at c = d = 50 in Table 2 (lines 414-417), which is why we recommend evaluating at several levels of c and d (Appendix J.3). Figure 13 (Appendix J) further shows that the average set sizes of EFACE stay small on the retained data and grow mainly on the forget data, while remaining below the evaluated level c = d = 50.
We thank the reviewer again for their time and effort, and we believe the changes stemming from this discussion meaningfully improve our submission. We hope these comments address the raised questions.
Official Comment by Reviewer dAFY
I thank the authors for their responses and the newly added modifications. I have increased my score to 3 (Weak Accept).
Follow-up on our responses
We thank the reviewer again for the thoughtful feedback and questions. We appreciate that the reviewer found our motivation interesting: a drop in top-1 accuracy may not be sufficient evidence of forgetting, because the true label can remain in the conformal prediction set. We also appreciate the reviewer's recognition of our evaluation across several baselines, datasets, and forgetting scenarios.
The reviewer noted only "minor or fixable concerns about soundness", and we addressed each of them in our response and the revision:
These changes improved the paper and its positioning. We would appreciate it if the reviewer could confirm that our responses resolve these concerns, or tell us what remains and we will be happy to address it.