- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a confidence-based sample selection framework to accelerate existing machine unlearning methods. It dynamically selects retain and forget examples while preserving the underlying unlearning objectives. The authors provide a motivation and evaluate the framework on vision classication and language models, reducing computational time in several settings.
Critical strengths
-
The framework integrates sample selection into several existing unlearning methods.
-
Several configurations achieve time savings while maintaining similar model utility.
-
The evaluation includes computation trajectories and prediction-distribution diagnostics.
Critical weaknesses
-
Key relevant papers were neither discussed nor compared. RUM [1] groups forget examples by memorization for sequential unlearning, while SULI [2] uses predictive entropy-ranked to order forget subsets for iterative soft relabeling. Neither work is discussed or compared, leaving it unclear what DSS’s dynamic confidence-based selection adds beyond these existing sample scheduling strategies. Bad Teacher [3] has already been updated using a combination of forget data and a subset of retain data.
-
The theoretical comparison relies on retrained-model confidence, whereas DSS selects samples using current confidence during unlearning. The fixed-source results do not establish why this changing score identifies useful subsequent updates in the actual dynamic procedure.
-
The MIA results and setup lack transparency, key implementation details of the MIA have not been disclosed, and the results cannot be used to substantiate the arguments. And, the privacy leakage risks associated with excluding certain forget set from the update process were not analyzed.
[1] RUM: What makes unlearning hard and what to do about it (NeurIPS 2024)
[2] SULI: Fine-Grained and Efficient Self-Unlearning with Layered Iteration (IJCAI 2025)
[3] Bad TeacherCan Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher (AAAI 2023)
Questions for authors
-
What does DSS’s dynamic selection add beyond RUM’s memorization-based grouping and SULI’s uncertainty-based staged unlearning?
-
What evidence validates current confidence for sample selection during unlearning, given that RUM’s confidence proxy is averaged over the original training trajectory?
-
Does reducing the retain computation suffice to achieve similar gains, and what exactly is gained by the additional step of filtering the forget data?
2 comments
Thank you for the revisions and additional experiments. However, my main concerns remain unresolved.
The paper still lacks comparisons with the newly cited methods, zero-shot and few-shot unlearning methods. The motivation and claimed advantages remain unconvincing.
The new ablation in Table 4 only evaluates NegGrad+ and does not support the broader claims across different unlearning methods.
The response to Q2 does not address my concern about the reliability of current confidence, especially given the findings of RUM. The result of table 26 further raises doubts about this selection rule, and the theoretical analysis still does not justify its effectiveness.
Still failed to change a stronger MIA.
Response to Reviewer ViBB
Thank you for your helpful comments. We have revised the manuscript accordingly and address your questions below.
Q1. RUM and SULI organize unlearning through example groups or stages. DSS instead recomputes selection from the current unlearning model and updates participation in both the retain and forget sets. Examples can enter or leave the selected set, while the base method retains its loss formulation, targets and update rules.
RUM groups forget examples for sequential unlearning. SULI combines uncertainty-based ordering with its own soft-label adjustment procedure. DSS separates the selection policy from the base unlearning method and evaluates that policy across six methods. DSS+ further changes the selection policy by keeping admitted examples and rescoring only those not yet selected. The distinction concerns which pools are selected, when selection is updated and how it is coupled to the underlying unlearning procedure.
Section 2 (Data selection for unlearning) now discusses RUM and SULI alongside DSS. Sections 3.2 and 3.3 specify DSS's refresh procedure and DSS+'s persistent selection, respectively.
Q2. We chose current true-label confidence primarily for its simplicity. It is directly available from a model forward pass and does not require storing or averaging scores over the original training trajectory. It also reflects the model that is about to perform the next unlearning updates.
EL2N and GraNd can also be used as scores in the same selection procedure, as evaluated in Figure 4. Confidence and EL2N use closely related information and produce similar results. Confidence therefore serves as a simple default within the selection procedure.
Section 3.2 now states that true-label confidence is obtained from a forward pass without storing or averaging the original training trajectory. Alternative scoring rules are described in Appendix A.3 and compared in Section 5.2, Figure 4.
Q3. We compared FULL, retain-only selection, forget-only selection and both-pool selection for NegGrad+ on CIFAR-10 and CIFAR-100 with ResNet-18, using ten paired deletion and unlearning requests. The initial model, deletion request, learning rate and base training recipe are shared across policies within each request. The unfiltered pool remains complete.
On CIFAR-100, retain-only and both-pool selection use nearly the same computation, 1.856 and 1.852 PFLOPs, and select similar numbers of retain examples. Their Avg. Gaps are 16.078 and 9.999 pp, respectively. Thus, filtering the forget pool changes the outcome beyond the savings from reducing retain computation alone. Both-pool selection also gives a lower Gap than either one-pool policy on CIFAR-10.
Table R3. Selection-side comparison for NegGrad+ with ResNet-18 under random forgetting. Means over ten paired deletion and unlearning requests. The values are taken from main-text Table 4. Mean selected (R, F) gives the average numbers of available retain and forget examples per epoch.
The expanded selection-side comparison has been added to Section 5.2 (Which set to select), Table 4, including the individual metrics, computation and mean selected counts. The result descriptions explain why reducing retain computation alone does not reproduce the both-pool outcome in these comparisons.