- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how predictions on forget samples change after retraining without those samples, aiming to approximate these outputs without access to retain data. Its empirical analysis examines which samples change predicted class, how probability mass is redistributed, and how confidence changes when predictions remain stable. Based on these observations, the authors propose Rank-Spread Flip (RSF), which uses entropy-based sample selection, mild confidence calibration, and minimal top-class probability adjustments to construct synthetic targets for forget-only distillation. Experiments across five image-classification datasets with 10% and 20% random deletion evaluate alignment with retrained outputs, forgetting, retained utility, membership inference, and runtime. The reported results generally show improved trade-offs over the evaluated baselines, although substantial utility gaps remain in some settings.
Critical strengths
Useful empirical analysis of retrained outputs. The paper examines which forget samples change predictions and how their probability distributions shift, while accounting for variability across retraining seeds. This provides useful guidance for designing unlearning targets beyond simply forcing misclassification or reducing confidence uniformly.
Promising results under forget-only updates. Across five image-classification datasets and two deletion fractions, RSF generally achieves a better forgetting–utility trade-off than the evaluated baselines. These results support the practical value of selectively adjusting soft targets, although gaps to retraining remain.
Critical weaknesses
The methodological novelty is overstated. Approximating retraining without the forget data is a standard objective of machine unlearning, and forget-only distillation with modified targets already exists in methods such as UNDIAL. Moreover, ignoring the numerical margin, RSF’s changing branch is equivalent to lowering the highest logit to the runner-up logit while leaving the others unchanged. The contribution therefore lies primarily in the empirical observations, sample selection, and adjustment rules, rather than a new formulation or redistribution mechanism. However, the target-level ablation uses an oracle change mask, and matched end-to-end comparisons do not sufficiently isolate the benefit of these specific design choices.
The evaluation does not adequately distinguish output adjustment from removal of sample-specific information. Prior work has demonstrated that classifier changes can produce apparently successful unlearning while leaving recoverable information in internal representations. RSF directly optimizes modified output targets; its distributional comparisons and reported membership-inference results are useful, but do not rule out this failure mode. Since the task is random-sample deletion, the relevant concern is residual sample-specific information relative to retraining—not whether class information remains decodable. Without complementary representation-based privacy or recovery evaluations against retrained controls, the evidence supports improved output matching more clearly than removal of training-data influence.
Questions for authors
Clarification of the methodological contribution: Is the changing branch equivalent, up to the numerical margin, to lowering the top logit to the runner-up logit? Under matched optimization settings and without oracle change masks, what evidence isolates the gains from entropy-based selection over random selection and simpler fixed-strength logit adjustments?
Evidence addressing residual information: Do you have evidence that RSF reduces sample-specific information beyond changing the model’s readout—for example, representation-based membership inference or controlled recovery tests compared with retrained models? Class-label probing alone would not resolve this concern in the random-deletion setting.
Clarification of model selection: What exact criterion selects and the reported checkpoints? Does it use any retrained-model information, and what data would the selection procedure require in deployment?
2 comments
Response to Reviewer kJ7C
Thank you for your time and for the constructive feedback throughout the discussion.
Thank you for the update. Overall I think this is a good paper and should be accepted. I have therefore increased my score accordingly.