- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work studies on what membership-style evidence can and cannot establish in machine unlearning. It makes a controlled study of what a standard action-error membership score measures, a test of whether score-based rules can recognize retain-only retraining, and a statistical framing of a hard-pair total-variation bound on joint coverage. The central negative result is well-designed.
Critical strengths
The paper separates three statistical objects that unlearning audits routinely conflate: a model distribution, a fixed checkpoint, and the trajectory scores a checkpoint produces. It shows that the pairwise tagged-KS distance is only an upper-bounded surrogate for distributional distance in Eq. (5). The empirical consequence is decisive for a currently popular evaluation practice. The combination of “the rule rejects the bad thing” and “the rule also rejects the good thing” demonstrates the failure mode that per-task unlearning papers do not check.
Critical weaknesses
- The gap to prior work is not established where it matters most, which makes it hard to judge whether the contribution is a discovery or a confirmation. The paper should states the difference among Zhang et al. (2025), Yang & Yeung (2026), and this paper.
- The empirical base is narrow. All results are on two MuJoCo locomotion tasks. As the paper mentioned, “these are resampling slots, not additional independent trained models” and “repeated resampling cannot replace additional independent models”. The intervals reflect trajectory-level resampling rather than the model-level variability that the claim is about. Similarly, the strongest new evidence where 10/10 secret-reference transfers is established with only 4 secret checkpoints per environment.
Questions for authors
- Could you report a model-level sensitivity analysis, e.g. leave-one-model-out across the three sources per role, or the spread of the positive-pass probability across the individual source combinations, so that the 0.134/0.289 estimates are not presented solely with resampling-level uncertainty?
- This work argues that “rejecting full-data comparators coexists with frequent rejection of retraining controls”. Is there a decision rule you would recommend instead that reports diagnostics without inviting the equivalence interpretation?
Thank you for reconsidering the score, and for the question that led to the correction.
We agree that the original development study should have caught this, and we would like to be precise about what changed. On the same simulated draws, correct accounting moves the tagged-KS outcomes for retraining controls from 0.237/0.381/0.382 to 0.143/0.842/0.015 (authorize/abstain/reject; Table 2). Rejections become abstentions, not authorizations. With three retrains, the rule still does not recognize retraining, so that part of the submitted conclusion stands. What changed is the mechanism and the failure mode: the rule does not wrongly reject retraining because of the information in the score, but cannot decide because three retrains are too few. We withdrew the earlier attribution, and the revised abstract and introduction state the correction.
The original validation missed the error for the reason the paper warns about. It checked that full-data models were rejected, and they were rejected under both accountings. Only the retraining controls, examined at the model level as you asked, exposed the defect.
The revised evidence no longer depends on that development study. We tested the mechanism on 67 newly trained independent models. Imposing the slot construction reproduces the failure (rejection 0.498), and source-level accounting removes it (0.044) (Table G.3). All later results use independently trained models as units, with whole-model permutation tests and model-bootstrap intervals. The discussion-period analyses fixed their gates and predictions before scoring.