-
The corrected detector is never shown to find correlated records that are actually present, so the near-zero results carry little information. The main statistic is a median over all deleted subjects in which a subject with no detected correlated record counts as zero, so it equals zero whenever fewer than half of the subjects have a detection, whatever the size of the real effect (Appendix B.5). This is the situation on all three datasets that read zero. At the same time, none of the five planted records passes the authors own pre-set criterion of being detected in at least half of the runs with an effect above their threshold for a meaningful effect (Section 5.2). The planted record built to have an effect well above that threshold is detected in 1 of 72 runs, and when it is forced into the analysis the estimate recovers about one ninth of its known effect (Appendix B.4). Appendix F.3 adds that in 137 of 186 subject and seed combinations the remaining data already predicts the deleted subject as well as its noise allows, so no retained record, not even an exact copy, could improve the prediction. Subtracting the mean removes false detections, but the paper does not show that it keeps true ones, and a detector that flags nothing would produce the same tables. Since Section 9 and Box 1 ask practitioners to adopt this detector, I do not think the evidence supports that recommendation yet. I also note that where records are detected, several subjects on MIMII, Paderborn and the larger PTB-XL pool show effects above the paper's threshold, and the all-subject median hides exactly these cases.
-
The formal results add little that is new, and the privacy harm being measured is not established. Proposition 1 holds by definition: under exact unlearning the two compared models are identical, so a record-level audit detects nothing at all, not only nothing about correlated records. That deleting a record gives no protection against inference from correlated data that remains is already known from work the paper cites [1], [2]. Theorem 1 is the standard low-rank (Woodbury) update for ridge regression, and the mean subtraction is the post-processing step proposed in [3]. On the threat side, "reconstruction" here means how well a linear head forecasts the next value of the deleted subject's signal, and three of the four datasets contain no personal data. The harms used as motivation in Section 1, re-identifying a patient or revealing a machine's operating state, are not tested. Section 8 also states that a positive reading neither shows a deletion failure nor justifies removing other subjects' data, which leaves open what an auditor should do when the measure is positive.
Minor issues:
-
The main text depends heavily on a very large appendix and uses many terms of its own (for example "predicted floor", "admissible pairs", "abstention band" and "window-clustered relabeling") without defining them where they first appear. Sections 5.2, 6.4 and 6.5 were hard to follow even after several readings.
-
Section 8 refers to a certificate from Appendix E that the main text never introduces, and Appendix E.5 says it covers no leak measured in the study. It should either be explained in the main text or dropped.
-
Related work in the main text (Section 7) is a single paragraph, and the actual positioning against prior work is left to Appendix D.
-
In the natural-data experiments the three seeds change only the random control and the planted records (Appendix F.1), so they are not repeated measurements of the effect. Section 4.2 reports a three-seed mean and should say this.
References
[1] R. Chourasia and N. Shah. Forget unlearning: Towards true data-deletion in machine learning. In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023.
[2] M. Humbert, B. Trubert, and K. Huguenin. A survey on interdependent privacy. ACM Computing Surveys, 52(6), 2020.
[3] J. Mu and P. Viswanath. All-but-the-top: Simple and effective postprocessing for word representations. In International Conference on Learning Representations (ICLR), 2018.