acceptodds
Under review as a conference paper at ICLR 2027

Batch-Norm Statistics: The Unrecorded Convention in Unlearning Audits

Abstract

Unlearning benchmarks release checkpoints with tables of accuracies and pass/fail verdicts, and a reader treats each entry as a property of the released checkpoint. A batch-normalized checkpoint, however, also ships normalization statistics fitted from data. An audit's retrained reference fits them on retained data by construction, while the unlearned checkpoint ships whatever its own run left, and no release we audit records how. We show that the published numbers of such checkpoints, and some verdicts, also depend on this fitting convention. We reproduce each checkpoint's published numbers and then refit only its statistics on retained data at byte-identical weights. The refit raises forget accuracy beyond the spread MU-Bench reports across training seeds on 47 of the 221 reproducible MU-Bench checkpoints we test. A method's average does not predict which ones. In a targeted test, removed data in the statistics does not explain which checkpoints move. Releases should therefore state how their statistics were fitted, and a reader can check one checkpoint on a single GPU in under an hour.

Primary area:
datasets and benchmarks
Keywords:
machine unlearning, evaluation, reproducibility, benchmarks, batch normalization, released checkpoints, audit validity, measurement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BN7aOpenReview ↗
Rating
2
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the issue of untuned batch-norm statistics in the machine unlearning scenario.

Specifically: machine unlearning methods act on a model to remove the impact of training on a data subset "forgotten data". This procedure may leave the batch-normalization statistics "stale", i.e. that the layer-wise mean and variance buffers do not correspond to the relevant activation means and variances on the data remaining after the "forgotten data" is removed.

A simple well-known procedure to remedy this is to re-tune the buffers on the data that remains [1].

The paper investigates the impact of the buffer re-tuning on two public checkpoint releases (MU-Bench and Unlearning Comparitor), covering seven unlearning methods, plus two self-trained populatons. The method SalUn is excluded because the published forget accuracies could not be reproduced.

Re-tuning the batch-norm statistics does alter the accuracy on the different data partitions beyond seed noise, for roughly a fifth of the checkpoints tested. A handful of pass/fail verdicts (benchmark-specific rules for whether an unlearning was succssful or not), but method rankings are unchanged. Which checkpoints are affected, and why, is left unclear.

Consequently, the paper recommends that benchmarks and checkpoints are released with information about how the batch-norm statistics are treated.

[1] Y. Wu and J. Johnson. Rethinking "Batch" in BatchNorm. arXiv:2105.07576, 2021. https://arxiv.org/abs/2105.07576

Critical strengths

1. Careful reproduction of third-party checkpoints.

I appreciate the paper's effort to get a proper baselines before starting. The paper is a good demonstration of the challenge in reproducing other results in the literature.

Critical weaknesses

1. The paper does not establish why its findings matter, or what causes them.

The problem setting and setup is clear and well-explained. As soon as the paper turns to the Results section, reading and understanding the contributions becomes extremely challening and it is difficult as a reader to extract takeways (generally, or for unlearning specifically).

As far as I understand, the paper's own results suggest that benchmark conclusions largely survive refit: method rankings by forget accuracy are unchanged (lines 399–401), most flipped verdicts sit at the threshold (lines 372–374), and the authors describe the verdict result as "an existence result on four checkpoints" (line 465). The cause of the shift is left open ("no procedural cause identified", line 462), even though the abstract states a negative conclusion about removed data in the statistics.

Given the framing (provided in my summary of the paper) I would have expected a statement similar to: "For methods of class X, the forget, retain, and test accuracy were shifted (significantly) like A, leading to a systematic change in benchmark conclusions for benchmarks B. However for class Y no such change occurs. We investigate this, and find it is because [...]"

It is also fine if the results really are inconclusive. If so, this should be stated up-front, not only in the Limitations section.

This critical weakness 1. is problematic for me as a) it is already well-know in the literature that batch-norm statistic tuning can have a systematic effect [1,2], and b) the particular problem in batch-norm unlearning, specifically constrained to convolutional batch-norm models, whereas modern architectures typically use RMS/LayerNorm [3,4], and so the paper needs to make it clear why the problem matters.

[1] Y. Wu and J. Johnson. Rethinking "Batch" in BatchNorm. arXiv:2105.07576, 2021. https://arxiv.org/abs/2105.07576

[2] P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov and A. G. Wilson. Averaging Weights Leads to Wider Optima and Better Generalization. UAI, 2018. https://arxiv.org/abs/1803.05407

[3] J. L. Ba, J. R. Kiros and G. E. Hinton. Layer Normalization. arXiv:1607.06450, 2016. https://arxiv.org/abs/1607.06450

[4] B. Zhang and R. Sennrich. Root Mean Square Layer Normalization. NeurIPS, 2019. https://arxiv.org/abs/1910.07467

Questions for authors

1. [Clarification] Based on your results, does anything an unlearning researcher cares about change after a refit?

Refitting appears to raise all of forget, retain and test accuracies, while benchmarks judge checkpoints on comparisons (e.g. the gap between forget and test accuracy, or method rankings).

How often does the gap itself move beyond some threshold? Does any method family show a consistent change in benchmark conclusions?

A systematic change for some class of methods would directly address my main concern (Critical Weakness 1) and could change my assessment.

2. [Clarification] What do you think is causing the shift?

The paper names two possible explanations: removed data surviving in the shipped statistics ("deletion failure"), or statistics that are stale relative to the final weights (lines 053, 077). The abstract rules out the first (lines 021–022), based on the targeted removed-image test in Section 4.2: at the final, fixed weights, the statistics are refit with and without removed images swapped in, and the accuracies barely change (lines 311–316, Table 2).

I am not sure this test can rule deletion failure out. The shipped statistics of some methods were updated on batches made only of removed images while unlearning was running (lines 308–309), i.e. at earlier weights, whereas the test only adds removed images at the final weights. The test is also run only on MU-Bench (lines 306–307), not on Unlearning Comparator's class deletion (lines 206–207), where removed data should be easiest to detect.

Could you either show that the test would detect removed data in statistics accumulated during unlearning, or narrow the claim in the abstract? If the cause is stale statistics, please say so explicitly. If removed data does survive in the statistics, that would be a genuine unlearning finding and could change my assessment.

3 comments
AuthorsOpenReview ↗

Response to Reviewer BN7a (1/2): what changes after a refit

Thank you for the careful reading. We agree with both main points and answer them directly.

Q1 / W1 — does anything an unlearning researcher cares about change? Mostly no, and we will say so up front instead of in the Limitations.

We computed what you ask for from the per-checkpoint results already in the supplement: the shift of the signed gap d_f − d_t under the refit, decided by the same rule as the paper (all ten refits beyond the 1.2 pp margin).

MU-Bench groupmethodnd_f displacedgap displacedmean gap shift (pp)
CIFAR-100 R50bad teaching18121+0.61
CIFAR-100 R50NegGrad4580−0.05
CIFAR-100 R50random label800+0.27
CIFAR-10 R50bad teaching3054+0.60
CIFAR-10 R50NegGrad45100+0.00
CIFAR-10 R50random label4511+0.34
CIFAR-10 R34bad teaching1030−0.14
CIFAR-10 R34NegGrad1030+0.07
CIFAR-10 R34random label1050+0.24
total221476

The gap moves past the margin on 6 of 221 checkpoints, against 47 for forget accuracy alone. Across the 221, the forget and test shifts correlate at r = 0.97, and among the 47 displaced checkpoints the median gap shift is 0.34 pp. So most of the movement is common to all three accuracies, as you suspected. With a two-sided sign test the 6 do not survive the false-discovery control we use elsewhere, so we report them as a count only. All 6 move the same way (forget accuracy rises more than test accuracy), 5 are bad teaching, and 4 are the CIFAR-10 ResNet-50 bad-teaching checkpoints the refit raises by +11 to +20 pp.

No method family shows a consistent change in benchmark conclusions: NegGrad's mean gap shift is within 0.1 pp of zero in every group, rankings by forget accuracy are unchanged, and 12 of the 16 flipped verdicts are threshold effects. What does change is narrower: individual published rows (47 of 221), and 4 verdicts.

We will add this table, and restate the contribution in the abstract and introduction as what it is: a reproducibility result about per-checkpoint numbers in released tables, with benchmark-level conclusions largely intact.

Why it still matters, given [1, 2] and the move to Layer/RMSNorm. We cite both [1] and [2] and use the estimator of [1]; we do not claim that recomputing statistics changing accuracy is new. What was open is whether tables the field already cites for third-party checkpoints survive it, row by row. The scope is batch-normalized classifiers only, as stated in Limitations. That scope is still where audits against a retrained reference are run: both public releases with per-checkpoint tables are BN ResNets, and 19 of the 42 papers in our rule-based literature sample evaluate BN convolutional models against a retrained reference, of which 4 mention the statistics.

AuthorsOpenReview ↗

Response to Reviewer BN7a (2/2): cause of the shift, new class-deletion control

Q2 — what causes the shift, and can the Section 4.2 test rule out deletion failure? You are right that it cannot, and your suggestion led to a result we did not have. We ran the test on Unlearning Comparator's class deletion (new Appendix G, Table 14; 6 checkpoints × 10 deleted classes × 3 draws). At fixed weights we add images of the deleted class to the refit and compare two directions in the space of running means: the one in which the class moves the fitted statistics, and the one in which the shipped statistics differ from a retained-only fit. A projection of 1 means the shipped statistics differ from a retained-only fit exactly as a fit containing the class at its 10% share does.

checkpoint (10 deleted classes each)cosine with deleted-class directionclasses > 0projection (median)refit shift, retain test (pp)
original model (positive control)+0.9310 of 100.98−0.75
retrained reference−0.063 of 10−0.07+0.02
CF-k+0.037 of 100.02+0.05
AdvNegGrad+0.4210 of 107.8+1.89
SalUn+0.017 of 100.31+2.22
SSD+0.379 of 1011.9+11.36
  • The test detects removed data when it is there. The original model's statistics were fitted with the class present, and the test recovers it (cosine +0.93, projection 0.98, draw-to-draw sd 0.006). CF-k has the same batch-norm inputs as the original model but statistics refreshed on retained data, and shows nothing.
  • Removed data does survive in the shipped statistics of two of four methods. AdvNegGrad is aligned on all ten classes and every draw, about eight times as far along the class direction as the class at its share would place it, as statistics averaged largely over deleted-class batches would be. SSD is aligned on nine classes; its code takes no training-mode pass, so it ships pre-unlearning statistics that contain the class by construction.
  • It is not the whole story. The class direction carries about a fifth of the squared offset for both methods, and SalUn shifts as far under the refit (+2.22 pp) with no trace of the class. With the class at its share in the fit, retain accuracy moves by at most 0.13 pp and forget accuracy stays at 0% (all but SalUn).

So both explanations occur: stale statistics, and statistics that carry removed data. We do not separate how much of a shift each causes, and we no longer claim that removed data "does not explain" the movement. The abstract now reads: "Adding removed images to the refit at the final weights does not reproduce the shift on MU-Bench, where the removed data is a random subset. On class deletion, a control that recovers the deleted class in the original model's statistics finds it in the shipped statistics of two of four methods." The same narrowing is made in the introduction, Section 4.2, the limitations and the conclusion. The control reads a direction at the final weights, on one release; SalUn's ten checkpoints are included although only two reproduce their published numbers.

All changes in the revised PDF are in blue.

Reviewer_BN7aOpenReview ↗

Response To Authors

Thank you for your detailed responses and running the additional experiments.

Q1. Based on your results, does anything an unlearning researcher cares about change after a refit?

Thank you for running this experiment. If I understand correctly, the conclusion is:

  1. refitting batch-norm statistics moves individual checkpoint numbers
  2. benchmark conclusions remain unchanged (rankings are unchanged, and the gap moves on 6 of 221 checkpoints, which does not survive the false discovery rate control). Despite this somewhat "negative" result, I do think "does tuning batch-norm change anything?" is a fair question, and the paper now gives a clear answer. I appreciate that this is now stated up front.

My remaining concern here is scope - the main counts come from one benchmark, on CIFAR with ResNets. I understand the authors discuss this in the limitations section.

Q2. What do you think is causing the shift?

Thank you for the class-deletion control experiment, and for narrowing the claims of the paper.

As I understand it, the control refits each checkpoint's statistics with and without images of the deleted class, and checks whether the shipped statistics differ from the class-free refit in the same way that adding the class does [L1228–1239]. The control works when the weights are unchanged: it finds the deleted class in the original model's statistics [L1239–1241]. My question was about the harder case, where the statistics are collected while unlearning changes the weights (e.g. AdvNegGrad). There, I am not yet convinced the control can tell "the class is in the statistics" apart from "the weights were changed to forget the class". For a few reasons:

  1. The control does not look for the class directly, only for this matching pattern. As far as I understand, unlearning a class tends to change the weights most where the network responds to that class, so out-of-date statistics could be off in those same places with no class data in them.
  2. The control's numbers depend strongly on the weights. SSD ships its pre-unlearning statistics, which contain the class by construction [L1271–1272], yet its projection is 11.92 against 0.98 for the original model (Table 14).
  3. SalUn shows no match even though its statistics are updated on removed-only batches [L499–500], so the readout under moving weights does not seem well understood either way.

As a simple check, give SSD's weights statistics known to contain no class data (e.g. computed at the original weights on retained data only) and rerun the test: if the match remains, it comes from the weights; if it disappears, from the removed data. For AdvNegGrad, the equivalent is rerunning it with statistics updated only on retained data.

Until then, I would move the SSD caveat [L1271–1272] from Appendix G into the main text, next to the "two of four methods" claim.

Score

I am raising Informativeness from 1 to 2: the paper now gives a clear answer to a fair question, I learned something useful from it. I am keeping my Rating at 2. In my review I named what would change my assessment:

  1. a systematic change in benchmark conclusions (Q1), or
  2. removed data surviving in the statistics (Q2). The answer to Q1 is no, and Q2 is not yet established, as far as I can tell. If the checks above show that removed data does survive in the shipped statistics, that would be the genuine unlearning finding I asked about, and I would raise my score.

I thank the authors again for their responses during this rebuttal.

Reviewer_xzVQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether the reported results of released machine unlearning models depend on the batch norm statistics shipped with them. Using checkpoints from two public benchmarks, MU-Bench and Unlearning Comparator, the authors keep the weights fixed, recompute the batch norm statistics on the retained training data and evaluate again. For many checkpoints, forget accuracy changes by more than the normal variation across random seeds and some checkpoints switch between passing and failing the benchmark. The authors attribute this to statistics that no longer match the weights after unlearning, rather than to leftover information from the removed data, and recommend that releases report how these statistics were computed.

Critical strengths
  1. The experiments are careful and easy to verify. The authors first check that they can reproduce the published numbers and they use models that were never unlearned as controls.

  2. The paper makes a useful point: averages at the method level in unlearning benchmarks can hide large changes in individual checkpoints.

Critical weaknesses
  1. I am not convinced by the test in Section 4.2 that is meant to show the removed data is not the cause. If the forget set is a random sample of the training data, then removed and retained images should give very similar batch norm statistics at the final weights, so the test would show no difference in either case. The more likely way removed data affects the statistics is during unlearning itself, when some methods update the running averages using batches made only of removed images. Testing at the final weights does not capture this. The paper's own results also point the other way: methods that update statistics this way shift more (Table 19), and checkpoints from the random-label method on CIFAR-10 ResNet-50 do react to the removed images. Two scenarios "Out of date" and "affected by removed data" can also be true at once. Without a check showing the test can detect removed data when it does matter, I think the paper can only imply that removed and retained images look alike at the final weights. That is a much weaker than the stated significance in Section 6.

  2. The contribution feels smaller than the paper presents it. It is already known that recomputing batch norm statistics changes accuracy. The recomputation also raises retain and test accuracy, so a large part of the effect looks like normal recalibration after the weights have changed, not something specific to forgetting. The shipped statistics are simply what the released code produces by default, so describing them as an undocumented choice seems too strong. The paper also does not say which choice an audit should use. Finally, the main counts come from one benchmark, on CIFAR with ResNets only, and two of the four verdict flips with a wide margin happen only because the recomputation fixes models that had poor accuracy to begin with. Checking each checkpoint individually is useful, but the overall contribution is modest.

Questions for authors
  1. Are the MU-Bench forget sets random samples of the training data? If they are, what result would you expect from the Section 4.2 test if removed data really did affect the statistics? Can you show the test works on a case where it should find an effect? At least with respect to Unlearning Comparator.

  2. How does the conclusion of Section 4.2 fit with the finding in Section 4.4 and Appendix K that methods updating statistics on removed-only batches shift more? Rerunning a few bad-teaching or NegGrad checkpoints with statistics updated only on retained data, or kept frozen, would test this directly.

  3. Since the recomputation raises retain and test accuracy too, how many checkpoints still change significantly if you look at the gap between forget and test accuracy instead of forget accuracy alone?

  4. Which way of computing batch norm statistics should an audit use? Should recomputing them count as part of the unlearning method, given that it needs retained data? And should the retrained reference model be treated the same way?

  5. Adding a third benchmark or a dataset other than CIFAR to the main counts would make the general contribution much stronger.

3 comments
AuthorsOpenReview ↗

Response to Reviewer xzVQ (1/2): the removed-image test, new class-deletion control

Thank you; the review identifies the weakest inference in the paper and we agree with most of it.

W1, Q1, Q2 — the Section 4.2 test. MU-Bench's forget sets are random subsets of the training data (2% to 10%), and we agree with your reading: at the final weights, removed and retained images are then nearly exchangeable for the statistics, so the test is expected to be null whether or not removed data shaped the shipped statistics during unlearning. We ran the check you ask for on Unlearning Comparator's class deletion (new Appendix G, Table 14; 6 checkpoints × 10 deleted classes × 3 draws). We add images of the deleted class to the refit and compare the direction in which the class moves the fitted statistics with the direction in which the shipped statistics differ from a retained-only fit. A projection of 1 means the shipped statistics differ exactly as a fit containing the class at its 10% share does.

checkpoint (10 deleted classes each)cosine with deleted-class directionclasses > 0projection (median)refit shift, retain test (pp)
original model (positive control)+0.9310 of 100.98−0.75
retrained reference−0.063 of 10−0.07+0.02
CF-k+0.037 of 100.02+0.05
AdvNegGrad+0.4210 of 107.8+1.89
SalUn+0.017 of 100.31+2.22
SSD+0.379 of 1011.9+11.36

The test works where it should: it recovers the class in the original model's statistics (cosine +0.93, projection 0.98) and finds nothing in the retrained references or in CF-k, whose batch-norm inputs equal the original model's but whose statistics were refreshed on retained data. And your expectation is borne out: the deleted class is present in the shipped statistics of AdvNegGrad (all ten classes, about eight times the class's share) and SSD (nine classes). "Out of date" and "affected by removed data" do hold at once. The class direction carries about a fifth of the squared offset in both, and SalUn shifts as far (+2.22 pp) with no trace of the class, so removed data in the statistics is one contributor and not the explanation of the shift.

We have therefore withdrawn the claim that removed data "does not explain which checkpoints move" from the abstract, introduction, Section 4.2, limitations and conclusion (changes in blue), and Section 4.2 now states what the MU-Bench test can and cannot detect and points to the method-level result of Table 19. We did not rerun bad teaching or NegGrad with frozen or retained-only statistics. The closest existing evidence is the frozen-statistics twin design on our self-trained SVHN population (Appendix F and H): suppressing only the statistics update at bitwise-identical weights changes test accuracy by −9.5 to −75.8 pp, and fine-tuning on retained data only pays that cost too, so a mismatch between statistics and moved weights is sufficient without removed data.

(Continued in part 2/2. All changes in the revised PDF are in blue.)

AuthorsOpenReview ↗

Response to Reviewer xzVQ (2/2): gap, contribution, convention, scope

Q3 — the forget–test gap. From the per-checkpoint results in the supplement, the signed gap d_f − d_t moves past the 1.2 pp margin on all ten refits for 6 of the 221 checkpoints, against 47 for forget accuracy alone: 4 bad teaching on CIFAR-10 ResNet-50 (gap shift +4.30 to +5.73 pp), 1 bad teaching on CIFAR-100 (+1.53 pp) and 1 random label on CIFAR-10 ResNet-50 (+2.50 pp). All six move in the same direction. The forget and test shifts correlate at r = 0.97 across the 221, and the median gap shift among the 47 displaced checkpoints is 0.34 pp. With a two-sided sign test the six do not pass our false-discovery control, so this is a count, not a certified one. Your reading is correct: a large part of the effect is ordinary recalibration after the weights moved. We will add the per-method table and say this in the main text.

W2 — size of the contribution. We accept the points individually. Recomputing statistics changing accuracy is known and cited (Izmailov et al.; Wu & Johnson); the new part is limited to third-party published tables, decided per checkpoint. "Unrecorded" is meant literally — the shipped statistics are what the released code leaves by default and no release says so — and we will remove wording that reads as a deliberate undocumented choice. Two of the four decisive flips are Unlearning Comparator checkpoints that failed the retain-accuracy bar and pass after the refit; the paper says so, and we will say it where the count of four first appears.

Q4 — which convention should an audit use? Our view: report both, and name the one the verdict uses. If a single one must be chosen, the comparison against a retrained reference is matched when both sides carry statistics fitted on retained data at their final weights, i.e. the refit under the training transform. The reference can be treated identically at no cost: refitting Unlearning Comparator's ten references moves retain accuracy by +0.01 pp. Because the refit needs retained data, it should count as part of the unlearning method whenever the method's claim is made under a retained-data budget; an audit that scores shipped statistics is evaluating the method as its authors ran it, which is equally defensible. That is why the paper asks for the convention to be recorded and stops short of prescribing one, but we will add this paragraph.

Q5 — a third benchmark. We found no third release that publishes a per-checkpoint table an outsider can reproduce; that is why the two additional populations (SVHN and CIFAR-100, VGG-16-BN) are self-trained and pooled with neither release. The main counts therefore rest on MU-Bench, CIFAR and ResNets, and we state that limit.

I thank the authors for the detailed responses and additional experiments. The new class-deletion control and SVHN experiments are helpful and address several of my concerns.

However, the authors still have not directly tested how updating batch normalization statistics using only removed data affects the released checkpoints. The results showing larger shifts for certain methods are interesting, but they do not establish that this is what causes the changes in individual checkpoints. Also only 6 of the 221 checkpoints show a change in the forget-test accuracy gap beyond the chosen margin and this result does not remain significant after correcting for multiple comparisons.

Overall, I appreciate the authors' efforts and the more careful presentation of their findings. However, the revised paper makes a narrower contribution, mainly showing how batch normalization statistics affect the reproducibility of unlearning results. My concerns about the paper's novelty and the extent to which the findings apply more broadly remain.

Reviewer_wLACOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the impact of batch-normalization (BN) running statistics on machine unlearning evaluations. The authors propose a refitting procedure that recomputes BN statistics using retained training data while keeping model weights unchanged.

Critical strengths
  1. Rigorous experimental design. The authors isolate the effect of BN statistics by keeping model weights byte-identical during recalibration. They validate their evaluation pipelines against published results and use repeated refits, statistical significance tests, and appropriate controls to distinguish systematic changes from sampling noise.
  2. Investigation of competing explanations. The experiments distinguish the potential influence of forgotten data in BN statistics from mismatches between statistics and final model weights. The results suggest that stale statistics may be an important factor, providing useful insights into the underlying mechanism.
Critical weaknesses
  1. Incomplete causal explanation. Although the experiments suggest that stale BN statistics rather than residual forgotten data explain many observed changes, the evidence remains indirect. The paper does not systematically track the evolution of BN statistics during unlearning or directly establish how statistical staleness causes the observed accuracy shifts.
  2. Limited evidence regarding actual forgetting quality. Most conclusions rely on changes in classification accuracy and benchmark-specific pass/fail criteria. These changes do not necessarily imply changes in privacy leakage or the actual removal of training-data influence.
Questions for authors
  1. The paper aims to unlearn certain training images, but why would unlearning lead to improved performance on other classification tasks?
  2. Can BN refitting reliably unlearn certain classes? What advantages does this approach offer over simply masking the classes to be forgotten?
  3. The paper demonstrates that recalibrating BN statistics can significantly change unlearning evaluation outcomes. However, it remains unclear whether these changes reflect meaningful differences in the removal of private information or merely changes in output behavior.
1 comment
AuthorsOpenReview ↗

Response to Reviewer wLAC

Thank you for the review. Several questions suggest the paper did not make its scope clear enough, so we start there.

Q1, Q2 — the refit is a measurement, not an unlearning method. The paper does not propose batch-norm refitting as a way to unlearn, and it does not unlearn classes; a comparison with masking the forgotten classes therefore does not apply. The refit takes a checkpoint someone else already unlearned and released, keeps its weights byte-identical, and recomputes only the normalization statistics, to ask whether the published audit numbers of that checkpoint depend on how those statistics were fitted.

That also answers why accuracy rises on data that was never targeted. Unlearning moves the weights, while the running statistics a checkpoint ships were accumulated as the weights were moving (or on unusual batches) and need not match the final weights. A mismatch degrades every prediction, not only those on removed data. Recomputing the statistics at the final weights removes the mismatch, so retain and test accuracy recover together with forget accuracy: across the 221 MU-Bench checkpoints the forget and test shifts correlate at r = 0.97. On models where nothing was removed the refit does almost nothing (+0.21 pp on MU-Bench's original model, +0.01 pp on Unlearning Comparator's retrained references). We will state this at the start of the Results.

W1 — the causal explanation is indirect. We agree, and the revision both narrows the claim and adds one direct measurement. We do not track the statistics during unlearning. On Unlearning Comparator's class deletion, where removed data is identifiable, a new control (Appendix G, Table 14) recovers the deleted class in the original model's statistics (cosine +0.93 between the class direction and the shipped offset), finds none in the retrained references (−0.06), and finds it in the shipped statistics of two of the four methods (AdvNegGrad +0.42 on all ten classes, SSD +0.37 on nine), where it carries about a fifth of the offset. SalUn shifts as far with no such trace. So stale statistics and statistics that carry removed data both occur, and the paper no longer says that removed data does not explain the movement; it says we do not separate the two contributions.

W2, Q3 — forgetting quality. We agree that accuracy and pass/fail changes do not establish a change in privacy leakage, and the paper does not claim one: the conclusion states that the refit "supports claims about how reproducible audit numbers are, not about deletion safety." What we could measure is in Appendix J. On MU-Bench the standard membership attack is at chance on these checkpoints (median 51.8% against a 55% readability floor), so there is no usable privacy readout. On Unlearning Comparator, which ships a retrained reference per class, one method of four (SSD) moves 0.0167 of the distance from the reference back toward the original model in its confidence on the deleted class, in the same direction on 10 of 10 classes; the other methods stay at noise. On MU-Bench's zero-retrain-forgetting score, 21 of 72 checkpoints move past the release's own spread (three refits, not certified). The honest summary is that the changes we establish are changes in the audit's output numbers; whether they reflect a difference in removed information is open except for that one small SSD effect. We will move this statement from the appendix into the main text.

All changes in the revised PDF are in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.