acceptodds
Withdrawn from ICLR 2027

Exposure Leaves a Trace A Provenance Audit for Machine Unlearning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce a provenance audit that tests whether an unlearning method removed the of training on the forget examples, rather than only suppressing their outputs. Behavioral parity with the masked reference cannot distinguish suppression from removal. The audit has two stages. First, the exposure-subspace stage constructs a stream-matched training contrast: two models start from one checkpoint and follow the identical training stream, differing only in the loss weight of the forget examples. We define their activation difference as the exposure effect under this matched training intervention, which controls for shared initialization, data order, and training randomness. Removal and insertion show that a low-rank exposure subspace causally affects target behavior under the specified interventions across Llama-3.2-1B, Qwen2.5-1.5B, and Qwen2.5-7B. In the controlled setting, the exposure subspace does not show a pooled advantage over a target-predictive control, as expected when target-correlated activation can coincide with the exposure effect because the forget examples are the target's only source. On pretrained models, where target-predictive directions can also reflect base knowledge, the exposure subspace transfers more behavior. Across seven core unlearning objectives, all six that reach behavioral parity leave a post-unlearning residual whose transplantation into the masked reference recovers 0.15–0.42 of the suppressed behavior. The residual norms remain comparable in scale to retain-only and replace-relabel controls that do not reach parity. An RMU variant supplied with oracle entanglement labels reduces the measured audit diagnostics to the matched-random level, albeit with a large activation displacement. Behavioral parity and removal of the measured training effect are therefore distinct outcomes.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine unlearning, unlearning auditing, interpretability, causal intervention, activation patching, representation analysis, safety

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_LqatOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on a machine unlearning method evaluation task. Machine Unlearning aims to let the model forget target domain knowledge and keep its performance on other (retain) data. Whether the effect of training data remains in the model or not is always an important question in the security domain. The authors propose a provenance audit method to address this issue. By detecting the difference between two different forget-example loss weights, this work builds a stream-matched training contrast. As shown in the experimental results, behavioral unlearning does not necessarily imply the removal of the underlying training effect.

Critical strengths
  1. The motivation of this paper is clear and insightful. To test whether an unlearning method removed the effect of training on the forgotten examples or just had worse performance on these examples.

  2. The proposed method is clear and easy to follow.

  3. The experimental results in Section 4 are comprehensive and show that the identified exposure subspace causally influences target behavior across multiple model families and scales.

Critical weaknesses
  1. The novelty of this work is not sufficient. In my view, the contribution is more of an engineering-driven improvement in characterization rather than a genuinely novel scientific insight into the mechanism.

  2. There is a significant gap between the experimental setting and real-world unlearning. Can the authors propose more results under unlearning knowledge acquired during a large-scale pretraining setting?

  3. Lacks formal guarantees on why this measurement can detect the remaining information in the remaining data.

  4. The most important result in the abstract part (15-42%) may be over-interpreted. In my view, this is a normalized target-score margin change (Eq. 4).

  5. The authors have not sufficiently demonstrated that the residual represents the truly retained influence of the data to be forgotten.

  6. Several equations are not numbered, which makes them difficult to reference. The authors are encouraged to number these equations for better readability.

Questions for authors

See in Weakness.

Reviewer_ELFPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a two-stage activation audit to test whether approximate unlearning removes fine-tuning changes caused by forget data (DfD_f) or only suppresses outputs. First, from a shared checkpoint M0M_0, two runs (MfullM_{\text{full}} and MmaskedM_{\text{masked}}) process identical minibatches with DfD_f's loss weight set to 11 vs. 00; SVD on their activation differences yields a low-rank subspace SS, tested via activation patching on synthetic facts across a 4-layer model, Llama-3.2-1B, Qwen2.5-1.5B, and Qwen2.5-7B. Second, unlearned models MuM_u stopped at output parity with MmaskedM_{\text{masked}} are audited inside SS. Across six parity-reaching core methods on 1B/1.5B models, transplanting the residual du(x)=PS(hu(x)−hmasked(x))d_u(x) = P_S(h_u(x) - h_{\text{masked}}(x)) into MmaskedM_{\text{masked}} recovers 15%–42% of the forgotten score gap. An RMU variant using oracle entanglement labels reduces this recovery to random levels, but only via a ∼30×\sim 30\times activation shift and worse perplexity.

Critical strengths
  1. Transparent falsification reporting across 11 objectives: Although the main takeaways echo prior work, the authors evaluate 11 unlearning variants under sealed test splits and pre-specified falsification checks, openly reporting negative outcomes (e.g., where MeanDiff matches their subspace or TOFU fails their adequacy threshold).
Critical weaknesses
  1. Impractical audit setup, limited novelty, and experimental confounds:

    • Unrealistic setup & limited novelty: Section 1 motivates the method for model providers handling real-world deletion requests (e.g., GDPR). However, the audit mimics TOFU's synthetic benchmark setup by requiring both a pre-exposure checkpoint M0M _ 0 and training MmaskedM _ {\text{masked}} on the retain data across the full training stream—which costs as much compute as retraining from scratch. In realistic unlearning settings (such as WMDP, Appendix F.4.4), M0M _ 0 and MmaskedM _ {\text{masked}} do not exist; and whenever a provider can afford to train MmaskedM _ {\text{masked}}, approximate unlearning and auditing are unnecessary because the provider can simply deploy MmaskedM _ {\text{masked}}. Meanwhile, the main takeaways—that behavioral unlearning suppresses outputs without erasing internal representations, and that RMU masks knowledge via large activation shifts rather than deletion—were already established without needing M0M _ 0 or MmaskedM _ {\text{masked}} [1, 2, 3] (notably, [1] is uncited).
    • Methodological & empirical gaps: First, the transplant test risks false positives: if an unlearned model MuM _ u genuinely erased how its final layers read from SS while leaving intermediate layer ll unchanged, taking du(x)d _ u(x) from layer ll and inserting it into MmaskedM _ {\text{masked}} (whose final layers still read SS) would still recover the answer. Moreover, neither control baseline (Retain-only, Replace-relabel) reaches behavioral parity, leaving unresolved how much of du(x)d _ u(x) simply reflects initializing unlearning at MfullM _ {\text{full}}. Second, in the retain-overlap conditions (REDUNDANT, INFERABLE, Table 8), SS fails to beat a simple MeanDiff probe that needs no MmaskedM _ {\text{masked}}, and MmaskedM _ {\text{masked}} barely differs from standard MretrainM _ {\text{retrain}} (76% vs. 71%, Appendix B). Finally, beyond synthetic nonsense strings, injected TOFU fails the adequacy gate (∣TE∣<2|\text{TE}| \lt 2), MUSE recovers only 5%–10% on 99 items at 1B, and the primary 7B audit uses 1 seed at a single layer (where 4 of 7 methods miss parity and GA fails) despite Appendix D.3 showing that 7B requires two-layer patching.
  2. Poor structure, needlessly convoluted notation, and non-self-contained main text:

    • Disorganized narrative and missing setup: Section 1 jumps abruptly between topics and omits explaining Stage 2 in prose on page 2, jumping straight from validating SS to an undefined "RMU variant." Section 3 ("Experimental Setup") is only 7 lines long and never explains what M0M _ 0 actually is—hiding until page 26 that the LLM runs use Instruct checkpoints and that a scratch-trained 4-layer toy transformer (used in Figure 2 and for the 75%/65% headline claims) even exists—while Section 4 relies on ~10 Appendix tables and figures for core results.
    • Unnecessarily opaque notation: Section 2.1 introduces heavy notation (ωr\omega _ r, zz, wi(z)w _ i(z), Mωr(z)M ^ {\omega _ r}(z), q,pq, p) just to express a simple loss mask in Eq. (1) before immediately discarding those symbols. Similarly, hq,ph _ {q,p} is never formally defined in the main text (its actual intervention site is buried on p. 27), changes notation without warning (h(x)h(x) vs. hq,p(M,x)h _ {q,p}(M,x) vs. hfull(x)h _ {\text{full}}(x)), and is mixed with unused baselines (Mretrain,MreplaceM _ {\text{retrain}}, M _ {\text{replace}}) and Stage-2 equations (Eq. 3).

References:

  • [1] Arditi & Chughtai (2024). Unlearning via RMU is mostly shallow. AI Alignment Forum / LessWrong.
  • [2] Deeb & Roger (2024). Do unlearning methods remove information from language model weights? arXiv:2410.08827.
  • [3] Łucki et al. (2025). An adversarial perspective on machine unlearning for AI safety. TMLR.
Questions for authors

Clarifications:

  1. Since training MmaskedM _ {\text{masked}} costs as much as full retraining and already produces a clean model without DfD _ f (and cannot be run when M0M _ 0 is unavailable), how is this method useful as a practical provider audit (Section 1) rather than only as an analysis metric on synthetic benchmarks—and what new insight does it offer beyond prior work [1, 2, 3] that does not require MmaskedM _ {\text{masked}}?
  2. Suppose an unlearning method achieved parity by erasing how layers above ll read from SS while leaving layer ll untouched: wouldn't transplanting du(x)d _ u(x) from layer ll into MmaskedM _ {\text{masked}} falsely flag retained behavior? Also, what happens if Replace-relabel is trained until it reaches full behavioral parity?
  3. Given that single-layer patching at layer 23 misses most of the exposure effect on Qwen2.5-7B whereas layers 23 + 27 recover 86%–91% (Appendix D.3), why was the 7B unlearning audit (Section 4.3) only evaluated at layer 23?

Suggestions:

  • Rewrite Sections 1–3 so the main paper is clear and self-contained: replace Eq. (1)'s throwaway notation with direct Lfull\mathcal{L} _ {\text{full}} and Lmasked\mathcal{L} _ {\text{masked}} definitions, explicitly define hh and M0M _ 0 (including the 4-layer toy model and Instruct checkpoints) in Section 3, cite Arditi & Chughtai (2024), and move primary tables out of the Appendix.
Reviewer_njECOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes an unlearning audit protocol to distinguish behavioral suppression from the removal of training-data effects. The protocol includes several steps: a) construct matched training runs with and without forget-data exposure, b) identify an exposure-related activation subspace using SVD, c) examine whether unlearned models retain relevant residuals in this subspace. Experiments across several LLMs and unlearning methods suggest that such residuals can persist even after behavioral parity is achieved.

Critical strengths

The authors study an important question about the differences between behavioral forgetting and internal-effect removal, through extensive experiments. The proposed activation interventions provide interesting evidence that some residual signals associated with the forget data can continue to influence model behaviour after unlearning.

Critical weaknesses
  1. The proposed audit indeed found forget-data-related activation residuals inside the unlearned model vs the oracle (M_masked) and demonstrates that they can influence model behavior, under their matched-checkpoint setting. However, this difference does not necessarily indicate unsuccessful unlearning. Exact unlearning is generally defined through distributional equivalence to retraining, rather than matching a particular retrained checkpoint. For example, two retain-only models trained with different random seeds / procedures may legitimately have different internal representations, but neither is exposed to the forget examples. Could such differences also produce substantial residuals under the proposed audit? I would suggest comparing independently trained retain-only models, or continuing to fine-tune M_masked without forget examples (to test if additional optimization alone can produce such residuals), and applying the same residual audit. This would also give a useful reference point about what levels of activation residuals / behavioral recovery we should expect under this audit, even without any exposure to the forget data.

  2. I found the paper's writing is in general rather confusing and a bit hard to follow, particularly the introduction and the methodology section. I think the presentation needs substantial improvement.

  3. This is a secondary concern so i give it less weight in my overall assessment. The authors motivate the work (partly) through relearning attacks / adversarial prompting, but do not directly show whether models with larger residuals are more vulnerable to such attacks, or whether reducing the residuals improves robustness to these attacks. I think evaluating this relationship would largely strengthen the practical motivation of the proposed audit.

Questions for authors

Please see the weaknesses above. In particular, I would appreciate clarification on whether the proposed audit could also detect substantial residuals between two retain-only models, simply because they followed different optimization paths.

Reviewer_m5mnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a provenance audit for machine unlearning to determine whether unlearning removes the influence of forget data or merely suppresses it in the outputs. The authors train two models from the same checkpoint using the same data stream, differing only in the forget-example loss weight (1 vs. 0). They define the resulting activation difference as the exposure effect, fit a low-rank subspace to it, and validate the subspace through removal and insertion experiments. They then stop each unlearning method at behavioral parity with the masked model and measure the residual exposure effect. Across six methods, inserting this residual into the masked model recovers 15–42% of the forgotten behavior.

Critical strengths
  • Well-controlled experimental design. The paired runs differ only in the forget-example loss weight, avoiding confounding effects from different data orders or random seeds.
  • Careful validation. The authors evaluate the subspace against matched random subspaces, cross-fact donors, held-out seeds, and a sealed test split. They also clearly report negative results.
Critical weaknesses
  • The audit does not distinguish successful from failed unlearning. Retain-only and replace-relabel do not reach behavioral parity, yet leave residual transfer comparable to or larger than parity-reaching methods (33–35% vs. 15–25% on Llama-1B; lines 354–362). Since every method starts from M_full, the residual may simply measure the distance from M_full rather than incomplete unlearning. Moreover, the authors do not evaluate models that never saw the forget data, leaving the false-positive rate unknown.

  • The benefit of the paired training contrast is unclear. In the controlled setting, the exposure subspace offers no pooled advantage over the simple MeanDiff baseline (lines 308–316). Its advantage appears only in certain conditions and on pretrained models. Furthermore, the masked, replace, and retrain references yield similar removal transfer (76%, 79%, and 71%; Appendix B). The audit also requires training M_masked, which is already an unlearned model, limiting its practical value for deletion requests (§1).

  • Limited evidence and presentation issues. The main 75%/65% results come from a 4-layer toy model (Table 20). On pretrained models, subspace removal is lower: 43% on Llama-1B and 28% at one site on Qwen-7B (Table 6). The main 7B audit also uses only one seed (Appendix D.2). Many analyses are post hoc (Appendix F.1), and the notation is dense. Finally, §1 reports a "28–36×" displacement for oracle RMU, whereas §4.3, Appendix D.5, and Table 16 report approximately 30× (29.6 and 30.1). These numbers should be made consistent.

Questions for authors
  • What does the audit report for models that never saw the forget data, such as M_retrain or M_masked trained with a different seed?

  • How does a subspace fitted using a standard retraining contrast compare with the stream-matched subspace?

  • Does the residual predict how quickly forgotten facts can be recovered through relearning attacks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.