acceptodds
Withdrawn from ICLR 2027

RECAST: Beyond Weight Reinitialization for Machine Unlearning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Machine unlearning is commonly evaluated through the behavior a model exhibits after data is removed. Yet similar behavioral outcomes can be achieved through completely different model modifications. This raises a question: what kind of internal state does an unlearning procedure actually produce? We study this question across classification and generative concept removal, comparing unlearned models on their outputs, their internal state, and how quickly forgotten behavior returns. Across two class-level classification benchmarks, methods with similar forgetting at the output reach substantially different internal states, often staying close to the original model, and differ in how fast the forgotten behavior comes back. Motivated by these differences, we introduce RECAST (REtain-guided Contraction via Alignment Scaled Transformation), a retain-guided weight intervention that uses gradient–weight alignment to continuously contract structural weight groups. RECAST combines strong behavioral agreement with the retrained reference and slower recovery of forgotten behavior than most evaluated baselines. Under repeated deletion requests, it is the only evaluated method whose features end close to a model retrained on the remaining classes. We further extend RECAST to nudity concept removal in FLUX, where it strongly suppresses the concept under both standard and adversarial prompts while preserving benign generation quality. Together, our results show that behavioral outcomes, internal state, and recoverability capture complementary properties of unlearned models, and position retain-guided continuous contraction as an alternative to reset-based weight interventions.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine unlearning, initialization, information erasure

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_etqsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents RECAST, which stands for Retain-Guided Contraction via Alignment Scaled Transformation. RECAST is an unlearning method that measures the alignment between gradients of the retain loss and model weights. Based on this alignment, it contracts weight groups more strongly when they are less aligned. The unlearning procedure consists of two stages. In the first stage, weight groups are contracted according to their measured alignment, determined through cosine similarity. This contraction is bounded by a contraction hyperparameter. In the second stage, called consolidation, the model is fine-tuned again on the retain set only. The authors provide experiments on class-level unlearning as well as concept unlearning in a generative model setting.

Critical strengths
  1. The paper presents results on class-level unlearning. The reported results are favorable, as the proposed method consistently ranks among the two best-performing methods across all metrics. It is also valuable that the authors evaluate the proposed method in a generative model setting, demonstrating promising results and utility in a setting other than class-level unlearning.
  2. The mathematical analysis provided in the appendix is a valuable addition, and the proofs and propositions appear sound. Furthermore, the extensive experimental protocols provided in the appendix are beneficial for reproducibility.
Critical weaknesses
  1. Presentation and Overall Readability The most significant weakness of the paper is its presentation. In its current form, the manuscript appears unfinished and not yet ready for submission. While many of the issues listed below would be relatively minor in isolation, their accumulation substantially affects the overall quality and readability of the paper. In particular, the paper does not guide the reader well through its content. Several abbreviations are used without being properly introduced or explained, sometimes requiring the reader to search through the paper or appendix to understand their meaning. The following issues should be addressed:
    • Figure and table formatting: Figure 3 and Table 4 are poorly integrated into the surrounding text, leaving awkward white space behind or overflowing the page limit. Minor point: the column separator in Table 1 is misaligned between the behavioral and internal profile sections.
    • Unexplained abbreviations: e.g.: FT, NCC, NC3, LMC, and SH,… Please introduce abbreviations when you first use them. Also, is ROFT the same as FT?
    • Broken hyperlinks: Every occurrence of "CIFAR-10/preactivation ResNet-18" appears to contain a broken hyperlink associated with the word "preactivation."
    • Minor notation issue: The repeated use of "FLUX.1 [dev]" instead of simply "FLUX" appears somewhat unnatural and could be reconsidered.
    • Minor citation issue: The citation for the Random Label method appears to be incorrect and should refer to [1] to credit the original idea.
    • Incomplete Appendix
      • Table 11 refers to ablation studies, but it is unclear where these results are presented in the paper or whether they are included at all.
      • Table 12 is entirely empty.
      • Table 14 contains annotations such as "(seq.; deferred)" which appear to be leftover annotations.
      • Table 15 contains a "Correlation" row that is almost entirely empty across the reported settings.
      • Table 18 similarly contains a "Corrected" row that is almost entirely empty.
      • Fig. 4 is just a screenshotted Table from another paper which is never referenced in the text.
  2. Baseline Selection
    • The selection of baselines is inconsistent across Tables 1–3. For instance, SSD, NegGrad, and SHs are missing from Table 2, while other baselines are absent from Tables 1 or 3. Why are not all baselines evaluated throughout? A consistent selection of baselines is essential for a transparent and fair evaluation. Otherwise, it raises concerns about potential cherry-picking of results.
    • Some relevant baselines are not considered. Although mentioned in the limitations, at least L1 sparsity-based unlearning methods should be included due to their similarity in dampening model weights, and potentially SCRUB as well. For the FLUX experiment, ConceptPrune [2] would be a relevant comparison, as it identifies critical model parameters and zeros them out (instead of using contraction as you propose).
  3. Incremental Novelty The main novelty appears to be the application of the direction-aware principle of DASH to the unlearning setting. Using only the retain set during unlearning has been explored before (e.g., FT, EU-k, CF-k, L1-Sparse), while other methods have also proposed weight dampening followed by a fine-tuning phase, as acknowledged in the paper. Therefore, the main contribution appears to be a new variant of existing approaches rather than a fundamentally new method. However, this would be less concerning if the proposed method consistently outperformed closely related baselines, which further emphasizes the importance of the previous point (see Baseline Selection).

[1] Graves, Laura, Vineel Nagisetty, and Vijay Ganesh. "Amnesiac machine learning." Proceedings of the AAAI conference on artificial intelligence. Vol. 35. No. 13. 2021.

[2] Chavhan, Ruchika, Da Li, and Timothy Hospedales. "Conceptprune: Concept editing in diffusion models via skilled neuron pruning." arXiv preprint arXiv:2405.19237 (2024).

Questions for authors
  • Line 203: You claim that "The contraction itself is agnostic to whether the forget subset is class- or sample-defined." Where is this demonstrated? This also ties into the debate regarding class- vs. instance-level unlearning. How does your method perform in the more common instance-level unlearning setting?
  • What does MIAc\mathrm{MIA}_c measure? What is meant by "confidence" in this context?
Reviewer_fHuKOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes an unlearning method only using retain data in two stages. The first stage utilizes gradient-weight alignment to contract structural weights. The following stage is to fine tune the model with retain data. The method is validated on classification and concept unlearning tasks.

Critical strengths

None

Critical weaknesses

The weight contraction idea is inherited from DASH. The only modifications are introducing this idea into an unlearning scenario and calculating contraction operator using static retain data, which significantly weakens the novelty of this paper.

Questions for authors

They main concern is about the novelty of this paper. Some minor issues are listed below.

  1. Since the solution uses only retain data, how the performance depends on the size of retain data? What if the size of retain data is huge?
  2. Table12 is blank.
Reviewer_r5h1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a machine unlearning method that removes the influence of forgotten data through weight contraction rather than parameter reinitialization. The authors argue that existing weight-intervention methods may unnecessarily disrupt useful representations by resetting selected parameters, and propose using information from retained data to determine which parts of the model should be preserved. The method has two stages. At first, it computes gradients of the retain loss at the original model checkpoint and measures the cosine similarity between each structural weight group and the negative retain gradient. Groups with higher alignment are preserved to a greater extent, while groups with lower alignment are contracted more aggressively. Then, the contracted model is finetuned using retained data to restore utility.

Critical strengths

S1. The observation that models with similar behavioral forgetting can have very different internal representations and relearning dynamics is interesting. I appreciate the authors' attempt to evaluate these aspects jointly rather than relying only on forget accuracy.

S2. The proposed method is relatively simple and intuitive, Namely, using retained data to guide weight contraction, rather than identifying parameters through forget gradients and resetting them.

Critical weaknesses

W1. The paper needs substantial improvements in presentation and organization. Figure 3 and Table 4 are awkwardly positioned. While the main text is less then the page limit, many important details are deferred to a very long appendix. Some tables like Table 12 are missing values, and some calims of experiments and results actually do not exist e.g., Table 11.

W2. The method assumes that weight groups less aligned with the retain loss descent direction can be contracted more aggressively while preserving information relevant to retained data. However, it is not clear that this alignment provides a reliable indication of which weights should be weakened for unlearning. For example, parameters with small retain gradients may already be useful for the retained task, rather than being unimportant. Similarly, retained and forgotten information may share parameters that have strong alignment with the retain gradient. The proposed criterion therefore does not necessarily distinguish information that should be forgotten from information that should be preserved. More importantly, I could not find some concrete experiments that could potentially show that the alignment-based contraction is actually better than simpler alternatives. The authors mention a comparison against mean-matched uniform contraction in Appendix B.9 (Table 11), but the table only describes the ablation axes and does not report the corresponding results. I think this is a particularly important missing experiment. Without comparing against uniform, random, or magnitude-based contraction under comparable intervention strengths, it is difficult to determine whether the improvements come from the proposed alignment criterion or simply from contracting model weights followed by retain-only fine-tuning.

W3. The contribution of the different stages of RECAST is also NOT ablated. The method performs a one-shot contraction followed by an optimization phase on retained data. However, it is unclear how much of the final forgetting performance comes from the contraction itself, how much comes from the subsequent consolidation, and whether the proposed gradient-based selection is necessary at all. I would expect a more systematic comparison between contraction alone, retain-only fine-tuning alone, and their combination, ideally under comparable computational budgets. Similarly, varying the contraction strength and the amount of subsequent fine-tuning would help clarify whether the method identifies a useful unlearning direction or primarily provides a different initialization for optimization.

W4. I think the recovery results are among the most interesting findings of the paper, but the experimental evaluation is too limited to support a general claim of improved resistance to relearning. In Table 3, RECAST requires approximately 15.7 epochs on average to recover 90% of the pretrained forget accuracy, compared with 7.3 for WIG and 4.0 for DEL. But this is limited to a single optimizer and training configuration.

W5. The main experimental evaluation is relatively narrow. The tinyimagenet-200 comparison uses one fixed random 20-class forget split and a single pretrained resnet50 checkpoint.

W6. I have concerns about the baseline comparisons. The paper includes several relevant methods, but important baselines are absent from the main tinyimagenet comparison. In particular, SCRUB and Spotter are closely related to the paper's claims about unlearning, internal representations, and resistance to relearning. Spotter is included in an additional selected-checkpoint comparison in Appendix I, but not in the main multi-seed comparison or the recovery evaluation. SCRUB also appears in supplementary experiments, but is not included in the main comparison.

W7. For FUX experiments agin it is difficult to determine how much of the improvement comes specifically from RECAST. the FLUX implementation uses target-concept prompts, source-model generations, SVD-based restriction, and concept-remapping optimization. These additional components make a lot of changes to the method and its data-access assumptions. For example, Table 4 shows that the SEMU control without contraction already reduces unsafe detections from 593 to 41, while RECAST reduces them further to 16. Although the additional improvement is interesting, much of the suppression appears to come from the underlying concept-remapping pipeline. I would like to see more controlled experiments isolating the effect of gradient-guided contraction, particularly comparisons against equally strong uniform or random contraction within the same low-rank subspace.

W8. The efficiency of the proposed method is not evaluated. Since one motivation for approximate unlearning is avoiding expensive retraining, I would expect a clear comparison of computational costs. Appendix B.9 includes an efficiency table (Table 12), but the wall-clock, memory, and speedup measurements are missing!

Questions for authors

Q1. Can the authors provide some quantitative comparisons between alignment-guided contraction and simpler alternatives, such as mean-matched uniform contraction or random group-wise contraction?

Q2. How sensitive are the recovery results to the choice of optimizer, learning rate, and training schedule?

Q3. Why are SCRUB and Spotter not included in the main recovery comparison, given their relevance to the paper's claims?

Q4. Can the authors provide the missing efficiency measurements from Table 12 and the quantitative outcomes of the ablations listed in Table 11? These seem important for assessing both the practical benefits and the necessity of the proposed design choices.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Withdrawal of Submission

Dear Reviewers,

We sincerely thank you for the time and effort you have dedicated to reviewing our submission, as well as for your valuable and constructive feedback.

We have decided to withdraw our submission from ICLR 2027.

We have also uploaded a revised version of the manuscript incorporating corrections and improvements, which is available on OpenReview for anyone interested in the updated version.

We appreciate your comments and will take them into account as we continue improving our work.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.