acceptodds
Withdrawn from ICLR 2027

Feature Reflection for Machine Unlearning without Retain Data

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Machine unlearning aims to remove designated knowledge while preserving a model's remaining capabilities. This preservation becomes more challenging when retain data are unavailable and unlearning relies solely on the pretrained model and the forget data. To overcome this challenge, we revisit unlearning from a Bayesian perspective and identify two requirements: reducing fit to the forget data while preserving pretrained knowledge. Motivated by these requirements, we introduce Feature Reflection (FR), a simple mechanism that turns bounded-below loss minimization into theoretically grounded forgetting. Specifically, FR reflects the features of the model being unlearned with respect to their pretrained counterparts, perturbs the reflection with multiplicative Gaussian noise, and minimizes the loss on the noisy reflected features. Theoretical analysis establishes that reducing the loss on reflected features raises a lower bound on the forget loss, explicitly connecting bounded-below minimization to forgetting. Further analysis shows that noise perturbation induces a curvature-weighted regularizer on deviations from pretrained features, promoting knowledge preservation. With a simple design and theoretical support, FR outperforms competitive retain-free unlearning baselines over eight vision datasets under different data settings, as well as on the WMDP benchmark.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_S1FQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Feature Reflection (FR), a retain-data-free unlearning method that uses only the pretrained model and forget data. FR reflects the current model’s features around their pretrained counterparts, adds multiplicative Gaussian noise, and minimizes the classification loss on the resulting features. The authors show that, under a fixed linear head, reducing the clean reflected-feature loss raises a lower bound on the original forget loss, while a second-order analysis interprets the noise as a curvature-weighted regularizer on feature deviation from the pretrained model. FR is evaluated on CLIP across eight vision datasets under full-data, few-shot, and long-tailed settings, and on Zephyr-7B-β using WMDP and MMLU. The reported results show a stronger forgetting–retention trade-off than the evaluated baselines, supported by ablations on reflection depth, noise level, regularization variants, and training dynamics.

Critical strengths
  1. Simple and practically useful retain-free formulation. Feature Reflection converts forgetting into ordinary loss minimization on reflected intermediate features, requiring only the pretrained model and forget data. This is computationally straightforward and addresses an important setting in which retain data are unavailable for parameter updates.
  2. Broad and generally consistent empirical validation. The method is evaluated across multiple CLIP architectures, eight vision datasets, several data regimes, and a language-model unlearning benchmark. The reported results and ablations consistently suggest a favorable forgetting–retention trade-off, indicating that the core mechanism is not limited to a single dataset or model family.
Critical weaknesses
  1. FR’s novelty over task-vector negation is limited. Clean FR is an online, feature-space version of fine-tune-and-negate, and the two procedures are equivalent for a linear or tangent model. Reusing the same Jacobian removes TA’s cross-model Jacobian term, but it does not guarantee that the update increases the original forget loss because qq and qrq_r can differ. The paper’s clean analysis also does not directly apply to the noisy objective used in the experiments. The theoretical distinction from task-vector negation is therefore narrower than the paper suggests.
  2. The visual evaluation is unnecessarily limited to CLIP. The main theoretical requirement is a fixed linear classification head, which can also be satisfied by freezing the head of a standard supervised model. The paper could therefore have evaluated FR on conventional ResNet or ViT classifiers and standard class- or sample-unlearning benchmarks. Since all vision results come from the CLIP task-arithmetic setting, it is unclear whether the gains transfer to other models and unlearning settings. The baseline comparison is also incomplete: the paper omits linearized variants of NegMerge and Task Simplex, as well as recent retain-free methods such as POUR and DELETE. These omissions make it difficult to assess either the method’s generality or its advantage over prior work.
Questions for authors

Clarification requests:

  1. Under a linear or tangent approximation, fθ(x)≈fθ0(x)+J0(θ−θ0)f_\theta(x)\approx f_{\theta_0}(x)+J_0(\theta-\theta_0), clean FR appears to be equivalent, step by step, to fine-tuning on DfD_f and then negating the resulting task vector. Is this interpretation correct? If not, which term makes the two procedures different?
  2. Equation 7 replaces TA’s cross-model Jacobian product with JJ⊤JJ^\top, but the alignment is q⊤JJ⊤qrq^\top JJ^\top q_r, which need not be nonnegative when q≠qrq\neq q_r. What guarantee does FR provide on the direction of an individual update? How does this argument extend to the noisy objective actually used in the experiments?
  3. The method is described as retain-free, but retain validation data are used to select the scaling coefficient and hyperparameters. How would these choices be made when no retain data are available at all, and how sensitive are the reported results to this model-selection procedure?

Evidence needed for the current claims:

  1. Can the authors evaluate FR on a conventional supervised model, such as a ResNet or ViT with a frozen linear head, using a standard class- or sample-deletion benchmark and a retrained-without-DfD_f reference? The theory is not specific to CLIP, so this experiment would help determine whether the reported gains transfer beyond the CLIP task-arithmetic setting.
  2. Can the authors compare against linearized variants of NegMerge and Task Simplex, and against recent retain-free methods such as POUR and DELETE on a common benchmark? These are close comparisons needed to assess FR’s advantage over prior work.
Reviewer_Z4VKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

For the machine unlearning scenario where the retain data are always not access, the authors design a feature reflection method to reflect the features of the model being unlearned with respect to their pretrained counterparts. Specifically, they first perturb the reflection using multiplicative Gaussian noise, and then minimize the loss on the noisy reflected features. In this way, the forgetting can be realized using a lower bound on the forget loss and the utility retention can be promoted by a curvature-weighted regularizer on deviations from pretrained features. Also, the authors provide some theoretical interpretation. To validate the effectiveness of the proposed feature reflection, experiments are conducted on the image datasets using CLIP model and WMDP benchmark with large language model. Overall, the trading-off between the forgetting and utility is better.

Critical strengths

(1) The implementation of the proposed feature reflection is simple and easily implemented since it used fixed classification head and the frozen feature as the reference.

(2) The authors provide the theoretical support to explain the relationship between the clean reflected loss and a lower bound on the forget loss under a fixed linear head.

(3) As for the experiments, the authors provides analysis among multiple CLIP architectures and language-model with a shared selection protocol. The results seem to be promising and the ablation studies and training dynamic analysis are impressive.

Critical weaknesses

(1) In proposition 4, the authors claim that the clean reflected loss can be minimized to a lower bound on the forget loss. Howevre, in the real implementation, the noise is added during the training phase and the parameters are rescaled. The authors should give more explanation on how the theoretical result explains the final model’s forgetting performance.

(2) Proposition 4 states that the forget loss tends to infinity as the clean reflected loss approaches zero, and the bounded training objective alone does not rule out excessive forgetting or damage to other capabilities. It is confusing what controls the behavior in the real implementation. It seems that feature normalization, a fixed training budget, noise regularization, or post-training parameter scaling can be the reason. The authors should clear this problem.

(3) As for the knowledge preservation, it seems that the authors only use a second-order expansion to show that noise approximately penalizes deviations from pretrained features on forget examples. This does not directly validate taht the model can keep its behavior on the retain data. Also, figures 4 and 6 support reduced feature deviation on forget data, but do not directly test this connection on retained data.

(4) The author do not use the retain data to optimize the model, but the retain data such as validation dataset is still needed to select the best model. Thus, FR still have dependence on the retained data. This is opposite to the original motivation.

(5) It is unfair to compare with the adapted baseline methods and can not truly validate the effectiveness of the proposed method. Apparently, FR also uses the retained validation data to operate the model selection. Some baselines are adapted by removing their retain-data losses, which changes their original objectives and may weaken their preservation mechanisms.

Questions for authors

(1) Does the bound in Proposition 4 closely track the real forget loss during noisy training and after parameter scaling? Could you provide experiment results showing this relationship?

(2) What keeps the model from forgetting too much and harming the retain ability? How do the number of training steps, noise, and parameter scaling help control this? The authors should clarify these points.

(3) How accurate is the second-order approximation in Proposition 5? When models achieve similar forgetting performance, does adding noise also reduce feature changes on retained data?

(4) If these is also no access to the retained validation data, whether the proposed method can also work efficiently?

Reviewer_1dY7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Machine unlearning (MU) aims to remove certain knowledge from a model, while preserving the rest, which could be challenging if there is no access to the retain data. The authors address this challenge using a Bayesian perspective, and introducing a new mechanism called Feature Reflection (FR). This method aims to reduce the loss on the reflected features of the model, and the authors provide a theoretical connection between reducing this loss and forgetting. Additional noise perturbation is also applied to encourage knowledge preservation. The experimental results show competitive performance across several different benchmarks compared to baselines.

Critical strengths
  • The paper is structured well, and the method has a simple implementation.
  • The proposed method generally achieves the best forget accuracy, mainly on the CLIP models (Table 1 and 2), while achieving better or on-par performance on retain data. This shows the ability of the proposed method to successfully unlearn the forget data.
Critical weaknesses
  • In Proposition 5, it is not fully clear why the weighted term can be interpreted as a regularizer to avoid deviation from the original parameters. For this conclusion to be true, it is assumed that Hii≥0H_{ii}\ge0 , but this assumption is not discussed or elaborated, and it is not clear if this assumption is held regardless of where in the network the features are obtained. Secondly, it would be helpful to elaborate on why the first order term is 0, which can make the proposition and its conclusions more clear. Since this is a core part of the method, clarifying this would help strengthen the paper.

  • Some of the results for baseline numbers reported in Table 1 are not consistent with the original baseline numbers. The original NegMerge paper reports Forget accuracy 8.03 and 6.60 for ViT-B/32 and ViT-B/16, which are much lower that the numbers reported in Table 1. While FR still outperforms them in terms of Forget accuracy, it would be helpful to clarify why this could be the case.

Questions for authors

Can the authors clarify the assumptions and conclusions of proposition 5, and also explain how baseline numbers were obtained across their experiments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.