acceptodds
Under review as a conference paper at ICLR 2027

Do unlearning algorithms learn the right features?

Abstract

Machine unlearning is commonly evaluated with standard post-deletion metrics that compare an unlearned model to one retrained on the remaining data. Such snapshot evaluations can conceal consequential differences in the features the two models encode. We prove that if an unlearned and a retrained model agree on their outputs but encode different features, a single gradient step on their prediction heads can break this agreement. We further show that deleting an arbitrarily small fraction of the data can cause retraining to select qualitatively different features, so matching retraining can require learning features that the full model never had. We then introduce a measure of feature recovery that separates missing from extraneous features. Across eight unlearning methods, models that perform well on standard metrics often fail to recover the retrained features. This mismatch predicts deviations from retraining under subsequent relearning and downstream adaptation, and directly correcting it systematically reduces these deviations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine unlearning, learned representations, feature learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QMF4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks a question the unlearning literature has largely dodged: when an unlearned model matches a retrained model on standard output metrics, has it actually recovered the features retraining selects, and does that mismatch matter once training resumes? The authors make four moves: Why features matter; Deletion can require learning new features; A directional feature-recovery metric and feature mismatch to behavior under adaptation

Critical strengths

Genuinely fresh framing. Snapshot output agreement isn’t enough is a simple, compelling thesis, and the paper nails it down properly rather than leaving it as rhetoric.

Deleting an arbitrarily small fraction can require learning a feature the full model never had is an interesting idea, cleanly proved, and knocks the subtractive view of unlearning on the head. The GA runtime ratio is a tidy quantitative statement.

Critical weaknesses

The metric hangs entirely on one feature operationalisation. Everything leans on Jiang et al.’s PCA-response clustering, and it’s sensitive to K , γ_corr , κ , and the complete-linkage choice. The paper ablates k and group size and argues qualitative robustness, which helps but doesn’t close the issue. would sparse-autoencoder features, linear probes, or RSA yield the same method ordering? The paper doesn’t test an alternative feature definition, only an alternative similarity measure (CKA). That’s the single biggest vulnerability from my piont.

Theory and empirics are decoupled. The XOR construction shows deletion can require new features; it isn’t claimed to be the mechanism behind the ViT/CIFAR-100 results, and the gap between a quadratic network under gradient flow and a fine-tuned ViT is large. The theory section is valuable in its own right but reads a bit like a separate paper bolted onto the empirical study.

Questions for authors

Does the method ranking under D_feat line up with a different feature definition, say sparse autoencoders or linear-probe directions, at least on the main ViT-Tiny/CIFAR-100 setup? Even one alternative would go a fair way towards shoring up the metric’s credibility.

For the predictive scatters in Fig. 2, are the relationships actually significant with only about n≈10 method-level points, and does the ranking hold up under leave-one-method-out?

Does the partial-relearning conclusion still stand under a second protocol, like full fine-tuning rather than a fresh head on a frozen representation? The frozen-representation design is neat, but it’s also the most favourable case for the paper’s mechanism.

In the XOR model, how touchy are Thm 4.2/4.3 to the exactly-balanced scaling and the quadratic architecture? A quick note on robustness would help bridge theory and practice.

Reviewer_CByQOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether unlearning methods recover the same features as retraining. The authors provide theoretical analysis and introduce feature discrepancy metrics to compare different unlearning methods. The results show that methods performing well on standard unlearning metrics may still fail to recover retrained features, affecting relearning and downstream performance.

Critical strengths
  1. The paper provides an interesting theoretical analysis showing that models with similar predictions may learn different features and behave differently during further training. It also shows that removing even a small amount of training data can change the features learned through retraining.

  2. The authors evaluate different unlearning methods across different model architectures, providing a broader comparison of their feature-recovery behavior.

Critical weaknesses

1. Limited experimental diversity. The experiments mainly focus on CIFAR-100 and SVHN. Although the authors evaluate different architectures and deletion fractions, the evaluation does not cover more challenging datasets or sequential unlearning requests. This makes it difficult to assess whether the observed feature-recovery differences between unlearning methods would also hold in more complex unlearning settings.

2. Sensitivity of the feature discrepancy metrics. The proposed metrics depend on PCA-based feature extraction, clustering, and matching thresholds. However, the paper does not provide sufficient analysis of how these choices affect the results. Since the main conclusions rely on these metrics, it is important to establish whether the observed differences between unlearning methods remain consistent under different parameter settings.

Questions for authors
  1. How sensitive are the proposed feature discrepancy metrics to the number of PCA components, clustering parameters, and matching thresholds? Would the main results remain consistent under different settings?

  2. How would the results change if the authors considered more challenging datasets or sequential unlearning requests? Would the differences between unlearning methods still hold?

  3. Minor presentation comments. The paper is generally understandable, but the presentation could benefit from some improvements. The main contributions could be summarized more clearly in the introduction, and the methodology and experiments could be better organized. Some appendix figures are difficult to read due to small text, and the names and abbreviations of unlearning methods should be used consistently throughout the paper. There are also a few typos and formatting issues, such as “fro” instead of “for” in line 95, “IN” instead of “In” in line 96, and a missing space before “or” in line 102. Addressing these issues would improve the paper's overall clarity and readability.

Reviewer_mXEHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates whether approximate unlearning methods recover the internal features of models retrained without the forget data. The authors provide theoretical results showing why prediction-level agreement may not persist under further training and why unlearning can require learning new features. They propose feature-recovery metrics and evaluate eight unlearning methods on image classification tasks. The results suggest that standard unlearning performance does not necessarily reflect feature recovery, and that representation mismatch can affect subsequent relearning and downstream performance.

Critical strengths
  1. The paper makes a useful distinction between forgetting information and recovering the features that retraining would have learned. I particularly liked the theoretical observation that removing data may require learning new features rather than just suppressing existing ones.

  2. The distinction between missing and unmatched features is useful. The experiments are beyond standard unlearning evaluations, and the feature intervention results provide additional evidence that these differences matter for subsequent model behavior.

Critical weaknesses
  1. I think the novelty of some contributions is limited and needs to be adjusted in the paper. Prior work has already shown that successful output-level forgetting may leave different internal representations and that forgotten information can re-emerge during further training For example From Dormant to Deleted (Siddiqui et al., 2025), An Illusion of Unlearning? (Gao et al., 2026), and Unlearning Isn't Deletion (Xu et al., 2026). The proposed feature metric also builds on the model-defined feature framework of Jiang et al. (2024). While I find the focus on missing retraining features interesting, I am not fully convinced how much new insight the proposed metric provides beyond existing representation similarity measures such as CKA. A direct comparison of their ability to predict subsequent behavior would help better illustrate the novelty and value of the proposed metric.

  2. The experimental validation does not support all the broader arguments about subsequent training and feature recovery. Most adaptation experiments freeze the backbone and only train a new classification head. It is unclear whether the same conclusions hold under full-model finetuning, where representations can change. Also, while the theory shows that retraining may learn new features, the empirical results mainly demonstrate representation mismatch rather than directly showing that acquiring new features is necessary for successful unlearning.

Questions for authors
  1. How does the proposed feature discrepancy compare with simpler representation similarity measures, such as CKA, in predicting relearning and downstream performance? Does the directional formulation provide a clear advantage?

  2. Do the observed relationships between feature mismatch and subsequent performance also hold when finetuning the full model rather than only a linear head?

  3. Can the authors provide more direct empirical evidence that retraining learns new features absent from the full model, and that failing to recover these specific features explains the observed unlearning failures?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.