acceptodds
Under review as a conference paper at ICLR 2027

Matryoshka Machine Unlearning in Classification and Generation

Abstract

Machine Unlearning (MU) removes the effect of data on a trained model without retraining from scratch. Earlier works operate on the full embedding width for unlearning. Therefore, nothing in them prevents deleted content from remaining decodable at a narrower feature scale. To address this, we propose Matryoshka Machine Unlearning (MMU), a nested prefix embedding that freezes the source model as a privileged multi-scale teacher. The student tries to mimic the teacher without having access to teacher-privileged information. The main idea is to arm the model with richer representations via coarse-to-fine nested embeddings. MMU is a plug-and-play method applicable on top of any MU method. Across three datasets, two backbones, three types of deletion, and fourteen unlearning baselines, MMU is the only method without a failure mode. Under class deletion, MMU drives forgetting accuracy to exactly 0% across all six dataset–backbone configurations while maintaining the highest retained accuracy among all baselines. Its real-world practicality has been demonstrated on the Physical Artificial Intelligence (PAI) dataset we collected. Additionally, for the image-generation task of removing dangerous content, it matches the best baseline, as shown in Fig. 1.

Primary area:
learning theory
Keywords:
Machine Unlearning, Representation Learning, Nested Learning, Image generation, Image Classification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ac4QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Matryoshka Machine Unlearning (MMU), which applies unlearning objectives to multiple nested prefixes of a model’s representation. A frozen copy of the original model serves as a teacher. On retained data, the student combines supervised learning with distillation; on forgotten data, it is pushed away from the teacher through a margin-limited KL objective. The authors also extend the approach to diffusion models by applying related objectives to noise predictions at different channel widths. Experiments cover three classification datasets, two backbones, three deletion granularities, and concept suppression in image generation. The strongest results concern class deletion and retained utility, while subclass and instance deletion remain less successful.

Critical strengths
  1. The paper addresses a meaningful limitation of output-based unlearning with a concrete method. Changing a model’s final prediction does not establish that information is no longer recoverable from its representation. Applying constraints at several nested widths is a reasonable attempt to address this issue. The construction is straightforward, and auxiliary classifiers can be removed after training, making the approach worth investigating further.
  2. The class-deletion results show a promising balance between suppression and retained utility. In Table 2, MMU achieves zero class-forget accuracy across all six dataset–backbone combinations while maintaining competitive retained accuracy. The paper also evaluates per-example attacks, relearning, and sequential deletion, providing a broader assessment than forget accuracy alone. These findings support MMU as a potentially useful class-deletion method, although they do not establish the broader claims made in the abstract.
Critical weaknesses
  1. The central representation-level claim is not directly tested, and the contribution of nesting is not isolated. The paper is motivated by the possibility that deleted information remains decodable from short representations after the final prediction changes. However, its objectives constrain predictions through trainable classifiers. In MMU-1, each width has an independent classifier, so the heads could absorb the changes while the encoder remains informative. Figure 3 evaluates the existing readouts rather than newly trained probes; SCRUB also achieves approximately zero forget accuracy at the widths shown. MMU additionally differs from SCRUB in its margin-limited objective, width weighting, and search space. Table 12 lists 96 candidate configurations for MMU versus 24 for SCRUB. The reported controls do not clearly separate these factors. Without a matched single-width hinge baseline and frozen-representation probing against Retrain, I cannot determine whether the main proposed mechanism accounts for the observed improvements.
  2. Important experimental results are insufficiently reconciled, and the overall claims exceed the evidence. Table 1 reports Retrain forget-set values of 5.21% and 7.65% for random deletion, which the accompanying text interprets as classification accuracy, despite test accuracies of 94.44% and 92.16%. Table 5 instead reports Retrain forget accuracies of 94.27% and 92.58%, with MMU remaining at 100% in both settings. The Table 1 values may reflect an error-rate convention, but the manuscript does not establish this or explain the different MMU outcomes. This ambiguity directly affects the claimed effectiveness of random deletion. Other results also show substantial limitations: MMU’s subclass-deletion U-LiRA AUC remains around 96% on CIFAR-100, and Table 8 reports reappearance of the first deleted condition in 3/16 samples after sequential deletion. Although the conclusion acknowledges some limitations, these findings do not support the abstract’s claim of having no failure mode. The evidence currently supports advantages in particular operating regimes rather than reliable unlearning across deletion granularities.
Questions for authors

These questions concern the interpretation and support of the current claims; their answers could change my assessment.

  1. Metric and protocol clarification: Does DfD_f in Table 1 denote accuracy or error rate? Do Tables 1 and 5 use different model variants, training configurations, or selection procedures? Please specify the differences and explain why MMU retains 100% forget-set accuracy in Table 5.
  2. Attribution of the improvement: What benefit does nesting provide over a single-width method using the same hinge objective, data access, training schedule, and hyperparameter-search budget? Do the existing experiments include a direct comparison that isolates this contribution?
  3. Evidence for representation-level removal: Have you evaluated newly trained probes on frozen representations at each width, using Retrain as the reference? If not, would you agree that the current evidence supports improved behavior under the evaluated readouts, rather than establishing that the deleted information is no longer decodable?
4 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for identifying the missing controls. We conducted the requested matched single-width hinge ablation and frozen-representation probing, together with head-sharing, margin-sensitivity, and robustness experiments. Gap = |forget accuracy − Retrain forget accuracy|.

W1: Does nesting itself improve unlearning? We held the hinge objective, data, training schedule, and 96-configuration grid fixed; only the supervised widths differ.

Table 1. Gap to Retrain (percentage points, test split; lower is better).

Dataset / BackboneClass: FullClass: NestedSubclass: FullSubclass: Nested
RTI / ResNet-180.00.09.32.3
RTI / CNN49.90.16.33.7
CIFAR-100 / ResNet-1830.70.052.342.7
CIFAR-100 / CNN67.30.046.739.3

Nesting reduces the class-deletion gap by 30.7–67.3 points in the three configurations where full width fails and improves all four subclass configurations by 2.7–9.7 points. Retain-accuracy changes remain within ±1.2 points across 12 settings. Instance deletion shows no consistent improvement.

Robustness. For margins δ ∈ {16, 64, 256, 1024}, nesting reaches zero validation class-deletion gap at δ ≥ 256 in all four configurations, versus one for full width. Hyperparameter-grid success rates (class gap ≤ 2 pts, utility within 2 pts) are 28/27/9/5% for nesting versus 4/0/0/0% for full width (RTI ResNet-18/CNN, CIFAR-100 ResNet-18/CNN).

The effect persists on Matryoshka-trained sources whose prefixes receive explicit classification supervision: the three nontrivial class-deletion gaps change from 58.7→0.0, 41.3→0.0, and 67.7→6.3.

Can the auxiliary heads absorb forgetting? We compared independent per-width heads against a single tied head under Eq. 6 on Matryoshka-trained sources.

Table 2. Class-deletion gap to Retrain (percentage points).

Dataset / BackboneIndependent (Eq. 6)Shared (Eq. 6, tied head)
RTI / ResNet-180.00.3
RTI / CNN0.056.7
CIFAR-100 / ResNet-180.058.3
CIFAR-100 / CNN6.368.3

These shared-head results are for the tied-head hinge control, not the separately evaluated MMU-2 variant.

Fresh probes. Linear probes trained on held-out non-member data recover forgotten-class information from frozen features. On RTI/ResNet-18, full-width probe-accuracy excess over Retrain is 40.6 pts for MMU, 39.9 for SCRUB, and 40.4 for Original.

W2: Which claims are supported? We agree that the statement of having "no failure mode" exceeds the evidence. We will remove it and distinguish predictive agreement with Retrain, membership privacy, and feature-level information removal.

Class deletion. The nested-hinge formulation attains a 0.0–0.1-point gap to Retrain across RTI and CIFAR-100 with both CNN and ResNet-18, maintaining competitive retained utility. The controlled full-width hinge fails substantially in three configurations.

RTI subclass deletion. Retrain is the appropriate reference, not zero accuracy: a model that never saw a deleted subclass can still recognize it through features learned from retained sibling subclasses.

Table 3. RTI subclass forget accuracy (%) and gap to Retrain (pts).

MethodResNet-18 accuracyGap ↓CNN accuracyGap ↓
Retrain86.00.089.00.0
Bad Teacher30.056.044.344.7
SCRUB (96)91.75.794.35.3
Nested hinge (96)85.72.392.73.7

Gaps are computed per seed before averaging. MMU comes closest to Retrain in both settings, whereas Bad Teacher substantially over-forgets. MMU's membership-inference accuracy is also near Retrain: approximately 50.6–53.4%, versus 49.2–51.1% for Retrain.

CIFAR-100 subclass deletion. The nested hinge still under-forgets, with gaps of approximately 39–43 points and substantial membership leakage (the submitted U-LiRA AUC is near 96%).

Instance deletion. Membership-inference accuracy remains approximately 60–68% for MMU, versus roughly 50% for Retrain. Several baselines also retain membership signals, and nesting provides no consistent improvement for individual-record removal.

Sequential deletion. In the generation experiment, the first removed condition reappears in 3/16 samples after sequential deletion. We agree that this constitutes incomplete removal and will report it as a limitation.

Revised claims. MMU provides strong predictive class-deletion results, improved RTI subclass agreement with Retrain, and a controlled benefit from nested supervision. It does not establish universal success across deletion granularities, membership privacy, or resistance to fresh feature probes.

Q1

Metric. In Table 1, the D_f column reports forget error (100 − accuracy, the "UA" convention of SalUn); Table 5 reports forget accuracy. The manuscript's text wrongly interpreted Table 1 as accuracy. We will use forget accuracy throughout and state the definition in every caption. A consistent conversion leaves the gap to Retrain unchanged: |(100 − a) − (100 − b)| = |a − b|.

Protocol: the two tables are different experiments.

Table 1Table 5
Trials10 (seeds 2–11), each with its own source and Retrain1 (seed 2)
Hyperparametersselected on pilot seed 1 using validation metricsfixed, no search
MMU variantnested MMU, as in the submissiona full-width, two-teacher variant (no nesting; a retain-fine-tuned teacher supplies attraction targets; lr 1e-3, T = 1, margin 0.002)
Metricforget errorforget accuracy

The Retrain values therefore differ for two reasons: the metric, and averaging. Table 1's 5.21% error is the 10-trial mean (94.79% accuracy), while Table 5's 94.27% is the seed-2 trial alone (5.73% error). At 50% forgetting, the corresponding figures are 92.35% (mean) and 92.58% (seed 2). Labelling the Table 5 variant "MMU" was misleading; the revision will name it separately.

The MMU row of Table 1 is wrong. The reported means and gaps do not match our logs, and we found no run that produced them. The logged ten-trial values are:

Random deletionReported (error)Logged forget accuracyLogged errorGap to RetrainD_rTest (gap)
10%4.19 ± 0.00100.00 ± 0.000.00 ± 0.005.21100.00 ± 0.0095.02 ± 0.19 (0.58)
50%4.39 ± 0.5399.61 ± 0.530.39 ± 0.537.2699.82 ± 0.3193.81 ± 0.81 (1.64)

Why MMU stays at 100% forget accuracy, and why the standard deviation is zero. Both MMU variants leave the predictions on the forgotten examples unchanged. The source model fits its training set perfectly. Randomly deleted examples follow the same distribution as retained ones, and MMU's preservation term keeps them at the teacher's prediction. At 10% forgetting this happened in all ten trials, so the forget error is exactly 0.00 each time; the zero standard deviation is real. GA, BE and BS behave the same way, which is why they also show 0.00 ± 0.00. High forget accuracy is not by itself a failure in random deletion, because Retrain still classifies 94.79% of the forgotten examples correctly. But MMU's forget accuracy stays 5.21 points above Retrain's, which is the same as leaving the model untouched.

Consequence. With the corrected values, MMU has no advantage in random forgetting. Its gaps (5.21 and 7.26) are among the largest in Table 1. ℓ1-sparse comes closest to Retrain at both ratios (gaps 3.25 and 0.08), followed by SalUn at 10% (4.59) and RL at 50% (2.00). We will correct Table 1, remove the "MMU best (4x)" summary and the related claims, and state that MMU is not effective for random instance deletion.

Q2

We performed the requested controlled comparison with identical data, training schedule, hinge objective, and 96-configuration search budget.

Nesting reduces class-deletion gaps from 49.9→0.1, 30.7→0.0, and 67.3→0.0 in the three configurations where full width fails. It also improves all four subclass configurations by 2.7–9.7 points, with retain-accuracy differences within ±1.2 points.

Margin-sensitivity and hyperparameter-grid analyses further confirm that these improvements are not confined to a single selected configuration. This directly isolates the effect of nested supervision under the bounded objective.

Q3

Yes. We trained linear probes on frozen width-m representations using held-out non-member data, then evaluated them on forgotten examples against the corresponding Retrain probes.

For RTI/ResNet-18 class deletion:

MethodFull-width probe excess over Retrain
Original40.4 pts
MMU40.6 pts
SCRUB (96)39.9 pts
Bad Teacher41.2 pts

Deleted-class information remains linearly recoverable. We therefore agree that the evidence supports forgetting under trained predictive readouts, not complete removal of recoverable information from the representation.

Revisions. We will adopt Eq. 6 with independent per-width heads, add controlled nesting and head ablations, report matched-budget SCRUB, include fresh probes, correct the random-forgetting inconsistencies, and narrow the unsupported claims.

We thank you again for your feedback. We believe these experiments directly address the reviewer's questions and provide stronger evidence for MMU's nested optimization. We would be grateful if the reviewer would reconsider their score.

Thank you for the additional experiments and clarifications. The matched single-width ablation provides useful evidence that nested supervision improves class-deletion performance, and I appreciate the authors' efforts to correct the experimental results and narrow their claims.

However, the new linear-probing results show that forgotten-class information remains highly recoverable from MMU's frozen representations, with performance nearly identical to the original model. This leaves my main concern about representation-level forgetting unresolved. Moreover, the limitations in subclass, instance, and random deletion remain.

Overall, the rebuttal improves the empirical support for the method but does not sufficiently address my fundamental concerns about its core motivation and contribution.

AuthorsOpenReview ↗

Follow-up: Representation-Level Forgetting

We conducted an additional experiment to determine whether MMU can reduce recoverable forgotten-class information, rather than only changing predictions under its trained readouts.

Method. We augment MMU (Eq. 6) with a probe-adversarial objective. At each nested width, a linear probe is periodically refitted using data available to the unlearner (DfD_f and 10% of DrD_r). The encoder is penalized when the probe assigns the deleted class a probability above a threshold τ\tau. Retrain is not used in this loss. We select λ\lambda and τ\tau using validation data.

We compare three variants: the original MMU, MMU with the adversary applied only at full width, and MMU with the adversary applied at every width. We evaluate frozen representations using fresh linear and two-hidden-layer MLP probes, fitted on validation data and evaluated on deleted training examples and deleted-class test images. Experiments cover four class-deletion settings and three seeds.

Probe excess over Retrain on deleted examples (accuracy difference in percentage points, averaged across widths; linear / MLP; lower is better):

SettingMMUAdv. full widthAdv. every width
RTI · ResNet-1848.3 / 45.441.6 / 37.639.5 / 35.6
RTI · CNN48.7 / 42.935.1 / 30.632.9 / 29.4
CIFAR-100 · ResNet-1840.6 / 38.734.9 / 32.332.1 / 27.7
CIFAR-100 · CNN33.1 / 32.226.2 / 23.724.5 / 22.4

1. Representation-level recoverability decreases, but remains substantial.

Compared with the original MMU, the per-width adversary reduces probe excess in all 12 seed–setting pairs, for both linear and MLP probes. Average reductions are 10.4 and 11.0 points on deleted examples, respectively. The corresponding reductions on deleted-class test images are 8.6 and 7.6 points.

The improvement extends beyond linear probing, but does not establish information removal. Probe excess remains substantial relative to Retrain, and stronger probes could recover additional information.

2. Nested adversarial supervision provides a small additional benefit.

Applying the adversary at every width outperforms applying it only at full width in 11/12 paired comparisons on deleted examples, for both probe families. Average improvements are approximately 2.2–2.3 points, increasing to 3.6–4.1 points at the narrowest width.

This provides evidence that per-width supervision can improve representation-level probe resistance. However, the improvement is small relative to seed variability, and we do not claim a statistically established advantage.

3. Most of the reduction comes from the adversarial objective itself.

On CIFAR-100 CNN, a full-width hinge model equipped with the same adversary achieves nearly identical probe excess to MMU-adv (24.4 / 22.3 versus 24.5 / 22.4). However, its deployed class-forgetting gap is 63.7 points, compared with 0.0 for MMU-adv.

Thus, the adversarial objective accounts for much of the reduction in recoverability, while the nested formulation retains its advantage in achieving class forgetting under the deployed readouts.

4. Utility and limitations.

MMU-adv maintains a 0.0-point deployed class-forgetting gap across all four settings. Relative to Retrain, test accuracy decreases by 1.9 and 2.5 points on RTI ResNet-18 and CNN, respectively, and by 0.2 and 0.4 points on CIFAR-100. The RTI CNN result exceeds our 2-point utility tolerance on the test set.

These are exploratory three-seed experiments using a limited eight-configuration adversarial grid. We have not established complete representation-level removal. The earlier limitations concerning subclass, instance, and random deletion also remain; these experiments do not resolve them.

Implications for our contribution.

We agree with the reviewer that the original MMU objective should not be presented as removing forgotten information from the representation. Our evidence now supports a more precise distinction:

  • Nested optimization: The matched single-width ablation establishes that nesting improves the effectiveness of the bounded forgetting objective under deployed predictive readouts.
  • Representation-level probing: The original MMU leaves substantial forgotten-class information recoverable, despite successful deployed class forgetting.
  • Adversarial extension: Explicit per-width adversarial supervision consistently reduces recoverability under the evaluated probes, but the remaining excess demonstrates incomplete removal.

We will incorporate these findings and revise the motivation and claims accordingly, distinguishing deployed predictive forgetting, membership privacy, and representation-level recoverability. We appreciate the reviewer's insistence on this distinction. The additional experiments do not fully resolve the concern, but they quantify the remaining limitation and clarify what the nested formulation contributes.

Reviewer_deBpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that existing unlearning methods only enforce deletion at the final classifier, so deleted content can still be read out from a narrower prefix of the same embedding vector. To fix this, it borrows the nested prefix idea from Matryoshka representations. A frozen copy of the source model acts as teacher, the student is pulled toward it on retained data and pushed away from it on forgotten data with a hinge margin, and this loss is applied at every prefix width so that deletion happens at every granularity at once. The method, called MMU, is evaluated on class, subclass and instance deletion over CIFAR-10, CIFAR-100 and a self collected haptic dataset with a small CNN and ResNet-18, plus a nudity suppression experiment in Stable Diffusion.

Critical strengths

(1) The evaluation protocol is broader than most unlearning papers. Three deletion granularities, three datasets, two backbones, validation based selection for every baseline, and a per example attack against a paired retrained model is the right way to audit approximate unlearning, and the authors are candid about where their method still leaks.

(2) The extension to diffusion models is worked out with care. The observation that zero padding truncated latent channels through an unmodified decoder injects a constant shift through the normalization layers is correct and not obvious, and restricting the decoder statistics to the prefix is the right fix.

Critical weaknesses

(1) The claim that motivates the whole paper is asserted but never shown. The paper says prior methods leave deleted content decodable from a short prefix of the embedding. The only evidence offered is one figure that evaluates each prefix with a sliced classifier. But for a model that was never trained with nested readouts, the first few coordinates of its embedding have no special meaning, so slicing the classifier is not a measurement of what an attacker could decode. More importantly, that same figure shows SCRUB and the retrained model already near zero forget accuracy at every width, so the failure mode the method is meant to fix does not appear for the closest baseline. Without a probe based leakage test on the baselines, MMU reduces to SCRUB plus a hinge margin applied several times, and neither of those two ingredients is ablated on the main benchmarks. In particular there is no row with a single full width readout, and the only ablation is on random forgetting, where MMU coincides with the source model anyway.

(2) The numbers also do not hold together. The random forgetting table and the retrain calibrated table both report forget accuracy for the same configuration, yet the two values cannot be reconciled under any single definition of forget accuracy. In the first of those tables, moreover, three baselines reach zero forget accuracy with perfect retain accuracy and test accuracy above the retrained model, which is not a plausible outcome for those methods. The hyperparameter table then shows that MMU searched four times as many configurations as SCRUB, with the choice of readout head and loss weighting folded into model selection rather than ablated, so part of the margin over SCRUB may simply be search budget.

(3) Finally, the evaluation is too small for a claim about representation capacity. The only backbones are a four layer CNN with a 64 dimensional embedding and a ResNet-18 trained from scratch on CIFAR. By contrast, the baselines the paper compares against were evaluated at Tiny-ImageNet or ImageNet scale in their own papers, and the Matryoshka idea itself was demonstrated on ResNet-50 and ViT at ImageNet scale, which is the regime where a short prefix might plausibly carry something. The haptic dataset is self collected from a virtual environment with twenty participants and has no stated release plan, so the real world claim rests on simulated data nobody can check, and the participant video comes with no consent or approval statement. As for the generation experiment, it compares against SalUn and a random mask only, on 60 prompts, and the random mask actually forgets better, which undercuts the abstract's claim of matching the best baseline. The claim of being the only method without a failure mode likewise does not survive the subclass deletion results, which are far from the retrained model and worse than a baseline in every cell.

Questions for authors

(1) Please reconcile the two tables that report forget accuracy for the same random forgetting configuration, state which definition each uses, and explain the zero standard deviation over ten trials.

(2) How do three baselines reach zero forget accuracy, perfect retain accuracy and test accuracy above the retrained model in the random forgetting table?

(3) Were SCRUB and fine-tuning given a search budget comparable to MMU, and does the readout head choice change the conclusions?

(4) Was ethics approval or informed consent obtained for the participant video, and will the dataset be released?

5 comments
AuthorsOpenReview ↗

Response 1/3

We thank the reviewer for the detailed and technically precise assessment. The review identifies two important questions: whether prior methods leave forgotten information accessible at narrower embedding widths, and whether nesting itself improves unlearning beyond existing teacher–student objectives. We tried to address them in the following experiments:

To investigate these questions, we conducted additional experiments on RTI and CIFAR-100, using ResNet-18 and CNN, three deletion granularities, and three random seeds. Within each setting, methods share the same source checkpoint, Retrain reference, validation-only selection protocol (2-point utility tolerance), and held-out test evaluation.

W1: Is the motivating representation-leakage failure observed?

We agree that slicing the classifier of a conventionally trained network does not, by itself, establish whether forgotten information is recoverable from its truncated features. We therefore conducted two complementary experiments.

1. Prefix evaluation on standard and Matryoshka-trained sources.

We evaluated the same forgotten examples using both conventional source models and Matryoshka-trained source models. For the latter, Original and Retrain were trained with classification supervision at every embedding width, making each prefix a functioning classifier with retain accuracy within 0.5 percentage points of full-width accuracy.

Across the examined baselines—including fine-tuning, Bad Teacher, SSD, SalUn, UNSIR, UniCLUN, and SCRUB—we did not observe the hypothesized systematic failure in which a narrower sliced readout exposes a deleted class that is suppressed at full width. In particular, budget-matched SCRUB achieves zero forget accuracy across the evaluated widths for class deletion on Matryoshka-trained sources.

2. Fresh linear probes on frozen representations.

To test recoverability independently of the original classifier, we trained fresh linear probes on frozen width-m features using held-out non-member data. We then evaluated these probes on the forgotten examples and compared them against probes trained on Retrain.

Table A. RTI/ResNet-18 class deletion: full-width probe-accuracy excess over Retrain (percentage points).

MethodProbe excess
Original40.4
SCRUB (96)39.9
MMU40.6
Bad Teacher41.2

The probes recover substantial deleted-class information from the representations of every method shown. MMU and SCRUB exhibit nearly identical full-width probe recoverability, despite suppressing deleted-class predictions through their trained classifiers. These findings establish an important distinction: enforcing forgetting on predictive readouts does not guarantee removal of recoverable information from the underlying features. This limitation applies to MMU as well as the evaluated baselines.

Consequences for the paper. We will remove the claim that prior methods leave deleted content decodable from a short prefix, together with the sliced-classifier figure used as evidence for it. The revised paper will present nesting as an optimization constraint and will justify it by the ablations in Part 2, not by leakage in baselines. We will also state that no approximate method we tested, MMU included, removes the deleted class from the features.

W1: Does nesting itself contribute beyond SCRUB?

To directly address the missing ablations, we evaluated full-width and nested representations under both unbounded and margin-limited forgetting objectives.

All hinge comparisons use the same 96-configuration grid and validation-only selection rule. The SCRUB comparison is included separately because its optimization schedule and tuned distillation parameters differ.

Table B. Controlled objective comparison: absolute forget-accuracy gap to Retrain (percentage points, lower is better).

Dataset / BackboneDeletionSCRUB (96)Submitted MMUFull-width hingeNested hinge
RTI / ResNet-18Class0.00.00.00.0
RTI / CNNClass0.00.149.90.1
CIFAR-100 / ResNet-18Class0.00.030.70.0
CIFAR-100 / CNNClass0.00.067.30.0
RTI / ResNet-18Subclass5.72.09.32.3
RTI / CNNSubclass5.33.06.33.7
CIFAR-100 / ResNet-18Subclass41.039.352.342.7
CIFAR-100 / CNNSubclass35.337.746.739.3

The most important comparison is between the full-width hinge and nested hinge, which isolates the contribution of nested supervision under the same bounded objective. This establishes a controlled contribution: nested supervision substantially improves the effectiveness of the bounded forgetting objective, rather than merely adding multiple outputs to the existing teacher–student formulation.

Q3: Were SCRUB and fine-tuning given comparable search budgets?

Yes. Please check the code. We will add an explicit sentence to the revised version stating this.

Does the head formulation change the conclusions?

Yes, and we investigated it directly.

We compared independent per-width heads against a single shared sliced head on Matryoshka-trained source models, where each source prefix was explicitly supervised.

Table D. Matryoshka-trained sources: class-deletion gap to Retrain (%) under the 96-configuration hinge ablation.

Dataset / BackboneIndependent headsShared sliced head
RTI / ResNet-180.00.3
RTI / CNN0.056.7
CIFAR-100 / ResNet-180.058.3
CIFAR-100 / CNN6.368.3

The shared-head formulation fails substantially in three of four configurations, even though the source model has meaningful prefixes.

Q1: Why do Tables 1 and 5 report different forget accuracies?

In Table 1, the reported values are consistent with forget classification error (100 − accuracy), whereas Table 5 reports forget classification accuracy. For example, Retrain's Table 1 values correspond to accuracies of 94.79% at 10% forgetting and 92.35% at 50% forgetting. Table 5 reports 94.27% and 92.58%, respectively.

Importantly, converting both a method and Retrain from accuracy to error does not change their absolute gap:

|error(method) − error(Retrain)| = |accuracy(method) − accuracy(Retrain)|.

Thus, the labeling error changes the interpretation of the D_f column, but does not automatically invalidate a consistently computed absolute-gap ranking.

Q2: How can GA, BE, and BS achieve zero forget values with perfect retained accuracy?

Under the forget-error interpretation, a reported value of D_f = 0.00 means that the model correctly classifies the forgotten examples, not that it has successfully forgotten them. This explains why GA, Boundary Expanding, and Boundary Shrink can simultaneously report zero forget error, 100% retained training accuracy, and test accuracy around 94.9%. These values indicate that the deleted examples remain correctly classified. The slightly higher test accuracy relative to Retrain is also not inherently paradoxical: Retrain was trained without the deleted data, whereas the approximate methods were initialized from the source model trained on the original dataset.

W3: Experimental scale and generality

Please note that prior works such as SCRUB, Salun, Bad Teacher, SSD, etc., are limited to these architectures and datasets. This is a widely accepted protocol in the MU community. We agree that the current classification experiments do not establish how MMU scales to larger architectures such as ResNet-50 or ViT on ImageNet-scale datasets. We didn't claim we have a universal MU method that can close the unlearning problem for every setting and architecture. Our experiments directly support the proposed objective on the evaluated CNN and ResNet-18 architectures. We will restrict the corresponding representation-capacity claims to this setting and explicitly identify larger-scale generalization as an open question.

Q4: RTI dataset, informed consent, and release

RTI dataset has ethical approval. However, we have not yet publicized the dataset. We cannot disclose which institution, as it is anonymous to us. The submission already states that RTI will be released upon acceptance. It will be available in our Git repository.

Generation: Forgetting–retention trade-off

Our generation experiment includes 90 held-out prompts, fixed before evaluation: 30 nudity prompts, 30 clothed-person prompts, and 30 unrelated-scene prompts. Each prompt is sampled with two noise draws, giving 60 generated outputs for the nudity-detection subset.

The results reveal a useful trade-off rather than a universal winner.

Table E. Held-out nudity suppression and retained-generation quality.

MetricMMUSalUnRandom mask
Nudity detection ↓3.3%0.6%0.0%
Retain CLIP ↑0.3230.3130.315
Retain LPIPS to Source ↓0.4780.5470.511

The random mask achieves stronger nudity suppression. However, MMU preserves retained-generation quality more effectively according to both retained CLIP similarity and LPIPS distance to the source model.

Clarifying the scope of MMU's claims

We also agree that the statement describing MMU as the only method without a failure mode is too broad. The additional experiments establish strong predictive class deletion and improvements over full-width bounded unlearning. However, the remaining limitations are important:

  • On CIFAR-100 subclass deletion, MMU remains substantially farther from Retrain than some alternative methods.
  • On instance deletion, membership-inference accuracy remains above the Retrain reference.
  • Fresh linear probes can recover deleted-class information from the features after unlearning.

We will therefore distinguish predictive agreement with Retrain, membership privacy, and feature-level information removal rather than treating them as interchangeable forms of successful unlearning.

Summary

The new experiments and revisions directly address the review's principal methodological concerns.

We provide controlled objective and nesting ablations, matched-budget SCRUB comparisons, expanded fine-tuning searches, margin-sensitivity analysis, head-sharing diagnostics, Matryoshka-trained source evaluations, and fresh linear-probe tests. We also clarify the implementation of the bounded objective and revise the motivation to reflect what the evidence establishes.

These experiments support a more precise contribution: MMU imposes independently enforced forgetting constraints across nested representation widths, making the bounded objective substantially more effective than its full-width counterpart in the evaluated settings. We will additionally correct the random-forgetting inconsistencies, clarify participant-data governance and reproducibility, and narrow the claims concerning scale, generation, and privacy.

We appreciate the reviewer's careful evaluation. We believe the additional experiments substantially strengthen the methodological evidence, and we would be grateful if the reviewer would reconsider the soundness and overall assessment in light of these results.

Reviewer_jQAeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors resort to the multiple embedding dimensions rather than the final-layer full representation to solve the approximate machine unlearning problem. They design teacher-student unlearning mechanism where the original model is the teacher and the unlearned model is the student. In this way, the retain data knowledge can be preserved and the forget data information can be pushed away supervised by the teacher model. Beside the image classification model, the authors also extend the machine unlearning task into the text-to-image generation task.

Critical strengths

(1) The proposed MMU design the training objective at different representation levels.

(2) The paper includes experiments on random forgetting, sequential deletion, relearning, and image generation.

Critical weaknesses

(1) The proposed method is closely similar to the implementation of existing method SCRUB, where the teacher-student unlearning mechanism is the main core. The primary difference is that the objectives of MMU utilize multiple embedding widths, making the novelty of the MMU is incremental and limited.

(2) The proposed MMU assumes that meaningful representations can be constructed by truncating the first m dimensions of pretrained feature embeddings. However, standard neural networks are not explicitly trained to organize their features in a nested or coarse-to-fine manner. therefore, smaller embedding prefixes may not preserve sufficient semantic information or provide reliable predictions. This is a critical problem that the authors should give more proof and explanation.

(3) The performance on subclass and instance-level unlearning are less convincing. For example, on CIFAR-10 with ResNet-18, MMU achieves 55.37 percent forget accuracy for subclass deletion, compared with 26.57 for Retrain and 28.40 for Bad Teacher. The authors should give more explanation of this situation based on their designed method.

Questions for authors

(1) MMU achieves good results in class-level unlearning setting, but the membership inference attacks remain effective for subclass- and instance-level unlearning. How do the authors explain these results, and can MMU still be considered successful in these settings?

(2) Can the forgotten information still be recovered by training a new linear classifier on the features of the unlearned model?

5 comments
AuthorsOpenReview ↗

Response 1/4

We thank the reviewer for the careful assessment and constructive questions. To address the concerns, we conducted additional controlled experiments on RTI and CIFAR-100, using ResNet-18 and CNN, three deletion granularities, and three random seeds. Within each experimental setting, methods share the same source checkpoints, Retrain references, validation-only selection protocol (2-point utility tolerance), and held-out test evaluation.

W1: How does MMU differ from SCRUB?

We agree that teacher–student distillation is an established mechanism in machine unlearning, including SCRUB and Bad Teacher. Our contribution is not the use of teacher–student learning itself, but the formulation of unlearning as simultaneous forgetting and preservation constraints across nested representation widths.

SCRUB applies its forgetting and preservation objectives to full-width predictions. MMU introduces a different optimization structure:

ComponentSCRUBMMU
Forgetting supervisionFull-width predictionsEvery nested embedding width (Eq. 6)
Teacher targetFull-width logitsWidth-aligned teacher targets (Eq. 3)
ReadoutsOne full-width classifierIndependent auxiliary heads per width
Forget objectiveKL maximizationMargin-limited KL hinge
Gradient constraintsOne full-width objectiveOverlapping constraints from nested prefixes
ApplicabilityClassificationClassification and conditional diffusion

More precisely, MMU minimizes the forgetting objective

Lforget=∑m∈Mλfmmax⁡(0,δm−K(m)),\mathcal L_{\mathrm{forget}}= \sum_{m\in\mathcal M}\lambda_f^m \max(0,\delta_m-K^{(m)}),

where K(m)K^{(m)} measures divergence from the frozen teacher at width mm. Unlike unconstrained KL maximization, the hinge has an explicit stopping target at each width. Furthermore, coordinate jj receives gradients from every prefix satisfying m≥jm\geq j, creating an ordered hierarchy of overlapping constraints.

This is a change to the unlearning objective, not simply an increase in the number of classifier outputs.

Controlled ablation: Is nesting responsible for the improvement?

To isolate the contribution of nesting, we kept the hinge objective, optimization procedure, and 96-configuration search grid fixed, changing only the representation widths. The full-width variant therefore uses the same bounded objective as MMU without nesting.

Table 1. Controlled nesting ablation: absolute forget-accuracy gap to Retrain (percentage points, lower is better).

Dataset / BackboneFull-width hingeNested hinge
RTI / ResNet-180.00.0
RTI / CNN49.90.1
CIFAR-100 / ResNet-1830.70.0
CIFAR-100 / CNN67.30.0

Nesting reduces the class-deletion gap by 30.7–67.3 points in the three configurations where the full-width hinge leaves substantial residual accuracy. These improvements arise under the same search budget and forgetting objective.

The advantage extends to subclass deletion, where nesting improves all four settings:

Table 2. Controlled nesting ablation on subclass deletion; gap to Retrain (percentage points, lower is better).

Dataset / BackboneFull-widthNested
RTI / ResNet-189.32.3
RTI / CNN6.33.7
CIFAR-100 / ResNet-1852.342.7
CIFAR-100 / CNN46.739.3

These improvements come without a meaningful utility penalty. Across all 12 deletion settings, the retain-accuracy change from nesting stays within ±1.2 percentage points; on class deletion, nesting maintains or improves retain accuracy in all four configurations.

The controlled results directly establish the contribution of nested supervision: when the loss, optimization protocol, and hyperparameter budget are held fixed, enforcing forgetting at multiple embedding widths substantially improves agreement with Retrain.

Margin sensitivity: Does nesting require carefully tuned forgetting strength?

To examine whether MMU's improvement depends on a particular forgetting margin, we swept δ∈{16,64,256,1024}\delta\in\{16,64,256,1024\} while keeping the remaining settings fixed. For each margin, we evaluated 24 candidate configurations over three seeds, selecting the smallest validation forget-accuracy gap subject to the same 2-point utility tolerance.

Table 3. Class-deletion gap to Retrain (%) across forgetting margins; lower is better.

Dataset / BackboneObjectiveδ=16δ=64δ=256δ=1024
RTI / ResNet-18Full-width65.953.20.00.0
Nested65.60.30.00.0
RTI / CNNFull-width64.154.854.853.1
Nested63.22.00.00.0
CIFAR-100 / ResNet-18Full-width66.759.327.327.3
Nested70.014.00.00.0
CIFAR-100 / CNNFull-width59.359.358.058.0
Nested61.340.00.00.0

At δ ≥ 256, the nested objective reaches zero class-deletion gap in all four configurations, whereas the full-width objective does so in only one.

The advantage is also evident at δ = 64. On RTI, nesting reduces the gap from 53.2 to 0.3 (ResNet-18) and from 54.8 to 2.0 (CNN). On CIFAR-100/ResNet-18, it reduces the gap from 59.3 to 14.0.

These results show that imposing multiple width-specific constraints can make the bounded forgetting objective effective over a wider range of margin values.

Hyperparameter robustness: How consistently does nesting succeed?

Beyond selecting the best configuration, we measured the fraction of the search grid achieving both a forget-accuracy gap of at most 2 points and retained utility within the 2-point tolerance.

Table 4. Hyperparameter-grid success rate (%) on class deletion; higher is better.

Dataset / BackboneFull-width hingeNested hinge
RTI / ResNet-184%28%
RTI / CNN0%27%
CIFAR-100 / ResNet-180%9%
CIFAR-100 / CNN0%5%

Nesting consistently increases the proportion of successful configurations. The largest improvements occur on RTI: 28% versus 4% on ResNet-18, and 27% versus 0% on CNN.

Thus, the controlled improvement is not restricted to an isolated favorable checkpoint: nested constraints increase the region of the hyperparameter space that achieves the desired forgetting–utility trade-off.

Comparison with SCRUB under a matched search budget

We also expanded SCRUB's hyperparameter search from 24 to 96 configurations, matching the budget of the nested-hinge objective. This is important because SCRUB's performance improves considerably with a larger search budget.

Table 5. Forget-accuracy gap to Retrain (%) for SCRUB and the nested-hinge objective; lower is better.

Dataset / BackboneDeletionSCRUB (24)SCRUB (96)Nested (96)
RTI / ResNet-18Class0.00.00.0
RTI / CNNClass49.90.00.1
CIFAR-100 / ResNet-18Class30.70.00.0
CIFAR-100 / CNNClass68.30.00.0
RTI / ResNet-18Subclass9.35.72.3
RTI / CNNSubclass6.35.33.7
CIFAR-100 / ResNet-18Subclass52.041.042.7
CIFAR-100 / CNNSubclass47.735.339.3

With an equal 96-configuration budget, SCRUB also reaches the Retrain target for class deletion in all four configurations. We therefore do not claim superiority over adequately tuned SCRUB in this setting.

The advantage of MMU becomes apparent on RTI subclass deletion:

  • ResNet-18: the gap decreases from 5.7 (SCRUB) to 2.3 (nested), a 59% reduction.
  • CNN: the gap decreases from 5.3 (SCRUB) to 3.7 (nested), a 31% reduction.

These improvements are achieved with comparable retained utility. On CIFAR-100 subclass deletion, both methods remain substantially farther from Retrain.

Taken together, the controlled ablation, margin sweep, robustness analysis, and matched-budget comparison support a specific contribution: nested supervision changes the optimization behavior of a bounded forgetting objective, improves its reliability across configurations, and can yield closer agreement with Retrain on subclass deletion. This is the distinction we will emphasize in the revised paper rather than claiming novelty from teacher–student distillation alone.

W2: Are truncated embeddings meaningful in standard pretrained networks?

We agree that a conventionally trained neural network is not guaranteed to organize its embedding coordinates in a meaningful coarse-to-fine order. However, MMU does not require truncated source embeddings to function as accurate classifiers without adaptation. Instead, MMU introduces independently trainable auxiliary heads for each width:

$ z^{(m)}(x)=W^{(m)}[f_\theta(x)]_{1:m}+b^{(m)}. $

Each auxiliary head is initialized from the corresponding slice of the original classifier and optimized during unlearning. Retained-data supervision makes the restricted readouts predictive, while the nested forgetting objective constrains their behavior on deleted examples. These heads are discarded afterward, leaving the deployed full-width architecture unchanged.

We performed two additional experiments to test this mechanism directly.

1. Experiments with explicitly Matryoshka-trained networks

To eliminate the concern that arbitrary prefixes may lack semantic meaning, we trained both Original and Retrain with classification supervision at every embedding width. This creates source models whose prefixes are already meaningful classifiers, with retain accuracy within 0.5 percentage points of their full-width accuracy.

We then repeated the nesting ablation using these source models.

Table 6. Class-deletion gap to Retrain (%) using Matryoshka-trained source models; lower is better.

Dataset / BackboneFull-width hingeNested hinge
RTI / ResNet-180.00.0
RTI / CNN58.70.0
CIFAR-100 / ResNet-1841.30.0
CIFAR-100 / CNN67.76.3

Nesting reduces the gap by 41.3–61.4 percentage points in the three configurations where the full-width hinge fails.

These results are particularly important because all prefixes were explicitly supervised during source training. Thus, the advantage of MMU is not confined to conventional pretrained networks with arbitrary prefix structure.

Together with the standard-source experiments reported in Part 1, these findings demonstrate that nested unlearning is effective across both source-training procedures.

2. Ablation of independent versus shared classifier heads

We further investigated whether independently trained heads are necessary by comparing:

  • MMU-1: a separate trainable classifier at every width.
  • MMU-2: one shared classifier whose weight matrix is sliced at each width.

On Matryoshka-trained sources, MMU-2 produces large class-deletion gaps in three of four configurations.

Table 7. Class-deletion gap to Retrain (%) for independent and shared heads.

Dataset / BackboneIndependent heads (MMU-1)Shared head (MMU-2)
RTI / ResNet-180.00.3
RTI / CNN0.063.5
CIFAR-100 / ResNet-180.057.3
CIFAR-100 / CNN6.367.0

The shared-head formulation fails substantially in three configurations, even though each embedding prefix has already been trained to provide meaningful predictions.

3. Why does sharing the head fail?

We performed a logit-level diagnostic on CIFAR-100/ResNet-18 to identify the mechanism.

After shared-head unlearning, the deleted-class logit receives:

+3.42⏟coords 1:128  +  (−14.21)⏟coords 129:512  =  −10.79\underbrace{+3.42}_{\text{coords } 1:128} \;+\; \underbrace{(-14.21)}_{\text{coords } 129:512} \;=\; -10.79

The negative contribution from later coordinates suppresses the final deleted-class prediction, while the first 128 coordinates still support that class.

Indeed, the shared-head model reduces full-width deleted-class accuracy to approximately 0%, while narrower prefixes can retain high accuracy on the forgotten class.

This illustrates a concrete failure of a shared readout: the final prediction can satisfy the forgetting objective through cancellation between early and late coordinates, without satisfying it at the narrower prefix.

Independent heads remove the reliance on a common classifier for every width, requiring the optimization to satisfy the width-specific forgetting constraints separately.

What these results establish

The experiments answer the reviewer's concern from two complementary directions:

  1. MMU can optimize restricted readouts starting from conventionally trained embeddings, without requiring a pretrained Matryoshka representation.
  2. When the source network is explicitly Matryoshka-trained, nested forgetting remains effective, and independent per-width heads are substantially more reliable than a shared head.

We will therefore make independent per-width heads the definitive MMU formulation, present the shared-head version as an ablation, and clarify the distinction between meaningful prefix readouts and feature-level information removal.

These results provide direct empirical support for the proposed nested construction and explain why simply truncating one shared classifier is insufficient.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.