acceptodds
Under review as a conference paper at ICLR 2027

How Learning Governs Unlearning across the Memorization–Generalization Spectrum

Abstract

While unlearning seeks to negate undesired capabilities acquired through learning, little research has examined how the way models learn shapes their subsequent unlearning. In this paper, we investigate this connection from the perspectives of memorization and generalization, the two most representative yet competing strategies that models employ during training. We first classify memorization- and generalization-heavy models using grokking in modular addition and compare their responses to unlearning, showing that the latter suffer greater retain damage, i.e., a larger performance drop on the retain set. Furthermore, we conduct a finer-grained analysis by introducing bucketed modular addition, in which the respective contributions of the two strategies can be explicitly controlled across the memorization–generalization spectrum. In this setup, we reaffirm that the same trend persists and is nearly monotonic. We further demonstrate that this relationship also holds in LLM unlearning across verbatim and factual recall settings. Finally, we provide two practical insights for developing better unlearning methods, highlighting the importance of accounting for learning dynamics in unlearning.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, Unlearning, Grokking, Generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LkDYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tries to understand the relation between machine unlearning and the learning mechanisms. Specifically, it tries to analyze the damage of retained knowledge across the memorization-generalization spectrum. Through controlled experiments and mechanistic analyses, the authors validate the hypothesis that generalization suffers greater retained-knowledge damage, and verify that the conclusions hold in LLM unlearning scenarios. Finally, the authors discuss two practical insights that extend from the main findings.

Critical strengths
  1. The toy experiments are well-designed; I like how the paper uses a controlled scenario to analyze the memorization-generalization spectrum. The experimental design and findings clearly address the hypothesis mentioned in Section 3 (Line 158). As the paper states, this work is the first attempt to understand unlearning from this perspective.
  2. Practical insights discussed in the discussion are useful to the unlearning community. I think this practical insight helps (Line 465, copied below) future work align and evaluate unlearning methods on more realistic settings and with reasonable goals.

Thus, retraining without the forget set is a gold standard for removing its training contribution, but does not always achieve the behavioral goal of suppressing outputs.

Critical weaknesses
  1. While this paper provides interesting analyses and insights, the practical impact is unclear. For instance, how does the finding help identify which knowledge is unlearnable for a model? How do we estimate the best forget-retrain tradeoff in a given scenario, and how do we achieve it? The paper states its insights in Section 6, no actionable insights are provided or tested. This limits the significance of the paper's findings.
  2. [Minor] Following the first point, I think this paper could benefit from existing benchmarks and tested models to see if the paper's findings hold in the community's common evaluations. Since I am not familiar with the existing machine unlearning work, I am not sure if this request is feasible. However, I think at least an attempt or discussion is needed because it will test whether this paper's findings generalize to these common scenarios.
Questions for authors

Questions:

  1. For toy experiments, I wonder if the experimental findings will change if we use a less capable model. As the model's complexity decreases, its memorization capability decreases as well. Therefore, I am curious how sensitive the conclusion is when testing on different models.

Suggestions

  1. For the clearness of this presentation, I think it will be better to put the hypothesis (copied below) in the Introduction or at the beginning of Sec 3. It will help readers to understand why memorization and generalization matter for unlearning; it will also make the goal of the experiments clear before stating the experiment results.

We hypothesize that unlearning the latter must damage this circuit, on which retained examples also depend, and thus incurs greater retain damage.

5 comments

We thank the reviewer for the thoughtful review, and for finding the toy experiments well designed and the practical insights useful to the unlearning community. Below, we first address the weaknesses and then answer the question and the suggestion.

On the practical impact of our findings

The reviewer notes that the lack of actionable insights limits the significance of our findings. Specifically, the reviewer asks how to identify which knowledge is unlearnable and to what extent, and how to achieve the best forget–retain trade-off accordingly.

While we focus largely on the science of how learning shapes unlearning, our insights point to a concrete practice for LLM unlearning in response to each question: (1) estimating how difficult a target is to unlearn before unlearning it, and (2) adjusting unlearning at the token level according to this difficulty.

  1. Estimating unlearning difficulty with the memorization floor.
    • Before unlearning a target, we suggest estimating its memorization floor, the amount of sample-specific information that must be memorized beyond the shared rule (Sec. 4.1). In LLMs, it can be estimated with the unintended memorization of Morris et al. (2026), which compares the model with a reference model that has not seen the target.
    • A higher floor predicts less retain damage in both of our LLM settings (Sec. 5) and on TOFU (Table R1 below), as it does in BMA (Sec. 4.2). The floor can therefore serve as a proxy for how difficult a target is to unlearn, available before any unlearning is run.
  2. Feasibility of token-level regularization guided by the memorization floor.
    • Since an LLM predicts text one token at a time, the memorization floor can be computed separately for each next-token prediction.
    • Some existing unlearning objectives also work token by token: these token-wise regularizers suppress each forget token with a different weight, for example according to how confidently the model still predicts it or whether it is a named entity.
    • In App. E.2, we compare eight such regularizers and find that the more closely a regularizer's weights align with the per-token floor, the more of the retain set it preserves (Spearman ρ = 0.76).
    • This supports the feasibility of guiding token-level regularization with the memorization floor, focusing suppression on what the model has memorized while sparing what it has generalized.

Taken together, we believe our findings are not confined to controlled experiments but carry concrete implications for unlearning in practice: they indicate in advance how costly it is to unlearn a target, and suggest how unlearning methods can reduce this cost.

On testing our findings on existing benchmarks

  • We appreciate this suggestion. We believe our LLM experiments (Sec. 5) already provide such a test on a public benchmark, since the Hubble suite (Wei et al., 2025) used there was released as a testbed for machine unlearning.
  • We further tested our findings on TOFU (Maini et al., 2024), a widely used unlearning benchmark. Using the Llama-3.2-1B-Instruct models from OpenUnlearning (Dorna et al., 2026), we split each forget10 author's QA pairs into halves with a higher and a lower memorization floor (3.93 vs. 2.29 bits per token on average), and unlearned each half with GA, NPO, and GD (Table R1).

Table R1. Retain damage when the forget per-token probability reaches 10% of its initial value. Retain-near is the other half of the same authors; retain-far is 100 QA pairs from other authors.

MethodRetain-near: High / Low floorRetain-far: High / Low floor
GA0.79 / 0.950.82 / 0.91
NPO0.79 / 0.950.82 / 0.91
GD0.51 / 0.650.50 / 0.67
  • For all three methods, unlearning the half with the higher memorization floor causes less retain damage, in line with BMA and the Hubble settings. We have added this experiment to the revised paper (App. D.4), where Figure 9 shows that the gap between the two halves widens as unlearning proceeds.

Q1. Would the conclusions change with a less capable model?

Model capacity can affect our two toy experiments differently, the modular addition experiment (Sec. 3) and BMA (Sec. 4), so we examine each separately.

  1. Modular addition (Sec. 3).

    • The binary comparison requires two models that fit the training set equally well but solve it differently, one by memorization and one by generalization. Such a pair is available only when generalization is delayed well after the training set is fit, i.e., when the model groks.
    • Grokking requires enough capacity to memorize the training set first: the network fits it with a memorizing solution, which is faster to learn, and only later moves to the more efficient generalizing solution under regularization such as weight decay (Nanda et al., 2023; Doshi et al., 2024).
    • A less capable model cannot memorize the training set, so we expect it to generalize directly without a memorizing stage. We tested this by training the Sec. 3 model at smaller widths (Table R2; see Figure 5 in App. B.5 of the revised paper for the full training curves).

    Table R2. Modular addition at different widths (range over three seeds, 60k epochs). The largest gap between training and test accuracy marks the memorizing stage.

    dmodeld_{model} (parameters)Max. train − test accuracyFinal train / test accuracy (%)
    128 (227k), paper0.96–0.98100 / 100
    64 (64k)0.94–0.97100 / 100
    32 (20k)0.62–0.80100 / 100
    16 (6.8k)0.29–0.3944–100 / 5–98
    8 (2.7k)0.19–0.2168–81 / 51–68
    • Models with dmodel≥32d_{model} \geq 32 fit the training set within a few hundred epochs and generalize thousands of epochs later. Models with dmodel≤16d_{model} \leq 16 show no such gap: their test accuracy rises together with training accuracy, or they fail to fit the training set. With less capacity, generalization follows fitting immediately, so the memorizing model required for the binary comparison cannot be obtained.
  2. BMA (Sec. 4).

    • BMA needs no such pair, since the bucket size sets the balance between memorization and generalization by design. We retrained all five BMA models at half the width (dmodel=128d_{model} = 128; all reach 100% training accuracy) and repeated RL at 5% forget over three seeds (Table R3; Figure 7 in App. C.4 of the revised paper).

    Table R3. Retain damage 1−Pr1 - P_r at Pf=0.1P_f = 0.1 under RL (5% forget; mean ± 95% CI over three seeds).

    Memorization floordmodel=256d_{model} = 256 (paper)dmodel=128d_{model} = 128
    0 bits0.874 ± 0.0080.896 ± 0.014
    2 bits0.804 ± 0.0370.723 ± 0.063
    4 bits0.717 ± 0.0770.653 ± 0.017
    6 bits0.484 ± 0.0310.639 ± 0.022
    8 bits0.291 ± 0.0410.595 ± 0.013
    • Retain damage again decreases monotonically with the memorization floor in the smaller model (Spearman ρ = −1 at both widths).
    • The smaller model, however, shows more retain damage at high floors (0.60 vs. 0.29 at 8 bits). Since it memorizes close to its capacity (initial retain probability 0.963 at 8 bits), its memorized examples appear to interfere more with each other, in line with the reviewer's intuition that memorization capability shrinks with model size.

Suggestion. Stating the hypothesis earlier

  • We agree that this hypothesis is the core of our work and deserves to be stated up front. In the revised paper, we now state it in the introduction (in blue): "We hypothesize that unlearning what a model has generalized must damage this shared mechanism, on which retained examples also depend, and thus incurs greater retain damage than unlearning what it has memorized."

References

Doshi, D., Das, A., He, T., & Gromov, A. (2024). To Grok or not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=UHjE5v5MB7

Dorna, V., Mekala, A. R., Zhao, W., McCallum, A., Kolter, J. Z., Lipton, Z. C., & Maini, P. (2026). OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https://openreview.net/forum?id=Gy67Zh5X1i

Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z. C., & Kolter, J. Z. (2024). TOFU: A Task of Fictitious Unlearning for LLMs. In First Conference on Language Modeling. https://openreview.net/forum?id=B41hNBoWLo

Morris, J. X., Sitawarin, C., Guo, C., Kokhlikyan, N., Suh, G. E., Rush, A. M., Chaudhuri, K., & Mahloujifar, S. (2026). How much can language models memorize? In Forty-third International Conference on Machine Learning. https://openreview.net/forum?id=bA6BgSbaUi

Wei, J. T.-Z., Godbole, A., Khan, M. A., Wang, R., Zhu, X., Flemings, J., Kashyap, N., Gummadi, K. P., Neiswanger, W., & Jia, R. (2025). Hubble: a Model Suite to Advance the Study of LLM Memorization. arXiv:2510.19811. https://arxiv.org/abs/2510.19811

Reviewer_EZp3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies how the learning dynamics of a model affect its unlearning behavior. The authors partition the "skills" a model has learned into two classes: (1) generalizing; and (2) memorizing. In the former regime, a model presumably applies this generalizing mechanism across many examples. In the latter, the model simply ingests an input-output relation. The paper finds that unlearning generalizing behaviors degrades performance outside of the unlearned set. On the other hand, unlearning memorized examples does not impact performance outside of the unlearned set as much.

Critical strengths

The model organisms studied in the paper are well chosen (specifically bucketed modular arithmetic). I feel overall that the experiments are well-designed and the claims are well-scoped and supported.

Critical weaknesses

The overarching claim seems tautological to me. For example, the authors define the terms memorization and generalization as

  1. In Line 112, "memorization denotes ... information tied to specific training samples"
  2. In Line 113, "generalization denotes learning a shared solution mechanism that applies across examples."

Then, by definition, if I unlearn the shared mechanisms that is applied across examples this would imply that performance would be damaged across all examples where this mechanism is applied. And the reverse direction would also hold for memorization. To this end, it is unclear to me what the motivation of the work is and its practicality. Like, presumably if a practitioner wanted to unlearn some behavior, they would first need to figure out whether it is a part of the generalizing vs. memorization behavior. Then, this could give some bound over to extent to which unlearning can be achieved without impacting performance on other data points?

Questions for authors
  • It is interesting to me (and also not intuitive) why for bucketed modular arithmetic there is a consistent delay between the model's test-bucket performance versus its test-within (bucket) performance. Is the model being warm-started here? Also, the loss function still teacher-forcing?
  • How do you think unlearning will look if the data we are trying to unlearn used some mixture of memorization vs. generalization (i.e. maybe we are trying to unlearn an entire data source)? Would there potentially be some sort of Pareto frontier between the retain accuracy and the ratio of the extent to which the data source encourages memorization versus generalization?
1 comment

We thank the reviewer for the careful review, and for finding the model organisms well chosen and the claims well-scoped and supported. Below, we first address the concerns raised under Weaknesses and then answer the reviewer's questions.

On whether the main claim is tautological

  • We would first like to clarify what unlearning optimizes. The unlearning objective is not set up to damage the shared mechanism. It only asks the model to stop producing the learned answers on the forget set Df\mathcal{D}_f, and the objective itself contains no account of any shared mechanism.
  • Our hypothesis is that if the model solves Df\mathcal{D}_f through a shared mechanism formed by generalization, unlearning achieves forgetting by the easiest route available, namely by damaging that mechanism. Since nothing in the objective requires this, whether it holds is an empirical question. We confirm it in a toy setting where the shared mechanism is known, showing that unlearning the grokked model collapses its Fourier circuit (Sec. 3.3).
  • The step after this, on the other hand, is where the reviewer's reasoning applies. Once the circuit is damaged, specifically by unlearning as confirmed above, the retain examples that rely on the same circuit are damaged as well, which follows directly from our definitions.
  • To summarize, the causal chain is unlearning → damage to the shared mechanism → retain damage. The second link is the part the reviewer correctly identifies as following from the definitions, whereas the first link is not guaranteed by them and therefore requires the empirical verification our experiments provide.

On motivation and practicality

  • Regarding the implications for the unlearning community, we often see that unlearning is understood like deleting a row from a database: knowledge can be erased by reversing how the model acquired it, i.e., by removing the training influence of Df\mathcal{D}_f, and retain damage arises only from the imperfection of this erasure.
  • Our results tell a different story. When generalization has absorbed the training influence of Df\mathcal{D}_f into building a shared solution mechanism (rather than storing it like a row in a database, i.e., memorizing it), there is no individual row for Df\mathcal{D}_f to delete in the first place, so unlearning has to act on the shared mechanism itself. Retain damage is then better understood as a matter of how the model learned Df\mathcal{D}_f than of a failure of erasure.
  • More broadly, we believe that progress in unlearning has been driven largely by methods that improve benchmark performance, while efforts to understand unlearning itself have been comparatively limited. Our work contributes to this understanding through the lens of learning, explaining why unlearning without retain damage is so difficult.
  • Indeed, Reviewer 9ov9 values our objective of understanding unlearning in terms of the generality of the circuits involved, and Reviewer LkDY finds our insights useful for aligning and evaluating unlearning methods with more realistic settings and reasonable goals. Given what they find valuable in our work, we expect it to be of value to the wider community.

Q1. Why is there a consistent delay in BMA between test bucket accuracy and test within-bucket accuracy? Is the model warm-started, and is the loss teacher-forced?

  • We understand the delay the reviewer refers to in Figure 3 (top row) as the gap between training accuracy (solid), which saturates early, and test bucket accuracy (dashed), which rises much later. This is grokking in bucket selection, as in modular addition (Sec. 3): the model first fits the training pairs, within-bucket indices included, and only later forms the shared mechanism that selects the correct bucket on unseen pairs.
  • Test within-bucket accuracy (dash-dot), by contrast, does not rise at any point: it stays at the 1/M1/M chance level throughout training, since the random within-bucket index cannot generalize.
  • The models are not warm-started: each is trained from random initialization.
  • Teacher forcing does not apply: each example is a single-token prediction. The model reads [a,b,=][a, b, =] and predicts one of the 256 answers at the final position, trained with cross-entropy on that position only.

Q2. How would unlearning look if the target mixes memorization and generalization, e.g., an entire data source? Would there be a Pareto frontier between retain accuracy and this mixture ratio?

  • This is exactly the setting BMA is designed for. Every BMA answer mixes the two: of its 8 bits, log⁡2M\log_2 M bits must be memorized and the rest follow from the shared rule, so the bucket size MM sets the mixture ratio (Sec. 4.1).
  • Figure 3 (bottom row) shows this frontier directly: each curve traces the forget–retain trade-off at one ratio, and as the memorized share grows, the curve generally moves closer to the ideal of complete forgetting with no retain damage.
Reviewer_9ov9OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

They study how off-target effects in unlearning are related to the generality of the circuits involved in computing the answer that is to be unlearned.

Their first toy setting for this is modular addition on a one layer transformer, ala Nanda et al. (2023) . They show that off-target damage from unlearning is greater later in training when the models’ answers are more influenced by the fourier feature based general modular addition circuits the model learns than the lookups. Since the mechanistic structure of the solutions networks learn in this setting is known, they are able to somewhat verify this causal story mechanistically, tracing out what part of the off-target damage originates with the key fourier modes. They show that filtering unlearning gradients that lie within the subspace of the key fourier modes (of the model pre-unlearning) substantially reduces off-target damage.

Their second toy setting is bucketed modular addition, a variation on the modular addition toy setting where the amount of information in each label that can be determined via general modular addition can be varied as desired, with the rest of the information coming from memorisation. They again show that settings in which more information is determined by general circuits show more off-target damage under unlearning. They also show that training the models to give specific different answers on the examples to be unlearned hurts performance more in cases where the new answer isn’t in the same bucket as the original, suggesting this damages the general mechanism that decides what bucket the answer is in.

Their final setting is llms. To measure how much each data point they study is determined via general vs. specific computational machinery in this setting, they compare the logit predictions of models from the Hubble suite (Wei et al., 2025) that have vs. haven’t been trained to memorise particular passages. They show how unlearning effects prediction on other data with semantic or linguistic similarities. Again, off-target damage appears greater in settings where the “memorisation floor” to generalisation seems higher. Further, damage is higher on data that shares a relation with the unlearned data point.

Critical strengths

I liked their idea of using modular addition as a toy setting for unlearning. This is a rare case where we have a good mechanistic understanding of what algorithms the models actually learn, so it is actually feasible to study what the unlearning changes mechanistically to a quite high level of detail, and they do conduct some of this analysis.

I also like their overarching objective of understanding unlearning behaviour in terms of the generality of the circuits involved in computing the answer we wish to modify.

Critical weaknesses

I think that the check of their story in the modular addition setting could and should have gone further than it did. This is a quite well-understood setting, so they could have traced out what the unlearning did to the fourier modular addition circuits neuron by neuron. Instead, they seem to just have looked at the subspace of key fourier frequencies, and I also don’t see them checking anywhere whether the location of that subspace shifted after unlearning, they just pick it before unlearning starts.

As a consequence, it seems possible for the unlearning to affect the modular addition circuits in ways their analysis doesn’t catch. For example, it seems to me that the unlearning might be able to affect upstream weights involved in calculating the key frequencies before they are formed, bypassing their filtering. Perhaps relatedly, their appendix B.4 seems to show that there’s still substantial retain set damage despite their intervention, more than in the memorising network. That seems like clear evidence that the key frequency subspace analysis is missing something essential to the story.

It also seems possible in principle for the unlearning to shift which neurons implement which key frequencies, possibly confounding the retained loss analysis in 3.3. I see no particular evidence for this, and it is perhaps unlikely, but my point is that many questions of this kind could likely have been answered here if they’d just gone a little deeper with the mechanistic analysis, which is very doable in this setting.

Questions for authors

Appendix B.4: It looks like there’s still retain set damage despite your intervention. If that’s because of damage to circuitry upstream of the key fourier features, filtering out gradients affecting those parts too should stop the remaining damage. Otherwise, it seems like there’s still more going on here than your causal story indicates. Do you have an idea of what the remaining damage is?

5 comments

We thank the reviewer for the constructive review and for valuing our mechanistic approach to understanding unlearning. We are especially grateful for the insight to trace unlearning at the level of the circuit's neurons, which proved key to the analysis below.

The reviewer points out that our analysis could have traced how unlearning actually damages the Fourier circuit instead of only measuring the key-frequency components of the logits. Indeed, FVE and restricted loss (Sec. 3.3, App. B.4) show that the circuit shared by forget and retain examples is damaged, but not how. The reviewer's question on App. B.4 points directly at this gap: substantial retain damage remains even after key-frequency gradient filtering.

Guided by this question and the testable hypotheses the reviewer proposes, we carried out a more detailed mechanistic analysis of the grokked model. Below, we present a fuller picture of

  1. how unlearning (RL) damages the circuit,
  2. how key-frequency filtering prevents this damage, and
  3. what causes the retain damage that remains after filtering.

We then return to the reviewer's hypotheses.

Part 1. How unlearning (RL) damages the circuit

Motivated by Hong et al. (2024), we divide the circuit into three components that unlearning could act on, in the order of computation:

  • Embedding output. The embeddings write each operand as Fourier features, cos⁡/sin⁡(ωka)\cos/\sin(\omega_k a) and cos⁡/sin⁡(ωkb)\cos/\sin(\omega_k b). This is the most upstream part of the circuit.
  • Neuron activations. Attention and the MLP input weights combine the embedded operands into neuron activations that contain cos⁡/sin⁡(ωk(a+b))\cos/\sin(\omega_k(a+b)). For each input, the activations are the coefficients of a linear combination of the neurons' readout weights.
  • Readout weights (WoutW_{out}, WUW_U). These are fixed across inputs. Each neuron's readout over the 113 answers is approximately a wave cos⁡(ωkc+ϕ)\cos(\omega_k c + \phi). Summed with the activations as coefficients, these waves form ∑kαkcos⁡(ωk(a+b−c))\sum_k \alpha_k \cos(\omega_k(a+b-c)), which peaks at the correct answer c=a+bc = a+b.

To find which component's damage is critical, we patch each one in two directions. First, we restore it in the unlearned model to its original value, i.e., its value before unlearning (Unlearned ← Original). Second, we inject its unlearned value into the original model (Original ← Unlearned).

Table R1. Retain accuracy (%) before → after patching, for RL on the grokked model (mean ± 95% CI over the five DfD_f seeds).

ComponentRestore (Unlearned ← Original)Inject (Original ← Unlearned)
Embedding output8.1 → 16.1 ± 1.8100.0 → 98.8 ± 0.4
Neuron activations8.1 → 91.1 ± 1.3100.0 → 6.0 ± 3.1
Readout weights8.1 → 4.2 ± 1.8100.0 → 89.5 ± 1.5

The neuron activations account for most of the damage. Restoring them alone brings retain accuracy back to 91.1%, and injecting them alone into the original model reduces it to 6.0%. The other two components matter little: restoring the embedding output or the readout weights leaves the unlearned model broken, and injecting either one leaves the original model largely intact.

Unlearning thus damages the circuit through the neuron activations, that is, how the Fourier features of the inputs are combined, while changes to the embedding output and readout weights play only a minor role. We thank the reviewer for pointing us to this analysis, which reveals where in the circuit unlearning acts.

Part 2. How key-frequency filtering prevents this damage

To measure the circuit on its own, we use the restricted logits ZKZ^K of App. B.2 (Eq. 11), i.e., the logits projected onto the key-frequency components. We report circuit-only accuracy, the accuracy of arg⁡max⁡cZab,cK\arg\max_c Z^K_{ab,c}, together with the restricted loss (Eq. 13) on the retain set.

Table R2. Circuit-only performance on the retain set, original → unlearned.

UnlearnedUnlearned + filtering (key)
Circuit-only accuracy (%)100.0 → 46.8 ± 8.4100.0 → 97.5 ± 1.8
Restricted loss0.00 → 2.34 ± 0.170.00 → 0.17 ± 0.17

After unlearning, the circuit alone answers fewer than half of the retain pairs correctly, whereas with filtering it still answers almost all of them. Filtering thus succeeds in preserving the circuit itself.

Since Part 1 located the damage in the neuron activations, we next ask whether filtering protects the circuit through them. We inject the unlearned neuron activations of each run into the original model and measure how much this patching deteriorates the original circuit.

Table R3. Circuit-only performance on the retain set after injecting unlearned neuron activations into the original model.

PatchingCircuit-only accuracy (%)Restricted loss
Original ← Unlearned100.0 → 5.2 ± 3.20.00 → 8.45 ± 3.00
Original ← Unlearned + filtering (key)100.0 → 100.00.00 → < 0.001

Activations from unlearning without filtering cannot drive the circuit at all, whereas activations from the filtered run drive it as well as the original activations do. Filtering thus protects the circuit at the same place where unlearning damages it: the neuron activations still combine the Fourier features of the inputs correctly.

Part 3. What causes the retain damage that remains after filtering

Although filtering preserves the circuit, retain accuracy with the full logits is still only 30.6%, compared with 97.5% from the circuit alone (Table R4). The circuit still computes the correct answer on almost all retain pairs, but something outside it overrides that answer. We therefore look at the part of the logits outside the key-frequency components, Z⊥=Z−ZKZ^\perp = Z - Z^K.

This off-circuit component is what implements forgetting. On the forget pairs themselves, the circuit alone still predicts the original answer 97.0% of the time, yet full-logit forget accuracy is 0.7% (Table R4). Since filtering removes the gradient signal for changing the circuit's contribution to the logits, unlearning makes the forget pairs wrong through Z⊥Z^\perp instead of through the circuit, i.e., through ZKZ^K.

Table R4. Accuracy (%) with the full logits and with the circuit alone (ZKZ^K), original → unlearned + filtering (key).

RetainForget
Full logits (ZK+Z⊥Z^K + Z^\perp)100.0 → 30.6 ± 4.1100.0 → 0.7 ± 0.1
Circuit only (ZKZ^K)100.0 → 97.5 ± 1.8100.0 → 97.0 ± 2.4

However, this off-circuit component grows not only on the forget pairs but on the retain pairs as well. The size (root-mean-square) of Z⊥Z^\perp relative to that of ZKZ^K rises from a marginal 0.17 in the original model to 0.93 on both the forget and the retain pairs. On the retain pairs, the off-circuit component is thus almost as large as the circuit itself and overrides the circuit's correct answer, explaining the retain damage that remains after filtering. In App. B.4 of the revised paper, we further show that this off-circuit component is also produced by the neuron activations.

To summarize, the retain damage that remains after filtering is interference rather than circuit damage. With the circuit protected, unlearning implements forgetting through an off-circuit component in the neuron activations, and this component spreads to all inputs, including the retain pairs.

On the hypotheses raised in the review

Q1. Did the key-frequency subspace shift during unlearning?

  • No. After unlearning, in all 10 runs (5 seeds × {unlearning, unlearning + filtering}), the five largest frequencies of the embedding are exactly K={14,35,42,52,55}K = \{14, 35, 42, 52, 55\}. In the logits, KK accounts for 99.5% (unlearning) and 99.1% (unlearning + filtering) of the variance along cos⁡/sin⁡(ω(a+b))\cos/\sin(\omega(a+b)) across all frequencies. The fixed KK used in our analysis therefore remains the circuit's subspace after unlearning.

Q2. Could unlearning bypass the filter through upstream weights, and would filtering those too stop the remaining damage?

  • Since the filter constrains only the logit gradient (App. B.3, Eq. 14), not how the resulting parameter updates change the logits, the circuit can still change indirectly, for example through upstream weights. The reviewer's concern is therefore legitimate.
  • However, Parts 1–3 show that this does not happen: filtering itself succeeds in protecting the circuit. The circuit alone still answers 97.5% of the retain pairs correctly after filtering (Table R2), and the filtered model's neuron activations, which include the effect of all upstream weights, drive the original circuit perfectly (Table R3). The remaining damage comes instead from the off-circuit component Z⊥Z^\perp (Part 3).
  • Since the circuit is already intact, this damage does not come from damaged circuitry upstream of the key Fourier features, and protecting the circuit further, for example by also filtering upstream gradients, would not address it.

Q3. Could unlearning change which neurons implement which key frequencies, confounding the restricted-loss analysis in Sec. 3.3?

  • No. Of the 454 neurons that are frequency-specific before unlearning, 98.1 ± 0.5% (unlearning) and 98.5 ± 1.4% (unlearning + filtering) keep the same dominant frequency afterwards. The restricted-loss analysis is therefore not confounded by neuron reassignment.

Revision

  • Following the reviewer's comments, we have strengthened the mechanistic analysis in the revised paper (changes in blue). We moved the causal intervention, previously in App. B.4, into the main text (Sec. 3.3), and added the analyses above (Parts 1–3 and the checks in Q1 and Q3) as a new App. B.4.

Reference

Hong, Y., Zou, Y., Hu, L., Zeng, Z., Wang, D., & Yang, H. (2024). Dissecting Fine-Tuning Unlearning in Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 3933–3941. https://aclanthology.org/2024.emnlp-main.228/

Comments on the paper

2 comments

Dear Reviewers,

The authors have responded four days ago and haven't received any feedback since then. Please, read the responses, comment whether your concerns were addressed and when appropriate adjust your score.

Dear Reviewers,

Thank you again for your thoughtful feedback. As the discussion period ends on November 18 (AoE), we briefly summarize how we addressed the main concerns.

  • Reviewer 9ov9 (depth of the mechanistic analysis): We traced the damage to the neuron activations and showed that the retain damage remaining after filtering is off-circuit interference, not circuit damage.
  • Reviewer EZp3 (tautology; motivation): The unlearning objective does not require damaging the shared mechanism, so whether it does is an empirical question. Our results explain retain damage as a consequence of how the model learned the forget set, rather than as a failure of erasure.
  • Reviewer LkDY (practical impact; benchmarks; model capacity): The memorization floor serves as a proxy for unlearning difficulty, and we confirmed our findings on TOFU and with smaller models. Thank you for confirming that your concerns have been addressed.

Reviewers 9ov9 and EZp3, we would be grateful to hear whether our responses have addressed your concerns, and we are happy to clarify any remaining points before the deadline.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.