acceptodds
Under review as a conference paper at ICLR 2027

Early Memories Don't Fade Easily: On the Difficulty of Unlearning Pretraining Knowledge

Abstract

Machine unlearning aims to remove the influence of specific training data from a deployed model. Existing benchmarks introduce the forget set through a brief fine-tuning stage after pretraining, so they primarily evaluate the removal of recently acquired knowledge. In practice, however, deletion requests may concern data encountered during pretraining itself. Whether these two settings are equally difficult remains poorly understood. We study how the stage at which knowledge is acquired affects its subsequent removal. We hypothesize that data encountered earlier in training can become more strongly integrated with representations that are subsequently reused, whereas data introduced later may induce more localized changes. We theoretically analyze this training-history effect, first characterizing how forget-set influence accumulates along the optimization trajectory and then showing, in a two-layer linear model, that earlier introduction of the forget data leads to greater parameter displacement from the retain-only solution through shared feature directions. To test this hypothesis, we train language models from scratch while controlling when the forget set is introduced. Across different models, unlearning methods, and evaluation metrics, we find that knowledge maintained from earlier stages of pretraining is consistently more difficult to remove than the same knowledge introduced later. These results show that two models can fit the same forget data similarly yet differ substantially in how that knowledge is represented in their parameters and how readily it can subsequently be removed, depending on when the data entered training. Our findings motivate benchmarks that explicitly account for training history and the development of unlearning methods that are effective across more general and realistic training settings.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
unlearning, llms, pre-training

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_DCGcOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when forgetting is harder: removing the knowledge early in pretraining or unlearn than knowledge introduced later. The authors present theoretical analyses and controlled experiments on Pythia-410M and TinyLlama-1.1B, evaluating four unlearning methods using perplexity and membership-inference attacks.

Critical strengths

The paper's central question addresses an important limitation of post-training unlearning benchmarks. Training models with controlled forget-data introduction is more informative than evaluating only conventional fine-tuning settings.

Critical weaknesses

My specific issues with the paper are the following:

(1) The paper mainly measures the forgetting process in terms of forget-set perplexity and membership inference AUC. Higher perplexity does not necessarily mean good unlearning, since the model trained with retaining only has a forget-set perplexity of 14.4014.40, and several other post-unlearning models have forget-set perplexities close to 101310^{13}.

I highly recommend to use some other interpretable measures such as verbatim memorization (VerbMem), KnowMem, and accuracy on forget/retain sets. Note that the retain utility also needs to be measured against similar forgetting level.

(2) The theoretical analysis fails to prove the claimed difficulty of the unlearning process. The theorem only demonstrates that parameter distance of an unlearning approach from the retain-only method could be increased if the data was introduced early in the learning phase. This happens in a special case for a two-layer linear network with restrictive assumptions, and the parameter distance does not necessarily capture the difficulty (or unlearning cost) of the unlearning problem.

Questions for authors

Please check the Critical Weaknesses setion.

5 comments

We thank the reviewer for acknowledging that our work reveals an important limitation of post-training unlearning benchmarks. We address the reviewer’s concerns in the following.


W1. New metric requested by the reviewer. Thank you for suggesting a generation-based evaluation of forgetting. We have added VerbMem [1] results in Table 3 measuring the mean ROUGE-L F1 overlap between generated continuations and the original forget text, scaled by 100.

RegimeSettingGAGDR\mathrm{GA}_{\mathrm{GDR}}SimNPORMUCE-U
Maintainedτ=0.2\tau=0.212.316.956.795.51
Maintainedτ=0.4\tau=0.411.434.435.435.41
Maintainedτ=0.6\tau=0.611.284.315.115.05
Maintainedτ=0.8\tau=0.89.402.884.184.36
Fine-tunedηtrain=10−5\eta_{\mathrm{train}}=10^{-5}1.152.252.623.97

The models begin with similar mean VerbMem scores (11.87 - 12.34). Across all four methods, post-unlearning VerbMem decreases as τ\tau increases. Within each method, every evaluated FT model has lower residual VerbMem than every maintained-exposure model. These results extend the observed training-history dependence beyond perplexity: under the tested method-specific budgets, earlier maintained exposure leaves more residual reproduction of the forget set.

In addition to the metric requested by the reviewer we also provided more empirical evidence in our response to the reviewer xhRW on the following (please see our complete response for the results):

  1. Optimization steps required for forgetting. Rather than relying on forget-set perplexity, we measure the number of optimizer steps required to reach a calibrated Min-K membership-inference AUC of 0.5, corresponding to the retain-only reference. The results show that earlier-maintained knowledge consistently requires more optimization steps

  2. Forgetting–utility tradeoff. We further evaluated the forgetting–utility tradeoff by constructing a Pareto frontier over all tested learning rates and intermediate checkpoints, measuring retained utility using the change in WikiText-2 perplexity from each model's initial value.

[1] Huang, Jing, Diyi Yang, and Christopher Potts. "Demystifying verbatim memorization in large language models." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

[2] Shi, W., Lee, J., Huang, Y., Malladi, S., Zhao, J., Holtzman, A., ... & Zhang, C. (2025, May). Muse: Machine unlearning six-way evaluation for language models. In International Conference on Learning Representations (Vol. 2025, pp. 27797-27818).


W2: Theoretical results and restrictive assumptions.

We agree that a larger parameter distance from the retain-only model does not, by itself, imply greater unlearning difficulty. The purpose of our theoretical results is to characterize how training history shapes parameter displacement from the retain-only model through shared feature directions, rather than to establish that this displacement directly determines unlearning cost. Instead, we investigate this relationship empirically through several experiments spanning different models, unlearning methods, and evaluation metrics.

Regarding the assumptions underlying our theoretical analysis, we emphasize that they are grounded in recent prior work (Assumptions 3.3 and 3.4 are taken from Feng et al. (2026) and the small initialization follows Gidel et al. (2019)). As is common in the literature, our theoretical analysis is intended to provide mathematical intuition for the observed phenomena rather than to establish their validity in general LLMs. Importantly, our theoretical analysis complements, rather than substitutes for, the extensive empirical evidence presented in our work.

To further improve the empirical evidence on unlearning difficulty, we expanded our experiments to a larger model as well (Pythia 2.8B). As the following results show, we see the same trends for membership inference attacks and the forget-set perplexity.

Membership Inference figure for Pythia 2.8B: https://shorturl.at/ULYEs

Thank you for the detailed response and additional experiments. However, two concerns remain:

(1) Evaluation: VerbMem is a useful addition, but it primarily measures text reproduction rather than retained knowledge. Knowledge-based evaluations (e.g., KnowMem or QA accuracy) would provide stronger evidence.

(2) Theory: I appreciate the authors' clarification that parameter distance does not necessarily imply unlearning difficulty. However, this remains my central theoretical concern. The results establish parameter displacement under restrictive assumptions, rather than a direct relationship with unlearning cost or difficulty. Therefore, the theoretical contribution should be framed as providing intuition rather than establishing the claimed difficulty.

Overall, my main concerns remain only partially addressed. I will maintain my current rating.

Thank you for evaluating our responses and clarifying the remaining concerns.

(1) Evaluation: To complement VerbMem, we added the results for KnowMem and QA accuracy as requested by the reviewer. We used Claude Sonnet 5 to generate 40 multi-choice questions where the blanks are characters or place names used in the novel we use as the forget set. We used the recent larger Pythia 2.8B model to evaluate the average QA accuracy and KnowMem score [1] before and after unlearning with 15 epochs of SimNPO. As the following table shows, by decreasing the value of τ\tau we get a higher accuracy and higher KnowMem score after unlearning by decreasing the value of τ\tau and the values are higher for τ<1\tau < 1 compared to finetuned models (despite comparable initial accuracies), which confirms that it is harder to unlearn for lower values of τ\tau.

ModelAcc (Before)Acc (After Unlearning)KnowMem
τ=0.6\tau=0.666.350.87.3
τ=0.8\tau=0.864.042.37.1
FT η=1e−5\eta = 1e-554.00.04.6
FT η=2e−5\eta = 2e-562.33.05.8
FT η=5e−5\eta = 5e-566.614.06.4

[1] Shi, W., Lee, J., Huang, Y., Malladi, S., Zhao, J., Holtzman, A., ... & Zhang, C. (2025, May). Muse: Machine unlearning six-way evaluation for language models. In International Conference on Learning Representations (Vol. 2025, pp. 27797-27818).

(2) Theory: To address this concern, we extended our theoretical analysis to establish a direct lower bound on the geometric unlearning cost C(τ)C(\tau) defined in Definition 3.1, under the assumptions of our two-layer linear model and therefore close the gap in our theoretical analysis. The following file is a standalone file containing the theoretical results and the complete proofs for the new theorems:

https://shorturl.at/vMs3h

We summarize the main results below.

1. Function-space monotonicity. We first show that the coordinatewise ordering underlying Corollary 3.8 also establishes an ordering in function space:

\geq \lVert f_{\tau_2}(x)-f_r(x)\rVert_2, \qquad \forall x,\quad \tau_1<\tau_2.$$ Unlike Euclidean parameter distance, this functional discrepancy is invariant to function-preserving reparameterizations. Thus, the stage-dependent ordering is not limited to parameter-space measurements. **2. Lower bound on geometric unlearning cost.** We establish a lower bound directly on the minimum feasible parameter displacement $C(\tau)$: $$C(\tau)\geq g_S\left((D_\tau-\varepsilon_r)_+\right), \qquad g_S(q)=\sqrt{2S+2q}-\sqrt{2S},$$ where $D_{\tau} = \lVert M_{\tau}^{\ast} - M_r^{\ast} \rVert_F$ denotes the end-to-end functional distance, $S$ is the maximum target singular value, and $\varepsilon_r = \rho_r + \sqrt{\rho_r^2 + \xi_r}$, with $\rho_r = \lVert M_r^{\ast} - A_r \rVert_F$. This result follows from the retain-loss constraint and applies to all parameter updates satisfying the unlearning constraints in Definition 3.1, without requiring the unlearned model to preserve the original balanced-factor structure. **3. Connecting the original parameter-distance theorem to unlearning cost.** Let $B_{\tau}$ denote the right-hand side of Equation (2) in Theorem 3.6, which establishes $\lVert \theta_{\tau}^{\ast} - \theta_r^{\ast} \rVert_F^2 \geq B_{\tau}$. Using the balanced-factor structure and coordinatewise feature bounds from the original theorem, we derive a lower bound $H_{\tau}$ on functional distance: $$D_\tau\geq H_\tau \quad\Longrightarrow\quad \boxed{C(\tau)\geq g_S\left((H_\tau-\varepsilon_r)_+\right).}$$ So, **when endpoint forget loss and retain-loss gap are matched across stages, this lower bound is non-increasing with $\tau$**. This establishes a direct theoretical connection between the training-history-dependent parameter displacement characterized by Theorem 3.6 and our defined geometric unlearning cost. Together, the new results establish the following theoretical progression: **Training history → Parameter displacement → Function-space displacement → Lower bound on geometric unlearning cost** Our additional experiments measuring optimizer steps, VerbMem, KnowMem, Q/A accuracy, forgetting–utility tradeoffs, and MIAs provide complementary evidence of practical unlearning difficulty across training histories. We will incorporate these theoretical results and empirical analyses into the revised manuscript to clarify the connection between our structural analysis and practical unlearning difficulty.
Reviewer_xg7EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work investigated how the timing of the forgetting set being introduced to the training data impacts unlearning difficulty. From optimization theory of smooth functions and deep linear network analysis, the authors proved that when the forget-dataset DfD_f is introduced earlier and remains in the training dataset until the end, the distance between the parameters found by gradient-based optimization with and without DfD_f are longer, compared to the case when DfD_f is introduced later. They also empirically demonstrated this phenomenon in the fine-tuning of LLMs and its relationship with unlearning performance.

Critical strengths

The paper motivates the empirical results with theoretical analyses in two settings: smooth function optimization and deep linear networks, where they characterize the bounds on the gap in the parameters after learning with or without forget-set (but see the Weaknesses and questions below for their limitations).

Critical weaknesses

Throughout the paper, both in the analytical and numerical parts, the forget set was introduced at iteration τT\tau T and remained in the training dataset until the end of training. This training procedure thus confounds the impact of the position in the learning process and the duration of learning. I find it problematic because (1) it is rather trivial that, when the forget set is included in the training set for a longer number of iterations, it moves the parameter away from the one without the forget set. (2) Claims regarding the position of encounter, such as “data encountered earlier in training can become more strongly integrated” in the Abstract, are somewhat misleading as they’re presented.

The authors motivated the analysis of the gap in the learning weights with and without the forget set ∥θr∗−θτ∗∥\lVert \theta_r^* - \theta_{\tau}^* \rVert as an upper bound of the unlearning cost C(τ)C(\tau). However, in Theorem 3.6, a lower bound of this quantity is evaluated, which does not inform anything about the original quantity of interest, C(τ)C(\tau). Additionally, in Prop. 3.2, an upper bound on the second term RTR_T was studied, although the first term also depends on τ\tau. Hence, the theoretical results do not appear to present technically sound upper bounds on the defined unlearning cost in either setting as presented.

Questions for authors

What is the justification for focusing on the second RTR_T factor in Prop. 3.2? Similarly, why was a lower bound studied in Theorem 3.6?

In the unlearning experiments depicted in Table 1, how much did the parameter move compared to ∥θ−θretain∥2\lVert \theta - \theta_{retain} \rVert_2 evaluated in Fig. 2a? I wonder whether unlearning algorithms follow Δθ=θr∗−θτ∗\Delta \theta = \theta_r^* - \theta_{\tau}^*, or find a solution with much smaller parameter displacement.

If you define θr∗=Train(Dr[0,T],Df[τT,(τ+dτ)T])\theta^*_r = Train(D_r^{[0,T]}, D_f^{[\tau T,( \tau +d\tau)T]}), do see observe similar conclusion? Here, dτd\tau is a short, fixed duration.

5 comments

We would like to thank the reviewer for their insightful comments. We address the reviewer's concern in the following:


W1, Q3: Why longer exposure.

As the reviewer mentioned, our perplexity-gated, maintained-exposure protocol does not focus solely on the timing of the first exposure. The reason is that, if the forget set were seen only once, the final checkpoints (corresponding to different values of and different fine-tuned models) would have substantially different forget-set perplexities, introducing a stronger confounding factor into our comparisons; achieving a lower perplexity on the forget set basically means the model better regenerates the text and therefore it is harder to unlearn. In our work, we wanted to go beyond this confounding factor and investigate whether models with comparable forget-set perplexity necessarily have comparable unlearning difficulty. We believe showing different levels of unlearning difficult at the same perplexity is not trivial and not studied in prior work; If longer exposure had led to lower loss (and therefore lower perplexity), the result would have been trivial., but we make sure that the final models achieve comparable perplexity on the forget set. In fact, loss- and likelihood-based statistics have long served as important signals for membership inference and memorization analysis. Foundational membership inference methods exploit differences in training loss or model confidence between members and nonmembers [1], while more recent approaches for LLMs build on token-level likelihood statistics, including Min-K% Prob and Min-K%++ [2,3]. The implicit assumption in these works is that same perplexity indicates the same level of memorization. However, our results demonstrate that models with similar initial forget-set perplexity can still exhibit substantially different responses to unlearning, indicating that acquisition history provides an additional dimension for understanding the persistence of learned information and unlearning difficulty.

Theorem 3.6 further motivates this approach by decomposing a lower bound on the parameter distance from the retain-only model into terms associated with forget-set and retain-set loss, and a term related to the shared features and the effect of maintained-exposure timing. To isolate the effect of the last term control the other two terms in our experiments. As we mentioned in our experiment section this theorem guided us in designing the experiments. We will further clarify this motivation in the revised manuscript.

[1] Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., & Tramer, F. (2022, May). Membership inference attacks from first principles. In 2022 IEEE symposium on security and privacy (SP) (pp. 1897-1914). IEEE.

[2] Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., ... & Zettlemoyer, L. (2024, May). Detecting pretraining data from large language models. In International Conference on Learning Representations (Vol. 2024, pp. 51826-51843).

[3] Zhang, J., Sun, J., Yeats, E., Ouyang, Y., Kuo, M., Zhang, J., ... & Li, H. (2025, May). Min-k%++: Improved baseline for pre-training data detection from large language models. In International Conference on Learning Representations (Vol. 2025, pp. 64845-64862).

W2: Unlearning cost and the parameter distance.

As the reviewer mentioned, Theorem 3.6 provides a lower-bound on the parameter distance from the retrain model. This theorem is used in Corollary 3.8 to show that the parameter distance is monotonic with respect to the stage the forget set is introduced to the model (). This has been further verified through empirical evidence (Figures 2 and 6).

We agree that parameter distance should not be equated with unlearning difficulty. Definition 3.1 introduces an idealized minimum parameter correction satisfying specified forget- and retain-loss constraints. Since the retain-only model is a feasible solution, its distance from the forget-exposed model provides an upper bound on this quantity. However, neither Theorem 3.6 nor Corollary 3.8 establishes monotonicity of the minimum unlearning cost. Instead, our work provides complementary theoretical and empirical findings: the theory characterizes history-dependent parameter displacement, while the experiments investigate whether models with different acquisition histories exhibit different resistance to practical unlearning methods.

To further improve the empirical evidence on unlearning difficulty, we expanded our experiments to a larger model as well (Pythia 2.8B). As the following results show, we see the same trends for membership inference attacks and the forget-set perplexity.

Membership Inference figure for Pythia 2.8B: https://shorturl.at/ULYEs

Forget-set perplexity figure for Pythia 2.8B: https://shorturl.at/ZbVeV

Q1: Theoretical results.

The purpose of our theoretical results is to show how the parameter distance from the retrain model is connected to the stage the forget-set is included in the training set. Proposition 3.2 considers a general differentiable model and characterizes this dependence through a sum of forget–retain gradient discrepancies and a remainder term RT. The remainder bound explicitly depends on through the factor (1-)T, which represents the number of training steps following the introduction of the forget set. Thus, earlier exposure allows the approximation error to accumulate over more steps, resulting in a potentially larger upper bound on RT. However, as the reviewer pointed out we could move the complete version of the proposition to the appendix and just mention that RT can be upperbounded by a term that also depends on the value of \tau.

As mentioned in our paper, although Proposition 3.2 provides a lower bound on the parameter distance that depends on , it does not establish the stronger conclusion about the monotonicity of the distance in . To establish this monotonicity, we move to a more restrictive setting of two-layer linear networks (which has been used in prior work as well [1]) where we could prove this stronger result (Corollary 3.8). To further use empirical evidence to show that this monotonicity holds in LLMs as well (Figure 2 and 6).

[1] Feng, L., Ghosal, G. R., Springer, J. M., Zhong, Z., & Raghunathan, A. (2026). Early Data Exposure Improves Robustness to Subsequent Fine-Tuning. arXiv preprint arXiv:2605.12705.


Q2: Parameter displacement.

We thank the reviewer for suggesting this interesting experiment. The following table shows the parameter displacement for each of the Pythia 410M unlearned models compared to their initial distance from the retrained model. For each unlearning method, we continue the iterations until MIA shows a calibrated AUC of 0.5 (same as the retrain model). Brackets are 95% bootstrap intervals over the evaluation set (1,000 paired resamples of the forget and non-member chunks) and the percentage shows the displacement with respect to distance from the retrain model.

As the table shows, the displacement is monotonic and increases as we decrease the value of \tau. However, these values are still much smaller than the initial distances from the retrained model which verifies the earlier point that the reviewer raised and is correctly characterized in our definition of the parameter distance being an upper-bound on the unlearning cost. We believe this additional result provides stronger evidence of the increase of unlearning cost with smaller values of \tau and motivates future research on deriving tighter lower bounds on the unlearning cost based on the parameter distance.

Methodτ = 0.2τ = 0.4τ = 0.6τ = 0.8Fine-tune η = 1e-5
‖θ_0 − θ_retain‖₂468.5323.2233.2166.54.9
GA+GD3.27 [3.25, 3.31] · 0.70%2.83 [2.81, 2.85] · 0.88%2.51 [2.48, 2.53] · 1.08%2.25 [2.22, 2.27] · 1.35%1.45 [1.42, 1.48] · 29.4%
SimNPO5.14 [5.01, 5.27] · 1.10%4.25 [4.13, 4.35] · 1.31%3.49 [3.43, 3.55] · 1.50%3.52 [3.41, 3.57] · 2.10%2.53 [2.48, 2.59] · 51.5%

I thank the authors for the rebuttal. Nevertheless, my concerns regarding the significance and soundness of the work remain only partially resolved.

I appreciate the authors' efforts in conducting additional numerical experiments on parameter displacement. However, the results, a roughly hundredfold gap in the distances, imply the vacuous nature of the bound C(τ)≤∥∣θτ∗−θr∗∥∣FC(\tau) \leq \|| \theta_{\tau}^{\ast} - \theta_{r}^{\ast} \||_F.

While it is common for theoretical bounds in deep learning systems to be vacuous, this weakens the connection between the theoretical and numerical results. Although I agree that the τ\tau-dependence of C(τ)C(\tau) is not necessarily trivial, the τ\tau-dependence of ∥∣θτ∗−θr∗∥∣F\|| \theta_{\tau}^{\ast} - \theta_r^{\ast} \||_F follows straightforwardly from the cumulative effect of learning, which also diminishes the significance of the theoretical contribution.

Reviewer_mG9cOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper examines how the timing of introducing forget data influences the difficulty of machine unlearning, revealing that even when models exhibit comparable initial fit to the forget data, models exposed to such data earlier typically diverge more from models trained exclusively on retain data and are more difficult to effectively unlearn by existing unlearning methods.

Critical strengths
  1. The definition of machine unlearning and the description of the process are clear, and the text is easy to read.
  2. The relationship between “parameter shifts in models trained at different stages of forgetting set injection and those in models that retain only the original data” is demonstrated through both theory and experiments.
  3. The experiment demonstrated that models with similar initial forgetting set perplexity may exhibit significantly different responses to the same forgetting algorithm, depending on when the forgotten data was introduced during the training process. This finding is valuable and insightful.
Critical weaknesses
  1. There is a lack of explanation regarding the key relationship between a larger parameter distance and harder unlearning.
  2. There are some claims with vague concepts, such as “more localized changes.”
Questions for authors
  1. The abstract states: “…whereas data introduced later may induce more localized changes.” This claim is not directly proven, or the statement is ambiguous. Corollary 3.8 and Section 5.1 show that the later the data is introduced, the smaller the distance between the retain model parameters, not that there are “more localized parameter changes”. These concepts are not equivalent.
  2. Why can the minimum parameter displacement serve as the unlearning cost? Further explanation is needed. Is the distance in parameter space sensitive to model reparameterization? Taking a two-layer linear network as an example, it is not difficult to construct models with identical functions but large parameter distances, such as W₁*W₂ = AW₁W₂A⁻¹, we know that W₁’ = AW₁, W₂’ = W₁A⁻¹, and the parameter distance between W₁’ W₂’ and W₁ W₂ can be very large. Furthermore, this cost is easily misunderstood. It is neither a computational cost nor the actual number of iterations required.
  3. This raises the next question that the author needs to further justify why a larger parameter distance equals harder unlearning or unlearning difficulty. Alternatively, the author should clarify the scope of the theoretical claim.
4 comments

We thank the reviewer for their constructive comments and positive assessment of our theoretical and empirical contributions. We address the concerns below.


W1, Q2, Q3: Larger parameter distance and unlearning.

We agree that a larger parameter distance from the retain-only model does not, by itself, imply greater unlearning difficulty, and that raw parameter distance is not invariant to function-preserving reparameterizations. Definition 3.1 introduces minimum parameter displacement as an idealized measure of the smallest parameter intervention satisfying specified forget- and retain-loss constraints. It is not intended to measure computational cost or optimizer iteration complexity. Since the retain-only parameters constitute a feasible solution, the initial parameter distance provides an upper bound on this idealized cost. However, this bound need not be tight, and our theoretical results do not establish that larger parameter distance necessarily implies greater unlearning cost. Instead, our theoretical analysis characterizes how training history shapes parameter displacement through shared feature directions, while our experiments investigate whether different acquisition histories are associated with different levels of practical unlearning difficulty.

Also we want to emphasize that within our theoretical setting, the reparameterization ambiguity you mentioned is avoided by the assumptions on initialization and optimization dynamics. Specifically, the target maps of the two datasets share singular vectors (Assumption [3.3]), and both models are trained by gradient descent from the same balanced initialization (Assumption [3.4]). Lemma A.1 establishes that their parameters remain balanced and aligned with the same singular-vector bases throughout training. Consequently, their parameter distance can be expressed entirely in terms of the differences between the square roots of the singular values of their end-to-end maps W2W1W_2 W_1. Therefore, within this restricted setting, the distance is determined by the learned functions rather than arbitrary choices of parameterization. Still, we acknowledge that this property does not hold for general neural networks or arbitrary function-preserving reparameterizations. Moreover, we conducted two additional analyses on Pythia 410M to address the reviewer’s concerns regarding parameter distance invariance, parameter displacement evaluation, and actual number of steps to reach the retrain model MIA AUC.

1. Function-space differences. We measured the KL divergence between the next-token distributions of the retain-only model and each trained model, averaged over tokens. Because this measure depends only on model outputs, it is invariant to function-preserving reparameterizations. On the forget set, KL divergence decreases monotonically. The ordering also holds after subtracting the divergence on general Pile text (0.205, 0.147, 0.100, and 0.089).

τ = 0.2τ = 0.4τ = 0.6τ = 0.8
Parameter distance ‖θ − θ_retain‖₂468.5323.2233.2166.5
KL on the forget set0.3760.2690.1900.158
KL on general Pile text0.1710.1220.0900.069

2. Actual unlearning effort. We measured the optimizer steps and parameter displacement produced by unlearning to reach a calibrated MIA AUC of 0.5, matching the retain-only reference under our calibration. As the following table shows, both quantities decrease monotonically with τ\tau.

ModelForget-set KL (nats/token)Unlearning steps to AUC 0.5 [95% CI]Unlearning displacement at AUC 0.5
τ=0.2\tau=0.20.376121.1 [119.6, 122.9]3.3
τ=0.4\tau=0.40.26998.2 [97.1, 99.2]2.8
τ=0.6\tau=0.60.19079.5 [78.2, 80.6]2.5
τ=0.8\tau=0.80.15869.4 [68.2, 70.7]2.2

We will revise the paper to emphasize the study of parameter distance as a description of how training history shapes the model and as part of the experimental setup. We will also add the function-space comparison and the measured unlearning steps and displacements.

W2 and Q1: “More localized changes" is vague.

We agree with both points. A smaller distance from the retain-only model is not the same as a more localized change, and the paper used "localized" without defining it. In the revision we will clarify what it means.

What "localized" means in our theory. In the two-layer linear model, both the stage-τ\tau model and the retain-only model have end-to-end maps of the form W2W1=Udiag(w)V⊤W_2W_1=U\mathrm{diag}(w)V^\top (Lemma A.1). The difference between them therefore splits over feature directions. We call a change localized when, at a given fit to the forget set, it stays in the directions specific to the forget set and does not extend into directions shared with the retain data. The analysis shows that late injection is more localized in this sense:

The simulation in Appendix C.1 confirms this on synthetic data for the actual distance, not just the bound. As τ\tau goes from 0 to 0.8, the contribution of the common block to the squared distance falls from 1.51 to 0.25, while the forget-specific and retain-specific blocks change only slightly, and in the opposite direction. With no shared features (kc=0k_c=0), the stage effect disappears. So at the same forget loss, late injection is mainly localized to the forget-specific directions, while early injection also moves the shared ones. Please note that these quantities depend only on the singular values of the end-to-end map, so this statement is unaffected by reparameterizations such as W1→AW1W_1\to AW_1, W2→W2A−1W_2\to W_2A^{-1}.

Thanks for your response, which has addressed my concerns. However, considering the overall novelty and merit of the manuscript, I maintain my current score.

We thank the reviewer for their continued engagement and for acknowledging that our responses have addressed their concerns.

We would also like to highlight several additional contributions introduced during the rebuttal. We expanded our experiments to a larger model (Pythia-2.8B) and an additional forget dataset (LUME), and introduced further evaluations using VerbMem, KnowMem, QA accuracy, and function-space KL divergence. We also analyzed the optimization steps required for unlearning, actual parameter displacement, and forgetting–utility tradeoffs across learning rates and stopping points. Furthermore, we strengthened our theoretical analysis by establishing a monotonic ordering in function-space discrepancy and deriving a direct lower bound on geometric unlearning cost under our two-layer linear model assumptions.

We have also submitted a revised version of the manuscript incorporating these additional results and clarifications addressing the reviewers' feedback.

We believe these additions substantially strengthen the paper's technical contributions and empirical significance, providing broader evidence that training history is an important and underexplored dimension of machine unlearning. We would be grateful if the reviewer would reconsider their overall assessment and consider increasing their support for our work in light of these improvements.

Reviewer_xhRWOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how acquisition history affects machine unlearning. It combines theoretical analysis of training trajectories and two-layer linear networks with experiments on Pythia-410M and TinyLlama-1.1B. Despite similar initial forget-set perplexity, models exposed to the target data earlier and repeatedly during pretraining generally respond less strongly to fixed unlearning procedures.

Critical strengths
  1. The paper questions whether successful unlearning of recently fine-tuned data transfers to pretraining knowledge. Matching initial forget-set perplexity provides a useful control.

  2. Theoretical analysis, synthetic verification, two model families, and four unlearning methods support the observation that endpoint fit alone does not explain unlearning response.

Critical weaknesses

The experiment does not isolate acquisition timing. Earlier conditions receive substantially more injections: 63 episodes at τ=0.2\tau = 0.2 versus 19 at τ=0.8\tau = 0.8 (Appendix B.5). Repetition, maintenance duration, and acquisition stage therefore vary together. The authors acknowledge this, but the central interpretation remains stronger than the control permits. Add an equal-exposure comparison across stages, alongside the existing matched-perplexity comparison, or narrow the claim to maintained-exposure history.

The evaluation establishes responsiveness more clearly than successful unlearning. Table 1 treats large perplexity increases as evidence of easier removal, although many values greatly exceed the retain-only reference of 14.40. Such overshooting does not establish retrain-equivalent behavior. Hyperparameters selected using the late-stage condition are also applied unchanged to all histories. Compare the compute needed to approach retrain-like behavior at matched retained utility, using early stopping and equally budgeted tuning.

Questions for authors
  1. Does earlier exposure worsen the attainable forgetting–utility tradeoff, or mainly require different learning rates and stopping points?
  2. What is the exact calibrated-AUC formula, and how are nonmembers selected? Please report raw AUC and uncertainty alongside the calibrated values.
4 comments

We thank the reviewer for recognizing the theoretical analysis, experimental results, and the theory-backed useful control in our experiments. We address the reviewer’s concerns in the following which we believe would make our work stronger:


W1: Acquisition stage versus cumulative forget-set exposure.

We agree that our perplexity-gated, maintained-exposure protocol does not focus on only the single first-exposure timing. The reason is that by only seeing the forget set once, the final checkpoints for different τ\tau values and fine-tuned model would have very different forget set perplexities which becomes a stronger confound in our comparisons. Achieving a lower perplexity on the forget set basically means the model regenerates it easier and therefore it is harder to unlearn. In our work, we wanted to go beyond this confounding factor and investigate whether models with comparable forget-set perplexity necessarily have comparable unlearning difficulty. Theorem 3.6 further motivates this approach by decomposing a lower bound on the parameter distance from the retain-only model into terms associated with forget-set and retain-set loss, and a term related to the shared features and the effect of maintained-exposure timing. To isolate the effect of the last term control the other two terms in our experiments. We will clarify this motivation in the revised manuscript.


W2 and Q1: Optimization Cost and Forgetting–Utility Tradeoff.

We thank the reviewer for this observation. We agree that a large increase in forget-set perplexity may not indicate the behavior of the retrain model, particularly when it substantially exceeds the retain-only model's perplexity. However, it helps us to measure the responsiveness of models with different training histories under the same unlearning procedure and computational budgets. We also presented results using multiple membership inference attacks (MIAs), which are commonly used to evaluate unlearning effectiveness.

To address the reviewer's concerns, we conducted additional experiments with GA+GD across four learning rates 5×10−6,10−5,2×10−5,5×10−55 \times10^{-6}, 10^{-5}, 2 \times10^{-5}, 5\times10^{-5}, evaluating intermediate checkpoints over a budget of up to 240 optimizer steps. We performed two complementary analyses:

  1. Optimization steps required for forgetting. Rather than relying on forget-set perplexity, we measure the number of optimizer steps required to reach a calibrated Min-K membership-inference AUC of 0.5, corresponding to the retain-only reference. The results show that earlier-maintained knowledge consistently requires more optimization steps (n.r. stands for never reached).
Unlearnerlrτ = 0.2τ = 0.4τ = 0.6τ = 0.8FT η=1e−5\eta = 1e-5
GA+GD5e-6n.r. (0.990)n.r. (0.964)222 [219, 225]193 [191, 196]102 [98, 106]
GA+GD1e-5121 [120, 123]98 [97, 99]80 [78, 81]69 [68, 71]27 [26, 29]
GA+GD2e-538 [37, 38]35 [34, 36]29 [28, 29]23 [22, 24]< 8
GA+GD5e-511.2 [10.9, 11.5]11.5 [11.3, 11.7]< 8< 8< 8

(first checkpoint is at step 8)

  1. Forgetting–utility tradeoff. We further evaluated the forgetting–utility tradeoff by constructing a Pareto frontier over all tested learning rates and intermediate checkpoints, measuring retained utility using the change in WikiText-2 perplexity from each model's initial value. Therefore, even after optimizing over learning rates and stopping points, the utility cost remains consistently ordered across acquisition stages at matched forgetting levels. So, models exposed to the forget set earlier and maintained for longer not only require more optimization steps but also incur greater retained-utility degradation to reach the same calibrated membership-inference threshold.

Pareto-frontier figure: https://shorturl.at/Jkiwt

Q2: Calibrated-AUC formula and uncertainty.

For each chunk xx, we compute the calibrated score as

scal(x;θ)=s(x;θ)−s(x;θretain)s_{\mathrm{cal}}(x;\theta)=s(x;\theta)-s(x;\theta_{\mathrm{retain}}),

where ss denotes the attack statistic (Min-K%, Min-K%++, or the negated zlib ratio). All statistics are oriented such that higher values indicate greater likelihood of membership. Here, θretain\theta_{\mathrm{retain}} is the retain-only model which serves as the reference model trained without injecting DfD_f. We compute the AUC by distinguishing forget-set chunks from nonmember chunks. For θretain\theta_{\mathrm{retain}}, all calibrated scores are zero yielding an AUC of exactly 0.5. Values below 0.5 indicate that the calibrated attack scores rank forget-set examples below nonmembers more often than above them. This calibration is common practice adopted by prior work [1,2,3].

We will incorporate these results into the revised manuscript and clarify the distinction between unlearning responsiveness and retrain-equivalent forgetting. We would like to thank the reviewer for their insightful comments and we hope our responses increase their support for our work.

[1] Shi, W., Lee, J., Huang, Y., Malladi, S., Zhao, J., Holtzman, A., ... & Zhang, C. (2025, May). Muse: Machine unlearning six-way evaluation for language models. In International Conference on Learning Representations (Vol. 2025, pp. 27797-27818).

[2] Watson, L., Guo, C., Cormode, G., & Sablayrolles, A. (2021). On the importance of difficulty calibration in membership inference attacks. arXiv preprint arXiv:2111.08440.

[3] Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettlemoyer, L., ... & Hajishirzi, H. (2024). Do membership inference attacks work on large language models?. arXiv preprint arXiv:2402.07841.

Reviewer_xhRWOpenReview ↗

Thanks for the thoughtful rebuttal

Thanks for running the extra GA+GD experiments. The step counts make the difference easier to see. At the same LR of 1e-5, it takes 121 steps for τ = 0.2 and 69 for τ = 0.8 to reach the same calibrated AUC. The Pareto plot is also helpful, since the earlier runs still seem to lose more utility at matched forgetting after sweeping LR and stopping points.

I get why you matched the initial perplexity, thats a useful control. The thing I would still clarify in the paper is that the earlier runs also got more injections. I would phrase the takeaway around data that was introduced earlier and kept in training for longer.

I am keeping my weak accept (3) and increasing my confidence. The new GA+GD sweep makes me more confident in the forgetting–utility gap, since it still shows up after tuning the learning rate and stopping point for each training history. I still think that’s a useful result for unlearning benchmarks. My main concern is that the earlier runs also got more injections, so we still can’t tell how much of the gap comes from timing alone.

We thank the reviewer for their continued support and constructive feedback.

We would also like to highlight several additional results presented in our response to other reviewers. On the theoretical side, we establish a monotonic ordering in function-space discrepancy and derive a direct lower bound on geometric unlearning cost under our two-layer linear model assumptions, strengthening the connection between training history and unlearning difficulty. On the empirical side, we introduce additional evaluations, including experiments on Pythia-2.8B, VerbMem, KnowMem, QA accuracy, optimization steps required for unlearning, and forgetting–utility tradeoffs. These results provide further evidence supporting our central findings.

We have also submitted a revised version of the manuscript incorporating these new theoretical and empirical results, along with clarifications and improvements addressing the reviewers' feedback.

We believe these additions substantially strengthen the paper and address several important concerns raised during the review process. We would be grateful if the reviewer would consider increasing their support for our work in light of these improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.