acceptodds
Under review as a conference paper at ICLR 2027

Forget What Comes Next: LLM Unlearning via Statistical Independence

Abstract

Large language models (LLMs) can memorize and reproduce fragments of their training data, raising privacy and legal concerns that motivate machine unlearning. Yet poisoned training examples can remain statistically detectable after unlearning, highlighting the need for a deeper theoretical understanding of how unlearning objectives relate to memorization. In this work, we introduce PINT, a novel algorithm for machine unlearning that uses the Hilbert-Schmidt Independence Criterion (HSIC) to reduce the statistical dependence between tokens to be forgotten and the internal representations used to predict them. We formally relate our HSIC-based objective to exact memorization through a computable upper bound, connecting the training objective directly to the model's ability to reproduce forgotten tokens. In extensive experiments, we compare forgetting-utility trade-offs across nine settings with models up to 7B parameters and demonstrate scalability to 70B parameter models. Our evaluations cover unlearning personal information, hazardous knowledge in biology and cybersecurity, copyrighted books and news articles, and Gaussian poisons. PINT achieves the strongest forgetting-utility trade-offs on most benchmarks. On Gaussian poisons, it is the only method to make the poison undetectable while maintaining near-original model utility.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine unlearning, large language models, statistical independence, HSIC, memorization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tv4TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PINT to understand how an LLM can forget a memorized sequence per token. PINT is an LLM unlearning objective that minimizes HSIC between prefix representations and next tokens while preserving retain-set representations. It derives an HSIC-based upper bound on exact memorization and reports strong forgetting–utility trade-offs across several benchmarks. This paper conducts extensive experiments and especially presents strong performance in Gaussian poison, but the evidence does not establish that the target data’s influence is truly removed.

Critical strengths
  1. PINT achieves competitive forgetting–utility trade-offs across multiple benchmarks, model families, and data types. The broad evaluation and Pareto-frontier analysis provide evidence that the method is effective beyond a single task.

  2. I like its validation on Gaussian poison. It is the only evaluated method that makes the injected poison statistically undetectable while preserving near-original utility. This is consequential because the poison test measures a trace of training-data influence more directly than standard output-based forgetting metrics.

Critical weaknesses
  1. I have main reservation on the novelty. The paper discusses prior uses of HSIC and statistical independence in machine unlearning, including HSIC-based model updates for image and graph classifiers (Han et al., 2024) and HSIC-based unlearning evaluation (Zhang et al., 2026). Representation-level objectives have also been widely studied for LLM unlearning. PINT may be the first to minimize HSIC specifically between prefix representations and next-token targets for LLM unlearning. However, this contribution primarily combines established ingredients in a new setting. The paper therefore offers a useful objective and strong empirical study, but its conceptual novelty appears incremental. I would like to see more insights rather than A+B.

  2. Insufficient evidence of genuine unlearning. The theoretical result only bounds exact next-token memorization on the observed forget positions. It does not establish equivalence to retraining or removal of the target data’s influence. Moreover, the experiments also lack strong recovery attacks, such as nonlinear probes, relearning tests, and adaptive extraction, so the forgotten information may remain recoverable.

Questions for authors
  1. Beyond applying HSIC to prefix representations and next-token targets, what is the substantive technical distinction between PINT and prior independence-based unlearning objectives?

  2. Is the theoretical connection to exact memorization specific to PINT, or could a similar result be derived for existing representation-level objectives?

  3. What evidence shows that reducing empirical next-token memorization removes the forget data’s influence, rather than merely suppressing its reproduction on the evaluated prefixes?

  4. How tight is the HSIC-based bound in absolute terms, and how stable is CθC_\theta across models and kernel configurations?

  5. Do PINT’s advantages persist under nonlinear probes, relearning attacks, adaptive extraction, and paraphrased or indirect prompts? Please also clarify whether the Pareto comparisons remain consistent across multiple seeds and under matched parameter-update scopes and tuning budgets.

2 comments
AuthorsOpenReview ↗

Response to Reviewer tv4T

We thank you for recognizing our broad evaluation and PINT's strong results on the Gaussian poisons. Below, we clarify the distinction from prior work, and, in addition to the existing evidence in the paper, we ran all the new experiments suggested in your review. Revisions are in GREEN, and those requested by several reviewers in GRAY. All works we cite are referenced in the paper.


W1, Q1: Clarification regarding prior works

We respectfully disagree and clarify that prior works use HSIC in a fundamentally different role. Zhang et al. (2026) do not unlearn: they use HSIC as a test statistic to evaluate existing unlearning methods on image classifiers. Han et al. (2024) use HSIC as a regularizer that preserves accuracy after data removal in an influence-function update of image and graph classifiers, not as the forget objective. In contrast, PINT formulates forgetting as removing the dependence between prefix representations and next tokens, and is, to the best of our knowledge, the first to minimize HSIC for LLM unlearning.

Beyond our empirical study, this formulation yields two contributions that neither prior work provides: 1) an upper bound on Exact Memorization (Theorem 3.2), and 2) a formal relation to representation collapse (Prop. 3.3), which we show is an unnecessary extreme that, on bioS, also removes closely related knowledge (RMU 19%, PINT 97%, Fig. 8).

Q2: Is the connection to EM specific to PINT?

Theorem 3.2 holds more broadly for every model θ\theta, so it bounds the memorization of any model. What is specific to PINT is that it directly minimizes the HSIC in the bound (Fig. 4). For other representation-level objectives, such as RMU, our Prop. 3.3 shows that collapsing representations onto a fixed target, as RMU does, is one extreme way to reduce this HSIC, which relates such objectives to our framework. However, collapse removes more than the memorized dependence and sacrifices utility that PINT preserves (Fig. 8).

W2, Q3 and Q5: Additional evidence of unlearning

We ran all the suggested experiments and would also like to clarify that our existing evaluations already go far beyond reproduction on the prefixes: 1) the statistical test for Gaussian poisons detects the data's influence on the gradients, 2) linear probes at every layer test whether the hidden states can predict the forgotten tokens after unlearning, 3) bioS asks the forgotten facts in unseen phrasings, 4) WMDP unlearns papers and code passages but tests the hazardous knowledge via multiple-choice questions, and 5) MUSE asks open questions about the unlearned books and news articles.

We ran the four attacks you suggest, each following prior work: 1) the relearning attack and 2) adaptive extraction by removing the unlearning direction, both by Łucki et al. (2024), 3) a nonlinear probe (Guo et al., 2024), and 4) TOFU's paraphrases by Maini et al. (2024). We tune each attack on the same grid for every method, and report its strongest result (App. E.6).

TOFU 10%Relearning ↓Adaptive extraction ↓Nonlinear probe ↓Paraphrases ↓
NPO+0.4 ± 0.3−0.4 ± 0.141.5 ± 0.533.8 ± 0.7
GradDiff+1.7 ± 1.30.3 ± 0.342.8 ± 0.732.8 ± 1.7
SimNPO+1.1 ± 1.00.4 ± 0.243.5 ± 0.333.2 ± 0.5
UNDIAL+0.9 ± 0.70.8 ± 0.536.2 ± 0.230.2 ± 0.7
RMU+25.1 ± 1.423.9 ± 2.435.8 ± 0.329.7 ± 0.5
PINT−1.2 ± 1.01.3 ± 0.733.1 ± 0.428.9 ± 1.0

PINT has the lowest value under relearning, the nonlinear probe and paraphrases, while adaptive extraction recovers at most 1.3% from all methods but RMU (23.9%). We use three seeds wherever computationally feasible, including all evaluations in this rebuttal. To demonstrate seed stability, we also reran every MUSE Books setting with Forget KnowMem between 0.2 and 0.4 using 3 seeds in new Fig. 12: PINT remains the strongest in new Tab. 11. All methods follow a shared protocol (App. C.2) over ~2,400 settings: the same nine runs on the Gaussian poisons and the same search procedure elsewhere.

Q4: Bound tightness and CθC_\theta stability

We would like to clarify that Fig. 4 already reports tightness in absolute terms across models. The existing kernel ablation (App. E.3) confirms PINT's stability across kernels. In App. E.2 (Fig. 11), CθC_\theta never cancels the decrease in HSIC. We additionally ran new experiments measuring CθC_\theta for the remaining kernels (App. E.3), on TOFU 10% at the retrained model's forget ROUGE:

KernelCθC_\theta rel. to original
Gaussian (PINT)1.00
Matérn-3/21.00
Laplacian0.98

We thank you for the feedback. Following the requests, we clarified the distinction from prior work and ran extensive new experiments, confirming PINT's strength. In light of the new evidence, please consider increasing your score.

Dear Reviewer,

Thank you for your detailed feedback. Your comments have further improved the quality of our work, deepened our analysis, and strengthened our evaluation of unlearning robustness.

We are writing to inquire whether you have any further questions we could clarify during the discussion period, which ends in 2 days. In our response, we clarified the distinction from prior work and substantially extended the evidence of unlearning with new experiments on relearning, adaptive extraction, nonlinear probes and paraphrased prompts, which we also added to the revised paper. We hope we have addressed your concerns. If so, we kindly ask you to consider updating your score.

Reviewer_BPZLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PINT, which reduces statistical dependence between final-layer prefix representations and ground-truth next tokens by minimizing HSIC on the forget set, while constraining retain-set representations to remain close to those of the original model. The method is straightforward to implement, and its effectiveness is supported by theoretical analysis and a reasonably extensive and diverse set of experiments.

Critical strengths
  1. Using HSIC as a training objective for LLM unlearning is well motivated and offers a useful perspective. By directly reducing statistical dependence between hidden representations and target tokens, PINT approaches unlearning from a perspective distinct from suppressing answer probabilities, with the potential to inform future work in this area.
  2. The paper introduces an independence baseline for Exact Memorization, explicitly accounting for chance matches attributable to token frequencies. It then derives an HSIC-based conditional upper bound on EM in excess of this baseline over a finite forget set, providing theoretical support for using HSIC as an unlearning objective.
  3. The paper provides detailed theoretical analysis and further discusses the connection between PINT and the established method RMU, helping clarify the rationale behind its design. Experiments covering multiple unlearning scenarios also provide broad empirical validation of the method’s performance.
Critical weaknesses
  1. The upper bound in Theorem 3.2 depends on the model-dependent quantity CθC_\theta, which is not directly controlled by the training objective. Although the authors examine its variation in Appendix E.2, reducing HSIC alone does not unconditionally guarantee a decrease in EM. I would therefore appreciate a clearer explanation of which changes in the representations drive unlearning in practice, and the respective roles of the reduction in HSIC and the change in (C_\theta) in explaining the observed decrease in EM. This would clarify how far the theoretical analysis explains the practical unlearning mechanism.
  2. Proposition 3.3 compares losses evaluated on the same representations, whereas the actual PINT and RMU implementations differ in their intervention layers, the locations of the retention constraints, and, on some tasks, the sets of parameters updated. The experiments therefore support the performance advantage of the overall training configuration, but do not fully separate the contributions of the HSIC forgetting term, the final-layer retention constraint, and the choice of trainable parameters. Distinguishing these contributions would strengthen the explanation of where the method’s advantages come from.
Questions for authors

No question for authors.

2 comments
AuthorsOpenReview ↗

Response to Reviewer BPZL

We are grateful for your positive assessment and careful reading of our theory, and we are glad that you find HSIC a well motivated and theoretically supported unlearning objective. Revisions based on your feedback are marked in BLUE, and changes requested by several reviewers in GRAY. We address both of your concerns below.


W1: Role of HSIC and CθC_\theta in unlearning

Thank you for this thoughtful question. As you note, the bound of Theorem 3.2 depends on HSIC and the model-dependent factor CθC_\theta. Appendix E.2 (Figure 11) already shows that the bound decreases together with HSIC at every checkpoint across forgetting strengths. Following your suggestion, we extended this analysis with two new experiments, which confirm that the observed decrease in EM is explained by the reduction in HSIC rather than by a change in CθC_\theta.

Changes in the representations: Before unlearning, prefix representations depend on their next token: those with the same next token lie closer than those with different ones, which allows the model to predict it. Unlearning should reduce this dependence while preserving the diversity of the representations. At the retrained model's level of Exact Memorization, we report the normalized variance E∥z−zˉ∥2/E∥z∥2\mathbb{E}\|z-\bar{z}\|^2 / \mathbb{E}\|z\|^2 of the forget set representations zz (0 means collapsed, 1 means as diverse as before unlearning) and their HSIC with the next tokens (0 means independent, 1 means as dependent as before unlearning), both relative to the original model:

TOFU 1%TOFU 5%TOFU 10%MUSE NewsMUSE Books
Norm. variance, retrained model1.000.990.990.950.97
Norm. variance, PINT0.990.991.000.950.93
HSIC, PINT0.440.480.450.390.36

PINT preserves 93-100% of the diversity of the representations, while their HSIC decreases to 36-48% of its original value. PINT thus unlearns by moving representations with the same next token apart while preserving their diversity, as illustrated in the new Figures 3 and 5 in the main paper.

Roles of HSIC and CθC_\theta: To separate the roles of HSIC and CθC_\theta in explaining the observed decrease in EM, we also report EM, HSIC\sqrt{\mathrm{HSIC}} and CθC_\theta after unlearning, relative to the original model:

TOFU 1%TOFU 5%TOFU 10%MUSE NewsMUSE Books
EM0.660.660.630.630.53
HSIC\sqrt{\mathrm{HSIC}}0.660.690.670.620.60
CθC_\theta1.111.031.001.071.00

On every benchmark, the decrease in HSIC explains the decrease in EM. Our theory thus explains the observed unlearning. We added both experiments to Appendix E.2 and E.5.


W2: Objective ablation

We agree that Proposition 3.3 compares the two forget terms on the same representations. For the best performance of each method and a fair comparison, each method in our experiments uses its own configuration, tuned in its original work and further by us (Appendix C.2), so on some settings the two differ in the intervention layer, the location of the retention constraint and the trainable parameters. To distinguish these contributions, we added a new ablation on MUSE Books, where the two configurations differ in all three components. Starting from PINT, each row replaces one more component with RMU's, until the last row is RMU itself. For each row, we tune the resulting objective, rerun every setting with Forget KnowMem between 0.2 and 0.4 using 3 seeds, and report the setting with the highest Retain KnowMem among those that forget at least as much as the retrained model (mean ± std over 3 seeds):

Forget termLayerTrained parametersRetain KnowMem ↑Forget KnowMem
Retrained model0.6960.303
PINTHSICfinalall layers0.673 ± 0.0090.289 ± 0.021
RMU'sfinalall layers0.615 ± 0.0060.242 ± 0.025
RMU's7all layers0.565 ± 0.0100.278 ± 0.025
RMURMU's7down-proj. of layers 5-70.632 ± 0.0070.250 ± 0.023

The ablation confirms the effectiveness of PINT's forget term: replacing HSIC with RMU's forget term lowers Retain KnowMem from 0.673 to 0.615 at the final layer and to 0.565 at layer 7. Restricting the trained parameters to RMU's recovers part of this loss (0.632), and PINT remains the strongest. Since a single configuration can depend on its hyperparameters, we additionally compare the full trade-off curves of all variants over all runs in new Figure 12. PINT has the largest area under the curve, confirming its consistent advantage. We added the ablation to Appendix E.3.


Thank you again for your time and for engaging closely with our theory, which helped us clarify how PINT unlearns and where its advantage comes from. In light of our clarifications, please consider increasing your score.

Dear Reviewer,

Thank you for your thoughtful feedback. Your comments have further improved the quality of our work and improved our analysis.

In our response, we added new analyses of HSIC and CθC_\theta and an ablation of PINT's components. We are writing to inquire whether you have any further questions we could clarify during the discussion period, which ends in 2 days. We hope we have addressed your concerns. If so, we kindly ask you to consider updating your score.

Reviewer_acuXOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an LLM unlearning method that minimizes the statistical dependence between prefix representations and their corresponding next tokens using HSIC, while preserving representations on the retain set. The authors derive an HSIC-based upper bound on exact memorization and evaluate the method across several unlearning benchmarks.

Critical strengths
  1. The authors formulate forgetting through statistical independence (rather than directly suppressing token probabilities / collapsing representations, like some other unlearning work does), which is an interesting perspective on unlearning.

  2. The theoretical connection between HSIC and Exact Memorization is interesting and provides some motivation for the proposed objective.

  3. The experimental evaluation is rather comprehensive. I particularly appreciate the extensive hyperparameter comparisons and the encouraging results on Gaussian poisons and the bioS benchmark.

Critical weaknesses
  1. My main concern is whether statistical independence is the right objective for unlearning.

The proposed method aims to reduce statistical dependence between prefix representations and their corresponding next tokens on the forget set, but is it necessarily desirable for unlearning? A model may correctly predict tokens from their preceding context because of generalizable language patterns rather than memorizing the training data. And this generalizable knowledge should be preserved. Driving this statistical dependence towards zero could potentially harm the model's ability to generate meaningful text on forget-set prompts.

This concern also seems reflected in Appendix E.1, where PINT sometimes produces disfluent or empty responses at stronger forgetting strengths. I would appreciate further discussion of why statistical independence is the appropriate target for unlearning.

  1. HSIC can encourage representation collapse. When most next-token labels in a minibatch are distinct, the token Gram matrix under the delta kernel approaches the identity. In the extreme case where all labels are distinct, making all prefix representations identical would minimize HSIC. This raises my question: does PINT (partly) achieve forgetting through representation collapse? Could the authors examine how the diversity of forget-set representations changes during unlearning, and whether the results depend on the distribution of next-token labels?

  2. Another concern is on the evaluation side. The authors use a Gaussian-poison detectability test to evaluate if PINT makes poisoned training examples undetectable. However, the reported detectability metric aggregates signed statistics, which potentially allows positive and negative deviations to cancel. Could the authors verify whether the full distribution matches the null distribution, for example, through a two-sided statistical test?

Questions for authors

Please see "Critical Weaknesses" for my questions and concerns. I would be happy to reconsider my rating if these concerns can be addressed.

5 comments
AuthorsOpenReview ↗

Response to Reviewer acuX

We thank the reviewer for their thoughtful feedback, particularly their recognition of the independence perspective on unlearning, its theoretical connection to Exact Memorization, and our comprehensive experimental evaluation. We revised the manuscript based on your feedback, marking the changes in BROWN and those requested by several reviewers in GRAY. We address all the raised concerns below.


W1: Statistical independence as unlearning objective

Thank you for this great question. We agree that a model may correctly predict tokens from their preceding context because of generalizable language patterns, and we agree that this generalizable knowledge should be preserved. We view statistical independence as a well-motivated target when it is regularized by a retain objective (Equation 2). Because both objectives share the model parameters, the retain objective acts as a regularizer on the update of the forget objective. Capabilities that the model uses on both sets, such as grammar, copying and reasoning, therefore remain constrained, and the update favors changes that reduce the dependence on the forget set at the smallest cost to the retain representations. The retain objective thus counterbalances the forget objective, so that the dependence is not driven to zero. The balance is evident throughout our results: PINT achieves the strongest forgetting-utility trade-off across a wide range of benchmarks and metrics (Figure 6). This supports independence as an objective that removes the targeted information while preserving generalizable knowledge. The forgetting strength λ\lambda controls PINT's answers to forget questions (Appendix E.1), and at the retrained model's level of forgetting, they are plausible alternatives (as in Figure 9). To make this reasoning explicit in the paper, we expanded the discussion in Section 3.2 of how the forget and retain objectives work together to remove memorization while preserving generalizable knowledge.


W2: Representation collapse

Thank you for this question. We formalize this behavior in Proposition 3.3: collapse is one way of achieving independence, however, it is the most extreme one. To confirm that PINT avoids collapse in practice, we ran a new experiment on how unlearning changes the diversity of the forget set representations, as you suggested. We measure the diversity of the forget set representations zz by their normalized variance E∥z−zˉ∥2/E∥z∥2\mathbb{E}\|z-\bar{z}\|^2 / \mathbb{E}\|z\|^2 and their dependence on the next tokens by HSIC. Both are relative to the original model, so 1 means unchanged by unlearning. A normalized variance of 0 means collapsed representations, and an HSIC of 0 means independence from the next tokens. For each benchmark, we also report how many of the next tokens in a minibatch appear more than once, on average. We report all of them at the retrained model's level of Exact Memorization, on the benchmarks that provide a retrained model:

TOFU 1%TOFU 5%TOFU 10%MUSE NewsMUSE Books
Next tokens appearing more than once81%70%66%90%92%
Normalized variance, retrained model1.000.990.990.950.97
Normalized variance, PINT0.990.991.000.950.93
HSIC, PINT0.440.480.450.390.36

PINT keeps 93-100% of the diversity in all settings, while its HSIC drops to 36-48% of the original model's. Even on TOFU 10%, where next tokens repeat least, PINT keeps all of the diversity (1.00). This confirms that PINT minimizes the dependence by rearranging the forget representations, instead of collapsing, as illustrated in the new Figure 5. We added these measurements to Appendix E.5, together with a new Figure 14 on how the diversity changes during unlearning.


W3: Verification of the Gaussian-poison test

Thank you for the careful reading. We agree that signed statistics could cancel. As suggested, we applied a two-sided Kolmogorov-Smirnov test of the full distribution against the null distribution, for the reported PINT result in Figure 1:

MeanStdpp-value
Null distribution01
Original2.992.69<0.001<0.001
PINT0.051.140.84

The test clearly detects the original model, while PINT's statistics are consistent with the null distribution, with no sign of cancellation. Appendix E.4 now reports this test. In addition, we complement it with a new Figure 13 of the full distributions for a visual comparison with the null distribution.


Thank you again for your insightful review and questions, which helped us meaningfully strengthen the paper. In light of our clarifications, please consider updating your score.

Dear Reviewer,

Thank you for your valuable feedback. Your comments have further improved the quality of our work and strengthened our results.

In our response, we clarified the motivation for independence and added new experiments on representation collapse and the Gaussian-poison test. We are writing to inquire whether you have any further questions we could clarify during the discussion period, which ends in 2 days. We hope we have addressed your concerns. If so, we kindly ask you to consider updating your score.

Thank you for the detailed responses and additional experiments. I really appreciate the authors' efforts, and I believe my second and third concerns have been addressed.

However, I remain unconvinced by the response to my first concern. I still question whether reducing the dependence between prefix representations and next tokens on the forget set is a correct objective for unlearning, as such dependence is naturally expected for all examples in general. And as mentioned in my initial review, the results in Appendix E.1 seems to reinforce this concern, where PINT produces much less fluent responses than the retrained model on TOFU 5% and 10% at matched forgetting levels. Therefore, I remain concerned that the observed forgetting is achieved (at least partly) by degrading useful behavior on forget-set prompts.

As this fundamental concern remains unresolved, I will maintain my original rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary by Authors

Dear AC and Reviewers,

We sincerely thank you for your time, effort and constructive feedback. We are encouraged that you recognized our comprehensive evaluation [acuX, BPZL, tv4T] and the strong results on Gaussian poisons [acuX, tv4T]. We also appreciate the recognition of statistical independence as a well-motivated perspective on unlearning [acuX, BPZL] that can inform future work [BPZL], and of the theoretical support [acuX, BPZL] that the connection between HSIC and Exact Memorization gives this objective.

In response to your suggestions, we added the following experiments and analyses, marked in the revised manuscript in BROWN\color{#8B4513}{\textbf{BROWN}} [acuX], BLUE\color{#0033CC}{\textbf{BLUE}} [BPZL], GREEN\color{#1B7A1B}{\textbf{GREEN}} [tv4T] and GRAY\color{#666666}{\textbf{GRAY}} [several reviewers]:

  • Independence as an unlearning objective [acuX] (Sec. 3.2, App. E.7): Our new experiments show that retraining without the forget set lowers the dependence on the forget set. PINT's objective targets the same change that retraining makes.
  • Probing and recovery attacks [tv4T] (Fig. 7, App. E.6): We ran extensive new experiments with recovery attacks, each tuned in the attacker's favor. PINT has the lowest value under both probes, relearning and paraphrases. Relearning and adaptive extraction, which recover 24-25% on RMU, recover at most 1.3% on PINT.
  • How PINT changes the representations [acuX, BPZL] (Fig. 5, 14, App. E.5): Our new experiments confirm that PINT removes the dependence on the next tokens while preserving the diversity of the representations throughout training and across different next-token distributions.
  • Two-sided poison test [acuX] (Fig. 13, App. E.4): We ran a new two-sided Kolmogorov-Smirnov test. It confirms that PINT makes the poisons undetectable in their full distribution.
  • Roles of HSIC and CθC_\theta [BPZL, tv4T] (App. E.2, E.3): We ran new experiments that compare HSIC and CθC_\theta after unlearning on every benchmark with a retrained model. They confirm that the decrease in HSIC explains the decrease in Exact Memorization, while CθC_\theta remains stable across kernels.
  • Objective ablation [BPZL] (Fig. 12, App. E.3): We ran a new ablation that replaces PINT's components with RMU's one at a time. It shows that PINT's forget term is the main source of its advantage.
  • Illustrations of the theory [acuX, BPZL] (Fig. 3, 5): Two new figures illustrate Theorem 3.2 and how PINT and RMU remove the dependence.

To help the AC assess each concern independently, we provide an overview of the concerns raised, their resolution and location in the revised manuscript:

ReviewerConcern / QuestionResolutionLocation
acuXStatistical independence as unlearning objectiveRetraining lowers the dependence on the forget set and leaves it unchanged on other data, our retain objective regularizes the forget update, strongest forgetting-utility trade-offs, higher fluency when unlearning in stepsSec. 3.2, Fig. 6, 9, App. E.1, E.7
acuXRepresentation collapseCollapse as the most extreme way to independence (Prop. 3.3), diversity and HSIC across next-token distributionsSec. 3.4, Fig. 5, 14, App. E.5
acuXSigned poison statistics could cancelTwo-sided KS test, full distributionsFig. 13, App. E.4
BPZLRoles of HSIC and CθC_\theta in the decrease in EMBound across forgetting strengths, diversity and HSIC of the representations, EM, HSIC and CθC_\theta after unlearningFig. 3, 4, 5, 11, App. E.2, E.5
BPZLContributions of the forget term, constraints and trainable parametersObjective ablationFig. 12, App. E.3
tv4TDistinction from prior independence-based objectivesClarified role of HSIC in prior work, bound on EM (Theorem 3.2), relation to collapse (Prop. 3.3)Sec. 2, 3.3, 3.4, Fig. 5
tv4TConnection to EM specific to PINT?Theorem 3.2 holds for any model, PINT minimizes its HSIC directly, Prop. 3.3 relates RMUSec. 3.3, 3.4, Fig. 4
tv4TEvaluation beyond reproduction on the evaluated prefixesGaussian-poison test, probes, unseen phrasings, multiple-choice and open questions, recovery attacksFig. 1, 6, 7, 8, App. E.6
tv4TRecovery attacksTwo probes, relearning, adaptive extraction, paraphrased questionsFig. 7, App. E.6
tv4TConsistency across seeds, update scopes and tuning budgetsThree-seed reruns, shared protocol over ~2,400 settings, tuned parametersResponse to tv4T, Tab. 12, Fig. 12, App. C.2
tv4TBound tightness, stability of CθC_\theta across models and kernelsAbsolute bound across models, CθC_\theta across forgetting strengths and kernelsFig. 4, 11, App. E.2, E.3

We believe that these additions address the concerns raised, and we thank you again for your valuable feedback, which has directly strengthened this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.