- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PINT to understand how an LLM can forget a memorized sequence per token. PINT is an LLM unlearning objective that minimizes HSIC between prefix representations and next tokens while preserving retain-set representations. It derives an HSIC-based upper bound on exact memorization and reports strong forgetting–utility trade-offs across several benchmarks. This paper conducts extensive experiments and especially presents strong performance in Gaussian poison, but the evidence does not establish that the target data’s influence is truly removed.
Critical strengths
-
PINT achieves competitive forgetting–utility trade-offs across multiple benchmarks, model families, and data types. The broad evaluation and Pareto-frontier analysis provide evidence that the method is effective beyond a single task.
-
I like its validation on Gaussian poison. It is the only evaluated method that makes the injected poison statistically undetectable while preserving near-original utility. This is consequential because the poison test measures a trace of training-data influence more directly than standard output-based forgetting metrics.
Critical weaknesses
-
I have main reservation on the novelty. The paper discusses prior uses of HSIC and statistical independence in machine unlearning, including HSIC-based model updates for image and graph classifiers (Han et al., 2024) and HSIC-based unlearning evaluation (Zhang et al., 2026). Representation-level objectives have also been widely studied for LLM unlearning. PINT may be the first to minimize HSIC specifically between prefix representations and next-token targets for LLM unlearning. However, this contribution primarily combines established ingredients in a new setting. The paper therefore offers a useful objective and strong empirical study, but its conceptual novelty appears incremental. I would like to see more insights rather than A+B.
-
Insufficient evidence of genuine unlearning. The theoretical result only bounds exact next-token memorization on the observed forget positions. It does not establish equivalence to retraining or removal of the target data’s influence. Moreover, the experiments also lack strong recovery attacks, such as nonlinear probes, relearning tests, and adaptive extraction, so the forgotten information may remain recoverable.
Questions for authors
-
Beyond applying HSIC to prefix representations and next-token targets, what is the substantive technical distinction between PINT and prior independence-based unlearning objectives?
-
Is the theoretical connection to exact memorization specific to PINT, or could a similar result be derived for existing representation-level objectives?
-
What evidence shows that reducing empirical next-token memorization removes the forget data’s influence, rather than merely suppressing its reproduction on the evaluated prefixes?
-
How tight is the HSIC-based bound in absolute terms, and how stable is across models and kernel configurations?
-
Do PINT’s advantages persist under nonlinear probes, relearning attacks, adaptive extraction, and paraphrased or indirect prompts? Please also clarify whether the Pareto comparisons remain consistent across multiple seeds and under matched parameter-update scopes and tuning budgets.
2 comments
Dear Reviewer,
Thank you for your detailed feedback. Your comments have further improved the quality of our work, deepened our analysis, and strengthened our evaluation of unlearning robustness.
We are writing to inquire whether you have any further questions we could clarify during the discussion period, which ends in 2 days. In our response, we clarified the distinction from prior work and substantially extended the evidence of unlearning with new experiments on relearning, adaptive extraction, nonlinear probes and paraphrased prompts, which we also added to the revised paper. We hope we have addressed your concerns. If so, we kindly ask you to consider updating your score.
Response to Reviewer tv4T
We thank you for recognizing our broad evaluation and PINT's strong results on the Gaussian poisons. Below, we clarify the distinction from prior work, and, in addition to the existing evidence in the paper, we ran all the new experiments suggested in your review. Revisions are in GREEN, and those requested by several reviewers in GRAY. All works we cite are referenced in the paper.
W1, Q1: Clarification regarding prior works
We respectfully disagree and clarify that prior works use HSIC in a fundamentally different role. Zhang et al. (2026) do not unlearn: they use HSIC as a test statistic to evaluate existing unlearning methods on image classifiers. Han et al. (2024) use HSIC as a regularizer that preserves accuracy after data removal in an influence-function update of image and graph classifiers, not as the forget objective. In contrast, PINT formulates forgetting as removing the dependence between prefix representations and next tokens, and is, to the best of our knowledge, the first to minimize HSIC for LLM unlearning.
Beyond our empirical study, this formulation yields two contributions that neither prior work provides: 1) an upper bound on Exact Memorization (Theorem 3.2), and 2) a formal relation to representation collapse (Prop. 3.3), which we show is an unnecessary extreme that, on bioS, also removes closely related knowledge (RMU 19%, PINT 97%, Fig. 8).
Q2: Is the connection to EM specific to PINT?
Theorem 3.2 holds more broadly for every model θ, so it bounds the memorization of any model. What is specific to PINT is that it directly minimizes the HSIC in the bound (Fig. 4). For other representation-level objectives, such as RMU, our Prop. 3.3 shows that collapsing representations onto a fixed target, as RMU does, is one extreme way to reduce this HSIC, which relates such objectives to our framework. However, collapse removes more than the memorized dependence and sacrifices utility that PINT preserves (Fig. 8).
W2, Q3 and Q5: Additional evidence of unlearning
We ran all the suggested experiments and would also like to clarify that our existing evaluations already go far beyond reproduction on the prefixes: 1) the statistical test for Gaussian poisons detects the data's influence on the gradients, 2) linear probes at every layer test whether the hidden states can predict the forgotten tokens after unlearning, 3) bioS asks the forgotten facts in unseen phrasings, 4) WMDP unlearns papers and code passages but tests the hazardous knowledge via multiple-choice questions, and 5) MUSE asks open questions about the unlearned books and news articles.
We ran the four attacks you suggest, each following prior work: 1) the relearning attack and 2) adaptive extraction by removing the unlearning direction, both by Łucki et al. (2024), 3) a nonlinear probe (Guo et al., 2024), and 4) TOFU's paraphrases by Maini et al. (2024). We tune each attack on the same grid for every method, and report its strongest result (App. E.6).
PINT has the lowest value under relearning, the nonlinear probe and paraphrases, while adaptive extraction recovers at most 1.3% from all methods but RMU (23.9%). We use three seeds wherever computationally feasible, including all evaluations in this rebuttal. To demonstrate seed stability, we also reran every MUSE Books setting with Forget KnowMem between 0.2 and 0.4 using 3 seeds in new Fig. 12: PINT remains the strongest in new Tab. 11. All methods follow a shared protocol (App. C.2) over ~2,400 settings: the same nine runs on the Gaussian poisons and the same search procedure elsewhere.
Q4: Bound tightness and Cθ stability
We would like to clarify that Fig. 4 already reports tightness in absolute terms across models. The existing kernel ablation (App. E.3) confirms PINT's stability across kernels. In App. E.2 (Fig. 11), Cθ never cancels the decrease in HSIC. We additionally ran new experiments measuring Cθ for the remaining kernels (App. E.3), on TOFU 10% at the retrained model's forget ROUGE:
We thank you for the feedback. Following the requests, we clarified the distinction from prior work and ran extensive new experiments, confirming PINT's strength. In light of the new evidence, please consider increasing your score.