acceptodds
Under review as a conference paper at ICLR 2027

Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix it

Abstract

Unlearning in large language models (LLMs) is typically evaluated at the output level, where a model appears to suppress sensitive or undesirable content. In this work, we show that such evaluations can create an illusion of forgetting: even when output-level leakage is eliminated, sensitive information can remain encoded in the model’s hidden representations. We first provide a theoretical analysis establishing a fundamental separation between output suppression and representational erasure. Specifically, we show that the decoder can be made arbitrarily insensitive to sensitive directions, driving output-level leakage to zero, while the hidden representations retain the underlying information. To empirically validate this phenomenon, we train generative probe decoders on hidden states across transformer layers, enabling layer-wise measurement of information leakage. Across three widely used benchmarks, TOFU, MUSE, and WMDP, and state-of-the-art unlearning methods, we find that substantial sensitive information remains recoverable from hidden representations, even when standard output-level metrics indicate successful unlearning. To address this gap, we propose Probe-Adversarial Representation Suppression (PARS), an unlearning objective that adversarially minimizes the extractable information from hidden representations. PARS directly targets representational leakage and provides significantly stronger guarantees of erasure under adversarial probing and relearning attacks, outperforming all evaluated baselines. Our results highlight a fundamental limitation of existing unlearning paradigms and suggest that true forgetting in LLMs requires controlling not only model outputs, but also the information encoded in hidden representations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Unlearning, Machine Unlearning, Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NYAcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the difference between output-level forgetting and representation-level knowledge erasure in LLM unlearning. The authors argue that existing methods may merely suppress sensitive outputs while retaining recoverable information in hidden representations. To substantiate this claim, they provide an information-theoretic analysis and introduce layer-wise generative probing to examine residual knowledge across three benchmarks. They propose Probe-Adversarial Representation Suppression (PARS), which adversarially reduces the recoverability of sensitive information from intermediate representations. Experiments suggest that PARS improves robustness against information extraction and relearning attacks while maintaining model utility.

Critical strengths
  • This paper identifies a meaningful gap between output-level suppression and residual-level knowledge removal.
  • The proposed solution, PARS, is straightforward and natural.
Critical weaknesses
  • My primary concern is the the insufficient differentiation from existing studies. The central observation that successful output-level forgetting does not necessarily imply genuine knowledge erasure has already been established in the literature [R1-R4]. For instance, Patil et al. (2024) demonstrate that seemingly deleted information can remain recoverable through extraction attacks, including attacks leveraging intermediate hidden states. The auditing tool of residual stream probing has also been extensively utilized.

  • A fundamental unresolved question is whether PARS genuinely removes sensitive knowledge from internal representations or simply makes that knowledge more difficult to recover using the selected adversarial probes. PARS is motivated by the assumption that reducing the predictive capability of adversarial probes can effectively eliminate sensitive information from hidden representations. However, minimizing the recoverability of information under a particular probe architecture and training procedure does not necessarily imply that the underlying information has been removed.

[Patil et al, 2024] Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks. ICLR 2024.

[R1] An Illusion of Unlearning? Assessing Machine Unlearning Through Internal Representations. AISTATS 2026.

[R2] Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs. ICML 2026.

[R3] Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputs. ICLR 2026.

[R4] Answer When Needed, Forget When Not: Language Models Pretend to Forget via In-Context Knowledge Unlearning. ACL Findings 2025.

Questions for authors

Q1: How do the authors ensure that the generative probes recover knowledge retained by the unlearned model, rather than learning sensitive information from their own training data?

Q2: What evidence supports the claim that PARS removes sensitive information rather than merely reducing its recoverability under the selected probe family?

Q3: What are the technical novelty of PARS beyond existing representation-level unlearning audits and relearning-based evaluations?

3 comments

We thank the reviewer for their insightful comments, particularly regarding novelty and the relationship to prior work. We clarify our contributions and address the questions below.

My primary concern is the the insufficient differentiation from existing studies. ... The auditing tool of residual stream probing has also been extensively utilized.

Response: Our novelty is not the observation itself, but the development of a unified framework that connects layer-wise generative extraction to an explicit representation-level unlearning objective. We clarify the technical distinctions below.

Compared to Patil et al. (ICLR 2024), our work differs in both the information extraction and the defense objective. Patil et al. introduces white-box extraction attacks based on intermediate-layer vocabulary projections and propose defenses such as Head Projection Defense and Max-Entropy Defense. Their approaches modify intermediate-layer output distributions. In contrast, our method trains a separate, layer-specific generative readout using answer-token supervision, allowing sensitive information to be reconstructed through free-form autoregressive generation rather than only identified through fixed vocabulary projections. This enables evaluation of sequence-level leakage, including factual answers and verbatim continuations.

More importantly, PARS does not optimize against a fixed projection or a predefined sensitive direction. Instead, it introduces a probe-adversarial objective where the probes are optimized to recover sensitive content, while the model is optimized to make that recovery more difficult across multiple intermediate layers. This distinction is central: the extraction mechanism itself becomes the adversary in the unlearning optimization. Unlike fixed-head suppression, PARS explicitly targets information recoverable by learned readouts while jointly optimizing the NPO-based forgetting objective and retain-set utility.

Our work differs from [R1–R4] in the operational question addressed and the resulting methodology.

Gao et al. [R1] studies feature–classifier misalignment and introduce class-mean-feature classifiers to improve representation-level forgetting. Their approach focuses on classifier alignment and feature discriminability. PARS instead targets the recoverability of generated sensitive content from intermediate representations through an adversarial generation objective.

Xu et al. [R2] studies the reversibility of unlearning through representation similarity, drift, and relearning analysis. Their approach characterizes when apparent forgetting remains reversible. PARS builds on this motivation but takes an optimization-oriented approach: rather than only measuring representational changes, it explicitly trains the model to resist intermediate-state extraction.

Chen et al. [R3] investigates whether unlearning leaves detectable fingerprints in model outputs and representations, including under forget-irrelevant prompts. Their primary objective is to detect whether unlearning has occurred. Our objective is different: identifying whether specific forgotten answers remain reconstructable and directly reducing that recoverability.

Takashiro et al. [R4] introduces in-context knowledge unlearning, enabling models to selectively withhold knowledge based on the query context. Also, their analysis reveals that information can persist in intermediate layers before being suppressed at the output. In contrast, PARS is designed to reduce intermediate-layer recoverability of designated forget-set information, rather than enabling context-dependent selective disclosure.

We have revised the manuscript to appropriately cite and discuss Patil et al. and [R1–R4], explicitly acknowledge their contributions, and clarify PARS's methodological novelty.

PARS is evaluated beyond the probe family used to define its training objective. We assess robustness through complementary attacks, including relearning, repeated probabilistic sampling, membership inference, and GCG jailbreaks. For example, on MUSE-News, PARS reduces post-relearning VerbMem from 41.3 (SAM) to 26.8, corresponding to a 35.1% reduction, while maintaining comparable retain-set performance (41.7 versus 42.6). These results provide evidence that the benefits of representation-aware optimization are not limited only to the training probe metric, although they do not establish resistance to every possible extraction mechanism.

A fundamental unresolved question is whether PARS genuinely removes sensitive knowledge from internal representations... imply that the underlying information has been removed.

Response: We agree that reduced recoverability under a specific probe family does not guarantee complete information erasure. PARS aims to suppress forgotten-answer recoverability through adaptive, layer-specific generative probes, rather than certify universal knowledge deletion. Importantly, PARS also demonstrates improved robustness beyond the probe-based objective, including under relearning, Leak@64, and GCG attacks, providing complementary evidence that its benefits are not limited to the training probes.

We have revised the manuscript to remove unsupported claims such as “genuine erasure” and “stronger guarantees of erasure,” replacing them with “reduced probe-accessible leakage and improved robustness under the evaluated attacks.” This clarification preserves our central contribution while precisely defining the scope of our empirical findings.

Q1. How do the authors ensure that generative probes recover knowledge retained by the unlearned model, rather than learning sensitive information from their own training data?

Response: Our primary goal is to evaluate how effectively sensitive information can be recovered from unlearned models and whether PARS reduces this recoverability, rather than proving that every recovered fact was independently memorized by the backbone. We freeze the backbone and apply the same generative probing protocol across all methods, training on 80% of forget QA pairs and evaluating on the held-out 20%. While probes may learn from their supervision, this standardized setup enables a meaningful comparison of susceptibility to supervised extraction. Importantly, PARS also demonstrates improvements beyond probing, including stronger resistance to relearning and GCG attacks. On MUSE-News, PARS reduces post-relearning VerbMem from 41.3 (SAM) to 26.8, a 35.1% improvement. We agree that retrained-model and entity-disjoint controls would help distinguish pre-existing knowledge from probe-acquired information. However, this distinction does not invalidate our central finding that PARS reduces recoverability under the evaluated extraction procedures.

Q2. What evidence supports the claim that PARS removes sensitive information rather than merely reducing its recoverability under the selected probe family?

Response: We do not claim that success against a particular probe family guarantees complete information erasure. However, PARS demonstrates improved robustness beyond the adversarial probes used during training, including against 75-step relearning, repeated stochastic decoding (Leak@64), and GCG jailbreak attacks. For example, on MUSE-News, PARS reduces post-relearning VerbMem from 41.3 (SAM) to 26.8 (PARS), a 35.1% reduction, while maintaining comparable retain-set utility. These results provide evidence that suppressing intermediate-state recoverability improves resistance to different attack mechanisms, rather than merely optimizing against the training probes. We acknowledge that finite evaluations cannot establish universal knowledge erasure. We clarify that our contribution is improved empirical resistance to sensitive-information extraction.

Q3. What is the technical novelty of PARS beyond existing representation-level unlearning audits and relearning-based evaluations?

Response: The technical novelty of PARS is in integrating learned, generation-based extraction directly into multi-layer unlearning optimization, rather than simply auditing hidden representations or evaluating robustness after unlearning.

Specifically, PARS differs from prior work in three ways:

  1. Unlike Patil et al. (2024), whose Head Projection and Max-Entropy defenses suppress sensitive token probabilities or increase entropy under fixed intermediate-layer vocabulary projections, PARS optimizes against learned, layer-specific generative probes. These probes are trained using answer-token supervision to reconstruct complete forgotten responses, making sequence-level recoverability the optimization target.

  2. Unlike Gao et al. [R1], who address feature–classifier misalignment through class-mean-feature classifiers, PARS directly targets the recoverability of sensitive answers from intermediate hidden states. Unlike Xu et al. [R2] and other representation-level audits, which characterize residual information or reversibility, PARS uses the extraction objective to update the model itself.

  3. Unlike post-hoc relearning evaluations, PARS incorporates extraction resistance into training. Specifically, its minimax objective optimizes generative probes to recover forgotten content while updating the model to increase their prediction loss across multiple layers, jointly with NPO-based forgetting and retain-set preservation.

Thank you for your response. After carefully reading your rebuttal, I find that the empirical clarifications alone do not sufficiently address my concerns, particularly in terms of the core claim and technical novelty of PARS. Furthermore, the distinctions from prior work appear to lie in empirical implementation details, rather than in methodological novelty. Therefore, I maintain my original assessment.

Reviewer_Z4BMOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper argues that unlearning evaluation largely ignores information leakage from hidden states, and proposes both a way to potentially measure it better and a new unlearning algorithm that achieves strong performance in the experimental setting considered. For the former, the authors introduce a generative probe decoder, which is trained on intermediate hidden states to recover forgotten information through generation, while for the latter the propose Probe-Adversarial Representation Suppression (PARS), an unlearning method that integrates an adversarial probing objective term into the standard NPO objective, with the goal of making reducing residual information left in the hidden states. Their experiments on TOFU and MUSe suggest that substantial information remains recoverable from hidden states even when the model appears to have forgotten it at the output level, while PARS has reported improvements over several baselines in unlearning performance and robustness to attacks (relearning/jailbreak).

Critical strengths

I believe the thesis of the paper, that current unlearning methods do not evaluate the information leakage of hidden states is well supported and a worthy problem to tackle.

Critical weaknesses
  • Mismatched Baselines: While you go over some of the appropriate baselines in the introduction (beginning of page 2), you never evaluate them! Furthermore, you make the substantial claim that these methods have no guarantees that the underlying information is fully removed from the representations (L116-L118), but you never evaluate this, and additionally, the same can be said for your method as well. Some (at the minimum) necessary baselines can be found in [1,2,3].
  • Theory Concerns: Lemma 1 is trivial decomposition, abd the additional assumptions in Theorem 1 seem to be incompatible with the deterministic inference setup in Equation 1. Specifically, for fixed model parameters and a fixed direction vv, Equation 1 and Lemma 1 give R=f(C;θ~u),ζ=⟨v,R−r0⟩−μS,ξ=(I−vv⊤)(R−r0),R = f(C ; \tilde{\boldsymbol{\theta}}_u), \qquad \zeta = \langle v, R - r_0 \rangle - \mu_S, \qquad \xi = (I - vv^\top) (R - r_0), where r0=E[R]r_0 = \mathbb{E}[R] and μS=E[⟨v,R−r0⟩∣S]\mu_S = \mathbb{E}[\langle v, R - r_0 \rangle \mid S]. Thus, (ζ,ξ)(\zeta,\xi) is a deterministic function of (C,S)(C,S). Nevertheless, Theorem 1 additionally assumes independence between (ζ,ξ)(\zeta,\xi) and (C,S)(C,S), and that ζ∼N(0,τ2)\zeta\sim\mathcal{N}(0,\tau^2). A random variable that is both a measurable function and independent of (C,S)(C,S) must be constant almost surely, leading to degenerate decompositions and also a violation of the assumed distribution of ζ\zeta, since its variance is 0.
  • Methodological Concerns: I have several experimental concerns: (1) There is no ablation comparing generative probes against non-generative probes, nor sufficient validation of the probes' reliability beyond as a probe. (2) double and triple runtime times are prohibitive, while the reported baseline runtime of 1h hours for TOFU on Llama 3.2 1B seems excessive (even more so consider the hardware used). In my experiments, training for 10 epochs (instead of your 6), with more optimization steps, takes only a few minutes on a single A100 GPU. (3) The RMU results are difficult to take at face value without a more thorough validation of the probes' performance, since RMU explicitly aims to make the representations of the forget set look like random vector, you don't reproduce the linear probing experiment used for comparison, but rather refer to a figure from another paper (L379), comparing effectivelly different things, and you suspiciously don't include your method in the WMDP analysis you already have done, for which RMU was developed for, while for the other datasets, it has been shown that RMU is mistuned and can work (e.g. see [4] for TOFU).
  • General: The writing quality is poor and rushed. Several plots are unclear and difficult to interpret, and important implementation details are unspecified.

[1] Yan, Han, Zheyuan Liu, and Meng Jiang. "Dual-space smoothness for robust and balanced llm unlearning." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Shen, William, et al. "LLM unlearning via neural activation redirection." Advances in Neural Information Processing Systems 38 (2026): 44253-44290.

[3] Yuan, Hongbang, et al. "Towards robust knowledge unlearning: An adversarial framework for assessing and improving unlearning robustness in large language models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 24. 2025.

[4] Yüce, Gizem, Giorgos Nikolaou, and Nicolas Flammarion. "Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance." arXiv preprint arXiv:2606.06320 (2026).

Questions for authors

See weaknesses. I have other questions as well but they are secondary to addressing the weaknesses.

Note to the authors: I am very open to discussing with you but my apriori opinion is that the paper need substantial revisions, maybe even beyond what a rebuttal can accommodate.

3 comments

We appreciate the reviewer's careful assessment and valuable suggestions. We address the theoretical, methodological, and experimental concerns below.

Mismatched Baselines: While you go over some of the appropriate baselines in the introduction (beginning of page 2), ... the same can be said for your method as well. Some (at the minimum) necessary baselines can be found in [1,2,3].

Response: We thank the reviewer for highlighting these relevant baselines. We clarify that neither PARS nor the cited methods guarantees complete information erasure. Our claim concerns improved empirical robustness against sensitive-information recovery, and we revised L116–L118 accordingly.

Importantly, we additionally evaluated PRISM [1] on MUSE-News, a directly relevant representation-level unlearning baseline, and compared performance at nearly identical retained utility.

MethodForget KnowMem ↓Leak@64–RS ↓Retain KnowMem ↑
PRISM [1]46.289.641.9
PARS40.180.641.7

PARS reduces forget-set KnowMem by 6.1 points and Leak@64 by 9.0 points, while maintaining comparable retained utility (41.7 vs. 41.9). This provides direct evidence of improved forgetting and repeated-sampling robustness relative to PRISM.

The technical distinction lies in the optimization objective. PRISM [1] promotes representation- and parameter-space smoothness, LUNAR [2] redirects neural activations, and LAU [3] optimizes against adversarial latent perturbations. PARS instead uses adaptive, layer-specific generative probes to directly minimize forgotten-answer recoverability, jointly with NPO forgetting and retain-set preservation.

Our contribution is improved empirical robustness through generation-supervised representation-level unlearning, rather than a claim of universal information erasure.

Theory Concerns: Lemma 1 is trivial decomposition, abd the additional assumptions in Theorem 1 seem to be incompatible with the deterministic inference setup in Equation 1. ... A random variable that is both a measurable function of and independent of (C,S)(C,S) must be constant almost surely, leading to degenerate decompositions and also a violation of the assumed distribution of ζ\zeta, since its variance is 00.

Response: We thank the reviewer for identifying this mathematical issue. We agree that the original assumption (ζ,ξ)⊥(C,S)(\zeta,\xi)\perp(C,S) is incompatible with nondegenerate residuals under deterministic inference. We have replaced Theorem 1 with a corrected, more general quantitative theorem and revised its proof.

Importantly, the revised theorem does not require independence or Gaussian assumptions on the residuals, nor stochastic hidden-state computation. Let H=(ζ,ξ)H=(\zeta,\xi) denote the residual components. We establish

I(S;R)≥I(S;μS+ζ),I(S;R)\geq I(S;\mu_S+\zeta),

\mathbb E[\operatorname{Var}(\mu_S\mid H)].$$ **Under the weaker condition $H\perp S$, which is compatible with deterministic representations,** the output bound simplifies to $$I(S;Z)\leq\Delta(v)^2\operatorname{Var}(\mu_S).$$ With an additional Gaussian assumption on $\zeta$ and appropriate entropy-power regularity conditions, we also recover the original form of the representation-information lower bound. **To explicitly demonstrate consistency with deterministic inference, we provide a nondegenerate construction** in which $R=f(C)$ is deterministic, $H\perp S$, and $\Delta(v)=0$, yet $I(S;Z)=0$ while $I(S;R)>0$. The revised proof uses **data processing, KL convexity, and decoder sensitivity**, without requiring independent noise in the hidden-state computation. We have also corrected $I(S;Z\mid C)$ to $I(S;Z)$ and clarified that Lemma 1 is an elementary orthogonal decomposition. > (1) There is no ablation comparing generative probes against non-generative probes, nor sufficient validation of the probes' reliability beyond as a probe. **Response:** Non-generative linear probes are a valid audit, and we now discuss them in the revision. However, non-generative probes are built for classification: they map a hidden-state embedding to a label (e.g., forget vs. retain) and never produce the answer text, so they can miss whether the actual answer is still recoverable. Our generative probe is a stronger test because it decodes the answer text from the hidden states. On TOFU, it recovers 8–12% of the forget answers from NPO, SimNPO and RMU at layers 12–14, versus about 0% for PARS.

(2) double and triple runtime times are prohibitive, ...with more optimization steps, takes only a few minutes on a single A100 GPU.

Response: We thank the reviewer for pointing this out. There is an error in the submitted version: our TOFU runs used NVIDIA A40 GPUs, not A100s. We corrected the hardware and runtime table in the revised paper. A40s are much slower than A100s, which explains the gap with your timing. PARS's extra cost comes from the inner probe updates at each outer step, which we now report alongside the baselines on the same hardware.

(3) The RMU results are difficult to take at face value ... for which RMU was developed for, while for the other datasets, it has been shown that RMU is mistuned and can work (e.g. see [4] for TOFU).

Response: We tuned RMU and evaluated it with our generative probe on both TOFU and WMDP-Bio. On TOFU, the probe recovers 10–12% of the forget answers at layers 12–14, above RMU's own output (8.25%). On WMDP-Bio, RMU performs well under a non-generative probe, but our generative probe still recovers answer text from its intermediate layers. This shows that RMU hides the answers at the output and at its target layer but does not remove them from the model.

Thank you for your clarifications! Nevertheless, the baseline and RMU points where not addressed adequately, and the triviality part of the theory point remains, as well as the fact that I don't know how reasonable it is to assume H⊥SH \perp S. I will increase the soundness score to a 2, but retain the original assessment.

Reviewer_Ppm7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper Summary

This paper argues that output-level evaluation of LLM unlearning can create an "illusion of forgetting": sensitive information may remain recoverable from hidden representations even when the model no longer generates it. The paper makes three contributions.

  1. Information-theoretic analysis. Using a semantic variable SS and a decomposition of the hidden representation along a direction vv, the paper shows the following. If the decoder is ε\varepsilon-insensitive to vv, output-level leakage is bounded by I(S;Z∣C)≤ε2Var(μS)I(S; Z \mid C) \le \varepsilon^2 \mathrm{Var}(\mu_S), while the information encoded in the representation, I(S;R)I(S; R), remains bounded below by a quantity that depends only on the backbone. Output suppression and representational erasure can therefore diverge.

  2. Layer-wise generative probing. A probe decoder is trained at each layer on 80% of the forget QA pairs and evaluated by free-form generation on the remaining 20%. Across TOFU, MUSE-News, and WMDP-Bio, and for methods including NPO, SimNPO, RMU, SURE, and SAM, the authors report that substantial sensitive information remains recoverable from intermediate layers despite successful output-level unlearning.

  3. PARS. The paper proposes an unlearning objective that augments the NPO forget loss and retain loss with an adversarial term. This term maximizes the loss of optimal probes at multiple intermediate layers, making forget content less decodable from hidden states.

Validation. PARS is evaluated on TOFU and MUSE-News using output-level forget metrics, Min-K%++ MIA, Leak@64, relearning attacks, and GCG jailbreaks. The authors report stronger robustness than the baselines at comparable utility; for example, post-relearning VerbMem on MUSE-News is reduced by 61% relative to SAM.

Critical strengths
  1. Important and timely question. Whether output-level suppression implies genuine removal is central to the reliability of LLM unlearning evaluation. The paper frames this gap clearly and supports it with concrete examples, such as Table 1, where answers suppressed at the output remain decodable at layer 28.
  2. Useful evaluation tool. Layer-wise generative probing complements existing multiple-choice linear probes. It shows which content is decodable at each layer, and the observation that RMU appears robust under linear probing but not under generative probing is informative.
  3. Broad evaluation. The probing study covers three widely used benchmarks (TOFU, MUSE-News, WMDP-Bio) and a range of representative methods, including the relearning-robust SAM and the representation-level RMU. PARS is evaluated under several complementary attacks (membership inference, Leak@k, relearning, and GCG jailbreaks) rather than output-level metrics alone.
  4. Promising robustness gains on MUSE-News. PARS reduces post-relearning VerbMem substantially compared with all baselines, including SAM, which is specifically designed for relearning robustness.
  5. Transparency about cost. The computational and memory overhead of PARS is explicitly reported.
Critical weaknesses
  1. The experiments do not validate the specific mechanism described by the theory. Theorem 1 characterizes suppression through an ε\varepsilon-insensitive final linear decoder WW. The empirical phenomenon, however, is different: information is decodable at intermediate layers (e.g., layer 28 in Table 1) but absent from the output at layer 31, with several transformer blocks in between. The paper does not measure Δ(v)\Delta(v) for the unlearned models' decoders, nor does it identify which directions are suppressed. The probing results therefore show that information remains decodable at intermediate layers, but they do not establish that existing methods suppress outputs through decoder insensitivity, as the conjecture and theory suggest.

  2. The probe may act as a relearning attack, and a retrain control is missing. Each probe is an n×dn \times d linear map initialized from the unlearned model's decoder and trained with cross-entropy on 80% of the forget QA pairs. This is effectively relearning restricted to a single head. In TOFU forget10, each author has 20 QA pairs with overlapping facts, so the probe can learn answers directly from its training split, and the held-out 20% is not truly unseen. The key control is absent: the same probe trained on the retrained model, which never saw the forget data. Without it, residual information in the representations cannot be distinguished from information acquired by the probe itself. Moreover, because the probe is still linear (essentially a tuned lens with autoregressive decoding), the claim that it is "strictly stronger" than linear probing reflects its output space and training data rather than greater capacity.

  3. The figures provide weaker evidence than the text suggests. On TOFU (Fig. 2), probe scores peak around 10–12, which is comparable to rather than clearly above the methods' output-level scores. On WMDP-Bio (Fig. 6), the LLM-judge axis spans only 0–0.25, even though scores are defined on a 1–4 scale, and most probe scores are close to zero. The evaluation splits are small (80 QA pairs on TOFU), and no variance across probe training runs or seeds is reported.

  4. The evaluation of PARS is partly circular, and several claims do not match the tables.

    • Circular evaluation. PARS is trained against linear probes fitted on the forget set and evaluated with probes of the same family. This leaves open the possibility of gradient masking. In related settings, adversarially removing an attribute from representations has been shown to leave it recoverable by freshly trained post-hoc probes [2]. Evaluation should therefore use probes unseen during training, such as MLP probes, layers outside Lp\mathcal{L}_p, or different initializations. In addition, how the inner problem ϕ⋆(θ)\phi^\star(\theta) is solved (number of probe updates per step, re-initialization) is not described.
    • Claims that do not match the tables. The abstract claims "perfect forgetting under ES" on TOFU, but Table 3 reports an ES of 13.0. It also claims an 82.7% reduction in Leak@64, whereas PARS achieves 73.7 against 84.3 for the original model and 75.9 for NPO.
    • Limited gains on TOFU. Post-relearning ES on TOFU is identical to that of NPO, its base method (20.4).
    • Questionable MIA interpretation. The MIA score of −104.6 is an extreme outlier (the baselines range from −1.8 to −6.3). An abnormally low Min-K%++ score may itself distinguish the model from a retrained one, yet no retrain reference is provided.
    • Leakage under sampling remains high. On MUSE-News, Leak@64 remains high for all methods, including PARS (80.6).
  5. Limited novelty positioning and experimental scope.

    • Prior work on hidden-state extraction and defense. Patil et al. [1] already showed that "deleted" facts can be extracted from intermediate hidden states. They also proposed whitebox defense objectives that suppress the information wherever it appears in the model, which is close in spirit to PARS. This paper cites [1] only as a jailbreaking method, which misrepresents its relevance.
    • The central conjecture has been studied before. Hong et al. [6] show, via parametric knowledge traces, that existing unlearning methods barely alter the underlying concept vectors and mainly suppress them at inference time. Xu et al. [7] use representation-level analysis to show that unlearned information is suppressed rather than erased and remains readily recoverable. Neither work is cited, and the paper should clarify what it adds beyond them.
    • Missing connection to concept erasure. PARS is closely related to the concept-erasure and adversarial-removal literature (e.g., adversarial attribute removal [2], INLP [3], LEACE [4]), and the probe is closely related to the tuned lens [5]. None of these works is discussed. Recent representation-level approaches to robust unlearning, such as CIR [10] and the concurrent MCU [9], would also be natural points of comparison.
    • Limited scope and utility evaluation. PARS is evaluated only on TOFU (with a 1B model) and MUSE-News, not on WMDP. RMU is missing as a baseline on MUSE-News. Utility is measured only on retain-set QA, without general capability benchmarks such as MMLU or fluency measures, even though maximizing probe loss at intermediate layers could affect general representations. All results come from single runs. The training times reported in the text and in Table 6 are inconsistent with each other.

[1] Vaidehi Patil, Peter Hase, Mohit Bansal. Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks. ICLR 2024.

[2] Yanai Elazar, Yoav Goldberg. Adversarial Removal of Demographic Attributes from Text Data. EMNLP 2018.

[3] Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, Yoav Goldberg. Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection. ACL 2020.

[4] Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, Stella Biderman. LEACE: Perfect Linear Concept Erasure in Closed Form. NeurIPS 2023.

[5] Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, Jacob Steinhardt. Eliciting Latent Predictions from Transformers with the Tuned Lens. arXiv:2303.08112, 2023.

[6] Yihuai Hong, Lei Yu, Haiqin Yang, Shauli Ravfogel, Mor Geva. Intrinsic Test of Unlearning Using Parametric Knowledge Traces. EMNLP 2025.

[7] Xiaoyu Xu, Xiang Yue, Yang Liu, Qingqing Ye, Huadi Zheng, Peizhao Hu, Minxin Du, Haibo Hu. Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs. ICML 2026.

[8] Aghyad Deeb, Fabien Roger. Do Unlearning Methods Remove Information from Language Model Weights? arXiv:2410.08827, 2024.

[9] Zeguan Xiao, Xuanzhe Xu, Yong Wang, Jian Yang, Yanqing Hu, Yun Chen. Robust LLM Unlearning Against Relearning Attacks: The Minor Components in Representations Matter. arXiv:2605.11685, 2026.

[10] Filip Sondej, Yushi Yang. Collapse of Irrelevant Representations (CIR) Ensures Robust and Non-Disruptive LLM Unlearning. arXiv:2509.11816, 2025.

Questions for authors
  1. Linking experiments to the theory. Can you measure Δ(v)\Delta(v) for the unlearned models' decoders along directions that carry forget information (e.g., directions identified by the probes)? Is the suppression between layer 28 and the output at layer 31 attributable to the final linear decoder, or to the intervening transformer blocks? Such evidence would directly support the conjecture that existing methods rely on decoder insensitivity.

  2. Controls for the probing study.

    • Could you report probe results on the retrained model, which never saw the forget data, under the same protocol? This is necessary to separate residual information in the representations from information the probe learns from its own training split.
    • In TOFU, how much factual overlap exists between the probe's 80% training split and the 20% evaluation split? Does the conclusion hold when the split is made at the author level?
    • Could you compare against a probe of the same capacity trained on a randomly initialized or unrelated head, as a baseline for how much the probe can learn on its own?
  3. Reading the figures.

    • In Fig. 2 (TOFU), for each method, how do the peak probe scores compare numerically with the corresponding output-level scores?
    • In Fig. 6 (WMDP-Bio), how is the LLM-judge score normalized, given that the judge produces scores on a 1–4 scale while the axis spans 0–0.25?
    • Could you report variance over probe training runs and seeds?
  4. PARS details and evaluation.

    • How is the inner problem ϕ⋆(θ)\phi^\star(\theta) solved in practice? Please specify the number of probe updates per step, whether probes are re-initialized, and the learning rate.
    • Does PARS remain robust against probes not seen during training, such as MLP probes, layers outside Lp\mathcal{L}_p, or freshly initialized probes trained for longer?
    • Which result supports the claim of "perfect forgetting under ES", and how is the 82.7% Leak@64 reduction computed?
    • Given its extreme value (−104.6), how should the MIA score be interpreted relative to a retrained reference?
  5. Scope and positioning.

    • Could you evaluate PARS on WMDP, include RMU as a baseline on MUSE-News, and report general utility (e.g., MMLU, fluency on forget prompts)?
    • How does PARS differ from concept-erasure approaches such as INLP [3] and LEACE [4], and from the extraction attacks and defenses of Patil et al. [1]?

Minor comments

  • The training times reported in Section B.2 and Table 6 are inconsistent with each other (TOFU and MUSE-News appear to be swapped).
  • There are typesetting artifacts, e.g., "\ perfect forgetting" and "leak@\64".
  • Several cited works have since been published and should be cited by their venues, for example NPO (COLM 2024), SimNPO (NeurIPS 2025), SAM (ICML 2025), MUSE (ICLR 2025), WMDP (ICML 2024), and Patil et al. (ICLR 2024).
5 comments

We thank the reviewer for their detailed and constructive feedback. We address each concern below and clarify the corresponding revisions and additional analyses.

The experiments do not validate the specific mechanism described by the theory. ... that existing methods suppress outputs through decoder insensitivity, as the conjecture and theory suggest.

Response: Theorem 1 establishes that decoder insensitivity is a sufficient mechanism for output suppression without representational erasure. It does not imply that all unlearning methods necessarily suppress information through the final decoder.

To examine this distinction, we further analyze MUSE-News leakage relative to the original model:

LayerSimNPO − OriginalNPO − Original
25+1.00+1.93
26−8.67−6.67
27+1.33−2.83
28+4.84+3.17
31 (output)−22.30−26.40

These results provide empirical support for the output–representation separation characterized by Theorem 1. At layer 28, both NPO and SimNPO retain substantial recoverability of sensitive information, with leakage even exceeding that of the original model. Yet their output-level leakage decreases by 26.4 and 22.3 points, respectively. This contrast demonstrates that successful output suppression can coexist with sensitive information remaining accessible in late intermediate representations, rather than requiring its removal throughout the network.

Theorem 1 provides a quantitative explanation for how such a separation can arise. Specifically, it establishes that sensitivity of the final decoder to information-bearing directions can be reduced without eliminating the information encoded in hidden representations. The theorem characterizes a sufficient mechanism, not an assumption that all suppression must occur exclusively in the final linear layer.

Importantly, the theoretical contribution and empirical findings establish the same central conclusion: output-level forgetting is insufficient to guarantee representational forgetting. Theorem 1 formalizes this separation, while the experiments demonstrate its practical relevance across existing unlearning methods.

The probe may act as a relearning attack, and a retrain control is missing. ... the claim that it is "strictly stronger" than linear probing reflects its output space and training data rather than greater capacity.

Response: Our probing framework is intended as a supervised extraction attack, rather than a passive measurement of memorization. We freeze the unlearned backbone and train only a lightweight readout on 80% of the forget QA pairs, evaluating on the held-out 20%. The objective is to measure how readily sensitive information can be recovered from frozen representations under a fixed supervision budget, rather than prove that every recovered fact was independently memorized before probe training.

We acknowledge that overlapping facts across TOFU questions may allow the probe to learn from its supervision. However, applying the same probe architecture, training budget, and evaluation protocol across unlearning methods provides a consistent comparison of their susceptibility to supervised extraction. Importantly, PARS's effectiveness is also supported by independent evaluations, including relearning, Leak@64, and GCG attacks, rather than relying solely on the trained probes.

Finally, we agree that our probe remains linear and does not have greater parameter-level expressivity than conventional linear readouts. Its distinction is the evaluation task: it enables free-form autoregressive reconstruction of sensitive answers rather than only multiple-choice classification.

The figures provide weaker evidence than the text suggests. On TOFU (Fig. 2), probe scores peak around 10–12, which is comparable to rather than clearly above the methods' output-level scores. On WMDP-Bio (Fig. 6), the LLM-judge axis spans only 0–0.25, even though scores are defined on a 1–4 scale, and most probe scores are close to zero.

Response: On WMDP-Bio (Fig. 6), the judge's 1–4 rubric is mapped to a 0–1 leakage score (1→0, 2→0.25, 3→0.5, 4→1), the same scale and judge as in Leak@k [11]. Low absolute values are expected for strict free-form judging of long technical answers. In Fig. 7 of Leak@k [11], RMU's leakage is around 0.05 under the same judge. Our probe scores for unlearned models (0.06–0.10 at the best layer) are slightly higher and on a similar scale. Most layers score close to zero because the early layers do not yet encode the answer in a decodable form, and only the later layers carry it. Since our probe scores are close to RMU's output-level performance in Leak@k, this scale is reasonable.

The evaluation splits are small (80 QA pairs on TOFU), and no variance across probe training runs or seeds is reported.

Response: Across four independent seeds on TOFU, forget-set ES is 12.56 ± 1.97 for PARS vs. 19.75 ± 3.21 for NPO (mean ± std), so the gap is larger than the seed variance.

[11] Reisizadeh et al. Leak@k: Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding. 2025.

Circular evaluation. PARS is trained against linear probes fitted on the forget set and evaluated with probes of the same family. .... In addition, how the inner problem ϕ⋆(θ)\phi^\star(\theta) is solved (number of probe updates per step, re-initialization) is not described.

Response: PARS does not optimize against a fixed probe. Instead, probes adapt to evolving representations through alternating optimization: the inner updates train probes to recover forgotten answers, while the outer updates suppress this recoverability alongside NPO forgetting and retain objectives.

Our evaluation extends beyond the training probes, including 75-step relearning, Leak@64, and GCG attacks. On MUSE-News, following identical relearning attacks, PARS reduces forget-set VerbMem from 41.3 (SAM) to 26.8 (PARS), a 35.1% reduction, while maintaining comparable retain-set KnowMem (41.7 vs. 42.6). This supports robustness beyond the probe-based objective.

Regarding implementation, PARS approximates ϕ⋆(θ)\phi^\star(\theta) through alternating gradient steps rather than exact inner optimization. Probes are updated to minimize answer-prediction loss, followed by a model update with probe parameters fixed. We clarified the number of inner steps, initialization strategy, and learning rates in the manuscript.

Claims that do not match the tables. The abstract claims "perfect forgetting under ES" on TOFU, but Table 3 reports an ES of 13.0. It also claims an 82.7% reduction in Leak@64, whereas PARS achieves 73.7 against 84.3 for the original model and 75.9 for NPO.

Response: We thank the reviewer for identifying these inconsistencies. These were accidental reporting errors in the abstract. The values in Table 3 are correct. Specifically, PARS achieves an ES of 13.0 on TOFU, rather than perfect forgetting. Similarly, the reported 82.7% reduction in Leak@64 was incorrect; the actual scores are 73.7 for PARS, 84.3 for the original model, and 75.9 for NPO. We apologize for these mistakes and revised the manuscript to ensure consistency with the reported results. While these corrections require more precise quantitative claims, the results continue to support PARS's effectiveness under the evaluated metrics.

Limited gains on TOFU. Post-relearning ES on TOFU is identical to that of NPO, its base method (20.4).

Response: While PARS and NPO achieve identical post-relearning ES (20.4) on TOFU, PARS demonstrates stronger performance across several other forgetting and robustness metrics on the same benchmark with higher retain utility.

Questionable MIA interpretation. The MIA score of −104.6 is an extreme outlier (the baselines range from −1.8 to −6.3). An abnormally low Min-K%++ score may itself distinguish the model from a retrained one, yet no retrain reference is provided.

Response: The Min-K%++ result should be interpreted as a forget-set membership score, rather than membership-inference attack accuracy. The substantially lower score for PARS (−104.6) indicates stronger suppression of the forgotten examples under this likelihood-based measure.

Leakage under sampling remains high. On MUSE-News, Leak@64 remains high for all methods, including PARS (80.6).

Response: While Leak@64 remains high across all methods, PARS achieves the lowest leakage (80.6), outperforming NPO (90.9), SURE (88.7), and SAM (83.3). PARS reduces leakage by 10.3 points comparing to NPO while maintaining nearly identical retain-set utility (41.7 vs. 41.8). Since PARS builds on NPO, this improvement supports the effectiveness of its additional representation-level suppression objective. Thus, although repeated-sampling leakage remains challenging, PARS provides a measurable improvement in robustness with comparable utility.

Prior work on hidden-state extraction and defense. Patil et al. [1] already showed that "deleted" facts can be extracted from intermediate hidden states. They also proposed whitebox defense objectives that suppress the information wherever it appears in the model, which is close in spirit to PARS. This paper cites [1] only as a jailbreaking method, which misrepresents its relevance.

Response: Patil et al. [1] establishes the recoverability of deleted facts from intermediate hidden states and proposed white-box defenses. The work is directly relevant to ours, and we will correct its characterization and discuss both their attacks and defenses.

PARS differs from Patil et al. in three technical aspects:

  1. Adaptive rather than fixed readouts: Patil et al.'s Head Projection and Max-Entropy defenses operate on intermediate vocabulary distributions obtained using the model's fixed projection head. PARS instead trains layer-specific generative probes that adapt to the evolving representations during unlearning.

  2. Answer-level rather than token-level suppression: Patil et al. target individual answer-token probabilities or intermediate distribution entropy. PARS uses answer-token supervision and autoregressive reconstruction to optimize against the recoverability of complete forgotten responses.

  3. Joint unlearning optimization: Patil et al. primarily study targeted factual model editing. PARS integrates multi-layer adversarial extraction directly into an NPO-based forget–retain objective, explicitly balancing reduced information recoverability with retained utility.

Thus, our novelty is not the discovery of hidden-state leakage or the general idea of defending against it, but an adaptive, generation-supervised, multi-layer adversarial objective for LLM unlearning. We will clarify this distinction and appropriately acknowledge Patil et al.'s contributions.

The central conjecture has been studied before. ... Neither work is cited, and the paper should clarify what it adds beyond them.

Response: Hong et al. [6] and Xu et al. [7] provide important evidence that apparent forgetting does not necessarily imply information erasure. We will cite both works and clarify our contributions relative to them.

PARS differs in both its evaluation methodology and its unlearning objective:

  1. Hong et al. [6] identify persistent parametric concept vectors using vocabulary projections and investigate their direct ablation. In contrast, our generative probes are trained to reconstruct forgotten answers from intermediate hidden states, directly measuring answer-level recoverability rather than tracking predefined concept vectors.

  2. Xu et al. [7] analyze representational changes using PCA, CKA, and Fisher information, alongside relearning experiments. These methods characterize representational drift and reversibility, whereas our probes directly measure whether forgotten content can be recovered through autoregressive generation.

  3. Unlike both approaches, PARS incorporates this recoverability into the unlearning objective itself. It jointly optimizes against adaptive, layer-specific generative probes while preserving the NPO forget–retain objectives, aiming to reduce residual leakage rather than merely detect it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.