- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the difference between output-level forgetting and representation-level knowledge erasure in LLM unlearning. The authors argue that existing methods may merely suppress sensitive outputs while retaining recoverable information in hidden representations. To substantiate this claim, they provide an information-theoretic analysis and introduce layer-wise generative probing to examine residual knowledge across three benchmarks. They propose Probe-Adversarial Representation Suppression (PARS), which adversarially reduces the recoverability of sensitive information from intermediate representations. Experiments suggest that PARS improves robustness against information extraction and relearning attacks while maintaining model utility.
Critical strengths
- This paper identifies a meaningful gap between output-level suppression and residual-level knowledge removal.
- The proposed solution, PARS, is straightforward and natural.
Critical weaknesses
-
My primary concern is the the insufficient differentiation from existing studies. The central observation that successful output-level forgetting does not necessarily imply genuine knowledge erasure has already been established in the literature [R1-R4]. For instance, Patil et al. (2024) demonstrate that seemingly deleted information can remain recoverable through extraction attacks, including attacks leveraging intermediate hidden states. The auditing tool of residual stream probing has also been extensively utilized.
-
A fundamental unresolved question is whether PARS genuinely removes sensitive knowledge from internal representations or simply makes that knowledge more difficult to recover using the selected adversarial probes. PARS is motivated by the assumption that reducing the predictive capability of adversarial probes can effectively eliminate sensitive information from hidden representations. However, minimizing the recoverability of information under a particular probe architecture and training procedure does not necessarily imply that the underlying information has been removed.
[Patil et al, 2024] Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks. ICLR 2024.
[R1] An Illusion of Unlearning? Assessing Machine Unlearning Through Internal Representations. AISTATS 2026.
[R2] Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs. ICML 2026.
[R3] Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputs. ICLR 2026.
[R4] Answer When Needed, Forget When Not: Language Models Pretend to Forget via In-Context Knowledge Unlearning. ACL Findings 2025.
Questions for authors
Q1: How do the authors ensure that the generative probes recover knowledge retained by the unlearned model, rather than learning sensitive information from their own training data?
Q2: What evidence supports the claim that PARS removes sensitive information rather than merely reducing its recoverability under the selected probe family?
Q3: What are the technical novelty of PARS beyond existing representation-level unlearning audits and relearning-based evaluations?
3 comments
A fundamental unresolved question is whether PARS genuinely removes sensitive knowledge from internal representations... imply that the underlying information has been removed.
Response: We agree that reduced recoverability under a specific probe family does not guarantee complete information erasure. PARS aims to suppress forgotten-answer recoverability through adaptive, layer-specific generative probes, rather than certify universal knowledge deletion. Importantly, PARS also demonstrates improved robustness beyond the probe-based objective, including under relearning, Leak@64, and GCG attacks, providing complementary evidence that its benefits are not limited to the training probes.
We have revised the manuscript to remove unsupported claims such as “genuine erasure” and “stronger guarantees of erasure,” replacing them with “reduced probe-accessible leakage and improved robustness under the evaluated attacks.” This clarification preserves our central contribution while precisely defining the scope of our empirical findings.
Q1. How do the authors ensure that generative probes recover knowledge retained by the unlearned model, rather than learning sensitive information from their own training data?
Response: Our primary goal is to evaluate how effectively sensitive information can be recovered from unlearned models and whether PARS reduces this recoverability, rather than proving that every recovered fact was independently memorized by the backbone. We freeze the backbone and apply the same generative probing protocol across all methods, training on 80% of forget QA pairs and evaluating on the held-out 20%. While probes may learn from their supervision, this standardized setup enables a meaningful comparison of susceptibility to supervised extraction. Importantly, PARS also demonstrates improvements beyond probing, including stronger resistance to relearning and GCG attacks. On MUSE-News, PARS reduces post-relearning VerbMem from 41.3 (SAM) to 26.8, a 35.1% improvement. We agree that retrained-model and entity-disjoint controls would help distinguish pre-existing knowledge from probe-acquired information. However, this distinction does not invalidate our central finding that PARS reduces recoverability under the evaluated extraction procedures.
Q2. What evidence supports the claim that PARS removes sensitive information rather than merely reducing its recoverability under the selected probe family?
Response: We do not claim that success against a particular probe family guarantees complete information erasure. However, PARS demonstrates improved robustness beyond the adversarial probes used during training, including against 75-step relearning, repeated stochastic decoding (Leak@64), and GCG jailbreak attacks. For example, on MUSE-News, PARS reduces post-relearning VerbMem from 41.3 (SAM) to 26.8 (PARS), a 35.1% reduction, while maintaining comparable retain-set utility. These results provide evidence that suppressing intermediate-state recoverability improves resistance to different attack mechanisms, rather than merely optimizing against the training probes. We acknowledge that finite evaluations cannot establish universal knowledge erasure. We clarify that our contribution is improved empirical resistance to sensitive-information extraction.
Q3. What is the technical novelty of PARS beyond existing representation-level unlearning audits and relearning-based evaluations?
Response: The technical novelty of PARS is in integrating learned, generation-based extraction directly into multi-layer unlearning optimization, rather than simply auditing hidden representations or evaluating robustness after unlearning.
Specifically, PARS differs from prior work in three ways:
-
Unlike Patil et al. (2024), whose Head Projection and Max-Entropy defenses suppress sensitive token probabilities or increase entropy under fixed intermediate-layer vocabulary projections, PARS optimizes against learned, layer-specific generative probes. These probes are trained using answer-token supervision to reconstruct complete forgotten responses, making sequence-level recoverability the optimization target.
-
Unlike Gao et al. [R1], who address feature–classifier misalignment through class-mean-feature classifiers, PARS directly targets the recoverability of sensitive answers from intermediate hidden states. Unlike Xu et al. [R2] and other representation-level audits, which characterize residual information or reversibility, PARS uses the extraction objective to update the model itself.
-
Unlike post-hoc relearning evaluations, PARS incorporates extraction resistance into training. Specifically, its minimax objective optimizes generative probes to recover forgotten content while updating the model to increase their prediction loss across multiple layers, jointly with NPO-based forgetting and retain-set preservation.
Thank you for your response. After carefully reading your rebuttal, I find that the empirical clarifications alone do not sufficiently address my concerns, particularly in terms of the core claim and technical novelty of PARS. Furthermore, the distinctions from prior work appear to lie in empirical implementation details, rather than in methodological novelty. Therefore, I maintain my original assessment.
We thank the reviewer for their insightful comments, particularly regarding novelty and the relationship to prior work. We clarify our contributions and address the questions below.
Response: Our novelty is not the observation itself, but the development of a unified framework that connects layer-wise generative extraction to an explicit representation-level unlearning objective. We clarify the technical distinctions below.
Compared to Patil et al. (ICLR 2024), our work differs in both the information extraction and the defense objective. Patil et al. introduces white-box extraction attacks based on intermediate-layer vocabulary projections and propose defenses such as Head Projection Defense and Max-Entropy Defense. Their approaches modify intermediate-layer output distributions. In contrast, our method trains a separate, layer-specific generative readout using answer-token supervision, allowing sensitive information to be reconstructed through free-form autoregressive generation rather than only identified through fixed vocabulary projections. This enables evaluation of sequence-level leakage, including factual answers and verbatim continuations.
More importantly, PARS does not optimize against a fixed projection or a predefined sensitive direction. Instead, it introduces a probe-adversarial objective where the probes are optimized to recover sensitive content, while the model is optimized to make that recovery more difficult across multiple intermediate layers. This distinction is central: the extraction mechanism itself becomes the adversary in the unlearning optimization. Unlike fixed-head suppression, PARS explicitly targets information recoverable by learned readouts while jointly optimizing the NPO-based forgetting objective and retain-set utility.
Our work differs from [R1–R4] in the operational question addressed and the resulting methodology.
Gao et al. [R1] studies feature–classifier misalignment and introduce class-mean-feature classifiers to improve representation-level forgetting. Their approach focuses on classifier alignment and feature discriminability. PARS instead targets the recoverability of generated sensitive content from intermediate representations through an adversarial generation objective.
Xu et al. [R2] studies the reversibility of unlearning through representation similarity, drift, and relearning analysis. Their approach characterizes when apparent forgetting remains reversible. PARS builds on this motivation but takes an optimization-oriented approach: rather than only measuring representational changes, it explicitly trains the model to resist intermediate-state extraction.
Chen et al. [R3] investigates whether unlearning leaves detectable fingerprints in model outputs and representations, including under forget-irrelevant prompts. Their primary objective is to detect whether unlearning has occurred. Our objective is different: identifying whether specific forgotten answers remain reconstructable and directly reducing that recoverability.
Takashiro et al. [R4] introduces in-context knowledge unlearning, enabling models to selectively withhold knowledge based on the query context. Also, their analysis reveals that information can persist in intermediate layers before being suppressed at the output. In contrast, PARS is designed to reduce intermediate-layer recoverability of designated forget-set information, rather than enabling context-dependent selective disclosure.
We have revised the manuscript to appropriately cite and discuss Patil et al. and [R1–R4], explicitly acknowledge their contributions, and clarify PARS's methodological novelty.
PARS is evaluated beyond the probe family used to define its training objective. We assess robustness through complementary attacks, including relearning, repeated probabilistic sampling, membership inference, and GCG jailbreaks. For example, on MUSE-News, PARS reduces post-relearning VerbMem from 41.3 (SAM) to 26.8, corresponding to a 35.1% reduction, while maintaining comparable retain-set performance (41.7 versus 42.6). These results provide evidence that the benefits of representation-aware optimization are not limited only to the training probe metric, although they do not establish resistance to every possible extraction mechanism.