acceptodds
Withdrawn from ICLR 2027

PAUSE: PII Unlearning Benchmark for Attribute- and Subject-Level Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models can memorize and reproduce personally identifiable information (PII), motivating machine unlearning to remove requested information from trained models. However, existing evaluations largely focus on removing entities or concepts, while fine-grained PII deletion, such as removing a specific phone number or address, remains underexplored. We introduce PAUSE, a scope-aware benchmark that jointly evaluates target forgetting and out-of-scope preservation. PAUSE considers two types of deletion requests: fact-level requests, which target a specific PII fact, and subject-level requests, which target all deletion-eligible PII facts associated with a subject, represented as . To characterize the scope of unlearning, PAUSE evaluates unlearning performance across four scopes: Target, Same Subject, Co-document, and Global. The benchmark spans seven document-level datasets across general, financial, medical, and legal domains and evaluates 11 unlearning methods across three LLMs. Our benchmark assess whether unlearning methods can fulfill real-world deletion requests while preserving information beyond the requested scope.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
personally identifiable information, machine unelarning, regulation, finance, medical

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ac37OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

PAUSE evaluates whether LLM unlearning can remove requested personal information while preserving information outside the deletion scope. It represents information as subject–attribute–value facts and supports fact-level and subject-level requests. Evaluation separates Target, Same Subject, Co-document, and Global facts. The experiments cover seven corpora, three model families, and eleven unlearning methods. The main finding is that Global preservation scores can obscure damage to related, unrequested information. Additional analyses examine shared values and generation failures.

Critical strengths
  1. The four evaluation scopes reveal local preservation failures that a single Global score can miss.

  2. Paired fact-level and subject-level requests allow deletion scopes to be compared using the same selected subjects.

  3. The document-to-fact mapping and experiments across seven corpora provide a foundation for studying selective PII unlearning.

Critical weaknesses
  1. The reported alignment checks do not establish correct ownership, and independent semantic annotation accuracy is not reported. Can the authors provide a human audit and verify the main findings on a checked subset?

  2. Relevant PII works (KnowUnDo) already test whether deleting private attributes harms other information about the same person. PAUSE should clarify what its evaluation adds beyond this test and UnlearnPII’s assessment of non-target PII retention.

Questions for authors
  1. Existing structured data can already support fact-level and subject-level deletion requests. It remains unclear what additional discovery PAUSE reveals through its document-based construction, beyond grouping facts into different evaluation scopes. Can a controlled comparison using the same facts and requests demonstrate this added diagnostic value?

  2. The evaluation omits PII-focused methods such as MemFlex, PPA and PERMU_tok without discussing their applicability. The DPO and RMU adaptations are also underspecified, and a common epoch budget does not establish comparable forgetting strength. Do the preservation gaps persist with a suitably adapted PII baseline at comparable levels of target forgetting?

Reviewer_9YRFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PAUSE, a scope-aware machine unlearning benchmark designed to evaluate Personally Identifiable Information (PII) deletion in Large Language Models (LLMs). Unlike prior unlearning benchmarks that focus on coarse entity-level or concept-level removal with largely disjoint forget/retain sets, PAUSE frames PII deletion around fine-grained (subject, attribute, value) facts.

Critical strengths
  1. Demonstrating that high "Global" retention scores can coexist with severe "Same Subject" and "Co-document" collateral damage is an impactful and necessary contribution to the field.

  2. Evaluating 11 methods across 3 model families and 7 corpora

Critical weaknesses
  1. Co-document evaluation is one of the paper's key conceptual pillars, but 4 out of the 7 corpora are marked as N/A due to single-subject structures or joint-filing deduplication. This narrows the empirical conclusions regarding document-level coupling primarily to Enron, OpenPII, and AMNESIA.

  2. Section 3.2 points out the blind spots of ROUGE-L recall, the primary quantitative tables (Tables 3, 4, 10–13) still exclusively report ROUGE-L differences under deterministic greedy decoding with fixed context prefixes. The benchmark lacks adversarial or robust probing (e.g., re-phrased queries, jailbreak prompts, or membership inference attacks) to evaluate whether "unlearned" PII has truly been erased or merely suppressed under the exact training prompt format.

  3. Major portions of the attribution and fact extraction pipelines rely on gemma-4. While string spans are verified programmatically, no quantified human agreement or error analysis is provided to assess how often semantic misattributions propagate into the "ground truth" evaluation sets.

Questions for authors

See Weaknesses

Reviewer_wNJuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PAUSE, a benchmark for evaluating LLM unlearning methods, particularly targeted towards removing knowledge of personally identifiable information (PII). Representing each PII item as a (subject, attribute, value) triplet, the benchmark mainly considers two setups: (1) fact-level unlearning, where only specified facts are targeted for unlearning, and (2) subject-level unlearning, where all PII items associated with the target subject is in scope for unlearning. The benchmark incorporates seven datasets across four domains (general, finance, medical, and legal). The effect of unlearning is measured across four domain scopes with increasing distance from the unlearning targets: Target, Same Subject, Co-document, and Global. Experiments study a cohort of 11 unlearning methods with 3 LLM model families, mainly tested with the ROUGE-L metric measured against the correct answer. Results show that Same-Subject and Co-document facts suffer from greater knowledge degradation than Global, and that most unlearning methods replace the target value with another value of the same format rather than removing it entirely.

Critical strengths
  • [S1] Useful evaluation scopes. The split between Co-document and Same Subject is a useful perspective on analyzing collateral damage due to unlearning. The overlap between the forget set and the retain set may be in terms of overlapping semantics/keywords such as names of people and places, or also in terms of statistical coupling during model optimization (being in the same batch/document). It is possible that two cases require different mitigation techniques, and the benchmark allows that study.
  • [S2] Broad domain coverage. PAUSE covers a wide variety of datasets spanning through 4 different domains, which brings great potential in studying how unlearning behavior varies across domains.
Critical weaknesses
  • [W1] Lack of reference points for evaluation. The main evaluations uses ROUGE-L against reference answers, but it is unclear what number should count as successful unlearning. Does achieving zero ROUGE-L mean successful unlearning? Table 5 seems to show this is not the case. What if the retain-only gold-standard model (which I believe can be tuned under the current setup) shows non-zero ROUGE-L? In that case, the "ideal" number would depend on that reference point. Without proper comparison against the base model prior to unlearning and the retain-only gold-standard model with proper metrics, the conclusions from the benchmark are hard to interpret.

  • [W2] Weak novelty. There already exist work that studies unlearning targeted towards atomic facts from knowledge bases or knowledge graphs, some examples being [A] and [B]. The finding that facts related to the unlearning targets (either in surface form or co-existence in the data point) is also studied in knowledge editing literature [C]. It would be better if the paper can posit PAUSE against those works and clarify what scope is exactly novel in this work.

  • [W3] Unclear practical motivation for fact-level unlearning. It is not clear whether the fact-level unlearning setup aligns well with how unlearning would be done under real-world deployment. Under regulations such as GDPR, most unlearning requests would come in at a document-level or subject-level: could the authors clarify what real-world scenarios would require the fact-level granularity studied in this work?

  • [W4] Limited insight from experimental results. The experimental results are overall what you would expect and have been extensively studied already with existing benchmarks. Below are some examples.

    • Table 3 and Figure 2 show the tradeoff between unlearning the forget set vs. collateral damage on the retain set, which is the main known challenge in unlearning literature.
    • Table 4 shows how co-document knowledge preservation is harder than global. Figure 3 shows that facts that share a value with Target PII are less well-preserved. These are somewhat expected since co-occurence with the forget target in a document and token sharing means the model has learned correlations between them.

    Overall, the results do not convey much information on top of what is already known. More interesting questions to ask would be:

    • If existing benchmarks are as coarse and disjoint as the authors claim, what specific unlearning behaviors do these benchmark miss, and does PAUSE reveal such behavior with finer unlearning granularity?
    • Considering that PAUSE incorporates a wide variety of domains, do different domains lead to different unlearning behaviors or challenges?

[A] Wu et al., Evaluating Deep Unlearning in Large Language Models. arXiv 2024.
[B] Qiu et al., How Data Inter-connectivity Shapes LLMs Unlearning: A Structural Unlearning Perspective. KDD 2024 GenAI Evaluation Workshop.
[C] Cohen et al., Evaluating the Ripple Effects of Knowledge Editing in Language Models. TACL 2024.

Questions for authors
  • [Q1] For Table 4, shouldn't the "difficulty" of knowledge preservation be measured relative to the base model's performance? Simply comparing the ROUGE-L reduction with Co-document samples vs. the same reduction with Global samples may not accurately reflect the difficulty when the base model's performance differs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.