PAUSE: PII Unlearning Benchmark for Attribute- and Subject-Level Evaluation

Large language models can memorize and reproduce personally identifiable information (PII), motivating machine unlearning to remove requested information from trained models. However, existing evaluations largely focus on removing entities or concepts, while fine-grained PII deletion, such as removing a specific phone number or address, remains underexplored. We introduce PAUSE, a scope-aware benchmark that jointly evaluates target forgetting and out-of-scope preservation. PAUSE considers two types of deletion requests: fact-level requests, which target a specific $(\text{subject},\text{attribute},\text{value})$ PII fact, and subject-level requests, which target all deletion-eligible PII facts associated with a subject, represented as $(\text{subject},\ast ,\ast )$. To characterize the scope of unlearning, PAUSE evaluates unlearning performance across four scopes: Target, Same Subject, Co-document, and Global. The benchmark spans seven document-level datasets across general, financial, medical, and legal domains and evaluates 11 unlearning methods across three LLMs. Our benchmark assess whether unlearning methods can fulfill real-world deletion requests while preserving information beyond the requested scope.