TASE: Ultra-Sparse Vector-Level Unlearning via Token-Anchored Surgical Editing

Current approximate unlearning methods aim to suppress the influence of target information and preserve general capabilities. Achieving both objectives remains challenging since Gradient Ascent incurs substantial training cost and inconsistent retained utility, while parameter-efficient LoRA-based approaches still struggle to achieve strong forgetting without sacrificing retained utility. Building on our observation that sparse edits to only a small set of forget-associated token embeddings suffice to alter target outputs, we propose TASE, a token-anchored approximate unlearning method with extreme parameter efficiency. TASE exploits token distribution differences between the forget and retain sets to construct a small anchored token set that maximizes forget-sample coverage while minimizing retain-side interference. Embedding-only surgical editing then updates only the corresponding embedding rows, achieving targeted knowledge suppression and retained-utility preservation through ultra-sparse vector-level unlearning. The anchored-token sparse update strategy further supports token-based unlearning requests, reducing data exposure in remote model update settings. Experiments on BERT, Phi-1.5B, and Llama-2-7B across classification, TOFU, and MUSE demonstrate competitive forgetting and retained utility across encoder and decoder architectures. TASE modifies 2-5 orders of magnitude fewer parameters than existing approaches, while reducing peak GPU memory and GPU-hours by at least 30%.