Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs

Unlearning in large language models (LLMs) requires precisely removing specific information from a pre-trained model, e.g. to delete private data or harmful knowledge acquired during pre-training. However, existing unlearning methods often fall short under thorough evaluation. This paper makes three contributions. First, we introduce JensUn, a novel unlearning approach which leverages the Jensen-Shannon Divergence (JSD) as the training objective for both forget and retain sets. We provide theoretical arguments why JSD outperforms KL-Divergence, and show in extensive experiments that JensUn achieves the best forget-utility trade-off and demonstrates strong resilience to benign relearning, outperforming existing methods. Second, for a precise unlearning evaluation, we introduce LPF, a curated dataset of less prominent facts that provides a realistic unlearning scenario. Third, to comprehensively test unlearning methods, we (i) employ an LLM as semantic judge instead of the standard ROUGE score, and (ii) evaluate worst-case unlearning over various paraphrases and input formats. Our improved evaluation reveals that many unlearning methods are less effective than previously thought.