Filtering Memorization from Parameter-Space in Diffusion Models

Low-Rank Adaptation (LoRA) enables efficient customization of diffusion models, but may also memorize and reproduce sensitive or copyrighted training content. This risk is particularly relevant in LoRA-sharing ecosystems, where adapters are distributed without their training data or training pipelines. Existing mitigation methods typically require access to training data, intervention during training, or control over inference, making them unsuitable when only a public pretrained backbone and a third-party LoRA are available. We propose Base-Anchored Filtering (BAF), a training- and data-free framework for post-hoc memorization mitigation. BAF spectrally decomposes the LoRA update and measures each component's alignment with the pretrained backbone's principal subspace, preserving strongly anchored components while suppressing weakly anchored components associated with instance-specific memorization. Experiments across three datasets, three diffusion backbones, and 20 community-released LoRAs show that BAF consistently reduces memorization while preserving generation quality, demonstrating that pretrained parameter geometry provides a useful structural prior for filtering memorization from released LoRAs. Our code is in the supplementary material.