Erased But Not Gone: How Model Merging Weakens Concept Erasure in Diffusion Models
Abstract
Concept erasure suppresses a target concept in a diffusion model, but the edited weights may later be merged with another checkpoint. We study whether suppression survives composition with a public style or domain model. Averaging weakens the erasure update as it incorporates the partner, whereas additive task arithmetic keeps that update intact. We probe these changes separately and compare merge operators across suppression and partner-appearance transfer. Our study covers three object concepts in Stable Diffusion 1.5, with a 95-setting checkpoint/partner grid and a balanced 33-setting strength sweep. Attenuation restores target-concept evidence, but partner addition can also restore it without weakening the erasure update. At common minimum appearance-transfer targets, addition improves normalized suppression retention by 26.8 percentage points over interpolation; the tested parameter guards add 1.0 point over addition. These paired comparisons are retrospective and depend on the measured appearance estimator. The results identify strength-tuned addition as an important baseline and show why preserving an update in parameter space does not guarantee preservation of its behavioral effect.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.