- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper noticed a limitation for text-to-image generation models that existing methods require fixed adapters for each concept variant and struggle to handle multiple concepts. To address this issue, authors propose HyperErase, a framework for concept erasure based on hypernetwork-driven promptconditioned parameter synthesis. HyperErase uses LoRA updates as teacher models, and train a hypernetwork to map textual descriptions to prompt-specific LoRA updates. HyperErase also follows a decoupled rectification strategy that disentangles LoRA tokens into subspaces and leverages teacher-derived canonical priors for inference-time correction. Experiments validate that HyperErase improves erasure effectiveness and images quality against baselines, and can provide LoRA parameters within a single forward pass during inference.
Critical strengths
- HyperErase is an interesting method for multi-concept erasure. HyperErase can handle multiple concepts using only one adapter, which is more flexible than using multiple LoRAs.
- Experiments include different types of concepts, and also include adversarial prompts on FLUX, showing the framework can be adapted to different scenarios and different models.
Critical weaknesses
- There are some existing works on using hypernetworks to generate LoRA weights for concept unlearning, but they are not compared. For example, UnHype in ICML 2026 (https://arxiv.org/abs/2602.03410) already proposed the hypernetworks framework for multi-concept erasure, and the overall idea is closely related. Without comparing against existing works on topic, it is hard to tell the advantages of the method and hard to show novelty.
- The computational complexity is not clear. HyperErase avoids optimization during inference, but still requires collecting ESD checkpoints for each concept and a hypernetwork. The paper should discuss the training cost, inference latency and memory complexity compared with methods that use LoRAs.
- The performance gains over the teacher models (LoRAs) are not sufficiently explained. HyperErase hypernetwork is trained by teacher LoRAs parameters from ESD, and sometimes it achieves better performance than teacher models. The authors explain these improvements are attributed to knowledge transfer across concepts, but no evidence is provided to support the statement.
- The paper does not compare with LoRA merging. Multiple LoRA weights can be merged to handle multiple concepts in one prompt, which is basically similar idea with using hypernetwork. Since LoRAs are already obtained and can be merged without additional parameter generation overhead during inference, experiments for model merging should be conducted to compare the performance and efficiency.
Questions for authors
Please see weaknesses section above.
3 comments
Response to W3:
Thank you for your comments. Here we provide a more detailed explanation of this phenomenon. We argue that, as a jointly trained multi-concept conditional parameter generator, HyperErase learns a shareable concept-to-erasure parameter mapping from multiple concepts and their teacher optimization trajectories; the shared representations and parameter generation prior formed through joint training enable the model to extract more stable erasure directions than a single teacher checkpoint, yielding better erasure–preservation trade-offs.
To validate this explanation, we compared HyperErase trained on a single concept versus trained on 14 concepts, and used each to generate the effective LoRA for the same target concept. Both PGs used exactly the same reference, namely the average effective update of 50 teacher checkpoints corresponding to the target concept—the teacher trajectory center:
=\frac{1}{50}\sum_{s=1}^{50}B_sA_s.$$ We report the following two directional consistency metrics. The scale-aligned distance allows an optimal global scale coefficient before comparison, thereby avoiding interference from LoRA magnitude differences in direction comparison: $$d = \min _{\alpha} \frac{ \\| \alpha \Delta W _{\mathrm{PG}} - \overline{\Delta W} _{\mathrm{church}} \\| _F} { \\| \overline{\Delta W} _{\mathrm{church}} \\| _F}, \quad \alpha ^{*} = \frac{ \langle \Delta W _{\mathrm{PG}}, \overline{\Delta W} _{\mathrm{church}} \rangle } { \\| \Delta W _{\mathrm{PG}} \\| _F ^{2} }.$$ Cosine similarity measures the angular consistency: $$\mathrm{CosSim} = \frac{ \langle \Delta W _{\mathrm{PG}}, \overline{\Delta W} _{\mathrm{church}} \rangle } { \\| \Delta W _{\mathrm{PG}} \\| _F \\, \\| \overline{\Delta W} _{\mathrm{church}} \\| _F }.$$ The experimental results are shown in the table below: | Type | Scale-aligned distance ↓ | Cosine with trajectory center ↑ | |---|---|---:| | HyperErase with single concept | 0.1836 | 0.9830 | | HyperErase with 14 concept | 0.1175 | 0.9931 | As can be seen, compared with erasing a single concept, HyperErase trained on 14 concepts reduces the scale-aligned distance to the same church teacher trajectory center by 36.0%, while the cosine similarity increases from 0.9830 to 0.9931. This indicates that multi-concept joint training generates target concept erasure directions closer to the stable center of the teacher trajectory, providing direct parameter-space evidence for our judgment: the full HyperErase can indeed transfer shared concept-to-parameter generation knowledge through joint multi-concept learning, thereby obtaining more accurate and stable functional erasure directions than learning the target concept alone. # Response to W4: We appreciate your suggestion. We conducted multi-concept erasure using model merging and report the comparison results between our method and model merging on Nudity below. As can be seen, directly applying model merging causes the concept erasure to completely fail. This is because model merging essentially dilutes the erasure capability for individual concepts; as the number of concepts increases, the model's generation ability is entirely disrupted, manifesting as very poor CLIP and FID scores as shown in the table. The high erasure rate in the table, however, is a consequence of the model's generation capability being destroyed and its inability to produce normal images. | Model | Total | FID | CLIP | |:---:|:---|:---:|:---:| | SD v1.4 | 621 | 17.84 | 31.08 | | ESD Single | 357 | 17.06 | 30.54 | | model merging | 1 | 246.59 |19.60 | | HyperErase with ESD | 359 | 16.96 | 30.57 | For a more comprehensive comparison, we apply SPEED and MACE separately for single-concept erasure, then use HyperErase to perform multi-concept erasure on both SPEED and MACE respectively, comparing the results against the native multi-concept erasure outcomes of these two methods. This allows us to verify the erasure effectiveness of our multi-concept approach compared to SPEED and MACE, as well as demonstrate the generality of our method. However, these experiments are not yet complete; we will append the results below once finished. # Response to Ethics Concerns: We understand your concerns. The field of concept erasure aims to prevent generative models from producing unsafe content, which is why our research necessarily involves nudity and NSFW content. However, our objective is to prevent the generation of such content rather than disseminate it, and we have taken care to implement protective measures in our writing. Moreover, previous papers in this area have not encountered such ethics concerns. Nevertheless, we agree that your suggestion is meaningful. Therefore, we are happy to include an ethics discussion section in the appendix, following the practice of other fields with ethics concerns, to address data sources, visual masking, safety considerations, and related issues.Extra comparison with MACE Multi, SPEED Multi and Model Merging
Below, we supplement the comparison results with multi-concept erasure methods MACE, SPEED, and Model Merging. First, it should be noted that since the modules trained or edited by MACE and SPEED differ from those of ESD and rely on using the modules specified in their respective papers, they cannot be directly transferred to the modules trained by ESD. Therefore, we trained HyperErase using the single-concept erasure trajectories of the corresponding modules of MACE and SPEED, and compared them with their respective multi-concept erasure methods. For Model Merging, we directly used the ESD-trained LoRA for merging and experimentation. The results are shown in the following three tables, respectively.
| Model | Nudity Total | Object (%) | Artist (%) | FID | CLIP |
|---|---|---|---|---|---|
| SPEED Multi | 363 | 50.0 | 10.0 | 19.13 | 30.94 |
| HyperErase | 256 | 42.5 | 3.3 | 19.03 | 31.07 |
| Model | Nudity Total | Object (%) | Artist (%) | FID | CLIP |
|---|---|---|---|---|---|
| MACE Multi | 35 | 0.0 | 0.02 | 19.82 | 27.73 |
| HyperErase | 51 | 0.8 | 0.02 | 19.79 | 31.06 |
| Model | Nudity Total | Object (%) | Artist (%) | FID | CLIP |
|---|---|---|---|---|---|
| model merging | 1 | 0.0 | 0.00 | 246.59 | 19.60 |
| HyperErase | 359 | 3.92 | 0.08 | 16.96 | 30.57 |
As can be seen, compared with the multi-concept erasure results of SPEED, HyperErase achieves better performance in both erasure effectiveness and generation capability metrics. Compared to MACE, although the erasure capability is slightly weaker than that of MACE, the CLIP metric of MACE drops significantly (the CLIP of the original model is 31.08). We argue that the superior multi-concept erasure results of MACE may be attributed to the degradation of generation capability. Directly applying model merging causes the concept erasure to completely fail. This is because model merging essentially dilutes the erasure capability for individual concepts; as the number of concepts increases, the model's generation ability is entirely disrupted, manifesting as very poor CLIP and FID scores as shown in the table. The high erasure rate in the table, however, is a consequence of the model's generation capability being destroyed and its inability to produce normal images. Overall, HyperErase's multi-concept erasure capability is on par with those of MACE and SPEED, and it can serve as a fairly general multi-concept erasure complement.
We thanks reviewer's thoughtful comments. We provide detailed responses to your concerns below.
Response to W1:
HyperErase differs significantly from the UnHype method. UnHype directly trains a CLIP-conditioned hypernetwork by aligning local gradients with a manually designed forgetting target gradient field, thereby avoiding the need to collect pre-computed teacher LoRA weights. While this approach effectively improves erasure performance and reduces cost, it also limits its erasure capability to its own training objective and the inductive biases and capacity of its network architecture, preventing it from effectively leveraging the capabilities of existing single-concept erasure methods. In contrast, the contribution of HyperErase lies in providing a general multi-concept erasure framework for existing single-concept erasure methods. By training on the checkpoints generated by these single-concept methods after completing concept erasure, HyperErase offers a universal multi-concept erasure solution for methods that excel at single-concept erasure but cannot scale to multiple concepts. Precisely because of this, the erasure performance of HyperErase is inherently correlated with the gold baseline, and our experiments are therefore compared primarily against the gold baseline.
To address your question, we present in the table below the nudity erasure results under the multi-concept erasure setting, using ESD and RECE as teacher models, respectively. In our main experiments, for a fair comparison, we uniformly adopt ESD as the teacher model. Due to the inherent limitations of ESD itself, the results of HyperErase may appear inferior to those of UnHype. However, in the ablation study, when we fit LoRA from the edited weights generated by RECE and use it as the teacher model, HyperErase's erasure performance surpasses that of UnHype. We do not claim that HyperErase outperforms UnHype in all aspects. Rather, we aim to demonstrate that, unlike UnHype, our method can serve as a general multi-concept erasure complement to existing concept erasure methods, thereby achieving composability and upgradability that UnHype does not possess.
Response to W2:
The GPU training time, number of training epochs, inference latency and peak VRAM consumption are reported as follows:
The time cost here does not include LoRA collection. The cost of collecting LoRAs depends on the single-concept method. For ESD, single-concept erasure typically takes 10 to 20 minutes.