acceptodds
Withdrawn from ICLR 2027

HyperErase: Scale-Calibrated Hypernetwork for Multi-Concept Erasure in Text-to-Image Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advances in text-to-image (T2I) generation have substantially improved visual synthesis, but have also raised increasing safety concerns due to their potential to generate harmful or undesirable content. Existing concept erasure methods predominantly follow a static weight paradigm, producing a single frozen adapter that struggles to adapt to diverse prompt variations and suffers from parameter interference when scaling to multiple concepts. We propose HyperErase, a framework for concept erasure based on hypernetwork-driven prompt-conditioned parameter synthesis. Our approach first reframes concept erasure as prompt-conditioned parameter amortization and trains a hypernetwork to map textual descriptions to prompt-specific LoRA updates, eliminating the need for per-prompt gradient optimization or manual LoRA merging. To further improve the stability and precision of synthesized adapters, we develop a decoupled rectification strategy, which disentangles LoRA tokens into pattern and scale subspaces, applies a square-root transform to curb multiplicative over-scaling, and leverages teacher-derived canonical priors for inference-time correction. Extensive experiments across major concept categories demonstrate that HyperErase consistently improves the trade-off between erasure effectiveness, image quality, and semantic alignment, achieving performance comparable to gold-standard single-concept baselines. Furthermore, the resulting models can provide specialized LoRAs for each input prompt variation in a single forward pass without requiring gradient updates during inference. These principled and flexible framework offers a new paradigm for concept erasure in T2I models.

Primary area:
generative models
Keywords:
Text-to-Image, Concept Erasure

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BUdnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper noticed a limitation for text-to-image generation models that existing methods require fixed adapters for each concept variant and struggle to handle multiple concepts. To address this issue, authors propose HyperErase, a framework for concept erasure based on hypernetwork-driven promptconditioned parameter synthesis. HyperErase uses LoRA updates as teacher models, and train a hypernetwork to map textual descriptions to prompt-specific LoRA updates. HyperErase also follows a decoupled rectification strategy that disentangles LoRA tokens into subspaces and leverages teacher-derived canonical priors for inference-time correction. Experiments validate that HyperErase improves erasure effectiveness and images quality against baselines, and can provide LoRA parameters within a single forward pass during inference.

Critical strengths
  1. HyperErase is an interesting method for multi-concept erasure. HyperErase can handle multiple concepts using only one adapter, which is more flexible than using multiple LoRAs.
  2. Experiments include different types of concepts, and also include adversarial prompts on FLUX, showing the framework can be adapted to different scenarios and different models.
Critical weaknesses
  1. There are some existing works on using hypernetworks to generate LoRA weights for concept unlearning, but they are not compared. For example, UnHype in ICML 2026 (https://arxiv.org/abs/2602.03410) already proposed the hypernetworks framework for multi-concept erasure, and the overall idea is closely related. Without comparing against existing works on topic, it is hard to tell the advantages of the method and hard to show novelty.
  2. The computational complexity is not clear. HyperErase avoids optimization during inference, but still requires collecting ESD checkpoints for each concept and a hypernetwork. The paper should discuss the training cost, inference latency and memory complexity compared with methods that use LoRAs.
  3. The performance gains over the teacher models (LoRAs) are not sufficiently explained. HyperErase hypernetwork is trained by teacher LoRAs parameters from ESD, and sometimes it achieves better performance than teacher models. The authors explain these improvements are attributed to knowledge transfer across concepts, but no evidence is provided to support the statement.
  4. The paper does not compare with LoRA merging. Multiple LoRA weights can be merged to handle multiple concepts in one prompt, which is basically similar idea with using hypernetwork. Since LoRAs are already obtained and can be merged without additional parameter generation overhead during inference, experiments for model merging should be conducted to compare the performance and efficiency.
Questions for authors

Please see weaknesses section above.

3 comments

We thanks reviewer's thoughtful comments. We provide detailed responses to your concerns below.

Response to W1:

HyperErase differs significantly from the UnHype method. UnHype directly trains a CLIP-conditioned hypernetwork by aligning local gradients with a manually designed forgetting target gradient field, thereby avoiding the need to collect pre-computed teacher LoRA weights. While this approach effectively improves erasure performance and reduces cost, it also limits its erasure capability to its own training objective and the inductive biases and capacity of its network architecture, preventing it from effectively leveraging the capabilities of existing single-concept erasure methods. In contrast, the contribution of HyperErase lies in providing a general multi-concept erasure framework for existing single-concept erasure methods. By training on the checkpoints generated by these single-concept methods after completing concept erasure, HyperErase offers a universal multi-concept erasure solution for methods that excel at single-concept erasure but cannot scale to multiple concepts. Precisely because of this, the erasure performance of HyperErase is inherently correlated with the gold baseline, and our experiments are therefore compared primarily against the gold baseline.

To address your question, we present in the table below the nudity erasure results under the multi-concept erasure setting, using ESD and RECE as teacher models, respectively. In our main experiments, for a fair comparison, we uniformly adopt ESD as the teacher model. Due to the inherent limitations of ESD itself, the results of HyperErase may appear inferior to those of UnHype. However, in the ablation study, when we fit LoRA from the edited weights generated by RECE and use it as the teacher model, HyperErase's erasure performance surpasses that of UnHype. We do not claim that HyperErase outperforms UnHype in all aspects. Rather, we aim to demonstrate that, unlike UnHype, our method can serve as a general multi-concept erasure complement to existing concept erasure methods, thereby achieving composability and upgradability that UnHype does not possess.

ModelTotalFIDCLIP
SD v1.462117.8431.08
UnHype13117.3331.03
ESD Single35717.0630.54
HyperErase with ESD35916.9630.57
RECE Single6817.7430.66
HyperErase with RECE7317.4830.88

Response to W2:

The GPU training time, number of training epochs, inference latency and peak VRAM consumption are reported as follows:

TypeNumber
GPUA6000 * 1
Training Time266:32 min
Inference latency (100 images)303 ms (original model: 301 ms)
Epoch8000
Peak VRAM Consumption15376 MB

The time cost here does not include LoRA collection. The cost of collecting LoRAs depends on the single-concept method. For ESD, single-concept erasure typically takes 10 to 20 minutes.

Response to W3:

Thank you for your comments. Here we provide a more detailed explanation of this phenomenon. We argue that, as a jointly trained multi-concept conditional parameter generator, HyperErase learns a shareable concept-to-erasure parameter mapping from multiple concepts and their teacher optimization trajectories; the shared representations and parameter generation prior formed through joint training enable the model to extract more stable erasure directions than a single teacher checkpoint, yielding better erasure–preservation trade-offs.

To validate this explanation, we compared HyperErase trained on a single concept versus trained on 14 concepts, and used each to generate the effective LoRA ΔWPG=BA\Delta W_{\mathrm{PG}}=BA for the same target concept. Both PGs used exactly the same reference, namely the average effective update of 50 teacher checkpoints corresponding to the target concept—the teacher trajectory center:

=\frac{1}{50}\sum_{s=1}^{50}B_sA_s.$$ We report the following two directional consistency metrics. The scale-aligned distance allows an optimal global scale coefficient before comparison, thereby avoiding interference from LoRA magnitude differences in direction comparison: $$d = \min _{\alpha} \frac{ \\| \alpha \Delta W _{\mathrm{PG}} - \overline{\Delta W} _{\mathrm{church}} \\| _F} { \\| \overline{\Delta W} _{\mathrm{church}} \\| _F}, \quad \alpha ^{*} = \frac{ \langle \Delta W _{\mathrm{PG}}, \overline{\Delta W} _{\mathrm{church}} \rangle } { \\| \Delta W _{\mathrm{PG}} \\| _F ^{2} }.$$ Cosine similarity measures the angular consistency: $$\mathrm{CosSim} = \frac{ \langle \Delta W _{\mathrm{PG}}, \overline{\Delta W} _{\mathrm{church}} \rangle } { \\| \Delta W _{\mathrm{PG}} \\| _F \\, \\| \overline{\Delta W} _{\mathrm{church}} \\| _F }.$$ The experimental results are shown in the table below: | Type | Scale-aligned distance ↓ | Cosine with trajectory center ↑ | |---|---|---:| | HyperErase with single concept | 0.1836 | 0.9830 | | HyperErase with 14 concept | 0.1175 | 0.9931 | As can be seen, compared with erasing a single concept, HyperErase trained on 14 concepts reduces the scale-aligned distance to the same church teacher trajectory center by 36.0%, while the cosine similarity increases from 0.9830 to 0.9931. This indicates that multi-concept joint training generates target concept erasure directions closer to the stable center of the teacher trajectory, providing direct parameter-space evidence for our judgment: the full HyperErase can indeed transfer shared concept-to-parameter generation knowledge through joint multi-concept learning, thereby obtaining more accurate and stable functional erasure directions than learning the target concept alone. # Response to W4: We appreciate your suggestion. We conducted multi-concept erasure using model merging and report the comparison results between our method and model merging on Nudity below. As can be seen, directly applying model merging causes the concept erasure to completely fail. This is because model merging essentially dilutes the erasure capability for individual concepts; as the number of concepts increases, the model's generation ability is entirely disrupted, manifesting as very poor CLIP and FID scores as shown in the table. The high erasure rate in the table, however, is a consequence of the model's generation capability being destroyed and its inability to produce normal images. | Model | Total | FID | CLIP | |:---:|:---|:---:|:---:| | SD v1.4 | 621 | 17.84 | 31.08 | | ESD Single | 357 | 17.06 | 30.54 | | model merging | 1 | 246.59 |19.60 | | HyperErase with ESD | 359 | 16.96 | 30.57 | For a more comprehensive comparison, we apply SPEED and MACE separately for single-concept erasure, then use HyperErase to perform multi-concept erasure on both SPEED and MACE respectively, comparing the results against the native multi-concept erasure outcomes of these two methods. This allows us to verify the erasure effectiveness of our multi-concept approach compared to SPEED and MACE, as well as demonstrate the generality of our method. However, these experiments are not yet complete; we will append the results below once finished. # Response to Ethics Concerns: We understand your concerns. The field of concept erasure aims to prevent generative models from producing unsafe content, which is why our research necessarily involves nudity and NSFW content. However, our objective is to prevent the generation of such content rather than disseminate it, and we have taken care to implement protective measures in our writing. Moreover, previous papers in this area have not encountered such ethics concerns. Nevertheless, we agree that your suggestion is meaningful. Therefore, we are happy to include an ethics discussion section in the appendix, following the practice of other fields with ethics concerns, to address data sources, visual masking, safety considerations, and related issues.
AuthorsOpenReview ↗

Extra comparison with MACE Multi, SPEED Multi and Model Merging

Below, we supplement the comparison results with multi-concept erasure methods MACE, SPEED, and Model Merging. First, it should be noted that since the modules trained or edited by MACE and SPEED differ from those of ESD and rely on using the modules specified in their respective papers, they cannot be directly transferred to the modules trained by ESD. Therefore, we trained HyperErase using the single-concept erasure trajectories of the corresponding modules of MACE and SPEED, and compared them with their respective multi-concept erasure methods. For Model Merging, we directly used the ESD-trained LoRA for merging and experimentation. The results are shown in the following three tables, respectively.

ModelNudity TotalObject ASRtASR_t(%)Artist ASRASR(%)FIDCLIP
SPEED Multi36350.010.019.1330.94
HyperErase25642.53.319.0331.07
ModelNudity TotalObject ASRtASR_t(%)Artist ASRASR(%)FIDCLIP
MACE Multi350.00.0219.8227.73
HyperErase510.80.0219.7931.06
ModelNudity TotalObject ASRtASR_t(%)Artist ASRASR(%)FIDCLIP
model merging10.00.00246.5919.60
HyperErase3593.920.0816.9630.57

As can be seen, compared with the multi-concept erasure results of SPEED, HyperErase achieves better performance in both erasure effectiveness and generation capability metrics. Compared to MACE, although the erasure capability is slightly weaker than that of MACE, the CLIP metric of MACE drops significantly (the CLIP of the original model is 31.08). We argue that the superior multi-concept erasure results of MACE may be attributed to the degradation of generation capability. Directly applying model merging causes the concept erasure to completely fail. This is because model merging essentially dilutes the erasure capability for individual concepts; as the number of concepts increases, the model's generation ability is entirely disrupted, manifesting as very poor CLIP and FID scores as shown in the table. The high erasure rate in the table, however, is a consequence of the model's generation capability being destroyed and its inability to produce normal images. Overall, HyperErase's multi-concept erasure capability is on par with those of MACE and SPEED, and it can serve as a fairly general multi-concept erasure complement.

Reviewer_UVzxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the HyperErase, a framework for concept erasure based on hypernetwork-driven prompt-conditioned parameter synthesis. They first reformulate concept erasure as prompt-conditioned parameter amortization and train a hypernetwork to map textual descriptions to prompt-specific LoRA updates, eliminating the need for per-prompt or manual LoRA merging. To enhance the stability and precision of synthesized adapters, they design a decoupled rectification strategy, which disentangles LoRA tokens into pattern and scale subspaces, applies a square-root transform to curb multiplicative over-scaling, and leverages teacher-derived canonical priors for inference-time correction.

Critical strengths
  1. The paper reformulates concept erasure as a conditional parameter generation problem, where a hypernetwork avoids repeated inference-time fine-tuning and LoRA merging. Its separation of parameter patterns and magnitudes also mitigates high-dimensional regression errors.
  2. Extensive experiments show that HyperErase achieves erasure performance close to the single-concept ESD teacher while maintaining competitive FID and CLIP scores.
  3. The paper is easy to understand.
Critical weaknesses
  1. The novelty is limited. UnHype[1] already uses a text-conditioned hypernetwork to dynamically generate LoRA parameters for single- and multi-concept unlearning, directly challenging the claim that this is the first hypernetwork-based erasure framework. Besides, DyME [2] proposes dynamically composing concept-specific LoRAs at inference time with bi-level orthogonality constraints, showing that dynamic multi-concept erasure itself is not a novel formulation. Though HyperErase introduces a different parameter regression and calibration strategy, its fundamental methodological advances over these closely related approaches remain insufficiently established. A direct comparison of these two works and a clearer clarification of their different contributions are necessary for establishing the novelty of the paper. [1] UnHype: CLIP-Guided Hypernetworks for Dynamic LoRA Unlearning, ICML 2026. [2] DyME: Dynamic Multi-Concept Erasure in Diffusion Models with Bi-Level Orthogonal LoRA Adaptation, CVPR 2025.
  2. The claim of prompt-specific adaptation lacks sufficient evidence. Since the hypernetwork is trained on concept-specific teachers paired with concept-specific prompts, different prompts for the same concept likely share identical parameter targets. Without a concept-aware LoRA routing baseline or tests on multi-concept and indirect prompts, it remains unclear whether the model truly learns prompt-level adaptation rather than merely mapping concepts to precomputed LoRAs.
  3. The experimental comparison is too narrow. The main quantitative baselines are limited to ESD variants, which suffer from severe image quality degradation, thus creating a weakness in comparison. There is a lack of direct evaluation of representative multi-concept methods such as MACE.
  4. The ablation study is incomplete and lacks efficiency analysis. Table 7 only compares the complete method with the undecoupled and uncorrected version, and fails to isolate the contributions of each component. This article also omits the quantitative measurements of inference overhead, hypernetwork parameter counting, and training costs, all of which are at the core of its claimed computational advantages.
Questions for authors

Please refer to the weakness.

3 comments

We thanks reviewer's thoughtful comments. We provide detailed responses to your concerns below.

Response to W1:

HyperErase differs significantly from the UnHype method. UnHype directly trains a CLIP-conditioned hypernetwork by aligning local gradients with a manually designed forgetting target gradient field, thereby avoiding the need to collect pre-computed teacher LoRA weights. While this approach effectively improves erasure performance and reduces cost, it also limits its erasure capability to its own training objective and the inductive biases and capacity of its network architecture, preventing it from effectively leveraging the capabilities of existing single-concept erasure methods. In contrast, the contribution of HyperErase lies in providing a general multi-concept erasure framework for existing single-concept erasure methods. By training on the checkpoints generated by these single-concept methods after completing concept erasure, HyperErase offers a universal multi-concept erasure solution for methods that excel at single-concept erasure but cannot scale to multiple concepts. Precisely because of this, the erasure performance of HyperErase is inherently correlated with the gold baseline, and our experiments are therefore compared primarily against the gold baseline.

To address your question, we present in the table below the nudity erasure results under the multi-concept erasure setting, using ESD and RECE as teacher models, respectively. In our main experiments, for a fair comparison, we uniformly adopt ESD as the teacher model. Due to the inherent limitations of ESD itself, the results of HyperErase may appear inferior to those of UnHype. However, in the ablation study, when we fit LoRA from the edited weights generated by RECE and use it as the teacher model, HyperErase's erasure performance surpasses that of UnHype. We do not claim that HyperErase outperforms UnHype in all aspects. Rather, we aim to demonstrate that, unlike UnHype, our method can serve as a general multi-concept erasure complement to existing concept erasure methods, thereby achieving composability and upgradability that UnHype does not possess.

ModelTotalFIDCLIP
SD v1.462117.8431.08
UnHype13117.3331.03
ESD Single35717.0630.54
HyperErase with ESD35916.9630.57
RECE Single6817.7430.66
HyperErase with RECE7317.4830.88

Although both HyperErase and DyME carry the term "Dynamic", their methodological mechanisms are fundamentally different. DyME addresses the problem of selecting and composing LoRAs from a pretrained, concept-specific LoRA library. Its "Dynamic" nature lies in determining the correct erasure subset at inference time, which is essentially a combination of fixed LoRAs. In contrast, the dynamics of HyperErase operate at the parameter level: its hyper-network generates a personalized LoRA conditioned on the input prompt, rather than selecting or combining from a fixed LoRA pool.

Furthermore, DyME relies on keyword matching; therefore, it completely fails on prompts that do not contain explicit keywords yet still involve target concepts, especially for Nudity datasets and adversarial attack benchmarks. This limitation is also reflected in their experiments: they do not evaluate nudity erasure at all. Since their code is not publicly available, we are currently unable to conduct a direct empirical comparison.

We also note a likely bibliographic problem in your review. The review cites "[2] DyME: Dynamic Multi-Concept Erasure in Diffusion Models with Bi-Level Orthogonal LoRA Adaptation, CVPR 2025." Based on our search, this paper was only submitted to ICLR 2026 and subsequently withdrawn; it currently exists only as a preprint, and we found no evidence of a CVPR 2025 publication. We respectfully ask the reviewer to verify the citation.

Response to W2:

We understand your concerns. However, the issues with a concept-aware LoRA routing baseline are the same as those of DyME (raised by W1): it must rely on target-matching methods for routing. This makes it difficult to correctly distinguish and route when erasing abstract concepts, especially Nudity and Artist Style. Moreover, the performance of a concept-aware LoRA routing baseline still depends on and is unlikely to surpass the gold baseline, and our main text already provides extensive comparisons and analysis against the gold baseline. Therefore, we believe that a comparison with a concept-aware LoRA routing baseline is unnecessary.

Response to W3:

We appreciate your valuable feedback, and we find your concerns highly meaningful. We are currently conducting controlled experiments using SPEED. In addition, we have also performed experiments with MACE's multi-concept method and Marging for comparative analysis. However, these experiments are not yet complete; we will append the results below once finished.

For a more comprehensive comparison, we apply SPEED and MACE separately for single-concept erasure, then use HyperErase to perform multi-concept erasure on both SPEED and MACE respectively, comparing the results against the native multi-concept erasure outcomes of these two methods. This allows us to verify the erasure effectiveness of our multi-concept approach compared to SPEED and MACE, as well as demonstrate the generality of our method.

For SPEED, since it is inherently an editing method, we follow the same erasure protocol as RECE. We first fit the edited weights from SPEED into corresponding LoRAs, and then proceed with multi-concept erasure. For Marging, we directly utilize the concept erasure LoRAs from ESD for marging and subsequently evaluate the results.

Response to W4:

Below, we present the experimental results for w/o Square-Root Transform, without decoupling, and with decoupling. When using logarithmic encoding without decoupling, systematic bias accumulates as the number of concepts increases, causing the LoRA generated by the Hyper-network to lose its erasing capability and interfere with normal image generation. This ultimately manifests as erasure results approaching those of the original model, with a severe degradation in CLIP and FID scores.

TypeTotalFID (↓)CLIP (↑)
SD v1.462117.8431.08
ESD Single35717.0630.54
w/o Square-Root Transform and w/o decouple60318.5228.87
w Square-Root Transform and w/o decouple46417.7730.35
w Square-Root Transform and w decouple35916.9630.57
AuthorsOpenReview ↗

Extra comparison with MACE Multi, SPEED Multi and Model Merging

Below, we supplement the comparison results with multi-concept erasure methods MACE, SPEED, and Model Merging. First, it should be noted that since the modules trained or edited by MACE and SPEED differ from those of ESD and rely on using the modules specified in their respective papers, they cannot be directly transferred to the modules trained by ESD. Therefore, we trained HyperErase using the single-concept erasure trajectories of the corresponding modules of MACE and SPEED, and compared them with their respective multi-concept erasure methods. For Model Merging, we directly used the ESD-trained LoRA for merging and experimentation. The results are shown in the following three tables, respectively.

ModelNudity TotalObject ASRtASR_t(%)Artist ASRASR(%)FIDCLIP
SPEED Multi36350.010.019.1330.94
HyperErase25642.53.319.0331.07
ModelNudity TotalObject ASRtASR_t(%)Artist ASRASR(%)FIDCLIP
MACE Multi350.00.0219.8227.73
HyperErase510.80.0219.7931.06
ModelNudity TotalObject ASRtASR_t(%)Artist ASRASR(%)FIDCLIP
model merging10.00.00246.5919.60
HyperErase3593.920.0816.9630.57

As can be seen, compared with the multi-concept erasure results of SPEED, HyperErase achieves better performance in both erasure effectiveness and generation capability metrics. Compared to MACE, although the erasure capability is slightly weaker than that of MACE, the CLIP metric of MACE drops significantly (the CLIP of the original model is 31.08). We argue that the superior multi-concept erasure results of MACE may be attributed to the degradation of generation capability. Directly applying model merging causes the concept erasure to completely fail. This is because model merging essentially dilutes the erasure capability for individual concepts; as the number of concepts increases, the model's generation ability is entirely disrupted, manifesting as very poor CLIP and FID scores as shown in the table. The high erasure rate in the table, however, is a consequence of the model's generation capability being destroyed and its inability to produce normal images. Overall, HyperErase's multi-concept erasure capability is on par with those of MACE and SPEED, and it can serve as a fairly general multi-concept erasure complement.

Reviewer_HmcfOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a hypernetwork-based framework for multi-concept erasure by formulating it as prompt-conditioned parameter generation. It generates prompt-specific LoRA updates without gradient optimization at inference time. Experiments show that HyperErase achieves erasure performance close to that of the gold-standard baseline while maintaining non-target generation performance.

Critical strengths
  • The paper presents a practical approach by formulating concept erasure as prompt-conditioned LoRA generation. Using a single hypernetwork, HyperErase generates erasure adapters based on input prompts without requiring explicit adapter selection or additional gradient optimization at inference time.
  • The paper provides relatively extensive experiments across different adversarial attacks and concept erasure categories. It also evaluates the applicability of the proposed framework using additional backbones and different erasure teachers.
Critical weaknesses

[W1] Insufficient differentiation from prior hypernetwork-based unlearning methods. UnHype [1] also generates dynamic unlearning LoRAs using a hypernetwork, but avoids collecting precomputed teacher LoRA weights. In contrast, HyperErase requires concept-specific teacher checkpoints for training. The paper does not discuss UnHype, making it difficult to assess the distinct contributions and relative advantages of HyperErase.

[W2] While the paper presents HyperErase as a framework for multi-concept erasure, its Mix-setting experiments are limited to SD v1.4. The experiments on FLUX and alternative erasure teachers focus on nudity erasure, leaving it unclear whether the method maintains its multi-concept erasure and preservation performance in these settings.

[W3] Limited analysis of computational costs and practical benefits. HyperErase requires collecting concept-specific teacher checkpoints and training an additional hypernetwork. Although the paper reports checkpoint counts and training settings, it lacks quantitative comparisons of training costs, inference latency, and storage requirements, making it difficult to assess whether the practical benefits justify the additional overhead.

[W4] Comparisons with stronger multi-concept erasure methods are missing. While the paper highlights interference and non-target degradation in static/merged-weight erasure, its comparisons mainly focus on ESD variants. Comparisons with recent methods such as SPEED [2] would help clarify the additional benefits of prompt-conditioned parameter generation over strong multi-concept erasure baselines.


[1] Wójcik, Piotr, et al. "Unhype: Clip-guided hypernetworks for dynamic lora unlearning.", ICML, 2026.

[2] Li, Ouxiang, et al. "Speed: Scalable, precise, and efficient concept erasure for diffusion models.", ICLR, 2026.

Questions for authors
  1. Are out-of-family prompts included in the hypernetwork training data? If not, how are the claimed near-zero updates achieved?

  2. For the MS-COCO FID/CLIP evaluation, is a target-conditioned LoRA reused across COCO prompts, or is each COCO prompt passed through the hypernetwork to generate its own LoRA?

  3. The FLUX experiments collect prompt-LoRA pairs by continuing ESD training at a lower learning rate after erasure. Is the same procedure used for SD v1.4?

  4. Could the authors clarify the validation protocol used for hyperparameter selection? Was the validation set separate from the test set?

4 comments

We thanks reviewer's thoughtful comments. We provide detailed responses to your concerns below.

Response to W1:

HyperErase differs significantly from the UnHype method. UnHype directly trains a CLIP-conditioned hypernetwork by aligning local gradients with a manually designed forgetting target gradient field, thereby avoiding the need to collect pre-computed teacher LoRA weights. While this approach effectively improves erasure performance and reduces cost, it also limits its erasure capability to its own training objective and the inductive biases and capacity of its network architecture, preventing it from effectively leveraging the capabilities of existing single-concept erasure methods. In contrast, the contribution of HyperErase lies in providing a general multi-concept erasure framework for existing single-concept erasure methods. By training on the checkpoints generated by these single-concept methods after completing concept erasure, HyperErase offers a universal multi-concept erasure solution for methods that excel at single-concept erasure but cannot scale to multiple concepts. Precisely because of this, the erasure performance of HyperErase is inherently correlated with the gold baseline, and our experiments are therefore compared primarily against the gold baseline.

To address your question, we present in the table below the nudity erasure results under the multi-concept erasure setting, using ESD and RECE as teacher models, respectively. In our main experiments, for a fair comparison, we uniformly adopt ESD as the teacher model. Due to the inherent limitations of ESD itself, the results of HyperErase may appear inferior to those of UnHype. However, in the ablation study, when we fit LoRA from the edited weights generated by RECE and use it as the teacher model, HyperErase's erasure performance surpasses that of UnHype. We do not claim that HyperErase outperforms UnHype in all aspects. Rather, we aim to demonstrate that, unlike UnHype, our method can serve as a general multi-concept erasure complement to existing concept erasure methods, thereby achieving composability and upgradability that UnHype does not possess.

ModelTotalFIDCLIP
SD v1.462117.8431.08
UnHype13117.3331.03
ESD Single35717.0630.54
HyperErase with ESD35916.9630.57
RECE Single6817.7430.66
HyperErase with RECE7317.4830.88

Response to W2:

Below, we report additional multi-concept erasure results using FLUX.1 Schnell (a faster variant of FLUX) as the base model and multiple single-concept erasure LoRAs erased by Flowerase-RL as teachers. Experiments are validated on Nudity, Object, and Artist concepts, with results summarized in the table. It can be observed that HyperErase's multi-concept erasure performance does not degrade significantly compared to the single-concept teachers.

ModelNudity TotalObject ASRtASR_tArtist ASRASR
FLUX 1.Schnell602//
FlowErase-RL Sngle530.770.04
HyperErase580.950.04

Response to W3:

The GPU training time, number of training epochs, inference latency and peak VRAM consumption are reported as follows:

TypeNumber
GPUA6000 * 1
Training Time266:32 min
Inference latency (100 images)303 ms (original model: 301 ms)
Epoch8000
Peak VRAM Consumption15376 MB

The time cost here does not include LoRA collection. The cost of collecting LoRAs depends on the single-concept method. For ESD, single-concept erasure typically takes 10 to 20 minutes.

Response to W4:

We appreciate your valuable feedback, and we find your concerns highly meaningful. We are currently conducting controlled experiments using SPEED. In addition, we have also performed experiments with MACE's multi-concept method and Marging for comparative analysis. However, these experiments are not yet complete; we will append the results below once finished.

For a more comprehensive comparison, we apply SPEED and MACE separately for single-concept erasure, then use HyperErase to perform multi-concept erasure on both SPEED and MACE respectively, comparing the results against the native multi-concept erasure outcomes of these two methods. This allows us to verify the erasure effectiveness of our multi-concept approach compared to SPEED and MACE, as well as demonstrate the generality of our method.

For SPEED, since it is inherently an editing method, we follow the same erasure protocol as RECE. We first fit the edited weights from SPEED into corresponding LoRAs, and then proceed with multi-concept erasure. For Marging, we directly utilize the concept erasure LoRAs from ESD for marging and subsequently evaluate the results.

Response to Q1:

Out-of-family prompts are not included in the training data of HyperErase. Near-zero updates are inherently determined by the prompt-conditioned amortized generation itself. Since the Hyper-network is trained exclusively on target concepts to be erased, the erasure mapping learned by the Hyper-network is concentrated on those target concepts. For inputs of irrelevant concepts, the output magnitude of the Hyper-network naturally remains near-zero.

It should also be noted that, although HyperErase is a general multi-concept erasure framework, its actual erasure effectiveness and near-zero updates still depend on the erasure capability of the teacher. This is also why the main experiments are primarily compared against the gold baseline ESD. By replacing the gold baseline, the erasure performance of HyperErase will correspondingly improve or degrade.

Response to Q2:

We appreciate your pointing this out. We indeed did not explicitly clarify this in the main text. For Nudity erasure, we generate a LoRA for every input prompt including MSCOCO and evaluate accordingly. Specifically, for each input prompt, a LoRA is first generated via HyperErase, and then this LoRA is used together with the base model to generate images. For Object and Artist, to more accurately assess the generated LoRA's effectiveness, we follow different protocols for different metrics. For ASRtASR_t prompts, we normally generate personalized LoRAs for these prompts. However, for Object's ASRkASR_k and the MSCOCO dataset, on one hand, since our multi-concept method simultaneously erases all target concepts, if we were to follow this standard pipeline when testing the target ASRkASR_k, all concepts would be erased, making it impossible to correctly reflect ASRkASR_k. On the other hand, we also want to compare the preservation capability of the generated LoRA against the gold baseline. Therefore, when evaluating ASRkASR_k and MSCOCO for Object and Artist, we first generate a LoRA using the target concept prompt, and then test all these prompts with that LoRA. We will add this clarification in the revised version, along with a table showing the erasure metrics and generation capability metrics when HyperErase is applied in full.

Response to Q3:

Yes, for any base model, our method collects prompt--LoRA pairs by continuing training at a lower learning rate after erasure. For editing-based methods, we first apply the method to complete single-concept erasure, then fit the edited weights into a LoRA, and subsequently perform training.

Response to Q4:

The training set used by our method consists of the gold baseline's training set and the corresponding LoRAs. The validation set, as shown in the appendix, is a dataset commonly used in the concept erasure field, differs from the training set, and is also the test dataset used by the gold baseline. During training, we did not set aside a test set for hyperparameter tuning; we only output the loss in the logs for debugging purposes. Our method is not tuned for specific datasets. We also did not perform hyperparameter optimization; most hyperparameters are the same as those used in ESD. The lower learning rate used for obtaining LoRAs was simply set to one order of magnitude smaller than ESD's learning rate.

AuthorsOpenReview ↗

Extra comparison with MACE Multi, SPEED Multi and Model Merging

Below, we supplement the comparison results with multi-concept erasure methods MACE, SPEED, and Model Merging. First, it should be noted that since the modules trained or edited by MACE and SPEED differ from those of ESD and rely on using the modules specified in their respective papers, they cannot be directly transferred to the modules trained by ESD. Therefore, we trained HyperErase using the single-concept erasure trajectories of the corresponding modules of MACE and SPEED, and compared them with their respective multi-concept erasure methods. For Model Merging, we directly used the ESD-trained LoRA for merging and experimentation. The results are shown in the following three tables, respectively.

ModelNudity TotalObject ASRtASR_t(%)Artist ASRASR(%)FIDCLIP
SPEED Multi36350.010.019.1330.94
HyperErase25642.53.319.0331.07
ModelNudity TotalObject ASRtASR_t(%)Artist ASRASR(%)FIDCLIP
MACE Multi350.00.0219.8227.73
HyperErase510.80.0219.7931.06
ModelNudity TotalObject ASRtASR_t(%)Artist ASRASR(%)FIDCLIP
model merging10.00.00246.5919.60
HyperErase3593.920.0816.9630.57

As can be seen, compared with the multi-concept erasure results of SPEED, HyperErase achieves better performance in both erasure effectiveness and generation capability metrics. Compared to MACE, although the erasure capability is slightly weaker than that of MACE, the CLIP metric of MACE drops significantly (the CLIP of the original model is 31.08). We argue that the superior multi-concept erasure results of MACE may be attributed to the degradation of generation capability. Directly applying model merging causes the concept erasure to completely fail. This is because model merging essentially dilutes the erasure capability for individual concepts; as the number of concepts increases, the model's generation ability is entirely disrupted, manifesting as very poor CLIP and FID scores as shown in the table. The high erasure rate in the table, however, is a consequence of the model's generation capability being destroyed and its inability to produce normal images. Overall, HyperErase's multi-concept erasure capability is on par with those of MACE and SPEED, and it can serve as a fairly general multi-concept erasure complement.

[W1] Thank you for your response regarding UnHype. HyperErase differs from UnHype by learning from existing single-concept erasure methods rather than directly optimizing an erasure objective. However, HyperErase is likewise constrained by its hypernetwork capacity and teacher quality. Moreover, Appendix B.2 notes that RECE-to-LoRA fitting is not always successful. Since the rebuttal table reports the result labeled "Best" in Table 13 without this designation, it would be helpful to clarify how representative this result is.

In addition, the comparison table reports only nudity performance under the 14-concept setting. Since the rebuttal states that the contribution lies in extending single-concept erasers to the multi-concept setting, the comparison should also cover the other target concepts erased in the same setting, not nudity alone. Finally, the claim of the “first hypernetwork-based CE framework” should be qualified.


[W2] Thank you for the additional results on FLUX.1 Schnell. The table reports erasure metrics only. Since my concern also covered preservation, could the authors add FID/CLIP and, if available, non-target preservation results for this setting?


[Q1] I am still unsure how near-zero updates arise for out-of-family prompts. Eq. 7 renormalizes the predicted patterns, while Eqs. 8–9 use teacher-derived scale targets. Moreover, Appendix C.4 describes calibrating update norms to teacher norms.

Could the authors clarify how scale prediction and norm calibration are handled for out-of-family prompts, and report the effective update norms ∥B^A^∥F\|\hat{B}\hat{A}\|_F for out-of-family prompts (e.g., MS-COCO) compared with in-family prompts?


[Q2] Thank you for the clarification. Since prompt-conditioned inference is a central contribution of HyperErase, could the authors provide the promised full-pipeline erasure and preservation results during the rebuttal?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.