acceptodds
Withdrawn from ICLR 2027

Concept Score Relearning: A Unified Cross-Architecture Attack on Concept Erasure

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Concept erasure aims to suppress undesirable knowledge in text-to-image generative models. However, existing robustness evaluations typically rely on relearning attacks tailored to specific model architectures. We study concept reactivation across two substantially different generative paradigms: noise-prediction U-Nets and flow-matching Transformers. We introduce **Concept Score Relearning (CSR)**, a unified parameter-level framework that reactivates erased concepts by optimizing each model within its native prediction space. CSR requires no external target-concept image dataset and applies the same concept-directed objective to both U-Net-based Stable Diffusion and Transformer-based FLUX. Experiments across diverse concepts and multiple erasure methods demonstrate consistent concept reactivation across both architectures, highlighting the cross-architecture applicability of CSR and the persistent recoverability of apparently erased concepts. For strict nudity, CSR reaches average ASRs of 50.47% on FLUX and 40.29% on Stable Diffusion, consistently ranking first across all evaluated safety settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
AI Safety

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VjKbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets a gap in concept erasure for text to image models. Erasure methods are usually judged without any cross architecture test. The authors propose Concept Score Relearning (CSR), a parameter level relearning attack. CSR takes a frozen, unerased public reference model, written as phi. At the same latent and the same timestep or noise level it computes the difference between the conditional and the null prediction. The authors evaluate on SD-1.4 and FLUX.1-dev, over six erasure methods (ESD, EA, AC, EAP, CP, MACE) and five concept groups (nudity, violence, artistic style, and the entity, abstraction and relationship categories). They compare against four attack baselines (UnlearnDiffAtk, P4D, Ring-A-Bell, ReFLUX), report CLIPScore, FID and LPIPS for utility, and run a blinded human study with 20 participants. Strict nudity ASR rises from 7.60 percent after erasure to 50.47 percent on FLUX, and from 8.60 percent to 40.29 percent on SD-1.4. Violence rises from 21.33 percent to 71.44 percent on FLUX, and from 39.04 percent to 59.92 percent on SD-1.4. The paper reports first place across the evaluated settings. One concept update takes about 2.4 minutes on SD-1.4 and about 5 minutes on FLUX. The stated conclusion is that suppression after erasure does not imply resistance to later parameter level relearning.

Critical strengths
  1. The paper correctly notes that existing parameter level relearning attacks are tied to one architecture. ReFLUX (Jiang et al., 2025) relies on FLUX velocity prediction and on specific attention mechanisms.

  2. The study spans six erasure methods, two architectures, five concept groups and four attack baselines. It adds three utility metrics and a human study, all documented in detail. Appendix A.3.6 states the threat model precisely. The attacker holds the erased weights, the target concept prompt, compute, and the unerased reference checkpoint, and does not need to know the erasure algorithm.

Critical weaknesses
  1. Liu and Zhang (2025, arXiv 2505.16174) propose a parameter level, concept directed relearning attack on U-Nets. The paper cites it once in related work. At eta equal to 1, CSR is in essence matching the reference model's own conditional prediction, which overlaps strongly with that setting. Since eta equal to 1 is much weaker than eta equal to 3, a reader cannot tell what CSR adds on the SD side beyond that method. The cross architecture story is missing its most important control.

  2. Each of the 12 main settings is run once. The appendix states that every cell comes from a single seed 0 attack run. The only error bars in the whole paper come from the human study. The authors themselves write in Appendix A.7.4 that small differences between nearby settings should not be read as statistically meaningful. The default hyperparameters, eta equal to 3 and N equal to 40, were chosen on those same single runs. Two curves show how large the wobble can be. The FLUX anchor sweep gives a lower score at N equal to 10 than at N equal to 5, and the FLUX budget curve moves 21.40, then 17.89, then 22.11, then 38.25. The abstract still says the method consistently ranks first across all evaluated safety settings. The human study reports only means and standard deviations and runs no test at all, while the main text claims the appendix provides a full study design and statistical analysis. On concept reactivation the two methods sit at 4.62 plus or minus 0.91 and 4.52 plus or minus 0.78, so the intervals overlap.

  3. ReFLUX is a LoRA fine tune and needs a real target concept image dataset. CSR updates about 2.51 billion parameters on FLUX, which is 21.1 percent of the Transformer, for 200 steps at a learning rate of 1e-4, and it updates the entire U-Net on SD-1.4 for 500 steps at 1e-5. The learning rates differ by a factor of ten and the step counts by a factor of two and a half. The appendix justifies this only by memory and hardware limits. On the SD side, ReFLUX is a port that the authors retuned, and two loss terms calibrated for FLUX were zeroed out. MACE on FLUX keeps only the CFR stage. EAP on FLUX uses an approximate configuration with no direct equivalent. EA on SD-1.4 is reduced from a bi-level optimization to a single prior preservation term. Only ReFLUX shares the white box parameter level threat model with CSR, since the other three baselines attack a frozen victim from prompt space or from a black box. The claim of first place therefore mixes threat models. No baseline uses an unerased reference model as its supervision source, so none of them is matched to CSR on that axis.

  4. CLIPScore, FID and LPIPS cover only ESD and EA, and only three concept blocks, namely nudity, violence and Van Gogh. Picasso and the other four erasure methods are absent. The conclusion is still written as preserving general generation utility. The paper has no limitations section.

Questions for authors
  1. At eta equal to 1, CSR matches the reference model's conditional prediction. That setting overlaps strongly with the U-Net concept directed noise prediction target of Liu and Zhang (2025, arXiv 2505.16174). Can you run that method as an SD side baseline? If the real insight is the gain from eta equal to 3 over eta equal to 1, from 17.89 to 32.98 percent, then how much of the increment over that baseline comes from the target construction and how much from the extrapolation hyperparameter?

  2. The main tables and the ablations are single seed 0 runs, and the paper has no error bars outside the human study. Can you report ASR distributions over at least three attack seeds? The abstract numbers of 50.47 and 40.29 percent are unweighted means over six erasure methods. Can you also give a method weighted view?

  3. On FLUX, CSR updates about 2.51 billion parameters, which is 21.1 percent, for 200 steps at 1e-4. On SD-1.4 it updates the whole U-Net for 500 steps at 1e-5. ReFLUX only trains LoRA adapters. How does ReFLUX perform at the same trainable parameter count and the same step count?

  4. The appendix says the SD-1.4 port of ReFLUX zeroed two losses that were calibrated for FLUX. Can you report the version with those terms restored, or at least quantify what zeroing them does to ReFLUX itself?

  5. The gap between eta equal to 1 and eta equal to 3 suggests that the concept direction is weakened after erasure and needs extrapolation. Can you characterize dcd_c quantitatively, for instance the cosine similarity or the norm ratio between the post erasure direction and the reference direction as a function of the timestep?

  6. CSR needs the unerased reference checkpoint phi. Since an attacker in this position could simply re-distill from phi, what does CSR establish beyond that simpler route?

  7. You state that the two prediction spaces are not equivalent and that each direction lives only in its own space, which I accept. Then what exactly is unified?

  8. The main text says the appendix provides a full study design and statistical analysis, but Appendix A.8 reports only participant level means and standard deviations and runs no test. The design is paired, so can you add a paired test, an effect size and a sample size justification?

  9. Have NudeNet at threshold 0.6, the binary Q16 classifier, WikiArt Top-3, and the 90 percent CLIP decision threshold used for the miscellaneous categories been validated?

  10. ReFLUX needs a real target concept image dataset while CSR needs only self generated anchors. Can you make that difference explicit in the main text rather than presenting it only as a capability?

Reviewer_ShQwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CSR—a unified, parameter-level attack applicable to both diffusion-based U-Net architectures and flow-matching-based Transformer architectures—designed to recover concepts that have been erased from these models. The approach leverages the discrepancy between "conditional" and "unconditional" predictions from an unerased reference model to guide the target model in relearning the erased concepts. Experiments conducted on SD-1.4 and FLUX.1-dev models demonstrate that this method can successfully reactivate concepts, even when those concepts have been removed using various erasure techniques or span a wide range of categories.

Critical strengths
  1. CSR provides a unified re-learning objective for diffusion-based U-Nets and flow-matching-based Transformers, without requiring external datasets.
  2. This paper covers a variety of erasure methods and conceptual categories, and includes a manual evaluation that demonstrates the effectiveness of the attacks.
Critical weaknesses
  1. The paper's core objective is a scaled version of the "difference between conditional and unconditional predictions," followed by distillation based on MSE. The conceptual distinction between this approach and existing methods such as "score-guided relearning" or formulations based on classifier-free guidance (CFG) has not yet been fully clarified.
  2. The CSR attack relies on the following premises:
    1. The attacker possesses the original, unerased model, which is inherently capable of generating the target concepts.
    2. The attacker must perform additional training on the erased model to recover these concepts. if the attacker already has the original model, why needs the additional cost of recovering the erased model? Why not simply use the original model instead? The paper need to justify the value of this white-box attack scenario.
  3. The paper does not prove that what is recovered is residual knowledge already present in the model, rather than knowledge re-injected via the original model.
Questions for authors

see critical weaknesses

Reviewer_YMkwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the robustness to relearning in concept erasure, pointing out that existing evaluations mostly focus on the immediate suppression effect and prompt-level circumvention (characterizing inference-time robustness), while overlooking the risk that erased concepts may remain recoverable through subsequent parameter updates. The authors systematically evaluate the proposed method across two substantially different generative architectures and six concept-erasure methods.

Critical strengths

1- This paper elevates the robustness evaluation from prompt-level circumvention to parameter-level recovery through relearning. If concept erasure methods fail to withstand parameter-level recovery, their safety guarantees lack long-term validity. This issue holds significant practical relevance for open-weight models.

2- By self-generating anchors from the reference model, the method eliminates the need to construct external image datasets for target concepts, thereby enhancing its practical applicability.

Critical weaknesses

1- The core method of the paper relies on an assumption: the attacker has access to a public, unerased reference model that is completely homologous to the victim model. However, this is a fragile assumption in real-world security scenarios. If the attacker already possesses the unerased model, they can directly generate the target images, which diminishes the actual threat value of attacking the erased model.

2- The paper compares CSR (a parameter-level attack requiring white-box access to both the erased model and the unerased reference model) with black-box prompt optimization baselines (such as Ring-A-Bell).

Questions for authors

1- Why are ablation experiments based on a single run? That doesn't seem convincing enough.

2- Given that CSR relies on a perfect unerased reference model, will CSR's performance degrade if there is a slight weight or architectural mismatch between the reference model and the erased model?

3- In Section 2.3, the authors enumerate a large number of literatures on inferency-time attacks, but only categorize them and list them, without in-depth analysis of the fundamental theoretical limitations of these methods in evaluating concept erasure.

4- The authors evaluated the effectiveness of the approach on two model architectures. Can you test it on a different backbone on the same architecture?

5- Figure 2 does not express the core idea of the paper and needs further optimization. The current version is a little rough.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.