acceptodds
Withdrawn from ICLR 2027

Training-Free Concept Erasure via Text-Projection Weight Editing for Visual Autoregressive Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual autoregressive (VAR) models have emerged as a competitive paradigm for high-resolution text-to-image generation. However, their growing generative capability raises concerns about undesirable content. Concept erasure aims to remove such concepts from pretrained models while preserving their general generation capability. Existing approaches to concept erasure in VARs either require time-consuming fine-tuning or introduce additional overhead through inference-time intervention. To address these limitations, we propose **TRACE** (**TRA**ining-Free **C**oncept **E**rasure), a training-free framework based on text-projection weight editing for VARs. Specifically, TRACE edits the linear layer of the text-projection module and traces concept-induced responses across different scales to construct scale-specific concept subspaces. TRACE then applies closed-form low-rank corrections to the text-projection weights to modulate components of the target concept from the projected text representations along these subspaces. We further introduce TRACE-Lite, a lightweight variant, which uses a single concept subspace and weight edit across all scales, completing concept erasure in about 3.40 seconds on a single GPU. Extensive experiments demonstrate that TRACE effectively erases target concepts while preserving generation quality.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
visual autoregressive model, concept erasure, text-to-image generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cKSPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TRACE, a training-free concept erasure method for visual autoregressive text-to-image models. Using paired prompts that respectively include and exclude the target concept, TRACE compares the model's predicted distributions across different generation scales and constructs scale-specific concept subspaces accordingly. It then applies closed-form low-rank weight editing to suppress components of text-projected representations within these subspaces, thereby reducing the generation of target concepts. The paper also introduces TRACE-Lite, which uses a shared concept subspace across scales to reduce editing costs.

Experiments on the Infinity-2B model evaluate concept erasure across three categories: nudity, artistic styles, and copyrighted characters. TRACE is compared with various fine-tuning-based, closed-form editing-based, and inference-time intervention methods. The results demonstrate that TRACE effectively reduces the generation of target concepts while preserving overall image quality, text alignment, and the ability to generate non-target concepts. Although TRACE-Lite adopts a simplified shared-subspace design, it achieves competitive erasure and preservation performance while substantially reducing editing overhead. Ablation studies further support the contributions of unit normalization, response weighting, and scale-specific subspace design to the method's performance.

Critical strengths
  1. The method addresses a consequential efficiency limitation. TRACE avoids per-concept gradient-based fine-tuning and the additional conditional forward passes used by inference-time erasure. TRACE-Lite is particularly attractive: its reported editing cost is substantially below S-VARE, while achieving competitive erasure and better COCO preservation metrics. This could make frequent concept updates practically feasible.

  2. The scale-aware response measurement is a sensible adaptation to autoregressive generation. Conditioning target and neutral predictions on identical preceding visual tokens controls an important source of variation. The resulting response weights connect subspace construction to the generator’s behavior rather than relying entirely on text-feature geometry. This is the most compelling methodological contribution.

Critical weaknesses
  1. Limited evaluation of safety robustness. The safety evaluation is restricted to a filtered nudity-related subset of I2P and uses NudeNet-based detection metrics. Using additional evaluation sets, such as Ring-A-Bell and MMA-Diffusion, would strengthen this evidence.

  2. Insufficient coverage of general object-category erasure. The evaluation covers nudity, one artistic style, and one copyrighted character, but does not include systematic erasure of ordinary object categories. While Pikachu provides evidence for entity-specific erasure, it does not establish that the method generalizes to broader categories with substantial variation and semantic overlap.

Questions for authors
  1. Can the authors evaluate the methods on additional evaluation sets, such as Ring-A-Bell[1] and MMA-Diffusion[2]? Please report the Attack Success Rate.

  2. Can the authors evaluate erasure across several ordinary object categories, such as church, golf ball, garbage truck ,and so on? Please report target-recognition success, preservation of unrelated objects and generation quality.

  3. Can the authors add an additional multi-concept test experiment? For example, mix target-neutral prompt pairs from multiple target concepts into a single calibration set, perform joint erasure, and then evaluate the same edited model using held-out prompts for each concept separately. How does this affect attack success, preservation quality, and computational cost?

[1] Tsai Y L, Hsu C Y, Xie C, et al. Ring-a-bell! how reliable are concept removal methods for diffusion models?[C]//International Conference on Learning Representations. 2024, 2024: 41543-41554.

[2] Yang Y, Gao R, Wang X, et al. Mma-diffusion: Multimodal attack on diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024: 7737-7746.

Reviewer_kJzuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how to reduce the computational cost of concept erasure in visual autoregressive text-to-image models and proposes TRACE, a training-free method for editing text projection weights. Its core idea is to use the prediction responses to target and neutral prompts at different generation scales to weight the differences between their text representations, construct scale-specific concept subspaces, and then modify the text projection weights through closed-form low-rank updates. The method performs response estimation during a calibration stage before generation, requiring neither gradient-based fine-tuning nor additional forward passes under target and neutral conditions during generation. The authors also propose TRACE-Lite, which uses a shared subspace to further reduce calibration costs. Experiments based on Infinity-2B compare the proposed method with existing methods in terms of erasure effectiveness, preservation of generation capabilities, and computational cost across three targets: nudity, Van Gogh style, and Pikachu.

Critical strengths
  1. I find the lightweight editing approach useful. TRACE restricts its changes to the text projection layer and suppresses concepts through a closed-form low-rank update, without gradient-based fine-tuning or additional conditional forward passes during generation. The runtime results in Table 3 show lower editing costs than S-VARE, while avoiding the extra computation that ScaleErasure requires for each generated image.

  2. The results in Table 1 show a favorable balance between concept erasure and generation quality. On nudity and style erasure, TRACE achieves better erasure metrics than the compared methods while keeping FID and CLIP scores on COCO-30K close to those of the original model. These results support the effectiveness of the lightweight weight-editing approach.

Critical weaknesses
  1. The technical motivation is not sufficiently grounded in the limitations of existing closed-form editing methods. The paper primarily motivates TRACE through the cost of fine-tuning and the overhead of inference-time interventions. However, UCE [1] already requires neither gradient-based training nor additional conditional forward passes. Table 3 also shows that the adapted UCE completes editing in 4.83 seconds. TRACE’s main contribution relative to such methods should therefore lie in improving the trade-off between concept erasure and content preservation. Although the paper reports performance gains, it does not analyze why existing closed-form editing methods leave residual target concepts or damage unrelated content in VAR models, or explain how these limitations motivate text-projection editing and scale-specific subspaces. This leaves the rationale for the proposed design and its contribution over existing closed-form approaches insufficiently established.

  2. The construction of the concept subspace lacks sufficient justification, while the closed-form derivation covers only the subsequent weight update. TRACE relies on a subspace constructed through prompt differences, JS-based response weighting, and residual SVD to identify the target concept. Differences between whole-prompt representations may capture semantic changes beyond the target concept, and a larger change in model predictions does not necessarily indicate that a direction more accurately represents that concept. Equations (10)–(17) take this subspace as given and derive the minimum weight change satisfying the scaling constraint; they provide no justification for the preceding concept-identification procedure. The analytical solution therefore rests on a key empirical assumption: the selected directions separate the target semantics from content that should be preserved. This assumption directly determines the selectivity of the edit, yet remains insufficiently supported.

  3. The experiments provide limited evidence for selective erasure and its applicability across concepts and prompts. The style and copyright experiments consider only Van Gogh’s style and Pikachu, respectively, while the safety evaluation omits adversarial prompt tests such as MMA and Ring-A-Bell, which were included in S-VARE [2]. Moreover, FID and CLIP scores on COCO primarily assess general generation quality and text alignment; they do not directly measure whether other objects, actions, and backgrounds are preserved in prompts containing the erased concept. For copyright erasure, CLIP-E and CLIP-S measure similarity to the names of the target and non-target characters, respectively. They do not quantify how often the target character remains recognizable or how erasure affects other content within the same image. The qualitative examples offer some supporting evidence, but these capabilities lack systematic evaluation, limiting confidence in the method’s erasure effectiveness and preservation of unrelated content.

References [1] Gandikota, R., Orgad, H., Belinkov, Y., Materzyńska, J., & Bau, D. (2024). Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 5111–5120). [2] Zhong, X., Zhou, Y., Zhang, Z., Li, J., Sun, Y., Chen, B., Xia, S.-T., Wang, X., & Xu, K. (2026). Closing the safety gap: Surgical concept erasure in visual autoregressive models. In The Fourteenth International Conference on Learning Representations.

Questions for authors
  1. What limitations of the adapted UCE motivate TRACE’s design? UCE is already evaluated on the same VAR backbone and requires no gradient-based training. What specific shortcomings motivate editing the text-projection weights and constructing scale-specific subspaces? Please explain how these choices address those shortcomings and which comparisons or ablations support this rationale.

  2. What evidence supports the semantic selectivity of the concept subspace? Equations (10)–(17) establish the minimum weight change for a fixed subspace. The effectiveness of selective erasure therefore depends on whether that subspace accurately captures the target concept. How does the construction account for non-target differences between paired prompts, such as changes in clothing, attributes, or scene details? What analysis or empirical evidence shows that these differences do not lead to unintended edits?

  3. Is unrelated content preserved within prompts containing the target concept? The COCO metrics and CLIP scores for other characters assess general generation quality and preservation on separate prompts. Do you have quantitative results showing whether other objects, actions, and backgrounds remain consistent with the requested content when the target concept is erased from the same prompt?

  4. How does erasure perform under indirect descriptions and adversarial prompts? Please clarify whether the evaluation includes prompts that invoke the target through its attributes without naming it, or prompts designed to recover the erased concept. Any available results, including comparisons with the main baselines, would help establish the scope of the reported erasure performance.

Reviewer_Y3ttOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TRACE (TRAining-Free Concept Erasure), a training-free concept erasure method for visual autoregressive (VAR) text-to-image models. TRACE builds concept subspaces from target-neutral prompt pairs in the output space of the text-projection layer, weights each pair by its concept-induced response at each generation scale, and applies a closed-form low-rank correction to the text-projection weights. The paper also proposes TRACE-Lite, which uses a single subspace shared across all scales.

Critical strengths
  1. The proposed method is simple and training-free: it edits a single linear layer in closed form, which makes erasure fast and adds no extra forward passes at inference.

  2. The experimental results demonstrate the effectiveness of the proposed method on NSFW, style, and copyright erasure, with stronger erasure than S-VARE and ScaleErasure while keeping COCO FID and CLIP close to the original model .

Critical weaknesses
  1. The evaluation is too narrow to verify that TRACE can be used for practical concept erasure:
    • Each task has only one target concept (nudity, Van Gogh, Pikachu), and only Infinity-2B is used, while ScaleErasure also tests Infinity-8B.
    • There is no evaluation on adversarial prompts or multi-concept erasure, which S-VARE already reports (e.g., Ring-A-Bell [1]).
    • (k,α)(k, \alpha) differ across tasks and seem to be selected on the test prompts (Figures 4 and 5), so it is unclear how to set them for a new concept without a test set. Results on more concepts, adversarial prompts, and hyperparameters selected on a separate validation set would make the method more convincing.
  2. The ablation in Table 2 may overstate the gain of the scale-specific design. All rows use unit normalization, and the first row (style Acc. 21.25%) is weaker than TRACE-Lite in Table 1 (17.50%), which does not use it. Scale-specific subspaces reduce NSFW ASR from 3.51% to 2.63% and style Acc. from 17.50% to 15.00%, from a single run. Adding TRACE-Lite to Table 2 and reporting variance over seeds would make this gain more convincing.

[1] Tsai, Yu-Lin, et al. "Ring-a-bell! how reliable are concept removal methods for diffusion models?." International Conference on Learning Representations. Vol. 2024. 2024.

Questions for authors
  1. Eq. 17 seems equivalent to applying (I−αPs)(I - \alpha P_s) to the projected text representations after removing boutb_{out}. What is the difference between TRACE and training-free embedding projection methods such as SAFREE [2]?
  2. In Figure 4, α=1\alpha = 1, which removes the concept component, still gives over 30% NSFW ASR, and the final setting α=2.0\alpha = 2.0 reverses it. Why is the reversal needed, and does it affect benign prompts related to the concept?
  3. The style test prompts are held-out pairs from the same LLM-generated pool as the calibration pairs (Lines 713-714). How does TRACE perform on style prompts written differently, e.g., artwork titles without the artist name?
  4. How many prompts are in the final I2P and style test sets after filtering (Lines 702-714)?

Minor:

  1. MM denotes both the number of prediction units (Eq. 5) and the number of generated images (Appendix A.2).
  2. The GPU used for the runtimes in Table 3 is not reported.

[2] Yoon, Jaehong, et al. "Safree: Training-free and adaptive guard for safe text-to-image and video generation." International Conference on Learning Representations. Vol. 2025. 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.