- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper address the problem of concept negation in text-to-image generation. The authors show that the T5 encoder successfully identifies negated prompts, but the observation from cross-attention visualization implies that the token-wise cross attention inevitably activates the negated tokens in the attention map. The authors manipulate the value vectors, by suppressing the negated concept in the embedding. Not only does the method suppress the explicitly mentioned negated token, but it also obtain embeddings from synonyms (ghost tokens) of the negated concept, and find the shared, concept direction to be suppressed. This suppression is also applied to context (non-negated) tokens, to ensure suppression leaking. The negated concepts are filtered out and synonyms are obtained with small language models, as the task is straightforward. The experimental results show that the method successfully suppresses negated concept without critical loss in overall generation quality.
Critical strengths
- The method is useful for practical scenarios, with effective and well-designed solution in suppressing negated concepts.
Critical weaknesses
-
Missing discussion and comparison against an extremely close work [1]. This work studies the exact same task, with a similar approach, which even uses SVD to suppress the negated tokens. However, it is not cited or discussed in the paper.
-
Task Definition and benchmarks: I think there are some controversial points.
- Sec.3: i) I am not sure if generating an unmentioned object is a critical problem. For instance, if a prompt is given to generate a road, I consider it is okay to generate trees in the background, but according to the definition here, this is considered wrong. I think it is only wrong to generate ones explicitly said not to generate. ii) I think the word "empty" could be considered a negating word, something like "nothing to be here", but in the examples of Fig.3, the authors consider it to be not a negation word. iii) I wonder how the example of 'A mailbox on an empty street with no houses around' is a 'attribute error', rather than an 'unmentioned object error', whereas 'A stop sign on a road without other signs' is an 'unmentioned object error'.
- Negation benchmark: I think the custom-built benchmark in this paper could be considered flawed. For instance, in the top row example of Fig.8, the prompt requires to generate a minion character, while excluding minion character. This is logically contradictory, making it nearly impossible to generate any reasonable, prompt-aligned image. This makes the definition of the task unclear.
- Image quality evaluation: MMD is a metric to quantify the distribution distance, rather than an explicit image quality metric. For image quality assessment, I suggest using a metric like HPSv3 [2], and see the before/after qualities of the images. If the authors intended to check the image distribution change before/after, it would have been more natural to use CMMD[3], which uses CLIP encoder space for MMD, or FID would have been enough. Although using an advanced metric with advanced encoders are ideal, it should always have a well-established, commonly used metric along with it, to ensure fairness.
-
Some important explanations seem to be missing. See Questions section.
[1] Li et al., Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models, ICLR 2024
[2] AI et al., HPSv3: Towards Wide-Spectrum Human Preference Score, ICCV 2025
[3] Jayasumana et al., Rethinking FID: Towards a Better Evaluation Metric for Image Generation, CVPR 2024
Questions for authors
-
How are the ghost tokens processed, as in order to obtain their embeddings, they would have to be fed to the language model as a prompt, with other texts. While the negated token is in the input prompt, so its embedding could be obtained while processing that prompt, ghost tokens are not in the prompt, so I wonder how it is acquired.
-
The explained method seems to be using a fixed text embedding, as in the ancestral Stable Diffusion Models. I wonder how the methods work on MM-DiT architectures, especially as their embeddings evolve through layers, and how it applies in joint attention.
-
Is the intervention applied in all layers, all timesteps?
-
What is the numbers reported in Table 1? Is it success rates?
-
Have the authors considered manipulating the attention maps instead of the value vectors? I get that manipulating the value vectors work, but I wonder how the authors got motivated to manipulate that, instead of the attention map, which was the fundamental problem of the baselines.
5 comments
(2) We sincerely thank the reviewer for raising this important concern. We would like to clarify the motivation and intended objective of this benchmark.
First, this setting is motivated by prior research on dependent concept negation. Dong et al. (NeurIPS 2023) explicitly investigate strongly correlated positive and negative concepts, using an example in which the prompt requests Mickey Mouse eating ice cream while excluding Disney characters. This illustrates a challenging setting where the desired and forbidden concepts are inherently related. Our IP benchmark follows this motivation to investigate whether a generator can suppress a strongly associated identity while preserving the remaining scene.
Second, our objective is not to satisfy both contradictory identity requirements. Instead, as formulated in Section 4, our method prioritizes the explicit exclusion constraint while preserving as much compatible contextual information as possible. For the Minion example, the desired output is an alternative character performing the requested action in the original setting, without reproducing the Minion identity. This provides a feasible solution to the constrained generation problem. Importantly, our evaluation follows this objective. Context preservation is evaluated after replacing the named identity with a generic figure, while negation prevention independently evaluates whether the forbidden identity appears. Therefore, an alternative character can satisfy both requirements by preserving the action and scene while excluding the Minion identity.
Finally, we intentionally include these highly correlated cases as stress tests of concept suppression, including scenarios involving conflicting user instructions or red-teaming. Such prompts can arise in practice, and a generation system should have a well-defined strategy for handling them rather than necessarily reproducing the forbidden concept. We will revise the benchmark description and Figure 8 to explicitly distinguish this constrained concept-exclusion setting from ordinary negation and clarify that its objective is to preserve compatible scene content under a prioritized exclusion constraint. We appreciate the reviewer's suggestion, which helps us clarify the scope of the benchmark. We also built an IP subset whose context does not name the forbidden entity (e.g., “a character”, not “a Minion”), which removes the logical conflict. Our method reduces NP from 32.08 to 0.00 on Infinity and from 28.30 to 11.32 on SD3. For more information, kindly redirect our reviewer to Nhwh Q6(1).
(3) The authors sincerely thank you for your comments. We apply the additional test using HPSv2.1 and CMMD on the spurious ImageNet dataset. We apologize that we were unable to complete the HPSv3 evaluation due to limited remaining storage space on our server. Instead, we adopted the widely cited HPSv2.1 for our experiments. HPSv2.1 for every image, scored against the prompt with the negated clause removed, so that a correctly absent object is not penalized. CMMD is computed for our images and vanilla images generated.
| Model | Comparison | HPSv2.1 ↑ | CMMD ↓ |
|---|---|---|---|
| Infinity | Vanilla | 0.279 | – |
| Infinity | Ours | 0.280 | 0.154 |
| SD3 | Vanilla | 0.253 | – |
| SD3 | Ours | 0.262 | 0.073 |
Q1 The small language model first extracts the negated concept and generates its corresponding Ghost tokens. These tokens are then appended to the parsed contextual prompt as a suppression suffix. For example, the augmented prompt becomes "A living room. [SUPPRESS: sofa, couch, settee]". The entire augmented sequence is processed together in a single forward pass through the frozen text encoder, which is to obtain the embeddings of both the context tokens and the appended negated and Ghost tokens. The Ghost embeddings are obtained directly from the augmented prompt, not through a separate encoding process.
At each cross-attention layer, we construct the suppression subspace by applying SVD to the value vectors of the negated and Ghost tokens. We then remove the suppression-aligned components from the selected context-token values and zero out the value vectors of the negated and Ghost tokens.
Q2 Thank you for raising this question. Our method does not rely on fixed text representations when applied to SD3's MM-DiT architecture. Instead, at each MM-DiT block and denoising step, we extract the negated and Ghost-token value vectors from the current text representations, reconstruct the suppression subspace using SVD, and adaptively select the contextual tokens for intervention.
For joint attention, we modify only the text value vectors contributing to the conditional image-query attention outputs, while leaving the image value vectors and the query-key attention computations unchanged. Specifically, we remove the suppression-aligned components from selected contextual values and zero out the negated and Ghost-token values. This procedure is applied across all 24 MM-DiT blocks and 28 denoising steps in SD3.
Q3 Yes. For Infinity, we apply it in all 40 blocks and all 13 scales. For SD3, in all 24 joint-attention blocks and all 28 denoising steps.
Q4 Yes, it is the success rates, the higher the better.
Q5 We thank the reviewer for this insightful question. We considered attention-map intervention during the formulating of our method. Our motivation originates from Figure 5, where the negated token's attention map correctly localizes the unwanted object, yet the model still generates it. This suggests that the failure can be unwanted semantic content propagated through the attention pathway.
Our key motivation is to preserve spatial correspondence while suppressing unwanted content. Directly modifying attention maps could disrupt the spatial allocation of attention and affect the desired image layout. In contrast, intervening on value vectors allows us to remove unwanted semantic components while preserving the original query-key attention patterns. This separates where the model attends from what content is transferred.
We also examined NASA attention subtraction. As illustrated in Figure 2, this intervention degraded image quality in our tested autoregressive setting, further motivating a more selective value-space intervention.
[1] Li et al., Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models, ICLR 2024
We hope our responses have adequately addressed your concerns. We would be happy to provide any further clarifications or answer additional questions. Thank you again for your valuable feedback and thoughtful consideration!
We sincerely thank the reviewer for the thoughtful feedback and for bringing closely related work to our attention. Your comments helped us better define the scope of our task and clarify how our approach differs from existing suppression methods. We address each concern below.
W1 We thank the reviewer for pointing out this highly relevant work. We will include a detailed discussion and comparison in the revised manuscript.
Li et al. [1] suppress unwanted concepts by modifying text embeddings. Specifically, they apply SVD to a matrix containing the negative-target and end-of-text embeddings, reduce its singular values, and further optimize the text embeddings using attention preservation and suppression losses.
Our method differs in the idea of correlated contextual information and intervention mechanism. Our method specifically targets settings where desired contextual content is strongly correlated with the negated concept. We identify context-token value vectors aligned with the negation subspace and selectively remove their unwanted components. This allows us to suppress negation-related information distributed across contextual tokens while preserving their remaining content. In contrast, Li et al. primarily suppress unwanted information through text-embedding regularization and subsequent optimization.
Furthermore, our method preserves the original queries and keys, avoiding direct modification of spatial attention affinities to reduce unintended changes to image layout. Our intervention also has a closed-form solution, no need for gradient-based embedding optimization.
To provide a direct empirical comparison, we additionally evaluate Li et al. [1] on the Spurious ImageNet benchmark under the same evaluation setting. The comparison results are:
Our method matches Li et al. on NP while preserving context better at less than half the runtime, without gradient-based optimization.
W2 Thank you for these insightful comments.
(1) Unmentioned objects. Our goal in Section 3.1 is to investigate potential causes of negation failures, based on the hypothesis that negated object generation can be related to hallucination. We agree that generating an unmentioned object does not necessarily constitute an error. For example, generating trees in a road scene is acceptable unless their absence is explicitly required. Our hamburger example (lines 135-136), where fries appear despite not being requested, illustrates the model's tendency to introduce contextually associated objects, which can be undesirable when strict adherence to the prompt is expected. Our experiments suggest that this type of unintended generation is relatively uncommon in modern models such as Infinity and SD3. We will clarify that unmentioned object generation is a potential source of unwanted content rather than a failure.
The meaning of "empty." We agree that "empty" can express an implicit absence constraint, even without explicit negation words such as "no" or "without." Our current evaluation primarily focuses on explicit linguistic negation. Our framework can accommodate such expressions by extending the first-stage parser instructions to recognize "empty" as an absence constraint and identify the corresponding unwanted concept. We will clarify this distinction and the scope of our evaluation.
We agree that the distinction between attribute errors and unmentioned-object errors is not sufficiently clear in Fig. 3. Since these failure types can overlap, we will revise the illustrations and refine their definitions to better distinguish errors involving object attributes from those involving unwanted object presence. Specifically, we will clarify the definitions in Sec. 3.1 to distinguish cases: unmentioned object, where the prompt does not explicitly exclude the object; attribute error, where a property of a requested object is rendered incorrectly. We have also recalculated the accuracy under the revised categorization and confirmed that this correction does not affect our main conclusion: both SD and Infinity still struggle to follow negation constraints reliably. We have updated our manuscript, thank you!
We appreciate these suggestions, which help us clarify the intended scope of our diagnostic analysis.