acceptodds
Under review as a conference paper at ICLR 2027

Training-Free Concept Negation in Text-to-Image Synthesis

Abstract

Text-to-image (T2I) models often struggle to follow negation prompts involving logical negation, raising serious concerns in critical scenarios. Existing negative prompting methods fail to overcome the strong correlations between contextual and negated concepts, while negation-aware fine-tuning is costly and difficult to generalize. To address negation failures, we revisit the text-image fusion mechanism in generative models and propose a training-free, context-aware null space projection method that suppresses negated concepts while preserving image layout and visual quality. Specifically, our method modifies the value vectors of cross-attention to directly control content synthesis via projecting contextual components onto the null space of negated semantics. We additionally introduce ghost token augmentation that expands the search space of negated concepts to further suppress negated content. Our approach operates as a plug-in module without additional training. Experiments on challenging generation tasks, including our new Negation Benchmark featuring complex prompts with mixed positive and negated concepts, demonstrate significant improvements over open-source baselines and competitive performance with Nano Banana Flash.

Primary area:
generative models
Keywords:
Negation, Image generation, Visual AutoRegressive

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_obaWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper address the problem of concept negation in text-to-image generation. The authors show that the T5 encoder successfully identifies negated prompts, but the observation from cross-attention visualization implies that the token-wise cross attention inevitably activates the negated tokens in the attention map. The authors manipulate the value vectors, by suppressing the negated concept in the embedding. Not only does the method suppress the explicitly mentioned negated token, but it also obtain embeddings from synonyms (ghost tokens) of the negated concept, and find the shared, concept direction to be suppressed. This suppression is also applied to context (non-negated) tokens, to ensure suppression leaking. The negated concepts are filtered out and synonyms are obtained with small language models, as the task is straightforward. The experimental results show that the method successfully suppresses negated concept without critical loss in overall generation quality.

Critical strengths
  • The method is useful for practical scenarios, with effective and well-designed solution in suppressing negated concepts.
Critical weaknesses
  • Missing discussion and comparison against an extremely close work [1]. This work studies the exact same task, with a similar approach, which even uses SVD to suppress the negated tokens. However, it is not cited or discussed in the paper.

  • Task Definition and benchmarks: I think there are some controversial points.

    • Sec.3: i) I am not sure if generating an unmentioned object is a critical problem. For instance, if a prompt is given to generate a road, I consider it is okay to generate trees in the background, but according to the definition here, this is considered wrong. I think it is only wrong to generate ones explicitly said not to generate. ii) I think the word "empty" could be considered a negating word, something like "nothing to be here", but in the examples of Fig.3, the authors consider it to be not a negation word. iii) I wonder how the example of 'A mailbox on an empty street with no houses around' is a 'attribute error', rather than an 'unmentioned object error', whereas 'A stop sign on a road without other signs' is an 'unmentioned object error'.
    • Negation benchmark: I think the custom-built benchmark in this paper could be considered flawed. For instance, in the top row example of Fig.8, the prompt requires to generate a minion character, while excluding minion character. This is logically contradictory, making it nearly impossible to generate any reasonable, prompt-aligned image. This makes the definition of the task unclear.
    • Image quality evaluation: MMD is a metric to quantify the distribution distance, rather than an explicit image quality metric. For image quality assessment, I suggest using a metric like HPSv3 [2], and see the before/after qualities of the images. If the authors intended to check the image distribution change before/after, it would have been more natural to use CMMD[3], which uses CLIP encoder space for MMD, or FID would have been enough. Although using an advanced metric with advanced encoders are ideal, it should always have a well-established, commonly used metric along with it, to ensure fairness.
  • Some important explanations seem to be missing. See Questions section.

[1] Li et al., Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models, ICLR 2024

[2] AI et al., HPSv3: Towards Wide-Spectrum Human Preference Score, ICCV 2025

[3] Jayasumana et al., Rethinking FID: Towards a Better Evaluation Metric for Image Generation, CVPR 2024

Questions for authors
  • How are the ghost tokens processed, as in order to obtain their embeddings, they would have to be fed to the language model as a prompt, with other texts. While the negated token is in the input prompt, so its embedding could be obtained while processing that prompt, ghost tokens are not in the prompt, so I wonder how it is acquired.

  • The explained method seems to be using a fixed text embedding, as in the ancestral Stable Diffusion Models. I wonder how the methods work on MM-DiT architectures, especially as their embeddings evolve through layers, and how it applies in joint attention.

  • Is the intervention applied in all layers, all timesteps?

  • What is the numbers reported in Table 1? Is it success rates?

  • Have the authors considered manipulating the attention maps instead of the value vectors? I get that manipulating the value vectors work, but I wonder how the authors got motivated to manipulate that, instead of the attention map, which was the fundamental problem of the baselines.

5 comments

We sincerely thank the reviewer for the thoughtful feedback and for bringing closely related work to our attention. Your comments helped us better define the scope of our task and clarify how our approach differs from existing suppression methods. We address each concern below.

W1 We thank the reviewer for pointing out this highly relevant work. We will include a detailed discussion and comparison in the revised manuscript.

Li et al. [1] suppress unwanted concepts by modifying text embeddings. Specifically, they apply SVD to a matrix containing the negative-target and end-of-text embeddings, reduce its singular values, and further optimize the text embeddings using attention preservation and suppression losses.

Our method differs in the idea of correlated contextual information and intervention mechanism. Our method specifically targets settings where desired contextual content is strongly correlated with the negated concept. We identify context-token value vectors aligned with the negation subspace and selectively remove their unwanted components. This allows us to suppress negation-related information distributed across contextual tokens while preserving their remaining content. In contrast, Li et al. primarily suppress unwanted information through text-embedding regularization and subsequent optimization.

Furthermore, our method preserves the original queries and keys, avoiding direct modification of spatial attention affinities to reduce unintended changes to image layout. Our intervention also has a closed-form solution, no need for gradient-based embedding optimization.

To provide a direct empirical comparison, we additionally evaluate Li et al. [1] on the Spurious ImageNet benchmark under the same evaluation setting. The comparison results are:

Method (SD v1.4)CP ↑NP ↓Time / image
Vanilla90288.9 s
Li et al. (official code)842237.3 s
Ours902116.8 s

Our method matches Li et al. on NP while preserving context better at less than half the runtime, without gradient-based optimization.

W2 Thank you for these insightful comments.

(1) Unmentioned objects. Our goal in Section 3.1 is to investigate potential causes of negation failures, based on the hypothesis that negated object generation can be related to hallucination. We agree that generating an unmentioned object does not necessarily constitute an error. For example, generating trees in a road scene is acceptable unless their absence is explicitly required. Our hamburger example (lines 135-136), where fries appear despite not being requested, illustrates the model's tendency to introduce contextually associated objects, which can be undesirable when strict adherence to the prompt is expected. Our experiments suggest that this type of unintended generation is relatively uncommon in modern models such as Infinity and SD3. We will clarify that unmentioned object generation is a potential source of unwanted content rather than a failure.

The meaning of "empty." We agree that "empty" can express an implicit absence constraint, even without explicit negation words such as "no" or "without." Our current evaluation primarily focuses on explicit linguistic negation. Our framework can accommodate such expressions by extending the first-stage parser instructions to recognize "empty" as an absence constraint and identify the corresponding unwanted concept. We will clarify this distinction and the scope of our evaluation.

We agree that the distinction between attribute errors and unmentioned-object errors is not sufficiently clear in Fig. 3. Since these failure types can overlap, we will revise the illustrations and refine their definitions to better distinguish errors involving object attributes from those involving unwanted object presence. Specifically, we will clarify the definitions in Sec. 3.1 to distinguish cases: unmentioned object, where the prompt does not explicitly exclude the object; attribute error, where a property of a requested object is rendered incorrectly. We have also recalculated the accuracy under the revised categorization and confirmed that this correction does not affect our main conclusion: both SD and Infinity still struggle to follow negation constraints reliably. We have updated our manuscript, thank you!

We appreciate these suggestions, which help us clarify the intended scope of our diagnostic analysis.

(2) We sincerely thank the reviewer for raising this important concern. We would like to clarify the motivation and intended objective of this benchmark.

First, this setting is motivated by prior research on dependent concept negation. Dong et al. (NeurIPS 2023) explicitly investigate strongly correlated positive and negative concepts, using an example in which the prompt requests Mickey Mouse eating ice cream while excluding Disney characters. This illustrates a challenging setting where the desired and forbidden concepts are inherently related. Our IP benchmark follows this motivation to investigate whether a generator can suppress a strongly associated identity while preserving the remaining scene.

Second, our objective is not to satisfy both contradictory identity requirements. Instead, as formulated in Section 4, our method prioritizes the explicit exclusion constraint while preserving as much compatible contextual information as possible. For the Minion example, the desired output is an alternative character performing the requested action in the original setting, without reproducing the Minion identity. This provides a feasible solution to the constrained generation problem. Importantly, our evaluation follows this objective. Context preservation is evaluated after replacing the named identity with a generic figure, while negation prevention independently evaluates whether the forbidden identity appears. Therefore, an alternative character can satisfy both requirements by preserving the action and scene while excluding the Minion identity.

Finally, we intentionally include these highly correlated cases as stress tests of concept suppression, including scenarios involving conflicting user instructions or red-teaming. Such prompts can arise in practice, and a generation system should have a well-defined strategy for handling them rather than necessarily reproducing the forbidden concept. We will revise the benchmark description and Figure 8 to explicitly distinguish this constrained concept-exclusion setting from ordinary negation and clarify that its objective is to preserve compatible scene content under a prioritized exclusion constraint. We appreciate the reviewer's suggestion, which helps us clarify the scope of the benchmark. We also built an IP subset whose context does not name the forbidden entity (e.g., “a character”, not “a Minion”), which removes the logical conflict. Our method reduces NP from 32.08 to 0.00 on Infinity and from 28.30 to 11.32 on SD3. For more information, kindly redirect our reviewer to Nhwh Q6(1).

(3) The authors sincerely thank you for your comments. We apply the additional test using HPSv2.1 and CMMD on the spurious ImageNet dataset. We apologize that we were unable to complete the HPSv3 evaluation due to limited remaining storage space on our server. Instead, we adopted the widely cited HPSv2.1 for our experiments. HPSv2.1 for every image, scored against the prompt with the negated clause removed, so that a correctly absent object is not penalized. CMMD is computed for our images and vanilla images generated.

ModelComparisonHPSv2.1 ↑CMMD ↓
InfinityVanilla0.279–
InfinityOurs0.2800.154
SD3Vanilla0.253–
SD3Ours0.2620.073

Q1 The small language model first extracts the negated concept and generates its corresponding Ghost tokens. These tokens are then appended to the parsed contextual prompt as a suppression suffix. For example, the augmented prompt becomes "A living room. [SUPPRESS: sofa, couch, settee]". The entire augmented sequence is processed together in a single forward pass through the frozen text encoder, which is to obtain the embeddings of both the context tokens and the appended negated and Ghost tokens. The Ghost embeddings are obtained directly from the augmented prompt, not through a separate encoding process.

At each cross-attention layer, we construct the suppression subspace by applying SVD to the value vectors of the negated and Ghost tokens. We then remove the suppression-aligned components from the selected context-token values and zero out the value vectors of the negated and Ghost tokens.

Q2 Thank you for raising this question. Our method does not rely on fixed text representations when applied to SD3's MM-DiT architecture. Instead, at each MM-DiT block and denoising step, we extract the negated and Ghost-token value vectors from the current text representations, reconstruct the suppression subspace using SVD, and adaptively select the contextual tokens for intervention.

For joint attention, we modify only the text value vectors contributing to the conditional image-query attention outputs, while leaving the image value vectors and the query-key attention computations unchanged. Specifically, we remove the suppression-aligned components from selected contextual values and zero out the negated and Ghost-token values. This procedure is applied across all 24 MM-DiT blocks and 28 denoising steps in SD3.

Q3 Yes. For Infinity, we apply it in all 40 blocks and all 13 scales. For SD3, in all 24 joint-attention blocks and all 28 denoising steps.

Q4 Yes, it is the success rates, the higher the better.

Q5 We thank the reviewer for this insightful question. We considered attention-map intervention during the formulating of our method. Our motivation originates from Figure 5, where the negated token's attention map correctly localizes the unwanted object, yet the model still generates it. This suggests that the failure can be unwanted semantic content propagated through the attention pathway.

Our key motivation is to preserve spatial correspondence while suppressing unwanted content. Directly modifying attention maps could disrupt the spatial allocation of attention and affect the desired image layout. In contrast, intervening on value vectors allows us to remove unwanted semantic components while preserving the original query-key attention patterns. This separates where the model attends from what content is transferred.

We also examined NASA attention subtraction. As illustrated in Figure 2, this intervention degraded image quality in our tested autoregressive setting, further motivating a more selective value-space intervention.

[1] Li et al., Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models, ICLR 2024

We hope our responses have adequately addressed your concerns. We would be happy to provide any further clarifications or answer additional questions. Thank you again for your valuable feedback and thoughtful consideration!

Reviewer_ouULOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses negation failures in text-to-image models: a prompt such as "a living room without a sofa" can still produce a sofa when the negated object is strongly tied to the scene. The proposed fix is training-free. LLaMA-3.2-3B parses the prompt into context and negated concepts and generates synonyms of the negated concept ("Ghost tokens"). The value vectors of the negated and Ghost tokens span a negation subspace. Affinity to this set decides how many context tokens to edit, subspace energy decides which ones (Eqs. 5-7), and their subspace component is then suppressed with strength λ (Eq. 8). Experiments on Infinity-8B and SD3 use a new Negation Benchmark (53 IP, 65 Safety, 52 General prompts), Spurious ImageNet, I2P and an artistic-style set, with ablations, judge validation, timing and failure cases in the appendices.

Critical strengths
  1. Large gains on two different backbones, not only on the authors' IP prompts. In Table 3, the average Δ rises from 22.69 to 61.72 on SD3 and from 23.70 to 72.38 on Infinity, at some cost in context preservation on Infinity (88.05 to 84.34). I give the most weight to Table 4, because Spurious ImageNet does not depend on the IP subset I question below. There, NP falls from 55.50 to 24.70 on SD3 with CP essentially unchanged (95.70 vs. 95.80), and from 63.40 to 35.40 on Infinity with CP slightly higher (90.60 to 92.30). These results show a substantial reduction in unwanted content while preserving the scene.
  2. A practical design whose main added component is supported by an ablation. Building the negation subspace from the current prompt's value vectors and editing only selected context tokens gives plug-in control without retraining, on both a diffusion and an autoregressive model. In Table 9, adding Ghost tokens lowers NP from 42.0 to 30.0 and raises Δ from 52.0 to 63.0, while CP only moves from 94.0 to 93.0. The human checks of the judges (Appendix F) and of the parser (Table 2) also make the headline numbers easier to trust.
Critical weaknesses
  1. The IP evaluation needs a clearer definition, since it drives the main comparison with Nano Banana. Both IP examples in Appendix H name the character in positive_class ("Donald Duck gestures angrily in a messy cartoon kitchen.") and negate it in negative_class ("Disney / Donald Duck"). The IP scene template in Appendix F checks a "{scene_description}", but the paper does not show which string fills it. Replacing a named character while keeping the action and setting is a sensible task (§5.1 borrows such prompts from Dong et al., 2023). If scene_description is filled directly from positive_class, the CP and NP checks would conflict. I could not find an explanation of how the displayed labels are mapped to the judge inputs. Ours-AR leads Nano Banana by 69.26 points on ΔIP but trails it on ΔSF and ΔGE (by 9.70 and 12.89), so its macro-average lead in Table 3 comes from IP. The quality evidence has a similar gap: HCS (§5.4) asks which of two similarity-matched images came from the baseline. That is a source-identification test, not direct evidence of equal quality, and no trial counts or uncertainty are reported.
  2. The new components are not isolated, and key implementation details are missing. SAFREE already pairs adaptive token selection with projection away from a concept subspace, and GLoCE uses gated closed-form low-rank edits (both compared in Table 5). The combination here is distinct, but the experiments do not show which part drives the gains: the value-space edit, the Ghost construction, the two-stage selection or the AR adaptation. Table 9 shows the Ghost module helps as a whole, not whether that comes from a larger subspace, different token selection or the extra tokens themselves. Table 5 also needs more detail on the backbone, concept inputs and baseline tuning before the comparison can be assessed. I could not identify the default λ, rank cutoff, number of Ghost tokens, edited layers/heads/steps or the SD3 joint-attention hook in the paper. The Spurious ImageNet gains stand on their own, but both issues touch the central claims, so I lean toward weak reject for now. I would move to weak accept given the actual IP judge inputs and results under an explicit definition, a reconstructable configuration for both backbones, a small matched-input control isolating the value-space and Ghost contributions, and a quality claim scoped to what HCS measures.
Questions for authors
  1. Clarification. For the IP items in Appendix H, what exact strings go to the CP and NP judges, and how are potentially conflicting positive and negative requirements resolved? Do the IP results hold under that definition? Per-image joint success (context kept, negated concept absent) would also help.
  2. Clarification. Which λ is used in the main tables (only λ=1 removes the subspace component exactly)? Please also give the rank cutoff, number of Ghost tokens, edited layers/heads/steps, the handling of Ghost and original negation tokens in K, V and the attention mask, and which text values are edited in SD3, or point to them in the supplement.
  3. Targeted evidence. With backbone, parser output and Ghost set fixed, could you report λ=0 (all else unchanged), λ=1 and the default λ, and compare value-space projection with projecting the input embeddings or editing all candidate tokens?
  4. Clarification. What does ± mean in Tables 3-4, and how many images and seeds per prompt? Which backbone does Table 5 use, and did SAFREE and GLoCE get the same concept inputs and comparable tuning? How does the setting of Tables 7-9 (e.g., 93.0/30.0) differ from Ours-AR in Table 4 (92.30/35.40)?
  5. Clarification. How many HCS trials were run, from what candidate pool, and with what uncertainty? If a blind quality rating on unselected samples is not feasible, please narrow the quality claim.
5 comments

We sincerely thank the reviewer for the careful assessment and constructive suggestions. We appreciate your recognition of the results on Spurious ImageNet. Your questions about the IP benchmark and ablations helped us clarify the evaluation and examine our design choices more closely. We address each point below.

W1 The reviewer raises concerns about the justification for the IP subset, the evaluation procedure, and our discussion of image quality. Our reviewer breaks down this point in the question section. We kindly refer the reviewer to our responses to Q1 and Q5.

W2 We address this weakness in three parts.

(1) New components isolation: Table 9 isolates the Ghost contribution with the backbone, where we can see that with Ghost token augmentation, the Context Preservation is maintaining and the negation prevention show better performance. For more components analysis results, please see our response to Q3.

(2) Difference from SAFREE and GLoCE.

Our method differs from SAFREE in two key aspects, which are important for effective negation suppression and preserving desired content.

Intervention space: SAFREE modifies text embeddings before image generation. Our method directly intervenes on cross-attention value vectors. This distinction is important because unwanted concepts can still be introduced through contextual information during generation. Our method directly suppresses unwanted information entering the image generation process.

Token identification and intervention: SAFREE identifies influential tokens based on changes in the average prompt representation when individual tokens are excluded. However, this global representation does not directly reflect how individual tokens contribute unwanted content through cross-attention. In contrast, our method identifies context tokens whose value vectors strongly align with the negation subspace and selectively removes their unwanted components. Negated concepts can also be represented in other context tokens, even after the negated token itself is suppressed.

Our method also differs from GLoCE in two aspects:

GLoCE generates reference samples and extracts their hidden representations to construct concept-specific low-rank modules. Our method modifies value vectors on the fly during generation, without generating reference images or requiring this separate preparation stage.

Context-dependent suppression. GLoCE uses a gate to preserve non-target content, but its concept representations are estimated from generated reference samples. Our concern is that these samples can mix the target concept with recurring context. For example, if reference images of “sofa” repeatedly include living-room features, the estimated subspace could capture both the sofa and its surroundings. Suppressing these shared components risks removing desired living-room content along with the sofa. Our method instead identifies and suppresses unwanted components within the current prompt’s contextual value vectors, directly addressing context-dependent negation.

(3) Table 5 and experimental details. Please see our response to Q4 for details on the backbone, concept inputs, hyperparameters, and experimental settings.

Q1 For the IP evaluation, context preservation (CP) does not require preserving the named character's identity. We evaluate each generated image using two separate Gemini calls. For CP, we replace the named character in positive_class with "a figure," while preserving the original action, scene, and setting. For negation prevention (NP), we evaluate the presence of the negation identity specified by negative_class.

For the example in Appendix H, the instantiated judge inputs are:

CP ({scene_description}): "A figure gestures angrily in a messy cartoon kitchen."

NP ({char_name}): "Donald Duck."

Thus, CP evaluates whether the intended scene and actions are preserved, NP separately checks whether the forbidden character appears. The two requirements are not conflicting.

Ji=1[CPi=Yes∧NPi=No].J_i=\mathbf{1}[\mathrm{CP}_i=\mathrm{Yes}\land\mathrm{NP}_i=\mathrm{No}].

We average JiJ_i across images, obtaining a joint success rate as shown in the following table.

MethodIPGeneralSafety
Infinity14.1514.2360.92
SD320.9417.8850.46
NASA-I23.2130.1952.46
NAG19.4323.8556.46
Flux-K (SD images)28.6843.8552.00
Flux-K (Infinity images)23.0232.7661.69
Nano Banana Flash20.8870.1183.87
Ours (SD3)61.5167.8862.00
Ours (Infinity)86.5557.5875.38

Under joint success, our method leads on IP and is second to Nano Banana Flash on General and Safety. Nano Banana is a closed commercial system with strong edge on its private data, while our method is training-free on open backbones.

Q2 We use the hyperparameter settings and attention intervention below.

Table: λ\lambda used in the main tables.

DatasetInfinitySD3
IP1.251
Safety23
General22
Spurious ImageNet23
I2P23

For the selected context values, λ=1\lambda=1 removes their projection onto the suppression subspace, λ>1\lambda>1 reverses the corresponding component. As shown in Figure 10, this stronger intervention can improve negation suppression empirically.

Rank and Ghost-token counts. The suppression rank rr is determined adaptively from the suppression-token matrix, with the default tolerance of torch.linalg.matrix_rank. The observed ranks range from 1 to 36 for Infinity and 2 to 31 for SD3. The number of Ghost tokens is also prompt-dependent: 2–15 for IP, 1–13 for Safety, 1–13 for General, and 1–24 for Spurious ImageNet.

Edited layers, heads, and generation steps. For Infinity, we apply the intervention in the cross-attention modules of all 40 transformer blocks, across 28 attention heads and 13 autoregressive scales. For SD3, we apply it to all 24 joint-attention blocks, across 24 heads and 28 denoising steps.

Negated and Ghost tokens. We move the negation concept and Ghost tokens in a [SUPPRESS: ...] suffix as shown in the Figure 6 (2). At each affected attention layer, we construct the suppression basis from their value vectors and remove the suppression-aligned components from the selected context-token values. We then zero the value vectors of the negated token and its Ghost tokens. Their key vectors remain unchanged. Queries, and keys are not modified, and no additional attention masking is used. These tokens preserves spatial attention patterns but contribute no content through their zeroed values. The attention computation uses the modified values at each generation scale or denoising step.

SD3 intervention. In SD3's joint attention, we modify only the conditional image-query attention contribution from text context values. Image-side values, queries, keys, the attention mask, and the unconditional branch remain as in the original model.

Which SD3 text streams are edited? SD3 Medium has the transformer with a single text sequence of 333 positions: 77 CLIP positions, with the CLIP-L and CLIP-G hidden states of the same token concatenated along the feature dimension, followed by 256 T5 positions. Our method edits both segments.

We locate the negated words using character offsets for each of the three tokenizers. Thus, SS includes their positions in both the CLIP and T5 segments. Context candidates are selected from both segments using the same procedure. In each block, the text value projection maps all 333 positions into the same 1536-dimensional value space. We construct a single subspace from the value vectors at all positions in SS, combining CLIP and T5, and apply candidate selection and projection to both segments. We do not edit the pooled CLIP embedding.

Q3 We provide the ablation study of λ\lambda in Appendix B, Figure 11, which examines the effect of intervention strength on negation performance. In particular, λ=1\lambda=1 removes the negation aligned component, and λ>1\lambda>1 applies a stronger intervention.

Following the reviewer's suggestion, we further conduct controlled ablations on Spurious ImageNet, keeping the backbone, parser output, and Ghost-token set fixed. Specifically, we compare (1) input-embedding projection instead of cross-attention value-space projection and (2) editing all candidate context tokens without the selection rule. These comparisons isolate the contributions of the intervention space and selective token editing.

We ran all variants below on Spurious ImageNet with the same backbone (Infinity-8B), the same parser output and seed 256, one image per prompt, the results are following:

VariantCP ↑NP ↓
Vanilla (original prompt)9163
λ=0\lambda = 0 Zero Suppression set only, no context projection8648
λ=1\lambda = 19840
Project T5 text embeddings9438
Edit all candidates (no selection rule)9135
Ours λ=2\lambda = 29335
Negated concept + the same number of random nouns λ=2.5\lambda = 2.59332
Ours λ=2.5\lambda = 2.59330

We can observe that zeroing the suppression set alone λ=0\lambda = 0 reduces NP, while λ=1\lambda = 1 reduces NP further to 40 and improves CP. Further reductions in NP are observed at λ≥2\lambda \geq 2, with a modest decrease in CP. Value-space projection outperforms projection in the T5 embedding space. Our selection also achieves an improvement in CP compared with editing all candidates, while modifying fewer tokens. We also replaced the Ghost tokens with random nouns to isolate the effects of the Ghost tokens. We can clearly observe that our full method achieves the best NP performance. This further validates the effectiveness of our method's design.

Q4 (1) The reported values represent the mean ± standard deviation across 10 random seeds per prompt.

(2) We use the official implementations of SAFREE and GLoCE. For a fair comparison, SAFREE and GLoCE receive the same input prompts and negated concepts identified by the same parser as our method. We tuned each baseline’s hyperparameters on the validation split of each benchmark, selecting the setting with the max CP−NPCP-NP.

(3) Differences between Tables 7–9 and Table 4. Tables 7–9 report ablation results using a single fixed random seed =256, λ=2.5\lambda = 2.5, Table 4 reports results averaged across 10 random seeds per prompt. The ablation studies use a fixed seed to isolate the effects of individual components. We have clarified these evaluation settings in the revised manuscript.

Q5 As described in Section 5.4 (Lines 458–463), the Human Confusion Study (HCS) was conducted on Spurious ImageNet, where participants were not informed of the negated concepts in the prompts. The study involved two researchers who evaluated 100 image pairs, selected from the original and ours generations based on visual similarity. For each pair, participants were asked to identify which image was generated without intervention.

To quantify uncertainty, we reanalyzed the researchers' raw evaluation records and computed the 95% Wilson confidence interval. Across 200 trials with two raters and 100 pairs, the source was correctly identified in 113 trials, corresponding to 56.5%, with a 95% Wilson confidence interval of 49.6%–63.2%. This interval includes the 50% chance level, indicating that the study does not have above-chance identification performance. We agree that HCS evaluates source distinguishability rather than directly measuring perceptual image quality. We narrow our claim to state that the intervention does not produce a clearly distinguishable change in the evaluated similarity-matched image pairs. We also report MMD as evidence of distributional preservation between modified and unmodified generations.

We hope our responses address your concerns. We would be more than happy to answer any further questions you may have. Thank you once again for your thoughtful feedback!

Reviewer_NhwhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the well-known failure of text-to-image models to correctly follow negation prompts. They propose a training-free method which intervene on cross-attention value vectors. a small language model parses the prompt into contextual and negated concepts and generates synonyms which are referred as ghost tokens. The value vectors of the negated concept and its ghost tokens are decomposed by SVD into a negation subspace. Context tokens are scored by a temperature-scaled softmax to that subspace, and the top-k tokens by subspace energy have their negation component removed with strength λ. Along the method, they also proposed Negation Benchmark to evaluate models for negation prompt evaluation task.

Critical strengths
  1. The problem statement is well motivation. Negation failure is a practical weakness of current T2I generation models.

  2. Most negative guidance methods work for diffusion with CFG. The proposed method is quite lightweight approach in terms of overhead, espacially in case of autoregressive models.

  3. The Spurious ImageNet evaluation is a good choice to stress correlated concept pairs.

  4. PaliGemma2 is calibrated against 1,000 human-reviewed samples, and Gemini's agreement with human labels is reported per subset. The supplement contains code and the benchmark files.

Critical weaknesses
  1. If I understand correctly, the negated concept is identified by an external LLM that parses the sentence. The generator never has to understand negation as it receives a list of concepts to suppress, which makes the proposed method an engineered solution. The authors should state it as a concept suppression given a known concept rather than understanding negation.

  2. In Section 4.2, the edited set H_aff is drawn from the context tokens C. The ghost tokens are zeroed, but the value vector of the negated token is not modified at any point. Figure 5 shows that this token's attention map localises the object, so the formulation leaves the main path that injects the unwanted concept intact. but it is unclear to me that how the method as is working then. Maybe, there is some other step involved which I am missing here, either the negated token is removed from the prompt, or the description is incomplete.

  3. I will continue my above point here. with one negated word and a handful of synonyms, the subspace has rank at most |N| + g in a value space of dimension d. It is not obvious why removing a few directions from a few context tokens should remove a visual concept that the generator can also infer from context. The selection rule edits every candidate whose affinity exceeds the mean affinity, so roughly half the candidates are always modified, even when no context token is related to the negated concept. A no-negation control would show what this costs.

  4. The problem statement is good but simply showing t-SNE separation of 80 pairs that differ only by "no" or "without" shows that the encoder registers the presence of a negation word, not that it encodes its scope. The latter is what generation needs. The text contrasts T5 with CLIP, but no CLIP result is shown. "[SOS] pooling" is also not a standard T5 operation and should be explained.

  5. On a lighter note, Image-editing baselines are dismissed for requiring "... nearly double the computational resources." but the proposed pipeline takes 20.69 seconds against 10.25 seconds for vanilla Infinity, which is the same factor of about two.

Questions for authors
  1. In the method as described, the negated token's value is never edited. How is its contribution suppressed? Is the negated phrase removed from the prompt before encoding?
  2. What do you gain over (a) LLM rewriting alone and (b) rewriting plus a standard CFG negative prompt, on both Infinity and SD3?
  3. Why is λ > 1 needed if Equation 3 is the intended solution, and what λ, r and k produced the main tables?
  4. How does your method differ conceptually from SAFREE's token-subspace projection?
  5. How do you reconcile Infinity's 0.00% negation success in Table 1 with Tables 3 and 4?
  6. How does the method perform on an IP subset whose context does not name the forbidden entity, and on standard non-negated prompt benchmarks?
  7. How is the method defined for SD3's joint-attention architecture? Which layers, heads and timesteps are edited? I couldn't find this information in the paper.
8 comments

We sincerely thank the reviewer for the careful reading and thoughtful questions! Your comments helped us clarify how our method handles negated tokens and how it compares with prompt-based alternatives. Below, we address your concerns and questions.

W1 We thank the reviewer for these thoughtful comments. We agree that the image generator itself does not interpret negation independently in our framework. Our method uses a small language model to parse the input prompt and identify the tokens related to negated concepts. We then apply a closed-form update to guide T2I generation to satisfy the negation constraint. For the generator itself to understand negation, one would fine-tune on negation-aware data. However, collecting such data is challenging because conventional image captions typically describe what is present and rarely specify what is absent.

Our design is motivated by the difficulty T2I models can have in interpreting some logical constraints, such as negation. We assign the interpretation of negation to a language model and use the analysis to guide image generation. Our contribution addresses contextual correlations that make suppression difficult. Unwanted concept information can remain in contextual token values even after the negated token’s direct contribution is removed. We suppress both the direct contribution of the negated tokens and the suppression-aligned components of selected contextual values.

W2 Thank you for the clarification question, we will make the procedure and pipeline even clearer in the revision. Figure 5 shows that the model can localize the unwanted object but still generate it. This motivates our approach of suppressing the content carried by the value vectors while preserving spatial layout.

In terms of implementation, as illustrated in Step (2) of Figure 6, the negated token (TV) is included alongside the ghost tokens in the suppression-token set. After constructing the suppression subspace, the value vectors of all suppression tokens, including TV, are zeroed. Therefore, although TV's key can still produce spatially localized attention, its zeroed value contributes no content through the attention pathway. We will discuss this in the revision.

W3 We would like to clarify that the mean-affinity threshold does not select half the candidates, the selected fraction depends on how the affinities score distributed. Across 100 Spurious ImageNet prompts and all 40 cross-attention layers of Infinity-8B, the rule selected 30.7% of candidate tokens on average. On SD3 it selected 31% tokens.

Our method identifies the overlap between the negation subspace and context-token representations, and intervenes on tokens with the strongest projections onto this subspace. The assumption is that even a few directions can be effective if substantial concept-related information is concentrated in them. Table 9 provides empirical support, showing that ghost augmentation reduces unwanted object generation from 42% to 30%.

To further examine this assumption, we analyzed 100 prompts from Spurious ImageNet using Infinity-8B across all 40 cross-attention layers. For each prompt and layer, we constructed the suppression matrix from the corresponding value vectors and applied SVD to get the subspace basis UlU_l. We measured each context token’s projected magnitude and the share captured by the selected tokens:

ei=∥Ul⊤vi∥22e_i = \|U_l^\top v_i\|^2_2,

R=∑i∈Haffei∑i∈CeiR = \frac{\sum_{i \in \mathcal{H}_{\mathrm{aff}}} e _ i }{ \sum _ {i \in \mathcal{C}} e _ i}

The selected tokens (30.7%, ∣Haff∣∣C∣\frac{|\mathrm{H} _ {aff}|}{|\mathcal{C}|} of candidates) carry 69.9% of the projected energy. Because Haff\mathrm{H}_{aff} is chosen by energy, this shown that negation-aligned information is concentrated in a few context tokens. For reference, a random subset of the same size carries 29.9%.

Even if small, nonzero affinities can trigger interventions on unrelated context tokens, the resulting changes remain small when these tokens have small projections onto the suppression subspace. As indicated by Eq. (8), a small projected component leads to only a small update to the corresponding value vector.

Regarding the cost of edits when no context token is related: Q6(3) evaluates negations of objects unrelated to the scene (e.g., "a tench without a giraffe"). CP improved on Infinity and SD3, indicating that edits to weakly aligned tokens do not harm the scene. Q6(2) further shows the pipeline is inactive on non-negated prompts (GenEval).

W4 We use T5 because it is the text encoder used in Infinity’s original implementation. T5 outputs a sequence of token representations, which Infinity normalizes and processes with its own TextAttentivePool module. The term “[SOS] pooling” refers to this operation within Infinity. We will clarify this terminology.

The t-SNE visualization shows a clear separation between the representations of prompts and their negated counterparts. To test negation scope rather than the mere presence of a negation word, we built 80 pairs that use the same words in a different order: “an image with A but without B” and “an image with B but without A”. The t-SNE plots are in revised Appendix. The two sentences contain the same tokens but have different meanings. T5 separates the two orderings, which motivates us a treatment beyond prompt tuning.

Our discussion of CLIP’s limitations is based on prior studies reporting difficulties in handling negation [1,2]. We also provide t-SNE visualizations for the CLIP ViT-L/14 text encoder, covering both the original negation counterparts (“with A” vs. “without A”) and the minimal pairs containing identical tokens but different negation scopes (“with A without B” vs. “with B without A”). We can clearly observe that neither experiment shows a clear separation between the two groups, which is consistent with existing literature. The premise of Section 3 holds in a more precise way: the text encoder does register which object is negated, at the sentence level and in the token states that the generator attends to, yet the generator still draws the object (Fig. 5). This is why we intervene on the cross-attention values of these tokens.

[1] Alhamoud et al., “Vision-Language Models Do Not Understand Negation,” CVPR 2025.

[2] Park et al., “Know ‘No’ Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIP,” ICCV 2025.

W5 Thank you for pointing this out! We agree. Our reported runtime increases from 10.25 to 20.69 seconds. We will remove the statement dismissing image-editing methods for nearly doubling computational cost. Our method performs one-pass generation, whereas the performance of image editing methods can differ depending on the source image. We will revise our claim to clarify this distinction.

Q1 Yes. Our method uses two mechanisms to prevent the negation concept from generated: (1) We move the negated tokens and ghost tokens in to the suppression lists (2) We suppress its contribution by zeroing the value vectors of both the negated token and its corresponding ghost tokens in the generation.

Specifically, at each cross-attention layer, we compute the token keys and values, modify the selected context-token values, and zero the value vectors of the entire suppression-token block. The modified values are then used in the attention computation for image generation. Its zeroed value contributes no content through the attention pathway.

Q2 Following the reviewer’s suggestion, we evaluated two additional baselines on Spurious ImageNet: (1) LLM rewriting, which removes the negated content from the prompt. (2) LLM rewriting+ negative prompting, which supplies the unwanted concept as a separate negative prompt. Due to the time constraint, we test using a 1 image per prompt.

Table: Different methods on the Spurious ImageNet bench

ModelMethodCP ↑ (%)NP ↓ (%)
InfinityBaseline91.063.0
InfinityLLM rewriting86.046.0
InfinityLLM rewriting + Negative prompting88.039.0
InfinityOurs93.035.0
SD3Baseline95.049.0
SD3LLM rewriting94.032.0
SD3LLM rewriting + Negative prompting93.027.0
SD3Ours94.024.0

Our method achieves the lowest NP on both backbones at comparable CP. As the ablation in our response to Reviewer ouUL Q3 shows, the gain over rewriting comes from the context-value projection.

Q3 For λ>1\lambda>1, it means that it reverses the negation content, which is an over-relaxed extension to compensate. It subtracts more than the projected component, compensating for a subspace estimated from only a few tokens. This is motivated by an empirical observation (see Figure 10 && 11), where increasing λ\lambda shows stronger negation performance.

.

To replicate our experiments, please refer the following table. Neither rr nor kk is fixed or manually tuned, where rr is the rank of the negation matrix. k=∣{i:ai>mean(a)}∣k=|\{i:a_i > mean(a)\}| determines adaptively, the kk tokens with the highest projection energies are then edited. we will make it clear in our paper.

Table: λ\lambda used in the main tables.

DatasetInfinitySD3
IP1.251
Safety23
General22
Spurious ImageNet23
I2P23

Eq. (3) removes the component along the estimated subspace, but that subspace is built from only a few tokens and underestimates how much concept information the context values carry. λ\lambda>1 over-relaxes the projection to compensate. λ\lambda is selected on a held-out split of 5 additional prompts, no overlapping with the test benchmark.

Q4 Our method differs from SAFREE in two key aspects, which are important for effective negation suppression and preserving desired content.

(1) Intervention space: SAFREE modifies text embeddings before image generation. Our method directly intervenes on cross-attention value vectors. This distinction is important because unwanted concepts can still be introduced through contextual information during generation. Our method directly suppresses unwanted information entering the image generation process.

(2) Token identification and intervention: SAFREE identifies influential tokens based on changes in the average prompt representation when individual tokens are excluded. However, this global representation does not directly reflect how individual tokens contribute unwanted content through cross-attention. In contrast, our method identifies context tokens whose value vectors strongly align with the negation subspace and selectively removes their unwanted components. Negated concepts can also be represented in other context tokens, even after the negated token itself is suppressed.

These help our method target unwanted concept information more directly while limiting its impact on desired content. Consistent with this motivation, our experiments demonstrate stronger negation suppression on three of four benchmarks and better context preservation than SAFREE.

Q5 Thank you for pointing this out. The difference is that Table 1 and Tables 3–4 are for different evaluation purposes and cover different prompt distributions.

Table 1 is a diagnostic stress test designed to investigate potential causes of negation failures, where explicit negation logic represents a small subset of the benchmark. The 0.00% success rate indicates that Infinity are unable to follow the negation constraints.

Tables 3 and 4 provide scale evaluations across more diverse negation logic prompts. The consistently high NP$-$ values for both Infinity and Stable Diffusion 3 indicate that negated concepts are frequently generated.

Together, they motivate the need for more effective negation suppression.

Q6 We address these settings separately.

(1) IP subset without explicit entity names in the context. To evaluate this setting, we construct an additional IP subset by replacing explicit mentions of the forbidden entity in the positive context with generic descriptions (e.g., "a character"), while retaining the forbidden entity in the negation clause.

MethodCP (%)NP (%)
Infinity90.5732.08
Ours (Infinity)96.230.00
SD392.4528.30
Ours (SD3)86.7911.32

We can observe that our method significantly reduces the generation of negated concepts, while showing a moderate effect on CP for SD3.

(2) Standard non-negated prompts. Our method first analyzes the input prompt to identify the negated concept and its contextual information. When no negation is detected, the suppression intervention is inactive, leaving the original generation pipeline unchanged. To verify this behavior, we evaluate our method on GenEval, a standard benchmark containing 553 non-negated prompts. The GenEval benchmark consists of 553 prompts across six tasks: single object, two objects, counting, colors, position, and color attribution. For evaluation, Mask2Former detects objects using a confidence threshold of 0.3, increased to 0.9 for counting. An image is considered correct only when all task-specific requirements are satisfied: all requested objects must be detected, object counts must match exactly, colors are classified using CLIP ViT-L/14 on detected objects, and spatial relationships are verified using bounding-box centers with a margin of 0.1. Each task's accuracy is calculated as the proportion of images satisfying all requirements. The overall GenEval score is the average of the six task accuracies.

Comments on the paper

1 comment

The authors sincerely thank three expert reviewers for their careful reviews, constructive feedback, insightful questions, and fruitful discussions! Based on the reviewers' feedback, we have revised our submission accordingly. The major revisions are summarized as follows:

  • A controlled analysis of the text encoder, including a new t-SNE visualization of CLIP text embeddings, to better support our motivation (Section 3. Reviewer Nhwh).

  • An expanded discussion of the Negation benchmark, including its motivation and evaluation judges (Section 5.1. Reviewers ouUL, obaW).

  • An uncertainty analysis of the human confusion experiment (Section 5.4. Reviewer ouUL).

  • Refined claims in the Abstract and Section 4.2 (Reviewer ouUL).

  • An expanded comparisons with GLoCE and SAFREE (Appendix A.2. Reviewers Nhwh, ouUL).

  • Additional ablation experiments to further support our method's design choices (Appendix B. Reviewers Nhwh, ouUL).

  • A detailed analysis of contextual token selection for the projection (Appendix D. Reviewers Nhwh, ouUL, obaW).

  • An evaluation of our method on non-negated benchmarks (Appendix F.3. Reviewers Nhwh, ouUL).

  • A new experiment on the IP subset without explicit entity names in the positive context, including results from Nano Banana (Appendix F.4. Reviewers Nhwh, ouUL).

  • A detailed discussion of methods closely related to ours (Appendix F.5. Reviewer obaW).

  • A detailed explanation of our negation text-to-image generation pipeline, including implementation details and hyperparameters to ensure reproducibility (Appendix I. Reviewers Nhwh, ouUL, obaW).

We have marked all revised content in blue. We sincerely thank all three reviewers once again for their time, effort, and constructive feedback throughout the review process!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.