acceptodds
Withdrawn from ICLR 2027

NegT2IBench: When Negation Changes the Picture. A Polarity Benchmark for Text-to-Image Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-to-image (T2I) models are judged by benchmarks that measure whether requested content appears, but these benchmarks largely overlook the complementary ability to satisfy negated constraints, for example, generating “a non-red cup. ” Measuring negation raises challenges not faced by affirmation-based benchmarks and requires careful prompt and evaluation design. We introduce NegT2IBench, a benchmark of 4,800 prompts covering two attribute types and four relation categories. Prompts are organized by polarity: the number of positive statements that must hold and negated statements that must not, each ranging from 0 to 2. Varying the two independently separates the effect of negation from the effect of prompt complexity. Our detector-based scoring is reproducible, auditable, and pinpoints which requirement failed. On 600 images with three-annotator labels, it agrees with humans as closely as vision-language judges up to 30× larger, while using only a fraction of their GPU memory. Across eleven T2I models and 211,200 images, nine score lower on a single negated statement than on a single positive one. Per-statement scoring reveals that the loss is largest for color and near zero for proximity, and that 41.5% of failed statements render exactly what the prompt forbids. Rendering what a prompt asks for and withholding what it forbids are distinct capabilities that an aggregate compositional score cannot distinguish. NegT2IBench measures the latter directly, providing a controlled testbed for diagnosing negation failures and developing methods to overcome them.

Primary area:
generative models
Keywords:
Text-to-Image Generation, Image Compositionality, Benchmarking, Multimodal Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_L5jwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces NegT2IBench, a benchmark for evaluating negated attributes and relations in text-to-image generation. Its 4,800 prompts cover six semantic categories and eight combinations of positive and negated statements, controlling statement and object counts across polarity settings while requiring all named objects to remain present. A modular evaluator combines object detection, attribute classification, and geometric and depth-based checks to assess individual requirements. The paper validates the evaluator against human annotations on 600 images and evaluates eleven generation configurations on 211,200 images, analyzing polarity-dependent performance, compositional load, and failure types.

Critical strengths
  1. The benchmark supports more controlled comparisons of positive and negated requirements. Organizing prompts by polarity and statement load while preserving object counts reduces confounding from prompt complexity. Requiring every named object to remain present also prevents success through object omission. These choices improve the interpretability of comparisons across polarity settings.

  2. The evaluator provides traceable decisions and useful validation. Checking individual objects, attributes, and relations makes failures attributable to specific requirements. The human study compares multiple evaluators and question formulations, showing agreement comparable to strong VLM judges with lower memory use than the evaluated 8B–38B models. The question-wording analysis also provides practical evidence for configuring VLM-based evaluation.

Critical weaknesses
  1. The incremental contribution and resulting scientific insights are insufficient. Existing benchmarks, including GenAI-Bench and UniGenBench++, already evaluate negation. The proposed organization by polarity and load is useful, but the main findings largely quantify established difficulties with negation and satisfying multiple constraints. Polarity-dependent model rankings and category differences are more informative, yet the analyses do not establish what drives these differences or separate responses to negation from default generation tendencies. The acknowledged possibility of accidental success further limits their interpretation. Overall, I find the contribution primarily a more structured measurement of known failures, without sufficient additional insight to justify acceptance.

  2. The evaluator’s practical advantage is not established against sufficiently competitive compact baselines. Its image-level agreement with humans is 87.2%, compared with 86.5% for Qwen3-VL-8B using positively phrased per-element questions; the latter is also faster under the reported protocol. The memory reduction is meaningful, but the comparison omits modern 4B-class VLMs, leaving unclear whether a compact off-the-shelf judge could provide comparable or better accuracy at a similar memory cost. Moreover, 140 of the 600 validation images were used to fit evaluator thresholds. A held-out comparison with a compact VLM is therefore important for establishing the additional value of this specialized pipeline.

Questions for authors

Comparison with a compact VLM: How does the proposed evaluator compare with a modern 4B-class judge, such as Qwen3-VL-4B-Instruct, using the same positively phrased per-element questions as the stronger VLM baselines? Please compare both methods on the 460 images excluded from threshold fitting, reporting image-level and negated-statement agreement with uncertainty estimates, alongside measured peak GPU memory and per-image latency under a common hardware and timing protocol, with numerical precision specified. This comparison would materially affect my assessment of whether the specialized evaluator provides a meaningful accuracy–cost advantage over a compact off-the-shelf alternative.

1 comment

We thank the reviewer for the clear suggestions. We added the requested compact-judge comparison and neutral controls.

W1. Contribution and new insight. We agree that the findings should go beyond known negation difficulty. GenAI-Bench and UniGenBench++ include negation (lines 129–130), but without matched polarity, load or neutral controls (comparison table: reply to mjAc, W2). Our design controls category, polarity, and load while retaining all named objects (lines 185–195, 777–785), with checks validated by polarity. The new control addresses accidental success, identified in lines 399–403.

Neutral control. With the same objects and seeds but the tested condition omitted (11 models), negation makes the denied content appear more often for color, material, directional and depth (e.g., color 12.7% → 67.2%); no model shows a significant decrease. For proximity and comparative there is no measurable effect, so successes there match default generation. This also explains category differences: color is hardest because negation primes it most (full table: reply to NnSX, W4/Q3a).

What drives the ranking changes. Rankings flip with polarity (Table 3): FLUX.2-dev is 1st on one positive statement but 8th on one negated statement, while Janus-Pro goes from 8th to 1st. From the neutral images, we measure each model's priming: how much more often the denied value appears with the negated prompt than with the neutral one ("a non-red cup" vs. "a cup"), averaged over color and material. The gap is negated minus affirmative accuracy on relations, which share no prompts with the priming measure.

ModelRank: positive → negatedPriming (pp)Gap, relations (pp)
FLUX.2-klein2 → 11+50.0−35.8
FLUX.2-dev1 → 8+40.9−24.1
Qwen-Image3 → 9+60.2−17.8
Z-Image6 → 10+59.8−17.4
HunyuanImage5 → 4+38.3−19.1
BLIP3-o4 → 2+34.3−12.5
HiDream9 → 7+36.6+0.2
SD 3.57 → 3+34.7+3.2
Janus-Pro8 → 1+16.1+9.4
SD 1.510 → 6+9.9+13.9
SD 1.5 + NegationCLIP11 → 5+6.8+13.8

Models that drop under negation are the most primed. Across models, priming correlates with the gap (Spearman ρ = −0.83, p = 0.003) and with negated accuracy at equal affirmative accuracy (partial ρ = −0.69), while default rates barely differ (color 10–16% for all). These are correlations, not proof of a cause.

W2/Q. Compact judges on the 460 images excluded from threshold fitting. We agree this comparison is needed to assess practical value. We tested Qwen3-VL-4B-Instruct, InternVL3.5-4B, and Qwen3-VL-2B using the same positively phrased per-element questions as the larger judges. Agreement uses the three-annotator majority (submission lines 255–262); 95% intervals use 2,000 image bootstrap resamples, paired for differences. Negated-statement results cover 384 statements.

MethodImage agr.Image κ [95% CI]Δ image κNeg. agr.Negated κ [95% CI]Δ negated κPeak GiBs/image
Ours86.5%71.0 [64.1, 78.1]—78.6%57.2 [48.8, 65.1]—6.280.78
Qwen3-VL-8B85.4%68.0 [60.4, 75.0]+3.1 [−4.2, 10.4]74.0%48.3 [38.6, 57.7]+8.9 [−0.6, 18.4]16.810.49
Qwen3-VL-4B84.1%65.0 [57.6, 72.1]+6.0 [−1.5, 13.7]76.0%51.8 [42.4, 60.6]+5.4 [−4.5, 15.0]8.770.41
InternVL3.5-4B83.9%65.2 [57.9, 72.1]+5.8 [−1.9, 13.8]70.6%41.9 [32.5, 51.3]+15.2 [5.3, 25.4]*9.980.61
Qwen3-VL-2B78.5%52.9 [44.9, 60.6]+18.1 [9.1, 26.9]*67.4%37.1 [29.6, 45.0]+20.0 [10.5, 30.1]*4.410.36

κ on a 0–100 scale; Δ = ours minus judge, paired [95% CI]; * significant. On affirmative statements, Qwen3-VL-4B is better (Δ −9.9 [−15.3, −4.4]). Cost: one H200, batch 1, loading excluded, peak memory from torch.cuda.max_memory_allocated(); judges in BF16 (max 1,003,520 pixels); RF-DETR-Seg-XL, SAM 2.1-small and Depth-Anything-V2-Small in FP32, SigLIP 2 in FP16 autocast, TF32 off.

We agree that compact judges are competitive: Qwen3-VL-4B comes close to our evaluator and is faster per process. Our evaluator still offers lower memory than the 4B and 8B judges, a separate check per statement, and rules that do not depend on question wording. Wording matters for judges: negatively phrased questions lower the image κ of Qwen3-VL-4B by 6.2 points and of InternVL3.5-4B by 17.4.

Throughput at scale. Both methods timed identically, including model loading, on one H200:

Parallel processesOurs (s/image)Qwen3-VL-4B (s/image)
20.3900.267
80.1560.282

At 8 processes, our evaluator has about 1.8× the throughput, using 50.2 vs. 70.2 GiB in total; Qwen is faster at two processes. We will report both regimes and position our evaluator as an auditable, scalable alternative.

Reviewer_NnSXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

NegT2IBench evaluates whether text-to-image models satisfy negated attributes and relations while preserving the named objects. It contains 4,800 prompts across six categories and eight polarity settings, with statement and object counts controlled across polarity at a fixed load. A modular perception pipeline evaluates generated images, and the paper compares this evaluator with human annotations and alternative judges. Experiments across eleven model configurations report substantial differences between positive and negated prompt performance and provide a statement-level failure breakdown.

Critical strengths
  • The benchmark addresses a clear and useful evaluation gap. Requiring object presence while negating attributes or relations avoids success through object omission, and controlling statement and object counts makes the polarity comparisons more informative than unconstrained comparisons.
  • The paper provides meaningful evaluator validation, including human annotations, comparisons with alternative judges, polarity-specific agreement, and component ablations. Its disclosure of dead-zone behavior and limitations is also valuable.
Critical weaknesses

-The failure analysis does not adequately distinguish generator failures from propagated evaluator errors, and the headline evaluator validation is not fully held out.

The pipeline depends on detection, counting, masks, attribute classification, and geometric or depth estimates. Errors in these components can produce any of the reported failure labels. An undetected object may be present in the image; a negated violation may result from incorrect classification or geometry. Consequently, these labels describe evaluator decisions rather than verified causes of generator failure. Section 5’s statement that dead zones are the only cause attributable to the evaluator alone is not justified.

This concern is material to the reported finding that 41.5% of failed statements render forbidden content. Appendix B.5 reports 46 false passes and 31 false failures on 600 images, while Table 13 reports 78.2% agreement on negated elements. Whole-image agreement does not establish the correctness of the failure taxonomy or its proportions. A stratified human audit of failure labels, with stagewise attribution of evaluator mistakes, is needed to support these diagnostic claims.

  • The controls do not sufficiently separate a response to negation from default generation behavior or unresolved perception.

Positive-versus-negative comparisons at matched load are useful, but a neutral prompt omitting the tested attribute or relation is necessary to measure whether the negative instruction changes behavior beyond the model’s default output distribution. For example, “a cup” establishes how frequently the model avoids red without being instructed to do so. Selecting plausible attributes does not supply this baseline. The paper acknowledges the missing control, but leaving it to future work limits the interpretation of the current scores as measures of negation-following ability.

The relation evaluator also passes negated statements when no predicate is resolved. This is appropriate when the relation is genuinely absent, but uncertain perception does not establish absence. Table 19 reports that 55.5% of successful negated depth statements are dead-zone passes. Reporting this frequency does not establish their semantic correctness. Human validation of these cases and sensitivity analysis separating verified absence from unresolved measurements are needed before drawing strong conclusions about category-specific negation performance.

Questions for authors
  • Clarification and reanalysis: Were the remaining 460 human-annotated images untouched by all evaluator selection and calibration decisions? What are the evaluator agreement and polarity-specific error rates when the 140 calibration images are excluded?
  • Additional validation: Can a stratified human audit establish how often each reported failure category reflects a genuine generator error rather than a detection, counting, classification, mask, or geometry error? Does the conclusion about forbidden-content violations remain after this distinction?
  • Additional experiments: Do the negation findings persist relative to neutral prompts that preserve the objects but omit the tested attribute or relation? For depth, do the conclusions persist when unresolved cases are reported separately rather than automatically accepted for negated statements?
1 comment

We thank the reviewer for the detailed feedback. Each point below led to a new analysis or human audit.

W1/Q1. Validation excluding calibration images. We agree that validation should exclude fitting data. Yes: no evaluator decision used the 460 images; parameters were fitted only on the 140 (lines 347–348, 1557–1560). κ is on a 0–100 scale.

SubsetnAgreementκ [95% CI]
Held-out images46086.5%71.0 [64.1, 78.1]
Calibration images14089.3%75.3 [62.7, 86.2]
All images60087.2%72.0 [66.1, 77.7]
Inter-annotator, held-out46086.6%68.4
Held-out affirmative statements1,70790.2%65.2 [60.1, 70.1]
Held-out negated statements38478.6%57.2 [48.8, 65.1]

Held-out minus calibration is −4.3 κ [−17.1, 10.6], not statistically significant. On held-out images, the evaluator makes about 36 false passes and 26 false failures; statement error rates are 9.8% (affirmative) and 21.4% (negated).

W1–W3/Q2. Human validation of failure labels. We agree that any perception stage can introduce errors. We will correct the claim in lines 408–409 that only dead zones can be evaluator errors and describe the taxonomy as evaluator-assigned failure labels, with human confirmation reported separately.

Using the paper's evaluator, we checked all 671 failed statements against existing human labels. We excluded 70 attribute/relation judgments made inapplicable by missing or duplicated objects, leaving 601. Rate: share of evaluator failure labels that the three-annotator majority also judges as failed.

LabelBenchmark shareHuman-confirmedRate [95% CI]
Forbidden content present41.5%195/26075.0% [69.4, 79.9]
Positive requirement violated25.9%106/13479.1% [71.5, 85.1]
Over-count15.0%75/11565.2% [56.1, 73.3]
Undetected object12.6%41/5574.5% [61.7, 84.2]
Positive relation unresolved5.0%24/3764.9% [48.8, 78.2]

Weighted by benchmark share, humans confirm 74.0% [70.5, 77.6] of labels; forbidden content remains the largest group (42.0% [39.6, 44.3] of confirmed failures vs. 41.5%, lines 404–407).

The 160 unconfirmed labels trace to 2D geometry (47), depth (45), counting (40), detection (14) and attribute classification (14). The audit is post-stratified: confirmation per failure label, weighted by benchmark share.

W4/Q3a. Neutral control. We agree this baseline is needed. We generated 16,676 neutral images (same objects and seeds, tested attribute or relation omitted; all 11 models). Denied-content occurrence, same evaluator, valid object detections; differences are paired by prompt and seed, so they can differ slightly from the gap between rates:

CategoryNeutralNegatedDifference [95% CI]
Color12.7%67.2%+54.5 [52.0, 57.1]
Material26.9%43.3%+16.1 [14.0, 18.2]
Directional23.1%45.6%+22.4 [20.0, 24.9]
Depth37.5%45.8%+8.3 [6.1, 10.4]
Proximity47.1%47.1%+0.0 [−1.4, 1.6]
Comparative55.7%54.7%−0.9 [−2.8, 0.9]

Increases are significant for 11/11, 9/11, 8/11, and 6/11 models in the first four categories; no model shows a significant decrease. Naming a forbidden value or relation thus makes it more likely, rather than models reverting to default output. For proximity and comparative, negation has no measurable effect.

W5/Q3b. Unresolved depth. We agree that unresolved perception does not establish absence. We reanalyzed the rule in lines 234–237 and exposure in Table 19 (lines 1620–1628): 974/4,400 negated images pass only through unresolved depth. We also annotated 200 images with three annotators (130 unresolved and 40 resolved depth passes, 30 unresolved directional passes); 35 were excluded because an object did not appear exactly once. Combined with the original study, humans confirm 80.1% [73, 86] of unresolved depth passes (113/141), 96% [87, 99] of resolved ones (49/51) and 72% [54, 85] of unresolved directional passes (21/29). Load-1 depth accuracy:

ProtocolAffirmativeNegated
Original: unresolved negated cases pass41.7%44.0%
Unresolved cases fail41.7%21.8%
Unresolved negated cases excluded41.7%28.1%
Unresolved weighted by human confirmation41.7%39.6%

Exclusion changes the subset, and the weighted row is an estimate, not human-scored accuracy; unresolved confirmation also differs between old and new samples (21/34 vs. 92/107). We will report unresolved cases separately and withdraw the depth-advantage claim.

Reviewer_mjAcOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark that measures how well text-to-image models follow negation in prompts. Negation is known to be difficult for discriminative models such as CLIP, and prior benchmarks for image generation have mostly tested whether a negated object is left out. The benchmark consists of 4,800 prompts containing positive and negated attributes and relations. The authors also propose an evaluation protocol built from pretrained foundation models (e.g., SAM 2, SigLIP 2, and Depth-Anything V2). This protocol agrees with human judgments close to how they agree with other VLM judges. Overall, the results show that current models struggle substantially with negation: nine of eleven models score lower on a single negated statement than on a single positive one, and accuracy drops sharply as more statements are combined.

Critical strengths

1- Thorough validation of the evaluation protocol against human labels. 600 images were annotated by humans and used to benchmark the protocol against both VQA judges and similarity-based metrics.

2- Easy to use, interpretable, and lightweight. The evaluator is up to 30× smaller than the VLM judges it is compared with, yet performs on par with or better than them. It also checks each requirement separately and reports which one failed.

Critical weaknesses

1- Limited scope of relation types, which also limits how far the evaluation protocol generalizes. All four relation categories (directional, proximity, comparative, depth) are geometric, and they are often applied to object pairs with no natural connection (e.g., “the backpack is above the parking meter”). Testing such pairs is reasonable, but the benchmark should also include relations between semantically connected objects, such as interactions (“a person holding a cup”) or containment (“a cat inside a box”), as in the non-spatial category of T2I-CompBench++. Object-level negation (e.g., “a desk with no laptop”) is also missing. Because the evaluator verifies relations only through detection and geometry, it is unclear how it would extend to these relation types, which limits its use beyond the current benchmark.

2- Limited novelty. Negation in text-to-image generation has already been studied (NegGenBench, NEG-TTOI, LMD), and the evaluation pipeline follows GenEval’s detector-based design. The main contribution therefore narrows to the matched-polarity setup and the extension of negation from objects to attributes and relations.

Huang et al., T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-Image Generation, IEEE TPAMI, 2025.

Ghosh et al., GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment, NeurIPS 2023 .

Lian et al., LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models, TMLR 2024.

Questions for authors

In general, I think this work has good potential, and its value would increase substantially with a higher quality and more natural prompt set. My suggestions:

1- Add a paraphrased subset. Keep the templated prompts as the controlled core, but add LLM-paraphrased versions of a subset (e.g., “a cup that isn’t red,” “a cup in any color except red”), checked to preserve meaning. Showing that the conclusions hold across phrasings would confirm the results reflect negation rather than the “non-X” template. NegBench (Alhamoud et al., 2025), already cited in the paper, uses a similar pipeline.

2- Broaden relation and attribute coverage. Add non-geometric relations such as interactions (“holding”), containment (“inside”) and support (“on top of”), as in T2I-CompBench++, and extend the color vocabulary with missing values such as gray and gold.

3- Add a counter-prior subset. Include negations that are uncommon with real-world co-occurrences, such as “a polar bear not surrounded by ice” or “a banana that is not yellow.” This is closer to the applications scenarios and could be harder for the model to follow.

1 comment

We thank the reviewer for the helpful suggestions. We added paraphrase and counter-prior tests.

W1/Q2. Relation scope, object negation, vocabulary. We agree that broader relations would extend coverage; App. F lists them as future work. Natural combinations are included, e.g., yellow bananas and wooden tables and their negations, since Qwen3-30B keeps combinations that are plausible or unusual but depictable (App. A.1, lines 795–806). We chose relations that can be verified from 2D geometry and depth. Inside is often hidden unless the box is transparent, and holding and on top of need contact, which masks cannot confirm when objects overlap; on top of is also close to our above. Adding these relations would require a new evaluation approach, most likely with some 3D understanding to verify them. Object negation is covered by NegGenBench, Neg-TtoI, and LMD; we target a complementary study: negated attributes and relations while every named object stays in the image. Colors follow GenEval, and gray is already an answer-only value (Table 15, line 1473), so gray objects are not forced into another color. Values such as gold can be added; we use GenEval's vocabulary for comparability.

W2. Contribution relative to prior work. The table compares NegT2IBench with prior benchmarks.

BenchmarkNeg. / totalPrompt type (how negation is written)Negated targetPolarity matchedLoad variedNeutral controlAll objects kept
NegGenBench200 pairsPositive prompt + separate negative prompt as extra input (“bike” / “wheels”)Object parts✗✗Partial: run without the negative prompt✓
Neg-TtoI2,000/2,000COCO captions rewritten with “without”, “no”Object, scene✗✗✗✓
LMD10Template “a scene without [object]”Object✗✗✗✗
NegationCLIP107/107Short prompts with “not”Object, action, attribute✗✗✗✓
GenAI-Bench347/1,600Free-form prompts; negation is one skill tag among severalObject, attribute, action✗✗✗✗
UniGenBench++65/600Free-form prompts; negation is one test point among severalObject/part, action✗✗✗✗
TIIF-Bench235/5,000Free-form instructionsObject, action (color identifies the object)✗✗✗✗
PosEval (Neg)100/600GenEval template + “A is not left of B”One directional relation✓✗✗✓
NegT2IBench3,600/4,800Templates “a non-red cup”, “A is not above B”; same wording at every polarityColor, material; directional, proximity, comparative, depth✓✓ (1–4)✓ (new)✓

Prior benchmarks mostly negate objects, scenes or actions; only PosEval matches negated with positive prompts, for one directional relation, and none varies load under negation or has a full neutral control. NegT2IBench combines these controls for attributes and relations, keeping all objects at matched statement/object counts, with checks validated by polarity (11 models, 211,200 images). The new neutral control shows that negation attracts the forbidden content: denied color appears in 12.7% of neutral and 67.2% of negated images; for proximity and comparative, negation has no measurable effect (reply to NnSX, W4/Q3a). Our evaluator follows GenEval's detector-based design, but GenEval does not test negation; we add negated checks, material, depth, proximity and size relations, validated by polarity.

Q1. Paraphrases. We agree that wording needs testing. We rewrote all 1,200 negated attribute prompts, at every load, from “non-X” to “X that is not…” (52,800 images, 11 models). The conclusions hold: negated accuracy stays below affirmative for 11/11 models (color) and 8/11 (material) in both wordings, with overall accuracy rising from 14.8 to 20.5 (color) and 24.6 to 26.9 (material). Wording changes the gap's size in both directions. Load-1 accuracy change (pp):

ModelColorMaterial
FLUX.2-dev+16.8+13.7
FLUX.2-klein+11.8+6.2
HiDream+1.2−10.0
SD 3.5+3.0−9.1
Other 7 models−4.0 to +5.5−6.7 to +8.7

Q3. Counter-prior subset: denied value occurrence (%). We agree that counter-prior cases are useful. From the new neutral images, we measured each value's prior (how often it appears unprompted) and grouped Load-1 negations by it.

Neutral priorColorMaterial
Below 5%63.722.2
At least 50%79.685.6

Stronger priors increase failures, and negation amplifies the prior (a color drawn in under 5% of neutral images appears in 63.7% once denied). The benchmark already contains counter-prior examples: the denied color appears in 91.0% of images for “non-yellow banana”, 98.7% for “non-orange carrot”, 90.3% for “non-red apple” and 100% for “non-brown bear”. As the reviewer expected, counter-prior cases are harder, but priors alone do not explain failures. We report this subset separately in the revision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal summary

We thank the reviewers for their helpful feedback. We have added the following experiments and analyses:

  • Neutral controls: 16,676 new images across 11 models with no condition added. Compared with neutral prompts, negation increases forbidden content for color, material, directional relations, and depth. Proximity and comparative relations show no measurable change (NnSX, W4/Q3a).
  • Paraphrase test: We generated 52,800 new images from 1,200 rewritten prompts. Changing "non-X [object]" to "[object] that is not X"caused the scores to vary, but main findings remained consistent: negated prompts still scored lower than affirmative prompts for most models (mjAc, Q1).
  • Object-prior analysis using neutral images: Common object–attribute associations make negation more difficult, but failures also occur for attributes that are rarely generated by default (mjAc, Q3).
  • Held-out validation on 460 images and a human audit of 601 failure labels: Image-level agreement is 86.5%, and forbidden content is the largest group of human-confirmed failures (NnSX, W1–W3/Q1–Q2).
  • Comparison with three additional compact judges and throughput tests: Compact judges are competitive. Our evaluator uses less memory than the tested 4B and 8B judges and achieves higher throughput with eight parallel processes ("Throughput at Scale" and L5jw, W2/Q).
  • Human annotation of an additional 200 images and depth scoring checks clarify how unresolved cases affect depth results. These checks also support the correction to our interpretation (NnSX, W5/Q3b).
  • Comparison with prior benchmarks. We added a table showing how NegT2IBench combines matched affirmative and negated prompts, varying numbers of constraints, and neutral controls while keeping all named objects in the image (mjAc, W2).

We hope these new experiments and clarifications address the reviewers’ concerns, and we kindly ask reviewers to consider them when updating their reviews.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.