- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces NegT2IBench, a benchmark for evaluating negated attributes and relations in text-to-image generation. Its 4,800 prompts cover six semantic categories and eight combinations of positive and negated statements, controlling statement and object counts across polarity settings while requiring all named objects to remain present. A modular evaluator combines object detection, attribute classification, and geometric and depth-based checks to assess individual requirements. The paper validates the evaluator against human annotations on 600 images and evaluates eleven generation configurations on 211,200 images, analyzing polarity-dependent performance, compositional load, and failure types.
Critical strengths
-
The benchmark supports more controlled comparisons of positive and negated requirements. Organizing prompts by polarity and statement load while preserving object counts reduces confounding from prompt complexity. Requiring every named object to remain present also prevents success through object omission. These choices improve the interpretability of comparisons across polarity settings.
-
The evaluator provides traceable decisions and useful validation. Checking individual objects, attributes, and relations makes failures attributable to specific requirements. The human study compares multiple evaluators and question formulations, showing agreement comparable to strong VLM judges with lower memory use than the evaluated 8B–38B models. The question-wording analysis also provides practical evidence for configuring VLM-based evaluation.
Critical weaknesses
-
The incremental contribution and resulting scientific insights are insufficient. Existing benchmarks, including GenAI-Bench and UniGenBench++, already evaluate negation. The proposed organization by polarity and load is useful, but the main findings largely quantify established difficulties with negation and satisfying multiple constraints. Polarity-dependent model rankings and category differences are more informative, yet the analyses do not establish what drives these differences or separate responses to negation from default generation tendencies. The acknowledged possibility of accidental success further limits their interpretation. Overall, I find the contribution primarily a more structured measurement of known failures, without sufficient additional insight to justify acceptance.
-
The evaluator’s practical advantage is not established against sufficiently competitive compact baselines. Its image-level agreement with humans is 87.2%, compared with 86.5% for Qwen3-VL-8B using positively phrased per-element questions; the latter is also faster under the reported protocol. The memory reduction is meaningful, but the comparison omits modern 4B-class VLMs, leaving unclear whether a compact off-the-shelf judge could provide comparable or better accuracy at a similar memory cost. Moreover, 140 of the 600 validation images were used to fit evaluator thresholds. A held-out comparison with a compact VLM is therefore important for establishing the additional value of this specialized pipeline.
Questions for authors
Comparison with a compact VLM: How does the proposed evaluator compare with a modern 4B-class judge, such as Qwen3-VL-4B-Instruct, using the same positively phrased per-element questions as the stronger VLM baselines? Please compare both methods on the 460 images excluded from threshold fitting, reporting image-level and negated-statement agreement with uncertainty estimates, alongside measured peak GPU memory and per-image latency under a common hardware and timing protocol, with numerical precision specified. This comparison would materially affect my assessment of whether the specialized evaluator provides a meaningful accuracy–cost advantage over a compact off-the-shelf alternative.
We thank the reviewer for the clear suggestions. We added the requested compact-judge comparison and neutral controls.
W1. Contribution and new insight. We agree that the findings should go beyond known negation difficulty. GenAI-Bench and UniGenBench++ include negation (lines 129–130), but without matched polarity, load or neutral controls (comparison table: reply to mjAc, W2). Our design controls category, polarity, and load while retaining all named objects (lines 185–195, 777–785), with checks validated by polarity. The new control addresses accidental success, identified in lines 399–403.
Neutral control. With the same objects and seeds but the tested condition omitted (11 models), negation makes the denied content appear more often for color, material, directional and depth (e.g., color 12.7% → 67.2%); no model shows a significant decrease. For proximity and comparative there is no measurable effect, so successes there match default generation. This also explains category differences: color is hardest because negation primes it most (full table: reply to NnSX, W4/Q3a).
What drives the ranking changes. Rankings flip with polarity (Table 3): FLUX.2-dev is 1st on one positive statement but 8th on one negated statement, while Janus-Pro goes from 8th to 1st. From the neutral images, we measure each model's priming: how much more often the denied value appears with the negated prompt than with the neutral one ("a non-red cup" vs. "a cup"), averaged over color and material. The gap is negated minus affirmative accuracy on relations, which share no prompts with the priming measure.
Models that drop under negation are the most primed. Across models, priming correlates with the gap (Spearman ρ = −0.83, p = 0.003) and with negated accuracy at equal affirmative accuracy (partial ρ = −0.69), while default rates barely differ (color 10–16% for all). These are correlations, not proof of a cause.
W2/Q. Compact judges on the 460 images excluded from threshold fitting. We agree this comparison is needed to assess practical value. We tested Qwen3-VL-4B-Instruct, InternVL3.5-4B, and Qwen3-VL-2B using the same positively phrased per-element questions as the larger judges. Agreement uses the three-annotator majority (submission lines 255–262); 95% intervals use 2,000 image bootstrap resamples, paired for differences. Negated-statement results cover 384 statements.
κ on a 0–100 scale; Δ = ours minus judge, paired [95% CI]; * significant. On affirmative statements, Qwen3-VL-4B is better (Δ −9.9 [−15.3, −4.4]). Cost: one H200, batch 1, loading excluded, peak memory from
torch.cuda.max_memory_allocated(); judges in BF16 (max 1,003,520 pixels); RF-DETR-Seg-XL, SAM 2.1-small and Depth-Anything-V2-Small in FP32, SigLIP 2 in FP16 autocast, TF32 off.We agree that compact judges are competitive: Qwen3-VL-4B comes close to our evaluator and is faster per process. Our evaluator still offers lower memory than the 4B and 8B judges, a separate check per statement, and rules that do not depend on question wording. Wording matters for judges: negatively phrased questions lower the image κ of Qwen3-VL-4B by 6.2 points and of InternVL3.5-4B by 17.4.
Throughput at scale. Both methods timed identically, including model loading, on one H200:
At 8 processes, our evaluator has about 1.8× the throughput, using 50.2 vs. 70.2 GiB in total; Qwen is faster at two processes. We will report both regimes and position our evaluator as an auditable, scalable alternative.