acceptodds
Under review as a conference paper at ICLR 2027

Structured Counterfactual Contrast for Compositional Text-to-Image Generation: A Pilot Study of Prompt Negation and Candidate Reranking

Abstract

Text-to-image diffusion models often fail on compositional prompts involving attribute binding, spatial relations, and multiple objects. Such failures can be near-miss compositions: an image is broadly plausible but violates one local structural constraint, such as an attribute assignment or a relation direction. We study structured counterfactual contrast as an inference-time diagnostic and candidate-selection signal. We represent a prompt as a constraint graph and construct minimally perturbed counterfactuals through attribute swaps, relation inversions, and argument-role swaps. A frozen vision-language scorer then computes the margin between compatibility with the target prompt and compatibility with its nearest counterfactual alternative. We evaluate two lightweight instantiations without modifying the diffusion model. First, prompt-level counterfactual augmentation lowers the diagnostic margin in a matched pilot, indicating that textual negation is not a reliable way to express structured exclusions. Second, in a compute-matched three-candidate setting, counterfactual-aware reranking selects images with higher values of the margin than first-candidate and random-selection baselines. Because this margin is also the selection objective, we treat this result as a selection diagnostic rather than independent evidence of improved compositional fidelity. Our results establish a constrained empirical finding: explicit structured comparison is more promising than direct textual negation as an inference-time use of counterfactual information. We release a randomized method-blind evaluation protocol for assessing whether this signal translates into independently judged compositional fidelity, and identify latent-level counterfactual guidance as a direction for future work.

Primary area:
generative models
Keywords:
text-to-image generation, diffusion models, compositional generation, counterfactual reasoning, vision-language models, candidate reranking

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_tDfbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a framework called Counterfactual Constraint Diffusion (CCD) for compositional text to image generation. Although the paper formulates a latent level guidance update rule, it is left unimplemented.

Critical strengths

NA

Critical weaknesses
  1. The paper is an unfinished work with circular and tiny scale experiments. The core latent level guidance method described in Section 3.4 and Figure 1 is not implemented or tested at all. The actual experiments only test 12 and 20 toy prompts on a single model (SDXL Turbo). In Table 2, the only positive quantitative result is completely circular: candidate images are selected by maximizing the CLIP counterfactual margin score, and then evaluated using that exact same margin score. Picking the maximum of three numbers naturally yields a higher average than picking the first or a random number, which provides no evidence of improved image quality. Furthermore, the external human evaluation in Section 4.5 is merely a blank template with zero collected data, and no standard compositional benchmarks (such as T2I-CompBench or TIFA) or basic CLIP reranking baselines are evaluated.

  2. Severe formatting errors and fabricated or duplicated references make the manuscript unacceptable for publication. On Page 3, Figure 1 and half of Figure 2 are pasted together as a single image with baked in captions, followed by a duplicate LaTeX caption underneath. Figure 2 is then pasted again on Page 5 with double captions. Even worse, none of the 45 references are cited anywhere in the main text. The reference list itself contains massive duplications (for example, Rombach et al. appears three times as [1], [26], and [30], and pairs like [2]/[28], [3]/[27], [8]/[35], [9]/[36], [10]/[39], [11]/[38], and [12]/[37] are duplicates) alongside clear AI hallucinations and wrong authors.

Questions for authors

Why was this manuscript submitted to a top tier conference in such an unfinished state? Submitting an incomplete draft without basic proofreading shows a lack of respect for the peer review process and wastes reviewers' time.

Reviewer_zLR3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies compositional generation in diffusion models. It proposes representing prompts as constraint graphs and constructing counterfactual alternatives. The paper first introduces a latent-based approach, which is "not implemented," and then alternatively proposes two methods: (1) prompt-level counterfactual augmentation and (2) counterfactual-aware reranking, which selects the candidate that maximizes the margin. The method is tested on only 12 prompts using SDXL-Turbo. Results show that while prompt-level counterfactual augmentation decreases the margin and is ineffective, reranking increases the margin compared to some baselines.

Critical strengths

compositional generation is an important topic and limitation of current T2I system that the paper is studying

Critical weaknesses

[W1] Limited novelty and unimplemented main contribution. The paper proposes latent-level method in Eqs. 5&6, but L195 mentions "We do not instantiate this latent verifier or evaluate this update in the current experiments." It is unclear why this central component is not implemented or validated. Consequently, the actual contributions are limited to relatively straightforward prompt augmentation and CLIP-based candidate reranking. Even among the proposed 3 transformations in L157, it's not clear whether only one of them is utilized or all types of transformations in the actual method. Furthermore, the novelty of the proposed latent-level formulation relative to existing methods such as Contrastive CFG [1] and Contrastive Guidance [2] remains unclear, even if it were implemented. The paper provides no discussion of these closely related approaches.

[W2] The paper is extremely poor and is poorly written

  • No references are provided in the main text and no actual related works are discussed or cited in the related work section
  • The Figure 1 has two sub figure with labels (Fig1 and Fig2) and then in page 5 there is another Figure 2 which is essentially the same Fig 2 in page 3 I think.
  • the paper is not reproducible as no actual implementation details are provided in the paper beyond "All methods use the same resolution, sampling steps, and seed for a matched prompt–seed instance" in L 207. The Reference section contains numerous duplicated references: [1]/[26], [2]/[28], [8]/[35], [10]/[39]), and incorrect/incomplete citations, such as s [18], [19], [20], [44] and etc. Overall, the manuscript appears poorly prepared and insufficiently checked

[W2] Insufficient experimental validation and questionable evaluation metric. The results rely primarily on a CLIP-based counterfactual margin, which does not necessarily indicate whether the generated images are compositionally correct. In particular, Table 2 reports that reranking images by this margin produces higher values of the same margin, which is expected by construction and does not independently demonstrate improved compositional fidelity. The evaluation is also extremely limited: the prompt-level study uses only 12 prompts and 36 prompt–seed instances, while the reranking study uses 20 unique prompt forms across 60 evaluations. It is unclear how these prompts were selected or constructed. The paper lacks independent compositional evaluation, established compositional benchmarks, and comparisons against stronger reranking baselines.

[W3] Incomplete research and missing essential experiments. The paper reads more like a preliminary work than a completed paper. The latent-level guidance mechanism is neither implemented nor tested, and the necessary independent compositional evaluation is missing: paper requires actual metric to test wether images generated are compositionally correct or not and also requires humane val. In L314-316, it is explicitly mentioned "we will report blinded human results only after collecting ratings from at least three independent raters, including CCD win rate, paired bootstrap confidence intervals, a paired permutation test, and inter-rater agreement."

[1] ContrastiveCFG: Guiding Diffusion Sampling by Contrasting Positive and Negative Concepts [2] Contrastive Prompts Improve Disentanglement in Text-to-Image Diffusion Models

Questions for authors

Please see weaknesses

Reviewer_U2TwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates whether a special type of negative prompt used with a diffusion model, in a framework called Counterfactual Constraint Diffusion (CCD), could improve compositional generation. To illustrate, suppose a positive prompt (pp) (the generation target prompt) describes objects, attributes, and their relations, such as “red A is on blue B.” The special type of negative prompt (p′p') perturbs the attributes or relations, for example, “blue A is on red B.”

The paper proposes two ways to leverage the negative prompt. First, it simply appends “NOT [negative prompt]” to the positive prompt. Second, it proposes an evaluation score, the Counterfactual Contrast Score, based on the negative prompt, and a pipeline, counterfactual-aware reranking, for selecting the best candidate from images generated with different random seeds. Specifically, the score is the largest gap between s(xi,p)s(x_i,p) and s(xi,p′)s(x_i,p') across all constructed negative prompts, where ss is a VLM alignment score between a generated image xix_i and a prompt. The experiments show that appending “NOT [negative prompt]” reduces the score, while counterfactual-aware reranking achieves a higher score than first-candidate and random selection.

Critical strengths

S1. The paper addresses an interesting and important problem: compositional generation remains an unresolved challenge.

S2. Constructing negative prompts by swapping relations and attributes is an insightful idea.

Critical weaknesses

Score design

W1. The proposed score measures how much more strongly a generated image aligns with the target prompt than with the negative prompts. However, a large gap does not necessarily indicate strong alignment with the target prompt itself. The paper lacks a theoretical discussion of why the gap should be preferred over the absolute similarity s(x, p). Can an image have a large gap but low absolute target similarity? If so, is the gap still a suitable selection criterion? The paper also provides no empirical comparison showing that gap-based selection outperforms selection based on target similarity alone.



Experiments

W2. The pipeline lacks an independent evaluation metric. The reranking method selects images by maximizing the counterfactual contrast score and is then evaluated with that same score. This does not establish whether the pipeline improves compositional correctness. Although a human evaluation protocol is proposed, no results are reported.



W3. The experimental scope is limited to SDXL-Turbo. The paper reports that appending negative prompts fails on this model, but this may reflect limitations of its text encoder in handling negation. Testing additional models, with a clearer justification for their selection and their expected ability to interpret negation, would strengthen the findings.

W4. The paper provides no justification or ablation establishing which attribute and relation perturbations should be used. Attribute swaps, relation inversions, and argument swaps are proposed, but their individual contributions and the sensitivity of the results to the choice of counterfactuals are not examined.


W5. The latent-level CCD formulation is presented as a third way to use counterfactual contrast, but no experiments or theoretical results are provided to support it.

Questions for authors

Q1. Does reranking by s(x,p)s(x, p) alone (without the counterfactual term) select the same candidates as reranking by the margin? If not, how often do the two disagree, and which choice is correct on inspection?

Q2. Table 1 margins are on the order of 0.006 while Table 2 margins are on the order of 0.3. Are these computed with the same scorer and normalization? If so, what explains the difference in scale?



Q3. In the prompt-level experiment, the exclusion “not p′p'” is appended to the positive prompt. Did you also try placing p′p' in the negative-prompt slot of a model that supports classifier-free guidance?

Q4. SDXL-Turbo uses guidance scale 0, so the standard negative-prompt mechanism is unavailable on it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.