acceptodds
Under review as a conference paper at ICLR 2027

Unspoken, Controllable, Unavoidable: Implicit Co-occurring Objects is A Fundamental Property in Text-to-Image Generation

Abstract

Text-to-Image (T2I) models often generate objects that are not in the prompt: e.g., a cup on a table when the prompt is simply β€œa photo of a cup”. We call those the Implicit Co-occurring Objects (ICO). In this work, we present the first systematic study of ICO across seven architecturally distinct T2I models spanning the years 2022-2026 with a strategic plan for the understanding and the control of them. We find that 34-50% of the generated images featuring common objects (56,000 across 80 categories), regardless of the model, contain at least one unprompted object, with strikingly consistent co-occurring patterns shared across the models. To understand where they come from and how models generate them, we investigate four hypotheses about the training data, text encoders, latent space trajectory planning, and sampling dynamics. Training data co-occurrence statistics are found to be the dominant factor, and the text encoder has a secondary effect. And the sampling parameters modulate the magnitude. Guided by the insights, we explore a group of controls of ICO and discover a simple yet effective two-component control method combining isolation phrasing with embedding steering that reduces the leakage rate to 0.19% with a 94.2% fidelity. For a theoretical understanding, we show that having ICO is a fundamental property of the training data, labels, and the objective of T2I models: with co-occurrence statistics existing in the natural image distribution, there is a trade-off between the model-vs-data distribution divergence and the rate of ICO, meaning we cannot freely suppress ICOs without a loss of generation fidelity. Finally, we show that ICO carries a downstream cost: object detectors trained on T2I-generated images exhibit up to an 18.8% gap in recall compared to ICO-free data. We are the first to establish a comprehensive understanding of ICO in T2I models. Our findings provide new insight into this intrinsic property of diffusion-based image synthesis.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
image generation, diffusion models, text-to-image, context, co-occurrence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DLwYOpenReview β†—
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work proposes a systematic study of Implicit Co-occurring Objects (ICO) in text-to-image (T2I) models, demonstrating that ICOs are observed in the generations of recent, diverse T2I architectures. The authors investigate several hypotheses regarding ICOs' sources, eventually indicating a strong impact of co-occurrences in the training data, early denoising steps, and standard sampling hyperparameters. The authors propose several control approaches that reduce leakage with small degradation in generation fidelity. Additionally, the authors show that ICOs cannot be freely suppressed without degrading fidelity when training data already contains object co-occurrences.

Critical strengths

I think the paper's main motivation is strong. It is worth investigating the phenomenon of implicit co-occurring objects (ICOs), which can be harmful as the biased outputs of text-to-image models are often used as training data for downstream tasks (e.g., object detection). The authors, using 7 diverse T2I pipelines on 80 COCO categories, show that the analyzed models generate ICOs.

Critical weaknesses

W1. Lack of novelty

I consider some of the main claims from this work to be incremental relative to prior studies, raising concerns about novelty.

Some of the proposed hypotheses are not new and have already been investigated in similar problems involving Diffusion Models. Precisely:

  • Hypothesis 1 - training data patterns. Liu et al. [1] study concepts that are generated in images despite not being specified in the prompt and attributed them to the training data.
  • Hypothesis 4 - sampling dynamics. Roos et al. [2] present a correlation between the model's bias level and the sampling hyperparameters: the guidance scale 𝑀 and the number of timesteps.

Also, the authors claim that their work is the first systematic study, but there are studies that weaken this claim. Wu et al. [3] investigate "unguided objects" in Stable Diffusion for biases - the objects that are generated but not mentioned in the prompt, along with their co-occurrence.

The method related to steering text embeddings has already been used in a similar setup. Koh et al. [4] investigate entangled content and introduce a delta vector to modify text embeddings.

[1] Liu, Zhili, et al. "Implicit concept removal of diffusion models." ECCV 2024.

[2] Roos, Nathan, et al. "How I Met Your Bias: Investigating Bias Amplification in Diffusion Models." WACV 2026.

[3] Wu, Yankun, et al. "Stable diffusion exposed: Gender bias from prompt to image." AIES 2024.

[4] Koh, Eunseo, et al. "Translation of Text Embedding via Delta Vector to Suppress Strongly Entangled Content in Text-to-Image Diffusion Models." ICCV 2025.

W2. Presentation issues

It is problematic to understand the paper during reading, i.a.:

  1. There are many details missing in the description. E.g., what are II and DD in the co-occurrence rate metric?
  2. In Stage 1 (Section 5), the authors design controls inspired by the observations from Section 4. To me, it's not clear which observations suggest which methods.
  3. Missing descriptions of methods used in Stage 3.
  4. Some hyperparameter values are provided without reasoning: Spearman ρ\rho, detector thresholds.

W3. Lack of motivation for some choices

  1. The choice of methods for Stage 3 seems to be ad hoc. No reasoning is provided for why they are good candidates.
  2. Choice of metrics is unusual. The authors provide their own for the object co-occurrence. Why do you not compare methods in overall generation quality (FID)? Also, why are standard concept unlearning metrics (Unlearning Accuracy, Retain Accuracy) not used for concept-erasure comparisons?
  3. What's the goal of continuous control in this exact setup? I cannot find information about the motivation.
Questions for authors

Q1. Please explain how your experiments and claims relate to my novelty concerns from W1.

Q2. Could you fill in the presentational gaps that I mentioned in W2?

Q3. Please provide motivation for the choices I listed in W3.

5 comments

Thank you for your valuable feedback. Below we address the weaknesses.

Response to W1: novelty

We thank the reviewer for these pointers. The works are relevant, and we agree that several individual components of our study, such as probing the training data, varying sampling parameters or editing text embeddings, have precedents. We will discuss each work explicitly and clarify our novelty with respect to it in the revision.

We would like to emphasis that the contribution we claim is the systematic study as a whole: ICOs measured at scale (seven architecturally distinct models from 2022 to 2026, 80 categories, 56,000 images, four independent probes) and compared against real-world co-occurrence statistics; four hypotheses on their origin tested side by side on the same data, including a causal training intervention; controls derived from these findings and compared with nine concept-erasure methods; and a theoretical account of why suppression trades off against faithfulness to the data. To our knowledge, no prior work brings these together for this problem, and none offers such a theory.

Below we note how each work differs from ours.

Liu et al. (2024) and H1. They fine-tune away unwanted artifacts such as watermarks. On the contrary, ICOs are not artifacts, they are natural scene objects that track real-world co-occurrence. And our control is training-free.

Roos et al. (2026) and H4. Our H4 agrees with their finding that guidance and timesteps modulate bias, and extends it to object co-occurrence. We will add a discussion in the revision.

Wu et al. (2024) and "first systematic study". They study how unguided objects differ across gendered prompts, while we study ICOs themselves, at a large scale, against real-world co-occurrence rates.

Koh et al. (2025) and embedding steering. SSDV adds a delta vector to one word's token, while we project the target direction out of the whole prompt embedding. We have conducted an analysis: at matched leakage, SSDV costs more quality (ImageReward βˆ’0.69 vs. βˆ’0.17). We will include a detailed discussion in the revision.

Response to W2: presentation

We will improve our draft and clarify the confusing points.

(1) Notation in the co-occurrence rate.

  • π’Ÿ is the set of MS-COCO 2017 train+val images with at least one instance annotation. That is 122,218 human-annotated photos (117,266 train + 4,952 val).
  • For an image I ∈ π’Ÿ, we write c ∈ I when category c is annotated in I. This is presence per image.
  • So Ο€(c, o) = |{I ∈ π’Ÿ : c ∈ I ∧ o ∈ I}| / |{I ∈ π’Ÿ : c ∈ I}| is the fraction of COCO photos containing c that also contain o.

We will state this explicitly in Section 2.

(2) Which observations motivate which controls. Each Stage 1 control targets one finding from Section 4. We will add this mapping to the Stage 1 table.

Stage 1 controlObservation it targets (Section 4)
Isolation phrasingH1: models reproduce training-data co-occurrence, and fine-tuning on isolated objects on a plain surface causally reduces leakage. Isolation phrasing requests that same low-co-occurrence part of the data (an isolated object on a plain background) at inference time.
Template diversityH1 (inference-time proxy): if leakage is tied to one overrepresented caption pattern ("a photo of"), paraphrasing it should change the leak rate.
Embedding-space steeringH2: text embeddings carry a weak co-occurrence signal, so we remove the prompt embedding's component along the leak-object direction.
Negative promptingH2 + H4: guidance modulates leakage (H4), and its effectiveness tracks the text encoder's co-occurrence signal (H2).
Early CFG suppressionH3 + H4: ICOs are planned early (20–30% of the steps for most models), so negative guidance is applied only before each model's median first-detection step.
Detect and correctH3: ICOs become detectable early, so at each model's median first-detection step we decode the preview image, detect the leak object, and apply negative guidance for the remaining steps only to images where it is detected.
Guidance and step reductionH4: higher guidance and more steps increase leakage.
Scheduler substitutionH4: the sampling trajectory modulates leakage.

(3) Description of Stage 3 methods. We will add an appendix subsection describing each method and its hyperparameters. In brief:

  • Loss-guided sampling: loss-based guidance in the spirit of universal guidance, with CLIP as the loss.
    • During the first 8 of 20 DDIM steps, the predicted clean image is decoded.
    • Its CLIP ViT-L/14 similarity to "a photo of a dining table" is backpropagated to the latent.
    • The latent is moved against that gradient (scale 40) before the regular denoising step.
  • Cross-attention suppression: the leak object never appears in the prompt, so there is no leak token to suppress.
    • A custom attention processor boosts the prompted category's token(s) by 6Γ—.
    • It scales every other ("unprompted") token by 0.3Γ—. The weights are then renormalised.
    • This is applied in all cross-attention layers at every step, so the image is pulled toward the prompted object alone.
  • Post-hoc inpainting: on images where OWLv2 detects a dining table, the detected boxes (dilated by 24 px, feathered) are inpainted with the model family's inpainting checkpoint (SD-1.5-inpainting, SDXL-inpainting-0.1, FLUX.1-Fill-dev). The inpainting prompt is the original prompt.
  • Post-hoc SDEdit: the finished image is re-noised to strength 0.75, past the point where ICOs are committed (Section 4). It is then denoised again with the original prompt, using the base model's own image-to-image pipeline. No negative prompt is used.
  • Best-of-N: N = 4 candidates are denoised jointly up to step 6 of 20.
    • OWLv2 checks their predicted clean images for a dining table, and only the first clean candidate is completed. If none is clean, candidate 0 is completed.
    • The selector uses the same detector as our evaluation, but it sees only the early preview, never the final image. A clean preview does not guarantee a clean final image, so the selection does not trivially optimise the evaluation metric.

(4) Hyperparameter choices. Sorry for the confusion.

  • Spearman threshold. Spearman ρ β‰₯ 0.3 is not a tuned hyperparameter. It is a conventional statistical boundary for a moderate correlation (0.1 small, 0.3 moderate, 0.5 large). We use it only as a descriptive label and report the exact ρ and p-values alongside it.
  • Detector threshold. We use OWLv2 with a fixed confidence threshold of 0.15 throughout.
  • Robustness. Our conclusions do not depend on this choice:
    • Replacing the fixed cutoff with an area-under-curve leak rate over thresholds 0.15–0.95 gives the identical ranking of all seven models by Excess Leakage Score.
    • On a stratified sample of 300 SD1.5 images (150 OWLv2-positive, 150 negative), OWLv2 agrees with BLIP in VQA mode ("Is there a dining table?") on 88.3% of images.

Response to W3: motivation

(1) Choice of Stage 3 methods. Stage 3 deliberately goes beyond our four hypotheses. It covers the main families of inference-time control:

  • gradient guidance (loss-guided sampling);
  • attention manipulation (cross-attention suppression);
  • post-hoc editing (inpainting, SDEdit);
  • search (best-of-N).

Where a method needed a timing choice (guided steps, checkpoint step, SDEdit strength), we reused the choices from Section 4 rather than tuning it. We will add this rationale to Section 6.

(2) Choice of metrics.

  • Counterparts of erasure metrics. Erasure metrics assume the concept is named in the prompt, which never holds for ICOs, so we report their counterparts:
    • 1 βˆ’ leakage for Unlearning Accuracy;
    • fidelity (the detection rate of the prompted object on the 79 non-target categories) for Retain Accuracy;
    • in addition, preservation of the target object when it is itself prompted.
  • Quality metrics. We added FD-DINOv2 (FID's FrΓ©chet distance on DINOv2 features, which agrees better with human judgment), ImageReward, CLIPScore, LPIPS diversity and Vendi. Please see our response to Reviewer giV8 W2 for a detailed table of comparison on those metrics.
  • Results. Our controls (steering, isolation, and the two combined) leave CLIPScore within 0.3 of the baseline and do not reduce diversity. The cost is moderate in ImageReward, and there is a measurable distribution shift (FD-DINOv2 to the baseline images).

(3) Goal of continuous control.

  • Why a dial. ICOs are not always unwanted. Context aids realism, and a synthetic-data pipeline may want to set a co-occurrence rate rather than remove it.
  • Mechanism. The dial is a single signed strength. On the six leak-prone prompts in SD1.5, it moves the dining-table rate monotonically from 50% to 8% at β‰₯93\geq93% fidelity (baseline 37%). The same rate needs a different strength for each prompted category, so the dial is calibrated per category.
  • Effect. Run on all COCO categories, the dial has a large range almost only for the leak-prone categories and close to none for most others. The dial modulates the ICOs a trained model already encodes in finer intervals.
Reviewer_giV8OpenReview β†—
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presets a systematic investigation of implicit co-occurring objects (ICOs) in text-to-image generation, objects that appear in generated images despite not being explicitly mentioned in the input prompt.

The authors present the paper across 7 T2I models and 80 object categories using 50k+ generated images. They investigate four potential contributing factors: training data co-occurrence statistics, text encoder representations, early denoising trajectories, and sampling dynamics. Based on these findings, they evaluate several inference-time control strategies, including isolation prompting and embedding steering, to suppress unwanted objects.

The paper also provides a theoretical analysis connecting object co-occurrence to distributional fidelity and studies the potential downstream effects of ICOs on object detection models trained with synthetic images.

Critical strengths

I think the paper provides an good study across seven T2I models with different architectures, text encoders, and sampling mechanisms. The large-scale evaluation and multiple object detection probes provide useful evidence that implicit object co-occurrence is widespread across existing generative models.

The investigation of training data, text encoders, and denoising dynamics offers interesting insights into the factors associated with ICOs. The proposed inference-time control strategies are straightforward and demonstrate meaningful reductions in unwanted object generation. The investigation of potential downstream effects also provides additional practical motivation for studying this phenomenon.

Critical weaknesses
  1. For Pinsker's inequality. I have the feeling that the mathematical argument under the assumptions, i.e. the interpretation as a fundamental limitation of T2I generation, maybe slightly overstated.

  2. Also, the paper needs to add a perceptual quality score. Also if we want to estimate fairly. Better also add a diversity and semantic consistency score.

Following 2. the downstream detection experiment also compares T2I training images against segmented objects pasted onto plain backgrounds. These datasets differ in multiple aspects beyond ICO frequency, thus a more carefully controlled comparison would be necessary to support the claimed downstream impact.

Questions for authors

See weakness

2 comments

We thank the reviewer for the valuable feedback. We are glad the reviewer found that our paper provides a good study and useful evidence on this interesting problem. Below, we address the weaknesses/questions.

Response to W1: Pinsker's inequality and the "fundamental" framing

We realize that calling this a fundamental limitation of (any) T2I generation overstates what the theory shows, and we will revise the framing to be more accurate.

Specifically, the theory shows that, by the data-processing and Pinsker inequalities, a model close to its training data cannot leak much less than the co-occurrence rate Ο€\pi in the data, and any intervention that pushes leakage below Ο€\pi must move the output away from the data by at least Ο€βˆ’β„“\pi βˆ’ \ell in total variation.

On the other hand, the bound applies to models trained on natural photographs, where objects co-occur. Thus, it is a property of the current training paradigm, not a limitation of any T2I generation. Indeed, fine-tuning on isolated-object composites (Ο€β‰ˆ0\pi \approx 0) brings leakage to 0.0% for five of the seven models (Section 6, "Changing Ο€\pi").

Revision. We will replace "fundamental property of T2I generation" in our draft with a precise statement:

Under standard training on natural photographs, a well-fit model reproduces object co-occurrence, and suppressing it at inference time leaves a non-negligible gap from the data distribution.

Furthermore, we will also state that the bound concerns one semantic dimension, object presence, not other aspects such as image quality or style, and that the back-solved Ξ΅ values are lower bounds on the KL divergence, not estimates of it.

Response to W2: quality, diversity and semantic consistency metrics

Thank you for the suggestion. Now we report the image quality, diversity, and semantic consistency of the control methods below. We will include this new analysis in the revision.

Reviewer's askMetric we addedWhat it measures
Perceptual qualityImageRewardHuman-preference quality score
Perceptual qualityFD-DINOv2 to real photosDistance from the distribution of real COCO photos: the same FrΓ©chet distance as FID, computed on DINOv2 instead of Inception features, which Stein et al. [1] found to match human judgments of image quality better
Semantic consistencyCLIPScoreImage–text agreement with the original prompt
Semantic consistencyTarget presenceRate at which the prompted object is detected
DiversityLPIPS diversityMean pairwise perceptual distance between images
DiversityVendi scoreEffective number of distinct images (DINOv2 features)

Setup. SD1.5 with dining table as the leak target, over all 80 COCO prompts Γ— 20 seeds. Each method is compared with plain SD1.5 (no control) generated with the same sampler and seeds: 20 steps for our methods, 50 steps for the comparison methods. Each method is listed under its matched baseline. FD-DINOv2 uses 3,887 real COCO photos as the reference.

MethodLeak %Target pres. %Preservation %CLIPScoreImageRewardFD-DINOv2 to realLPIPS div.Vendi
Baseline (20 steps)3.396.410024.590.2278970.7326.21
Steering (ours)0.693.310024.620.0559120.7276.72
Isolation + steering (ours)0.294.2524.29βˆ’0.07010000.7706.60
ESD2.896.410024.670.2879060.7376.06
SEGA2.296.110024.570.2959240.7426.06
UCE0.04.6018.10βˆ’2.25723640.5845.22
STEREO0.672.72521.95βˆ’0.4498630.75410.47

Preservation = rate at which a dining table is still generated when the prompt is "a photo of a dining table". Semantic consistency and diversity are preserved for our methods: CLIPScore stays within 0.3 of the baseline, and LPIPS diversity and Vendi match or exceed it. The measurable cost is in perceptual quality (ImageReward βˆ’0.17 to βˆ’0.30), which is the trade-off our theory predicts: suppressing co-occurrence moves outputs away from the data distribution. The erasure methods show the two extremes of this trade-off: ESD and SEGA preserve quality but barely reduce leakage (2.8% and 2.2%), while UCE and STEREO reach low leakage only by losing the target object (target presence 4.6% and 72.7%) and much of the image quality (ImageReward βˆ’2.257 and βˆ’0.449).

[1] Stein, George, et al. "Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models." Advances in Neural Information Processing Systems 36 (2023): 3732-3784.

Response to W2 (continued): controlled downstream comparison

Thanks for the suggestion. Below we show a more carefully controlled comparison on the downstream detection experiment.

Matched comparison: same image source, matched set sizes, ICO presence varied. Both training sets come from the same generator and the same prompt, "a photo of {c}". The ICO-present set contains images where OWLv2 detects another COCO object; the ICO-absent set contains images that are clean under both OWLv2 and RAM++. The sets are matched 1:1 on target size and instance count, with identical set sizes. Gap-of-gaps = Gap(ICO-present-trained)βˆ’Gap(ICO-absent-trained)Gap(\text{ICO-present-trained}) βˆ’ Gap(\text{ICO-absent-trained}). The test set remains the same. We report the mean over 3 training seeds. We observe a smaller gap-of-gaps in the matched comparison experiment. We will include an expanded version of the matched comparison in the revised PDF.

GeneratorDetector (LR)ICO-present R_iso / R_ctx / GapICO-absent R_iso / R_ctx / GapGap-of-gaps
Z-Image-TurboFaster R-CNN59.6 / 20.9 / 38.864.6 / 27.5 / 37.1+1.7
Z-Image-TurboYOLO11m52.9 / 16.4 / 36.551.3 / 24.3 / 27.0+9.5

More setup details:

  • New image pools. We generated new images for this experiment with the prompt "a photo of {c}" for the six leak-prone categories: 27,000 raw generations with Z-Image-Turbo (8 steps, guidance 0, 1024Β²). Each pool extends the original 1,000 seeds per category with new seeds, because clean images are rare.
  • Set assignment. ICO-present: OWLv2 (confidence β‰₯ 0.15) detects at least one non-prompted COCO object. ICO-absent: neither OWLv2 nor RAM++ finds any non-prompted object. Images containing another training category are excluded from both arms.
  • Matching. Images are paired 1:1 within strata of target instance count (1 vs. β‰₯2) Γ— 10 target-area bins, giving identical set sizes per the six categories: 2,022 per set for Z-Image-Turbo.
  • Training and test. Boxes are labeled by OWLv2 as in the original experiment. Faster R-CNN (ResNet-50-FPN, LR 0.005) and YOLO11m (AdamW, LR 0.001) are trained with 3 seeds each and tested on the same 2,408 real COCO images.
Reviewer_n9xdOpenReview β†—
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies implicit co-occurring objects (ICOs): objects generated by text-to-image models despite not being named in the prompt. The authors analyze 56,000 images across seven models and 80 COCO categories using multiple object probes. They investigate training-data associations, text-embedding geometry, denoising trajectories, and sampling settings, then evaluate inference-time controls and isolated-object LoRA adaptation. On SD1.5, isolation phrasing combined with embedding steering achieves 0.19% dining-table leakage with 94.2% prompted-object detection. The paper also derives bounds relating co-occurrence suppression to distributional divergence and evaluates downstream detection using synthetic versus isolated-composite training data.

Critical strengths
  1. Broad empirical characterization. The study covers seven generators, 80 object categories, and multiple probing methods, supplemented by threshold and agreement checks. This breadth supports the relevance of ICOs across the evaluated models and makes the empirical findings informative.
  2. Actionable investigation of control strategies. The paper connects its observations to comparisons of prompting, embedding steering, sampling changes, and training-data adaptation. Component comparisons and multi-target tests provide useful evidence about which controls work and where their limitations arise.
Critical weaknesses
  1. The theoretical interpretation is insufficiently supported. The bounds concern distributional divergence, whereas reported fidelity measures prompted-object detection. The theorem does not establish a trade-off for this metric, and back-solving the bound does not estimate KL divergence or demonstrate tightness.
  2. The downstream experiment does not isolate ICO effects. Comparing 6,000 generated images with 5,400 COCO-derived composites changes image source, backgrounds, and training-set size simultaneously. The recall differences therefore cannot be attributed specifically to ICOs.
  3. Reporting and reference inconsistencies. The abstract reports ICO rates of 34–50%, although Qwen-Image has 19.1%, and does not scope the 0.19% result to dining-table suppression on SD1.5. Three NeurIPS 2025 references are dated 2026. Section 5 broadly describes the compared methods as removing a concept from the model entirely, without distinguishing inference-time guidance, such as SEGA and SLD, from parameter-level unlearning.
Questions for authors
  1. Clarification of the theoretical claims. Are the back-solved Ξ΅ values intended as lower bounds rather than estimates of KL divergence? How do Proposition 1 and Corollary 1 support a trade-off involving the detector-based fidelity metric, particularly when isolation phrasing changes the conditioning prompt?
  2. Clarification and additional validation of downstream effects. Could you report the separate recalls on isolated and contextual test images and specify the subtraction direction? Is there a comparison using the same image source and matched training-set sizes that varies ICO presence? The recall breakdown is a clarification request; the matched comparison would provide additional validation of the causal attribution.
4 comments

We thank the reviewer for the constructive feedback and insightful questions. We are glad the reviewer found our empirical characterization and investigation of control strategies to be informative and useful. We will revise and clarify the reporting and reference inconsistencies. Below we address and discuss the questions and weaknesses.

Response to W1 / Q1: theory

Part 1: the back-solved Ξ΅ and the KL divergence.

This is a good point. We will clarify that the back-solved values are lower bounds on the KL divergence, not estimates of it. We will clarify this point in the revised draft.

On the other hand, the true DKL(Pdata(β‹…βˆ£c)β€–Pmodel(β‹…βˆ£c))D_{KL}(P_{data}(Β·|c) β€– P_{model}(Β·|c)) is extremely hard to compute: PdataP_{data} is unknown, and both distributions live in a high-dimensional image space. Proposition 1 makes this tractable for ICOs. Projecting both distributions through an object detector yields two Bernoulli distributions with measurable parameters Ο€(c, o) and β„“(c, o). The back-solved Ξ΅ β‰₯ 2(Ο€ βˆ’ β„“)Β² is thus a certificate, cheap to compute for every (prompt, object, model) triple: no model with the observed leakage rate can be closer to the data than this.

A tighter bound from the same framework. Notably, the first step of our proof, the data-processing inequality, gives another bound that is as easy to measure:

DKL(Bern(Ο€)βˆ₯βˆ₯Bern(β„“))≀DKL(Pdata(β‹…βˆ£c)βˆ₯βˆ₯Pmodel(β‹…βˆ£c))D_{\mathrm{KL}}\big(\mathrm{Bern}(\pi)\|\|\mathrm{Bern}(\ell)\big) \leq D_{\mathrm{KL}}\big(P_{\mathrm{data}}(\cdot\mid c)\|\|P_{\mathrm{model}}(\cdot\mid c)\big)

Pinsker's form 2(Ο€βˆ’β„“)22(\pi βˆ’ \ell)^2 can never exceed 2, while the Bernoulli KL has no such cap. For example, 58% of COCO photos of cake contain a dining table (Ο€\pi = 0.58). If a model shows a table in only 1% of its cake images (β„“\ell= 0.01), its KL divergence from the data must be at least 0.66 by Pinsker's bound, but at least 2.02 by the Bernoulli KL bound. If the model never shows a table (β„“\ell β†’ 0), the Bernoulli bound grows without limit.

This sharpens the trade-off. Likelihood-based diffusion training minimizes an upper bound on the forward KL: DKL(Pdataβ€–Pmodel)D_{KL}(P_{data} β€– P_{model})[1]. A model that fully suppresses an object the data contains (β„“\ell = 0 while Ο€\pi > 0) is therefore penalized without limit by its own training objective. We will add this bound to Section 6 and report both bounds in the appendix.

[1] Song, Yang, et al. "Maximum likelihood training of score-based diffusion models." Advances in neural information processing systems 34 (2021): 1415-1428.

Part 2: the bound vs. the detector-based fidelity metric.

We realize that the term "fidelity'' will cause confusion, as it can mean several things. Again, the full distance between a trained model and the real data is hard to measure. Our paper measures it along one semantic dimension: object presence. Leakage β„“(c,o)\ell(c, o) is the presence of unprompted objects. What we call "fidelity" in the original paper, Ο†(c)=β„“(c,c)\varphi(c) = \ell(c, c), is the presence of the prompted object, and we will rename it target presence to avoid the confusion. The bounds act on the presence of leaked objects. Target presence is a constraint: an intervention must keep it high, or suppression becomes trivial (a blank image has zero leakage). Object presence does not capture other dimensions in the full distribution distance, such as image quality or style, and we will clarify this clearly in the revision.

On Isolation phrasing. Rewriting the prompt cc into cβ€²cβ€² changes the model's behavior, but it does not escape the trade-off. The reference is always Pdata(β‹…βˆ£c)P_{data}(Β·|c), the photos the user asked for. Whatever an intervention does, its output is some Pintervene(β‹…βˆ£c)P_{intervene}(Β·|c), and Corollary 1 holds for any such distribution: TV(Pdata(β‹…βˆ£c),Pmodel(β‹…βˆ£cβ€²))β‰₯Ο€(c,o)βˆ’β„“TV(P_{data}(Β·|c), P_{model}(Β·|c')) \geq \pi(c, o) βˆ’ \ell. Isolation phrasing works because it routes the model to a part of the training data where co-occurrence is low, such as clean product shots in a studio scene. It is the inference-time counterpart of our "Changing Ο€\pi" experiment. The cost is the distance from typical photos of c. On SD1.5 (six leak-prone categories, 600 images per setting), isolation phrasing cuts leakage from 33.7% to 14.5% and keeps target presence at 97.2%, but raises the perceptual score of FD-DINOv2 to real COCO photos by 69. The prompted object is still there, but the image no longer looks like a typical photo of c.

Response to W2 / Q2: downstream

Recall breakdown and subtraction direction. Gap=Risoβˆ’RctxGap = R_{iso} βˆ’ R_{ctx}, the recall drop from isolated to contextual test images. Gap-of-gaps = Gap(T2I-trained)βˆ’Gap(composite-trained)Gap(\text{T2I-trained}) βˆ’ Gap(\text{composite-trained}), so a positive value means the T2I-trained detector loses more recall in context. Below is the detailed breakdown of the original experiment shown in the paper. Test set: 2,408 real COCO images (849 isolated, 1,559 contextual); recall at IoU β‰₯ 0.25; recalls in %, gaps in pp.

GeneratorDetector (LR)T2I-trained R_iso / R_ctx / GapComposite-trained R_iso / R_ctx / GapGap-of-gaps
SD1.5YOLO11m (AdamW, 0.001)55.5 / 19.9 / 35.559.7 / 31.9 / 27.8+7.7
SD1.5Faster R-CNN (0.001)81.9 / 44.1 / 37.792.3 / 67.7 / 24.6+13.1
SD1.5Faster R-CNN (0.005)69.8 / 30.5 / 39.375.9 / 54.0 / 21.8+17.5
SD1.5Faster R-CNN (0.01)54.2 / 29.8 / 24.410.4 / 4.7 / 5.6+18.8
Z-Image-TurboYOLO11m (AdamW, 0.001)50.8 / 16.3 / 34.559.7 / 31.9 / 27.8+6.6
Z-Image-TurboFaster R-CNN (0.001)74.2 / 31.2 / 43.092.3 / 67.7 / 24.6+18.4
Z-Image-TurboFaster R-CNN (0.005)42.2 / 13.9 / 28.379.0 / 55.5 / 23.5+4.8
Z-Image-TurboFaster R-CNN (0.01)27.7 / 10.5 / 17.210.4 / 4.7 / 5.6+11.5

Matched comparison: same image source, matched set sizes, ICO presence varied. Thank you for the suggestion. We now present a new comparison below. Both training sets come from the same generator and the same prompt, "a photo of {c}". The ICO-present set contains images where OWLv2 detects another COCO object; the ICO-absent set contains images that are clean under both OWLv2 and RAM++. The sets are matched 1:1 on target size and instance count, with identical set sizes. Gap-of-gaps = Gap(ICO-present-trained)βˆ’Gap(ICO-absent-trained)Gap(\text{ICO-present-trained}) βˆ’ Gap(\text{ICO-absent-trained}). The test set remains the same. We report the mean over 3 training seeds. We observe a smaller gap-of-gaps in the matched comparison experiment. We will include an expanded version of the matched comparison in the revised PDF.

GeneratorDetector (LR)ICO-present R_iso / R_ctx / GapICO-absent R_iso / R_ctx / GapGap-of-gaps
Z-Image-TurboFaster R-CNN59.6 / 20.9 / 38.864.6 / 27.5 / 37.1+1.7
Z-Image-TurboYOLO11m52.9 / 16.4 / 36.551.3 / 24.3 / 27.0+9.5

More setup details:

  • New image pools. We generated new images for this experiment with the prompt "a photo of {c}" for the six leak-prone categories: 27,000 raw generations with Z-Image-Turbo (8 steps, guidance 0, 1024Β²). Each pool extends the original 1,000 seeds per category with new seeds, because clean images are rare.
  • Set assignment. ICO-present: OWLv2 (confidence β‰₯ 0.15) detects at least one non-prompted COCO object. ICO-absent: neither OWLv2 nor RAM++ finds any non-prompted object. Images containing another training category are excluded from both arms.
  • Matching. Images are paired 1:1 within strata of target instance count (1 vs. β‰₯2) Γ— 10 target-area bins, giving identical set sizes per the six categories: 2,022 per set for Z-Image-Turbo.
  • Training and test. Boxes are labeled by OWLv2 as in the original experiment. Faster R-CNN (ResNet-50-FPN, LR 0.005) and YOLO11m (AdamW, LR 0.001) are trained with 3 seeds each and tested on the same 2,408 real COCO images.

Thank you for clarifying the KL lower-bound interpretation and the target-presence metric. The recall breakdown and same-generator, size-matched comparison improve the downstream evidence. My remaining concern is causal attribution: selecting images by ICO presence may also change visual properties beyond the matched target size and count. The reported three-seed means also leave the stability of the differences unclear. I therefore maintain my original rating.

We thank the reviewer for the follow-up. We are glad the clarifications on the KL lower bound and on target presence were helpful, and that the recall breakdown and the matched comparison improved the downstream evidence.

Seed stability. Gap-of-gaps in the matched comparison, over 3 training seeds (mean Β± std):

GeneratorDetectorGap-of-gaps (pp)
Z-Image-TurboFaster R-CNN+1.7 Β± 6.1
Z-Image-TurboYOLO11m+9.5 Β± 2.2

The YOLO11m difference is stable across seeds (95% t-interval: +4.0 to +15.0 pp). The Faster R-CNN difference is smaller considering the variance. We report both results with their standard deviations in the revised paper.

Causal attribution. We agree that selecting images by ICO presence cannot fully separate ICOs from other visual properties that come with them, such as scene clutter. The matched comparison fixes the generator, prompt, target size, instance count and set size, but not every property correlated with ICO presence. We note that, for a synthetic-data pipeline, the practically relevant quantity is the effect of generated images that contain ICOs, together with what comes with them, and this is what the matched comparison measures. We have revised Section 7 and the abstract accordingly. The downstream results are now presented as evidence consistent with an ICO-related cost, not as an isolated causal estimate.

Downstream experiment and the scope of the paper. We would like to emphasize that the downstream experiment supports the paper's motivation: that ICOs in synthetic training data can carry over to downstream detectors. The paper's main contributions, however, do not depend on it: the measurement across seven models, the analysis of the four hypotheses, the controls, and the theory.

We thank the reviewer again for the careful review, which made the paper more rigorous.

Comments on the paper

1 comment
AuthorsOpenReview β†—

General Response

We thank all reviewers for their time and constructive feedback. We have uploaded a revised PDF with all changes are highlighted in blue.

Strengths recognized by the reviewers

  • Strong motivation. "The paper's main motivation is strong", and ICOs are "worth investigating" because synthetic images are often used as training data (DLwY).
  • Broad, informative empirical study. "A good study across seven T2I models with different architectures, text encoders, and sampling mechanisms", providing "useful evidence that implicit object co-occurrence is widespread" (giV8); a "broad empirical characterization" that "makes the empirical findings informative" (n9xd).
  • Insight into origins and contributing factors. The analysis of training data, text encoders, and denoising dynamics "offers interesting insights into the factors associated with ICOs" (giV8).
  • Actionable controls. An "actionable investigation of control strategies" with "useful evidence about which controls work and where their limitations arise" (n9xd); the controls "are straightforward and demonstrate meaningful reductions in unwanted object generation" (giV8).

New experiments and analysis

  1. Quality, diversity, and semantic consistency (giV8, DLwY). We report ImageReward, FD-DINOv2, CLIPScore, LPIPS diversity, and Vendi for our controls and the erasure methods (Table A.11). Our controls preserve CLIPScore and diversity, at a moderate cost in perceptual quality. This is consistent with our theory's predictions.
  2. Matched downstream comparison (n9xd, giV8). With 27,000 generations, we built matched ICO-present and ICO-absent training sets that differ only in ICO presence. The gap-of-gaps stays positive (+1.7 / +9.5 pp).
  3. Another theoretical bound (n9xd). The data-processing step of our proof gives a Bernoulli-KL bound, and we have included the results of both bounds in the revision.
  4. Comparison with SSDV (DLwY). At matched leakage, SSDV costs more image quality than our embedding steering (ImageReward βˆ’0.69 vs. βˆ’0.17).

Revisions to the paper

  • Theory framing (giV8, n9xd). We replaced "a fundamental property of T2I generation" with a scoped statement: under standard training on natural photographs, a well-fit model reproduces object co-occurrence, and suppressing it at inference time leaves a non-negligible gap from the data distribution. We now state that the back-solved Ξ΅ values are lower bounds on the KL divergence, not estimates of it. We renamed "fidelity" to "target presence" and explain why rewriting the prompt (isolation phrasing) does not escape the trade-off.
  • Novelty and positioning (DLwY). The related work now discusses Liu et al. (2024), Roos et al. (2026), Wu et al. (2024), and Koh et al. (2025) explicitly. The introduction states that our contribution is a systematic study as a whole, and we revised the title to reflect this: Unspoken, Controllable, Unavoidable: A Systematic Study of Implicit Co-occurring Objects in Text-to-Image Generation.
  • Clarity (DLwY) and Reporting accuracy (n9xd). We have made revisions to the paper to improve clarity and consistency.

We sincerely thank the reviewers for their feedback, which has made the paper more rigorous and refined. We believe the revised paper offers a useful foundation for understanding and controlling implicit co-occurring objects in text-to-image generation, and for building more reliable synthetic-data pipelines.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.