acceptodds
Under review as a conference paper at ICLR 2027

Probing the Generative Worldview: Uncovering Implicit Visual Priors in T2I Models

Abstract

Text-to-image (T2I) models must resolve a fundamental challenge: natural language is radically underspecified compared to the visual world. A prompt like “a man waiting on the side of the road” only conveys a handful of visual attributes, yet the T2I model must commit to many specific choices across different visual dimensions such as weather, lighting, and clothing to depict the scene. To characterize how models fill in what is left unsaid, we introduce implicit visual priors, the systematic tendencies T2I models exhibit over unspecified visual attributes, and argue that previously studied biases are a subset of this broader concept. Consequently, we propose ProbeGen, a bottom-up framework that discovers these priors directly from generated images without pre-specified classes or hypotheses. Unlike top-down approaches that anticipate potential biases, our observation-driven method surfaces priors across diverse visual dimensions. Through extensive experiments, we find that implicit visual priors are highly pervasive, with models imposing multiple simultaneous defaults and resolving the same underspecification in measurably distinct ways.

Primary area:
generative models
Keywords:
Text-to-Image Generation, Implicit Visual Priors, Visual Attribute Extraction, Model Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nZdVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The draft presents a method for implementing implicit priors for a T2I model to compensate for missing information in the text prompt. Through extensive experiments, the authors find that implicit visual priors are highly pervasive, with models imposing multiple simultaneous defaults and resolving the same underspecification in measurably distinct ways.

Critical strengths

None

Critical weaknesses

None

Questions for authors

Although the authors described some of the limitations of this method, such as the method’s perceptiveness being bounded by the VLM used for attribute extraction, it should be clearly addressed: 1) what the practical limitations are concerning large test datasets, in contrast to the very limited dataset that the authors used, and 2) what the theoretical foundation of this work is and its limitations.

1 comment

We thank the reviewer for these questions regarding dataset scalability and theoretical grounding. Below is a point-by-point response:

[Q1] what the practical limitations are concerning large test datasets, in contrast to the very limited dataset that the authors used

We would like to clarify that our evaluation does not rely on a limited dataset. While our main benchmark comprises 1,0001{,}000 prompts, each prompt is selected as a K-Means centroid on text embeddings across five distinct supercategories to maximize semantic coverage, and sampling m=20m=20 seeds per prompt yields 20,00020{,}000 images per main model (48,00048{,}000 images total across all evaluated models). Algorithmically, the main stages of our pipeline execute independently per prompt, allowing them to parallelize trivially and scale linearly as O(n⋅m)O(n \cdot m) with prompt count nn; thus, ProbeGen faces no method-specific bottleneck on larger datasets, and the compute budget grows naturally.

[Q2] what the theoretical foundation of this work is and its limitations.

Formally, ProbeGen estimates a generative model's conditional distribution over unentailed visual factors. Given an input prompt tit_i, sampling random seeds induces a conditional image distribution PG(I∣ti)P_G(I \mid t_i), and we quantify the resulting unentailed attributes a∉Entailed(ti)a \notin \text{Entailed}(t_i) (Eq. 1) empirically via f(a∣ti)f(a \mid t_i) (Eq. 2) and PG(i)P_G^{(i)} (Eq. 7). Prevalent implicit visual priors correspond to high-probability modes over these unentailed attributes, while Lift and JSD/JkJ_k capture intra-prompt attribute dependencies and cross-model distributional divergence, respectively. By design, ProbeGen characterizes what a model internally defaults to without benchmarking against a real-world or uniform reference distribution. Consequently, it does not attempt to separate benign defaults and co-occurrences from harmful stereotypes, motivating our broader framing around "implicit visual priors."

We will incorporate these clarifications into the final version of the paper.

Reviewer_5NP4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines “implicit visual priors” as visual attributes that are filled in by the image generation model when the prompt used to generate the image is underspecified. The authors argue that existing open-set bias methods are top-down and propose a bottom-up method that discovers priors directly from generated images called ProbeGen. First, they generate 20 seeds per prompt. Then Gemma 4 12B captions each image and extracts the attributes from the image.

Then an NLI model refines those attributes by:

  1. removing attributes the prompt already contains
  2. splitting the remaining attributes into skip-bigrams
  3. re-checking the pooled attributes against every caption
  4. merging the synonyms, and
  5. propagating hypernyms.

They run this on 1,000 MS-COCO captions across five domains using SDXL and FLUX.1-schnell. In addition, they also show with SD3.5-L and Z-Image Turbo on 200 People prompts. The paper finds that a prompt has ~6 strong priors on average and that the two models diverge. Using the top 3 Food priors as negative prompts raises pairwise diversity by 11%.

Critical strengths
  1. The question asked in the paper is very interesting from both interpretability and actionable perspectives. The writing is pretty clear too.

  2. There is a lot of engineering effort in designing the main method, with all the filters for prompt attributes, merging synonyms, etc. The authors give exact premise and hypothesis formats, thresholds and VLM prompts, which helps with reproducibility.

  3. The small subsection on an actionable application is also appreciated, the paper could be improved more by showing more evidences of this kind.

Critical weaknesses
  1. There is a major overlap of findings with GRADE (Rassin et al. 2024), that also study the "default behaviors" (or as the current paper terms it, "implicit visual priors") in T2I models when prompted with underspecified details. The major difference as I understand is that ProbeGen discovers the attributes from captions, while GRADE has an LLM propose the axes. Also, ProbeGen extends this to scene-level attributes and co-occurrence. Citing GRADE and positioning against the rich prior literature would improve the paper.

  2. The paper does not show how bottom-up discovery finds priors that are unique and are not found in top-down approaches. The authors suggest that “this bottom-up approach discovers priors that no LLM would anticipate” . This is not shown in the paper with an evidence or experiment to back up the claim. The direct test would be to provide only the prompt to an LLM and ask it to predict the most likely unspecified attributes. Then report what fraction of ProbeGen’s strong priors this LLM misses. This claim is very interesting and would very much benefit with an experiment to back it up.

  3. FLUX.1-schnell is timestep-distilled, and SDXL is a base model run with CFG. The paper claims “models capable of higher image fidelity may actually enforce a narrower set of implicit priors.” but this can also be due to “distilled models losing diversity.” This diversity drop from time step distillation is documented in prior work, for example Gandikota et al. (2025), "Distilling Diversity and Control in Diffusion Models", plus DMD and LADD literature.

  4. The diversity gain measured by Dinov2 needs some control baselines. For example, a random negative prompt and a generic diversity method like CADS or LLM prompt expansion. In addition to it, some more metrics would benefit the story: what is the prompt alignment like (CLIPScore or VQAScore); what is the quality? (maybe some human preference scores or FID). Prompt alignment seems really important to measure as hacking diversity is easy when the prompt is no longer followed.

  5. I understand that Figure 3 is to that scale to preserve the page limit. But this makes it really difficult to parse the figure. This is not a critical weakness in any sense, just a suggestion for a more relaxed paper limit of 10.

Questions for authors
  1. What fraction of strong priors does an LLM predict from the prompt alone?

  2. Why does the captioning prompt ask for racial cues, and how do the People results change without that instruction?

  3. What happens if you replace Gemma with a different VLM? This would show which priors belong to the generator and which to the extractor.

2 comments

We thank the reviewer for their constructive feedback and insightful suggestions. Below is a point-by-point response:

[W1] There is a major overlap of findings with GRADE (Rassin et al. 2024), that also study the "default behaviors" (or as the current paper terms it, "implicit visual priors") in T2I models when prompted with underspecified details. The major difference as I understand is that ProbeGen discovers the attributes from captions, while GRADE has an LLM propose the axes. Also, ProbeGen extends this to scene-level attributes and co-occurrence. Citing GRADE and positioning against the rich prior literature would improve the paper.

We thank the reviewer for bringing GRADE (Rassin et al., 2024) to our attention. We will cite and position against GRADE in Related Work in the final version of the paper. As the reviewer noted, unlike GRADE, ProbeGen does not rely on an LLM to pre-specify concepts or attribute axes; instead, it discovers open-ended priors directly from generated images. Additionally, ProbeGen characterizes joint visual templates via co-occurrence analysis, quantifies cross-model divergence and overlap, and demonstrates downstream actionability by using discovered priors to improve semantic diversity.

[W2] The paper does not show how bottom-up discovery finds priors that are unique and are not found in top-down approaches. The authors suggest that “this bottom-up approach discovers priors that no LLM would anticipate” . This is not shown in the paper with an evidence or experiment to back up the claim. The direct test would be to provide only the prompt to an LLM and ask it to predict the most likely unspecified attributes. Then report what fraction of ProbeGen’s strong priors this LLM misses. This claim is very interesting and would very much benefit with an experiment to back it up.

[Q1] What fraction of strong priors does an LLM predict from the prompt alone?

We kindly refer the reviewer to Appendix A5, where we evaluate this exact question by comparing ProbeGen against OpenBias (i.e., a representative top-down approach that queries an LLM with only the prompt to propose unspecified bias). Quantitatively, OpenBias proposes and detects an average of 2.2 distinct attributes per prompt, whereas ProbeGen uncovers 6.1 distinct strong priors (f≥0.8f \ge 0.8) per prompt. Qualitatively, Table A3 illustrates this difference: for "A picture of a person on the side of the road," the top-down LLM only anticipates human-centric categories (Person Gender, Person Clothing, Person Emotion), while missing all tonal, compositional, and scene-level defaults surfaced by ProbeGen (Grayish tone, Two-lane road, Grassy areas on both sides, Sense of scale and solitude).

[W3] FLUX.1-schnell is timestep-distilled, and SDXL is a base model run with CFG. The paper claims “models capable of higher image fidelity may actually enforce a narrower set of implicit priors.” but this can also be due to “distilled models losing diversity.” This diversity drop from time step distillation is documented in prior work, for example Gandikota et al. (2025), "Distilling Diversity and Control in Diffusion Models", plus DMD and LADD literature.

We thank the reviewer for raising this point. We refer the reviewer to Appendix A3 (Table A1), which evaluates an undistilled base model (SD-3.5 Large) alongside an aggressively distilled model (Z-Image Turbo), showing that undistilled SD-3.5 Large already concentrates priors more strongly than SDXL, and timestep distillation further amplifies mode concentration. In the final paper, we will revise Section 4.3 to qualify this claim and cite Gandikota et al. (2025) along with the DMD/LADD literature.

[W4] The diversity gain measured by Dinov2 needs some control baselines. For example, a random negative prompt and a generic diversity method like CADS or LLM prompt expansion. In addition to it, some more metrics would benefit the story: what is the prompt alignment like (CLIPScore or VQAScore); what is the quality? (maybe some human preference scores or FID). Prompt alignment seems really important to measure as hacking diversity is easy when the prompt is no longer followed.

We thank the reviewer for these helpful suggestions. We clarify that Section 4.4 is intended as a lightweight proof-of-actionability rather than a specialized diversification method competing with CADS or LLM prompt expansion. In fact, ProbeGen is complementary to such approaches by explicitly identifying which implicit defaults a model exhibits. As illustrated qualitatively in Figure 4, negative-prompting the discovered defaults breaks the white-plate template while preserving prompt adherence and visual quality. In the final paper, we will further supplement Table 4 with prompt-alignment (e.g., CLIPScore/VQAScore) and image quality metrics.

[W5] I understand that Figure 3 is to that scale to preserve the page limit. But this makes it really difficult to parse the figure. This is not a critical weakness in any sense, just a suggestion for a more relaxed paper limit of 10.

We thank the reviewer for this helpful suggestion. We agree that Figure 3 is dense and will reformat it for better readability in the final version of the paper.

[Q2] Why does the captioning prompt ask for racial cues, and how do the People results change without that instruction?

We thank the reviewer for this question. We included that instruction because VLMs often omit visible racial or ethnic descriptors by default to remain neutral on sensitive attributes. Without this instruction, ProbeGen would miss genuine demographic defaults that the T2I model imposes (such as East Asian features in Section 4.2) and could not be meaningfully benchmarked on race against demographic classifiers like FairFace (Appendix A4).

[Q3] What happens if you replace Gemma with a different VLM? This would show which priors belong to the generator and which to the extractor.

We thank the reviewer for this insightful question. Three existing results show that our discovered priors reflect T2I model behavior rather than purely VLM artifacts. First, holding the VLM fixed still yields substantial cross-model divergence, with mean JSD=0.488\text{JSD} = 0.488 between SDXL and FLUX.1 (Table 3) and JSD=0.504\text{JSD} = 0.504 between Z-Image Turbo and SD-3.5 Large (Appendix A3). This would not occur if the extracted priors were dominated by Gemma's own defaults. Second, our extractions align with independent non-Gemma evaluators, including FairFace (Table A2) and OpenBias (Table A4). Third, negative-prompting the top-3 discovered priors yields a +11.12%+11.12\% diversity gain measured via DINOv2 (Table 4), confirming they capture real visual defaults. Nevertheless, as noted in our Limitations, extraction remains bounded by the VLM, and we will add a cross-VLM comparison on a representative subset in the final version of the paper.

Reviewer_Pgf6OpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigets how text-to-image (T2I) models render those visual contents that are not specified in text prompts. The problem is interesting and important for understanding T2I models.

The paper finds that some attributes that are not specified in short prompts repeats even when changing the seeds. Also, some categories (e.g., food) share common attributes.

However, the number of tested models is small and relatively outdated. This issue is critical since this paper is for reaching some conclusions that are expected to be valid for the text-to-image paradigm.

Critical strengths
  1. The problem itself is interesting. Observing, understanding, and auditing T2I models from this perspective is important.

  2. The finding that suppressing discovered defaults can improve visual diversity is interesting. ( However, this may be not pratical considering that modern T2I models typically use reprompt/recaption to expand the length of user prompt).

Critical weaknesses
  1. The number of tested models is small and the selected models are relatively outdated.

  2. Modern T2I models are mainly pretrained on dense captions, rather than the short prompts listed in the paper. Moreover, short user prompts are commonly transformerd to dense caption, namely recaption or reprompt, for better generation results. This paper does not contextualize the paper position and its short-prompt evaluation setting in relation to dense-caption training and prompt rewriting.

  3. The empirical findings are mostly expected. T2I models share defaults, different models favor different priors, and suppressing common defaults increases diversity. While systematically auditing these "bias" is useful, the analysis offers limited insight into when and why they arise. This weakness is also highly related to the second weakness.

Questions for authors
  1. How do the findings change when short prompts are rewritten or expanded into detailed descriptions using common prompt rewriting models/methods?
1 comment

We thank the reviewer for their thoughtful review. Below is a point-by-point response:

[W1] The number of tested models is small and the selected models are relatively outdated.

We thank the reviewer for this feedback. We clarify that ProbeGen is proposed as a model-agnostic auditing framework rather than an exhaustive model benchmark. In addition to SDXL and FLUX.1-schnell in the main text, Appendix A3 evaluates two more recent architectures: SD-3.5 Large and Z-Image Turbo (a late-2025 single-stream DiT model). Across these four distinct architectures (48,00048{,}000 total images), implicit visual priors do not disappear in newer models. In fact, they intensify, with People MCmeanMC_{\text{mean}} and Nstrong, meanN_{\text{strong, mean}} rising from SDXL (0.9450.945 / 4.74.7) to SD-3.5 Large (0.9650.965 / 6.36.3) and Z-Image Turbo (0.9890.989 / 10.010.0). To make this clearer, we will highlight the Appendix A3 models more prominently in the main text in the final version of the paper.

[W2] Modern T2I models are mainly pretrained on dense captions, rather than the short prompts listed in the paper. Moreover, short user prompts are commonly transformerd to dense caption, namely recaption or reprompt, for better generation results. This paper does not contextualize the paper position and its short-prompt evaluation setting in relation to dense-caption training and prompt rewriting.

We clarify that our evaluation uses natural-language scene descriptions (MS-COCO captions), reflecting how users realistically prompt T2I models. Regardless of whether models are trained on dense captions, natural prompts remain underspecified across visual dimensions such as camera angle, lighting, and color palette, which is the setting ProbeGen is designed to audit. Furthermore, evaluating generators directly without an LLM rewriter isolates the T2I model's intrinsic behavior rather than conflating it with a rewriter's textual defaults. Prompt rewriting does not eliminate implicit priors either; it merely shifts those assumptions to the rewriter, and ProbeGen applies equally to such end-to-end pipelines.

[W3] The empirical findings are mostly expected. T2I models share defaults, different models favor different priors, and suppressing common defaults increases diversity. While systematically auditing these "bias" is useful, the analysis offers limited insight into when and why they arise. This weakness is also highly related to the second weakness.

While high-level intuitions about model defaults may seem natural in hindsight, ProbeGen makes them concrete and reveals non-obvious patterns. Specifically, our analysis quantifies the extent of these constraint (e.g., 5–7 strong priors per prompt on average in SDXL/FLUX.1), reveals co-occurrence templates, and show that models resolve the same underspecification very differently, e.g., sharing a median of only one top five prior in People and Vehicles (J5=0.111J_5 = 0.111) while converging on similar defaults in Food and Furniture. As for "when and why they arise," ProbeGen is by design a black-box auditing framework, since attributing these priors to their origins requires access to training data, curation pipelines, and post-training details that are undisclosed for most modern T2I models.

[Q1] How do the findings change when short prompts are rewritten or expanded into detailed descriptions using common prompt rewriting models/methods?

Evaluating rewritten prompts audits an end-to-end system (Rewriter + T2I model) rather than isolating the intrinsic visual priors of the image generator itself. Fundamentally, prompt rewriting does not eliminate implicit visual priors; it merely shifts these assumptions to the language model. When an LLM expands a prompt, it must make concrete choices about unspecified visual dimensions (e.g., injecting specific demographics), which the T2I model then renders. In practice, our framework can audit such pipelines directly without modification. In such a setting, the discovered priors would reflect the joint defaults of both the rewriter and the image generator.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.