acceptodds
Withdrawn from ICLR 2027

Scaling Properties of Text Conditioning in Visual Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We study empirical scaling properties for text conditioning in visual generation. Such properties have rarely been measured because diffusion loss does not scale with the number of tokens in natural-language prompts. Surprisingly, we find that the converged diffusion loss scales with the amount of structured language in the prompt. To quantify structured language, we adapt two complementary measures: a white-box likelihood metric (GPG) and a black-box attribute metric (ED). Across controlled training runs, the converged diffusion loss decreases approximately linearly with GPG and follows a power law with ED. Guided by these scaling properties, we improve diffusability by constructing structured prompts with semantic and geometric annotations derived from images, and improve promptability by training a prompter through supervised fine-tuning, cold-start, and verifier-gated on-policy distillation. The resulting system outperforms all evaluated open-weight models on nearly every compositional, reasoning, and world-knowledge benchmark, while matching or surpassing the strongest closed-weight models on most evaluations.

Primary area:
generative models
Keywords:
Scaling laws for image generation.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BtkMOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies text conditioning for text-to-image diffusion and argues that caption information, rather than caption length, is the relevant axis for scaling. It introduces structured prompts (SP), a typed JSON caption interface, and two measures of caption informativeness: Grounded Perplexity Gain (GPG) and Effective Detailness (ED). In a 15-configuration BAGEL sweep, the authors calibrate converged diffusion MSE against GPG and ED to characterize text-conditioning scaling within that setup. They frame generation quality as Diffusability x Promptability: SP annotations improve diffusability, while an LLM prompter trained to map user requests to SPs improves promptability. The evaluation includes controlled caption sweeps, held-out spatial/field variants, matched natural-language retraining on Qwen-Image, prompter-training ablations, and an inference-time refine-render-judge loop. The paper reports results on GenEval, GenEval2, DPG-Bench, TIIF, WISE, and T2I-CoReBench. The evaluation spans several settings, but the scaling claim remains specific to the tested recipe, and proprietary components limit reproducibility.

Critical strengths
  1. The controlled design tests whether caption organization matters independently of caption length. It fixes the backbone and training budget, evaluates 15 caption configurations using two complementary metrics, tests held-out variant prediction, and includes matched NL retraining. Together, these results support the claim that caption organization contributes beyond caption length.

  2. The end-to-end evaluation spans GenEval, GenEval2, DPG-Bench, TIIF, WISE, and T2I-CoReBench, and includes matched controls. The SP system outperforms an NL interface trained with the same Qwen-Image architecture, data, stages, and budget. This comparison makes additional training alone a less plausible explanation for the gains.

Critical weaknesses
  1. The scaling claim extends beyond the evidence. The reported properties are fitted to a single BAGEL sweep of 15 configurations, with each configuration trained once, and are explicitly dependent on the judge, extractor, and training recipe. Qwen-Image is not re-fitted, and the held-out variants come from the same annotation suite. These results do not yet justify treating the fitted properties as general.

  2. The study provides limited evidence on reproducibility and confound control. The pipeline relies on proprietary VLMs and APIs, a Qwen3.5-397B prompter, and large annotation infrastructure. Corpus size, annotation quality, judge variance, and seed-level training variance are not reported in sufficient detail. GPG and ED also depend on frozen judges and extractors, leaving the central calibration without independent validation.

Questions for authors
  1. Can the authors evaluate whether the GPG/ED calibration transfers across backbones, for example, Qwen-Image or another DiT family? If the coefficients do not transfer, how should the claim about scaling properties be scoped?

  2. What are the exact training corpus sizes, the number of L10 SPs, and the annotation quality statistics?

  3. For the key BAGEL sweep points, can the authors report run-to-run or seed-level variance and compare it with the reported held-out MAEs of 5.0e-4 and 8.1e-4?

  4. Are GPG/ED evaluated using judges and extractors held out from the calibration fit? Can the authors report results on an independent validation set, or at least assess judge robustness for the end-to-end benchmark gains?

  5. Can the authors add human evaluation or a stronger judge-robustness analysis for the main benchmark comparisons, given that most gains are measured by automated judges?

Reviewer_XKS6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the text conditioning of visual generative models. The authors’ question is mainly about structured prompts (SP) vs. natural language (NL) prompts. With length-matched settings, empirical findings demonstrate that SP is better at both training and inference. The authors explain this as scaling properties of text conditioning, using grounded perplexity gain and effective detailness. Given these arguments, the authors propose both inference-time and training-time techniques that leverage SP. Experimental results demonstrate that the proposed method outperforms the base text-to-image generative models.

Critical strengths

The authors analyzed the effect of information in text conditioning on visual generative models. This has been implicitly demonstrated in prior art (e.g., multi-level captioning for text-to-image/video training), as well as by Gutflaish et al. (2025) and Fan et al. (2025), but analyses using GPG and ED have not been presented before.

Critical weaknesses
  • Novelty. The authors state that they introduce the structured prompt in L151–L153. However, at least two previous works have proposed some form of structured prompts. These works already demonstrate that structured captions help include fine details.
    • Gutflaish et al., “Generating an Image From 1,000 Words: Enhancing Text-to-Image With Structured Captions” arXiv:2511.06876
    • Fan et al., “InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Caption” CVPR’25.
  • The authors should clearly separate “structured” vs. “natural” and token length vs. information. These axes are silently merged across experiments. It is unclear whether “structured” is better than “natural,” or whether detailed information is more important than the number of tokens.
  • The scaling law is tested using a single model. It may not hold for other models. To demonstrate a general scaling law, it should be tested using multiple model families, including LLMs with visual decoders, MMDiT, and cross-attention models.
Questions for authors
  • Have you tested converting structured prompts to natural language? Or vice-verse. This may use fewer or similar number of tokens. To claim “structured” vs. “natural,” this comparison is necessary. Could you add NL-from-SP and SP-from-NL experiments?
  • Could you give examples of SP and NL given token counts?
  • Does the “Ours” row in Table 1 include CoT? If so, did the baseline (Qwen-Image) also use CoT?
Reviewer_bgpZOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors first study how text conditioned image generation scale relative to textual information. They used 2 metrics the likelihood gain from in caption using the image (GPG) and Effective Detailness (ED) - the precision and recall of caption attributes. They show that adding fields to a structured prompt (increasing information) improves generation compared to just increasing text/prose length without adding any information. The diffusion loss falls linearly in GPG and power law in ED.

Based on structured text, the authors then splits generation quality into two parts:

  • Diffusability captures how much information is present in structured text, where increasing the richness improves generation during training. To improve this, they annotate images into structured json with global, per element and cross element fields using VLM.

  • Promptablity is about how well a user prompt can be converted into a structured json using LLM. They improve it by scaling the model and training it in three stages - SFT, cold-start, and distillation from a teacher that can see the image.

The authors showed their model performs better then all the models they compared with, including GPT-Image-1 and Nano Banana on multiple datasets.

Critical strengths
  • The authors clearly addressed an important question: scaling information in text for better image generation
  • Splitting the problem into diffusability and promptability makes it easy to study and reason about them separately. The prompter training is well thought out, especially the teacher that can see the image.
  • Very thorough analysis and great results. They are able to match or beat GPT-Image-1 and Nano Banana
Critical weaknesses

None

Questions for authors
  • In Table 4 - structured L10 has much higher GPG and ED compared to NL dense L10. Why is that?
  • Is it structure information or just information richness? If we convert Structure L10 into a dense prompt with same GPG/ED, the performance could be same.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.