acceptodds
Withdrawn from ICLR 2027

COLOR ANNOTATIONS IN TEXT-TO-IMAGE PRETRAINING: AN EMPIRICAL STUDY

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Numerical color annotations provide a route to precise color control in text-to-image pretraining, but how their design affects generation with and without explicit color inputs remains unclear. We investigate this question by training diffusion transformers from scratch on a shared BLIP3-o subset, keeping the architecture and training hyperparameters fixed. We compare region-level palettes extracted from image pixels with object colors predicted jointly with color-enriched descriptions. With benchmark colors explicitly supplied, our pixel-annotated model reaches 86.97% accuracy on the evaluated GenColorBench numerical-color task. Its scores exceed published NumColor results in the three color categories compared, under the respective evaluation settings. Within our palette-trained configurations, pixel-extracted annotations give the strongest supplied-color fidelity, including prompts requiring multiple colors on multiple objects. This ranking reverses when palettes are withheld from matched prompts while descriptions, target colors, and scoring remain fixed. The pixel-annotated model also gains more in image quality from receiving palettes, whereas predicted annotations support stronger texture binding and compositional generation. These results demonstrate the effectiveness of color-annotated pretraining for precise numerical control. They also show that, in this comparison, the strongest control of supplied colors accompanies the greatest palette dependence and does not translate into the strongest color generation from descriptions.

Primary area:
generative models
Keywords:
Image Generation, Diffusion Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8BXNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents an empirical study of color annotations in text-to-image pretraining. The authors compare two annotation strategies: Pixel, which extracts region-level color palettes using K-means clustering, and Predicted, which employs a fine-tuned multimodal captioner to predict object colors and generate color-enriched descriptions. An additional Predicted-Leaf configuration is introduced to examine the effect of reduced palette exposure.

Critical strengths
  1. The paper investigates a practically relevant problem concerning the role of color annotations in text-to-image pretraining.

  2. The experiments cover several aspects of color generation, including numerical color fidelity, object-color binding, compositional generation, and sensitivity to color conditions.

Critical weaknesses
  1. Limited methodological novelty and insufficiently developed research contributions. The proposed annotation strategies largely rely on established techniques, including K-means clustering for palette extraction, multimodal captioning for semantic color descriptions, and structured conditioning for diffusion models. The paper does not introduce a substantially new learning algorithm, architecture, or color representation mechanism.

  2. The experimental comparisons do not isolate the effects of color annotation strategies. This is my primary concern. The Pixel and Predicted configurations differ in several important factors beyond how color palettes are obtained. Specifically, their textual descriptions, frequency of explicit color words, palette lengths, and potentially their region layouts are different. For example, Table 7 shows that the percentage of descriptions containing color words increases from 59.6% in Pixel to 82.8% in Predicted, while the average number of palette entries per element decreases from 4.98 to 1.86. These differences can independently affect both language-conditioned generation and numerical color accuracy.

  3. The conclusions regarding palette dependence are not sufficiently convincing. One of the central findings is that the Pixel configuration performs substantially worse when numerical color annotations are removed. However, this observation is not particularly surprising because the model is trained to exploit precise numerical color conditions.

Questions for authors

Can the authors provide controlled ablations in which the region layouts, descriptions, palette lengths, and color-word frequencies are fixed, while only the palette extraction strategy changes?

Reviewer_fwJ2OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates how different policies for adding color annotations to training captions during T2I pre-training affect the performance and behavior of the resulting generative model. The paper includes an analysis based on experiments conducted on a training subset of BLIP3-o.

Critical strengths

To me, this is a timely and important study. In particular, style elements such as color can be ambiguous to handle using only text prompts, so I think how such information is annotated in the dataset from the pre-training stage is important. The paper also provides a well-controlled comparison for this analysis, and to the best of our knowledge, studies on such data design from the pre-training stage are relatively rare. This is also a practically useful study.

I think the findings could be very useful for color-aware image generation applications as well as application-side settings such as post-training, personalization, and editing.

Many experiments are conducted, allowing us to observe the results under various settings.

Critical weaknesses

The actionable guidance from the perspective of actually designing the data is somewhat weak. The conclusion mainly remains at the trade-off that PIXEL is better for numerical control while PREDICTED is better for description-based generation. However, it does not answer how the two should be combined. The STAGED setting in the appendix, which uses a curriculum from PIXEL to PREDICTED, seems to directly address this question, but not very clear for the dataset preparation side.

Although I understand the resource limitations of the experiments, the generality is still somewhat limited. There is no analysis of how the observations change with model or data scale, which limits the scope of the conclusions. But still, the results look convincing for the scale considered in this work.

Overall, I think the paper can be difficult to follow for readers who are not already familiar with this field. In particular, there are many different settings and configurations, which makes the paper somewhat hard to read. I think the presentation and organization could be improved overall.

Questions for authors

See the first point of weaknesses. I am curious about the actionable guidance for building such color-award data annotation.

Reviewer_qTRgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how adding HEX color codes to the captions used for text-to-image pretraining affects color generation. The authors train the same diffusion transformer from scratch on a 13M BLIP3-o image subset with structured JSON captions, and compare two ways of making the color palettes. PIXEL takes k-means clusters directly from the image pixels in each box, and PREDICTED uses a fine-tuned Qwen3.5-4B captioner that predicts the object's own colors along with color-rich descriptions (a third variant, PREDICTED-LEAF, uses only leaf-level palettes). They validate on GenColorBench, T2I-CompBench++ and PRISM, testing with and without palettes and under palette perturbations. The main claim is that the annotation giving the best exact color control also depends most on getting explicit colors.

Critical strengths

The biggest strength is the setup. The authors train full models from scratch with the same architecture, data and hyperparameters, and include a NOCOLOR baseline trained the same way. That's expensive, and it means the differences between models come from the annotations rather than from different training setups, which is what makes the paper worth considering at all.

The second strength is testing the same prompts with and without palettes. This shows that the model with the best color accuracy (PIXEL) is also the one that breaks most when colors are not given, which a normal color benchmark would never show. This trade-off is the most useful finding of the paper for anyone building color-controllable models, and the paired bootstrap CIs make it fairly convincing.

Critical weaknesses
  1. PIXEL’s training palettes come from k-means (K=5) on the pixels, and the scorer also picks the biggest k-means (K=5) cluster to check the color. So PIXEL is basically trained to produce exactly what the scorer measures. PREDICTED learns the object’s real color, which can look different from the biggest pixel cluster because of lighting and shadows. So I’m not sure the PIXEL win is really better color control or just this match. Can the authors re-check with a different color extractor (for example a chroma-weighted median or a VLM judge) and show PIXEL still wins?

  2. authors didn't show how the generated data samples looks like ? Section 3.1 and Appendix A describe the fields in words little bit, but it does not clear the how the samples look like ? They didn't describe how the generated 6M samples looks like.

  3. Authors didn't describe what prompt they used to create the 6M samples description using Gemini 3.5 Flash-Lite? Appendix B.1 only says the records came “from our annotation pipeline”. The source of the 6M images, how the layout records were made, and the Gemini prompt are all missing.

  4. Authors make no promise to release the codebase, models, dataset or others things to reproduce their work upon acceptance. No code in supplementary or any anonymous files.

  5. There is no accuracy check of the Qwen captioner or the Gemini targets.

  6. As the authors build their model from scratch and train it which is very big. So, authors should clearly describe the thing like - the text encoder, autoencoder, model size and training hyperparameters and other details which are unspecified. Each configuration has one training run and the T2I-CompBench comparison uses one seed with no CIs.

Overall, paper has reproducibility concerns.

Questions for authors

see the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.