acceptodds
Under review as a conference paper at ICLR 2027

Text-to-Image Generative Modeling Without Language Pretraining

Abstract

Modern text-to-image (T2I) systems use internet-scale language pretraining for semantic conditioning. We ask whether language pretraining is necessary for strong T2I generation. We introduce **Xanthus**, a family of early-fusion multimodal autoregressive models pretrained entirely from scratch on GPIC, a dataset of 100M image-text pairs. **Xanthus-0.52B** outperforms LlamaGen-XL/8 (**48.0** vs. 63.5 gFD) with **18.2×** less pretraining compute. Further, **Xanthus-1.6B** outperforms JiT-T2I-XXL/8++ (**35.3** vs. 43.2 gFD) with **275×** less pretraining compute. Controlled data composition and scheduling experiments show that external text-only pretraining does not improve T2I generation. Under the same pretraining budget, **Xanthus** also learns image captioning from the paired image-text data while maintaining competitive T2I generation quality. We show that strong T2I systems can be trained from scratch without internet-scale language pretraining.

Primary area:
generative models
Keywords:
text-to-image generation, multimodal autoregressive modeling, language pretraining, multimodal pretraining, image generation, generative modeling

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_Xn4QOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a systematic analysis of the order in which training data is provided for training a unified model (without any language pretraining) for text-to-image generation. The paper found that unified baseline with prediction for both image and text together at each training step is better or on par for text-to-image generation task than other handcrafted tricks like: language pretraining, staged training (image-to-text first then text-to-image) or mixed training (some percentage of batch involved in image-to-text and some for text-to-image). However, mixed training do help for text generation tasks and can be thought of as an alternative. The authors did an adequate job in providing proper ablations and provide multiple models which show strong results.

Critical strengths
  1. The paper analyses quite well the order of data flow to a unified early fusion model. The results and analysis are quite interesting and is really relevant for the community.
  2. One of the great insights from the paper is about mixing text-to-image and image-to-text in the same batch: it helps in captioning and staging with I2T and T2I helps in generation (on par with baseline ordering). This is quite interesting!
Critical weaknesses
  1. I really appreciate the author’s effort in training a new 1D tokenizer for images. However, I feel that the comparison with the prior Cosmos tokenizer is a bit unfair in Table 1, as the paper trained its tokenizer on GPIC (unlike Cosmos) and tested on GPIC. The distribution gap is supposed to be greater, and Table 1 actually shows something obvious. It is also the same with the Text Tokenizer and Table 2; the things shown are quite obvious and don't add much to the paper. However, for both image and text tokenizers, the text seems to be quite over the top with suggestions of better performance due to some sort of cool improvements in training. I really think that these need to be toned down. Otherwise, I really appreciate the effort to train new tokenizers for both images and text!
  2. I really like the ablations of different ways of providing data to the model. However, I am not so sure I understand the color scheme for the horizontal bands in Figure 4. For example, in Staged: I2T then T2I, I am not so sure why the I2T part has dark orange and light green. Shouldn’t it be dark and light orange bands instead? Additionally, I am not so sure about the bands for Baseline T2I, too. A clarification regarding this figure is very important to improve the readability of the paper.
  3. Continuing with the previous point of data ordering for training, I was wondering if the authors explored changing the λimg\lambda_{img} value when using mixing. I would assume a higher value must be used when a higher amount of I2T is used. This might be one of the reasons why the conclusion Staging favors generation; mixing favors captioning is found. It would be super nice if the authors could comment on this! I totally understand the time limitations of the rebuttal, and as such, simple experiments with text-only explanations are more than enough!
  4. I really like that the authors went out of their way to train with language pretraining followed by T2I. However, I am not quite sure to understand why the authors used ClimbMix for the language pretraining. Given the main intention is to train a T2I model on GPIC, why is the language pretraining also not done on GPIC captions? I would assume that would have been the natural step. Again, I totally understand the constraints of the rebuttal, and simple experiments with text only explanations suffice as plausible answers.
  5. The use of perceptual LPIPS is quite common now to improve JiT baselines. I appreciate that the authors also included it in their pipeline. However, I was wondering, given it's a unified model with language information embedded, why a perceptual loss based on some external language models was not adopted too! I would be really curious to know about this from the authors.
  6. I really liked exploring the ideas of the paper and found the ablations quite interesting. However, I must admit it was a bit difficult to read with all the bullet points at places making me scrape for relevant information at times. Also the organization of the results is super difficult for me to parse. I would have preferred to have the ablations slowly building towards the final results and a proper story to link everything together; however, here it feels like a collection of results for me to figure out what they are, which should not be the case. I really encourage the authors to improve the writing and make it easy for future readers.
Questions for authors

As mentioned before I liked the ideas proposed in the paper, and I do think the ablations are quite good. However, apart from the difficulty in parsing the content of the paper, there are a few things that need a bit of clarification.

(Note: I totally understand the restrictions of the rebuttal, and simple text explanations and short experiments are more than enough)

  1. I really like the ablation of the dataset order. However, I really feel that the way Figure 4 is represented is incorrect (Weakness 2) and it would really help if the authors could provide clarification regarding this. Additionally, it would also really help if the authors could also provide further clarification on the ablation of tokenization where the comparison is a bit unfair as the test set of the paper is GPIC which is much closer to the proposed model which is trained on GPIC than other prior methods (Weakness 1).
  2. I appreciate the effort of the authors to provide experiments with language pretraining. However, the language pretraining is not done on GPIC. The authors don't provide a clarification regarding this, and this is quite important for future readers to know (Weakness 4). Additionally, the paper also seems to lack ablation on λimg\lambda_{img} when the amount of I2T is increased or decreased (Weakness 3) and also lacks ablation on different perceptual backbones based on LLMs (Weakness 5). It would be super nice if the authors could comment on this in their rebuttal.

Overall, I like the ideas of the paper, but I found the paper difficult to read, and it also requires some clarification regarding some of the experiments. As such, I recommend Weak Rejection!

7 comments
AuthorsOpenReview ↗

Response to Reviewer Xn4Q (1/2)

We thank the reviewer for their valuable time and feedback. The reviewer (1) raised concerns about the tokenizer comparisons, (2) requested clarification on Figure 4, (3) suggested increasing the conditional image-token loss weight, (4) requested clarification on the use of ClimbMix for language pretraining experiments, and (5) suggested exploring language-informed perceptual losses. We address each point below:

W1 + Q1: Unfair tokenizer comparisons

We thank the reviewer for the comment and suggestion. We agree that the comparisons in Tables 1 and 2 favor tokenizers trained on GPIC, whereas the counterparts were trained on larger, general-purpose datasets. To provide a broader comparison, we evaluate reconstruction quality on MS-COCO and ImageNet-1K, alongside GPIC. The improvements in PSNR and SSIM also generalize to MS-COCO and ImageNet-1K. Our goal was to evaluate the reconstruction quality of our visual tokenizer and the compression efficiency of both our vision and text tokenizers on GPIC. We did not intend to claim a better tokenizer design and we will clarify this in the final version.

GPICMS-COCOImageNet-1K
rFDPSNRSSIMrFIDPSNRSSIMrFIDPSNRSSIM
Cosmos-DI8x8 pretrained37.92625.380.7763.71024.400.7041.26524.480.701
Ours18.85225.610.7924.49225.690.7700.85625.940.757

 ~

W2 + Q1: Clarification on Figure 4

We would like to clarify a misunderstanding. Figure 4 is correct: the colors represent token types, not task types (T2I/I2T), as indicated in the legend and figure caption. We explain the two cases below.

  • Staged I2T: The model first predicts image tokens without text conditioning (I∣∅I \mid \varnothing, light green), followed by caption tokens conditioned on the image (T∣IT \mid I, dark orange). Therefore, the I2T stage correctly contains light green and dark orange bands, rather than two orange bands.
  • Baseline T2I: The model predicts caption tokens (T∣∅T \mid \varnothing, light orange), followed by image tokens conditioned on the caption (I∣TI \mid T, dark green). Caption dropout (pdrop=0.1p_{\mathrm{drop}} = 0.1) additionally trains unconditional image prediction (I∣∅I \mid \varnothing, light green).

We hope this clarifies the color scheme in Figure 4.

 ~

W3 + Q2: Effect of increasing λimg\lambda_{\mathrm{img}}

We thank the reviewer for this suggestion. We use λimg=0.5\lambda_{\mathrm{img}} = 0.5 across all configurations to prevent image tokens from dominating the training loss, as 1,024 of the 1,280 sequence tokens are image tokens. This is consistent with prior work (Emu3 [D]). Following the reviewer's suggestion, we conduct an additional experiment using Xanthus-0.52B with η=0.5\eta = 0.5, increasing the conditional image-token loss weight from 0.5 to 1.0 while keeping all other settings unchanged.

ConfigurationI2T fractionλimg\lambda_{img}gFD ↓Text NLL (T∣IT \mid I) ↓
Baseline T2I00.548.0218.414
Staged I2T → T2I0.50.554.2220.437
Mixing T2I + I2T0.50.575.971.009
Mixing T2I + I2T0.51.065.971.486

Results and takeaway. As the reviewer suggested, a larger conditional image-token loss weight improves gFD (75.97 → 65.97). But this setting does not match or outperform staged training (54.22). We attribute this difference to how model capacity is allocated during training. In staged training, the model first learns I2T and then trains exclusively on T2I. In mixed training, model capacity is shared between T2I and I2T throughout training. This explains why, despite identical conditional and unconditional image-token counts, staged training performs better on T2I generation, while mixed training performs better on captioning (text NLL: 20.437 vs. 1.486).

AuthorsOpenReview ↗

Response to Reviewer Xn4Q (2/2)

W4 + Q2: Why ClimbMix instead of GPIC captions for language pretraining?

We would like to clarify a misunderstanding: GPIC captions already provide language modeling supervision during Xanthus's T2I training. In Xanthus, the model autoregressively predicts GPIC caption tokens before image tokens, so language modeling on GPIC captions is already part of the T2I training objective (Section 3.2). However, GPIC contains only ~4.5B caption tokens, which is insufficient for Chinchilla-optimal language pretraining [E]. We therefore use ClimbMix to study whether language pretraining on external text-only data improves T2I generation. We will make this clear in the revised version.

W5 + Q2: Exploring language-informed perceptual losses

We thank the reviewer for the comment. Incorporating semantic losses from pretrained vision-language models, such as CLIP or VLMs, is a valuable direction. However, these models are typically pretrained on very large image-text datasets (e.g., CLIP uses 400M image-text pairs), which would introduce additional pretraining data and confound our controlled comparisons. Since our goal is to isolate the role of language pretraining in T2I generation, we avoided using such pretrained models. We hope this clarifies our design choice.

W6: Presentation and organization

We thank the reviewer for the detailed feedback. We will organize the experiments and results to address the two central questions: (i) Does language pretraining improve T2I generation? (ii) How do data composition and training schedules affect T2I generation?

 ~

References

[D] Wang, X., et al. "Multimodal Learning with Next-Token Prediction for Large Multimodal Models." Nature, vol. 650, 2026, pp. 327–33.

[E] Hoffmann, Jordan, et al. “Training Compute-Optimal Large Language Models.” NeurIPS 2022

I really thank the authors for their reply and would also like to apologize for the delayed response! Following the answers, I have some further comments and questions.

  1. I appreciate the authors' additional ablation for the tokenizer and also appreciate that the authors will change the wording in the text for the camera-ready version. Thank you very much!

  2. I understand the authors' response regarding Figure 4. However, my concern remains. The figure does show task types, too! Yes, it's not the colors of the tokens, but say, for example, "Staged: I2T, then T2I" there is clearly two task which are labelled I2T and T2I. This is where my confusion arises: why is the proposed method predicting image tokens without text conditioning for the I2T part? My assumption from the entire paper for the I2T task stage was that you provide the image tokens and make the model predict the text (and vice versa for the T2I task). If there is a need to predict image tokens during I2T, then the paper should provide clarity on why this is necessary and also provide ablation studies and/or explanations regarding this.

  3. I thank the authors for providing additional ablations for the λimg\lambda_{img}. This clearly shows that when doing "Mixing" experiments, this value needs to be studied more. However, this further makes the claim Staging favors generation; mixing favors captioning more vague. Mixing with a higher or lower fraction of I2T needs to have different λimg\lambda_{img} values. So the conclusion is no longer concrete! An explanation is quite necessary!

  4. Regarding the question about language-informed losses, I was particularly interested in knowing language-based losses where you have an external LLM and adopt a loss using the generated tokens in the external LLM space and the proposed method's text tokens. It would have been useful in the parts where only I2T was being solved. It would have also been particularly interesting to see if such a loss would have been useful in just language pretraining too. A comment on this would be really nice!

Overall, I appreciate the author's effort in the rebuttal and for stating that they would improve the text for the camera-ready version. However, there are still some clarifications needed. At this point, I am inclined towards a rating of 2.5 and am not yet convinced of a rating of 3. However, I will be eager to know the response of the authors!

Reviewer_HQEyOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether text-to-image models can learn effective text conditioning directly from paired image–text data without external text-only model pretraining. It trains the Xanthus model family and tokenizers from scratch, studies model scaling and training-data schedules on GPIC, and compares against modified LlamaGen and JiT-T2I baselines. It also reports compute accounting that includes language-model pretraining and examines replacing caption dropout with image-captioning supervision.

Critical strengths
  1. The study includes controlled experiments within the same Xanthus architecture, rather than relying only on comparisons between different model families. The extra-compute condition in Table 5 is particularly useful because it checks whether the negative result for language pretraining is simply due to reducing paired-data training.
  2. The baseline experiments explicitly examine image representation, sequence length, and perceptual supervision. Reporting these large effects separately helps avoid attributing every difference to language pretraining.
Critical weaknesses
  1. The main language-pretraining conclusion lacks a corresponding evaluation of prompt adherence. Table 5 primarily uses gFD and NLL. These measurements do not adequately establish whether language pretraining helps with complex instructions, counting, relations, or less common concepts. The alignment evaluation in Appendix C compares baseline T2I training with I2T mixing, but does not cover the language-pretraining ablations or the main late-fusion comparisons. The current evidence supports strong distributional generation performance on GPIC without language pretraining more directly than it supports the broader conclusion about text-conditioning quality.

  2. The 275× compute comparison needs a clearer distinction between training a complete system from scratch and reusing an available text encoder. Table 11 charges JiT the full pretraining cost of Qwen3. This is a valid accounting choice for constructing all components from scratch, but it does not describe the additional cost faced by someone using a public pretrained encoder. For example, the table lists 6.76×10206.76\times10^{20} generator FLOPs for JiT-T2I-XXL/8++, compared with 1.34×10211.34\times10^{21} total FLOPs for Xanthus-1.6B. The headline ratio should therefore be accompanied by the incremental T2I training cost so that its practical interpretation is clear.

Questions for authors
  1. How do the language-pretraining variants in Table 5 compare under a common text–image alignment evaluator and controlled tests of counting, relations, and attribute binding?
  2. Appendix B uses the baseline’s CFG scale for all configurations of a given size. Does the conclusion in Table 5 hold when each configuration selects its own CFG scale on validation data?
  3. Could you report complete from-scratch pretraining cost, incremental training cost when public encoders are reused, and generation-time inference cost side by side?
2 comments
AuthorsOpenReview ↗

Response to Reviewer HQEy (1/2)

We thank the reviewer for their valuable time and feedback. The reviewer (1) requested additional prompt-adherence and alignment evaluations for language-pretraining experiments and late-fusion baselines, (2) requested clarification on total pretraining, incremental training, and inference costs, (3) requested per-category GenEval results for language-pretraining experiments, and (4) asked whether individually tuning CFG scales affects our conclusions. We address each point below.

W1 (Part 1): Effect of language pretraining on prompt adherence

To evaluate the effect of language pretraining on prompt adherence, we report GenEval [A] and DPG-Bench [B] results for the language-pretraining configurations from Section 4.3 (Table 5).

Evaluation setup. For GenEval, we use PixelGen's rephrased prompts. For DPG-Bench, we use the original prompts.

ConfigurationLanguage fraction (α)GPIC gFD ↓GenEval ↑DPG-Bench ↑
Xanthus-0.52B048.0244.91 ±\pm 1.2576.84 ±\pm 0.27
Staged: Language → T2I0.151.5647.34 ±\pm 1.7276.02 ±\pm 0.67
Staged: Language → T2I0.252.7645.07 ±\pm 0.8776.03 ±\pm 0.76
Staged: Language → T2I0.565.1338.96 ±\pm 0.8872.65 ±\pm 0.32
Staged: Language → T2I (1.5× compute)0.549.2245.41 ±\pm 1.2174.25 ±\pm 0.58

Takeaway. (1) Under a fixed compute budget, allocating 50% of training to language pretraining worsens all three metrics. Even with a full T2I epoch (1.5× total compute), gFD (49.22 vs. 48.02) and DPG-Bench (74.25 vs. 76.84) remain worse, while GenEval is comparable (45.41 vs. 44.91). Overall, language pretraining provides no consistent improvement in compositional generation or prompt adherence.

 ~

W1 (Part 2): Prompt adherence compared with late-fusion baselines

We extend the Qwen3-VL-Reranker-8B alignment evaluation in Appendix C to include late-fusion baselines. We also report GenEval and DPG-Bench results for all models.

ModelGPIC gFD ↓GenEval ↑DPG-Bench ↑Qwen3-VL-Reranker (mean ±\pm std)
LlamaGen
LlamaGen-XL147.0130.0365.400.748 ±\pm 0.238
LlamaGen-XL/863.2930.4867.080.760 ±\pm 0.235
LlamaGen-XL-T5-Base/866.6928.4864.670.742 ±\pm 0.243
JiT-T2I
JiT-T2I-XXL/1668.6736.2772.690.838 ±\pm 0.209
JiT-T2I-XXL/16++55.7041.7976.580.852 ±\pm 0.199
JiT-T2I-XXL/8++43.2243.1776.850.845 ±\pm 0.206
Xanthus
Xanthus-0.1B144.1627.3166.360.801 ±\pm 0.226
Xanthus-0.25B72.5535.7272.660.840 ±\pm 0.202
Xanthus-0.52B48.0244.9176.840.870 ±\pm 0.176
Xanthus-1.6B35.2853.4877.900.886 ±\pm 0.155
Xanthus-4B28.4356.3977.900.893 ±\pm 0.146

Takeaways. (1) Xanthus-0.52B outperforms LlamaGen-XL/8 on GenEval (44.91 vs. 30.48), DPG-Bench (76.84 vs. 67.08), and Qwen3-VL-Reranker alignment (0.870 vs. 0.760). (2) Xanthus-1.6B outperforms JiT-T2I-XXL/8++ on all three metrics: GenEval (53.48 vs. 43.17), DPG-Bench (77.90 vs. 76.85), and Qwen3-VL-Reranker (0.886 vs. 0.845).

 ~

W2 + Q3: Total pretraining, incremental training, and inference costs

For clarity, we report the best-performing LlamaGen and JiT-T2I models.

1. Pretraining costs. We separate language and image training FLOPs, with image FLOPs including image-generator and image-tokenizer training. Incremental cost is generator training time on 8×H100 GPUs, excluding language-model and tokenizer pretraining.

ModelLanguage FLOPsImage FLOPsTotal FLOPsgFDIncremental cost (8xH100 hours)
LlamaGen-XL/89.06 × 10²¹6.21 × 10²⁰9.68 × 10²¹63.2954.3
JiT-T2I-XXL/8++3.67 × 10²³6.76 × 10²⁰3.68 × 10²³43.2292.0
Xanthus-0.52B7.97 × 10¹⁹4.53 × 10²⁰5.33 × 10²⁰48.0256.1
Xanthus-1.6B2.40 × 10²⁰1.10 × 10²¹1.34 × 10²¹35.28110.0

Takeaway. As the reviewer noted, the 275× figure reflects total cost of building a T2I system from scratch. With public text encoders and tokenizers, incremental cost is comparable (110.0 vs. 92.0 8×H100 hours, Xanthus-1.6B vs. JiT-T2I-XXL/8++). We will add this to the final version.

2. Inference costs. We report wall-clock time for generating 50K images at 256×256 resolution on 8×H100 GPUs. LlamaGen inference time is estimated using vLLM, as its official implementation does not support vLLM for T2I generation. We measure JiT-T2I inference time using the official PixGen codebase. Xanthus inference time is measured using vLLM.

ModelGeneration time (s) ↓
LlamaGen-XL/8 (uses 1.22B Flan-T5-XL)1,264
JiT-T2I-XXL/8++ (uses Qwen3-1.7B)2,704
Xanthus-0.52B861
Xanthus-1.6B911
AuthorsOpenReview ↗

Response to Reviewer HQEy (2/2)

Q1: Text–image alignment and compositional generation in language-pretraining experiments in Table 5.

We thank the reviewer for the suggestion. We provide Qwen3-VL-Reranker-8B alignment scores and per-category GenEval results for the language-pretraining experiments in Table 5.

  1. Text–image alignment.
ConfigurationLanguage fraction (α)gFDAlignment ↑ (mean ±\pm std)
Xanthus-0.52B048.020.870 ±\pm 0.176
Staged: Language → T2I0.151.560.871$\pm$ 0.174
Staged: Language → T2I0.252.760.865 ±\pm 0.181
Staged: Language → T2I0.565.130.853 ±\pm 0.191
Staged: Language → T2I (1.5× compute)0.549.220.871 ±\pm 0.172
  1. GenEval category breakdown. We report per-category scores to assess the effects of language pretraining on counting, spatial relations, colors, and attribute binding.
ConfigurationSingle obj.Two obj.CountingColorsPositionColor attributionOverall ↑
Xanthus-0.52B83.7550.0026.5661.1729.7518.2544.91 ±\pm 1.25
Staged: Language → T2I (α = 0.1)86.2555.3020.9467.5527.5026.5047.34 ±\pm 1.72
Staged: Language → T2I (α = 0.2)85.9452.7822.5065.4326.0017.7545.07 ±\pm 0.87
Staged: Language → T2I (α = 0.5)80.0039.6519.3857.7122.2514.7538.96 ±\pm 0.88
Staged: Language → T2I (1.5× compute)85.9452.2724.0665.1627.7517.2545.41 ±\pm 1.21

Takeaway. Language pretraining does not consistently improve text–image alignment or compositional generation. (1) Qwen3-VL-Reranker scores remain comparable at smaller language-pretraining fractions but decrease at α = 0.5 (0.853 vs. 0.870). Even with 1.5× compute, alignment is comparable to the baseline (0.871 vs. 0.870). (2) GenEval category-level results are mixed, with no consistent improvements in counting, spatial relations, or attribute binding when adding language pretraining.

 ~

Q2: Effect of CFG scale selection on language pretraining results

In Table 5 of the main paper, we use the same CFG scale (cfg=6.0) across all configurations. To assess whether this affects our conclusions, we additionally sweep CFG scales for each configuration on GPIC Validation-50K and report the corresponding gFD.

ConfigurationgFD (cfg=6.0) ↓gFD (individually tuned CFG) ↓
Xanthus-0.52B48.0248.02 (cfg=6.0)
Staged: Language → T2I (α = 0.1)51.5651.51 (cfg=6.5)
Staged: Language → T2I (α = 0.2)52.7652.76 (cfg=6.0)
Staged: Language → T2I (α = 0.5)65.1364.67 (cfg=7.0)
Staged: Language → T2I (1.5× compute)49.2248.97 (cfg=6.5)

Takeaway. Individually tuning CFG scales does not change our conclusions. All language-pretraining configurations have worse gFD than the baseline (48.02), even with 1.5× total compute (48.97).

 ~

References

[A] Ghosh, Dhruba, Hannaneh Hajishirzi, and Ludwig Schmidt. "Geneval: An object-focused framework for evaluating text-to-image alignment." NeurIPS 2023

[B] Hu, Xiwei, et al. "Ella: Equip diffusion models with llm for enhanced semantic alignment." arXiv preprint arXiv:2403.05135 (2024).

Reviewer_hqLHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper examines whether internet-scale language pretraining is necessary for strong text-to-image (T2I) generation. The authors propose Xanthus, a family of early-fusion autoregressive Transformers (0.1B–4B) trained entirely from scratch on GPIC, a dataset of 100M image–text pairs. All components, including both tokenizers, are learned without external text-only data. Xanthus-0.52B outperforms LlamaGen-XL/8 (48.0 vs. 63.5 gFD) with 18.2× less total pretraining compute. Xanthus-1.6B surpasses JiT-T2I-XXL/8++ (35.3 vs. 43.2 gFD) with 275× less compute. Controlled ablations show that adding text-only pretraining does not improve generation quality. Replacing caption-dropped examples with image-to-text data further enables captioning without degrading generation. These results suggest that paired image–text supervision alone can suffice for effective text conditioning. The findings, however, are limited to GPIC, 256×256 resolution, and the evaluated model families.

Critical strengths
  • The paper reports total pretraining compute in FLOPs, including the language-model pretraining cost of late-fusion baselines, which is usually treated as external. The decomposition into language and vision compute (Fig. 1, Table 11) makes the cost comparison transparent.
  • Baselines are retrained on the same data and strengthened with a shared image tokenizer, an LPIPS loss, and 8×8 patches. This makes the comparison with Xanthus fairer than one against off-the-shelf checkpoints.
Critical weaknesses
  • Evaluation is limited to the training domain. All models are trained and evaluated on GPIC captions. In-distribution gFD may not reflect the general language understanding that pretrained LMs provide, such as handling out-of-domain or compositional prompts. The findings of Q1 share this limitation, since they rely on the same in-domain evaluation. Evidence beyond GPIC would be needed to support the general claim that language pretraining is unnecessary.
  • The analysis does not support the scope of the claim. Q1 tests at most about 68B text-only tokens (Table 5), whereas the baseline LMs are pretrained on about 1T (FLAN-T5) and 36T (Qwen3) tokens. The finding that text-only pretraining does not help is thus established at a scale orders of magnitude smaller than the internet-scale pretraining the paper argues against. In addition, Q1 and Q2 are conducted at a single model size (0.52B). Since the model itself combines existing designs, the contribution rests largely on these findings, which makes their limited scope more consequential.
Questions for authors
  • Table 12 reports Qwen3-VL-Reranker alignment scores only for Xanthus configurations. Could the authors report the same scores for LlamaGen-XL/8 and JiT-T2I-XXL/8++, so that prompt adherence can be compared directly?
  • In Table 6, staged and mixed training at 0.5 use identical token counts, yet gFD differs substantially (54.2 vs. 76.0). What explains such a large gap from scheduling alone?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer hqLH (1/2)

We thank the reviewer for their valuable time and feedback. The reviewer (1) requested additional T2I evaluations beyond gFD, (2) requested experiments with internet-scale language pretraining for the Xanthus model, (3) requested Qwen3-VL-Reranker-8B alignment scores for LlamaGen and JiT-T2I baselines, and (4) requested an explanation for the performance differences between staged and mixed training. We address each point below:

W1: Evaluation beyond gFD

We provide additional results on GenEval [A] and DPG-Bench [B], which test compositional generation and prompt adherence on prompts outside GPIC.

Evaluation setup and results. For GenEval, we use the rephrased prompts from PixelGen. For DPG-Bench, we use the original prompts. We report GenEval and DPG-Bench scores for all baselines and Xanthus models, alongside GPIC gFD for comparison.

ModelGPIC gFD ↓GenEval ↑DPG-Bench ↑
LlamaGen-XL147.0130.0365.40
LlamaGen-XL/863.2930.4867.08
LlamaGen-XL-T5-Base/866.6928.4864.67
JiT-T2I-XXL/1668.6736.2772.69
JiT-T2I-XXL/16++55.7041.7976.58
JiT-T2I-XXL/8++43.2243.1776.85
Xanthus-0.1B144.1627.3166.36
Xanthus-0.25B72.5535.7272.66
Xanthus-0.52B48.0244.9176.84
Xanthus-1.6B35.2853.4877.90
Xanthus-4B28.4356.3977.90

Takeaway. Xanthus-0.52B outperforms all LlamaGen variants on both GenEval and DPG-Bench and achieves comparable performance to JiT-T2I-XXL/8++. Xanthus-1.6B achieves notably higher GenEval (53.48 vs. 43.17) and higher DPG-Bench (77.90 vs. 76.85) scores than JiT-T2I-XXL/8++. These results show strong compositional generation and prompt adherence capabilities of Xanthus models.

 ~

W2: Scope of language pretraining experiments

We note that our language pretraining experiments in Section 4.3 (Table 5) exceed the Chinchilla-optimal budget of 20 tokens per parameter, using 32–131 tokens per parameter. However, we agree with the reviewer that our token budget is substantially lower than those used to train internet-scale LLMs. To address this concern, we additionally train and evaluate Qwen3-0.6B pretrained on 36T text tokens.

Experiment setup and Results. We compare Qwen3-0.6B initialized from pretrained weights against random initialization, keeping the architecture, vocabulary, image tokenizer, GPIC training data, and T2I training budget fixed. We also train the pretrained model with a lower learning rate (1 × 10⁻⁴) to test whether smaller parameter updates better preserve its language modeling capabilities and improve T2I generation. The default learning rate (7.5 × 10⁻⁴) is used for other experiments. We report GPIC gFD, GenEval, and DPG-Bench scores below.

ConfigurationLanguage pretraining tokensgFD ↓GenEval ↑DPG-Bench ↑
Qwen3-0.6B (random initialization)None46.6149.71 ±\pm 0.2677.08 ±\pm 0.71
Qwen3-0.6B-CPT36T47.8146.09 ±\pm 2.0775.18 ±\pm 0.78
Qwen3-0.6B-CPT (lower lr)36T63.6242.52 ±\pm 1.3373.96 ±\pm 0.72

Takeaway. Pretraining on 36T text tokens does not improve T2I generation over random initialization (47.81 vs. 46.61 gFD). GenEval and DPG-Bench scores are also lower than those of the randomly initialized model, and reducing the learning rate further worsens all three metrics. These results are consistent with our findings in Section 4.3 and Table 5.

AuthorsOpenReview ↗

Author Response to Reviewer hqLH (2/2)

Q1: Qwen3-VL-Reranker alignment scores for baselines

We thank the reviewer for the suggestion. Below, we provide the requested Qwen3-VL-Reranker-8B alignment scores for all baselines and Xanthus models.

ModelQwen3-VL-Reranker (mean ± std)
LlamaGen
LlamaGen-XL0.748 ±\pm 0.238
LlamaGen-XL/80.760 ± 0.235
LlamaGen-XL-T5-Base/80.742 ± 0.243
JiT-T2I
JiT-T2I-XXL/160.838 ± 0.209
JiT-T2I-XXL/16++0.852 ± 0.199
JiT-T2I-XXL/8++0.845 ± 0.206
Xanthus
Xanthus-0.1B0.801 ± 0.226
Xanthus-0.25B0.840 ± 0.202
Xanthus-0.52B0.870 ± 0.176
Xanthus-1.6B0.886 ± 0.155
Xanthus-4B0.893 ± 0.146
Xanthus (I2T training, p_drop=0, η = 0.1)
Xanthus-0.1B0.806 ± 0.223
Xanthus-0.25B0.849 ± 0.195
Xanthus-0.52B0.872 ± 0.174
Xanthus-1.6B0.886 ± 0.156
Xanthus-4B0.890 ± 0.147

Takeaway. Xanthus-0.52B achieves higher alignment scores than LlamaGen-XL/8 (0.870 vs. 0.760), while Xanthus-1.6B outperforms JiT-T2I-XXL/8++ (0.886 vs. 0.845).

 ~

Q2: Effect of staged and mixing schedule on T2I generation

Although staged (β\beta=0.5) and mixed (η\eta=0.5) training use identical token counts, they differ in how the model learns the two tasks. In mixed training, the model learns T2I and I2T simultaneously, allocating capacity to both tasks throughout training. In staged training, the model first learns I2T and then focuses exclusively on T2I. This results in staged training achieving better T2I generation than mixed training (54.2 vs. 76.0 gFD), while mixed training achieves better captioning (1.009 vs. 20.437 conditional text NLL). We will make this clear in the revised version.

References

[A] Ghosh, Dhruba, Hannaneh Hajishirzi, and Ludwig Schmidt. "Geneval: An object-focused framework for evaluating text-to-image alignment." NeurIPS 2023

[B] Hu, Xiwei, et al. "Ella: Equip diffusion models with llm for enhanced semantic alignment." arXiv preprint arXiv:2403.05135 (2024).

I thank the authors for the comprehensive rebuttal. The additional GenEval/DPG-Bench results and the Qwen3-0.6B (36T) initialization experiment directly address my main concerns, therefore, I raise my score to weak accept. In the revised version, I encourage the authors to (1) clarify whether the Qwen3 initialization experiment used Qwen3’s original tokenizer and embeddings or the Xanthus vocabulary, as this affects how much pretrained knowledge is retained, and (2) report GenEval results with the original (non-rephrased) prompts.

AuthorsOpenReview ↗

Response to Reviewer hqLH

We thank the reviewer for the follow-up and for raising your recommendation to Weak Accept.

(1) Qwen3 initialization: The Qwen3-0.6B-CPT experiment uses Qwen3's original tokenizer and pretrained weights, including embeddings, expanded with 64K image-token embeddings for continual pretraining. The random-initialization control uses the identical vocabulary and architecture. We will state this explicitly in the final version.

(2) GenEval prompts: We will report GenEval results with the original (non-rephrased) prompts in the final version.

Thank you for your time.

Best,
Authors

Reviewer_mKofOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether strong text-to-image generation requires large-scale language pretraining. The authors introduce Xanthus, an early-fusion autoregressive model trained from scratch on 100M GPIC image-text pairs, including newly trained image and text tokenizers. Across model sizes from 0.1B to 4B, Xanthus achieves better reported gFD than the evaluated LlamaGen and JiT-T2I baselines. The paper also studies text-only pretraining and I2T training schedules, and shows findings such as replacing caption-dropped examples with I2T examples enables captioning while largely preserving generation quality.

Critical strengths

The paper asks an interesting and practically relevant question about how much language pretraining contributes to text-to-image generation.

The experimental effort is substantial, covering five model sizes, two baseline families, and several data composition and scheduling ablations. The authors also strengthen the baselines by matching image sequence lengths and introducing perceptual supervision, which makes the comparison more informative.

Replacing caption dropout with image-to-text supervision is a simple and useful observation. The paper is generally clear, well structured, and easy to follow.

Critical weaknesses

(1) Evaluation on standard T2I benchmarks is limited. The evaluation primarily relies on gFD and related distributional metrics on GPIC. Appendix C includes limited alignment scores, but no standard benchmark comparison with the baselines. Results on GenEval and DPG-Bench, together with human evaluation of visual quality and prompt adherence, would better show the model's generation capabilities.

(2) A controlled comparison between early and late fusion is missing. The comparisons with LlamaGen and JiT-T2I are useful, but also differ in backbone, pretraining trajectory etc. Comparing early-fusion and late-fusion Xanthus variants under matched data, image tokenization, training objectives, and compute budgets would better isolate the effect of fusion architecture. Language pretraining should ideally be varied independently, providing clearer insights into which design choices drive the gains.

(3) The novel empirical findings need clarification relative to prior work. Training T2I models without pretrained language encoders is already established as the paper acknowledges. More recent work, such as Han et al. (2026) (Towards Physics of Multimodal Pretraining), also studies early integration, training schedules, and cross-modal transfer, and reports benefits from language data for generation. The authors should consider clarifying which findings extend prior evidence and explain the differing conclusions about the utility of language data.

Questions for authors

Please see the weaknesses section above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer mKof (1/2)

We thank the reviewer for their valuable time and feedback. The reviewer (1) requested additional evaluations on standard T2I benchmarks, (2) requested a controlled comparison between early and late fusion, (3) requested experiments that vary language pretraining independently of image-text pretraining, and (4) asked for clarification on the novelty of our findings relative to Han et al. (arXiv Aug 2026) work. We address each point below:

W1: Evaluation results on GenEval and DPG-Bench

Below, we provide the requested evaluation results on GenEval [A] and DPG-Bench [B] for all Xanthus models and baselines.

Evaluation setup and results. For GenEval, we use the rephrased prompts from PixelGen. For DPG-Bench, we use the original prompts. Below, we report GenEval and DPG-Bench scores alongside GPIC gFD for comparison.

ModelGPIC gFD ↓GenEval ↑DPG-Bench ↑
LlamaGen-XL147.0130.0365.40
LlamaGen-XL/863.2930.4867.08
LlamaGen-XL-T5-Base/866.6928.4864.67
JiT-T2I-XXL/1668.6736.2772.69
JiT-T2I-XXL/16++55.7041.7976.58
JiT-T2I-XXL/8++43.2243.1776.85
Xanthus-0.1B144.1627.3166.36
Xanthus-0.25B72.5535.7272.66
Xanthus-0.52B48.0244.9176.84
Xanthus-1.6B35.2853.4877.90
Xanthus-4B28.4356.3977.90

Takeaway. Xanthus-0.52B outperforms all LlamaGen variants on both GenEval and DPG-Bench and achieves comparable performance to JiT-T2I-XXL/8++. Xanthus-1.6B achieves notably higher GenEval (53.48 vs. 43.17) and higher DPG-Bench (77.90 vs. 76.85) scores than JiT-T2I-XXL/8++. These results show strong compositional generation and prompt adherence capabilities of Xanthus models.

 ~

W2 (part 1): A controlled comparison between early and late fusion is missing. Comparing early-fusion and late-fusion Xanthus variants under matched data, image tokenization, training objectives, and compute budgets would better isolate the effect of fusion architecture.

We thank the reviewer for the comment. We provide controlled comparisons addressing this concern in Section 4.1 (Tables 3 and 4) of the main paper, where we modify LlamaGen to match Xanthus's image-text training dataset, image tokenizer, and image sequence length. Both Xanthus and LlamaGen use prefix-conditioned autoregressive image generation. We copy the setup and results below, including GenEval and DPG-Bench results.

ConfigurationFusionDatasetImage tokenizer#Image tokensGenerationT2I trainingLanguage encodergFD ↓GenEval ↑DPG-Bench ↑
Xanthus-0.52BEarlyGPIC 100MOurs (FSQ, 8x8)1,024AR1 epochNone48.0244.9176.84
LlamaGen-XL/8LateGPIC 100MOurs(FSQ, 8x8)1,024AR1 epochFLAN-T5-XL63.2930.4867.08
LlamaGen-XL-T5-Base/8LateGPIC 100MOurs(FSQ, 8x8)1,024AR1 epochFLAN-T5-Base66.6928.4864.67

Takeaway. Under these matched settings, Xanthus-0.52B outperforms both late-fusion baselines on gFD, GenEval, and DPG-Bench.

AuthorsOpenReview ↗

Response to Reviewer mKof (2/2)

W2 (part 2): Language pretraining should ideally be varied independently, providing clearer insights into which design choices drive the gains.

We’d like to point the reviewer to Section 4.3 (Table 5) of the main paper, where we vary language pretraining under fixed and additional compute budgets. Further, we also provide a new continual pretraining experiment initialized from Qwen3-0.6B, pretrained on 36T text tokens.

1. Effect of text-only pretraining on T2I generation (Table 5 with GenEval and DPG-Bench results).

ConfigurationLanguage fraction (α)T2I traininggFD ↓GenEval ↑DPG-Bench ↑
Xanthus-0.52B01 epoch48.0244.91 ±\pm 1.2576.84 ±\pm 0.27
Staged: Language → T2I0.10.9 epoch51.5647.34 ±\pm 1.7276.02 ±\pm 0.67
Staged: Language → T2I0.20.8 epoch52.7645.07 ±\pm 0.8776.03 ±\pm 0.76
Staged: Language → T2I0.50.5 epoch65.1338.96 ±\pm 0.8872.65 ±\pm 0.32
Staged: Language → T2I (1.5× compute)0.51 epoch49.2245.41 ±\pm 1.2174.25 ±\pm 0.58

2. Effect of internet-scale language pretraining (new experiment). We compare Qwen3-0.6B initialized from a pretrained language model (36T text tokens) against random initialization, holding the architecture, vocabulary, image tokenizer, and T2I training budget fixed.

ConfigurationLanguage pretraininggFD ↓GenEval ↑DPG-Bench ↑
Qwen3-0.6B (random initialization)None46.6149.71 ±\pm 0.2677.08 ±\pm 0.71
Qwen3-0.6B-CPT36T tokens47.8146.09 ±\pm 2.0775.18 ±\pm 0.78
Qwen3-0.6B-CPT (lower lr)36T tokens63.6242.52 ±\pm 1.3373.96 ±\pm 0.72

Takeaways. (1) Under a fixed compute budget, allocating 50% of training to language pretraining worsens all three metrics. Even with a full T2I epoch (1.5× total compute), gFD (49.22 vs. 48.02) and DPG-Bench (74.25 vs. 76.84) remain worse, while GenEval is comparable (45.41 vs. 44.91). (2) Initializing from Qwen3-0.6B pretrained on 36T text tokens performs worse than random initialization across all three metrics. Together, these experiments show no consistent benefit from language pretraining for T2I generation quality or prompt adherence.

W3: Novelty of findings relative to prior work

(1) Contribution of our work. To our knowledge, ours is the first work to systematically quantify the role of language pretraining in T2I generation through controlled comparisons across early- and late-fusion T2I systems. Our findings provide guidance on allocating pretraining compute across language and vision for T2I generation.

(2) Comparison with Han et al. [C] (arXiv Aug 2026). We thank the reviewer for pointing out this work. Han et al. study unified multimodal pretraining. In contrast, our work focuses on the role of language pretraining in T2I generation and includes controlled comparisons against late-fusion T2I systems with separately pretrained text encoders (LlamaGen, JiT-T2I), which Han et al. do not consider. On the role of language pretraining, our findings are consistent with theirs: under a fixed token budget, extending language-only pretraining before introducing visual data degrades visual generation (Fig. 11), and joint training outperforms sequential training (Fig. 12). The language benefits reported in their Sec. 3.1.1 come from adding language data on top of a fixed visual data budget, which increases total compute. Our work instead studies data schedules under a fixed compute budget.

References

[A] Ghosh, Dhruba, Hannaneh Hajishirzi, and Ludwig Schmidt. "Geneval: An object-focused framework for evaluating text-to-image alignment." NeurIPS 2023

[B] Hu, Xiwei, et al. "Ella: Equip diffusion models with llm for enhanced semantic alignment." arXiv preprint arXiv:2403.05135 (2024).

[C] Han, Junlin, et al. "Towards physics of multimodal pretraining: Knowledge flow, modality synergy, early unification, and recipes." arXiv preprint arXiv:2608.05000 (2026).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author–Reviewer Discussion Summary

Dear AC and Reviewers,

As the author–reviewer discussion ends within the hour, we summarize reviewer concerns, the requested experiments and/or clarifications provided, and their takeaways.

Thesis. We study whether strong text-to-image (T2I) generative models require internet-scale language pretraining. Through controlled experiments and comparisons against early- and late-fusion T2I systems, we show that strong T2I models can be trained from scratch, without language pretraining.

Reviewer concerns and responses:

  • T2I benchmarks (mKof, hqLH, HQEy). We report GenEval and DPG-Bench for all Xanthus models and baselines. Takeaway: Xanthus-0.52B outperforms all LlamaGen variants on both benchmarks and is comparable to JiT-T2I-XXL/8++. Xanthus-1.6B outperforms JiT-T2I-XXL/8++ on GenEval (53.48 vs. 43.17) and DPG-Bench (77.90 vs. 76.85).
  • Internet-scale language pretraining (mKof, hqLH, HQEy). We ran a controlled experiment comparing Qwen3-0.6B initialized from 36T-token pretrained weights against random initialization, with the architecture, vocabulary, image tokenizer, and T2I budget fixed. Takeaway: Language pretraining does not improve gFD (47.81 vs. 46.61), GenEval (46.09 vs. 49.71), or DPG-Bench (75.18 vs. 77.08) (pretrained vs. random). This is consistent with our ablations in Section 4.3 (Table 5).
  • Controlled early- vs. late-fusion comparison (mKof). Section 4.1 (Tables 3–4) matches the dataset, image tokenizer, image sequence length, and AR objective. Takeaway: Xanthus-0.52B outperforms both late-fusion LlamaGen variants on gFD, GenEval, and DPG-Bench.
  • Image–text alignment (hqLH, HQEy). We report Qwen3-VL-Reranker-8B scores for all baselines. Takeaway: Xanthus-0.52B outperforms LlamaGen-XL/8 (0.870 vs. 0.760), and Xanthus-1.6B outperforms JiT-T2I-XXL/8++ (0.886 vs. 0.845).
  • Compute accounting (HQEy). We agree that the 275× figure reflects building the full system from scratch. With public encoders reused, incremental training cost is comparable (110.0 vs. 92.0 8×H100 hours, Xanthus-1.6B vs. JiT-T2I-XXL/8++), while Xanthus-1.6B achieves better gFD (35.28 vs. 43.22). We will report both from-scratch and incremental costs in the final version.
  • Other requested items. Per-configuration CFG tuning (HQEy), the λ_img ablation and I2T clarifications (hqLH, Xn4Q), clarification of Figure 4 and the I2T training design (Xn4Q), tokenizer evaluation on MS-COCO and ImageNet-1K (Xn4Q), and positioning relative to Han et al. (arXiv August 2026) (mKof).

We hope these results and clarifications address the concerns raised by the reviewers.

Thank you for your time and consideration.

Best,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.