- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes a systematic analysis of the order in which training data is provided for training a unified model (without any language pretraining) for text-to-image generation. The paper found that unified baseline with prediction for both image and text together at each training step is better or on par for text-to-image generation task than other handcrafted tricks like: language pretraining, staged training (image-to-text first then text-to-image) or mixed training (some percentage of batch involved in image-to-text and some for text-to-image). However, mixed training do help for text generation tasks and can be thought of as an alternative. The authors did an adequate job in providing proper ablations and provide multiple models which show strong results.
Critical strengths
- The paper analyses quite well the order of data flow to a unified early fusion model. The results and analysis are quite interesting and is really relevant for the community.
- One of the great insights from the paper is about mixing text-to-image and image-to-text in the same batch: it helps in captioning and staging with I2T and T2I helps in generation (on par with baseline ordering). This is quite interesting!
Critical weaknesses
- I really appreciate the author’s effort in training a new 1D tokenizer for images. However, I feel that the comparison with the prior Cosmos tokenizer is a bit unfair in Table 1, as the paper trained its tokenizer on GPIC (unlike Cosmos) and tested on GPIC. The distribution gap is supposed to be greater, and Table 1 actually shows something obvious. It is also the same with the Text Tokenizer and Table 2; the things shown are quite obvious and don't add much to the paper. However, for both image and text tokenizers, the text seems to be quite over the top with suggestions of better performance due to some sort of cool improvements in training. I really think that these need to be toned down. Otherwise, I really appreciate the effort to train new tokenizers for both images and text!
- I really like the ablations of different ways of providing data to the model. However, I am not so sure I understand the color scheme for the horizontal bands in Figure 4. For example, in Staged: I2T then T2I, I am not so sure why the I2T part has dark orange and light green. Shouldn’t it be dark and light orange bands instead? Additionally, I am not so sure about the bands for Baseline T2I, too. A clarification regarding this figure is very important to improve the readability of the paper.
- Continuing with the previous point of data ordering for training, I was wondering if the authors explored changing the value when using mixing. I would assume a higher value must be used when a higher amount of I2T is used. This might be one of the reasons why the conclusion
Staging favors generation; mixing favors captioningis found. It would be super nice if the authors could comment on this! I totally understand the time limitations of the rebuttal, and as such, simple experiments with text-only explanations are more than enough! - I really like that the authors went out of their way to train with language pretraining followed by T2I. However, I am not quite sure to understand why the authors used ClimbMix for the language pretraining. Given the main intention is to train a T2I model on GPIC, why is the language pretraining also not done on GPIC captions? I would assume that would have been the natural step. Again, I totally understand the constraints of the rebuttal, and simple experiments with text only explanations suffice as plausible answers.
- The use of perceptual LPIPS is quite common now to improve JiT baselines. I appreciate that the authors also included it in their pipeline. However, I was wondering, given it's a unified model with language information embedded, why a perceptual loss based on some external language models was not adopted too! I would be really curious to know about this from the authors.
- I really liked exploring the ideas of the paper and found the ablations quite interesting. However, I must admit it was a bit difficult to read with all the bullet points at places making me scrape for relevant information at times. Also the organization of the results is super difficult for me to parse. I would have preferred to have the ablations slowly building towards the final results and a proper story to link everything together; however, here it feels like a collection of results for me to figure out what they are, which should not be the case. I really encourage the authors to improve the writing and make it easy for future readers.
Questions for authors
As mentioned before I liked the ideas proposed in the paper, and I do think the ablations are quite good. However, apart from the difficulty in parsing the content of the paper, there are a few things that need a bit of clarification.
(Note: I totally understand the restrictions of the rebuttal, and simple text explanations and short experiments are more than enough)
- I really like the ablation of the dataset order. However, I really feel that the way Figure 4 is represented is incorrect (Weakness 2) and it would really help if the authors could provide clarification regarding this. Additionally, it would also really help if the authors could also provide further clarification on the ablation of tokenization where the comparison is a bit unfair as the test set of the paper is GPIC which is much closer to the proposed model which is trained on GPIC than other prior methods (Weakness 1).
- I appreciate the effort of the authors to provide experiments with language pretraining. However, the language pretraining is not done on GPIC. The authors don't provide a clarification regarding this, and this is quite important for future readers to know (Weakness 4). Additionally, the paper also seems to lack ablation on when the amount of I2T is increased or decreased (Weakness 3) and also lacks ablation on different perceptual backbones based on LLMs (Weakness 5). It would be super nice if the authors could comment on this in their rebuttal.
Overall, I like the ideas of the paper, but I found the paper difficult to read, and it also requires some clarification regarding some of the experiments. As such, I recommend Weak Rejection!
7 comments
Response to Reviewer Xn4Q (2/2)
W4 + Q2: Why ClimbMix instead of GPIC captions for language pretraining?
We would like to clarify a misunderstanding: GPIC captions already provide language modeling supervision during Xanthus's T2I training. In Xanthus, the model autoregressively predicts GPIC caption tokens before image tokens, so language modeling on GPIC captions is already part of the T2I training objective (Section 3.2). However, GPIC contains only ~4.5B caption tokens, which is insufficient for Chinchilla-optimal language pretraining [E]. We therefore use ClimbMix to study whether language pretraining on external text-only data improves T2I generation. We will make this clear in the revised version.
W5 + Q2: Exploring language-informed perceptual losses
We thank the reviewer for the comment. Incorporating semantic losses from pretrained vision-language models, such as CLIP or VLMs, is a valuable direction. However, these models are typically pretrained on very large image-text datasets (e.g., CLIP uses 400M image-text pairs), which would introduce additional pretraining data and confound our controlled comparisons. Since our goal is to isolate the role of language pretraining in T2I generation, we avoided using such pretrained models. We hope this clarifies our design choice.
W6: Presentation and organization
We thank the reviewer for the detailed feedback. We will organize the experiments and results to address the two central questions: (i) Does language pretraining improve T2I generation? (ii) How do data composition and training schedules affect T2I generation?
References
[D] Wang, X., et al. "Multimodal Learning with Next-Token Prediction for Large Multimodal Models." Nature, vol. 650, 2026, pp. 327–33.
[E] Hoffmann, Jordan, et al. “Training Compute-Optimal Large Language Models.” NeurIPS 2022
I really thank the authors for their reply and would also like to apologize for the delayed response! Following the answers, I have some further comments and questions.
-
I appreciate the authors' additional ablation for the tokenizer and also appreciate that the authors will change the wording in the text for the camera-ready version. Thank you very much!
-
I understand the authors' response regarding Figure 4. However, my concern remains. The figure does show task types, too! Yes, it's not the colors of the tokens, but say, for example, "Staged: I2T, then T2I" there is clearly two task which are labelled I2T and T2I. This is where my confusion arises: why is the proposed method predicting image tokens without text conditioning for the I2T part? My assumption from the entire paper for the I2T task stage was that you provide the image tokens and make the model predict the text (and vice versa for the T2I task). If there is a need to predict image tokens during I2T, then the paper should provide clarity on why this is necessary and also provide ablation studies and/or explanations regarding this.
-
I thank the authors for providing additional ablations for the . This clearly shows that when doing "Mixing" experiments, this value needs to be studied more. However, this further makes the claim
Staging favors generation; mixing favors captioningmore vague. Mixing with a higher or lower fraction of I2T needs to have different values. So the conclusion is no longer concrete! An explanation is quite necessary! -
Regarding the question about language-informed losses, I was particularly interested in knowing language-based losses where you have an external LLM and adopt a loss using the generated tokens in the external LLM space and the proposed method's text tokens. It would have been useful in the parts where only I2T was being solved. It would have also been particularly interesting to see if such a loss would have been useful in just language pretraining too. A comment on this would be really nice!
Overall, I appreciate the author's effort in the rebuttal and for stating that they would improve the text for the camera-ready version. However, there are still some clarifications needed. At this point, I am inclined towards a rating of 2.5 and am not yet convinced of a rating of 3. However, I will be eager to know the response of the authors!
Response to Reviewer Xn4Q (1/2)
We thank the reviewer for their valuable time and feedback. The reviewer (1) raised concerns about the tokenizer comparisons, (2) requested clarification on Figure 4, (3) suggested increasing the conditional image-token loss weight, (4) requested clarification on the use of ClimbMix for language pretraining experiments, and (5) suggested exploring language-informed perceptual losses. We address each point below:
We thank the reviewer for the comment and suggestion. We agree that the comparisons in Tables 1 and 2 favor tokenizers trained on GPIC, whereas the counterparts were trained on larger, general-purpose datasets. To provide a broader comparison, we evaluate reconstruction quality on MS-COCO and ImageNet-1K, alongside GPIC. The improvements in PSNR and SSIM also generalize to MS-COCO and ImageNet-1K. Our goal was to evaluate the reconstruction quality of our visual tokenizer and the compression efficiency of both our vision and text tokenizers on GPIC. We did not intend to claim a better tokenizer design and we will clarify this in the final version.
We would like to clarify a misunderstanding. Figure 4 is correct: the colors represent token types, not task types (T2I/I2T), as indicated in the legend and figure caption. We explain the two cases below.
We hope this clarifies the color scheme in Figure 4.
We thank the reviewer for this suggestion. We use λimg=0.5 across all configurations to prevent image tokens from dominating the training loss, as 1,024 of the 1,280 sequence tokens are image tokens. This is consistent with prior work (Emu3 [D]). Following the reviewer's suggestion, we conduct an additional experiment using Xanthus-0.52B with η=0.5, increasing the conditional image-token loss weight from 0.5 to 1.0 while keeping all other settings unchanged.
Results and takeaway. As the reviewer suggested, a larger conditional image-token loss weight improves gFD (75.97 → 65.97). But this setting does not match or outperform staged training (54.22). We attribute this difference to how model capacity is allocated during training. In staged training, the model first learns I2T and then trains exclusively on T2I. In mixed training, model capacity is shared between T2I and I2T throughout training. This explains why, despite identical conditional and unconditional image-token counts, staged training performs better on T2I generation, while mixed training performs better on captioning (text NLL: 20.437 vs. 1.486).