acceptodds
Withdrawn from ICLR 2027

Know, then Draw: A New Learning Paradigm for Visual Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual generation requires both high-level semantic knowledge and fine-grained pixel details. We propose a learning paradigm that assigns these roles to autoregressive pretraining for knowledge acquisition and diffusion post-training for high-fidelity rendering. Within this framework, we study the objective, representation, and scaling of knowledge acquisition and rendering. Controlled comparisons identify autoregression as the stronger pretraining objective for downstream rendering, improving both learning efficiency and final generation quality. We further find that AR-only generation quality does not necessarily correlate with final performance; semantic quality matters more. This motivates a purely semantic tokenizer trained with language supervision alone, without an image decoder or reconstruction loss. We establish compute-optimal scaling laws for knowledge acquisition and rendering, and find that autoregressive pretraining lowers the fitted asymptotic rendering loss compared with pixel diffusion models trained from scratch. Guided by the laws, we train a 3B-parameter text-to-image model with only 6,600 H100 GPU-hours, achieving 86 on GenEval and 87 on DPG-Bench and validating the proposed paradigm at scale.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Visual Generation, Scaling Law, Autoregressive and Diffusion Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_F9AoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes splitting text-to-image generation into two stages. The first stage is Knowledge Acquisition where a text conditioned AR model is pretrained to predict discrete visual tokens. In the second stage, the AR backbone is frozen and pixel space diffusion models reuse the attention projections and are post trained with flow matching.

Critical strengths
  • The framing of the paper is useful. Instead of asking AR vs Diffusion, the authors focus on specific strengths and use them in collaboration.
  • The dissociation between reconstruction or AR only quality and final quality is a practically useful observation for tokenizer design
  • A decoder free tokenizer trained only through language supervision is an interesting idea.
Critical weaknesses
  • The post training renderer receives codes extracted from the same image it denoises. The from scratch pixel diffusion model only receives text. A lower velocity MSE is therefore expected from the extra information about the target alone, independent of any benefit from pre training. Section 3.3 acknowledges that the comparison does not isolate the two factors, but the abstract and conclusion still attribute the lower floor to AR pretraining.

  • The fitted asymptotes also come without uncertainty estimates. The post-training allocation exponent ranges from 0.45 to 0.70 across estimators (Tab 9), which suggests parametric fit is weakly identified.

  • There are inconsistencies in proposed tokenizer and what the evidence favors. The Ar -> PDM row in Table 1 (65.51/71.11) is the semantic-recon configuration and not the semantic-lm as shown in table 2. Semantic-LM is actually lower on both geneval and DPG.

  • Semantic quality determines final performance is asserted rather than measured. No measure of semantic quality is reported (eg. linear probing, retrieval, or VQA), and conclusion rests on three tokenizers that differ in several ways at once.

  • There is a plausible alternative explanation of why recon gets worse after post training. Recon codes nearly determine the image(rFID 1.04), so renderer trained on GT codes may learn to copy them and hence is brittle to AR errors. This would be consistent with recon getting worse post training. Coarser semantics may leave more to the renderer and may simply tolerate rollout errors better. The paper does not discuss or distinguish between these explanations.

  • The compute-efficiency claim does not consider the tokenizer training (A.1). More fundamentally, a substantial part of the "knowledge" comes from Qwen2.5-VL-32B features rather than the AR pre-training.

Questions for authors
  • Can you separate pretraining from conditioning in the asymptote comparison? Either of these would do: i) a from-scratch pixel diffusion model conditioned on the same ground-truth codes, or ii) post-training loss evaluated with AR-generated codes.

  • Which tokenizer does the final model use? If it is Semantic-Recon, how should readers interpret the emphasis on Semantic-LM?

  • What is the measured compute for tokenizer training and for the frozen AR forward pass during post-training, and what is the measured wall-clock time of the final run?

  • GenEval and DPG-Bench measure prompt alignment, not fidelity. Since rendering quality is a central claim, an FID, human-preference, or text-rendering evaluation would help

Reviewer_5EsHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper separates visual generation into autoregressive pretraining over discrete visual tokens and pixel-diffusion post-training using a frozen AR backbone. It investigates pretraining objectives, semantic tokenizers—including one trained without image reconstruction—and scaling across the two stages. Experiments include controlled smaller models, extensive scaling sweeps, and a roughly 3B generator reporting 0.86 GenEval and 87.78 DPG-Bench with approximately 6,600 H100 GPU-hours.

Critical strengths

1.A meaningful insight into visual tokenization. Table 2 shows that reconstruction quality and AR-only generation quality need not predict downstream generation performance. Semantic-LM supports useful generation without an image decoder or reconstruction objective, providing evidence for separating semantic prediction from rendering.

2.A substantial empirical investigation. The paper combines matched final-model-size and image-exposure comparisons with broad scaling sweeps and independent optimization searches. This provides useful evidence for the practical viability of the proposed pipeline.

Critical weaknesses

1.The headline evaluation uses benchmark feedback for prompt selection. Appendix A.9 explicitly states that prompt variants are selected using evaluation scores under fixed seeds. The reported results therefore do not establish performance under an independently fixed evaluation procedure, and comparisons with published baseline scores may be optimistically biased. This materially weakens the large-scale validation.

2.The experiments do not isolate the claimed advantages of AR pretraining. Section 3.1 changes the underlying pretrained visual representation alongside the pretraining objective. Furthermore, Section 3.3 compares the fitted loss floor of target-image-conditioned rendering with that of text-only generation. Although the paper acknowledges the latter limitation, this comparison cannot attribute the lower floor specifically to AR pretraining or establish better attainable text-to-image performance.

Questions for authors

1.Evaluation clarification and validation: How many prompt variants were evaluated, and were they selected separately for each benchmark prompt? What results are obtained with a rewriting policy fixed without benchmark-score feedback and evaluated on fresh seeds?

2.Controlled objective comparison: Does the AR advantage persist when the underlying visual representation is held fixed across pretraining objectives?

3.Loss-floor interpretation: Does the lower fitted asymptote persist when a renderer trained from scratch receives the same target-image tokens as the pretrained renderer?

Reviewer_f4iUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a two-stage text-to-image generation framework. First, a frozen text encoder encodes the input text, and an autoregressive (AR) model generates discrete visual tokens. A diffusion model then conditions on these tokens and the text to generate an image through iterative denoising in pixel space. The two stages share attention weights; during diffusion training, the AR parameters are frozen, and only the newly introduced diffusion modules are trained. The core idea is to learn visual content through AR training and render fine-grained details through diffusion.

The authors further compare different tokenizers and introduce Semantic-LM, whose codebook is trained solely through language supervision, allowing the AR prediction targets to forgo image reconstruction. They then separately fit scaling laws for AR pretraining and diffusion post-training and use these to derive an approximately 3:2 compute budget allocation between the two stages.

The experiments include comparisons across generation paradigms, tokenizer ablations, and scaling analyses across multiple model sizes. The final model is trained with a reported budget of approximately 6,600 H100 GPU-hours and achieves 0.86 on GenEval and 87.78 on DPG-Bench under the evaluation protocol used in the paper.

Critical strengths

The paper proposes an effective two-stage image generation framework. It introduces diffusion modules to render images conditioned on AR-generated visual tokens and reuses the frozen AR backbone’s attention weights, enabling parameter sharing between the two stages. The method achieves strong GenEval and DPG-Bench scores within a limited training budget, demonstrating the effectiveness of this design.

Critical weaknesses
  1. There is a gap between the high quantitative scores and the visual quality of the showcased examples. Despite the strong GenEval and DPG-Bench scores, Figure 1 exhibits noticeable geometric distortions, such as the irregular contour of the circular component at the front of the locomotive and the distorted rim and handles of the woven basket in the bedroom scene. High-fidelity generation therefore remains insufficiently validated. Visual comparisons with baselines and human evaluations would strengthen the evidence.

  2. The necessity of the additional AR stage and its cost–benefit trade-off remain insufficiently validated. Unlike Emu3, which autoregressively generates visual tokens directly from text tokens, this method uses a separate Qwen text encoder, followed by AR visual token generation and diffusion rendering. Compared with directly conditioning diffusion on encoded text, this design introduces an additional sequential generation stage. Although the paper reports quantitative improvements, it lacks a direct diffusion baseline matched for model capacity and training budget, as well as comparisons of end-to-end inference latency. The available evidence is therefore insufficient to establish whether these gains justify the added architectural complexity and inference overhead.

Questions for authors

Inference efficiency is not evaluated. The method requires sequential AR token generation and multi-step diffusion sampling, but the paper does not report end-to-end generation latency or comparisons with baselines. It therefore remains unclear whether its training efficiency advantages come at the cost of additional inference overhead.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.