acceptodds
Under review as a conference paper at ICLR 2027

PLANNING AND RENDERING IN CONCERT: DEEPFUSION OF AUTOREGRESSIVE LAYOUTS AND DIFFUSION FOR VISUAL TEXT GENERATION

Abstract

Generating text-rich images from prompts requires both textual fidelity and the coherent integration of text into the surrounding image. An explicit layout can provide structured guidance about what text should appear and where, but a well-formed plan alone does not guarantee that the renderer will realize it faithfully. Existing layout-based AR–diffusion systems typically optimize planning and rendering separately, preventing the planner’s representations from being adapted jointly with image synthesis. We introduce DuetGen, an autonomous visual text generator built on DeepFusion, which jointly learns autoregressive planning and continuous diffusion rendering. DeepFusion conditions a diffusion transformer on the planner’s prompt and bbox–content hidden states, allowing rendering supervision to shape the representations connecting textual plans with visual outputs. Its joint objective combines autoregressive plan supervision, text-region-weighted diffusion learning, and auxiliary coordinate supervision to maintain structured planning, emphasize text-bearing regions, and improve the spatial precision of planner representations. During inference, Phase-Aware Attention Modulation strengthens the correspondence between image regions and their matched coordinate and content states, facilitating region-specific execution of the generated plan. With a 2B planner and a 4B single-stream DiT, DuetGen achieves 0.8293 word accuracy on CVTG-2K and 0.938 accuracy on LongText-Bench, closely matching the substantially larger Qwen-Image on both benchmarks. These results demonstrate the value of jointly learned planning representations and region-specific rendering for autonomous visual text generation

Primary area:
generative models
Keywords:
visual text generation, text rendering, text-to-image generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PjGzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DuetGen, an autonomous visual text generation framework that integrates autoregressive layout planning with continuous diffusion-based image generation. The proposed DeepFusion architecture connects a 2B autoregressive planner and a 4B diffusion transformer through jointly optimized planner representations.

Critical strengths

The paper addresses an important challenge in visual text generation: producing accurate text while maintaining consistency between the intended layout and the rendered image. Unlike conventional systems that separately optimize layout planning and image synthesis, DuetGen enables diffusion supervision to influence the planner's hidden representations.

Critical weaknesses
  1. Insufficient evaluation of diverse and challenging visual text generation.
  • Limited evidence under challenging visual layouts. The qualitative examples in Figure 3 mainly contain relatively short phrases and titles. Moreover, Table 5 reports an OCR accuracy of 69.93 on the StyledTextSynth subset, indicating substantial remaining errors in another text-rendering setting. More comprehensive evaluations on dense paragraphs, overlapping text regions, and complex typography are necessary to support the broader claims of the paper.
  • Insufficient analysis of textual complexity. Table 1 evaluates CVTG-2K with only 2–5 text regions, while Table 2 reports a single aggregate accuracy on LongText-Bench. The paper does not provide a systematic breakdown by text length, vocabulary rarity, special characters, mixed alphanumeric strings, or complex multi-line layouts. Consequently, it remains unclear whether the reported improvements generalize to more challenging and linguistically diverse text-rendering scenarios.
  1. The result is not SOTA.
  • DuetGen does not outperform Qwen-Image on the overall results of CVTG-2K or LongText-Bench. Qwen-Image achieves slightly higher word accuracy, NED, and CLIPScore. Although DuetGen uses fewer parameters, its performance advantage over the strongest existing model remains unestablished.
  • Qwen-Image is excluded from both the qualitative comparisons in Figure 3 and the human evaluation in Table 7.
  1. Insufficient evidence for the necessity and effectiveness of the proposed joint optimization.
  • In Table 3, diffusion-only planner optimization actually reduces word accuracy from 0.7032 to 0.6788, while adding autoregressive plan supervision improves it only marginally to 0.6832. The substantial improvement to 0.8309 appears after introducing auxiliary coordinate supervision. This raises the question of whether the gains primarily originate from stronger spatial supervision rather than the proposed joint planning–rendering optimization.
  1. The writing of the paper lacks proper standardization.
  • The text in the Figure 2 is not clear.
  • The use of periods and commas in the formula is incorrect.
Questions for authors

See weakness and solve my concerns.

Reviewer_H7EzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce DuetGen, an autonomous visual text generator that deeply integrates autoregressive layout planning and continuous diffusion rendering via DeepFusion. DeepFusion jointly learns prompt-and-plan representations through a combination of autoregressive plan supervision, text-region-weighted diffusion learning, and auxiliary coordinate supervision. It further strengthens region-specific plan execution at inference time with Phase-Aware Attention Modulation (PAM). With a 2B planner and a 4B single-stream DiT, the method reports 0.8293 word accuracy on CVTG-2K and 0.938 accuracy on LongText-Bench, closely matching the substantially larger Qwen-Image.

Critical strengths
  1. DeepFusion provides a coherent integration of planning and continuous rendering through jointly learned representations, coordinated objectives (plan, text-region-weighted diffusion, and auxiliary coordinate losses), and region-specific execution via PAM. This design directly addresses the limitations of separated AR–diffusion systems and discrete fully autoregressive models, and the ablations (joint vs. separate training, spatial probe readability, and diffusion-side transfer) supply supporting evidence that the joint interface improves planning–rendering consistency.
Critical weaknesses
  1. Limited competitiveness of the results. Recent methods report higher accuracy on LongText-Bench, for example GenClaw [1] (0.989) and Emu3.5 [2] (0.976). These are general-purpose generation models rather than specialized text-rendering systems. A purely numerical comparison is not entirely fair, and DuetGen's results are close to Qwen-Image with a much smaller model. Still, the gap to the current state of the art weakens the practical and academic significance of the contribution.

[1] GenClaw: Code-driven agentic image generation. arXiv preprint arXiv:2605.30248, 2026.

[2] Emu3.5: Native multimodal models are world learners. arXiv preprint arXiv:2510.26583, 2025.

Questions for authors
  1. The description of PAM is difficult to follow. Additional clarity on the precise implementation and ablation of its design choices would help.
  2. Could the authors evaluate DuetGen on larger or more recent planner and diffusion backbones to assess whether the DeepFusion approach generalizes and can reach more compelling absolute results?
  3. Could the authors more explicitly demonstrate why the deep-fusion design is necessary and what concrete benefits it provides, particularly in resolving the issues of the Separated and Sequential generation paradigms illustrated in Figure 1?
Reviewer_jmsrOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on text rendering and proposes a framework that jointly optimizes structured AR planning and continuous diffusion rendering. Specifically, it conditions a diffusion transformer on the planner's prompt and bbox–content hidden states. The joint training objective combines autoregressive planning supervision, text-region-weighted diffusion learning, and auxiliary coordinate supervision to preserve structured planning capabilities, emphasize text-bearing regions, and improve the spatial precision of planner representations. The proposed method achieves competitive performance with Qwen-Image on two benchmarks.

Critical strengths
  1. The authors make a substantial effort to train both the planner and the generative model from scratch and jointly optimize them, achieving strong quantitative results.
  2. The paper provides comprehensive ablation studies, including analyses of the three loss components, planner scale, text-region weighting and PAM, backbone transfer, and PAM design.
  3. The paper is clearly written and easy to follow. The proposed method is also relatively simple and straightforward to implement.
Critical weaknesses
  1. I have some concerns regarding the novelty of the proposed method. Specifically, L_plan follows a standard autoregressive training objective, while L_diff adopts a standard diffusion loss with higher weights assigned to text regions. I would appreciate further clarification on the motivation and benefits of introducing L_aux.
  2. The method appears to rely on quotation marks to identify the text to be rendered. It is unclear how well the model handles open-ended prompts, particularly when the target text is not explicitly marked or enclosed in quotation marks.
  3. The paper could benefit from comparisons with more recent text-to-image generation models, including both autoregressive-based and diffusion-based approaches.
  4. The generated images appear to lack fine-grained details and exhibit somewhat less vivid colors. For example, in the third and fourth rows of Figure 3, although the target text is rendered correctly, the model seems to sacrifice visual details and color richness to achieve accurate text rendering.
Questions for authors
  1. The paper mentions that generic text-to-image pretraining was conducted using only 16 NVIDIA A800 GPUs. Given that the proposed 4B model is randomly initialized and trained from scratch, I am curious about the approximate training time required to achieve satisfactory performance. Additionally, why did the authors choose to train the model from scratch rather than fine-tune an existing text-to-image model?
  2. In Equation (3), which token positions do H_p and H_s correspond to? The current description seems insufficiently clear.
  3. How sensitive is the proposed method to text length? Although the paper demonstrates results with different text regions, it seems to lack an investigation into the impact of text length on rendering performance.
  4. The paper repeatedly emphasizes the concept of "DeepFusion", but provides limited explanation of this concept and does not appear to include relevant references. Could the authors elaborate on what DeepFusion specifically refers to in this work and provide appropriate citations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.