acceptodds
Withdrawn from ICLR 2027

ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-to-image (T2I) models can produce visually compelling images, yet they remain limited on open-world tasks that require complex semantic understanding, multi-step reasoning, and the integration of external world knowledge. Existing efforts introduce agent capabilities into image generation, but they either prescribe a fixed workflow or place only a subset of the open-world image generation process under agent control. Consequently, reasoning, tool invocation, and image generation are not coordinated by a single policy. We propose ToolArtist, a fully model obtained by post-training a Unified Multimodal Model (UMM). ToolArtist dynamically orchestrates reasoning, external tool use, and native image generation within one unified policy. During Supervised Fine-Tuning (SFT), we equip a teacher agent with search tools alongside an image-generation tool. We then convert the collected trajectories into a UMM compatible format, where the image-generation tool is concealed while the resulting generated images are retained. During Reinforcement Learning (RL), we develop an agentic RL infrastructure for UMMs and introduce Reason-Act-Draw GRPO (RAD-GRPO), which uses complementary intent and quality rewards to jointly optimize the model. Experiments show that placing the entire open-world image-generation process under an agent policy consistently outperforms approaches with fixed pipelines or only partially agent-controlled components. We release the training data and the complete post-training infrastructure.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agentic Image Generation, Reinforcement Learning, Unified Multimodal Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XmonOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ToolArtist post-trains Emu3.5 for knowledge-grounded image generation through reasoning, external search and native visual output. It uses teacher trajectories for SFT, then applies GRPO with caption-intent and image-quality rewards to text and image tokens. The proposed policy also permits further search and redrawing after an image has been generated. Evaluation covers WISE and WorldGenBench-Humanities.

Critical strengths
  1. Factual image generation is a useful application, and adapting both evidence acquisition and visual synthesis to the task is a reasonable direction.

  2. The system reports competitive results, especially on WorldGenBench-Humanities. The WISE improvement from 0.74 after SFT to 0.79 after RL supports the effectiveness of the overall RL recipe. The source-summary ablation also offers a useful finding about retrieval quality.

Critical weaknesses
  1. Limited novelty. Unify-Agent already integrates search, grounded recaptioning and native image generation within a unified model. The proposed workflow mainly adds a post-generation review and refinement loop. The RL formulation also offers limited methodological novelty: it uses GRPO with text-intent and image-quality rewards, a combination already used by Gen-Searcher Extending the updates to image-generation tokens changes the scope of optimization, but does not introduce a distinct optimization method. Overall, the contribution appears to be an incremental combination of existing ideas, which falls short of the claimed new paradigm. What substantive methodological advance goes beyond these extensions?

  2. The experiments do not establish the value of the added components. The WISE improvement from 0.74 after SFT to 0.79 after RL supports the complete training recipe, but does not show that updating the generator is beneficial or that reviewing and revising generated images helps. There is neither a control that disables the review loop nor a frozen-generator baseline that optimizes only search and captions. Comparisons with other systems also change the backbone, data and retrieval tools. Could the authors isolate these two additions under comparable settings, reporting image quality before and after revision and the additional generation cost? These controls are central to assessing whether the limited methodological changes offer a meaningful practical improvement.

Questions for authors

See Weaknesses.

Reviewer_EemEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ToolArtist, an agentic image-generation system built by post-training the unified multimodal model Emu3.5. A single autoregressive policy produces reasoning, invokes text and image search tools, and natively generates images. The training pipeline first collects teacher trajectories using search tools and an external image generator, then converts the generated images into native visual-token supervision for the UMM. Subsequently, Reason–Act–Draw GRPO (RAD-GRPO) optimizes complete trajectories using complementary generation-intent and image-quality rewards, together with auxiliary rewards. ToolArtist achieves WISE and WorldGenBench-Humanities scores of 0.79 and 22.10, respectively, outperforming the reported non-proprietary baselines overall. The paper also reports ablations on reinforcement learning and source-aware image-search summaries.

Critical strengths
  1. A coherent formulation of search and native generation under one policy. Treating both tool invocation and image generation as policy actions provides a clear formulation for learning the complete task trajectory. The distinction between policy-generated tokens and environment observations is explicitly incorporated into both SFT and RL objectives.

  2. A practical method for constructing multimodal agent supervision. Converting external image-generation calls into caption and visual-token targets allows teacher trajectories to supervise a UMM’s reasoning, tool use, and native generation within the same sequence. The resulting 7000+ trajectory dataset and accompanying training infrastructure could be useful resources.

  3. Useful evidence about retrieval context. The source-summary ablation shows that identifying what a retrieved image actually depicts is important for factual generation. The qualitative comparisons also expose remaining errors rather than showing only fully successful examples.

Critical weaknesses
  1. The benefit of integrating reasoning, tool use, and image generation into a single policy is not directly established. The paper’s central claim is that reasoning, search, and native image generation benefit from being controlled within one unified policy. However, the experiments do not compare this design against a modular counterpart using the same UMM for reasoning and search while invoking image generation as a separate tool, even when that tool is implemented by the same UMM. Such a comparison would hold model capacity and generator capability fixed while testing whether keeping generation within the ongoing policy provides an advantage over an explicit handoff. Without this control, the results do not establish whether unified execution itself improves performance over separating the reasoning/search process from generation.

  2. The distinction from closely related agentic image generation systems is insufficiently developed.
    Although SearchGen and GenEvolve are cited, neither receives a dedicated methodological comparison or quantitative evaluation. This matters because SearchGen also co-trains a search agent and a generator. ToolArtist’s shared autoregressive policy is a meaningful architectural distinction, but the manuscript does not sufficiently separate this distinction from the broader claim of jointly improving search and generation. The current comparisons with Unify-Agent and GenSearcher therefore leave the scope of the novelty and empirical advantage incompletely established.

  3. Inference cost and the fairness of computational budgets are unclear.
    The appendix reports approximately four tool calls and 20.5k input tokens per SFT trajectory, but these are teacher-data statistics, not inference-time measurements for ToolArtist. Moreover, one tool call can contain multiple search queries and trigger additional webpage-reading operations. The paper does not report statistics about evaluation-time search queries, reader calls, reasoning tokens, image-generation attempts, or end-to-end latency. Without these measurements and the corresponding baseline budgets, it is difficult to determine whether the reported advantage reflects a better decision policy or greater retrieval and generation expenditure.

  4. Context management and generation quality under growing multimodal histories are insufficiently characterized.
    The method accumulates retrieved text, reference images, and previous generations in the context, yet the manuscript does not specify the effective context limit or history-management policy, or analyze image generation quality as histories become longer.

Questions for authors
  1. Have you evaluated a modular variant that uses the same UMM checkpoint for reasoning and search but invokes that same UMM separately as an image-generation tool? With equivalent access to the generation caption, retrieved evidence, and relevant history, matched search and drawing budgets, and the same opportunity for revision, does native generation within the continuing policy perform better? Please specify any differences in training or context access that would affect this comparison.

  2. Beyond sharing an autoregressive backbone, what capabilities or optimization mechanisms distinguish ToolArtist from more related works like SearchGen and GenEvolve?

  3. The effectiveness of the proposed method would be more clear if the authors can provide inference statistics, for example, what are the mean and median numbers of text-search queries, image-search queries, LLM Reader calls, reasoning tokens, and image-generation attempts per task? What are the maximum interaction and token budgets, and the end-to-end latency under the reported hardware?

  4. What fraction of evaluated tasks contain more than one drawing action, and what is the score change from the first to the final image on those tasks?

Reviewer_pxjEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on knowledge-intensive text-to-image generation in open-world settings and proposes ToolArtist, which unifies reasoning, external tool use, and native image generation under a single UMM policy. The method is based on Emu3.5. During SFT, a teacher agent equipped with text/image search and an external image-generation tool collects trajectories. Image-generation tool calls are then hidden and converted into native visual captions and image tokens, yielding 7,132 training trajectories. During RL, RAD-GRPO combines an intent reward evaluating the final caption and a quality reward evaluating the final image to optimize complete trajectories end to end. ToolArtist outperforms existing open-source UMMs and partially agentic methods on WISE and WorldGenBench-Humanities.

Critical strengths
  1. Clear problem definition and writing: The authors treat image generation as an action autonomously selected by the policy and unify it with reasoning and search within the same UMM.
  2. Novel methodology and effective experimental support: RAD-GRPO uses intent and quality rewards to guide generation planning and the final image, respectively. It shows substantial improvements over existing agentic methods on WorldGenBench-Humanities. The ablation on source-aware summarization also provides a valuable reference.
  3. The training data and complete SFT/RL infrastructure are open-sourced, and the appendix provides full prompts and trajectory examples, making an important contribution to community development and follow-up research.
Critical weaknesses
  1. In Table 1, how do the authors explain the performance drop relative to the baseline on the Time dimension of WISE? Could you provide insights into the underlying causes?

  2. The SFT target images come from Nano Banana Pro, which achieves WISE 0.87 / WGB 28.62 without search. How much of the performance improvement comes from distilling the teacher’s images rather than agentic behavior itself?

  3. Could the authors analyze inference-time calls to each tool to clarify the contribution of the agentic system to UMM image generation? How does ToolArtist perform when the number of tool calls is limited? I suggest that the authors include more quantitative analysis of tool use in the ToolArtist system in the appendix.

Questions for authors

Please see the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.