acceptodds
Under review as a conference paper at ICLR 2027

CanvasAgent: Enabling Complex Image Creation and Editing via Visual Tool Orchestration

Abstract

Complex image creation and editing often exceed a single generation or editing model: a request may require synthesizing, localizing, segmenting, editing, compositing, reading text, and enhancing within one workflow. Such requests are complex in three respects: they involve long-horizon, state-dependent execution, heterogeneous tools whose outputs feed subsequent operations, and multiple visual assets, including multiple input images. Existing multimodal tool-use agents mostly target perception, search, or domain-specific editing, or plan over a single image without training, and large-scale supervision for executable, multi-asset creation trajectories remains scarce. We introduce CanvasCraft, a large-scale multimodal tool-use dataset, and CanvasAgent, an agent that learns to orchestrate heterogeneous visual tools while tracking intermediate visual assets. CanvasCraft contains fully annotated executable trajectories and RL task specifications stratified by reasoning difficulty, trajectory length, and tool diversity. CanvasAgent is trained with SFT and then GRPO, using a hybrid reward that combines outcome- and process-level signals. On CanvasCraft-Test (250 tasks), CanvasAgent raises the overall reward of its Qwen3-VL-8B base from 0.426 to 0.821, improving both final-image alignment and trajectory quality, and human evaluation shows the same trend.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Agents, Visual Tool Orchestration, Image Creation and Editing, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EbxtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper focuses on training MLLM to use pre-constructed tools to generate and edit images. The paper first defines a toolset covering 11 tools used for processing images. The paper then proposes CanvasCraft, a 140k scale dataset including a series of tool operations to process images and 10k RL dataset for further boosting reasoning quality. Using the constructed CanvasCraft, the work then proposes CanvasAgent, a multimodal LLM based on Qwen3-VL model for first reasoning proper tools, then propose executable image editing trajectories. To train CanvasAgent, the paper first SFT on the 140k dataset, then uses RL which combines LLM scores focusing on alignment, aesthetics and trajectory quality with rule-based scores focusing on executability and efficiency. Experiments show the CanvasAgent has higher alignment score than baselines including open-source MLLM and private image edit models.

Critical strengths
  • The paper contributes a relatively large-scale tool-based image editing dataset for the community. The dataset contains both images and the editing trajectories.
  • Combining the low-level editing tool with existing image generation/editing model in the tools, which goes beyond prior work such JarvisArt, Lightroom-based work.
Critical weaknesses
  • Important categories of baseline missing. In the main experiment, Table 1, the paper does not compare proposed CanvasAgent with any tool-based MLLM agent works that train MLLM to call image editing tools. For example, JarvisArt [1], JarvisEvo [2], MIRA [3], etc. At least JarvisArt has open-sourced the model weights and demo for comparison. Given that the work also uses natural language instruction and input images that can be passed to these works, why CanvasAgent cannot be compared with them? There is indeed an advantage that CanvasAgent can proces multiple images, but the comparison on single image can be conducted. In addition, Besides using open-source MLLM, the paper should also consider using GPT/Claude private MLLM, providing the documentation of the tools and necessary instruction, and ask them to do zero-shot image editing leveraging the pre-defined tools. This is also a set of baselines; testing if the existing cutting-edge MLLM can already do this task without training. Third, the paper should also consider comparison with some MLLM agentic work such as [5].

  • The difficulty of training dataset in CanvasCraft is limited. For the editing trajectory, CanvasCraft leverage Qwen3.5-Plus to carry out the editing step by step. It is good that the paper performs filtering to ensure correctness, but the complexity of trajectory is bounded by the ability of Qwen3.5-Plus.

  • The method to train CanvasAgent is very similar to existing tool-based image editing work. Related work such as JarvisArt [1], RetouchIQ [4] have demonstrated for tool-based editing task, we can do first SFT, then RL, with variations on the reward designs. So this actually challenges the second contribution: is the whole method new? Or are the method differences mainly on the reward design?

  • The tool set complexity is limited. JarvisArt has more than 200 tools in its tool set. While the JarvisArt is based on Lightroom and inherits Lightroom's tool complexity, a tool set of only 11 tools might be too small and less challenging for today's MLLM to learn.

[1] JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agent [2] JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimization [3] MIRA: Multimodal Iterative Reasoning Agent for Image Editing [4] RetouchIQ: MLLM Agents for Instruction-Based Image Retouching with Generalist Reward [5] PhotoArtAgent: Intelligent Photo Retouching with Language Model-Based Artist Agents

Questions for authors

These questions are directly from the weakness sections. Specifically:

  1. Are there specific reasons why the work cannot be compared with the three categories of work mentioned above?
  2. Does using Qwen3.5-Plus to generate trajectories limit the complexity of the current CanvasCraft SFT dataset?
  3. What methodological novelty does CanvasAgent introduce beyond the existing SFT+RL framework and task-specific reward adaptations?
  4. Would you agree that 11 tools might be a too small action space? If so, do you plan to expand it? If not, how would it cover a broader categories of edits and be sufficiently challenging for the models (which makes me curious more about how the zero-shot GPT/Claude would perform in this task)?
2 comments

We thank the reviewer for recognizing CanvasCraft's executable trajectories and the combination of generative models with source-image operations. This setting and training resource must be distinguished from optimization novelty, and stronger comparative evidence is needed.

1. Trained agents, strong planners, and agentic workflows.

We agree with single-image comparisons on supported tasks; multi-image support does not remove the need for overlapping-task comparisons. JarvisArt, JarvisEvo, and MIRA are relevant trained agents, and PhotoArtAgent is a relevant agentic editing workflow.

Our tools cover generation, editing, grounding, segmentation, extraction, OCR, compositing, geometric transformations, and super-resolution. Comparisons should define compatible subsets before inspecting results, identify unsupported operations, and report multi-asset tasks separately. Shared inputs or different toolkits alone do not settle comparability.

A strong GPT/Claude planner should receive the same tool documentation, input assets, and intermediate feedback under comparable execution budgets. This tests whether it can solve the tasks without CanvasCraft training, which current untrained baselines cannot establish for stronger planners.

We plan these compatible-task and strong-planner comparisons. Table 1 shows training benefits in the shared environment: alignment rises from 0.493 for Qwen3-VL-8B-Instruct to 0.613 after SFT and 0.869 after SFT+RL. It does not establish superiority over existing trained agents or stronger planners.

2. Does the teacher limit CanvasCraft-SFT?

The teacher can limit retained SFT coverage: tasks Qwen3.5-Plus cannot execute successfully may be filtered out. Execution-based quality control improves reliability but does not remove this limitation.

Teacher identity alone does not establish that retained tasks are simple. Construction starts with tool-chain templates and dependencies, followed by image selection and instruction construction. The teacher executes these in the environment, producing structured arguments, intermediate assets, and final outputs. Difficulty should be assessed from these operations and dependencies.

CanvasCraft-SFT includes atomic operations and composed workflows for reliable invocation and asset handling. Table 7 reports 3,808 Multi-tool Hard trajectories among 142,798 samples; the full dataset size is not the number of complex planning examples.

CanvasCraft-RL provides 10K task specifications without reference trajectories for exploring alternative actions. The complete system need not reproduce the teacher's sequences, but this does not resolve teacher-dependent SFT coverage. We will clarify both points and distinguish difficulty annotations from demonstrated generalization.

3. Contribution beyond SFT+RL and reward adaptation.

We claim no new SFT objective or GRPO optimizer. The contribution is formulating and training executable orchestration across heterogeneous tools and coexisting assets using CanvasCraft's real-execution supervision. Task-specific rewards support this system; that description alone does not establish a new optimization method.

Calls must reference currently available assets, satisfy operation-specific constraints, and produce reusable outputs. Grounding returns boxes, SAM returns masks, Extract consumes a mask and source image, and edits or composites can feed later actions. Action validity is checked against the evolving asset state, while outcome terms assess instruction satisfaction and visual coherence.

The contribution concerns executable supervision and multi-asset policy learning. Ablations support the combined design: without outcome rewards, trajectory score is 0.907 but alignment is 0.320; without process rewards, alignment falls to 0.421 and rule-based score to 0.290. These do not isolate every state-aware check or establish reward superiority over all alternatives. We will emphasize the dataset, execution setting, and demonstrated training benefit in the contribution statement.

4. Is an 11-tool action space too small?

Tool counts are not directly comparable across interface granularities: Lightroom operations and our heterogeneous interfaces represent different actions. Each action also requires source assets and structured arguments, with validity depending on earlier outputs. Eleven tool names do not imply eleven possible actions.

Complementary capabilities do not automatically establish difficulty for the strongest MLLMs; the proposed GPT/Claude comparison provides that empirical check. We will avoid using tool count or raw trajectory length as a proxy for planning difficulty. Expansion should address missing user-facing capabilities, not tool count alone.

The authors agree that comparisons with the relevant baselines are needed, but no additional experimental results were provided during the rebuttal. Given the missing comparisons with directly relevant baselines, I cannot recommend acceptance.

Reviewer_Kp5HOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces CanvasCraft and CanvasAgent for image creation and editing through visual tool orchestration. CanvasCraft contains approximately 140K executable SFT trajectories and 10K RL task specifications. CanvasAgent is trained with SFT followed by GRPO, using a hybrid reward that evaluates both the final image and the execution process. Results on 250 held-out tasks show improvements in instruction alignment and tool execution. The main contribution is the executable trajectory dataset and training system, rather than a new planning or optimization method.

Critical strengths
  1. CanvasCraft records real tool executions rather than merely model-generated descriptions of plausible trajectories. It includes structured arguments, intermediate assets, and final images, with checks for execution success, valid asset references, and instruction–output consistency.

  2. The reported gains extend beyond tool-call formatting. The reported gains go beyond execution scores, and the reward ablations are informative. Under the paper’s evaluation, adding RL raises alignment from 0.613 to 0.869 and aesthetics from 0.711 to 0.762. The improvements are therefore not limited to tool-call formatting or execution. The ablations show a clear mismatch when the outcome reward is removed: the trajectory score reaches 0.907, but alignment drops to 0.320. Removing the process reward also hurts alignment and execution-related scores. These results show that a high process score does not guarantee a correct image and support combining outcome and process rewards in this setting.

Critical weaknesses
  1. The claimed task complexity is not convincingly demonstrated. Fig. 3 demonstrates multistage editing, but not clearly complex planning: the user specifies the read–translate–reconstruct–insert sequence and supplies the street scene. It remains unclear which decisions require intermediate visual feedback or why simpler editing approaches would fail. The qualitative evidence is also limited, lacking representative side-by-side baseline comparisons across task types and analysis of successes and failures. In the main example, OCR misses a text error, and the authors acknowledge that the subsequent correction cannot be attributed to that check. This offers little support for observation-driven correction or the broader claims about complex planning.

  2. The size of the toolkit is emphasized, but its necessity is not established. Eleven tool interfaces do not necessarily imply eleven distinct planning challenges. Grounding→SAM→Extract, for example, could be merged into one operation without changing the task. Reported tool diversity and trajectory length may thus reflect API granularity rather than task difficulty. This concern is reinforced by reverse-engineering SFT instructions from predefined tool chains, which risks tailoring tasks to the toolkit rather than demonstrating its necessity for natural user requests. Without comparisons against reduced tool sets, composite tools, or tool-category ablations, the full toolkit’s advantage over a simpler design remains unclear.

  3. The reward explicitly favors multistep execution, which weakens the planning claims. The trajectory judge assigns zero to any trajectory containing exactly one tool call, regardless of the quality of the final image. It also rewards coverage of the annotated expected tool set. These rules do not adequately distinguish skipping necessary steps from finding a simpler valid solution. Nor is there sufficient justification for treating the annotated tool set as necessary for every successful solution. Process terms account for 60% of the overall reward. Consequently, learning to make additional calls and cover the expected tools can improve the reported score without necessarily improving planning. The efficiency penalties do not resolve the more basic question of why a correct, efficient single-call solution should automatically receive zero trajectory credit. A reward that already prefers multistep execution cannot, by itself, establish that multistep execution is better.

  4. The missing baselines prevent attribution of the gains to tool orchestration. The comparisons mainly involve general-purpose multimodal models without task-specific training and image models evaluated without interactive trajectories. These baselines show that specialized training helps, but leave the central alternatives untested. Would repeated calls to the same editing backend already achieve similar results? How would a stronger general-purpose planner perform with the same tools? Does the full system retain an advantage under comparable runtime or inference budgets? The ablations remove training stages or reward components, but do not directly test the contribution of intermediate visual observations, tool diversity, or fine-grained orchestration. Given that the training procedure is standard SFT followed by GRPO, these are not peripheral analyses; they are needed to establish the paper’s main contribution.

Questions for authors

1. Do the gains persist without the hard preference for multistep execution?
Please ablate the zero-score rule for single-call trajectories and the expected-tool coverage penalties, and report final-image quality under an independent evaluator alongside changes in tool usage. How is a successful solution scored when it does not use the annotated expected tools?

2. What does the full toolkit contribute beyond simpler alternatives?
Please compare the full system with an iterative editor using the same backend and with composite tools that encapsulate fixed subroutines. Which tool categories materially improve task completion, and do those improvements remain worthwhile after accounting for runtime or execution cost?

3. What evidence shows that the agent changes its decisions in response to intermediate results?
Can the authors provide examples in which different intermediate outcomes for the same task lead to different, appropriate subsequent actions, or a controlled comparison without intermediate visual observations? More side-by-side generation and editing results would also help identify the specific constraints that simpler approaches fail to satisfy.

1 comment

We thank the reviewer for recognizing real-execution supervision and distinguishing gains from their causes. RL improves alignment from 0.613 to 0.869 and aesthetics from 0.711 to 0.762 under the reported evaluation, but these gains and ablations do not establish the necessity of every tool, interface, or observation.

1. Do gains persist without multistep or expected-tool preferences?

The single-call zero trajectory-score rule was used only in early RL and removed later. It discouraged premature stopping during multi-step learning and affected the trajectory component, not final-image rewards or the entire reward. Later RL did not automatically zero single-call trajectory scores through this rule.

We will clarify the schedule, label the scope of the prompt in Appendix A.3, and distinguish early-training, later-training, and evaluation configurations. This is not an ablation against training without the rule; current experiments do not isolate its contribution.

Expected-tool coverage is a separate concern. Annotations do not prove every listed tool necessary for every valid solution. A successful alternative omitting a key annotated tool can receive a lower process score through trajectory assessment and the missing-tool penalty despite high final-image scores.

Averaging action validity over calls does not eliminate these preferences; Table 2 removes process terms together without isolating them. A targeted comparison would remove both preferences separately and jointly, independently evaluate final images, and report tool usage. This has not been performed. Current scores reflect the specified objective; they do not prove that longer or reference-matching solutions are better.

2. What does the full toolkit add over iterative editing or composite tools, and at what cost?

Our interfaces expose complementary controls: Crop selects source pixels by coordinates, Extract isolates masked content, Rotate/Flip apply specified transformations, and Overlay positions existing content. Grounding, SAM, and OCR return boxes, masks, and text for later calls, which ImageEdit does not directly return.

This distinction does not establish a performance gap: repeated editing may satisfy many final-image requests without these representations. The relevant comparison uses the same editing backend, input assets, and comparable execution budget, evaluated on requested output constraints.

Composite tools are legitimate alternatives. Grounding, SAM, and Extract expose independently reusable outputs, but a combined operation can simplify a subroutine and coexist with primitives. Current results do not establish superior granularity. Tool-category ablations and composite-tool comparisons would distinguish capability benefits from API-call counts.

Existing baselines and Table 2 establish training benefits, not the separate effects of toolkit breadth or fine-grained orchestration. Call counts are insufficient cost measures: generation, editing, OCR, and scripts have different costs. Current tables do not establish an advantage at matched runtime or inference cost.

We plan to evaluate a stronger general-purpose planner with the same toolkit and comparable budgets. Simpler-toolkit comparisons should report independent final-image quality, generation/editing calls, execution time, and resource usage. Pending those controls, we claim effective learned orchestration here, not superiority of the full toolkit or granularity over every simpler design.

3. Do intermediate visual results change subsequent decisions?

Figure 3's instruction gives the high-level read-translate-reconstruct-insert sequence, but no tool names, arguments, or asset bindings. The policy constructs the executable workflow. This supports multi-image execution and cross-asset dependencies, with limited evidence for autonomous high-level decomposition.

As Section 4.4 states, OCR missed the text error; the later correction cannot be attributed to it. We will describe the case accordingly. Access to intermediate observations alone does not establish their causal benefit.

Direct evidence would be appropriate action changes following different intermediate outcomes for the same task, or a comparison without intermediate visual observations that retains instructions, execution history, and asset identifiers for valid calls. The paper lacks that controlled comparison.

We plan side-by-side outputs across task types from existing evaluations, with execution traces and both successes and failures. These will identify satisfied or missed constraints rather than infer useful feedback from a checking call. They can clarify behavior; the causal value of visual feedback still requires the controlled evidence above.

Reviewer_MD3xOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces CanvasAgent, a multimodal agent for complex image creation and editing. The authors also construct CanvasCraft, which is a dataset that has around 140K executable supervised fine-tuning trajectories and 10K RL tasks. The agent is trained using a 2 stage SFT and GRPO framework and has a hybrid reward that combines final-image alignment, aesthetic quality, trajectory quality and rule-based execution validity. The system operates with 11 visual tools and they cover generation, editing, grounding, segmentation, OCR, compositing and some other operations.

Critical strengths
  • The paper addresses a nice complex problem in image creation and editing.
  • It provides a large-scale dataset of approx 140K executable trajectories and 10K RL tasks.
  • The experiments show substantial improvements over the base model.
  • The ablation studies demonstrate the complementary benefits of SFT, RL and the hybrid reward also. The multi-stage case study is also informative.
Critical weaknesses
  • The same LLM judge is used for both RL training and automatic evaluation. Now this can introduce some bias in the reported improvements. The human eval is also limited and has just 12 tasks.
  • Although RL improves multi-tool planning, the paper provides limited analysis of how well the agent generalizes to longer or more complex tool chains beyond those represented in the training data.
  • The evaluation relies entirely on the authors' synthetic benchmark. There is no independent benchmark or comparison against existing trained tool-orchestration systems that can demonstrate generalization.
  • The reward function assigns zero trajectory score to single-tool solutions regardless of their quality. This I fell can encourage unnecessary tool calls and penalize valid and efficient workflows.
  • All results come from a single training run and there is no reported training variance or sensitivity analysis for the reward weights. This I feel can limit confidence in the robustness of the improvements.
Questions for authors
  • Can you evaluate CanvasAgent using an independent LLM judge or maybe do a larger human study. Pls also include some image-only baselines.
  • Please evaluate on some independent benchmark or maybe real user instructions and compare it with existing trained tool-orchestration systems.
  • Why does the trajectory judge assign zero score to all single-tool solutions like even when they satisfy the task? How sensitive are the results to this rule?
  • Can the authors please report multiple training runs or maybe do a sensitivity analysis of the reward weights?
  • Please also address the weaknesses.
2 comments

We thank the reviewer for recognizing CanvasCraft, the gains over the base model, and the complementary roles of SFT, RL, and the hybrid reward. We distinguish benchmark performance, evaluator and training robustness, and generalization below.

1. Independent evaluation, human evaluation, and image-only baselines.

Using Qwen3.5-Plus for both RL rewards and evaluation can favor policies adapted to that judge. Automatic scores establish gains under that objective, not robustness across evaluators.

The blinded study covers 12 randomly selected tasks rated by 15 annotators: CanvasAgent achieves task alignment of 3.97, versus 2.68 and 2.93 for the untrained 8B and 32B baselines. Its limited coverage and omission of SFT-only and image-only models mean it cannot independently establish all Table 1 comparisons.

Table 1 already includes Qwen-Image-2.0, Wan2.7-Image, and GPT-Image-2, receiving all input images when applicable. CanvasAgent has higher alignment under the reported judge, while these models have higher aesthetic scores. We will state this trade-off explicitly, without claiming an unqualified improvement in image quality.

We plan to rescore existing outputs, including these image-only baselines, with multiple MLLMs distinct from the training judge. Evaluators will receive the instruction, input images, and final output, without model identities or execution narratives. Results will be reported separately for each evaluator. This evaluation remains to be completed.

2. Generalization, independent tasks, and trained-agent comparisons.

CanvasCraft-Test is a held-out split of CanvasCraft-RL. It tests unseen tasks within that construction process, not transfer to independently constructed benchmarks or workflows beyond the training distribution. Likewise, training tasks annotated as requiring nine or more steps establish coverage, not length extrapolation. We will distinguish these claims.

Selecting among coexisting assets, passing boxes and masks between tools, and maintaining cross-tool dependencies explain the intended difficulty but still require task-level evidence. We plan to analyze performance by reasoning difficulty, expected trajectory length, tool diversity, and input-image count, reporting held-out tool combinations separately from within-distribution results.

The distinctive toolkit does not justify omitting trained-agent comparisons. Our proposed comparison uses predefined compatible subsets and reports broader multi-asset tasks separately. Independently sourced instructions are also needed to test transfer beyond synthetic task construction; current results do not establish this external generalization.

3. The single-call zero-score rule and efficient valid solutions.

An implementation detail was insufficiently described: the single-call zero trajectory-score rule was enabled only during early RL and removed later. It discouraged premature termination while the policy learned multi-step execution. It affected the trajectory component, not final-image scores or the entire reward. Later RL did not automatically reject single-call trajectories through this rule.

We will revise the reward description and label the scope of the prompt in Appendix A.3, documenting early-training, later-training, and evaluation configurations separately. This schedule clarification is not a controlled ablation. Current experiments do not isolate the rule's contribution, so they cannot establish its necessity or whether removing it from the outset would leave performance unchanged.

A correct, efficient solution should not be deemed unsuccessful merely for using fewer calls. Expected-tool annotations and associated process penalties can still favor one route. Their effect must be distinguished from final-image task satisfaction; annotations do not prove every listed tool necessary.

4. Training variance and reward-weight sensitivity.

Weights were chosen heuristically: 0.3 for alignment, 0.1 for aesthetics, 0.2 for trajectory quality, and 0.4 for rule-based validity. They prioritize instruction satisfaction and valid execution, but their optimality and insensitivity to perturbation have not been established.

Table 2 supports combining outcome and process rewards and using SFT before RL; it does not replace multiple-seed training or weight-sensitivity analysis. Results come from a single training run, without those additional robustness results. We will state this limitation and avoid presenting component ablations as evidence of training stability or weight robustness.

Thank you for the rebuttal. However, I feel that the rebuttal only acknowledges my concerns rather than addressing them. Therefore, I can not increase my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.