- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper focuses on training MLLM to use pre-constructed tools to generate and edit images. The paper first defines a toolset covering 11 tools used for processing images. The paper then proposes CanvasCraft, a 140k scale dataset including a series of tool operations to process images and 10k RL dataset for further boosting reasoning quality. Using the constructed CanvasCraft, the work then proposes CanvasAgent, a multimodal LLM based on Qwen3-VL model for first reasoning proper tools, then propose executable image editing trajectories. To train CanvasAgent, the paper first SFT on the 140k dataset, then uses RL which combines LLM scores focusing on alignment, aesthetics and trajectory quality with rule-based scores focusing on executability and efficiency. Experiments show the CanvasAgent has higher alignment score than baselines including open-source MLLM and private image edit models.
Critical strengths
- The paper contributes a relatively large-scale tool-based image editing dataset for the community. The dataset contains both images and the editing trajectories.
- Combining the low-level editing tool with existing image generation/editing model in the tools, which goes beyond prior work such JarvisArt, Lightroom-based work.
Critical weaknesses
-
Important categories of baseline missing. In the main experiment, Table 1, the paper does not compare proposed CanvasAgent with any tool-based MLLM agent works that train MLLM to call image editing tools. For example, JarvisArt [1], JarvisEvo [2], MIRA [3], etc. At least JarvisArt has open-sourced the model weights and demo for comparison. Given that the work also uses natural language instruction and input images that can be passed to these works, why CanvasAgent cannot be compared with them? There is indeed an advantage that CanvasAgent can proces multiple images, but the comparison on single image can be conducted. In addition, Besides using open-source MLLM, the paper should also consider using GPT/Claude private MLLM, providing the documentation of the tools and necessary instruction, and ask them to do zero-shot image editing leveraging the pre-defined tools. This is also a set of baselines; testing if the existing cutting-edge MLLM can already do this task without training. Third, the paper should also consider comparison with some MLLM agentic work such as [5].
-
The difficulty of training dataset in CanvasCraft is limited. For the editing trajectory, CanvasCraft leverage Qwen3.5-Plus to carry out the editing step by step. It is good that the paper performs filtering to ensure correctness, but the complexity of trajectory is bounded by the ability of Qwen3.5-Plus.
-
The method to train CanvasAgent is very similar to existing tool-based image editing work. Related work such as JarvisArt [1], RetouchIQ [4] have demonstrated for tool-based editing task, we can do first SFT, then RL, with variations on the reward designs. So this actually challenges the second contribution: is the whole method new? Or are the method differences mainly on the reward design?
-
The tool set complexity is limited. JarvisArt has more than 200 tools in its tool set. While the JarvisArt is based on Lightroom and inherits Lightroom's tool complexity, a tool set of only 11 tools might be too small and less challenging for today's MLLM to learn.
[1] JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agent [2] JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimization [3] MIRA: Multimodal Iterative Reasoning Agent for Image Editing [4] RetouchIQ: MLLM Agents for Instruction-Based Image Retouching with Generalist Reward [5] PhotoArtAgent: Intelligent Photo Retouching with Language Model-Based Artist Agents
Questions for authors
These questions are directly from the weakness sections. Specifically:
- Are there specific reasons why the work cannot be compared with the three categories of work mentioned above?
- Does using Qwen3.5-Plus to generate trajectories limit the complexity of the current CanvasCraft SFT dataset?
- What methodological novelty does CanvasAgent introduce beyond the existing SFT+RL framework and task-specific reward adaptations?
- Would you agree that 11 tools might be a too small action space? If so, do you plan to expand it? If not, how would it cover a broader categories of edits and be sufficiently challenging for the models (which makes me curious more about how the zero-shot GPT/Claude would perform in this task)?
2 comments
The authors agree that comparisons with the relevant baselines are needed, but no additional experimental results were provided during the rebuttal. Given the missing comparisons with directly relevant baselines, I cannot recommend acceptance.
We thank the reviewer for recognizing CanvasCraft's executable trajectories and the combination of generative models with source-image operations. This setting and training resource must be distinguished from optimization novelty, and stronger comparative evidence is needed.
1. Trained agents, strong planners, and agentic workflows.
We agree with single-image comparisons on supported tasks; multi-image support does not remove the need for overlapping-task comparisons. JarvisArt, JarvisEvo, and MIRA are relevant trained agents, and PhotoArtAgent is a relevant agentic editing workflow.
Our tools cover generation, editing, grounding, segmentation, extraction, OCR, compositing, geometric transformations, and super-resolution. Comparisons should define compatible subsets before inspecting results, identify unsupported operations, and report multi-asset tasks separately. Shared inputs or different toolkits alone do not settle comparability.
A strong GPT/Claude planner should receive the same tool documentation, input assets, and intermediate feedback under comparable execution budgets. This tests whether it can solve the tasks without CanvasCraft training, which current untrained baselines cannot establish for stronger planners.
We plan these compatible-task and strong-planner comparisons. Table 1 shows training benefits in the shared environment: alignment rises from 0.493 for Qwen3-VL-8B-Instruct to 0.613 after SFT and 0.869 after SFT+RL. It does not establish superiority over existing trained agents or stronger planners.
2. Does the teacher limit CanvasCraft-SFT?
The teacher can limit retained SFT coverage: tasks Qwen3.5-Plus cannot execute successfully may be filtered out. Execution-based quality control improves reliability but does not remove this limitation.
Teacher identity alone does not establish that retained tasks are simple. Construction starts with tool-chain templates and dependencies, followed by image selection and instruction construction. The teacher executes these in the environment, producing structured arguments, intermediate assets, and final outputs. Difficulty should be assessed from these operations and dependencies.
CanvasCraft-SFT includes atomic operations and composed workflows for reliable invocation and asset handling. Table 7 reports 3,808 Multi-tool Hard trajectories among 142,798 samples; the full dataset size is not the number of complex planning examples.
CanvasCraft-RL provides 10K task specifications without reference trajectories for exploring alternative actions. The complete system need not reproduce the teacher's sequences, but this does not resolve teacher-dependent SFT coverage. We will clarify both points and distinguish difficulty annotations from demonstrated generalization.
3. Contribution beyond SFT+RL and reward adaptation.
We claim no new SFT objective or GRPO optimizer. The contribution is formulating and training executable orchestration across heterogeneous tools and coexisting assets using CanvasCraft's real-execution supervision. Task-specific rewards support this system; that description alone does not establish a new optimization method.
Calls must reference currently available assets, satisfy operation-specific constraints, and produce reusable outputs. Grounding returns boxes, SAM returns masks, Extract consumes a mask and source image, and edits or composites can feed later actions. Action validity is checked against the evolving asset state, while outcome terms assess instruction satisfaction and visual coherence.
The contribution concerns executable supervision and multi-asset policy learning. Ablations support the combined design: without outcome rewards, trajectory score is 0.907 but alignment is 0.320; without process rewards, alignment falls to 0.421 and rule-based score to 0.290. These do not isolate every state-aware check or establish reward superiority over all alternatives. We will emphasize the dataset, execution setting, and demonstrated training benefit in the contribution statement.
4. Is an 11-tool action space too small?
Tool counts are not directly comparable across interface granularities: Lightroom operations and our heterogeneous interfaces represent different actions. Each action also requires source assets and structured arguments, with validity depending on earlier outputs. Eleven tool names do not imply eleven possible actions.
Complementary capabilities do not automatically establish difficulty for the strongest MLLMs; the proposed GPT/Claude comparison provides that empirical check. We will avoid using tool count or raw trajectory length as a proxy for planning difficulty. Expansion should address missing user-facing capabilities, not tool count alone.