- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces WeAgent-MMGenEdit, a framework that retrieves, verifies, and organizes external evidence for image generation and editing. It trains both the agent and image model with SFT and RL, and provides training data and a bilingual benchmark. Component ablations and results on two additional public benchmarks support the effectiveness of the approach.
Critical strengths
-
Effective training supported by useful experiments. The paper develops a practical training recipe for both the agent policy and the image backend. Controlled comparisons and component ablations support their respective contributions. Results on KnowGen and Mind-Bench provide additional evidence that the improvements extend beyond the proposed benchmark.
-
Substantial data and evaluation contribution. The paper provides a structured pipeline for constructing multimodal tasks and collecting training trajectories, together with a bilingual benchmark covering generation and editing. Evaluation of the agent process, generation input, and final image offers a useful basis for diagnosing failures.
Critical weaknesses
-
The distinctive harness contribution and practical cost need clearer comparison. Several design ideas overlap with existing frameworks such as JarvisHub and Qwen-Image-Agent. While the ablations demonstrate the usefulness of individual components, the specific advance over these frameworks could be better established. The paper also lacks a quantitative comparison of inference costs with recent agentic generation methods, including the additional calls for visual verification.
-
Automatic evaluation needs further validation. The supplement clarifies benchmark construction and scoring, but does not report agreement between the automatic evaluator and human judgments of generated outputs. A human audit would strengthen confidence in the reported rankings, particularly for visual knowledge accuracy and editing quality.
Questions for authors
See weaknesses. It seems that some figures are ai-generated, can you replace them with better human-drafted figures?
3 comments
Response to Reviewer Gpc8 (part2)
W1-2. Inference cost of Policy Model. Thanks for the valuable comments! We report the average per-case cost on WeBench-MMGenEdit below.
Table R1. Inference cost per trajectory. Turns, calls, and tokens are averaged over all 300 cases of WeBench-MMGenEdit. Verification tokens are produced by the separately deployed VLM. W-Avg is with GPT-Image-2 (Gen/Edit).
| Method | Harness | Turns | Policy Output Tokens | Verify Calls | Verify Tokens | Integrate Calls | Integrate Tokens | W-Avg (Gen/Edit) |
|---|---|---|---|---|---|---|---|---|
| GenEvolve-8B | GenEvolve | 5.41 | 2,090 | – | – | – | – | 51.21 / 41.61 |
| GenSearcher-8B | GenSearcher | 4.36 | 4,281 | – | – | – | – | 53.33 / 44.03 |
| Qwen3-VL (30B-A3B) | WeHarness | 8.74 | 2,518 | 0.65 | 1,067 | 0.80 | 1,512 | 51.86 / 45.80 |
| Qwen-3.5 (35B-A3B) | WeHarness | 10.63 | 2,903 | 2.16 | 3,024 | 1.05 | 1,685 | 62.75 / 55.46 |
| Qwen-3.8 (27B) | WeHarness | 10.32 | 3,042 | 1.92 | 2,526 | 0.58 | 1,668 | 62.81 / 60.79 |
| Kimi-K2.6 (1T-A32B) | WeHarness | 9.00 | 4,537 | 1.78 | 1,138 | 0.97 | 1,971 | 65.73 / 65.50 |
| WeAgent-RL (30B-A3B) | WeHarness | 11.57 | 5,061 | 2.22 | 1,696 | 1.06 | 2,264 | 64.31 / 62.52 |
(i) Harness-level comparison. Compared with GenSearcher and GenEvolve, WeHarness adds verification tools (2.22 calls per trajectory) to check the identity and appearance of retrieved images, and integration tools (1.06 calls) to organize verified evidence into a rendered carrier. These additional calls mainly improve knowledge accuracy: with the same GPT-Image-2 backend, WeAgent-RL improves TKA/VKA over the best open-source system by +12.63/+4.86 on generation and +25.87/+19.23 on editing (+7.60/+18.49 W-Avg), and the ablation in Table 3 attributes +3.16/+2.89 W-Avg to verification and +2.83/+2.50 to integration.
(ii) Policy-level comparison. Within the same WeHarness, WeAgent-RL retrieves more accurately, verifies more actively (2.22 vs. 0.65 verification calls), and integrates evidence better than its base model Qwen3-VL, improving TKR/VKR/MEI by +25.30/+24.74/+21.01 on generation and +30.30/+23.72/+20.02 on editing (Table 2), and W-Avg by +12.45/+16.72. With comparable costs (11.57 turns and 9.0K tokens vs. 9.0–10.6 turns and 7.2–7.6K tokens), WeAgent-RL surpasses the similarly sized Qwen-3.5/3.8 and reaches ≈98%/95% of Kimi-K2.6. We will add this cost analysis, including wall-clock latency, to the revision.
W2. Agreement between automatic and human evaluation. We randomly sampled 50 cases (25 generation and 25 editing) and collected outputs from six representative systems built on two image backends (Qwen-Image and GPT-Image-2). 20 annotators independently scored each output on the same dimensions and 0–10 checklists used by GPT-5.5, without seeing system identities or judge scores.
Table R2. Human scores (mean over annotators): Qwen-Image-based systems.
| Model | Policy Model | Gen/Edit Model | Harness | Gen IA | Gen TKA | Gen VKA | Gen VQ | Gen Avg | Edit IA | Edit TKA | Edit VKA | Edit VQ | Edit Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen-Image (Gen/Edit) | N/A | Qwen-Image (Gen/Edit) | N/A | 3.1 | 1.4 | 2.2 | 4.6 | 2.8 | 3.4 | 1.9 | 2.1 | 4.5 | 3.0 |
| GenSearcher | GenSearcher-8B | Qwen-Image (Gen/Edit) | GenSearcher | 3.5 | 2.6 | 3.1 | 5.2 | 3.6 | 3.3 | 2.1 | 2.2 | 4.6 | 3.1 |
| WeAgent-RL + WeEdit-M-RL (Ours) | WeAgent-RL (30B-A3B) | WeEdit-M-RL | WeHarness | 5.3 | 4.2 | 4.5 | 6.1 | 5.0 | 5.1 | 3.9 | 4.5 | 5.9 | 4.9 |
Table R3. Human scores (mean over annotators): GPT-Image-2-based systems.
| Model | Policy Model | Gen/Edit Model | Harness | Gen IA | Gen TKA | Gen VKA | Gen VQ | Gen Avg | Edit IA | Edit TKA | Edit VKA | Edit VQ | Edit Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-Image-2 | N/A | GPT-Image-2 | N/A | 6.1 | 3.5 | 4.1 | 8.2 | 5.5 | 5.2 | 2.5 | 4.9 | 8.3 | 5.2 |
| Qwen3-VL + GPT-Image-2 | Qwen3-VL (30B-A3B) | GPT-Image-2 | WeHarness | 6.3 | 3.7 | 4.4 | 8.3 | 5.7 | 5.1 | 2.7 | 5.1 | 8.2 | 5.3 |
| WeAgent-RL + GPT-Image-2 (Ours) | WeAgent-RL (30B-A3B) | GPT-Image-2 | WeHarness | 7.5 | 6.3 | 5.8 | 8.6 | 7.1 | 6.3 | 4.9 | 6.1 | 8.4 | 6.4 |
We then compare the system rankings induced by these human scores with those induced by GPT-5.5 on the full benchmark (Table 1). The relative rankings of the six systems under GPT-5.5 and human evaluation are highly consistent on both generation and editing, including the textual knowledge accuracy and visual knowledge accuracy dimensions, showing that the automatic evaluation is well aligned with human judgments.
Q1. AI-generated figures. Thank you for the suggestion. We will refine all figures, redraw Figures 2 and 3 as high-resolution and vector graphics.
Response to author
Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. The clarifications and new results are helpful and have largely addressed my questions.
I will maintain my original positive assessment of the paper. Best wishes to the authors!
Response to Reviewer Gpc8 (part1)
We thank the reviewer for recognizing our effective two-sided training recipe, the comprehensive comparisons and ablations, the generalization to KnowGen and Mind-Bench, and the data and benchmark contributions. We address the concerns below.
W1-1. Distinction from JarvisHub and Qwen-Image-Agent. We thank the reviewer for pointing out these concurrent works and agree that they share some design ideas with WeAgent-Harness, such as tool-based search and artifact management. However, there remain some significant differences:
We will add this discussion to the related work in the revision.