acceptodds
Under review as a conference paper at ICLR 2027

WeAgent-MMGenEdit: A Full-Stack Recipe for Multimodal Agentic Image Generation and Editing

Abstract

Image generation and editing models have advanced rapidly, yet remain unreliable when prompts require external world knowledge. Bounded and long-tail parametric knowledge prevents direct or reason-then-generate approaches from recovering the required facts and visual appearances. Existing agentic generation and editing methods mitigate this limitation with retrieval tools, yet remain constrained by insufficient visual verification, overloaded policy models, and weak integration of retrieved textual and visual evidence. To address these limitations, we present WeAgent-MMGenEdit, a full-stack recipe including a multimodal harness, a scalable data construction pipeline, a comprehensive benchmark, and post-training methods for the agent policy and image backend. We first introduce WeAgent-Harness, a multimodal runtime with persistent evidence management and dedicated verification and integration tools that organize retrieved multimodal evidence into a dense carrier. Upon this, we develop a scalable pipeline for prompt synthesis and agentic trajectory collection, yielding 23K supervised trajectories and 14.7K RL tasks with three-layer verifiable checklists. We further introduce WeBench-MMGenEdit, a bilingual benchmark covering both knowledge-intensive image generation and multi-image editing. Finally, a two-sided post-training recipe based on SFT and RL improves the agent policy and image backend. Together, WeAgent-MMGenEdit enables a 30B-total/3B-active policy to outperform similarly sized policy models and approach the performance of a 1T-parameter agent.

Primary area:
generative models
Keywords:
Agentic RL, Image Generation, Image Editing

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Gpc8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces WeAgent-MMGenEdit, a framework that retrieves, verifies, and organizes external evidence for image generation and editing. It trains both the agent and image model with SFT and RL, and provides training data and a bilingual benchmark. Component ablations and results on two additional public benchmarks support the effectiveness of the approach.

Critical strengths
  1. Effective training supported by useful experiments. The paper develops a practical training recipe for both the agent policy and the image backend. Controlled comparisons and component ablations support their respective contributions. Results on KnowGen and Mind-Bench provide additional evidence that the improvements extend beyond the proposed benchmark.

  2. Substantial data and evaluation contribution. The paper provides a structured pipeline for constructing multimodal tasks and collecting training trajectories, together with a bilingual benchmark covering generation and editing. Evaluation of the agent process, generation input, and final image offers a useful basis for diagnosing failures.

Critical weaknesses
  1. The distinctive harness contribution and practical cost need clearer comparison. Several design ideas overlap with existing frameworks such as JarvisHub and Qwen-Image-Agent. While the ablations demonstrate the usefulness of individual components, the specific advance over these frameworks could be better established. The paper also lacks a quantitative comparison of inference costs with recent agentic generation methods, including the additional calls for visual verification.

  2. Automatic evaluation needs further validation. The supplement clarifies benchmark construction and scoring, but does not report agreement between the automatic evaluator and human judgments of generated outputs. A human audit would strengthen confidence in the reported rankings, particularly for visual knowledge accuracy and editing quality.

Questions for authors

See weaknesses. It seems that some figures are ai-generated, can you replace them with better human-drafted figures?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Gpc8 (part1)

We thank the reviewer for recognizing our effective two-sided training recipe, the comprehensive comparisons and ablations, the generalization to KnowGen and Mind-Bench, and the data and benchmark contributions. We address the concerns below.

W1-1. Distinction from JarvisHub and Qwen-Image-Agent. We thank the reviewer for pointing out these concurrent works and agree that they share some design ideas with WeAgent-Harness, such as tool-based search and artifact management. However, there remain some significant differences:

  • JarvisHub is a canvas-native harness for long-horizon creative production across images, videos, audio, UI, and slides. It treats an editable canvas as the shared project state and emphasizes human-steerable orchestration, version management, and recovery. However, it does not target knowledge-intensive evidence verification, involves no policy or image-model training, and is demonstrated through qualitative case studies without a quantitative benchmark.
  • Qwen-Image-Agent is a training-free, generator-agnostic framework that bridges the context gap of T2I models through context-aware planning and grounding (reason, search, memory, and feedback), orchestrated by a strong MLLM. However, it does not post-train either the policy model or the image model, and relies mainly on post-generation feedback rather than verifying retrieved visual evidence before generation.
  • WeAgent-MMGenEdit targets knowledge-intensive generation and multi-image editing with three specific advances:
    1. WeAgent-Harness. Unlike approaches that mainly rely on planning or post-generation feedback, WeAgent-Harness verifies and structures evidence before generation. Retrieved images are explicitly inspected by a separately deployed VLM (Vision Analyze) before entering the generation input, reducing the verification burden from the policy's context. The verified evidence is then integrated into an HTML-rendered carrier that explicitly encodes entity–attribute–image–layout relations.
    2. Two-sided post-training. Rather than orchestrating fixed models, we apply agentic SFT and checklist-grounded RL to lift a 30B-A3B policy to ≈98%/95% of Kimi-K2.6 (1T-A32B) on generation/editing, while multi-reference SFT and Diffusion-NFT RL adapt the image model to the heterogeneous references produced by the agent.
    3. WeDataset-MMGenEdit and WeBench-MMGenEdit. The dataset provides ~23K SFT trajectories and 14.7K RL tasks with verifiable three-level checklists; the benchmark contains 300 human-audited bilingual cases, with hidden visual references in 84% of cases, multi-image inputs in 69.3% of editing cases, and three-level evaluation of the agentic process, generation input, and final image, enabling quantitative, process-level diagnosis.

We will add this discussion to the related work in the revision.

AuthorsOpenReview ↗

Response to Reviewer Gpc8 (part2)

W1-2. Inference cost of Policy Model. Thanks for the valuable comments! We report the average per-case cost on WeBench-MMGenEdit below.

Table R1. Inference cost per trajectory. Turns, calls, and tokens are averaged over all 300 cases of WeBench-MMGenEdit. Verification tokens are produced by the separately deployed VLM. W-Avg is with GPT-Image-2 (Gen/Edit).

MethodHarnessTurnsPolicy Output TokensVerify CallsVerify TokensIntegrate CallsIntegrate TokensW-Avg (Gen/Edit)
GenEvolve-8BGenEvolve5.412,090––––51.21 / 41.61
GenSearcher-8BGenSearcher4.364,281––––53.33 / 44.03
Qwen3-VL (30B-A3B)WeHarness8.742,5180.651,0670.801,51251.86 / 45.80
Qwen-3.5 (35B-A3B)WeHarness10.632,9032.163,0241.051,68562.75 / 55.46
Qwen-3.8 (27B)WeHarness10.323,0421.922,5260.581,66862.81 / 60.79
Kimi-K2.6 (1T-A32B)WeHarness9.004,5371.781,1380.971,97165.73 / 65.50
WeAgent-RL (30B-A3B)WeHarness11.575,0612.221,6961.062,26464.31 / 62.52

(i) Harness-level comparison. Compared with GenSearcher and GenEvolve, WeHarness adds verification tools (2.22 calls per trajectory) to check the identity and appearance of retrieved images, and integration tools (1.06 calls) to organize verified evidence into a rendered carrier. These additional calls mainly improve knowledge accuracy: with the same GPT-Image-2 backend, WeAgent-RL improves TKA/VKA over the best open-source system by +12.63/+4.86 on generation and +25.87/+19.23 on editing (+7.60/+18.49 W-Avg), and the ablation in Table 3 attributes +3.16/+2.89 W-Avg to verification and +2.83/+2.50 to integration.

(ii) Policy-level comparison. Within the same WeHarness, WeAgent-RL retrieves more accurately, verifies more actively (2.22 vs. 0.65 verification calls), and integrates evidence better than its base model Qwen3-VL, improving TKR/VKR/MEI by +25.30/+24.74/+21.01 on generation and +30.30/+23.72/+20.02 on editing (Table 2), and W-Avg by +12.45/+16.72. With comparable costs (11.57 turns and 9.0K tokens vs. 9.0–10.6 turns and 7.2–7.6K tokens), WeAgent-RL surpasses the similarly sized Qwen-3.5/3.8 and reaches ≈98%/95% of Kimi-K2.6. We will add this cost analysis, including wall-clock latency, to the revision.

W2. Agreement between automatic and human evaluation. We randomly sampled 50 cases (25 generation and 25 editing) and collected outputs from six representative systems built on two image backends (Qwen-Image and GPT-Image-2). 20 annotators independently scored each output on the same dimensions and 0–10 checklists used by GPT-5.5, without seeing system identities or judge scores.

Table R2. Human scores (mean over annotators): Qwen-Image-based systems.

ModelPolicy ModelGen/Edit ModelHarnessGen IAGen TKAGen VKAGen VQGen AvgEdit IAEdit TKAEdit VKAEdit VQEdit Avg
Qwen-Image (Gen/Edit)N/AQwen-Image (Gen/Edit)N/A3.11.42.24.62.83.41.92.14.53.0
GenSearcherGenSearcher-8BQwen-Image (Gen/Edit)GenSearcher3.52.63.15.23.63.32.12.24.63.1
WeAgent-RL + WeEdit-M-RL (Ours)WeAgent-RL (30B-A3B)WeEdit-M-RLWeHarness5.34.24.56.15.05.13.94.55.94.9

Table R3. Human scores (mean over annotators): GPT-Image-2-based systems.

ModelPolicy ModelGen/Edit ModelHarnessGen IAGen TKAGen VKAGen VQGen AvgEdit IAEdit TKAEdit VKAEdit VQEdit Avg
GPT-Image-2N/AGPT-Image-2N/A6.13.54.18.25.55.22.54.98.35.2
Qwen3-VL + GPT-Image-2Qwen3-VL (30B-A3B)GPT-Image-2WeHarness6.33.74.48.35.75.12.75.18.25.3
WeAgent-RL + GPT-Image-2 (Ours)WeAgent-RL (30B-A3B)GPT-Image-2WeHarness7.56.35.88.67.16.34.96.18.46.4

We then compare the system rankings induced by these human scores with those induced by GPT-5.5 on the full benchmark (Table 1). The relative rankings of the six systems under GPT-5.5 and human evaluation are highly consistent on both generation and editing, including the textual knowledge accuracy and visual knowledge accuracy dimensions, showing that the automatic evaluation is well aligned with human judgments.

Q1. AI-generated figures. Thank you for the suggestion. We will refine all figures, redraw Figures 2 and 3 as high-resolution and vector graphics.

Reviewer_Gpc8OpenReview ↗

Response to author

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. The clarifications and new results are helpful and have largely addressed my questions.

I will maintain my original positive assessment of the paper. Best wishes to the authors!

Reviewer_VKZ1OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents WeAgent-MMGenEdit, a framework for knowledge-intensive image generation and editing comprising four core components: WeAgent-Harness (a multimodal agent runtime for retrieving and verifying external evidence), WeDataset-MMGenEdit (a pipeline producing SFT trajectories and RL tasks with verifiable checklists), WeBench-MMGenEdit (a bilingual evaluation benchmark), and a two-sided post-training paradigm applying SFT and RL to both the agent policy and image generator. Paired with GPT-Image-2, WeAgent-MMGenEdit boosts the baseline Qwen3-VL-30B-A3B policy's weighted average score by 12.45 points in generation and 16.72 points in editing, approaching the performance of Kimi-K2.6 (1T-A32B).

Critical strengths

Well-Motivated Problem: Existing agentic benchmarks rarely address knowledge-intensive multi-image editing. WeBench-MMGenEdit fills this gap, with 69.3% of editing cases involving multiple input images and 84% requiring hidden visual references.

Solid Engineering Contribution: The framework introduces end-to-end training pipelines and RL mechanisms for both the planner and image generator, validated through comprehensive evaluation across diverse baseline models.

Critical weaknesses

Potential unfair backend baselines

The reported performance gains appear confounded by backend post-training/fine-tuning. Original image backends (e.g., Qwen-Image) inherently lack strong support for multi-image editing, meaning the observed improvements may largely stem from backend adaptation rather than the proposed agent framework itself. Notably, under a strictly controlled setting using the identical Qwen-Image backend, WeAgent-RL shows only a marginal gain over GenEvolve-8B (33.54 vs. 32.63 W Avg in generation). The authors need to decouple policy-level contributions from backend tuning effects, evaluate open-source baselines under a unified backend (e.g., GPT-Image-2) for fair comparison, and clarify how each baseline handles multi-image inputs.

Limited Generalizability of WeAgent-Harness Across Weaker Policies

The effectiveness of WeAgent-Harness seems heavily dependent on the reasoning capacity of the underlying policy model. Empirical results show that pairing the harness with weaker policies (e.g., Qwen3-VL-30B-A3B) leads to degraded editing performance, even falling behind direct prompting with frontier models like GPT-Image-2 and Gemini-3.1-Flash-Lite. This suggests that the harness introduces reasoning or coordination overhead that weaker backbones cannot handle. The authors should formally discuss this operational boundary and analyze the minimum base-model capability required for the harness to yield positive gains.

Questions for authors
  • What exact models are used for the expert policy (Sec. 4.1) and the RL reward judge (Sec. 5.2)? Is the judge from the same family as the evaluation model (GPT-5.5)?

  • In Table 1, does Baseline 3 use Qwen-Image or Qwen-Image-Edit for generation? How much of WeEdit-M’s gain comes solely from this backend discrepancy?

  • How were the multi-objective reward weights in Section 6.2 selected (heuristic, grid search, or val set)? How sensitive is performance to these coefficients?

  • Figures 2 & 3 suffer from severe legibility issues (tiny fonts/compression). Also, may contain some AI generated content. Moreover, Figure 3 has discrepancies with Section 3.2 (e.g., mismatched tool names, undefined modules). Please redraw these figures in high-res vector format and unify all terminology.

3 comments
AuthorsOpenReview ↗

Response to Reviewer VKZ1 (part1)

We thank the reviewer for recognizing the well-motivated problem of knowledge-intensive multi-image editing and the end-to-end training pipelines for both the planner and the image generator. We address the concerns and questions below.

W1. Decoupling policy contributions from generation backend. (i) Same Qwen-Image backend. On editing, WeAgent-RL surpasses GenEvolve-8B by +13.57 W-Avg (38.50 vs. 24.93; TKA +12.36, VKA +19.74). On the backend-independent Agentic Chain Score and Prompt Score (Table 2), WeAgent-RL surpasses GenEvolve-8B by +13.00/+26.08 and +9.29/+27.20 on generation/editing. Therefore, we attribute the +0.91 gain over GenEvolve in generation (33.54 vs. 32.63) to the generation model ceiling. (ii) Same GPT-Image-2 backend. We supplement the results that all open-source systems with GPT-Image-2 generation backend, keeping their released policies and harnesses, as shown in table below:

Table R1. Generation results on WeBench-MMGenEdit with a unified GPT-Image-2 backend.

MethodPolicy ModelHarnessIATKAVKAVQW-Avg
GenSearcher + GPT-Image-2GenSearcher-8BGenSearcher64.8340.7148.6785.6453.33
GenEvolve + GPT-Image-2GenEvolve-8BGenEvolve61.4238.9448.9183.6551.21
Mind Brush + GPT-Image-2GPT-5.1Mind Brush67.1647.0152.0186.1556.71
Qwen3-VL + GPT-Image-2 (Baseline 1)Qwen3-VL (30B-A3B)WeHarness63.9238.8245.6785.5251.86
WeAgent-RL + GPT-Image-2 (Ours)WeAgent-RL (30B-A3B)WeHarness73.6759.6456.8787.9364.31
Δ vs. best open-source––+6.51+12.63+4.86+1.78+7.60

Table R2. Editing results on WeBench-MMGenEdit with a unified GPT-Image-2 backend.

MethodPolicy ModelHarnessIATKAVKACPVQW-Avg
GenSearcher + GPT-Image-2GenSearcher-8BGenSearcher51.9722.3843.7373.5982.1144.03
GenEvolve + GPT-Image-2GenEvolve-8BGenEvolve47.1220.3536.8769.6280.3441.61
Mind Brush + GPT-Image-2GPT-5.1Mind Brush44.8018.8936.7164.0278.9039.89
Qwen3-VL + GPT-Image-2 (Baseline 1)Qwen3-VL (30B-A3B)WeHarness50.7424.5348.0174.2080.9445.80
WeAgent-RL + GPT-Image-2 (Ours)WeAgent-RL (30B-A3B)WeHarness64.4748.2562.9679.7882.8762.52
Δ vs. best open-source––+12.50+25.87+19.23+6.19+0.76+18.49

WeAgent-RL leads the best open-source system by +7.60/+18.49 W-Avg, mainly on TKA/VKA, and its gap over GenEvolve-8B widens from +0.91 to +13.10 once the backend bottleneck is removed. These gains stem from our core contributions: WeAgent-Harness for explicit multimodal evidence retrieval, verification, and integration, together with agentic SFT and checklist-grounded RL.

AuthorsOpenReview ↗

Response to Reviewer VKZ1 (part2)

W2. Effectiveness of WeAgent-Harness under different policies. We agree this boundary should be discussed. Any agentic harness requires basic planning and tool-calling ability from the policy, which is why GenSearcher and GenEvolve also post-train their base models. Table R3 summarizes results from Table 1:

Table R3. W-Avg of different policies in WeHarness with GPT-Image-2, compared with direct GPT-Image-2 (48.79/48.99).

Policy in WeHarnessGen W-Avg (Δ vs. direct)Edit W-Avg (Δ vs. direct)
Qwen-3.5 (35B-A3B)62.75 (+13.96)55.46 (+6.47)
Qwen-3.6 (35B-A3B)62.91 (+14.12)56.58 (+7.59)
Qwen-3.8 (27B)62.81 (+14.02)60.79 (+11.80)
WeAgent-RL (30B-A3B)64.31 (+15.52)62.52 (+13.53)
Kimi-K2.6 (1T-A32B)65.73 (+16.94)65.50 (+16.51)

The harness improves generation for all policies and editing for most of them. As shown in Table 2, the untrained Qwen3-VL obtains only 35.48/28.03 Agentic Chain Avg and 32.17/21.80 Prompt Score Avg on generation/editing, i.e., it fails to accurately acquire and deliver the required evidence. In contrast, similarly sized Qwen-series models and Kimi-K2.6 have stronger agentic ability and therefore obtain clear gains on both generation and editing within WeHarness.

Closing this gap is exactly the goal of our agentic post-training. SFT and RL raise these scores to 59.17/52.71 and 55.54/50.85 (+23.68/+24.68 and +23.36/+29.05), improving the W-Avg of the base model by +12.45/+16.72 on generation/editing and reaching ≈98%/95% of Kimi-K2.6 (1T-A32B) with about 3% of its parameters. We will add this boundary analysis to the revision.

Q1. Expert policy and RL reward judge. Both the expert policy for trajectory collection (Sec. 4.1) and the RL reward judge (Sec. 5.2) use Kimi-K2.6. It is from a different model family than the evaluation judge (GPT-5.5).

Q2. Details of the Qwen Backend used by Baseline 3. Qwen-Image (Gen/Edit) routes requests by input type: requests without reference images go to Qwen-Image, and requests with reference images go to Qwen-Image-Edit-2509, because Qwen-Image-Edit does not support generation without reference images. WeEdit-M is post-trained from Qwen-Image-Edit-2509 and uses the same routing rule with Baseline 3. The gain of WeEdit-M is isolated in Table 1 and Table 3 by fixing the policy (WeAgent-RL) and changing only the backend: Qwen-Image (Gen/Edit) → WeEdit-M-SFT → WeEdit-M-RL improves W-Avg from 33.54/38.50 to 38.65/46.24 to 47.39/49.98 on generation/editing. This +13.85/+11.48 gain comes from image-side post-training.

Q3. Reward weights in Sec. 6.2. The weights were set heuristically based on the main failure modes of multi-reference editing. Instruction accuracy (0.25) and text accuracy (0.35) receive the largest weights because instruction violations and unreadable or incorrect text are the most frequent failures in knowledge-intensive editing; in practice, we found that keeping their combined weight within 50%–60% (0.60 here) works well. Preservation, aesthetics, and relative quality (0.10/0.15/0.15) act as regularizers against degraded preservation and visual quality. Decomposing the reward into separate dimensions also prevents a single holistic score from masking severe failures.

Q4. Figures 2 and 3. Thank you for pointing this out. We will redraw Figures 2 and 3 as high-resolution vector graphics with larger fonts, and align the terminology of Figure 3 with Sec. 3.2.

Reviewer_VKZ1OpenReview ↗

Response to Authors' Rebuttal

Thank authors for the detailed rebuttal and the updated experiments. My concern on the unfair backend baselines is fully addressed. I think the results with GPT-Image-2 backend are convincing since this is already the best backend we can possibly have. Please update the results in the finalized version. For the policies, I accept authors' opinion that the tool itself is also important. I think the post-trained backend will also be useful for the community. I really appreciate authors' feedback. Therefore I raise my score to 4.

Reviewer_VuZLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This is a solid work on knowledge-intensive image generation and editing. The proposed WeAgent-Harness combines seven tool interfaces, a persistent multimodal evidence store, a separately deployed VLM for visual verification, and HTML-rendered evidence carriers to support grounded generation and multi-image editing. The authors also construct WeDataset-MMGenEdit, comprising approximately 23K SFT trajectories and 14.7K RL tasks collected and filtered within the harness, and WeBench-MMGenEdit, a benchmark of 300 human-audited English and Chinese tasks.

Critical strengths

1.structured evidence carriers address errors in visual evidence selection and cross-modal association, offering a practical approach to knowledge-intensive generation and editing. 2. The three-level evaluation distinguishes failures in the agentic process, generation input, and final image, providing useful diagnostic insights beyond final-output scoring. 3. The extensive comparisons, component ablations, and experiments across multiple image backends provide good empirical support for the proposed framework and post-training recipe.

Critical weaknesses
  1. How does the agent resolve conflicting information across retrieved sources or inconsistencies between images and textual descriptions? The mechanism for preventing unreliable evidence from propagating into the final image is unclear.
  2. To what extent is final image quality constrained by the layout and readability of the HTML-rendered evidence carrier? A comparison against structured text with reference images would help establish the specific benefit of rendering the evidence into an image.
  3. The provided manuscript lacks the appendix repeatedly referenced in the main text. Including it in the revision is necessary to assess methodological details and reproducibility.
  4. Agreement between GPT-5.5 scores and human expert judgments is not reported. Such validation would strengthen confidence in the automated evaluation.
  5. The benchmark scoring protocol needs further clarification, particularly checklist scoring, WAvg weights, and how hidden visual references are used during evaluation.
Questions for authors

Please see weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer VuZL (part1)

We thank the reviewer for recognizing our work as a solid contribution to knowledge-intensive image generation and editing, particularly its structured evidence carriers, three-level evaluation, and extensive empirical validation. We address the five concerns below.

W1. When the textual and the visual are in conflict. In WeAgent-Harness, retrieved content is stored with stable IDs and source information, rather than being forwarded directly to the image generator. The agent can cross-check textual sources via Web Search/Extract, verify image contents with Vision Analyze, and use Search by Image to inspect identity and source. It then integrates the better-supported evidence into the final generation input. This reduces the risk of inconsistent information.

W2. Benefit of HTML-rendered integration. The comparison with structured text and reference images is included in Table 3 (harness ablation, Row 2):

Generation inputGeneration W-AvgEditing W-Avg
Structured text + reference images (w/o HTML integration)49.0343.30
HTML-rendered carrier (Ours)51.8645.80
Δ+2.83+2.50

The rendered carrier makes entity–attribute–layout associations explicit, whereas structured text leaves more of these relations for the image model to infer.

W3. Appendix. We apologize for the confusion. The appendix is available in the Supplementary Material file (25 pages, Secs. 11.1–11.9), rather than being appended to the main PDF. It provides harness/tool specifications, dataset and benchmark construction, RL objectives, training configurations, experimental details, and example trajectories. In the revision, we will append it directly after the main paper.

W4. Alignment between GPT-5.5 evaluations and human evaluations. Thanks for the valuable comments. We randomly sampled 50 cases (25 generation and 25 editing) and collected outputs from six representative systems built on two image backends (Qwen-Image and GPT-Image-2), covering direct generation/editing, open-source agentic methods, harness-based baselines, and our models. 20 annotators independently scored each output on the same dimensions and 0–10 checklists used by GPT-5.5, without seeing system identities or judge scores.

Table A1. Human scores (mean over annotators) on the 50-case subset: Qwen-Image-based systems.

ModelPolicy ModelGen/Edit ModelHarnessGen IAGen TKAGen VKAGen VQGen AvgEdit IAEdit TKAEdit VKAEdit VQEdit Avg
Qwen-Image (Gen/Edit)N/AQwen-Image (Gen/Edit)N/A3.11.42.24.62.83.41.92.14.53.0
GenSearcherGenSearcher-8BQwen-Image (Gen/Edit)GenSearcher3.52.63.15.23.63.32.12.24.63.1
WeAgent-RL + WeEdit-M-RL (Ours)WeAgent-RL (30B-A3B)WeEdit-M-RLWeHarness5.34.24.56.15.05.13.94.55.94.9

Table A2. Human scores (mean over annotators) on the 50-case subset: GPT-Image-2-based systems.

ModelPolicy ModelGen/Edit ModelHarnessGen IAGen TKAGen VKAGen VQGen AvgEdit IAEdit TKAEdit VKAEdit VQEdit Avg
GPT-Image-2N/AGPT-Image-2N/A6.13.54.18.25.55.22.54.98.35.2
Qwen3-VL + GPT-Image-2Qwen3-VL (30B-A3B)GPT-Image-2WeHarness6.33.74.48.35.75.12.75.18.25.3
WeAgent-RL + GPT-Image-2 (Ours)WeAgent-RL (30B-A3B)GPT-Image-2WeHarness7.56.35.88.67.16.34.96.18.46.4

We then compare the system rankings induced by these human scores with those induced by GPT-5.5 on the full benchmark (Table 1). The relative rankings of the six systems under GPT-5.5 and human evaluation are highly consistent on both generation and editing, showing that the automatic evaluation is well aligned with human judgments.

AuthorsOpenReview ↗

Response to Reviewer VuZL (part2)

W5. Benchmark scoring and hidden references. As described in the Supplementary Material (Sec. 11.3), we use task-specific checklists scored on a 0–10 scale by separate judges at three levels: (i) Agentic Chain (TKR/VKR/MEI), (ii) Generation Input (TKS/VKS), and (iii) Final Image (IA/TKA/VKA/VQ, plus CP for editing). Each checklist item is scored on a 0–10 scale, and item scores are averaged within each dimension. The first two levels then average their dimensions. For final images, the weights are:

  • Generation: W-Avg = 0.20 IA + 0.35 TKA + 0.35 VKA + 0.10 VQ.
  • Editing: W-Avg = 0.20 IA + 0.30 TKA + 0.30 VKA + 0.10 CP + 0.10 VQ.

These weights emphasize knowledge accuracy. Hidden visual references are available in 252/300 cases (84%) only to the evaluator—not to the tested agent or image generator—to assess whether target appearances are faithfully reproduced.

Reviewer_VuZLOpenReview ↗

Response to Authors

I thank the authors for their thoughtful responses and detailed explanations. I have no further questions, and I would encourage the authors to include the additional experiments presented in the rebuttal in the final version of the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.