acceptodds
Under review as a conference paper at ICLR 2027

Can Image Agents Complete Visual Commissions? A Benchmark and Modular Framework

Abstract

Visual commissions require image agents to integrate heterogeneous source materials, preserve requirements across dependent stages, and produce coordinated deliverables. Existing evaluations examine specific capabilities and execution settings, leaving end-to-end commission completion less understood. We introduce Visual Commission Bench (VCB), comprising 600 tasks across 12 categories and three complementary tiers, from capability-focused tasks to complete commissions. VCB uses 5,039 requirement checks and separate visual quality ratings to assess content correctness, visual realization, constraint preservation, and delivery completeness without prescribing an internal workflow. We present V-Protocol, a reference agent that dynamically composes reusable task, cognitive, and expression modules at inference time. Task modules organize delivery, cognitive modules derive content, and expression modules guide visual realization. The modules exchange structured intermediate artifacts, while a shared execution harness tracks artifact revisions and dependencies, manages tool execution, and checks completion. With the same controller and image generator as the agent baselines, V-Protocol achieves 69.0% strict task success on VCB, compared with 42.0% for SCOPE. On Mind-Bench, strict accuracy increases from 39.2% to 45.4%. The gallery for V-protocol could be found in Figure1.

Primary area:
generative models
Keywords:
Image Generation Agents, Multimodal Agents, Capability Composition, Agentic AI, Tool-Augmented Generation, Modular Agents, Inference-Time Adaptation, Self-Extensible Systems

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gGvjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether image agents can complete visual commissions that require integrating heterogeneous source materials, maintaining constraints across dependent stages, and coordinating multiple deliverables. It introduces Visual Commission Bench (VCB), comprising 600 tasks across 12 categories and 5,039 requirement checks, together with V-Protocol. VCB evaluates final deliverables through requirement satisfaction, strict task success, and visual quality. V-Protocol combines task, cognitive, and expression modules with a dynamic operation graph and a shared execution harness to manage tool execution, artifact revisions, dependencies, and online verification. Experiments compare task completion across systems on VCB, WISE-Verified, and Mind-Bench. Component analyses and execution-trace observations further examine module selection and runtime enforcement.

Critical strengths
  1. The problem formulation addresses a practical need. The paper extends evaluation beyond individual image generation to complete visual commissions, emphasizing information extraction from multiple source files, dependent operations, preservation of evolving constraints, and coordinated delivery. This setting more closely reflects practical workflows than evaluating prompt adherence or the quality of a single image alone.

  2. The benchmark has substantial scope. VCB includes 600 tasks, 12 categories, 45 subcategories, 4,288 source files, and 5,039 requirement checks. Evaluating requirement satisfaction separately from visual quality helps identify cases in which images appear satisfactory but the overall commission remains incomplete. This benchmark contribution is potentially useful independently of the proposed reference agent.

  3. Explicit execution-state management has practical value. Supplementary Figure 5 illustrates the relationships among task specification, module composition, operation-graph execution, verification feedback, and delivery, while distinguishing stale artifact dependencies from invalidated verification reports. Table 2 reports high task completion rates, and Appendix D provides descriptive analyses of execution traces and controller usage that offer concrete observations about reliability in multistage execution.

Critical weaknesses

Major Weaknesses

  1. The methodological novelty of V-Protocol and its module-selection claims are not sufficiently established. The framework combines three module families, capability matching, operation graphs, artifact revision management, and online verification for visual commissions, but its key technical additions over existing skill orchestration and workflow execution systems remain unclear. In particular, the Introduction claims that module combinations are selected dynamically using effect profiles estimated on separate development tasks. However, Section 4.2 and supplementary Figure 5 describe capability matching, configured priorities, capability coverage, and model-based selection, with module-effect estimates used only for offline diagnostics. The module-budget experiments in Appendix C do not establish the benefit of using effect profiles for selection either. The technical claims should be aligned with the implementation, the mechanisms distinguishing V-Protocol from the closest systems should be specified, and controlled comparisons should separate the contributions of modular guidance, candidate filtering, and effect-based selection.

  2. Although the overall workflow is illustrated, the key decision mechanisms remain insufficiently specified for reproduction. Supplementary Figure 5 clarifies task specification, module composition, guarded execution, verification feedback, and revision invalidation. However, Appendix A still primarily outlines the responsibilities of the model and the harness, without providing complete pseudocode, a module inventory with its actual guidance, executable input/output schemas, candidate-ranking rules, or key prompt templates. The available materials do not sufficiently specify how operation nodes are created and revised, module applicability is assessed, stale dependencies are resolved, or semantic verification feedback triggers editing, regeneration, or replanning. These implementation details should be provided, together with a complete execution example containing the actual task specification, operation graph, module assignments, artifact revisions, and verification results.

  3. The comparability of baseline access capabilities and execution conditions remains insufficiently documented. Appendix B.2 standardizes the controller and image backend but notes that shared backends do not imply equal call counts or execution time. Main-text Figure 4 and supplementary Figures 18 and 20 attribute GEMS failures to its inability to read files autonomously, although file access is a core prerequisite of VCB. These comparisons may therefore conflate system adaptation with methodological differences. The paper should report each method’s file and tool access, main-experiment budgets, and retry and recovery rules, and compare methods under shared capability interfaces and explicit budgets. Although Table 7 reports controller usage and session duration, it excludes earlier attempts, subsequent manual recovery, and some model calls; caching and service load are also not matched. These statistics do not support a rigorously controlled comparison of total cost or latency.

  4. The ablations are confounded by subset selection and execution history, preventing isolation of the components’ causal contributions. Appendix C states that the diagnostic subset was selected using historical Full performance rather than as an independent held-out sample. Full reuses historical outputs from earlier runtime versions and recovery runs, and exact equivalence of historical model weights was not verified despite the shared controller identifier. In contrast, the new variants use frozen implementations and receive one execution per task within limits of 256 steps, eight generation calls, and eight editing calls, without subsequent output-driven recovery runs. Differences between Full and the variants in main-text Table 4 and supplementary Table 5 may thus reflect subset selection, runtime versions, model versions, and recovery opportunities. Although the authors appropriately describe these results as exploratory diagnostics, they do not establish the independent effects of the core components. All configurations should be rerun on a prespecified task set selected independently of Full performance, using the same frozen model and runtime, budgets, and recovery rules, with uncertainty reported across repeated runs.

  5. The absence of an independent human study of final deliverables leaves evaluation validity insufficiently established. Human involvement in Section 3.2 concerns reviewing requirement checklists and does not replace independent assessment of generated outputs. Section 3.3 uses the same Qwen3.8-Flash-Next model to evaluate both requirement satisfaction and visual quality, and the supplementary material reports neither corresponding agreement with human judgments nor user evaluation. VCB emphasizes complete visual commissions, for which readability, layout, consistency across deliverables, and practical usability have subjective aspects. It is therefore unclear whether higher ReqPass, TaskPass, and VQ scores correspond to task completion that human evaluators would accept. A human study should use stratified task sampling and conceal system identities. Multiple independent raters should assess requirement satisfaction against the requests and source materials, as well as visual quality, consistency across outputs, and overall readiness for delivery. Designers or intended users could be included for C-tier tasks. Inter-rater agreement, agreement between automated and human assessments, and paired comparisons with the strongest baseline should be reported, together with confidence intervals.

Minor Weaknesses

  1. The tier-level analysis would benefit from more detailed task-structure statistics. S, M, and C differ in both task categories and execution scope, and the authors already clarify that they do not represent intrinsic difficulty levels. Reporting the distributions of source-file counts, applicable requirement checks, deliverable counts, and dependency depth for each tier would help interpret TaskPass differences. The decline across tiers alone cannot separate the effects of task type from those of dependency complexity.

  2. The Introduction contains redundant text and typographical errors. Two consecutive paragraphs on page 2 begin with “Existing evaluations have extended image generation...” and substantially overlap; the text also contains “require require.” The misspellings “conmmission” in Figure 2 and “achives” in the Figure 4 caption should be corrected. The repeated paragraphs should be consolidated and the manuscript carefully proofread.

  3. Citations and sources for published results should be made more explicit in the tables. Several benchmarks in Table 1 lack row-level citations, while the published GPT-Image-1.5, Nano Banana Pro, and Qwen-Image-Agent results in Table 3 lack itemized information on model versions, sources, and evaluation protocols. Complete references should be added to the tables or footnotes, with clarification of whether these results are directly comparable to the shared-backend experiments.

  4. The module-budget experiments introduce an additional intervention. Appendix C permits bypassing the expression-module prerequisite when the module budget leaves no room for an expression module, so the differences in Table 5 do not arise solely from module counts. The actual module distributions and diagnostic-subset size should be reported, and the effects of budget restrictions should be distinguished from those of relaxing the prerequisite.

Questions for authors
  1. Do module-effect profiles influence runtime selection in the evaluated implementation? If so, where do they enter candidate ranking or selection? If not, how should the corresponding claim in the Introduction be interpreted, and what mechanism distinguishes the implemented selector from capability matching followed by model-based selection?

  2. Are the complete module definitions, key prompt templates, input/output schemas, and operation-graph update procedures available? A representative execution trace showing module assignments, artifact revisions, and verification-triggered repairs would help establish whether the framework can be reproduced from the provided materials.

  3. What file-reading, retrieval, image-editing, and workspace capabilities were available to each baseline? In particular, was GEMS given a compatible way to access the source files? Please clarify the main-experiment budgets and retry and recovery policies for all methods.

  4. Are component comparisons available in which Full and all ablations use the same frozen controller weights, runtime, task set, budgets, and recovery policy? How many tasks are in the diagnostic subset, and what selection rule was applied using historical Full performance?

  5. Was any independent human evaluation of final deliverables conducted beyond checklist review? If available, please report the sampling procedure, evaluator qualifications, inter-rater agreement, agreement with automated judgments, and paired comparisons against the strongest baseline.

9 comments

We sincerely thank the reviewer for the highly recognization of VCB’s practical value and the benefits of V-Protocol’s execution-state management. We also appreciate the reviewer’s detailed comments, which have helped us refine the methodological description and clarify key implementation details. We have provided point-by-point responses to all comments and revised the relevant sections of the manuscript and supplementary material accordingly.

Major Weaknesses 1 and Q1: Novelty of V-Protocol

Major Weaknesses 1. The methodological novelty of V-Protocol and its module-selection claims are not sufficiently established. The framework combines three module families, capability matching, operation graphs, artifact revision management, and online verification for visual commissions, but its key technical additions over existing skill orchestration and workflow execution systems remain unclear. In particular, the Introduction claims that module combinations are selected dynamically using effect profiles estimated on separate development tasks. However, Section 4.2 and supplementary Figure 5 describe capability matching, configured priorities, capability coverage, and model-based selection, with module-effect estimates used only for offline diagnostics. The module-budget experiments in Appendix C do not establish the benefit of using effect profiles for selection either. The technical claims should be aligned with the implementation, the mechanisms distinguishing V-Protocol from the closest systems should be specified, and controlled comparisons should separate the contributions of modular guidance, candidate filtering, and effect-based selection.

Q1. Do module-effect profiles influence runtime selection in the evaluated implementation? If so, where do they enter candidate ranking or selection? If not, how should the corresponding claim in the Introduction be interpreted, and what mechanism distinguishes the implemented selector from capability matching followed by model-based selection?

A1: We thank the reviewer for highlighting the need to clarify V-Protocol's contributions and implementation details. We address this concern through the methodological contributions, runtime selection mechanism, and ablation studies below.

A1.1 Novelty Claims

V-Protocol is built around two complementary designs: adaptive module composition and runtime enforcement. These form the core of our methodological contributions and are implemented as follows.

Adaptive module composition. The system constructs a TaskSpec from the user request and publicly available source materials, and forms an operation graph using an applicable task template or model-based planning. Each operation specifies its goal, input and output artifacts, capability requirements, and permitted actions. Task, cognitive, and expression modules provide reusable guidance for task procedures, content reasoning, and visual expression, respectively. Multiple types of module guidance can be combined within the same operation, and modules can be reused across operations.

Runtime enforcement. Based on the current execution state, the harness checks input availability, dependency freshness, action permissions, and budgets, and registers generated artifacts and their revisions. Verification reports are bound to a specific image's identifier, revision, content digest, and verification criteria. If the image or relevant criteria change, an earlier report cannot directly serve as evidence that the current revision passes verification. Final delivery must also satisfy the specified file requirements and current completion conditions.

Implementing these designs involves more detailed mechanisms. For example, the modular guidance and candidate filtering identified by the reviewer are concrete mechanisms used to implement adaptive module composition.

We also compare V-Protocol with SCOPE, the most closely related system:

SystemA: Task planning and module organizationB: Execution control and completion conditions
SCOPEConditionally invokes retrieval, reasoning, and repair skills around a structured semantic specification within a predefined generation loop.Maintains state and validates inputs according to predefined stage rules, and terminates based on verification outcomes or the iteration budget.
V-ProtocolForms an operation graph according to task requirements, declares each node's inputs, outputs, and dependencies, and selects and combines module guidance for the nodes.The harness checks execution conditions against the current node and artifact states, and manages stage transitions, tool execution, and delivery completion.
Specific additions over SCOPEConstruction of the task operation graph and module configuration for its nodesConstrained execution of the task operation graph

A1.2 Module-Effect Profiles

Module-effect profiles do not influence runtime selection in the evaluated implementation. This mechanism was part of our initial design, but we subsequently removed it to avoid potentially favoring similar benchmark tasks and compromising the fairness of the comparison.

The implemented selection procedure is as follows:

  1. Retrieve candidate modules whose capabilities overlap with the requirements declared by the current operation node.
  2. Establish a deterministic candidate order using configured priority, the number of required capabilities covered, and module identifiers.
  3. The controller assesses applicability and selects suitable modules based on the current operation's goal, input artifacts, and the modules' applicability conditions.
  4. Load procedural guidance from the selected modules. Some support modules can also be preselected during method selection for the task and loaded at matching operation nodes.

Accordingly, we have removed the inconsistent description from the Introduction and revised the relevant text as follows:

We present V-Protocol as a reference agent for end-to-end visual commission completion. V-Protocol relies on two complementary designs: adaptive module composition and runtime enforcement. Adaptive module composition forms an operation graph and configures its nodes by ranking candidate modules, assessing their applicability, and selecting suitable guidance. Runtime enforcement executes the graph through a dedicated harness that enforces execution constraints, governs stage transitions, and manages tool execution.

This wording is consistent with Section 4.2, the supplementary framework figure, and the selection rules in the appendix.

A1.3 Ablations

The two original component ablations correspond to the two designs described above:

  • w/o Runtime Enforcement: Retains module selection and tool access while relaxing the specified execution gates.
  • w/o Adaptive Module Selection: Uses the same predefined module combination for all tasks while retaining the guidance text of those modules and task-planning capability.

To address the reviewer's request to distinguish modular guidance from candidate filtering, we add two fine-grained ablations:

  • w/o Module-specific Procedural Guidance: Replaces module-specific guidance text with interface descriptions while retaining metadata, interfaces, program templates, candidate recommendation, and runtime constraints.
  • w/o Capability-based Candidate Recommendation: Removes capability-overlap filtering and ranking by capability coverage from automatic candidate recommendation, while retaining module guidance, model-based selection, preselected support modules, and other loading paths.
Configurationw/ Module-specific Procedural Guidancew/ Capability-based Candidate RecommendationRP ↑TP ↑VQ ↑
w/o Runtime Enforcement——81.59 ± 0.5760.44 ± 0.793.760 ± 0.038
w/o Adaptive Module Selection×√88.55 ± 0.3164.44 ± 1.924.148 ± 0.081
w/o Adaptive Module Selection√×91.32 ± 0.4167.22 ± 2.554.068 ± 0.036
w/o Adaptive Module SelectionFixed×82.28 ± 0.1861.83 ± 0.933.865 ± 0.006
Full V-Protocol√√94.28 ± 0.1869.11 ± 0.674.202 ± 0.005

Fixed denotes the same predefined four-module combination for every task, retaining its guidance text. This is the fixed-module ablation described in Section 5.2 (Component analysis).

Full V-Protocol outperforms all the ablated configurations across RP, TP, and VQ, isolating and quantifying the performance gains attributable to the proposed modules.

Accordingly, Section 5.2 (Component analysis) describes the fine-grained ablation settings, and Table 4 reports the results.

Major Weaknesses 2 and Q2: Detail of V-Protocol

Major Weaknesses 2. Although the overall workflow is illustrated, the key decision mechanisms remain insufficiently specified for reproduction. Supplementary Figure 5 clarifies task specification, module composition, guarded execution, verification feedback, and revision invalidation. However, Appendix A still primarily outlines the responsibilities of the model and the harness, without providing complete pseudocode, a module inventory with its actual guidance, executable input/output schemas, candidate-ranking rules, or key prompt templates. The available materials do not sufficiently specify how operation nodes are created and revised, module applicability is assessed, stale dependencies are resolved, or semantic verification feedback triggers editing, regeneration, or replanning. These implementation details should be provided, together with a complete execution example containing the actual task specification, operation graph, module assignments, artifact revisions, and verification results.

Q2. Are the complete module definitions, key prompt templates, input/output schemas, and operation-graph update procedures available? A representative execution trace showing module assignments, artifact revisions, and verification-triggered repairs would help establish whether the framework can be reproduced from the provided materials.

A2: We thank the reviewer for the attention to reproducibility and the detailed suggestions for clarifying V-Protocol’s decision mechanisms and implementation details. We will make the code publicly available. In the revised supplementary material, we have expanded Appendix A as follows:

  • Appendix A.1 (Selection rule) specifies capability-based candidate retrieval, ranking by configured priority and capability coverage with module identifiers as the tie-breaker, and controller-assessed applicability.
  • Appendix A.2 (Schemas and operation-graph updates) presents the principal structured interfaces in Table 5 and pseudocode for the execution loop (Algorithm 1), module selection and loading (Algorithm 2), dependency freshness and verification-triggered repair (Algorithm 3), and operation-graph updates (Algorithm 4). The subsequent Controller and harness paragraph clarifies their respective roles.
  • Appendix A.3 (Actual Execution Trace) provides a recorded end-to-end C-tier workspace example, including the final TaskSpec, operation graph, node-level module assignments, image revisions, and verification results. Table 6 summarizes the execution steps and artifact versions, while Figure 6 shows the generated images, including the schedule before and after verification-triggered regeneration, together with the final delivered outputs.
Reviewer_K7qPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces Visual Commission Bench (VCB), a benchmark for evaluating whether image agents can complete realistic visual commissions rather than isolated generation requests. VCB contains 600 tasks covering capability-specific problems, multi-stage workflows, and full commissions. Tasks may involve documents, reference images, interaction histories, and multiple required deliverables. The authors also present V-Protocol, which combines reusable task, reasoning, and visual-expression modules with an execution harness for tracking artifacts and dependencies. With the same controller and image generator, V-Protocol reaches 69.0% TaskPass on VCB, compared with 42.0% for SCOPE. It also improves over the baselines on Mind-Bench.

Critical strengths

The strongest part of the paper is the benchmark. Existing image-agent benchmarks usually isolate generation, retrieval, or reasoning, whereas VCB tests whether an agent can preserve information and constraints across a longer workflow and deliver all requested artifacts. TaskPass is particularly useful because it exposes cases where most individual requirements are satisfied but the overall commission is still incomplete. The complete-system results are also strong. Under shared model backbones, the gap over SCOPE is large, and the Mind-Bench results suggest that the improvement is not limited to file management or output formatting.

Critical weaknesses

My main concern is the reliability of the evaluation. The checklists are largely produced with language models, and final requirement and quality judgments use Qwen3.8-Flash-Next. Humans review the checklists, but the paper does not report how many tasks were checked, how often humans disagreed with the evaluator, or whether the final system ranking survives human evaluation. This matters because the evaluated agent also uses a Qwen3.8-family controller. The component analysis is also difficult to interpret. The full system reuses historical outputs, including recovery runs, while the ablations use a later frozen runtime. Consequently, the reported differences cannot be attributed only to runtime enforcement or module selection.

Questions for authors

Questions for the Authors

  1. What is the agreement between the automated evaluator and human reviewers on a random subset of VCB? Does human evaluation preserve the system ranking?

  2. Can the full method and ablations be rerun with the same runtime, task subset, budgets, and fresh outputs?

  3. Which parts of VCB, including source files and evaluation annotations, will be released?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer K7qP

We really appreciate the reviewer for recognizing VCB’s focus on preserving information and constraints across extended workflows and delivering all requested artifacts. We also appreciate the recognition of TaskPass’s ability and of V-Protocol’s gains. The reviewer’s encouraging feedback has further motivated us to refine the work and strengthen its empirical support. In response to the reviewer’s suggestions, we have added human evaluation and agreement analyses, updated the component analysis, and clarified our plan to publicly release all VCB materials.

Critical Weaknesses and Q1: Human Review

Critical Weaknesses. My main concern is the reliability of the evaluation. The checklists are largely produced with language models, and final requirement and quality judgments use Qwen3.8-Flash-Next. Humans review the checklists, but the paper does not report how many tasks were checked, how often humans disagreed with the evaluator, or whether the final system ranking survives human evaluation. This matters because the evaluated agent also uses a Qwen3.8-family controller. The component analysis is also difficult to interpret. The full system reuses historical outputs, including recovery runs, while the ablations use a later frozen runtime. Consequently, the reported differences cannot be attributed only to runtime enforcement or module selection.

Q1. What is the agreement between the automated evaluator and human reviewers on a random subset of VCB? Does human evaluation preserve the system ranking?

A1: We thank the reviewer for helping us strengthen the reliability of our evaluation. We have added a human evaluation of the final deliverables on all 600 VCB tasks, covering all 12 categories and all three tiers.

Ten professional annotators each reviewed 72 tasks, with outputs from V-Protocol, SCOPE, and the original GEMS and Mind-Brush for each task. System identities and execution traces were concealed. Annotators could inspect the requests, source materials, reference images, and checklists. The resulting 720 task annotations include 120 tasks reviewed by two annotators. The annotators spent an average of 27 hours each, totaling 270 person-hours.

The results have been added to Appendix B.4:

SystemHuman–judge agreementHuman TaskPassAutomated TaskPassRank
V-Protocol91.57%56.67%69.00%1
SCOPE86.58%37.83%42.00%2
GEMS89.22%8.25%8.00%3
Mind-Brush90.85%0.67%0.33%4

These results provide human validation of the four-system ranking. Agreement compares human and automated requirement judgments, with an overall agreement rate of 89.51%. The human TaskPass ranking is V-Protocol > SCOPE > GEMS > Mind-Brush, matching the automated ranking. On the 120 doubly annotated tasks, agreement between annotators is 88.22%, with Krippendorff’s α = 0.758 and a 95% confidence interval of [0.719, 0.796].

The comparable human–human agreement (88.22%) and human–judge agreement (89.51%) rates further support the reliability of the automated requirement evaluation.

Q2: Ablations under matched conditions

Q2. Can the full method and ablations be rerun with the same runtime, task subset, budgets, and fresh outputs?

A2: Of course, we thank the reviewer for this helpful suggestion. We have expanded the component analysis to the complete 600-task VCB. The two ablated configurations use image base seeds 123, 124, and 125. We report the mean and sample standard deviation across three results for each configuration.

ConfigurationTasks per runRP (mean ± SD)TP (mean ± SD)VQ (mean ± SD)
w/o Runtime Enforcement60081.59 ± 0.5760.44 ± 0.793.760 ± 0.038
w/o Adaptive Module Selection60082.28 ± 0.1861.83 ± 0.933.865 ± 0.006
Full V-Protocol60094.28 ± 0.1869.11 ± 0.674.202 ± 0.005

Full V-Protocol outperforms both ablated configurations across RP, TP, and VQ, isolating and quantifying the performance gains attributable to the proposed modules. These results have been updated in Table 4.

Q3: Release scope

Q3. Which parts of VCB, including source files and evaluation annotations, will be released?

A3: We sincerely thank the reviewer for their interest in our benchmark. Our goal in introducing VCB is to broaden the scope of T2I-agent evaluation for the community. This also motivated our use of open-source models for agent execution and evaluation. Therefore, we will definitely release all VCB materials, including the original tasks, source files, checklists, automated evaluation results, and human annotations.

Dear Reviewer K7qP,

Thank you for your time and thoughtful reviews. Please be reminded that the author-reviewer discussion period ends on November 18.

Since the authors have procvided their feedback, please review the feedback and actively participate in the discussion. If you have any remaining questions or concerns, please raise them before the deadline so the authors have sufficient time to respond. Thanks.

Best regards,

AC

Thanks for the response from the authors. I maintain my positive score.

AuthorsOpenReview ↗

Thanks again for Reviewer K7qP and AC

Thank you for carefully considering our rebuttal and for your continued positive assessment. Your constructive feedback helped us strengthen the empirical evaluation, and we appreciate your recognition of VCB’s focus on end-to-end task completion. We hope VCB will serve as a useful resource for further research on reliable image agents.

Reviewer_ks66OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VCB, a benchmark comprising 600 tasks organized into 12 categories and three complementary tiers, spanning from individual capability tests to end-to-end delegated tasks. VCB is accompanied by an independent visual quality evaluation framework that assesses outputs along four dimensions: content correctness, visual implementation quality, constraint preservation, and delivery completeness. In addition, the paper introduces V-Protocol, a reference agent that dynamically assembles reusable task modules, cognitive modules, and presentation modules at inference time. These modules exchange structured intermediate artifacts, share a common execution framework, manage tool-calling processes, and validate task completion.

Critical strengths
  1. The paper extends the evaluation scope from single-image quality to end-to-end task completion, which better aligns with its central research objective.
  2. The paper adopts a multidimensional evaluation framework, helping ensure that high-quality visual outputs are not mistaken for successful task completion. 3.V-Protocol achieves a 69.0% strict task success rate on the VCB benchmark, substantially outperforming the SCOPE baseline, which achieves only 42.0%.
Critical weaknesses
  1. TaskPass relies on an LLM judge, meaning that evaluation errors could directly affect the key performance scores and the conclusions drawn from them. The authors should provide more extensive score validation results to better demonstrate the reliability of the judge.

  2. Some tool calls appear to have a substantial impact on the overall results in the existing comparisons. It would therefore be helpful to disentangle the contributions of tools themselves, such as file access and tool adaptation, from the gains attributable to the proposed modules. Establishing strong baselines under matched conditions would enable a more rigorous assessment of the modules’ actual contributions.

Questions for authors
  1. Could the authors conduct a human review of deliverables selected through stratified sampling and report the agreement rate and Cohen’s κ between human and model judgments?

  2. Could the authors rerun the full system and both variants under the same frozen runtime and identical resource budgets, using multiple random seeds, to clearly isolate and quantify the performance gains attributable to the proposed modules?

6 comments

We thank the reviewer for recognizing VCB’s focus on end-to-end task completion, the value of its multidimensional evaluation, and V-Protocol’s improvement in strict task. We also appreciate the suggestions for strengthening evaluator reliability and distinguishing material-access gains from framework contributions. In response, we have added human evaluation covering all 600 VCB tasks, access-adapted baselines, and further component analyses.

Critical Weaknesses 1 and Q1: Human Review

Critical Weaknesses 1. TaskPass relies on an LLM judge, meaning that evaluation errors could directly affect the key performance scores and the conclusions drawn from them. The authors should provide more extensive score validation results to better demonstrate the reliability of the judge.

Q1. Could the authors conduct a human review of deliverables selected through stratified sampling and report the agreement rate and Cohen’s κ between human and model judgments?

A1: We thank the reviewer for the interest in the benchmark and for highlighting the importance of evaluation reliability. We have added a human evaluation of the final deliverables on all 600 VCB tasks, covering all 12 categories and all three tiers.

Ten professional annotators each review 72 tasks, with outputs from V-Protocol, SCOPE, and the original GEMS and Mind-Brush for each task. System identities and execution traces are concealed. Annotators can inspect the requests, source materials, reference images, and checklists. The resulting 720 task annotations include 120 tasks reviewed by two annotators. The annotators spend an average of 27 hours each, totaling 270 person-hours.

The results have been added in Appendix B.4:

SystemHuman–judge agreementCohen’s κHuman TaskPassAutomated TaskPassRank
V-Protocol91.57%0.41056.67%69.00%1
SCOPE86.58%0.54437.83%42.00%2
GEMS89.22%0.7618.25%8.00%3
Mind-Brush90.85%0.4900.67%0.33%4

These results provide human validation of the four-system ranking. Agreement and Cohen’s κ compare human and automated requirement judgments. Overall agreement is 89.51%, with a pooled Cohen’s κ of 0.783. The human TaskPass ranking is V-Protocol > SCOPE > GEMS > Mind-Brush, matching the automated ranking. On the 120 doubly annotated tasks, agreement between annotators is 88.22%, with Krippendorff’s α = 0.758 and a 95% confidence interval of [0.719, 0.796].

The comparable human–human (88.22%) and human–judge agreement (89.51%) rates further support the reliability of our automated requirement evaluation.

Critical Weaknesses 2: Tool access and module contributions

Critical Weaknesses 2. Some tool calls appear to have a substantial impact on the overall results in the existing comparisons. It would therefore be helpful to disentangle the contributions of tools themselves, such as file access and tool adaptation, from the gains attributable to the proposed modules. Establishing strong baselines under matched conditions would enable a more rigorous assessment of the modules’ actual contributions.

A2.1: We agree that source-material access should be accounted for in the comparison. We add GEMS + access and Mind-Brush + access, preserving their native workflows while enabling on-demand access to the public VCB materials, and evaluate both on all 600 tasks.

  • GEMS + access: A read-only material-access interface is attached to the existing decomposition, planning, prompt-construction, verification, and refinement stages. The shared controller model within the native GEMS workflow can list directories, search and read text, and inspect source images and PDF pages. The retrieved information enters the current stage’s context, and GEMS continues with its native planning, generation, editing, and verification procedure.
  • Mind-Brush + access: The same interface is exposed during knowledge reasoning and knowledge review. Its controller can consult source materials while deriving content and reviewing its reasoning, then incorporate that evidence into subsequent generation and revision.

The overall results, now included in Table 2, are:

SystemRP ↑TP ↑VQ ↑
Mind-Brush12.690.332.308
Mind-Brush + access48.718.503.245
GEMS30.928.002.738
GEMS + access80.2937.834.003
SCOPE81.7842.003.982
V-Protocol94.4869.004.200

Material access substantially improves both baselines. GEMS + access approaches SCOPE in overall RP and TP and slightly exceeds its VQ. V-Protocol still achieves the highest overall TaskPass among these systems.

A2.2: Regarding our contribution beyond material access, V-Protocol outperforms both GEMS + access and SCOPE, which also have access to source materials. V-Protocol outperforms both GEMS + access and SCOPE, which also have access to source materials. This supports the contribution of our framework beyond material access itself. The component analysis in Section 5.2 and Table 4 further examines the contributions of adaptive module composition and runtime enforcement, including separate ablations of procedural guidance and candidate recommendation. On Mind-Bench, where file access is not required, V-Protocol also improves CSA from SCOPE’s 0.392 to 0.454.

Q2: Matched component ablations

Q2. Could the authors rerun the full system and both variants under the same frozen runtime and identical resource budgets, using multiple random seeds, to clearly isolate and quantify the performance gains attributable to the proposed modules?

A3: Of course, we thank the reviewer for this helpful suggestion. We have expanded the component analysis to the complete 600-task VCB. The two ablated configurations use image base seeds 123, 124, and 125. We report the mean and sample standard deviation across three results for each configuration.

ConfigurationTasks per runRP (mean ± SD)TP (mean ± SD)VQ (mean ± SD)
w/o Runtime Enforcement60081.59 ± 0.5760.44 ± 0.793.760 ± 0.038
w/o Adaptive Module Selection60082.28 ± 0.1861.83 ± 0.933.865 ± 0.006
Full V-Protocol60094.28 ± 0.1869.11 ± 0.674.202 ± 0.005

Full V-Protocol outperforms both ablated configurations across RP, TP, and VQ, isolating and quantifying the performance gains attributable to the proposed modules. These results have been updated in Table 4.

Dear Reviewer ks66,

Thank you for your time and thoughtful reviews. Please be reminded that the author-reviewer discussion period ends on November 18.

Since the authors have procvided their feedback, please review the feedback and actively participate in the discussion. If you have any remaining questions or concerns, please raise them before the deadline so the authors have sufficient time to respond. Thanks.

Best regards,

AC

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their careful reviews and shared recognition of the practical relevance of visual commission completion and the value of VCB. Their comments have helped us refine the presentation and strengthen the evaluation. Below, we summarize the additions addressing the common concerns. We provide point-by-point responses to each reviewer separately. The manuscript and the supplementary material have also been updated accordingly.

1. Human evaluation and evaluator reliability

We have added a human evaluation of the final deliverables on all 600 VCB tasks, covering all 12 categories and all three tiers.

Ten professional annotators each reviewed 72 tasks, with outputs from V-Protocol, SCOPE, and the original GEMS and Mind-Brush for each task. System identities and execution traces were concealed. Annotators could inspect the requests, source materials, reference images, and checklists. The resulting 720 task annotations include 120 tasks reviewed by two annotators. The annotators spent an average of 27 hours each, totaling 270 person-hours.

The results have been added to Appendix B.4*

SystemHuman–judge agreementHuman TaskPassAutomated TaskPassRank
V-Protocol91.57%56.67%69.00%1
SCOPE86.58%37.83%42.00%2
GEMS89.22%8.25%8.00%3
Mind-Brush90.85%0.67%0.33%4

Agreement compares human and automated requirement judgments. Overall agreement is 89.51%, with a pooled Cohen’s κ of 0.783. The human TaskPass ranking is V-Protocol > SCOPE > GEMS > Mind-Brush, matching the automated ranking. On the 120 doubly annotated tasks, agreement between annotators is 88.22%, with Krippendorff’s α = 0.758 and a 95% confidence interval of [0.719, 0.796]. The comparable human–human and human–judge agreement rates further support the reliability of the automated requirement evaluation.

2. Expanded component analysis

We have expanded the component analysis to the complete 600-task VCB. The two ablated configurations use image base seeds 123, 124, and 125. We report the mean and sample standard deviation across three results for each configuration.

ConfigurationTasks per runRP (mean ± SD)TP (mean ± SD)VQ (mean ± SD)
w/o Runtime Enforcement60081.59 ± 0.5760.44 ± 0.793.760 ± 0.038
w/o Adaptive Module Selection60082.28 ± 0.1861.83 ± 0.933.865 ± 0.006
Full V-Protocol60094.28 ± 0.1869.11 ± 0.674.202 ± 0.005

Full V-Protocol outperforms both ablated configurations across RP, TP, and VQ. These results have been updated in Table 4.

3. Source-material access for baselines

We add GEMS + access and Mind-Brush + access, preserving their native workflows while enabling on-demand access to the public VCB materials, and evaluate both on all 600 tasks.

  • GEMS + access: A read-only material-access interface is attached to the existing decomposition, planning, prompt-construction, verification, and refinement stages. The shared controller model within the native GEMS workflow can list directories, search and read text, and inspect source images and PDF pages. The retrieved information enters the current stage’s context, and GEMS continues with its native planning, generation, editing, and verification procedure.
  • Mind-Brush + access: The same interface is exposed during knowledge reasoning and knowledge review. Its controller can consult source materials while deriving content and reviewing its reasoning, then incorporate that evidence into subsequent generation and revision.

The overall results, now included in Table 2, are:

SystemRP ↑TP ↑VQ ↑
Mind-Brush12.690.332.308
Mind-Brush + access48.718.503.245
GEMS30.928.002.738
GEMS + access80.2937.834.003
SCOPE81.7842.003.982
V-Protocol94.4869.004.200

Material access substantially improves both baselines. GEMS + access approaches SCOPE in overall RP and TP and slightly exceeds its VQ. V-Protocol still achieves the highest overall TaskPass among these systems, retaining an advantage over baselines that also have source-material access.

AuthorsOpenReview ↗

To AC: Evidence Supporting Acceptance and Request for Clarification

Dear Area Chair,

Thank you for the time and care you have devoted to our submission, particularly for following up with the reviewers individually and encouraging their engagement with our rebuttal. We sincerely appreciate your efforts to facilitate a substantive discussion of our work.

We are writing to respectfully ask you to consider Submission 572 for acceptance in light of the additional evidence provided during rebuttal. We also seek your help in clarifying the specific concerns underlying Reviewer ks66’s unchanged weak-reject recommendation, so that we can address them directly.

Evidence Supporting Acceptance

VCB’s central contribution is to distinguish a visually convincing output from a successfully completed commission. An agent may produce a polished image while omitting a requested deliverable or violating a constraint from the source materials. VCB makes these failures measurable. During rebuttal, we added the following evidence:

  • Human validation of the main result. Ten professional annotators evaluated outputs from four systems across all 600 tasks, with system identities concealed. Human TaskPass was 56.67% for V-Protocol and 37.83% for SCOPE, compared with automated scores of 69.00% and 42.00%. Human judgments yielded lower absolute completion rates, while preserving V-Protocol’s advantage over SCOPE. At the individual-requirement level, human–judge agreement was 89.51%, with pooled Cohen’s κ of 0.783 (Appendix B.4).

  • Material-access-adapted baselines. Enabling source-file access increased GEMS’s automated TaskPass from 8.00% to 37.83%, demonstrating the importance of accounting for material access. Under automated evaluation, SCOPE achieved 42.00% and V-Protocol achieved 69.00%. These results provide a clearer comparison with stronger, access-adapted baselines (Table 2).

  • Expanded component comparisons. The component analysis now covers all 600 tasks. Reported mean TaskPass was 69.11% for the full system, 60.44% without runtime enforcement, and 61.83% with fixed module selection. These results extend the empirical analysis of the proposed components (Table 4).

Request for Clarification

Reviewer ks66 thanked us “for addressing my concerns” and maintained their weak-reject recommendation. Their follow-up leaves unclear whether an original concern remains decisive or whether another consideration supports the recommendation. We would appreciate your help in clarifying the remaining grounds, particularly regarding evaluator reliability and experimental controls, so that we can respond precisely.

We believe that VCB’s value, together with the additional empirical evidence, makes a strong case for acceptance. We would be grateful for your consideration of these results in your assessment of the revised paper.

Thank you again for your time and coordination throughout the discussion.

Best regards,
The Authors of Submission 572

I will ask Reviewer ks66 for clarification.

Best regards,

AC

AuthorsOpenReview ↗

Thanks Again!

Thank you for your time and help in seeking clarification from Reviewer ks66. We feel fortunate to have such a conscientious and responsive Area Chair guiding the discussion! We would be happy to provide any further clarification needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.