acceptodds
Under review as a conference paper at ICLR 2027

VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chaining

Abstract

Recent advancements extend Multimodal Large Language Models (MLLMs) beyond passive visual perception to actively composing external tools for complex visual tasks. Evaluating these capabilities requires examining how models select visual operations, compose them into multi-step workflows, and interpret intermediate outputs. We introduce VisualToolChain-Bench (VTC-Bench), a benchmark for evaluating multi-step visual tool composition in MLLMs. Our framework features 35 diverse visual operations and 680 curated problems structured across a nine-category functional taxonomy, each paired with reference execution trajectories that serve as diagnostic anchors for fine-grained analysis. Crucially, VTC-Bench supports a dual-paradigm evaluation protocol: a code-driven mode that tests programmatic tool orchestration, and an interface-driven mode that assesses structured tool invocation through predefined tool-sets. This design enables systematic diagnosis of where model capabilities diverge across execution paradigms. Extensive experiments on 19 leading MLLMs reveal critical limitations: models struggle with compositional visual tool use, with the best model achieving 51% accuracy. Multi-tool composition remains a fundamental bottleneck. Models concentrate their calls on a small subset of operations and make limited use of reference tools even when explicitly provided. By identifying these challenges, VTC-Bench provides a rigorous foundation for developing more capable visual agentic models.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Agentic Models, Visual Tool Use Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AyG6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces VTC-Bench, a benchmark of 680 visual questions with 35 OpenCV-based tools and expert-validated reference trajectories. It evaluates 19 model configurations under direct answering, code-based execution, and structured tool invocation. Results show uneven benefits from tools and difficulties in selecting, composing, and checking visual operations.

Critical strengths

The expert-validated reference trajectories and the comparison between code and interface modes are useful. They provide a concrete setting for examining how models choose and execute visual operations, beyond reporting final-answer accuracy alone.

Critical weaknesses
  1. The novelty needs a more precise justification. A single code interpreter can already execute dependent image-processing operations, so Table 1 compares tools at different levels of granularity. The reference trajectories and dual execution modes are useful additions, but the paper does not clearly establish what additional capability they reveal beyond existing visual tool-use benchmarks.

  2. The central diagnosis is stronger than the evidence supports. Low accuracy shows that the tasks are difficult, but does not identify tool composition as the main bottleneck. Using few tool types or departing from a reference trajectory is also not necessarily a failure. Section 4.3 acknowledges these limitations, including incomplete adoption of the supplied reference tools, but the abstract makes a stronger claim. The conclusions should distinguish observed behavior from its proposed explanation.

  3. The efficiency metric needs a clearer definition. Section 3.3 describes extracting an effective toolchain by tracing data flow, but does not explain how this works when an answer draws on several inspected images. A redundant operation may remain on the final data-flow path, while a discarded branch may still provide useful information. Without explaining these cases, the ratio is difficult to interpret as tool-use efficiency.

  4. Benchmark availability is unresolved. I could not find a benchmark repository or an explicit release commitment in the manuscript. Documenting prompts and hyperparameters does not make the exact questions, processed images, reference trajectories, and evaluation implementation reproducible. Since these resources are the main contribution, their availability is important to assessing the paper’s practical value.

Questions for authors
  1. What does function-level dependency evaluation reveal beyond executing dependent operations inside a code interpreter? Could you explain this using an existing example and clarify the criteria behind Table 1? This would help distinguish the contribution from differences in API packaging.

  2. Which existing results specifically support a failure of tool composition, as opposed to perception, parameter selection, or interpretation of tool outputs? Given the qualifications in Section 4.3, could the abstract and conclusions more clearly separate observations from explanations?

  3. Could you provide the procedure or pseudocode for extracting an effective toolchain, together with one worked trajectory? Please explain how it handles multiple inspected images, discarded branches, and redundant operations on the final data-flow path.

  4. Could the conclusions about open-source models be scoped more carefully? The general-purpose open-source evaluation covers only Qwen3-VL variants, and Section 4.1 uses different execution frameworks for different models. Please distinguish results for these configurations from broader claims about model capability.

  5. Will the complete benchmark and evaluation implementation be released, and on what timeline? Can an anonymous version be provided during review? If some components cannot be redistributed, please explain how users will obtain the exact evaluation inputs.

15 comments

We sincerely thank you for your detailed review and for recognizing the value of the expert-verified reference toolchains and the two execution modes. We particularly appreciate your questions about the contribution, tool composition, efficiency metric, and reproducibility; they have helped us clarify both the evidence and its interpretation.

Response to W1.

The novelty needs a more precise justification. A single code interpreter can already execute dependent image-processing operations, so Table 1 compares tools at different levels of granularity. The reference trajectories and dual execution modes are useful additions, but the paper does not clearly establish what additional capability they reveal beyond existing visual tool-use benchmarks.

Response:

Thank you for raising the distinction between tool-interface granularity and the capability being evaluated. We agree that a code interpreter can execute dependent operations. VTC-Bench examines how models construct visual solutions when later functions require data produced by earlier ones, using 35 visual operations, paired Code/Interface access, expert-validated references, and process analysis.

Table 1 separately reports the operation space, interaction modes, reference trajectories, multi-tool composition, and function-level dependency:

Table 1 itemCriterion explanation / existing markerInterpretation
ModeCode, Interface, or bothInteraction form supplied by the benchmark
#Tools and footnote †Code interpreter = 1†; VTC-Bench toolkit = 35Read the count with tool type; one interpreter does not mean one expressible operation
RTReference tool-call trajectoryProvision of reference tool-call trajectories
MTCMulti-tool composition for a single taskMultiple tools composed within a task
FLDFunction-level visual artifacts become processing inputs or parameters of subsequent functionsExamine the generation and use of masks, contours, coordinates, and related artifacts
TIR-Bench MTC/FLDBoth marked partial supportThe existing table recognizes programmatic composition and dependency
Unestablished markernot established under these criteriaDoes not assert theoretical impossibility

GTA, Agent-X, VisualToolBench, and AgentVista are marked as supporting MTC, while FLD is not established under the table’s criteria; TIR-Bench has partial support for both MTC and FLD. MTC describes composing multiple tools within a task. FLD identifies a prerequisite: a later function requires intermediate data produced by a preceding function. VTC-Bench organizes tasks, references, and process analysis around these operations and artifacts, allowing operation selection, artifact generation, and parameter connections to be examined separately.

The paper’s red-chess-piece counting example illustrates this distinction. Color filtering produces a mask, morphology cleans it, component analysis returns object centroids, and a drawing step uses those centroids for positional annotation. Drawing can still use the original image as its background, while its position parameters come from the preceding statistics function.

  • Function prerequisites constrain the model’s solution plan. In this workflow, component analysis operates on the processed target mask, and positional annotation depends on the centroids returned by that analysis. Selecting the functions includes establishing these prerequisites in the appropriate order; grouping their execution inside one interpreter preserves the same requirements.

  • References and execution records support analysis of how a solution is assembled. An expert-validated reference provides a feasible solution and identifies operations that can supply the information needed to answer the question. Recorded executions then show which functions and outputs the model actually used. This supports diagnosis of operation choice and data use while retaining full APR credit for valid alternative solutions.

Response to W2.

The central diagnosis is stronger than the evidence supports. Low accuracy shows that the tasks are difficult, but does not identify tool composition as the main bottleneck. Using few tool types or departing from a reference trajectory is also not necessarily a failure. Section 4.3 acknowledges these limitations, including incomplete adoption of the supplied reference tools, but the abstract makes a stronger claim. The conclusions should distinguish observed behavior from its proposed explanation.

Response:

We appreciate your suggestion to distinguish observed results more clearly from explanations of failure. We analyze composition capabilities through task performance and actual execution: APR scores the final answer, including correct short routes and valid alternative solutions. Using fewer tool types or departing from a reference is not itself a failure criterion. To clarify the evidence behind our interpretation, we build on the paper’s Direct/Code/Interface evaluation with two comparisons and examine the corresponding execution records.

The first uses a simple-tool strategy as a reference, evaluating all 680 questions under Direct (no tools), Crop+Zoom (only the Crop and Zoom In tools are provided, with repeated calls allowed), and Full (all 35 tools). For the closed-source model, we use Gemini 3.8 Flash: Gemini-3.0-Pro achieves the paper’s highest overall APR (51.18%), motivating our choice of a newer model from the same family. For the open-source model, we use Qwen3-VL-235B-A22B-Thinking, the best-performing open-source model evaluated in the paper, with an Interface APR of 38.09%.

These comparisons follow the paper’s scoring rules and judge prompt, using gpt-6-luna-official for assisted judging. Failures and empty answers score incorrect. Confidence intervals use paired bootstrap resampling over questions.

ModelDirectCrop+ZoomFullFull−Crop+Zoom (pp)95% CI (pp)
Gemini 3.8 Flash57.79% (393/680)60.74% (413/680)69.41% (472/680)+8.68[5.15, 12.21]
Qwen3-VL-235B-A22B-Thinking34.56% (235/680)34.85% (237/680)37.50% (255/680)+2.65[-0.88, 6.18]
  • The full workflow adds value beyond simple visual inspection. Gemini gains 8.68 percentage points over Crop+Zoom, with a 95% interval of [5.15, 12.21]. Crop+Zoom already permits repeated inspection through cropping and enlargement; the additional gain supports the value of the broader visual-processing workflow.

  • Qwen’s smaller net gain reflects both recovered and lost solutions. Its 2.65-point gain has an interval including zero. The paired outcomes contain 87 Full-only successes and 69 Crop+Zoom-only successes, so many additional solutions are offset by failures on questions solved under the simpler strategy. These outcomes motivate examining when a model benefits from a richer workflow and how it can preserve successful simpler routes.

The second comparison focuses more specifically on the composition workflow. Using Gemini 3.8 Flash on the 197 questions unsolved by both Direct and Crop+Zoom, we compare allowing composition with permitting only independent calls, with the same 35 tools available in both conditions. Full can read tool-generated intermediate images and make subsequent calls based on returned results. Independent calling, N, permits different tools, repeated calls, and parameter changes, but every image input is restricted to the original image. All actual results are provided together after calling finishes, with no further tool calls allowed.

QuestionsFullIndependent NNet correct gainGain (pp)95% CI (pp)McNemar p
19765/197 (32.99%)34/197 (17.26%)31+15.74[9.64, 21.83]3.12e-06
  • The controlled gain cannot be attributed to a larger tool set. Full answers 31 more questions correctly, a gain of 15.74 percentage points, while both conditions provide the same 35 tools. This supports the combined value of intermediate-image chaining and result feedback on this subset. Access to the individual operations alone does not recover the performance of the workflow that can build on earlier results.

  • Recorded trajectories show how successful solutions use this capability. Of the 38 Full-only successes, 36 trajectories use intermediate images across tool types. The performance advantage is thus accompanied by actual use of dependent processing, providing process evidence alongside the controlled comparison.

Intervals for these 197 questions use paired bootstrap resampling within task categories; McNemar tests are two-sided.

The pupil-distance example in Section 4.3’s “Case Study,” left panel, further illustrates what process diagnosis reveals. The model supplies estimated coordinates to Draw Circle and Draw Line, computes 60 and then 120 pixels, and finally guesses B (100 pixels). Drawing does not generate new pupil-localization information. The feasible workflow in the figure obtains circle centers through Threshold and Hough Circles and accounts for enlargement to obtain A (90 pixels).

  • Operation choice must address the information missing from the solution. In this failure, drawing guessed positions supplies no new evidence for those positions. The feasible workflow obtains locations through detection before measuring their separation. This gives a concrete role to workflow organization: choosing earlier operations that supply information needed by later computations.

In the revision, we will further distinguish empirical observations from their interpretation in the abstract and conclusions, explicitly connecting statements about composition difficulties to the controlled comparisons and process evidence.

Reviewer_yUubOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces VTC-Bench, a benchmark of 680 questions across nine tasks for evaluating visual tool composition with 35 operations. It provides expert-validated reference trajectories and supports both code-based and interface-based execution. Experiments on 19 models compare direct answering with tool use, with the best standard-setting accuracy reaching 51.18%. Prompt ablations, failure cases, and process audits examine tool selection, execution, and interpretation of intermediate results.

Critical strengths
  1. The paper addresses the selection and composition of interdependent visual operations, an important capability for visual agents. Providing both code-based and interface-based execution on the same tasks offers a useful setting for comparing how models perform under different forms of tool access.

  2. The evaluation covers 19 models and includes direct-answering baselines, prompt ablations, and manual audits of execution trajectories. The failure cases and process audits distinguish answer correctness from valid execution, providing useful evidence about errors in tool selection, interpretation, and verification.

Critical weaknesses
  1. The benchmark does not sufficiently establish how much performance depends on compositional tool use. Multi-step reference trajectories demonstrate possible solutions, but do not establish that comparable results require combining multiple tools. Without comparisons against simple tool-use baselines, the benchmark’s added value for evaluating composition beyond basic perception and image processing remains unclear.

  2. The error analysis remains too aggregated to explain the strongest models’ limitations. Failure categories are pooled across models and tasks, while the reference-tool experiment does not separate planning from execution because models often ignore the supplied guidance. Although the paper identifies meaningful failure patterns, it provides limited insight into why these failures occur and which bottlenecks future methods should address.

Questions for authors

Please see the Critical Weaknesses section for my main concerns.

5 comments

We sincerely thank you for your thoughtful review and for recognizing the value of interdependent visual operations, the two execution modes, and the 19-model evaluation. We especially appreciate your suggestions on simple-tool baselines and a more detailed error analysis, which have helped us explain the results more clearly.

Response to W1.

The benchmark does not sufficiently establish how much performance depends on compositional tool use. Multi-step reference trajectories demonstrate possible solutions, but do not establish that comparable results require combining multiple tools. Without comparisons against simple tool-use baselines, the benchmark’s added value for evaluating composition beyond basic perception and image processing remains unclear.

Response:

Thank you for suggesting a simple-tool baseline; this helps us make composition’s contribution more explicit. We use two complementary comparisons to examine whether simple operations achieve similar performance and how much composition adds when the toolkit is fixed. Execution-trace analysis complements these comparisons with process evidence.

The first experiment covers all 680 questions with Direct (no tools), Crop+Zoom (only the Crop and Zoom In tools are provided, with repeated calls allowed), and Full (all 35 tools). We select Gemini 3.8 Flash and Qwen3-VL-235B-A22B-Thinking as closed-source and open-source representatives, respectively. Gemini-3.0-Pro achieves the highest overall APR in the paper (51.18%), motivating our choice of a newer model from the same family. Qwen3-VL-235B-A22B-Thinking is the best-performing open-source model evaluated in the paper, with an Interface APR of 38.09%.

These comparisons follow the paper’s scoring rules and judge prompt, using gpt-6-luna-official for assisted judging. Failures and empty answers score incorrect. Confidence intervals use paired bootstrap resampling over questions.

ModelDirectCrop+ZoomFullFull−Crop+Zoom (pp)95% CI (pp)
Gemini 3.8 Flash57.79% (393/680)60.74% (413/680)69.41% (472/680)+8.68[5.15, 12.21]
Qwen3-VL-235B-A22B-Thinking34.56% (235/680)34.85% (237/680)37.50% (255/680)+2.65[-0.88, 6.18]

The following table reports paired outcomes on the same questions and counts the Full-only successes whose trajectories contain cross-type image dependencies:

ModelBoth correctFull onlyCrop+Zoom onlyBoth wrongNet Full gainCross-type paths / Full-only subset
Gemini 3.8 Flash365107481605995/107
Qwen3-VL-235B-A22B-Thinking16887693561834/87
  • Full improves over simple cropping and zooming. The gain is 8.68 percentage points for Gemini and 2.65 for Qwen, with Qwen’s interval including zero.

  • Additional successes include actual tool composition. Among the Full-only successes, 95/107 Gemini trajectories and 34/87 Qwen trajectories use intermediate images across tool types.

To further investigate tool-composition capabilities, the second experiment provides the same 35 tools in both conditions, allowing composition in one and only independent tool calls in the other. We evaluate Gemini 3.8 Flash on the 197 questions unsolved by both Direct and Crop+Zoom. Full permits tools to read intermediate images produced upstream and make further calls based on returned results. Independent calling, N, still permits different tools, repeated calls, and parameter changes, but every image input must be the original image. All actual results are returned together after calling finishes, with no further tool calls allowed.

QuestionsFullIndependent NNet correct gainGain (pp)95% CI (pp)McNemar p
19765/197 (32.99%)34/197 (17.26%)31+15.74[9.64, 21.83]3.12e-06
  • Composition adds value with the toolkit held fixed. Full answers 31 more questions correctly, a gain of 15.74 percentage points. With all 35 tools available, independent calling yields significantly lower accuracy on these 197 questions.

  • The gain accompanies use of intermediate outputs. Of the 38 Full-only successes, 36 trajectories use intermediate images across tool types. Together, the paired outcomes and execution records support the value of evaluating composition in VTC-Bench.

Intervals for these 197 questions use paired bootstrap resampling within task categories; McNemar tests are two-sided.

Response to W2.

The error analysis remains too aggregated to explain the strongest models’ limitations. Failure categories are pooled across models and tasks, while the reference-tool experiment does not separate planning from execution because models often ignore the supplied guidance. Although the paper identifies meaningful failure patterns, it provides limited insight into why these failures occur and which bottlenecks future methods should address.

Response:

We appreciate your suggestion to make the failure analysis more informative. We break down the appendix’s “Primary Failure Attribution” audit of 144 incorrect trajectories by model and task. The audit includes two trajectories per model–task pair across eight models and nine task categories, with one primary failure label per trajectory. Both tables refer to this same sample.

Model (N=18 each)Interpret.Select.Percept.Param.Early stopExtract.Repeat/trial
DeepEyes3720510
GPT-4o5611500
GPT-5.25551101
GPT-o310321101
GPT-o4-mini4411521
Gemini-3.0-Flash3731400
Gemini-3.0-Pro6133311
Qwen3-VL-235B-Thinking5233500
  • Interpreting tool outputs is a shared difficulty. Interpretation is the largest category overall (41/144), occurs in all eight models, and is the most frequent category for GPT-o3 and Gemini-3.0-Pro. This points to the value of checking whether a returned image or quantity supports the requested answer.

  • Similar accuracy can hide different failure patterns. Gemini-3.0-Pro and Gemini-3.0-Flash have similar Interface accuracy (51.03% and 50.74%) but differ in their most frequent errors: interpretation for Pro (6/18) and tool selection for Flash (7/18). These patterns suggest different priorities: verifying returned evidence for Pro and selecting operations that supply missing information for Flash.

The task breakdown makes these improvement priorities more specific:

Task category (N=16 each)Interpret.Select.Percept.Param.Early stopExtract.Repeat/trial
Attention focusing1650301
Chart8400211
Color3307300
Counting6612100
Math6341011
Measurement7140400
OCR3330421
Perceptual restoration2700700
Spatial reasoning5231500
  • Color: validate segmentation before using it. Color accounts for 7 of the 11 parameterization errors in the audit. Checking whether the mask retains the target and excludes background could guide adjustments to color ranges and thresholds.

  • Chart and Measurement: connect returned quantities to the question. Interpretation accounts for 15/32 incorrect trajectories in these two categories. Useful checks include matching the output to the correct object, coordinate system, and scale before forming an answer.

  • Perceptual Restoration: choose the next operation and when to stop. Tool selection and premature stopping account for 14/16 errors. This suggests using each result to assess whether the required visual information has been recovered and whether further processing is needed.

Evaluation data and code are in the anonymous repository.

Dear Reviewer,

As the discussion period is coming to a close, we wanted to follow up briefly on our rebuttal. We would be grateful to know whether our responses resolved your concerns. If any points remain unclear, or if further experiments or clarifications would be helpful, we would be glad to provide them in the time remaining. Thank you again for your time and for your thoughtful feedback on our paper.

Best regards,

The Authors

Reviewer_qRabOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VTC-Bench evaluates multimodal models on visual tool workflows using 35 OpenCV operations and 680 questions across nine task categories, each paired with an expert-reviewed executable reference trajectory. It supports code execution and structured API calls, alongside direct-answering baselines. Experiments on 19 models report a best accuracy of 51.18%, inconsistent benefits from tool access, and concentrated tool use. Prompt ablations, manual process audits, and a small GIMP pilot provide additional analysis. The paper aims to provide a benchmark for diagnosing tool selection and composition, although the evidence identifying composition-specific difficulty remains incomplete.

Critical strengths
  • A useful evaluation resource. The OpenCV toolkit, varied visual tasks, and expert-reviewed trajectories provide concrete material for examining how agents execute visual workflows.

  • Evaluation beyond final accuracy. The 19-model comparison and process audits distinguish answer correctness from supported execution. Allowing alternative valid solutions also avoids equating competence with reproducing one reference trajectory.

Critical weaknesses
  • The central construct—compositional tool use—is insufficiently validated. The experiments show that models struggle on the selected tasks, but do not clearly distinguish composition failures from weak perception, difficulty using individual operations, parameter errors, or incompatibility with the execution framework. A reference containing several dependent operations demonstrates one workable solution; it does not establish that those operations are necessary. APR likewise measures final answers rather than successful composition: 16 of 96 audited answer-correct trajectories had invalid processes. The appendix audits help, but do not establish that models fail to combine operations they can otherwise execute correctly. This directly affects the benchmark’s main claimed contribution and weakens the conclusion that composition is a fundamental bottleneck.

  • The execution and scoring protocols leave consequential uncertainty. Code implements raw operations while Interface invokes predefined wrappers, and their permitted computations and resource comparability are insufficiently specified. The answer-judging rules and agreement with human judgments are also missing. These gaps limit reproducibility and confidence in the interpretation of paradigm differences.

Questions for authors
  1. What fraction of tasks requires dependent multi-operation composition beyond direct perception or crop/zoom? What controlled results distinguish composition failures from failures of individual operations?

  2. How are capability limits and budgets enforced across Code and Interface? What implements “Sharpen,” and are auxiliary computations such as PCA/minAreaRect available in both modes?

  3. What are the exact answer-judging rules, their agreement with human labels, and the uncertainty of the principal performance comparisons?

9 comments

We sincerely thank you for your careful review and for recognizing the value of the evaluation resource, process analysis, and support for valid alternative solutions. Your specific questions about composition, execution settings, and scoring have helped us clarify the design and evidence behind our evaluation.

Response to W1.

The central construct—compositional tool use—is insufficiently validated. The experiments show that models struggle on the selected tasks, but do not clearly distinguish composition failures from weak perception, difficulty using individual operations, parameter errors, or incompatibility with the execution framework. A reference containing several dependent operations demonstrates one workable solution; it does not establish that those operations are necessary. APR likewise measures final answers rather than successful composition: 16 of 96 audited answer-correct trajectories had invalid processes. The appendix audits help, but do not establish that models fail to combine operations they can otherwise execute correctly. This directly affects the benchmark’s main claimed contribution and weakens the conclusion that composition is a fundamental bottleneck.

Response:

Thank you for highlighting the need to connect composition claims with execution evidence. Our controlled comparison shows that using intermediate images and tool feedback improves accuracy even when both conditions provide the same tools. We combine this result with execution audits to examine how operations contribute to a solution. APR measures final answers; reference toolchains and recorded executions help us examine operation selection, input connections, and result interpretation.

To assess tool-composition capabilities more closely, we conduct a controlled comparison of Direct (no tools), Crop+Zoom (only the Crop and Zoom In tools are provided), and Full (all 35 tools). We also analyze execution trajectories to understand how models use tools and their intermediate outputs. We evaluate Gemini 3.8 Flash and Qwen3-VL-235B-A22B-Thinking on the same 680 questions as closed-source and open-source representatives, respectively. Gemini-3.0-Pro achieves the highest overall APR in the paper (51.18%), so we continue with the Gemini family using the newer Gemini 3.8 Flash. For the open-source representative, we select Qwen3-VL-235B-A22B-Thinking, the best-performing open-source model evaluated in the paper, with an Interface APR of 38.09%.

These comparisons follow the paper’s scoring rules and judge prompt, using gpt-6-luna-official for assisted judging. Failures and empty answers score incorrect. Confidence intervals use paired bootstrap resampling over questions.

ModelDirectCrop+ZoomFullFull−Crop+Zoom (pp)95% CI (pp)
Gemini 3.8 Flash57.79% (393/680)60.74% (413/680)69.41% (472/680)+8.68[5.15, 12.21]
Qwen3-VL-235B-A22B-Thinking34.56% (235/680)34.85% (237/680)37.50% (255/680)+2.65[-0.88, 6.18]

To relate these gains to actual tool use, we report the paired Full/Crop+Zoom outcomes and cross-tool image dependencies below:

ModelBoth correctFull onlyCrop+Zoom onlyBoth wrongNet Full gainCross-type paths / Full-only subset
Gemini 3.8 Flash365107481605995/107
Qwen3-VL-235B-A22B-Thinking16887693561834/87
  • The gain extends beyond cropping and zooming. Gemini gains 8.68 percentage points over Crop+Zoom, whose operations already allow repeated visual inspection. This supports the value of the broader visual-processing workflow on these questions.

  • A small net gain can conceal substantial changes in which questions are solved. Qwen’s 2.65-point gain, whose interval includes zero, combines 87 Full-only successes with 69 Crop+Zoom-only successes. The richer workflow recovers many failed answers but also loses answers obtained under the simpler strategy. This suggests that preserving successful simple solutions is an important goal alongside enabling more complex workflows.

  • Successful trajectories contain actual cross-tool dependencies. Intermediate images pass between tool types in 95/107 Gemini and 34/87 Qwen Full-only successes. These records show how models use the broader workflow in their solutions, providing evidence beyond the existence of multi-step reference trajectories.

To examine composition while keeping the toolkit fixed, we conduct a controlled experiment on the 197 questions unsolved by both Direct and Crop+Zoom. Both conditions provide the same 35 tools, with one allowing tool composition and the other permitting only independent tool calls. We use Gemini 3.8 Flash for this experiment. Full allows sequential tool calls using intermediate images and feedback, while independent N applies tools only to the original image and returns all results together for the final answer.

QuestionsFullIndependent NNet correct gainGain (pp)95% CI (pp)McNemar p
19765/197 (32.99%)34/197 (17.26%)31+15.74[9.64, 21.83]3.12e-06
  • Tool availability alone does not explain the gain. Full answers 31 more questions correctly, a gain of 15.74 percentage points, with all 35 tools available in both conditions. The difference supports the combined value of using intermediate images and adapting subsequent calls to tool feedback on these 197 questions.

  • The advantage accompanies executed dependency chains. Among the 38 Full-only successes, 36 trajectories use intermediate images across tool types. The successful solutions thus commonly exercise the capability enabled by Full, connecting the accuracy gain to observed workflows.

Intervals for these 197 questions use paired bootstrap resampling within task categories; McNemar tests are two-sided.

The pupil-distance example in Section 4.3’s “Case Study,” left panel, illustrates a concrete workflow problem. The model supplies estimated coordinates to Draw Circle and Draw Line, computes distances of 60 and then 120 pixels, and finally guesses B (100 pixels). Drawing functions render supplied coordinates; they do not detect or correct pupil locations. The feasible workflow shown in the figure uses Threshold and Hough Circles to obtain centers (70,70) and (250,70), then accounts for the 2× enlargement to obtain 90 pixels.

  • Successful tool execution is insufficient for a supported measurement. Drawing can render the supplied coordinates correctly while leaving their accuracy unresolved. The analysis must therefore check how the coordinates were obtained and whether the subsequent distance calculation uses a consistent scale.

Response to W2.

The execution and scoring protocols leave consequential uncertainty. Code implements raw operations while Interface invokes predefined wrappers, and their permitted computations and resource comparability are insufficiently specified. The answer-judging rules and agreement with human judgments are also missing. These gaps limit reproducibility and confidence in the interpretation of paradigm differences.

Response:

We appreciate the opportunity to clarify the execution and scoring protocols. Code provides native code execution, while Interface provides predefined tool access. Both use the same tasks and answer evaluation, with tool use managed by their respective capability constraints and controllers:

AspectCodeInterface
Operation scopeGenerate Python/OpenCV programs under the appendix capability prompt35 registered tools specified in the paper’s tool table
Call granularityOne execution can contain several operationsOne tool call can combine preprocessing, computation, and statistics
Interaction budgetQwen-Agent: up to 20 rounds per attemptQwen-Agent: up to 20 rounds per attempt
Answer scoringDeterministic rule matching plus GPT-4o judgingDeterministic rule matching plus GPT-4o judging

A Code execution can contain several OpenCV operations, and an Interface call can include preprocessing and returned statistics. We therefore interpret round budgets in their respective interaction units. The paper compares task performance under these two tool-access paradigms.

Scoring uses the human-verified ground-truth answers. The procedure in Section 4.1 first applies rule matching, followed by GPT-4o for unmatched answers. For multiple-choice questions, the judge extracts an option for comparison with the gold answer; for open answers, it judges equivalence. Failures and empty answers score incorrect, while process validity receives a separate evaluation.

The anonymous repository includes the evaluation data, 680 images, reference toolchains, execution program, configurations, and tool code.

Comments on the paper

1 comment

Dear reviewers, the authors have posted their responses to your reviews. Please have a look and discuss ASAP as the author-reviewer discussion period ends at November 18 at 11:59pm AoE

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.