- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces VTC-Bench, a benchmark of 680 visual questions with 35 OpenCV-based tools and expert-validated reference trajectories. It evaluates 19 model configurations under direct answering, code-based execution, and structured tool invocation. Results show uneven benefits from tools and difficulties in selecting, composing, and checking visual operations.
Critical strengths
The expert-validated reference trajectories and the comparison between code and interface modes are useful. They provide a concrete setting for examining how models choose and execute visual operations, beyond reporting final-answer accuracy alone.
Critical weaknesses
-
The novelty needs a more precise justification. A single code interpreter can already execute dependent image-processing operations, so Table 1 compares tools at different levels of granularity. The reference trajectories and dual execution modes are useful additions, but the paper does not clearly establish what additional capability they reveal beyond existing visual tool-use benchmarks.
-
The central diagnosis is stronger than the evidence supports. Low accuracy shows that the tasks are difficult, but does not identify tool composition as the main bottleneck. Using few tool types or departing from a reference trajectory is also not necessarily a failure. Section 4.3 acknowledges these limitations, including incomplete adoption of the supplied reference tools, but the abstract makes a stronger claim. The conclusions should distinguish observed behavior from its proposed explanation.
-
The efficiency metric needs a clearer definition. Section 3.3 describes extracting an effective toolchain by tracing data flow, but does not explain how this works when an answer draws on several inspected images. A redundant operation may remain on the final data-flow path, while a discarded branch may still provide useful information. Without explaining these cases, the ratio is difficult to interpret as tool-use efficiency.
-
Benchmark availability is unresolved. I could not find a benchmark repository or an explicit release commitment in the manuscript. Documenting prompts and hyperparameters does not make the exact questions, processed images, reference trajectories, and evaluation implementation reproducible. Since these resources are the main contribution, their availability is important to assessing the paper’s practical value.
Questions for authors
-
What does function-level dependency evaluation reveal beyond executing dependent operations inside a code interpreter? Could you explain this using an existing example and clarify the criteria behind Table 1? This would help distinguish the contribution from differences in API packaging.
-
Which existing results specifically support a failure of tool composition, as opposed to perception, parameter selection, or interpretation of tool outputs? Given the qualifications in Section 4.3, could the abstract and conclusions more clearly separate observations from explanations?
-
Could you provide the procedure or pseudocode for extracting an effective toolchain, together with one worked trajectory? Please explain how it handles multiple inspected images, discarded branches, and redundant operations on the final data-flow path.
-
Could the conclusions about open-source models be scoped more carefully? The general-purpose open-source evaluation covers only Qwen3-VL variants, and Section 4.1 uses different execution frameworks for different models. Please distinguish results for these configurations from broader claims about model capability.
-
Will the complete benchmark and evaluation implementation be released, and on what timeline? Can an anonymous version be provided during review? If some components cannot be redistributed, please explain how users will obtain the exact evaluation inputs.
15 comments
Response to W2.
The central diagnosis is stronger than the evidence supports. Low accuracy shows that the tasks are difficult, but does not identify tool composition as the main bottleneck. Using few tool types or departing from a reference trajectory is also not necessarily a failure. Section 4.3 acknowledges these limitations, including incomplete adoption of the supplied reference tools, but the abstract makes a stronger claim. The conclusions should distinguish observed behavior from its proposed explanation.
Response:
We appreciate your suggestion to distinguish observed results more clearly from explanations of failure. We analyze composition capabilities through task performance and actual execution: APR scores the final answer, including correct short routes and valid alternative solutions. Using fewer tool types or departing from a reference is not itself a failure criterion. To clarify the evidence behind our interpretation, we build on the paper’s Direct/Code/Interface evaluation with two comparisons and examine the corresponding execution records.
The first uses a simple-tool strategy as a reference, evaluating all 680 questions under Direct (no tools), Crop+Zoom (only the Crop and Zoom In tools are provided, with repeated calls allowed), and Full (all 35 tools). For the closed-source model, we use Gemini 3.8 Flash: Gemini-3.0-Pro achieves the paper’s highest overall APR (51.18%), motivating our choice of a newer model from the same family. For the open-source model, we use Qwen3-VL-235B-A22B-Thinking, the best-performing open-source model evaluated in the paper, with an Interface APR of 38.09%.
These comparisons follow the paper’s scoring rules and judge prompt, using gpt-6-luna-official for assisted judging. Failures and empty answers score incorrect. Confidence intervals use paired bootstrap resampling over questions.
| Model | Direct | Crop+Zoom | Full | Full−Crop+Zoom (pp) | 95% CI (pp) |
|---|---|---|---|---|---|
| Gemini 3.8 Flash | 57.79% (393/680) | 60.74% (413/680) | 69.41% (472/680) | +8.68 | [5.15, 12.21] |
| Qwen3-VL-235B-A22B-Thinking | 34.56% (235/680) | 34.85% (237/680) | 37.50% (255/680) | +2.65 | [-0.88, 6.18] |
-
The full workflow adds value beyond simple visual inspection. Gemini gains 8.68 percentage points over Crop+Zoom, with a 95% interval of [5.15, 12.21]. Crop+Zoom already permits repeated inspection through cropping and enlargement; the additional gain supports the value of the broader visual-processing workflow.
-
Qwen’s smaller net gain reflects both recovered and lost solutions. Its 2.65-point gain has an interval including zero. The paired outcomes contain 87 Full-only successes and 69 Crop+Zoom-only successes, so many additional solutions are offset by failures on questions solved under the simpler strategy. These outcomes motivate examining when a model benefits from a richer workflow and how it can preserve successful simpler routes.
The second comparison focuses more specifically on the composition workflow. Using Gemini 3.8 Flash on the 197 questions unsolved by both Direct and Crop+Zoom, we compare allowing composition with permitting only independent calls, with the same 35 tools available in both conditions. Full can read tool-generated intermediate images and make subsequent calls based on returned results. Independent calling, N, permits different tools, repeated calls, and parameter changes, but every image input is restricted to the original image. All actual results are provided together after calling finishes, with no further tool calls allowed.
| Questions | Full | Independent N | Net correct gain | Gain (pp) | 95% CI (pp) | McNemar p |
|---|---|---|---|---|---|---|
| 197 | 65/197 (32.99%) | 34/197 (17.26%) | 31 | +15.74 | [9.64, 21.83] | 3.12e-06 |
-
The controlled gain cannot be attributed to a larger tool set. Full answers 31 more questions correctly, a gain of 15.74 percentage points, while both conditions provide the same 35 tools. This supports the combined value of intermediate-image chaining and result feedback on this subset. Access to the individual operations alone does not recover the performance of the workflow that can build on earlier results.
-
Recorded trajectories show how successful solutions use this capability. Of the 38 Full-only successes, 36 trajectories use intermediate images across tool types. The performance advantage is thus accompanied by actual use of dependent processing, providing process evidence alongside the controlled comparison.
Intervals for these 197 questions use paired bootstrap resampling within task categories; McNemar tests are two-sided.
The pupil-distance example in Section 4.3’s “Case Study,” left panel, further illustrates what process diagnosis reveals. The model supplies estimated coordinates to Draw Circle and Draw Line, computes 60 and then 120 pixels, and finally guesses B (100 pixels). Drawing does not generate new pupil-localization information. The feasible workflow in the figure obtains circle centers through Threshold and Hough Circles and accounts for enlargement to obtain A (90 pixels).
- Operation choice must address the information missing from the solution. In this failure, drawing guessed positions supplies no new evidence for those positions. The feasible workflow obtains locations through detection before measuring their separation. This gives a concrete role to workflow organization: choosing earlier operations that supply information needed by later computations.
In the revision, we will further distinguish empirical observations from their interpretation in the abstract and conclusions, explicitly connecting statements about composition difficulties to the controlled comparisons and process evidence.
We sincerely thank you for your detailed review and for recognizing the value of the expert-verified reference toolchains and the two execution modes. We particularly appreciate your questions about the contribution, tool composition, efficiency metric, and reproducibility; they have helped us clarify both the evidence and its interpretation.
Response to W1.
Response:
Thank you for raising the distinction between tool-interface granularity and the capability being evaluated. We agree that a code interpreter can execute dependent operations. VTC-Bench examines how models construct visual solutions when later functions require data produced by earlier ones, using 35 visual operations, paired Code/Interface access, expert-validated references, and process analysis.
Table 1 separately reports the operation space, interaction modes, reference trajectories, multi-tool composition, and function-level dependency:
GTA, Agent-X, VisualToolBench, and AgentVista are marked as supporting MTC, while FLD is not established under the table’s criteria; TIR-Bench has partial support for both MTC and FLD. MTC describes composing multiple tools within a task. FLD identifies a prerequisite: a later function requires intermediate data produced by a preceding function. VTC-Bench organizes tasks, references, and process analysis around these operations and artifacts, allowing operation selection, artifact generation, and parameter connections to be examined separately.
The paper’s red-chess-piece counting example illustrates this distinction. Color filtering produces a mask, morphology cleans it, component analysis returns object centroids, and a drawing step uses those centroids for positional annotation. Drawing can still use the original image as its background, while its position parameters come from the preceding statistics function.
Function prerequisites constrain the model’s solution plan. In this workflow, component analysis operates on the processed target mask, and positional annotation depends on the centroids returned by that analysis. Selecting the functions includes establishing these prerequisites in the appropriate order; grouping their execution inside one interpreter preserves the same requirements.
References and execution records support analysis of how a solution is assembled. An expert-validated reference provides a feasible solution and identifies operations that can supply the information needed to answer the question. Recorded executions then show which functions and outputs the model actually used. This supports diagnosis of operation choice and data use while retaining full APR credit for valid alternative solutions.