acceptodds
Under review as a conference paper at ICLR 2027

OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

Abstract

Multimodal large language models (MLLMs) have made rapid progress on visual reasoning, yet many tasks require evidence that is difficult to obtain through internal reasoning alone—our pilot study on 12 benchmarks shows that simply increasing the inference budget yields only modest gains—such tasks demand grounded perception, external knowledge, and numerical measurements obtained through heterogeneous tools, whose outputs must be made interpretable to the model and retained across reasoning steps. We introduce OpenVLHarness, a multimodal reasoning harness organized around a capability layer and persistent multimodal memory. The capability layer combines task-level tool interfaces with evidence rendering, providing consistent access to heterogeneous backends and converting their outputs into model-readable observations. Persistent multimodal memory retains the corresponding images, structured data, and web references as addressable artifacts throughout each query. A coding interface executes generated programs over stored data, enabling computations that combine results from multiple tools. Together, these mechanisms support an iterative process of acquiring, inspecting, and composing multimodal evidence. Evaluated on 23 benchmarks across perception, spatial reasoning, visual search, and multimodal question answering, OpenVLHarness consistently surpasses strong MLLM backbones and outperforms existing visual tool-use baselines across six open and proprietary backbones of different scales, including +11.2 with Qwen3-VL-8B, +9.2 with Kimi K3, and +11.2 with GPT-6 Sol. In addition, our controlled study on 12 benchmarks shows that even though OpenVLHarness incurs additional cost from multi-step tool use, giving the base GPT-6 Luna an even larger reasoning budget (1.27x the cost of OpenVLHarness) improves the average only from 49.6 to 51.5, compared with 62.9 for OpenVLHarness. These results suggest that additional reasoning alone does not substitute for acquiring and composing task-relevant multimodal evidence.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Tool-Use, Visual Agent, Harness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ihKpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OpenVLHarness is a training-free system for visual tool use. A capability layer standardizes tool interfaces and renders heterogeneous outputs into model-readable observations. A per-query artifact registry retains images, structured data, and web references for later calls, including computations delegated to a Python coding agent using the same backbone. Evaluation spans 23 benchmarks and six principal backbones, with additional comparisons against fine-tuned agents. The paper also studies reasoning budgets, diagnoses failed trajectories, and explores specialization through instruction edits and generated tools.

Critical strengths
  1. The empirical breadth and reasoning-budget study provide convincing evidence of practical utility. Improvements extend across model families and task domains: the 23-benchmark average rises from 35.99 to 47.14 with Qwen3-VL-8B and from 54.26 to 65.48 with GPT-6 Sol (Table 1). Three-run Qwen3 results strengthen this evidence; the full-system average standard deviations are 0.35 and 0.24 for the two scales (Appendix H). Table 12 also tests whether additional internal reasoning can recover the gains. On 12 benchmarks, higher-budget direct inference reaches 51.53 versus 62.85 for the harness, despite 1.27× average backbone-token spending. Within the reported accounting scope, this supports the value of acquiring task-relevant evidence beyond extending reasoning alone. The breadth of the gains and this complementary analysis are central to my positive assessment.

  2. The design supports evidence composition and practical adaptation without weight updates. Retaining masks and arrays alongside readable renderings lets generated code compose tool results directly; the backpack trajectory also shows reuse across cropping, image search, and source verification (Example 8). The paper connects diagnosis to adaptation: a critic analyzes 9,980 failures, with 92% agreement on 100 human-annotated samples, and two workflows improve performance without updating backbone weights (Sections 3.4–3.5). Harness search adds 4.1 points on ERQA, while generated tools add 8.2 on SparBench; Appendix E.3 specifies disjoint adaptation and evaluation splits. These experiments give a concrete path for adapting the harness to particular tasks and strengthen its value as a reusable system.

Critical weaknesses

The interpretation and reporting of the memory ablation need clarification. Table 2 gives full-minus-textual differences of 2.21/3.39 points, whereas Section 3.3 states 3.1/4.5. Appendix B.2 specifies 22 datasets and subsampling, while the full row reproduces the 23-dataset main-table averages; the earlier Overall entries also appear to use equal-domain averaging. Please reconcile these figures and specify a consistent evaluation population and aggregation rule. There is also an interpretation issue: textual memory replaces tool-generated images with descriptions of at most 160 tokens, while the full variant restores both native images and reusable handles. This is a meaningful test of the combined multimodal-memory design, but it does not establish how much comes from visual access versus subsequent reuse. I would read it as evidence for the combined mechanism. Section 3.3 already acknowledges that schemas and rendering are evaluated jointly; a similarly explicit interpretation here would make the component claims easier to assess.

Questions for authors
  1. Reporting clarification: Which dataset membership, sample counts, and Overall weighting apply to each row of Table 2? How should the differences between the table, Section 3.3, and Appendix B.2 be reconciled? This matters for assessing the magnitude of the component gains.
  2. Memory interpretation: Is the final ablation intended to measure the joint benefit of native visual evidence and persistent image references? Making that scope explicit would clarify the conclusion. Separating these effects would be useful follow-up work, rather than a prerequisite for assessing the complete system.
  3. Cost clarification: Appendix B.6 explicitly excludes tool execution, and three search-task cost estimates are lower bounds. Can you clarify the unrecorded cache-write costs and estimate the excluded perception, search, and auxiliary-model costs? Representative end-to-end latency would also help assess practical efficiency beyond model-token spending.
3 comments
AuthorsOpenReview ↗

Author Response (1/2)

We thank Reviewer ihKp for recognizing (1) the empirical breadth and reasoning-budget study as convincing evidence of practical utility, and (2) a design that supports evidence composition and practical adaptation without weight updates.

Q1. Which datasets, samples, and weights did Table 2 use?

A1. We apologize for the confusion. We would like to clarify that Table 2 uses the 22 datasets in Table 1 excluding MMSearch-Plus. All variants use identical seed-42 subsets capped at 500 examples per dataset, and results are averaged over three independent runs. Each domain score is the mean across its datasets, and Overall equally averages the four domain scores. Table 1 instead reports the mean across all 23 datasets.

While updating Table 1 to include MMSearch-Plus during final manuscript preparation, summary values from the main experiments were inadvertently copied into the baseline and full-system rows of Table 2. We have corrected both rows using the saved outputs from the ablation experiments. The corrected table is provided below.

VariantCounting and groundingSearch and deep researchGeneral VQA and hallucinationSpatial understandingOverall
Model8B / 32B8B / 32B8B / 32B8B / 32B8B / 32B
Baseline (no tool)27.65 / 31.5118.72 / 22.9057.46 / 61.9544.88 / 48.0237.18 / 41.10
+ Tool41.70 / 43.3021.44 / 24.8952.41 / 61.4147.83 / 52.8440.84 / 45.61
+ Capability layer46.73 / 46.2025.97 / 32.7456.05 / 63.6348.16 / 52.6344.23 / 48.80
+ Textual memory47.27 / 45.7426.92 / 34.4357.54 / 64.8447.98 / 51.4644.93 / 49.12
Full OpenVLHarness46.97 / 51.0537.89 / 40.8859.90 / 67.1350.14 / 55.2148.73 / 53.57

The corrected multimodal-memory gains are +3.80/+4.45. We have verified these results against the original outputs. In the revised manuscript, we have corrected the values of Table 2 and the gains in Section 3.3 (lines 371–372), and stated the datasets and weighting in the Table 2 caption (lines 389–390) and Appendix B.2 (lines 1401–1403).

Q2. Is the final ablation intended to measure the joint benefit of native visual evidence and persistent image references?

A2. We would like to clarify that the textual memory reported in the second-to-last row of Table 2 is exactly what is requested by the reviewer. When a tool generates an image, the model receives the image in its context and can inspect it directly (native visual evidence). In this ablation setting, persistent memory stores only a text description (thus model does not have access to persistent image references). Later tools cannot retrieve the original image through that reference. We have revised Appendix B.2 (lines 1366–1376 of the revised manuscript) to make clear when the model sees the image and what the harness retains for later calls.

AuthorsOpenReview ↗

Author Response (2/2)

Q3. Can you clarify the end-to-end latency and cost including tool execution?

A3. We thank the reviewer for raising this. We have now estimated the end-to-end cost for the 12 benchmarks of Table 12 and added it to the model-token cost. The tools fall into two groups:

  • Locally served GPU tools: SAM 3 (Visual Grounding), Depth Anything 3 (Depth Estimation and Camera Trajectory) and PaddleOCR-VL (OCR).
  • Search tools billed through the Serper API and auxiliary local model to summarize search results: Text Search, Image Search and Webpage Visit.

Specifically, we count every tool call in the Table 12 trajectories, and time each request on one RTX A6000 to estimate the total GPU hours. We then convert GPU time to dollars at \0.33perA6000−hour,followingthepriceatRunPodCommunityCloud.ForSerper,atextsearchorImageSearchrequestcosts1creditandawebpagescrapecosts2.WepricecreditsatSerper’svolumerateof0.33 per A6000-hour, following the price at RunPod Community Cloud. For Serper, a text search or Image Search request costs 1 credit and a webpage scrape costs 2. We price credits at Serper’s volume rate of \\0.30 per 1,000. The three search tools also use a local Qwen3-VL-8B to extract evidence from search results and webpages. To price it, we replayed 100 search calls from these trajectories (40 text searches, 30 image searches, and 30 webpage visits) on one RTX A6000 serving 8 requests at a time, as in our deployment. Each extraction request takes 1.39 GPU-seconds, which we price at the same \$0.33 per A6000-hour and scale by each domain's search usage. For Visual Sketchpad, we time its GPU tools the same way. These are GroundingDINO detection (including its 16-crop sliding-window variant), SoM segmentation and Depth Anything. Codex runs its code in a CPU sandbox, so its cost is model tokens only. We report the cost per 1,000 questions below.

DomainOurs: tokensOurs: GPU toolsOurs: Serper APIOurs: aux. modelOurs: totalCodexVisual SketchpadVisual Sketchpad + GPU tools
Perception$1.03$0.033$0.000$0.000$1.07 (1×)1.52×1.20–4.27×1.28–4.35×
Search & DR$1.07$0.006$0.494$0.172$1.74 (1×)0.84×0.58–1.81×0.59–1.82×
General VQA$0.71$0.019$0.140$0.049$0.92 (1×)1.29×0.49–2.67×0.53–2.70×
Spatial$1.04$0.133$0.000$0.000$1.18 (1×)1.34×1.01–3.95×1.04–3.98×
Average$0.96$0.048$0.159$0.055$1.22 (1×)1.25×0.82–3.18×0.86–3.21×

Including tools, Ours costs less than Codex on average (Codex is 1.25×). Ours sits at the low end of Visual Sketchpad’s range (0.86–3.21× with its GPU tools). Across the 12 benchmarks, Ours scores 62.85 on average, against 48.31 for Codex and 49.50 for Visual Sketchpad. We have added this analysis and the measurement details to Appendix B.6 of the revised manuscript (lines 1724–1727 and 1782–1808, Table 15).

It's worth noting that our harness does not introduce additional page-reading or extraction service fees such as Jina Reader. Webpage text comes from the Serper scrape priced above, and each local tool model fits on a single RTX A6000.

Cache writes. Table 13 leaves the cache-write cells blank for RealX-Bench, FVQA, and BC-VL because the usage saved for these runs recorded cache reads but did not separate cache writes from uncached input. The input that is not a cache read (3,610 / 2,857 / 5,204 tokens per question) was therefore listed entirely as uncached input and priced at the uncached rate, which is why Appendix B.6 calls these costs a lower bound. To recover the cache-write costs, we re-ran these three datasets with per-call logging. Almost all of this input is cache writes, with fewer than 10 uncached tokens per question, as on the other nine datasets. Priced at the cache-write rate:

DatasetOurs $/1kCost-matched Base costBase scoreOurs score
RealX-Bench0.881.35×22.6840.72
FVQA0.751.33×39.3363.33
BC-VL1.571.04×37.5950.13
Average (12 datasets)1.23×51.5362.85

These costs are 9–11% higher than the lower bounds in Table 12 (\0.79,0.79, \\0.68, and \$1.44). Cost-matched Base still spends more than Ours on all three datasets and scores 12.5–24.0 points lower. The cost table above already uses these token costs. We have filled in these entries in Table 14 (formerly Table 13, lines 1742–1748; caption lines 1778–1780) in the revised manuscript.

Thanks for authors' response which addresses my concerns. I will maintain my positive rating.

Reviewer_CrMhOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OpenVLHarness, a training-free multimodal agent framework integrating heterogeneous tools through a unified capability layer and persistent multimodal memory. Experiments across 23 benchmarks demonstrate consistent improvements over existing visual tool-use baselines.

Critical strengths
  • Practical Framework: The framework effectively combines different visual tools and multimodal memory for multi-step reasoning.
  • Extensive Experiments: The evaluation covers multiple models and benchmarks, showing consistent performance gains.
  • Detailed Analysis: The paper includes ablation studies, failure analysis, and qualitative examples to support its findings.
Critical weaknesses
  • Limited Novelty: The main components, including tool interfaces, persistent memory, and code execution, have been explored in prior agent frameworks. While their multimodal integration is useful, the paper should better clarify what is technically new and what generalizable insights it provides beyond combining existing components.
  • Unfair Baseline Comparison: As shown in Table 22, OpenVLHarness has access to more specialized tools and web search than existing baselines. This makes it difficult to determine how much of the improvement comes from the harness design rather than the additional tool capabilities.
  • Limited Ablation Studies: The capability layer combines tool descriptions and output processing, but their individual contributions are not evaluated.
  • Incomplete Cost Analysis: The reported costs do not include external tool execution and serving overhead, making it difficult to assess the actual efficiency.
Questions for authors
  • Can the authors compare against baselines with the same tool capabilities?
  • Can the authors provide more detailed ablations for individual components?
  • What are the total inference costs and latency, including external tools?

I am willing to adjust my score according to the rebuttal.

3 comments
AuthorsOpenReview ↗

Author Response (1/2)

We thank Reviewer CrMh for recognizing (1) the practical framework that effectively combines different visual tools and multimodal memory, (2) the extensive experiments showing consistent performance gains, and (3) the detailed analysis, including ablations, failure analysis, and qualitative examples.

Q1. The main components, including tool interfaces, persistent memory, and code execution, have been explored in prior agent frameworks. While their multimodal integration is useful, the paper should better clarify what is technically new and what generalizable insights it provides beyond combining existing components.

A1. Two critical design choices differentiate our framework from prior agent frameworks: (1) Capability abstraction without artificial tool scaffolding or pipelining, and (2) Illustrative + exact evidence representation with automatic environment updates.

  1. Capability abstraction without artificial tool scaffolding or pipelining. Visual Sketchpad’s sliding-window detector prescribes a fixed 16-crop routine around its zoom-in and grounding tools; OctoTools chains planning, command generation, and verification into hard-coded pipelines. Such hand-designed procedures work well on in-domain tasks, where humans have verified that the procedure succeeds, but they may constrain the model’s ability to solve harder or out-of-domain tasks creatively. Our core capabilities layer exposes only fundamental perception, search, and computation capabilities, letting the model create its own compositions with the coding agent tool. This design enables more flexible and adaptable tool-use behaviors as described in Related Work (lines 456–460 of the original manuscript).

  2. Illustrative + exact evidence representation with automatic environment updates. In prior harnesses, each tool call often returns either just a lossy visual illustration or an implicitly stored global variable name. This causes the orchestrator to commonly lose track of available artifacts and their exact data structures in longer interactions. In OpenVLHarness, we not only provide both high quality illustrative and exact evidence representations, but also provide an explicit environment update indicating latest artifact inventory with data structure information. This design aims to avoid model confusion in longer interactions.

The table below shows the difference across harnesses.

FrameworkCapability abstractionDual evidence representationAutomatic environment update
Visual Sketchpad✗◐✗
OctoTools◐◐✗
OpenVLHarness✓✓✓

✓ Built-in support · ◐ Partial or tool-specific support · ✗ No supported mechanism.

Experiment in A2 where all harnesses call the same backend services shows that the performance gap persists when tool backends and search access are matched. In this setting, OpenVLHarness scores 12.04–15.20 points higher than both baselines.

Empirically, harnesses with artificial tool scaffolding or pipelining tend to perform poorly on newly released datasets beyond the ones they are built for. We share results on PerceptionBench [1], a recently released challenging perception benchmark:

HarnessQwen3-VL-8BGPT-6 Luna
Base (no tool)21.550.1
Visual Sketchpad22.148.9
OctoTools21.051.7
OpenVLHarness30.256.1

Q2. Differences in search access and tools make it hard to isolate the effect of the harness. Can you compare baselines with the same tool capabilities?

A2. We evaluated Visual Sketchpad and OctoTools with the OpenVLHarness tools, including our perception backends and search services, on the same questions from the 23 Table 1 datasets:

SetupTool BackendsQwen3-VL-8BGPT-6 Luna
Visual SketchpadOriginal32.1542.78
Visual SketchpadOpenVLHarness31.9443.72
OctoToolsOriginal (Generalist)33.3140.59
OctoToolsOriginal (All Tools)34.7642.18
OctoToolsOpenVLHarness32.8543.03
OpenVLHarnessOpenVLHarness47.1455.76

For OctoTools, the Generalist Solver toolset is the most commonly reported configuration, since OctoTools selects additional tools separately for each task using validation performance. We have also added the full-toolkit comparison which slightly improves performance but causes extremely slow inference speeds.

These results show that access to the same perception backends and search services does not close the performance gap between baseline harnesses and OpenVLHarness. The revised manuscript reports these results in Appendix B.8 (Table 17, lines 1890–1912) and summarizes them in Section 3.2 (lines 328–330).

AuthorsOpenReview ↗

Author Response (2/2)

Q3. The ablation combines tool descriptions and output processing. Can you separate the effects of individual components?

A3. The + Capability Layer variant in our ablation table combines tool descriptions and output processing because a significant portion of tool descriptions describe how tool output gets processed. Here, we further ablate adding only output processing without updating tool descriptions:

VariantCounting and groundingSearch and deep researchGeneral VQA and hallucinationSpatial understandingOverall
Model8B / 32B8B / 32B8B / 32B8B / 32B8B / 32B
Baseline (no tool)27.65 / 31.5118.72 / 22.9057.46 / 61.9544.88 / 48.0237.18 / 41.10
+ Tool41.70 / 43.3021.44 / 24.8952.41 / 61.4147.83 / 52.8440.84 / 45.61
+ Output Processing44.05 / 45.8324.12 / 29.3153.16 / 61.9948.00 / 52.9142.33 / 47.51
+ Tool Descriptions46.73 / 46.2025.97 / 32.7456.05 / 63.6348.16 / 52.6344.23 / 48.80
+ Textual memory47.27 / 45.7426.92 / 34.4357.54 / 64.8447.98 / 51.4644.93 / 49.12
Full OpenVLHarness46.97 / 51.0537.89 / 40.8859.90 / 67.1350.14 / 55.2148.73 / 53.57

From the table we can see that output processing alone adds 1.5 points at 8B and 1.9 at 32B, and tool descriptions add a further 1.9 and 1.3. Both components help, and each gives roughly half of the capability layer’s total gain (3.4 and 3.2 points). We have added this split to Appendix B.2 of the revised manuscript (Table 8, lines 1404–1418) and summarized it in Section 3.3 (lines 363–365).

Q4. The cost analysis excludes tool execution and serving overhead. What are the total inference cost and latency, including external tools?

A4. We thank the reviewer for raising this. We have now estimated the end-to-end cost for the 12 benchmarks of Table 12 and added it to the model-token cost. The tools fall into two groups:

  • Locally served GPU tools: SAM 3 (Visual Grounding), Depth Anything 3 (Depth Estimation and Camera Trajectory) and PaddleOCR-VL (OCR).
  • Search tools billed through the Serper API and auxiliary local model to summarize search results: Text Search, Image Search and Webpage Visit.

Specifically, we count every tool call in the Table 12 trajectories, and time each request on one RTX A6000 to estimate the total GPU hours. We then convert GPU time to dollars at \0.33perA6000−hour,followingthepriceatRunPodCommunityCloud.ForSerper,atextsearchorImageSearchrequestcosts1creditandawebpagescrapecosts2.WepricecreditsatSerper’svolumerateof0.33 per A6000-hour, following the price at RunPod Community Cloud. For Serper, a text search or Image Search request costs 1 credit and a webpage scrape costs 2. We price credits at Serper’s volume rate of \\0.30 per 1,000. The three search tools also use a local Qwen3-VL-8B to extract evidence from search results and webpages. To price it, we replayed 100 search calls from these trajectories (40 text searches, 30 image searches, and 30 webpage visits) on one RTX A6000 serving 8 requests at a time, as in our deployment. Each extraction request takes 1.39 GPU-seconds, which we price at the same \$0.33 per A6000-hour and scale by each domain's search usage. For Visual Sketchpad, we time its GPU tools the same way. These are GroundingDINO detection (including its 16-crop sliding-window variant), SoM segmentation and Depth Anything. Codex runs its code in a CPU sandbox, so its cost is model tokens only. We report the cost per 1,000 questions below.

DomainOurs: tokensOurs: GPU toolsOurs: Serper APIOurs: aux. modelOurs: totalCodexVisual SketchpadVisual Sketchpad + GPU tools
Perception$1.03$0.033$0.000$0.000$1.07 (1×)1.52×1.20–4.27×1.28–4.35×
Search & DR$1.07$0.006$0.494$0.172$1.74 (1×)0.84×0.58–1.81×0.59–1.82×
General VQA$0.71$0.019$0.140$0.049$0.92 (1×)1.29×0.49–2.67×0.53–2.70×
Spatial$1.04$0.133$0.000$0.000$1.18 (1×)1.34×1.01–3.95×1.04–3.98×
Average$0.96$0.048$0.159$0.055$1.22 (1×)1.25×0.82–3.18×0.86–3.21×

Including tools, Ours costs less than Codex on average (Codex is 1.25×). Ours sits at the low end of Visual Sketchpad’s range (0.86–3.21× with its GPU tools). Across the 12 benchmarks, Ours scores 62.85 on average, against 48.31 for Codex and 49.50 for Visual Sketchpad. We have added this analysis and the measurement details to Appendix B.6 of the revised manuscript (lines 1724–1727 and 1782–1808, Table 15).

It's worth noting that our harness does not introduce additional page-reading or extraction service fees such as Jina Reader. Webpage text comes from the Serper scrape priced above, and each local tool model fits on a single RTX A6000.

Thank you for the detailed rebuttal. I have adjust my score accordingly.

Reviewer_ReFYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OpenVLHarness, a training-free framework for multimodal tool use built around a capability layer and persistent multimodal working memory. The capability layer provides task-level interfaces and converts heterogeneous tool outputs into model-readable observations. The memory mechanism retains images, structured data, and web references as addressable artifacts, enabling subsequent tool calls and a Python Coding Agent to reuse intermediate results. The paper also explores two specialization workflows: optimizing harness instructions and generating task-specific tools.

The evaluation spans 23 benchmarks covering counting and grounding, multimodal search, general visual question answering, and spatial understanding, with several open and proprietary backbones. The complete system achieves substantial improvements over direct inference. However, differences in tool availability across methods, confounded memory ablations, and inconsistencies in the ablation results make it difficult to establish the independent contribution of the proposed harness design.

Critical strengths

S1. The paper addresses an important practical problem with a sensible system design.

Multimodal tools produce heterogeneous outputs that must be both interpretable to the orchestrating model and accessible to downstream computation. The distinction between rendered observations and retained underlying artifacts is well motivated. For example, an annotated segmentation image supports visual inspection, whereas region-level depth calculations require the actual masks and numerical depth arrays. Integrating these two forms of access into a common framework is practically useful.

S2. The evaluation is broad, and the complete system substantially improves over direct inference.

The experiments cover multiple task domains and model families. In Table 1, OpenVLHarness improves the average score from 35.99 to 47.14 for Qwen3-VL-8B and from 40.10 to 52.51 for Qwen3-VL-32B, with substantial gains also reported for other backbones. Improvements appear across several benchmarks rather than depending on a single favorable dataset. The three-run evaluation and standard deviations for the main Qwen3-VL experiments provide additional support for the stability of the overall gains.

S3. The paper provides useful analyses beyond aggregate benchmark scores.

The supplementary analyses examine tool-call frequency, repeated and failed calls, and both corrected and newly introduced errors relative to direct inference. These measurements help characterize the benefits and costs of tool use. The detailed trajectories also reveal meaningful behaviors, including recovery from execution errors and the composition of perception, retrieval, and computation. I also appreciate that the appendix explicitly documents limitations and process errors in some trajectories.

S4. The inference-budget comparison and specialization studies raise useful research questions.

The paper investigates whether additional direct reasoning can recover the benefits of tool use and whether different failure patterns favor instruction optimization or tool generation. These are practically relevant questions. Although the current experiments do not fully establish the broader conclusions, they provide promising evidence and useful directions for further controlled evaluation.

Critical weaknesses

W1. The baseline comparisons do not adequately control tool availability and backend capability.

Appendix G and Table 22 indicate that OpenVLHarness has access to search tools, while none of the evaluated baselines has web-search access. This is a substantial difference on benchmarks explicitly requiring external information. These comparisons demonstrate the benefit of the complete system configuration, but do not isolate the quality of its orchestration, evidence rendering, or memory mechanism.

The OctoTools configuration is particularly limited: Appendix G.1 states that it uses only the Generalist Solution Generator. Visual Sketchpad uses its original Grounding DINO and Depth Anything backends, whereas OpenVLHarness uses different expert backends. Consequently, matching the backbone alone does not establish that performance differences arise from the harness design.

A matched-tool comparison with at least one strong baseline would substantially strengthen the attribution. Both methods should have access to the same relevant perception backends and search services, with broadly comparable interaction budgets. Although Appendix F includes standalone SAM 3 outputs for selected examples, a benchmark-level comparison with a simple expert-tool pipeline would also help quantify how much of the counting and grounding improvement comes from the underlying detector rather than adaptive tool composition. These are possible focused controls, rather than a request to conduct both comparisons across the full benchmark suite.

W2. The memory ablation conflates access to visual evidence with persistent artifact reuse.

According to Appendix B.2, the textual-memory variant replaces tool-generated images with descriptions of up to 160 tokens and does not provide reusable image handles. The full system introduces both actual visual attachments and persistent, addressable images.

Thus, this comparison simultaneously changes whether the model can inspect the tool-generated image and whether later calls can reuse it. For a tool such as Zoom-in, withholding the actual cropped image already changes the tool’s basic functionality. The resulting improvement therefore cannot be attributed specifically to persistence and cross-tool reuse.

A more informative control would preserve visual tool observations while disabling persistent image handles and subsequent cross-tool image reuse. If visual presentation and reuse are instead intended as a single integrated mechanism, this is a defensible design choice, but the scope of the contribution should be stated accordingly. In particular, Appendix B.2 claims that each ablation step isolates exactly one mechanism’s marginal contribution, which is stronger than the current comparison supports. Similarly, the capability-layer ablation changes both tool descriptions and output processing; the main text appropriately acknowledges that their individual contributions are not separated, and the appendix should use equally qualified language.

W3. The ablation results contain numerical and protocol inconsistencies that affect the central evidence.

Section 3.3 reports that the final multimodal-memory step adds 3.1 and 4.5 points. However, Table 2 gives:

  • Qwen3-VL-8B: 47.14 − 44.93 = 2.21 points.
  • Qwen3-VL-32B: 52.51 − 49.12 = 3.39 points.

More importantly, Appendix B.2 describes evaluation on 22 datasets, with datasets larger than 500 examples subsampled to 500. Yet the full-system overall scores in Table 2 exactly match the 23-dataset results in Table 1, and its domain averages also match those results up to rounding.

The baseline row appears to follow a different convention. For example, the six search benchmarks in Table 1 average approximately 15.55 for Qwen3-VL-8B, while Table 2 reports 18.40. The latter matches the average obtained by excluding MMSearch-Plus. In contrast, the full-system search average of 33.10 matches all six search benchmarks. The baseline overall score of 37.06 also matches an equal-weight average across the four domains, whereas the full-system score of 47.14 matches the 23-dataset macro-average.

These observations do not establish whether the underlying experiments used different protocols or whether the discrepancies arose during reporting. Nevertheless, they must be reconciled before differences between ablation rows can be interpreted reliably. The exact dataset lists, evaluation subsets, and aggregation formula for each row are needed, ideally accompanied by the existing per-dataset results. If these are reporting errors, a corrected table reconstructed from the existing runs would be an appropriate response; a full rerun should not be necessary solely to resolve them.

W4. The main-paper presentation of Example 17 should reflect the process errors disclosed in the appendix.

Figure 3(a) uses Example 17 to illustrate geometric computation through camera poses, depth, grounding, and code. However, Appendix F explicitly states that the generated code misinterprets the bounding-box format and selects the wrong cabinet. It further notes that the prediction of 1.1957 m being close to the 1.1 m reference is partly coincidental and should not be interpreted as a successful measurement.

I appreciate the authors’ explicit disclosure of these errors. The concern is that the main text presents the example as explaining the system’s gains and describes the underlying artifacts as supporting exact 3D computation, without carrying over this qualification. The example should either be replaced with an already verified computation or explicitly presented as a case where a numerically close answer masks an incorrect process. A clarification in the main text and figure caption would adequately address this presentation issue; I do not consider a new process-level audit necessary during rebuttal.

W5. The methodological distinction from existing tool-use frameworks remains insufficiently established by the current evidence.

The paper appropriately acknowledges prior mechanisms for referenceable intermediate outputs, persistent executable state, and visual feedback. Its contribution appears primarily to be the integration of these ideas into a unified multimodal framework.

Such a systems contribution can be valuable without introducing a new learning algorithm. However, the evidence should clarify what the proposed organization enables beyond an existing agent with the same tools, persistent variables, and accessible image files. This concern follows from W1 and W2 rather than constituting a separate request for additional experiments: a focused controlled comparison, or a more carefully bounded account of the contribution, would help address it.

W6. The cost analysis is informative but does not measure end-to-end efficiency.

The main text and Appendix B.6 appropriately state that the reported costs cover model tokens and exclude tool execution and judging. Appendix B.6 also acknowledges that some search-task costs are lower bounds because cache-write usage was not fully recorded.

It would nevertheless be helpful to clarify explicitly whether the auxiliary Qwen3-VL-8B model used for search summarization and webpage extraction is included in the accounting. The reported results support conclusions about the measured model-token expenditure within the tested reasoning settings; end-to-end efficiency, including auxiliary inference, tool execution, and wall-clock latency, remains unmeasured. I regard this as a clarification of scope rather than a request for a new cost or latency study during rebuttal.

Questions for authors

1. Matched-tool comparison. Can you provide a focused comparison with at least one baseline using the same relevant tool backends, search access, and comparable interaction budgets? Why was OctoTools restricted to the Generalist Solution Generator?

I do not expect a full rerun across all 23 benchmarks during rebuttal. A small, representative matched-tool comparison targeting one central claim would be informative, provided that the shared tools, evaluation subset, and interaction budgets are clearly specified.

2. Ablation results and aggregation. What are the exact datasets, sample subsets, and aggregation rules used for each row of Table 2? Please reconcile the discrepancies with Table 1 and the gains stated in Section 3.3. If the differences are reporting errors, corrected results and a clear explanation based on the existing runs would be sufficient.

3. Scope of the memory ablation. In Appendix B.2 (lines 1318–1324), the textual-memory variant replaces tool-generated images with text descriptions and removes reusable image references, while the full system restores both visual attachments and addressable image handles (lines 373–374). This comparison therefore changes two factors simultaneously: whether the model can directly inspect visual tool outputs, and whether those outputs can be retained and reused in subsequent tool calls. The observed improvement does not isolate the contribution of persistence and reuse from that of access to visual evidence.

Could you provide a control that preserves visual tool observations but disables persistent image handles and subsequent cross-tool image reuse? Alternatively, if visual presentation and reuse are intended as a single integrated mechanism, could you clarify this scope and qualify the claim that each ablation step isolates an individual mechanism’s marginal contribution (lines 1298–1300)?

4. Qualitative example. Could the main text and Figure 3(a) explicitly acknowledge the process errors in Example 17, or use a verified replacement if one is already available? A presentation correction would be sufficient for this concern; I do not expect a new process-level audit during rebuttal.

5. Codex configuration. Was code execution operational in the Codex experiments? Table 22 describes it as unavailable, whereas Appendix G.1 describes execution in a read-only sandbox. Appendix B.7 also states that a 2,048-token per-response limit is shared with OpenVLHarness, while Appendix C.6 reports 12,800 tokens per model call for the latter. Please clarify the actual execution capabilities and token limits used in each comparison.

6. Specialization protocol. Were validation scores or trajectories available to the agents generating subsequent candidates, or used for final selection? The paper states that tool-generation data are disjoint from the standard evaluation splits (lines 2563–2564). Since Table 21 describes tools generated from FREAK-MCQ failures but the reported evaluation uses FREAK-VQA, could you clarify whether this separation is enforced at the image and underlying question-source levels, rather than only by benchmark entries?

Overall Assessment

I see practical value in the proposed framework and find the broad improvements over direct inference encouraging. The separation of model-readable evidence from reusable underlying artifacts is sensible, and the extensive experimental effort is a strength.

However, my recommendation is currently Weak Reject. The main obstacles are the lack of a matched-tool framework comparison, the confounding in the memory ablation, and the unresolved inconsistencies in Table 2. These issues directly affect whether the experiments support the paper’s central contribution. The qualitative example and configuration ambiguities are additional concerns, but they can largely be addressed through corrections and clarification.

I do not expect a full benchmark rerun or an extensive new experimental suite during rebuttal. My priority is a convincing reconciliation of the ablation results and a focused response to the attribution concerns. A small controlled comparison would be helpful where feasible; where further experiments are impractical, a clear account of what the existing evidence establishes, together with appropriately narrowed claims, would still be valuable.

My current recommendation is provisional. If the authors’ supplementary evidence, explanations, and corrections convincingly address these concerns, I would be happy to raise my score.

5 comments
AuthorsOpenReview ↗

Author Response (1/4)

We thank Reviewer ReFY for recognizing (1) the sensible system design for an important practical problem, (2) the broad evaluation with substantial improvements, (3) the useful analyses beyond benchmark scores, and (4) the useful research questions raised by the inference-budget and specialization studies.

Q1. The baselines use different perception tools and lack web search. Can you compare systems with the same tools and budgets? Why did OctoTools use only the Generalist Solution Generator?

A1. We evaluated Visual Sketchpad and OctoTools with the OpenVLHarness tools, including our perception backends and search services, on the same questions from the 23 Table 1 datasets:

SetupTool BackendsQwen3-VL-8BGPT-6 Luna
Visual SketchpadOriginal32.1542.78
Visual SketchpadOpenVLHarness31.9443.72
OctoToolsOriginal (Generalist)33.3140.59
OctoToolsOriginal (All Tools)34.7642.18
OctoToolsOpenVLHarness32.8543.03
OpenVLHarnessOpenVLHarness47.1455.76

For OctoTools, the Generalist Solver toolset is the most commonly reported configuration, since OctoTools selects additional tools separately for each task using validation performance. We have also added the full-toolkit comparison which slightly improves performance but causes extremely slow inference speeds.

These results show that access to the same perception backends and search services does not close the performance gap between baseline harnesses and OpenVLHarness.

The revised manuscript reports these results in Appendix B.8 (Table 17, lines 1890–1912), summarizes them in Section 3.2 (lines 328–330), and explains the OctoTools configuration in Appendix G.1 (lines 8881–8883).

Q2. The textual memory ablation changes whether the model can inspect the tool-generated image and whether later calls can reuse it. Could you provide a control that preserves visual tool observations but disables persistent image handles and subsequent cross-tool image reuse?

A2. We would like to clarify a potential misunderstanding: the textual memory reported in the second-to-last row of Table 2 is exactly what is requested by the reviewer. In this ablation setting, when a tool generates an image, the model receives the raw image and can inspect it directly in context. However, tool-generated images are only replaced by a VLM-written text description when stored in memory.

This setup was explained under textual memory implementation details in Appendix B.2 (original manuscript lines 1318–1320), although we now realize our original wording did not rule out alternative interpretations. We have added a sentence clarifying that the replacement applies only to persistent storage, not to the image initially returned to the model (lines 367-369).

Q3. Table 2 conflicts with Section 3.3 and Table 1. Which datasets, samples, and weights does each row use?

A3. Thank you for catching these inconsistencies. Table 2 uses the 22 datasets in Table 1 excluding MMSearch-Plus. All variants use identical seed-42 subsets capped at 500 examples per dataset, and results are averaged over three independent runs. Each domain score is the mean across its datasets, and Overall equally averages the four domain scores. Table 1 instead reports the mean across all 23 datasets.

While updating Table 1 to include MMSearch-Plus during final manuscript preparation, summary values from the main experiments were inadvertently copied into the baseline and full-system rows of Table 2. We have corrected both rows using the saved outputs from the ablation experiments. The corrected table is provided below.

VariantCounting and groundingSearch and deep researchGeneral VQA and hallucinationSpatial understandingOverall
Model8B / 32B8B / 32B8B / 32B8B / 32B8B / 32B
Baseline (no tool)27.65 / 31.5118.72 / 22.9057.46 / 61.9544.88 / 48.0237.18 / 41.10
+ Tool41.70 / 43.3021.44 / 24.8952.41 / 61.4147.83 / 52.8440.84 / 45.61
+ Capability layer46.73 / 46.2025.97 / 32.7456.05 / 63.6348.16 / 52.6344.23 / 48.80
+ Textual memory47.27 / 45.7426.92 / 34.4357.54 / 64.8447.98 / 51.4644.93 / 49.12
Full OpenVLHarness46.97 / 51.0537.89 / 40.8859.90 / 67.1350.14 / 55.2148.73 / 53.57

The corrected multimodal-memory gains are +3.80/+4.45. We have verified these results against the original outputs. In the revised manuscript, we have corrected the values of Table 2 and the gains in Section 3.3 (lines 371–372), and stated the datasets and weighting in the Table 2 caption (lines 389–390) and Appendix B.2 (lines 1401–1403).

AuthorsOpenReview ↗

Author Response (2/4)

Q4. Figure 3(a) presents Example 17 as a successful measurement despite the errors in Appendix F indicating that this might be a coincidence. Can you correct the main text and caption or replace the example?

A4. We thank the reviewer for the careful reading. Example 17 shows the intended computation pipeline (camera poses → depth → grounding → code), but as Appendix F reports, the final code reads the boxes in the wrong format and measures the wrong cabinet, so the close answer is partly coincidental. We have revised Figure 3(a) caption to state the error. For a case where the same pipeline works as intended, see Example 16 in Appendix F: on another SparBench question, the agent combines camera poses and depth in code and the final answer of 2.4 m matches the reported reference.

Q5. The distinction from existing frameworks with the same tools and accessible intermediate results (persistent variables, and accessible images) is unclear.

A5. The main distinction between our framework and existing frameworks with the same tools is (1) Capability abstraction without artificial tool scaffolding or pipelining, and (2) Illustrative + exact evidence representation with automatic environment updates.

  1. Capability abstraction without artificial tool scaffolding or pipelining. Visual Sketchpad’s sliding-window detector prescribes a fixed 16-crop routine around its zoom-in and grounding tools; OctoTools chains planning, command generation, and verification into hard-coded pipelines. Such hand-designed procedures work on in-domain tasks, where humans have verified that the procedure succeeds, but they may constrain the model’s ability to solve harder or out-of-domain tasks creatively. Our core capabilities layer exposes only fundamental perception, search, and computation capabilities, letting the model create its own compositions with the coding agent tool. This design enables more flexible and adaptable tool-use behaviors as described in Related Work (lines 456–460 of the original manuscript).

  2. Illustrative + exact evidence representation with automatic environment updates. In prior harnesses, each tool call often returns either just a lossy visual illustration or an implicitly stored global variable name. This causes the orchestrator to commonly lose track of available artifacts and their exact data structures in longer interactions. In OpenVLHarness, we not only provide both high quality illustrative and exact evidence representations, but also provide an explicit environment update indicating latest artifact inventory with data structure information. This design aims to avoid model confusion in longer interactions.

The table below shows the difference across harnesses.

FrameworkCapability abstractionDual evidence representationAutomatic environment update
Visual Sketchpad✗◐✗
OctoTools◐◐✗
OpenVLHarness✓✓✓

✓ Built-in support · ◐ Partial or tool-specific support · ✗ No supported mechanism.

The experiment in A1 where all harnesses call the same backend services shows that the performance gap persists when tool backends and search access are matched. In this setting, OpenVLHarness scores 12.04–15.20 points higher than both baselines.

Empirically, harnesses with artificial tool scaffolding or pipelining tend to perform poorly on newly released datasets beyond the ones they are built for. We share results on PerceptionBench [1], a recently released challenging perception benchmark:

HarnessQwen3-VL-8BGPT-6 Luna
Base (no tool)21.550.1
Visual Sketchpad22.148.9
OctoTools21.051.7
OpenVLHarness30.256.1
AuthorsOpenReview ↗

Author Response (3/4)

Q6. Does the cost analysis include Qwen3-VL-8B search inference or measure end-to-end efficiency?

A6. We thank the reviewer for raising this. We have now estimated the end-to-end cost for the 12 benchmarks of Table 12 and added it to the model-token cost. The tools fall into two groups:

  • Locally served GPU tools: SAM 3 (Visual Grounding), Depth Anything 3 (Depth Estimation and Camera Trajectory) and PaddleOCR-VL (OCR).
  • Search tools billed through the Serper API and auxiliary local model to summarize search results: Text Search, Image Search and Webpage Visit.

Specifically, we count every tool call in the Table 12 trajectories, and time each request on one RTX A6000 to estimate the total GPU hours. We then convert GPU time to dollars at \0.33perA6000−hour,followingthepriceatRunPodCommunityCloud.ForSerper,atextsearchorImageSearchrequestcosts1creditandawebpagescrapecosts2.WepricecreditsatSerper’svolumerateof0.33 per A6000-hour, following the price at RunPod Community Cloud. For Serper, a text search or Image Search request costs 1 credit and a webpage scrape costs 2. We price credits at Serper’s volume rate of \\0.30 per 1,000. The three search tools also use a local Qwen3-VL-8B to extract evidence from search results and webpages. To price it, we replayed 100 search calls from these trajectories (40 text searches, 30 image searches, and 30 webpage visits) on one RTX A6000 serving 8 requests at a time, as in our deployment. Each extraction request takes 1.39 GPU-seconds, which we price at the same \$0.33 per A6000-hour and scale by each domain's search usage. For Visual Sketchpad, we time its GPU tools the same way. These are GroundingDINO detection (including its 16-crop sliding-window variant), SoM segmentation and Depth Anything. Codex runs its code in a CPU sandbox, so its cost is model tokens only. We report the cost per 1,000 questions below.

DomainOurs: tokensOurs: GPU toolsOurs: Serper APIOurs: aux. modelOurs: totalCodexVisual SketchpadVisual Sketchpad + GPU tools
Perception$1.03$0.033$0.000$0.000$1.07 (1×)1.52×1.20–4.27×1.28–4.35×
Search & DR$1.07$0.006$0.494$0.172$1.74 (1×)0.84×0.58–1.81×0.59–1.82×
General VQA$0.71$0.019$0.140$0.049$0.92 (1×)1.29×0.49–2.67×0.53–2.70×
Spatial$1.04$0.133$0.000$0.000$1.18 (1×)1.34×1.01–3.95×1.04–3.98×
Average$0.96$0.048$0.159$0.055$1.22 (1×)1.25×0.82–3.18×0.86–3.21×

Including tools, Ours costs less than Codex on average (Codex is 1.25×). Ours sits at the low end of Visual Sketchpad’s range (0.86–3.21× with its GPU tools). Across the 12 benchmarks, Ours scores 62.85 on average, against 48.31 for Codex and 49.50 for Visual Sketchpad. We have added this analysis and the measurement details to Appendix B.6 of the revised manuscript (lines 1724–1727 and 1782–1808, Table 15).

It's worth noting that our harness does not introduce additional page-reading or extraction service fees such as Jina Reader. Webpage text comes from the Serper scrape priced above, and each local tool model fits on a single RTX A6000.

Q7. Could Codex execute code, and what token limits did the comparisons use?

A7. Thank you for identifying these text ambiguities. We would like to clarify:

1. Code execution was operational for Codex.

Codex could execute model-written code in a sandbox, while shell-based access to our vision expert tools was unavailable. This distinction is described in Appendix B.7 (p. 33, lines 1738–1740) and Appendix G.1 (p. 163, lines 8786–8787).

In Table 22, “unavailable” describes the restriction on shell-based access to vision expert tools. We adjusted the entry to “Code execution (no vision expert tools)” to avoid any confusion in the camera ready version.

Shell-based access to vision expert tools was disabled because enabling them reduced empirical performance (averaged performance on the 22 Table 1 datasets, 100 random examples per dataset):

BackboneCode executionw/o Vision Expert Toolsw/ Vision Expert Tools
GPT-6 LunaEnabled41.8540.06

2. Both OpenVLHarness and Codex used 12,800 token limit cap.

We have carefully examined our experiment logs and confirmed that both Codex and our method used the same 12,800 token limit cap. We have adjusted the stale descriptions in B.7/G.1 and clarified C.6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.