- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OpenVLHarness is a training-free system for visual tool use. A capability layer standardizes tool interfaces and renders heterogeneous outputs into model-readable observations. A per-query artifact registry retains images, structured data, and web references for later calls, including computations delegated to a Python coding agent using the same backbone. Evaluation spans 23 benchmarks and six principal backbones, with additional comparisons against fine-tuned agents. The paper also studies reasoning budgets, diagnoses failed trajectories, and explores specialization through instruction edits and generated tools.
Critical strengths
-
The empirical breadth and reasoning-budget study provide convincing evidence of practical utility. Improvements extend across model families and task domains: the 23-benchmark average rises from 35.99 to 47.14 with Qwen3-VL-8B and from 54.26 to 65.48 with GPT-6 Sol (Table 1). Three-run Qwen3 results strengthen this evidence; the full-system average standard deviations are 0.35 and 0.24 for the two scales (Appendix H). Table 12 also tests whether additional internal reasoning can recover the gains. On 12 benchmarks, higher-budget direct inference reaches 51.53 versus 62.85 for the harness, despite 1.27× average backbone-token spending. Within the reported accounting scope, this supports the value of acquiring task-relevant evidence beyond extending reasoning alone. The breadth of the gains and this complementary analysis are central to my positive assessment.
-
The design supports evidence composition and practical adaptation without weight updates. Retaining masks and arrays alongside readable renderings lets generated code compose tool results directly; the backpack trajectory also shows reuse across cropping, image search, and source verification (Example 8). The paper connects diagnosis to adaptation: a critic analyzes 9,980 failures, with 92% agreement on 100 human-annotated samples, and two workflows improve performance without updating backbone weights (Sections 3.4–3.5). Harness search adds 4.1 points on ERQA, while generated tools add 8.2 on SparBench; Appendix E.3 specifies disjoint adaptation and evaluation splits. These experiments give a concrete path for adapting the harness to particular tasks and strengthen its value as a reusable system.
Critical weaknesses
The interpretation and reporting of the memory ablation need clarification. Table 2 gives full-minus-textual differences of 2.21/3.39 points, whereas Section 3.3 states 3.1/4.5. Appendix B.2 specifies 22 datasets and subsampling, while the full row reproduces the 23-dataset main-table averages; the earlier Overall entries also appear to use equal-domain averaging. Please reconcile these figures and specify a consistent evaluation population and aggregation rule. There is also an interpretation issue: textual memory replaces tool-generated images with descriptions of at most 160 tokens, while the full variant restores both native images and reusable handles. This is a meaningful test of the combined multimodal-memory design, but it does not establish how much comes from visual access versus subsequent reuse. I would read it as evidence for the combined mechanism. Section 3.3 already acknowledges that schemas and rendering are evaluated jointly; a similarly explicit interpretation here would make the component claims easier to assess.
Questions for authors
- Reporting clarification: Which dataset membership, sample counts, and Overall weighting apply to each row of Table 2? How should the differences between the table, Section 3.3, and Appendix B.2 be reconciled? This matters for assessing the magnitude of the component gains.
- Memory interpretation: Is the final ablation intended to measure the joint benefit of native visual evidence and persistent image references? Making that scope explicit would clarify the conclusion. Separating these effects would be useful follow-up work, rather than a prerequisite for assessing the complete system.
- Cost clarification: Appendix B.6 explicitly excludes tool execution, and three search-task cost estimates are lower bounds. Can you clarify the unrecorded cache-write costs and estimate the excluded perception, search, and auxiliary-model costs? Representative end-to-end latency would also help assess practical efficiency beyond model-token spending.
3 comments
Author Response (2/2)
Q3. Can you clarify the end-to-end latency and cost including tool execution?
A3. We thank the reviewer for raising this. We have now estimated the end-to-end cost for the 12 benchmarks of Table 12 and added it to the model-token cost. The tools fall into two groups:
- Locally served GPU tools: SAM 3 (Visual Grounding), Depth Anything 3 (Depth Estimation and Camera Trajectory) and PaddleOCR-VL (OCR).
- Search tools billed through the Serper API and auxiliary local model to summarize search results: Text Search, Image Search and Webpage Visit.
Specifically, we count every tool call in the Table 12 trajectories, and time each request on one RTX A6000 to estimate the total GPU hours. We then convert GPU time to dollars at \0.30 per 1,000. The three search tools also use a local Qwen3-VL-8B to extract evidence from search results and webpages. To price it, we replayed 100 search calls from these trajectories (40 text searches, 30 image searches, and 30 webpage visits) on one RTX A6000 serving 8 requests at a time, as in our deployment. Each extraction request takes 1.39 GPU-seconds, which we price at the same \$0.33 per A6000-hour and scale by each domain's search usage. For Visual Sketchpad, we time its GPU tools the same way. These are GroundingDINO detection (including its 16-crop sliding-window variant), SoM segmentation and Depth Anything. Codex runs its code in a CPU sandbox, so its cost is model tokens only. We report the cost per 1,000 questions below.
| Domain | Ours: tokens | Ours: GPU tools | Ours: Serper API | Ours: aux. model | Ours: total | Codex | Visual Sketchpad | Visual Sketchpad + GPU tools |
|---|---|---|---|---|---|---|---|---|
| Perception | $1.03 | $0.033 | $0.000 | $0.000 | $1.07 (1×) | 1.52× | 1.20–4.27× | 1.28–4.35× |
| Search & DR | $1.07 | $0.006 | $0.494 | $0.172 | $1.74 (1×) | 0.84× | 0.58–1.81× | 0.59–1.82× |
| General VQA | $0.71 | $0.019 | $0.140 | $0.049 | $0.92 (1×) | 1.29× | 0.49–2.67× | 0.53–2.70× |
| Spatial | $1.04 | $0.133 | $0.000 | $0.000 | $1.18 (1×) | 1.34× | 1.01–3.95× | 1.04–3.98× |
| Average | $0.96 | $0.048 | $0.159 | $0.055 | $1.22 (1×) | 1.25× | 0.82–3.18× | 0.86–3.21× |
Including tools, Ours costs less than Codex on average (Codex is 1.25×). Ours sits at the low end of Visual Sketchpad’s range (0.86–3.21× with its GPU tools). Across the 12 benchmarks, Ours scores 62.85 on average, against 48.31 for Codex and 49.50 for Visual Sketchpad. We have added this analysis and the measurement details to Appendix B.6 of the revised manuscript (lines 1724–1727 and 1782–1808, Table 15).
It's worth noting that our harness does not introduce additional page-reading or extraction service fees such as Jina Reader. Webpage text comes from the Serper scrape priced above, and each local tool model fits on a single RTX A6000.
Cache writes. Table 13 leaves the cache-write cells blank for RealX-Bench, FVQA, and BC-VL because the usage saved for these runs recorded cache reads but did not separate cache writes from uncached input. The input that is not a cache read (3,610 / 2,857 / 5,204 tokens per question) was therefore listed entirely as uncached input and priced at the uncached rate, which is why Appendix B.6 calls these costs a lower bound. To recover the cache-write costs, we re-ran these three datasets with per-call logging. Almost all of this input is cache writes, with fewer than 10 uncached tokens per question, as on the other nine datasets. Priced at the cache-write rate:
| Dataset | Ours $/1k | Cost-matched Base cost | Base score | Ours score |
|---|---|---|---|---|
| RealX-Bench | 0.88 | 1.35× | 22.68 | 40.72 |
| FVQA | 0.75 | 1.33× | 39.33 | 63.33 |
| BC-VL | 1.57 | 1.04× | 37.59 | 50.13 |
| Average (12 datasets) | 1.23× | 51.53 | 62.85 |
These costs are 9–11% higher than the lower bounds in Table 12 (\0.68, and \$1.44). Cost-matched Base still spends more than Ours on all three datasets and scores 12.5–24.0 points lower. The cost table above already uses these token costs. We have filled in these entries in Table 14 (formerly Table 13, lines 1742–1748; caption lines 1778–1780) in the revised manuscript.
Thanks for authors' response which addresses my concerns. I will maintain my positive rating.
Author Response (1/2)
We thank Reviewer ihKp for recognizing (1) the empirical breadth and reasoning-budget study as convincing evidence of practical utility, and (2) a design that supports evidence composition and practical adaptation without weight updates.
A1. We apologize for the confusion. We would like to clarify that Table 2 uses the 22 datasets in Table 1 excluding MMSearch-Plus. All variants use identical seed-42 subsets capped at 500 examples per dataset, and results are averaged over three independent runs. Each domain score is the mean across its datasets, and Overall equally averages the four domain scores. Table 1 instead reports the mean across all 23 datasets.
While updating Table 1 to include MMSearch-Plus during final manuscript preparation, summary values from the main experiments were inadvertently copied into the baseline and full-system rows of Table 2. We have corrected both rows using the saved outputs from the ablation experiments. The corrected table is provided below.
The corrected multimodal-memory gains are +3.80/+4.45. We have verified these results against the original outputs. In the revised manuscript, we have corrected the values of Table 2 and the gains in Section 3.3 (lines 371–372), and stated the datasets and weighting in the Table 2 caption (lines 389–390) and Appendix B.2 (lines 1401–1403).
A2. We would like to clarify that the textual memory reported in the second-to-last row of Table 2 is exactly what is requested by the reviewer. When a tool generates an image, the model receives the image in its context and can inspect it directly (native visual evidence). In this ablation setting, persistent memory stores only a text description (thus model does not have access to persistent image references). Later tools cannot retrieve the original image through that reference. We have revised Appendix B.2 (lines 1366–1376 of the revised manuscript) to make clear when the model sees the image and what the harness retains for later calls.