acceptodds
Under review as a conference paper at ICLR 2027

Visual Reasoning at the Harness Level: A Ladder for Controlled Evaluation

Abstract

Advances in foundation models are shifting coding and working agents from predefined tools toward general computational environments. Yet existing visual agents often couple model, harness, and training choices, making it difficult to determine how this transition affects visual reasoning. In this work, we introduce a unified evaluation framework to study how harness design allocates responsibility between the model and its environment. We instantiate this framework as a seven-rung harness ladder organized around expanding computational access and delegating state and evidence management. The ladder connects direct answering, predefined visual tools, generated code, and general workspaces under fixed models and tasks. We further construct (Visual Harness with Stratified Tasks for Agentic Reasoning), a benchmark of 600 tasks across five visual domains and four calibrated difficulty levels. Experiments across nine models show that code execution with a persistent kernel improves all four full-ladder models over fixed tools, whereas further workspace delegation offers no consistent advantage. Models with similar direct-answer performance benefit differently from computational access, motivating harness designs that expand computational freedom while retaining appropriate support for state and evidence. abstract

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual reasoning, visual agent, harness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wwVmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper compares harness configurations for visual reasoning while keeping the model and the tasks fixed. The configurations form a ladder that moves from direct answering, through predefined visual tools and generated code with a persistent kernel, to general workspaces where the model manages files and runs shell commands. The authors evaluate several models on VHStar, a new benchmark of open-ended image questions drawn from public sources and stratified by domain and difficulty. They report that code with a persistent kernel improves every model over fixed tools, while moving on to general workspaces gives uneven and often lower scores.

Critical strengths

Moving from fixed visual tools to a code environment helps every model the paper tests. Adding code execution with a persistent kernel to the fixed-tool setup raises all full-ladder models on the same tasks, and the gain holds after removing the operation menu or doubling the step budget, as in Table 10.

Critical weaknesses
  1. I am not convinced that the lower scores of the workspace rungs come from workspace delegation itself rather than from how these rungs are implemented. Each step after L3 changes several things at once, such as how images come back to the model, file storage and how answers are submitted, and L6 also enforces an access policy that is described only in Appendix E. Permission rejections hit almost half of the L6 tasks for GLM (Table 21), yet the scores are not split by whether a rejection happened.

On top of this, every result comes from one in-house harness that. It seems that the authors did not reproduce any existing harness system like opencode, and none of the code, prompts, tool descriptions, permission rules or judge prompt is released. So the current analysis results might not be generalizable across current harness systems, instead, is more reliable on the sketchy harness provided. Therefore it might be hard to generalize the conclusion to existing workspace harness systems.

  1. From analysis, it is unclear to me that the gain from L2 to L3 can be credited to the persistent kernel, as the Abstract does. That step adds code execution and kernel persistence together, as Appendix C.1 itself notes, and the one rung that changes kernel persistence alone, L4 to L5, barely moves the scores (Table 10). Most of the gain may simply come from being able to run code.
Questions for authors
  1. Which operations does the L6 permission policy block, for example the overlay inspections in the GLM case in Appendix E? What is the gap between L3 and L6 on the tasks where no rejection occurred?

  2. Appendix B.3 says the saved evaluation configuration includes the exact prompt. Can you release the full system prompt, tool descriptions, permission rules, submission rules and judge prompt for each rung, together with the harness code? Apart from the tool descriptions, is the prompt identical across rungs?

  3. If L6 is replaced by an existing workspace agent such as Opencode, or by a second independent implementation, does L3 still beat L6 on at least two of the full-ladder models?

  4. Can you report configurations that change one design axis at a time, for example L3 with explicit image reading added, or L4 with automatic image return?

2 comments
AuthorsOpenReview ↗

Response to Reviewer wwVm

We sincerely thank the reviewer for the careful and thorough review. We greatly appreciate your valuable questions and insightful comments on our work. We address your concerns below.

W1 & Q1. Detailed implementation and access policy

We thank the reviewer for the insightful suggestion. All six interactive configurations (L1–L6) are implemented on OpenCode v1.18.9 [1]. L0 is the direct-answer baseline. L6 is built entirely on OpenCode, but to keep the experiments fair and stable, we disable several additional OpenCode features, including planning, subagents, web search, and access to files outside the workspace. This lets the model perform vision-centric tasks in a stable sandbox using the supplied inputs, which are sufficient to solve the tasks. Disabling planning and subagent modes also helps us evaluate models under the same step budget. These settings standardize the evaluation conditions while preserving local computation and file access. The model can run Python through the shell, save an image in the task workspace, and inspect it with read. We implement L1–L6 on the OpenCode runtime by disabling file reads and writes, providing a kernel-based Python tool, or providing predefined visual tools designed by humans, depending on the specific level. We provide an anonymous link to our implementation in the following section so that reviewers can inspect it.

For GLM-5.3-Flash, we check the permission rejections and find that almost all arise from attempts to access external directories. The remaining rejections all result from web search tool calls, which occur in 1.4% of all cases. The intended directory for task files is /workspace/work, but GLM-5.3-Flash sometimes attempts to run commands or access files under common directories such as /data or /code. These paths lie outside the permitted /workspace directory, so the requests trigger the harness’s external-directory restriction. After each rejection, the agent returns to the designated sandbox directory and continues the task. We suspect that the model retains this behavior from post-training with a particular harness and starting directory. We do not observe this pattern in any of the other models. The table below compares L3 and L6 on the same tasks for which GLM records no L6 permission rejection.

ModelTasks without an L6 rejectionL3 score (%)L6 score (%)L3 − L6 (pp)
GLM-5.3-Flash31155.154.3+0.8

L3 still slightly outperforms L6 on tasks without permission rejections. This also shows that individual permission rejections do not substantially affect accuracy, as the model typically recovers quickly.

W1 & Q2. Code, prompts, and scoring rules

We thank the reviewer for requesting these materials. We release the code for all seven configurations, including the L0-L6 system prompts, the tool definitions, the permission and submission rules, and the exact answer-equivalence judge prompt. We also provide a configuration guide to support reproducibility. Anonymous repository

Most of the prompt text is shared across levels. The differences beyond tool descriptions explain how to use the available tools and submit the answer. For example, L3 tells the model to display Python figures and submit its answer in the final response (as it cannot write files to the sandbox directory). At L6, the prompt tells it to read saved images and write its answer to a specific file, with the final response as a fallback. These instructions reflect each level’s tool and submission requirements. For fair comparison, the VH-Star task prompts and the scoring rule for each task are the same across levels.

AuthorsOpenReview ↗

Response to Reviewer wwVm - Part 2

W1 & Q3. Performance with other harnesses

We fully understand the reviewer’s concern. A direct comparison between our OpenCode-based L3 and a complete harness such as Pi [2] also reflects differences in runtime prompts and system-level tools. Since the original L3 and L6 already use OpenCode, we implement both levels on Pi and evaluate them on the full benchmark. This allows us to compare L3 and L6 within each runtime. The results are shown below.

ModelRuntimeL3 score (%)L6 score (%)L3 − L6 (pp)
DeepSeek-V4.1OpenCode55.151.4+3.7
DeepSeek-V4.1Pi53.850.7+3.1
Qwen3.8-Flash-NextOpenCode58.554.5+4.0
Qwen3.8-Flash-NextPi58.854.5+4.3

L3 outperforms L6 for both models on both OpenCode and Pi, with similar score gaps.

W2 & Q4. Isolating the effects of kernel persistence and image return

We thank the reviewer for the insightful feedback. We first clarify that code execution and the kernel are introduced together at L3. Our L0–L6 ladder follows the development of visual agentic reasoning, from direct perception to simple code-based visual reasoning (PyVision [3], L3), and then to the current harness-based paradigm (OpenCode [1], L6). L6 typically uses a terminal tool to write and execute code, while earlier L3-style designs use an isolated Jupyter kernel to execute code and retain intermediate state. L2–L3 therefore measures the combined effect of code execution and kernel persistence. Similarly, L3–L4 introduces writable storage and explicit image reading together. In our implementation, the model must save a generated image to the workspace before reading it explicitly.

To examine kernel persistence, we compare L4 with L5, which starts a fresh Python interpreter for each call. Both configurations provide the same visual tools, writable files, explicit image reads, and submission rule. L4 scores 1.1–2.4 percentage points higher across the four models (Table 2), showing the benefit of kernel persistence when file storage is also available.

To examine image delivery, we add automatic image return to L4 while keeping the persistent kernel, file access, fixed visual tools, and submission rule the same. Both configurations are evaluated on the same 600 tasks. The results are shown below.

ModelL4 score (%)L4 + automatic image return score (%)Gain over L4 (pp)
DeepSeek-V4.149.854.9+5.1
Qwen3.8-Flash-Next59.661.8+2.2

Adding automatic image return improves scores by 5.1 percentage points for DeepSeek and 2.2 points for Qwen. These gains show that immediate visual feedback provides an additional benefit when code execution and kernel persistence are already available.

References

[1] OpenCode

[2] Pi

[3] PyVision: Agentic Vision with Dynamic Tooling

Reviewer_vQiGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors present a comprehensive, unified, graded analysis framework for studying how different visual harness designs affect model performance on visual environments and tasks. They also build VHStar, an evaluation benchmark covering multiple domains, aiming to provide a comprehensive analysis perspective. The study yields several interesting findings: a moderate degree of freedom and code execution improve model performance. But more freedom is not always better. Granting additional privileges such as state management does not necessarily bring gains. Moreover, the reasoning ability of models and the ability to exploit a harness appear to be decoupled.

Critical strengths
  • S1. Broad experimental coverage and fairly systematic analysis. The paper evaluates 9 models, builds VHStar by integrating 12 existing datasets across 5 domains. Beyond overall performance, it provides per-domain analyses, ablations, and token-efficiency analyses, making the expeirments quite comprehensive.
  • S2. Several valuable and insightful findings. For example, harness freedom is not always benificial; and the visual reasoning ability and the harness-utilization ability do not align. These findings offer useful reference for understanding agentic capabilities and designing effective harnesses.
  • S3. A clear and general harness evaluation framework. The paper organizes harness design along four dimensions (action, persistent state, observation, and control). They bring direct answering, fixed tools, code executaion, and a general-purpose workspace into a single unified comparison, providing a reusable analytical perspective for future harness design and evaluation.
Critical weaknesses
  • W1. The rationale for the seven-level harness division is insufficient. The proposed seven levels are not an established standard, and the paper does not adequately justify why the levels are in this way. More importantly, the conclusions may be confounded by the effectiveness and correctness of the harness implementations themselves.
  • W2. The conclusion that greater autonomy does not necessarily improve performance may be limited by the harness implementations and task selection. Performance differences across L3–L6 only show that the specific chosen harnesses perform differently on these tasks; they do not yet establish that harness with more freedom brings no benefit. In particular, VHStar consists mainly of static image QA tasks and may lack tasks that require file mangement and long-term state maintenance.
  • W3. The distinctions from prior work and the incremental contributions need to be clarified. Prior works such as PyVision and SpatialClaw have already studied the effects of code execuation and harness design in visual reasoning, and WildClawBench has compared different harnesses under fixed models and tasks. The paper's unique research questions and new findings relative to these works could be highlighted more prominently.
Questions for authors
  • Q1. What is the basis for the seven-level harness ladder? Is the grading grounded in an existing consensous in the field, and could the analysis instead be built on harnesses already proven effective?
  • Q2. Does the lack of further gains from L3 to L6 mainly reflect a mismatch between the specfic harness designs and the tasks? For example, moving from L3 to L6 not only changes autonomy but also removes the persistent kernel and predefined tools.
  • Q3. Are harnesses gains related to model scale? The paper shows clear differences across models and across generations at the same parameter scale. Could models of different sizes within the same family be used as controls to disentangle the effects of scale from those of training differences?
5 comments
AuthorsOpenReview ↗

Response to Reviewer vQiG

We sincerely thank the reviewers for their careful and thorough review. We greatly appreciate your valuable questions and insightful comments on our work. We will attempt to address your concerns below.

W1 & Q1. Rationale and Motivation for seven-level harness ladder design.

We thank the reviewers for the insightful suggestions. Our ladder design follows the development of modern agentic visual reasoning, from direct perception (LLaVA [1]), through predefined visual actions (Mini-o3 [2]) and model-written visual programs (PyVision [3]), to the general-purpose workspaces used by modern agent harnesses such as OpenCode [4] and Claude Code [5]. L3 and L6 represent two paradigms along this trajectory: traditional code-based thinking with images (L3, PyVision-style) and harness-based agentic visual reasoning (L6, OpenCode-style). At L3, the model writes Python in a persistent kernel and automatically receives the figures it displays. At L6, the model works through a shell and a persistent filesystem, choosing which intermediate results to save, reopen, and inspect.

We design the seven ladder configurations to make these transitions explicit. L0 answers directly from the input images, L1 adds zoom, a canonical example of a human-predefined visual tool, and L2 expands the tool menu. At L3, the model can write its own visual tools in Python. L4 retains the kernel and tool menu while introducing writable files and explicit image reads. L5 removes kernel persistence while preserving files across calls. Finally, L6 replaces the predefined visual tools and Python interface with bash and read. L4 and L5 thus let us examine the design choices connecting the two paradigms. We implement all six ladder designs (except L0) based on OpenCode's core runtime.

We agree that the comparison depends on how each configuration is implemented. Therefore, we check and release the implementation in an anonymous code repository, including the prompts, tool definitions, permissions, image-return policies, and answer-submission rules for each configuration. Anonymous code repository.

W2 & Q2. How representative are VHStar’s tasks, and how broadly do the findings generalize?

We thank the reviewer for asking how task selection affects our conclusions. First, we indeed focus on vision-centric tasks that require models to recognize fine details or judge spatial relationships in images. These tasks are traditionally viewed as primarily perceptual. Given the success of general-purpose harnesses in coding and other multi-step work [6, 7], we ask whether the same approach can support a range of visual reasoning tasks and where models may struggle in such a paradigm. Our results show that modern harnesses can improve advanced models’ performance on vision-centric tasks. However, the most effective harness design may differ from that for coding and other agent tasks.

Second, the reviewer asks whether VHStar’s emphasis on vision-centric questions leaves little need for file management or long-term state maintenance. This is a reasonable expectation for questions about static images. In our experiments, however, we discover that models can also save intermediate visual results to files and reuse them in later steps of the same task. For example, one model reloads a pixel-level score map from a .npy file to count matching shapes, while another model reopens a difference image to identify a changed grid cell. We present several case studies from the original trajectories to help reviewers trace how models generate, save, and reuse visual evidence across steps. These cases also illustrate several visual reasoning skills that models exhibit through code. Anonymous case studies. To complement these case studies, we analyze how often models explicitly read and reread derived images across 300 trajectories from two models on the same 150 tasks.

Model (L4)Runs that open a derived image / all runsRuns that reread identical image content / runs that open a derived image
Qwen3.8-Flash-Next115/150 (76.7%)0/115 (0%)
DeepSeek-V4.1-Flash135/150 (90.0%)1/135 (0.7%)
Overall250/300 (83.3%)1/250 (0.4%)

As we can see, models open a derived image in 250 of 300 runs (83.3%). Only one of those runs (0.4%) includes a second read of the same image content at a later step. The model can still see the image in context after the first read. Together, the cases and the frequency of these file reads show that the two advanced models we study can create visual artifacts and manage intermediate state even on vision-centric tasks.

AuthorsOpenReview ↗

Response to Reviewer vQiG - Part 2

W2 & Q2. follow-up

Additionally, one possible reason for why L3 outperforms L6 is how each harness handles intermediate results. L3 keeps Python variables in memory across calls and automatically adds displayed figures to the conversation. This adds overhead to the harness and uses additional visual tokens in the model’s context, but the model can reuse variables and inspect figures without reopening files. At L6, the model decides what to save and which files to reopen. In coding tasks, agents can reread notes and earlier code to guide their next decision [6]. In vision-centric tasks, the evidence may be spread across images, masks, and numerical arrays. Current models may find it harder to identify which result to revisit and use it to check their answer. Future work may improve L6 performance by training models to revisit relevant visual evidence or by designing harnesses that make multimodal evidence easier to retrieve.

W3. What does this study add beyond prior work?

We thank the reviewer for this important question and we want to provide a clarification below. As for previous studies, PyVision [3] evaluates model-written Python tools, and PyVision-RL [8] trains models to use them. Both use their own frameworks, so it remains unclear whether the gains transfer to other harnesses. SpatialClaw [9] uses code to solve spatial reasoning tasks. Its harness and geometry tools are tailored to that domain. WildClawBench [7] compares agent harnesses on long-horizon work tasks that include multimodal files. Its focus on workflow completion makes it difficult to isolate visual reasoning performance. Compared to these studies, VHStar focuses on tasks where solving the problem depends primarily on visual perception. We design VHStar as a benchmark for vision-centric tasks across five domains and four difficulty levels. We build a unified evaluation framework based on OpenCode to measure the effect of harness design with the model held fixed and to compare model capabilities under the same harness. As a result, we can provide the community with fair comparisons and broadly applicable experimental findings, which may be useful for future studies.

Based on this motivation and design, we find several patterns that these studies do not report. First, moving from predefined visual tools (L2) to model-written programs in a persistent Python kernel (L3) improves scores by 5.9–10.0 percentage points across all four models. This demonstrates the value of allowing advanced models to construct their own visual tools. Second, we find that giving the models more control over the workspace does not consistently improve visual reasoning scores further, which may be attributed to remaining performance bottlenecks in current models or harnesses. Third, we observe the emergence of zero-shot harness-based visual reasoning capabilities across model generations at the same parameter count. Specifically, Qwen3.5-, 3.6-, and 3.8-27B submit code containing a recognized operation on 6.5%, 20.2%, and 67.7% of L6 tasks, respectively (Table 14). The first two models score 8.8 and 0.3 points below direct answering at L6, while Qwen3.8-27B scores 17.1 points above it (Table 3). This also demonstrates that only the latest advanced models begin to exhibit zero-shot harness-based visual reasoning capabilities. Fourth, removing text feedback from tools lowers scores for all four models at both L3 and L6. Removing derived images also lowers scores for all four at L3 (Figure 3). This demonstrates that harness-based agentic visual reasoning relies not only on additional visual perception through derived images but also on useful statistics computed from visual data through code. In addition, we report further useful findings from broader model evaluations, efficiency evaluations, and analyses of code-based visual capabilities.

AuthorsOpenReview ↗

Response to Reviewer vQiG - Part 3

Q3. Can model scale be separated from training differences?

We thank the reviewer for asking how model scale and training differences contribute to the gains from a harness. We evaluate each Qwen model on the same VHStar tasks with direct answering and with the L6 workspace. This allows us to compare three model generations at a fixed size of 27B and two variants within the Qwen3.8 generation. We report these results in Table 3 of the paper and reorganize the relevant rows below for ease of comparison.

ModelDirect answering (L0)Workspace (L6)L6 gain over L0 (pp)
Qwen3.5-27B34.125.3−8.8
Qwen3.6-27B32.932.6−0.3
Qwen3.8-27B35.352.4+17.1
Qwen3.8-Flash-Next40.954.5+13.5

At fixed size, Qwen3.5-27B and Qwen3.8-27B differ by only 1.2 points in direct answering, yet their gains from L6 differ by 26.0 points (Appendix C.4). Qwen3.6-27B falls between them, scoring about the same with and without the workspace. The ability to benefit from L6 thus changes substantially across generations even at the same parameter count. Within Qwen3.8, both variants benefit from L6. The 27B model is dense [10] and Flash-Next uses a mixture-of-experts architecture [11], so this pair cannot isolate the effect of scale. Qwen3.5 offers more sizes [12], but its 27B checkpoint scores lower at L6 than with direct answering. A size comparison within that family therefore addresses a different question from how the gains observed in Qwen3.8 scale. We plan to extend the paired L0/L3/L6 evaluation when a comparably capable open-weight family offers several sizes with matched architectures and training recipes.

References

[1] Visual Instruction Tuning

[2] Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Search

[3] PyVision: Agentic Vision with Dynamic Tooling

[4] OpenCode

[5] Claude 3.7 Sonnet and Claude Code

[6] Effective harnesses for long-running agents

[7] WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation

[8] PyVision-RL: Forging Open Agentic Vision Models via RL

[9] SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning

[10] Qwen3.8-27B

[11] Qwen3.8-Flash-Next

[12] Qwen3.5

Reviewer_4t9WOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how the tools and execution environment around a fixed model affect visual reasoning scores. It defines seven harness configurations, ranging from direct answers to fixed visual tools, persistent code execution, files, and a shell. To compare them, the authors assemble VHStar, with 600 tasks from 12 sources, and test four models across all seven configurations and nine models at selected levels. Moving from fixed tools to persistent code execution raises scores by about 6–10 percentage points for all four full-ladder models, while broader interfaces bring no consistent further gain. The main contribution is a common evaluation protocol and evidence that more tool access does not always improve performance.

Critical strengths
  • The paper makes harness differences easier to evaluate. Prior work already shows that tools and interfaces matter, but model scores are hard to compare when these settings differ. The ladder tests complete configurations on common visual tasks, providing a useful protocol beyond another tool-equipped agent.
  • The main finding has consistent support. L2→L3 improves all four full-ladder models, with paired task-bootstrap intervals excluding zero, while none scores higher at L6 than L3. Checks using source-level averaging, observation channels, step budgets, and token costs strengthen the finding that broader access does not reliably help.
Critical weaknesses
  • The main comparisons do not establish stability across repeated runs. Each configuration is evaluated with one trajectory per task. The confidence intervals resample tasks while keeping those trajectories fixed; they do not measure variation across fresh runs. The reported gains are evidence for the tested runs, but repeated interactive runs are needed to show how consistently the configuration rankings hold.

  • Reuse and scoring need further verification. The paper describes the setup, but I could not locate a release link in the supplied materials to check the task manifest, harness implementation, and saved outputs. Some answers also rely on LLM equivalence judgments, for which I could not locate a human agreement audit. These gaps limit confidence in reuse and close score differences.

Questions for authors
  1. Check ranking stability. Please repeat L2/L3 and L3/L6 comparisons on a representative task subset and report variation across runs. Separate this uncertainty from the existing task-bootstrap intervals.
  2. Make the benchmark verifiable. Please provide or identify the task manifest, harness code and prompts, model versions, grader settings, and saved trajectories. State which source data can be redistributed and which must be reconstructed.
  3. Validate LLM-based scoring. Please report agreement with human scoring on a sample, including cases where nearby configurations receive different scores, and state whether corrections change the main comparisons.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 4t9W

We sincerely thank the reviewers for the thorough review. We greatly appreciate your recognition of our work and the valuable questions you raise. We will attempt to address your concerns below.

W1 & Q1: Consistency of comparisons across tasks and runs

Thank you for your insightful suggestions. To ensure the robustness and consistency of our comparisons and evaluations, we run and evaluate L2, L3, and L6 three times on the same 600 tasks, with all configurations using a temperature of 0.95. We report mean ± standard deviation across three runs and 95% confidence intervals computed from paired differences across runs. The results are shown below.

ModelL2 scoreL3 scoreL6 scoreL3 − L2, pp [95% CI]L3 − L6, pp [95% CI]
DeepSeek-V4.147.1 ± 1.655.6 ± 1.551.4 ± 1.9+8.5 [+4.1, +11.2]+4.2 [+1.0, +7.1]
GLM-5.3-Flash43.3 ± 1.853.9 ± 1.953.4 ± 2.4+10.6 [+6.0, +13.5]+0.5 [−3.1, +4.0]
Qwen3.8-Flash-Next52.4 ± 0.659.0 ± 1.454.5 ± 1.3+6.6 [+2.7, +9.1]+4.5 [+1.0, +7.6]

Most results show limited variation, and the paired confidence intervals support the same conclusions, confirming the consistency and robustness of our previous findings.

W2 (benchmark reuse) & Q2: Materials for reproducibility

Thank you for this suggestion. To facilitate reproducibility, we provide the evaluation code and the VH-Star benchmark in an anonymous repository. It includes all seven harness implementations, their prompts and tool policies, and the exact prompt for judging answer equivalence. The benchmark metadata lists the source, split, upstream record, and evaluator for each of the 600 tasks. The task index also records each source’s license and reconstruction instructions, including upstream access for ZeroBench and VTC-Bench. Appendices B.2 and B.3 document model deployments, reasoning settings, interaction budgets, and grading settings. We also provide several saved outputs in the README files to illustrate how the judge scores two responses to the same task.

W2 (scoring reliability) & Q3: Consistency and reliability of LLM-based evaluations

First, we understand the concern that grading errors may change the ordering of configurations with similar scores. To ensure the validity of our scoring, VH-Star follows each source benchmark’s answer requirements and scoring criteria. We apply the same scoring rule to each task across all configurations. For example, Spatial-IQ requires an exact integer match. TIR-Bench jigsaw tasks award partial credit for the fraction of pieces placed correctly. For tasks that require semantic judgment, we use a fixed LLM judge to compare the extracted answer with the reference, following the approach used in prior studies. The LLM judge prompts are available in the anonymous repository.

Second, to validate LLM-based scoring, we sample 100 tasks, each paired with a model response and a reference answer, and ask three human annotators to score each response against its reference. The sample includes cases where adjacent configurations receive different scores. We then compare their scores with those of the LLM judge. Overall agreement between the human annotators and the LLM judge is 97.7%. Even in the worst case, correcting the audited disagreements can shift a three-run mean difference by at most 0.39 percentage points, below the smallest nonzero gap of 0.5 points, which confirms that the rankings from LLM-based scoring are stable under these corrections.

Thank you for the new experiments and clear replies. The repeated runs, tests with a second runtime, and human checks of the scores address most of my concerns. I will keep my Weak Accept score. Please add these results to the paper and limit the claims to the tested tasks and harnesses. Please also make clear that L3 and L6 differ in several ways. The results do not show that L3 is better for every model.

Thank you very much for your thoughtful feedback and for taking the time to review our additional experiments. We appreciate your continued support for the paper. In the revised version, we will include the results from the repeated runs, the tests with a second runtime, and the human checks of the scores. We will limit our claims to the tasks and evaluation harnesses we tested, clarify the multiple differences between L3 and L6, and describe the comparison separately for each tested model.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

Dear Reviewers, AC, SAC, and PC,

Thank you for the careful and constructive reviews. We summarize our responses around three central questions: whether the study is reproducible and generalizes across harnesses, whether the reported comparisons and scores are reliable, and what the ladder measures and reveals about visual reasoning.

Paper Overview and Strengths

[Paper Overview]: VH-Star studies how harness design affects visual reasoning. It evaluates four models on 600 tasks from 12 source benchmarks, covering five visual domains and four difficulty strata. Its seven configurations (L0–L6) progressively introduce visual tools, model-written code, kernel persistence, file access, image delivery, and workspace interaction.

[Strengths]: The reviewers highlight three strengths:

  • Unified evaluation: We introduce a unified harness-based evaluation framework and compare seven configurations across models on identical tasks, budgets, and scoring rules, enabling controlled analysis of visual reasoning interfaces.
  • Usable benchmark: The benchmark contains 600 tasks from 12 sources across five visual domains and four difficulty strata, with task metadata, source information, and evaluation specifications for reuse.
  • Systematic analysis: We disentangle harness and model effects. The ladder links score differences to visual tools, code, state, and image feedback; under the same harness, we identify substantial differences across models and the emergence of harness-based visual reasoning in recent advanced foundation models.

Responses to Key Concerns

1. Reproducibility and Generalization (4t9W, wwVm)

[Reproducible release] (4t9W W2/Q2; wwVm W1/Q2): We release the core materials in the VH-Star review repository. It contains the 600-task index, source and license metadata, all seven configurations, prompts, tool definitions, image-return and permission rules, submission/model/grading settings, and the exact answer-equivalence judge prompt. Saved trajectories and case studies are included where permitted; otherwise, we provide upstream locators and reconstruction instructions.

[A second workspace runtime] (wwVm Q3): We implemented L3 and L6 in Pi and evaluated them on the full benchmark. L3 exceeds L6 by 3.1 and 4.3 points in Pi, compared with 3.7 and 4.0 points in OpenCode for the two tested models. The same ordering across runtimes supports generalization across harness implementations.

Together, these results establish VH-Star as a controlled framework for studying visual reasoning at the harness level, with a reusable benchmark and a common basis for comparing tools, code, state, feedback, and workspace interaction.

2. Reliability of Comparisons and Scoring (4t9W)

[Run-to-run stability] (4t9W W1/Q1): We repeated L2, L3, and L6 on the same 600 tasks for three runs. Score standard deviations are at most 2.4 points. The mean L3–L2 improvement ranges from 6.6 to 10.6 points, and the mean L3–L6 difference from 0.5 to 4.5 points. We report run-level variation separately from task-bootstrap confidence intervals.

[Human validation of the judge] (4t9W W2/Q3): On 100 response–reference pairs, including adjacent-score cases, three human annotators agree with the fixed LLM judge on 97.7% of cases. Correcting audited disagreements changes a three-run mean difference by at most 0.39 percentage points. The judge prompt and scoring configuration are released.

3. Ladder Motivation and Empirical Findings (vQiG, wwVm)

[Motivation for the seven levels] (vQiG W1/Q1): The ladder follows the progression of agentic visual reasoning: direct perception (L0), human-designed visual actions (L1–L2), model-written programs in a persistent kernel (L3), file-backed reasoning (L4–L5), and a shell-based workspace with explicit file inspection (L6). L4 and L5 bridge the code-kernel and workspace paradigms. All interactive configurations use the same OpenCode runtime, while tools, state, image delivery, and submission policies vary systematically.

[What the ladder reveals] (vQiG W3/Q2/Q3): Moving from predefined tools (L2) to model-written programs in a persistent kernel (L3) improves scores by 5.9–10.0 points across four models. More workspace freedom does not consistently improve scores, and the benefit of a workspace varies sharply across model generations. At 27B, Qwen3.5, Qwen3.6, and Qwen3.8 obtain L6 gains of −8.8, −0.3, and +17.1 points over direct answering; Qwen3.5-27B and Qwen3.8-27B differ by only 1.2 points in direct answering but by 26.0 points in their L6 gains. These results show that harness design and model training jointly shape visual reasoning performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.