- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper compares harness configurations for visual reasoning while keeping the model and the tasks fixed. The configurations form a ladder that moves from direct answering, through predefined visual tools and generated code with a persistent kernel, to general workspaces where the model manages files and runs shell commands. The authors evaluate several models on VHStar, a new benchmark of open-ended image questions drawn from public sources and stratified by domain and difficulty. They report that code with a persistent kernel improves every model over fixed tools, while moving on to general workspaces gives uneven and often lower scores.
Critical strengths
Moving from fixed visual tools to a code environment helps every model the paper tests. Adding code execution with a persistent kernel to the fixed-tool setup raises all full-ladder models on the same tasks, and the gain holds after removing the operation menu or doubling the step budget, as in Table 10.
Critical weaknesses
- I am not convinced that the lower scores of the workspace rungs come from workspace delegation itself rather than from how these rungs are implemented. Each step after L3 changes several things at once, such as how images come back to the model, file storage and how answers are submitted, and L6 also enforces an access policy that is described only in Appendix E. Permission rejections hit almost half of the L6 tasks for GLM (Table 21), yet the scores are not split by whether a rejection happened.
On top of this, every result comes from one in-house harness that. It seems that the authors did not reproduce any existing harness system like opencode, and none of the code, prompts, tool descriptions, permission rules or judge prompt is released. So the current analysis results might not be generalizable across current harness systems, instead, is more reliable on the sketchy harness provided. Therefore it might be hard to generalize the conclusion to existing workspace harness systems.
- From analysis, it is unclear to me that the gain from L2 to L3 can be credited to the persistent kernel, as the Abstract does. That step adds code execution and kernel persistence together, as Appendix C.1 itself notes, and the one rung that changes kernel persistence alone, L4 to L5, barely moves the scores (Table 10). Most of the gain may simply come from being able to run code.
Questions for authors
-
Which operations does the L6 permission policy block, for example the overlay inspections in the GLM case in Appendix E? What is the gap between L3 and L6 on the tasks where no rejection occurred?
-
Appendix B.3 says the saved evaluation configuration includes the exact prompt. Can you release the full system prompt, tool descriptions, permission rules, submission rules and judge prompt for each rung, together with the harness code? Apart from the tool descriptions, is the prompt identical across rungs?
-
If L6 is replaced by an existing workspace agent such as Opencode, or by a second independent implementation, does L3 still beat L6 on at least two of the full-ladder models?
-
Can you report configurations that change one design axis at a time, for example L3 with explicit image reading added, or L4 with automatic image return?
2 comments
Response to Reviewer wwVm - Part 2
W1 & Q3. Performance with other harnesses
We fully understand the reviewer’s concern. A direct comparison between our OpenCode-based L3 and a complete harness such as Pi [2] also reflects differences in runtime prompts and system-level tools. Since the original L3 and L6 already use OpenCode, we implement both levels on Pi and evaluate them on the full benchmark. This allows us to compare L3 and L6 within each runtime. The results are shown below.
| Model | Runtime | L3 score (%) | L6 score (%) | L3 − L6 (pp) |
|---|---|---|---|---|
| DeepSeek-V4.1 | OpenCode | 55.1 | 51.4 | +3.7 |
| DeepSeek-V4.1 | Pi | 53.8 | 50.7 | +3.1 |
| Qwen3.8-Flash-Next | OpenCode | 58.5 | 54.5 | +4.0 |
| Qwen3.8-Flash-Next | Pi | 58.8 | 54.5 | +4.3 |
L3 outperforms L6 for both models on both OpenCode and Pi, with similar score gaps.
W2 & Q4. Isolating the effects of kernel persistence and image return
We thank the reviewer for the insightful feedback. We first clarify that code execution and the kernel are introduced together at L3. Our L0–L6 ladder follows the development of visual agentic reasoning, from direct perception to simple code-based visual reasoning (PyVision [3], L3), and then to the current harness-based paradigm (OpenCode [1], L6). L6 typically uses a terminal tool to write and execute code, while earlier L3-style designs use an isolated Jupyter kernel to execute code and retain intermediate state. L2–L3 therefore measures the combined effect of code execution and kernel persistence. Similarly, L3–L4 introduces writable storage and explicit image reading together. In our implementation, the model must save a generated image to the workspace before reading it explicitly.
To examine kernel persistence, we compare L4 with L5, which starts a fresh Python interpreter for each call. Both configurations provide the same visual tools, writable files, explicit image reads, and submission rule. L4 scores 1.1–2.4 percentage points higher across the four models (Table 2), showing the benefit of kernel persistence when file storage is also available.
To examine image delivery, we add automatic image return to L4 while keeping the persistent kernel, file access, fixed visual tools, and submission rule the same. Both configurations are evaluated on the same 600 tasks. The results are shown below.
| Model | L4 score (%) | L4 + automatic image return score (%) | Gain over L4 (pp) |
|---|---|---|---|
| DeepSeek-V4.1 | 49.8 | 54.9 | +5.1 |
| Qwen3.8-Flash-Next | 59.6 | 61.8 | +2.2 |
Adding automatic image return improves scores by 5.1 percentage points for DeepSeek and 2.2 points for Qwen. These gains show that immediate visual feedback provides an additional benefit when code execution and kernel persistence are already available.
References
[1] OpenCode
[2] Pi
Response to Reviewer wwVm
We sincerely thank the reviewer for the careful and thorough review. We greatly appreciate your valuable questions and insightful comments on our work. We address your concerns below.
W1 & Q1. Detailed implementation and access policy
We thank the reviewer for the insightful suggestion. All six interactive configurations (L1–L6) are implemented on OpenCode v1.18.9 [1]. L0 is the direct-answer baseline. L6 is built entirely on OpenCode, but to keep the experiments fair and stable, we disable several additional OpenCode features, including planning, subagents, web search, and access to files outside the workspace. This lets the model perform vision-centric tasks in a stable sandbox using the supplied inputs, which are sufficient to solve the tasks. Disabling planning and subagent modes also helps us evaluate models under the same step budget. These settings standardize the evaluation conditions while preserving local computation and file access. The model can run Python through the shell, save an image in the task workspace, and inspect it with
read. We implement L1–L6 on the OpenCode runtime by disabling file reads and writes, providing a kernel-based Python tool, or providing predefined visual tools designed by humans, depending on the specific level. We provide an anonymous link to our implementation in the following section so that reviewers can inspect it.For GLM-5.3-Flash, we check the permission rejections and find that almost all arise from attempts to access external directories. The remaining rejections all result from web search tool calls, which occur in 1.4% of all cases. The intended directory for task files is
/workspace/work, but GLM-5.3-Flash sometimes attempts to run commands or access files under common directories such as/dataor/code. These paths lie outside the permitted/workspacedirectory, so the requests trigger the harness’s external-directory restriction. After each rejection, the agent returns to the designated sandbox directory and continues the task. We suspect that the model retains this behavior from post-training with a particular harness and starting directory. We do not observe this pattern in any of the other models. The table below compares L3 and L6 on the same tasks for which GLM records no L6 permission rejection.L3 still slightly outperforms L6 on tasks without permission rejections. This also shows that individual permission rejections do not substantially affect accuracy, as the model typically recovers quickly.
W1 & Q2. Code, prompts, and scoring rules
We thank the reviewer for requesting these materials. We release the code for all seven configurations, including the L0-L6 system prompts, the tool definitions, the permission and submission rules, and the exact answer-equivalence judge prompt. We also provide a configuration guide to support reproducibility. Anonymous repository
Most of the prompt text is shared across levels. The differences beyond tool descriptions explain how to use the available tools and submit the answer. For example, L3 tells the model to display Python figures and submit its answer in the final response (as it cannot write files to the sandbox directory). At L6, the prompt tells it to read saved images and write its answer to a specific file, with the final response as a fallback. These instructions reflect each level’s tool and submission requirements. For fair comparison, the VH-Star task prompts and the scoring rule for each task are the same across levels.