- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces VLM-in-Sandbox, a training-free inference framework for vision-language models that manages intermediate images generated within a sandbox via a "Visual Workspace." It utilizes an image ledger to track files and their provenance, limits visual context to two active slots, and employs an explicit "PROMOTE" operation to select evidence for subsequent review. The study evaluates the framework's overall accuracy and efficiency gains—relative to an append-only sandbox approach—across four models and seven benchmarks, while also analyzing the impact of visual evidence selection and retention strategies through matching ablation studies, pairing statistics, and actual inference latency analysis.
Critical strengths
- The problem is clearly defined and holds practical value. By decoupling the persistent storage of intermediate images from their immediate visibility, the approach directly addresses the issue of context bloat in multi-step visual reasoning. The framework requires no additional training and relies on no specific visual tools, offering strong potential for integration.
- Overall results are consistent. The evaluation spans open-source and closed-source models, as well as tasks involving search, mathematics, and general reasoning. Compared to the "Append-only" baseline, all four models showed improved weighted accuracy and a 11.7%–51.8% reduction in total token consumption, demonstrating the general applicability of the benefits.
- The controlled experiments and cost analyses are robust. A matched 2×2 experimental design controlled for variables such as tools, prompt compilers, and budgets; the full solution achieved a 1.83 percentage point gain in performance while reducing token usage by 18.6%. Additionally, the reporting of paired statistics, repeated runs, and end-to-end latency (including with prefix caching enabled) further strengthens the credibility of the conclusions.
Critical weaknesses
- The independent contribution of the active selection mechanism remains insufficient. Compared to the "Auto-Recent-2" baseline—which automatically retains the two most recent images—the full method yields an improvement of only 0.63 percentage points (statistically insignificant, p=0.185) and a further token reduction of merely 2.3%. Consequently, the existing evidence more strongly supports the concept of "limiting visual context" rather than explicitly favoring the PROMOTE mechanism over simple recency-based retention strategies.
- The extent of innovation and external competitiveness remains unclear. The contributions primarily lie in system-level integration regarding image persistence, indexing, and context management. Comparisons with related methods—such as PyVision and Visual Sketchpad—are limited to the interface level; the absence of quantitative benchmarks using identical models and computational budgets makes it difficult to assess the actual incremental value over existing frameworks.
- Validation regarding long-term evidence management is limited. Trajectories in the matching experiments averaged only about five steps; experiments involving expanded image sets relied on image replication and fixed-trajectory replay, which validate context expansion but fail to demonstrate genuine long-range reasoning. Furthermore, there is a lack of analysis regarding the frequency and utility of re-accessing images after they have been evicted from the active buffer, rendering the value of the persistent "revisit" mechanism somewhat indirect.
Questions for authors
See Weaknesses.
3 comments
W3. Earlier evidence is retrieved in model-driven trajectories
We understand that the important question is whether stored evidence remains useful after the active visual context has changed. Our design makes artifact lifetime independent of slot occupancy: replacing a view preserves its file and identifier. The context-expansion study measures request growth; the new Qwen3.8-27B trajectories examine the corresponding reasoning behavior by following each artifact through viewing, visual absence, and later re-entry:
| Full evaluation | Episodes with a true revisit | Retrieval events | Multi-artifact revisit episodes |
|---|---|---|---|
| All seven benchmarks, 6,350 episodes | 285 (4.49%) | 369 | 256 |
| Independent repeat, 1,260 episodes | 94 (7.46%) | 123 | 86 |
A true revisit requires at least one intervening model request in which the previously viewed image is absent from the complete visual input. The first run includes retrieval after up to eight such requests. Of its 256 multi-artifact revisit episodes, 231 finish correctly, 24 incorrectly, and one with an invalid answer.
These observations establish that the model exercises the persistence mechanism during complete task-solving runs: it processes intervening evidence and subsequently brings an earlier artifact back into context. This is the functional reason to preserve artifacts separately from the active visual buffer—earlier work remains available as the agent changes what it inspects.
Main-text Figure 6A–C (p. 10) summarizes three real Qwen3.8-27B trajectories. In A, the model retrieves a wide crop and redirects its search to the blue sail. In B, it retrieves an earlier cube view and combines the two views to infer the opposite face. C is a failure case: the model retrieves the height grid but misreads the rear-view direction. These recorded cases show how earlier evidence enters subsequent decisions, including both successful reuse and a remaining interpretation error.
Thank you for your response. This has addressed my concerns, and I will keep my positive rating unchanged.
Thank you for recognizing the practical problem and the controlled evaluation. We understand your central concern to be the incremental value of the visual workspace: what explicit selection adds to a small context, how the abstraction relates to existing visual tools, and whether persistent evidence is actually retrieved. The new stronger-model results sharpen the first point, while the related-work comparison and trajectory analysis clarify the other two.
Manuscript changes: Section 3.2 (pp. 7–8) adds matched evidence for the workspace's incremental benefit; Section 3.3 (p. 9) and Figure 6A–C (p. 10) document actual retrieval and subsequent evidence use.
W1. The stronger backbone benefits more from explicit selection
Your concern focuses on the small 0.63-point selection gain in the original GPT-4.1-mini study. We therefore repeated the capacity-matched comparison with Qwen3.8-27B. On the same 1,260-question collection, the gain rises to 2.06 points, and an independent repeat yields 2.14 points:
The original collection comprises V*Bench, MathVision-mini, and RealWorldQA. The first Qwen three-benchmark row is a subset of the seven-benchmark run. Each gain is a within-model comparison at the same two-slot capacity; values are calculated from unrounded correct counts. In the new study, original task images are common inputs, and Full uses explicit model selection for both dynamic slots.
Full improves on all seven benchmarks, with 124 rescues versus 31 regressions overall. These results support a more substantial role for selection with the stronger backbone and suggest that greater reasoning capability can help the model exploit control over its visual evidence. The components also have distinct benefits: bounding automatic retention saves 14.86% of solver tokens, while active selection adds 1.46 accuracy points at almost the same token cost as the recency policy. Together, they improve the accuracy–cost trade-off of the full system.
W2. A reusable mechanism for visual evidence management
We understand your request for a clearer account of the contribution beyond existing tool-based visual reasoning systems. Our contribution is to make the lifecycle of visual evidence explicit and controllable throughout inference. The Image Ledger gives generated artifacts stable identities and provenance; bounded active slots control which evidence is visible; and PROMOTE brings a stored artifact back when a later decision calls for it. These operations let the agent retain its intermediate work while selecting the evidence relevant to its current reasoning step.
Table 13 in Appendix D.3, “Comparison with Related Approaches” (p. 24), compares our mechanism with Visual Sketchpad, PyVision, and multimodal-memory approaches. It covers artifact persistence, identity and provenance, active visual budget, reactivation, and training requirements. Its main distinctions are:
These interfaces define a reusable evidence-management layer over artifacts produced by visual tools. The matched comparison targets its contribution directly: the model, tools, compiler, budget, and scoring are fixed, while visibility and retention change. With Qwen3.8-27B on seven benchmarks, Full improves over automatic retain-all exposure by 2.93 points with 15.39% fewer tokens, and over automatic two-image recency by 1.46 points. Comparing complete frameworks also mixes differences in tool generation and orchestration; our controls isolate the visibility and retention policies within a common environment. Table 13 locates the architectural contribution, and the matched results establish its accuracy–cost benefit.