acceptodds
Desk rejected at ICLR 2027

VLM-in-Sandbox: Visual Workspaces for Agentic Visual Reasoning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Sandboxed computer environments support multi-step reasoning with tools, executable programs, and persistent files, yet their extension from language models to vision-language models (VLMs) introduces a distinct state-management problem. Visual reasoning produces intermediate image-valued evidence—crops, masks, overlays, zoomed regions, and analytic renderings—that must remain addressable without accumulating unboundedly in multimodal context.We introduce VLM-in-Sandbox, a training-free framework for agentic multimodal reasoning in controlled computer environments. Its Visual Workspace registers generated artifacts in an image ledger, maintains a bounded active visual context, and lets the model explicitly promote selected evidence for subsequent inspection. This separates visual evidence generation, performed by sandbox tools, from visual evidence management.Across seven benchmarks and four VLMs, VLM-in-Sandbox achieves higher sample-weighted accuracy than Vanilla VLM and Append-only Sandbox for every model. A compiler-matched study on 1,260 examples isolates visibility from retention: the full policy reaches 66.27% accuracy with 18.6% fewer total tokens than the automatic, retain-all control. Paired evaluation of all 6,350 GPT-4.1-mini examples yields 302 rescues and 142 regressions relative to Original Append-only. A local vLLM study with prefix caching also reduces uncached prompt tokens, time to first token, and end-to-end latency, supporting explicit visual evidence management for sandboxed VLM agents.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Language Model, Visual Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HBmQOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VLM-in-Sandbox, a training-free inference framework for vision-language models that manages intermediate images generated within a sandbox via a "Visual Workspace." It utilizes an image ledger to track files and their provenance, limits visual context to two active slots, and employs an explicit "PROMOTE" operation to select evidence for subsequent review. The study evaluates the framework's overall accuracy and efficiency gains—relative to an append-only sandbox approach—across four models and seven benchmarks, while also analyzing the impact of visual evidence selection and retention strategies through matching ablation studies, pairing statistics, and actual inference latency analysis.

Critical strengths
  1. The problem is clearly defined and holds practical value. By decoupling the persistent storage of intermediate images from their immediate visibility, the approach directly addresses the issue of context bloat in multi-step visual reasoning. The framework requires no additional training and relies on no specific visual tools, offering strong potential for integration.
  2. Overall results are consistent. The evaluation spans open-source and closed-source models, as well as tasks involving search, mathematics, and general reasoning. Compared to the "Append-only" baseline, all four models showed improved weighted accuracy and a 11.7%–51.8% reduction in total token consumption, demonstrating the general applicability of the benefits.
  3. The controlled experiments and cost analyses are robust. A matched 2×2 experimental design controlled for variables such as tools, prompt compilers, and budgets; the full solution achieved a 1.83 percentage point gain in performance while reducing token usage by 18.6%. Additionally, the reporting of paired statistics, repeated runs, and end-to-end latency (including with prefix caching enabled) further strengthens the credibility of the conclusions.
Critical weaknesses
  1. The independent contribution of the active selection mechanism remains insufficient. Compared to the "Auto-Recent-2" baseline—which automatically retains the two most recent images—the full method yields an improvement of only 0.63 percentage points (statistically insignificant, p=0.185) and a further token reduction of merely 2.3%. Consequently, the existing evidence more strongly supports the concept of "limiting visual context" rather than explicitly favoring the PROMOTE mechanism over simple recency-based retention strategies.
  2. The extent of innovation and external competitiveness remains unclear. The contributions primarily lie in system-level integration regarding image persistence, indexing, and context management. Comparisons with related methods—such as PyVision and Visual Sketchpad—are limited to the interface level; the absence of quantitative benchmarks using identical models and computational budgets makes it difficult to assess the actual incremental value over existing frameworks.
  3. Validation regarding long-term evidence management is limited. Trajectories in the matching experiments averaged only about five steps; experiments involving expanded image sets relied on image replication and fixed-trajectory replay, which validate context expansion but fail to demonstrate genuine long-range reasoning. Furthermore, there is a lack of analysis regarding the frequency and utility of re-accessing images after they have been evicted from the active buffer, rendering the value of the persistent "revisit" mechanism somewhat indirect.
Questions for authors

See Weaknesses.

3 comments

Thank you for recognizing the practical problem and the controlled evaluation. We understand your central concern to be the incremental value of the visual workspace: what explicit selection adds to a small context, how the abstraction relates to existing visual tools, and whether persistent evidence is actually retrieved. The new stronger-model results sharpen the first point, while the related-work comparison and trajectory analysis clarify the other two.

Manuscript changes: Section 3.2 (pp. 7–8) adds matched evidence for the workspace's incremental benefit; Section 3.3 (p. 9) and Figure 6A–C (p. 10) document actual retrieval and subsequent evidence use.

W1. The stronger backbone benefits more from explicit selection

Your concern focuses on the small 0.63-point selection gain in the original GPT-4.1-mini study. We therefore repeated the capacity-matched comparison with Qwen3.8-27B. On the same 1,260-question collection, the gain rises to 2.06 points, and an independent repeat yields 2.14 points:

Model and evaluationAuto-Recent-2FullSelection gain
GPT-4.1-mini, original 1,260 questions65.63%66.27%+0.63 pp
Qwen3.8-27B, same 1,260 questions88.02%90.08%+2.06 pp
Qwen3.8-27B, independent repeat87.86%90.00%+2.14 pp
Qwen3.8-27B, all 6,350 instances88.41%89.87%+1.46 pp

The original collection comprises V*Bench, MathVision-mini, and RealWorldQA. The first Qwen three-benchmark row is a subset of the seven-benchmark run. Each gain is a within-model comparison at the same two-slot capacity; values are calculated from unrounded correct counts. In the new study, original task images are common inputs, and Full uses explicit model selection for both dynamic slots.

Full improves on all seven benchmarks, with 124 rescues versus 31 regressions overall. These results support a more substantial role for selection with the stronger backbone and suggest that greater reasoning capability can help the model exploit control over its visual evidence. The components also have distinct benefits: bounding automatic retention saves 14.86% of solver tokens, while active selection adds 1.46 accuracy points at almost the same token cost as the recency policy. Together, they improve the accuracy–cost trade-off of the full system.

W2. A reusable mechanism for visual evidence management

We understand your request for a clearer account of the contribution beyond existing tool-based visual reasoning systems. Our contribution is to make the lifecycle of visual evidence explicit and controllable throughout inference. The Image Ledger gives generated artifacts stable identities and provenance; bounded active slots control which evidence is visible; and PROMOTE brings a stored artifact back when a later decision calls for it. These operations let the agent retain its intermediate work while selecting the evidence relevant to its current reasoning step.

Table 13 in Appendix D.3, “Comparison with Related Approaches” (p. 24), compares our mechanism with Visual Sketchpad, PyVision, and multimodal-memory approaches. It covers artifact persistence, identity and provenance, active visual budget, reactivation, and training requirements. Its main distinctions are:

Approach in Table 13Artifact stateActive visual budgetReactivation
Visual Sketchpad / visual scratchpadsEpisode canvas or stateMethod-specificUpdate or regenerate
PyVisionSession artifactsNo general slot boundTool view or rerun
Multimodal memoryModel or memory stateMethod-specificRetrieval or update
VLM-in-SandboxFilesystem + ledger identity and provenanceTwo active slotsPROMOTE by artifact ID

These interfaces define a reusable evidence-management layer over artifacts produced by visual tools. The matched comparison targets its contribution directly: the model, tools, compiler, budget, and scoring are fixed, while visibility and retention change. With Qwen3.8-27B on seven benchmarks, Full improves over automatic retain-all exposure by 2.93 points with 15.39% fewer tokens, and over automatic two-image recency by 1.46 points. Comparing complete frameworks also mixes differences in tool generation and orchestration; our controls isolate the visibility and retention policies within a common environment. Table 13 locates the architectural contribution, and the matched results establish its accuracy–cost benefit.

W3. Earlier evidence is retrieved in model-driven trajectories

We understand that the important question is whether stored evidence remains useful after the active visual context has changed. Our design makes artifact lifetime independent of slot occupancy: replacing a view preserves its file and identifier. The context-expansion study measures request growth; the new Qwen3.8-27B trajectories examine the corresponding reasoning behavior by following each artifact through viewing, visual absence, and later re-entry:

Full evaluationEpisodes with a true revisitRetrieval eventsMulti-artifact revisit episodes
All seven benchmarks, 6,350 episodes285 (4.49%)369256
Independent repeat, 1,260 episodes94 (7.46%)12386

A true revisit requires at least one intervening model request in which the previously viewed image is absent from the complete visual input. The first run includes retrieval after up to eight such requests. Of its 256 multi-artifact revisit episodes, 231 finish correctly, 24 incorrectly, and one with an invalid answer.

These observations establish that the model exercises the persistence mechanism during complete task-solving runs: it processes intervening evidence and subsequently brings an earlier artifact back into context. This is the functional reason to preserve artifacts separately from the active visual buffer—earlier work remains available as the agent changes what it inspects.

Main-text Figure 6A–C (p. 10) summarizes three real Qwen3.8-27B trajectories. In A, the model retrieves a wide crop and redirects its search to the blue sail. In B, it retrieves an earlier cube view and combines the two views to infer the opposite face. C is a failure case: the model retrieves the height grid but misreads the rear-view direction. These recorded cases show how earlier evidence enters subsequent decisions, including both successful reuse and a remaining interpretation error.

Thank you for your response. This has addressed my concerns, and I will keep my positive rating unchanged.

Reviewer_pEPZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VLM-in-Sandbox, a training-free framework for agentic visual reasoning that enables vision-language models (VLMs) to interact with executable tools while efficiently managing intermediate visual evidence. Its main contribution is a Visual Workspace comprising an image ledger for persistent artifact storage and provenance tracking, a bounded active visual context with two slots (focus and aux), and an explicit PROMOTE operation that allows models to selectively reintroduce visual artifacts into their context. The authors evaluate the framework on seven multimodal benchmarks using four VLMs, comparing against vanilla inference and an append-only sandbox baseline. The proposed method consistently improves sample-weighted accuracy while reducing token consumption relative to append-only sandboxing.

Critical strengths
  1. A practical and well-motivated abstraction for managing intermediate visual evidence. The paper identifies an important systems challenge in tool-augmented VLM reasoning: intermediate visual artifacts must remain accessible without continuously occupying multimodal context.

  2. Comprehensive evaluation with meaningful controlled comparisons. The evaluation spans four VLMs and seven benchmarks, covering visual search, mathematical reasoning, and general visual understanding.

Critical weaknesses
  1. Although model-directed PROMOTE is presented as a defining component of the framework, the matched ablation shows only a 0.63 percentage-point improvement over the simpler Auto-Recent-2 baseline, which automatically retains the two most recent visual artifacts. This difference is not significant. These observations suggest that a substantial portion of the improvement may arise from bounded context and prompt management rather than the distinctive model-directed evidence-selection mechanism.

  2. Limited validation of the persistent visual memory capabilities that motivate the framework. The evaluation primarily uses static-image question-answering benchmarks, with relatively short reasoning trajectories (approximately five steps on average in the matched study). However, the proposed framework is particularly motivated by the need to preserve, retrieve, and revisit intermediate visual evidence over extended reasoning processes.

Questions for authors
  1. Could the authors report how frequently PROMOTE is invoked across all evaluated trajectories, particularly how often it retrieves an artifact that is no longer among the most recently generated images?

  2. Do the authors have closed-loop experimental results on tasks requiring the generation and subsequent reuse of multiple distinct visual artifacts, especially where correct reasoning depends on revisiting earlier evidence?

3 comments

Thank you for highlighting the distinction between keeping context compact and actively managing persistent evidence. We have strengthened the evidence for both parts: the new Qwen3.8-27B comparison shows a larger accuracy contribution from selection, and complete trajectory statistics show when the model retrieves earlier artifacts during task solving. Below, we explain what each result establishes for the mechanism you questioned.

Manuscript changes: Section 3.2 (pp. 7–8) quantifies selection gains at fixed visual capacity. Section 3.3 (p. 9) reports revisit statistics, and Figure 6B (p. 10) shows earlier and later artifacts being combined during task solving.

W1. Active selection adds accuracy at the same visual capacity

We understand the concern that the original 0.63-point difference left the contribution of PROMOTE unclear. The new comparison directly tests this contribution: Full and Auto-Recent-2 share the two-slot capacity and prompt compiler, while differing in the evidence-selection policy. On Qwen3.8-27B across all seven benchmarks, Full improves accuracy from 88.41% to 89.87%, with 124 rescues and 31 regressions.

SettingSelectionRetentionAccuracyMean total solver tokens
Matched-Auto-AllAutomaticAll generated images86.94%53,564
Matched-AppendModel-directedAll selected images87.69%50,998
Auto-Recent-2AutomaticLatest two images88.41%45,607
FullModel-directedTwo replaceable slots89.87%45,321

All four settings use the same 6,350 instances, tools, compiler, budgets, and scoring. Tokens are cumulative solver usage per instance. Original task images are common inputs; Full uses explicit selection for both dynamic slots in this study.

The comparison explains the two components' complementary roles. Limiting automatic retention saves 14.86% of tokens; adding model-directed selection at the same capacity then contributes 1.46 accuracy points at nearly unchanged token cost. On the original 1,260-question collection, the selection gain is 2.06 points, compared with 0.63 in the original GPT-4.1-mini study, and remains 2.14 points in an independent Qwen repeat. The larger gain in this stronger-model evaluation suggests that a capable backbone can use explicit control over visual evidence more effectively. Both the capacity limit and the selection policy contribute to the full system's performance.

W2. Persistence supports an evolving visual workspace

We appreciate your focus on preserving useful evidence as reasoning progresses. The relevant property is that an artifact's lifetime extends beyond its residence in the active slots. Our ledger preserves its file, identity, and provenance as the model shifts to other visual results, making it available for a later decision. A static input can therefore give rise to an evolving sequence of crops, diagrams, and other derived views.

We test this behavior in complete Qwen3.8-27B runs by following actual viewing, visual eviction, and retrieval:

Behavior in the seven-benchmark Full runObserved result
Episodes inspecting at least two distinct derived artifacts976
Episodes retrieving a previously viewed, evicted artifact285
Multi-artifact episodes with such retrieval256
Median / maximum interval of image absence before retrieval2 / 8 model requests

These observations exercise both parts of the design: the agent uses a bounded working set to inspect new results while retaining access to earlier ones. Figure 6B (p. 10) makes the benefit concrete: after inspecting intervening artifacts, the model retrieves an earlier cube view and combines it with the current view to infer the opposite face correctly. Slot replacement preserves the underlying file and identifier, so additional intermediate views can be processed without losing access to prior work. The full-run statistics and closed-loop case directly demonstrate the persistence-and-reuse behavior that motivates the workspace.

Q1. PROMOTE selects older evidence in ordinary benchmark execution

Yes. We now report PROMOTE usage over all Qwen3.8-27B Full trajectories, including successful, incorrect, and invalid-answer episodes. In the seven-benchmark Full run, PROMOTE is invoked in 20.96% of episodes, and 5.84% select an artifact outside the latest two generated images:

Full evaluationNInvokes PROMOTESelects outside latest twoTrue revisit
All seven benchmarks6,3501,331 (20.96%)371 (5.84%)285 (4.49%)
Independent repeat on original three1,260370 (29.37%)117 (9.29%)94 (7.46%)

Recency is determined by artifact generation order. A separate revisit label requires prior viewing and at least one intervening model request in which the same derived image is absent from the complete visual input. First inspection of an older artifact is counted separately: 118 episodes (1.86%) in the first run and 35 (2.78%) in the repeat; an episode may contain both types of event.

Thus, explicit selection is exercised during ordinary task solving and includes returns beyond the automatic recency window. For context, Matched-Append invokes PROMOTE in 19.98% of its first-run episodes and selects outside the latest two in 1.69%; the automatic settings expose images according to their respective policies.

Q2. Multi-artifact retrieval occurs in closed-loop reasoning

The new study contains the generation–inspection–retrieval loop you describe. The model chooses its next tool action from the preceding feedback, creates and inspects distinct visual artifacts, and can later retrieve an earlier image after it has left the active input. We identified 256 such multi-artifact revisit episodes in the seven-benchmark run and 86 in the independent repeat:

Full evaluationMulti-artifact revisit episodesCorrectIncorrectInvalid
All seven benchmarks, 6,350 episodes256231241
Independent repeat, 1,260 episodes867790

This makes the persistence mechanism observable within complete solutions: an early artifact survives on disk while other results occupy the workspace, then returns as visual input for a later decision. Artifact identity and the sequence of actual image inputs establish this continuity across intermediate tool actions.

Main-text Figure 6B (p. 10) shows this process in a real Qwen3.8-27B trajectory on MathVision-mini item 110. The first cube view shows two faces adjacent to a green triangle. After inspecting other artifacts, the model retrieves that view beside a second cube view showing two more neighbors. It combines the views to identify all four adjacent faces and concludes that the kangaroo is opposite the green triangle, correctly answering B. The retrieval and joint comparison are successive decisions in the recorded model trajectory.

Dear Reviewer, as the discussion period ends on November 18, we would appreciate hearing your thoughts on our rebuttal soon. This would give us time to address any remaining questions or run additional experiments if needed. Thank you!

Reviewer_3cuaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents VLM-in-Sandbox, a training-free framework for managing visual evidence during tool-assisted multi-step reasoning. Its Visual Workspace separates persistent image storage from the visual evidence presented to the model: an Image Ledger records images and their provenance; an explicit PROMOTE operation selects images for two replaceable active slots; and a prompt compiler combines the selected views with recent interaction history and a concise recap. Experiments across seven benchmarks and four VLMs demonstrate consistent improvements in aggregate accuracy over both direct inference and an append-only sandbox. In a comparative study involving 1,260 examples, the full method improves accuracy by 1.83 percentage points while reducing total token usage by 18.6% compared with automatically retaining all images.

Critical strengths
  1. The method design is clear, with the roles of key design components well-explained. The paper decouples the persistent storage of images from the visual context currently visible to the model; it uses an "image ledger" to maintain intermediate results and employs a "PROMOTE" operation to select images for two active slots—all without requiring additional training.

  2. The comparative experiments are robust, providing a solid basis for the observed improvements in accuracy. A 2×2 experimental design—holding the compiler, tools, budget, and scoring method constant—systematically evaluates the strategies for image selection and retention. Tested on 1,260 samples using GPT-4.1-mini, the full method achieved an accuracy of 66.27%, compared to 64.44% for the "Matched-Auto-All" baseline. This demonstrates that the primary conclusions are derived from systematic comparison rather than a few isolated success cases.

  3. Results across different models and real-world performance metrics underscore the method's practical value. The full method consistently outperforms the "Original Append-only" approach across all combinations of four models and seven benchmarks, indicating that the benefits are broadly applicable. In the aforementioned controlled experiments, the accuracy improvement was accompanied by an 18.6% reduction in total token usage. Furthermore, in local inference experiments using Qwen with prefix caching enabled, end-to-end latency decreased from 39.2 seconds to 34.1 seconds—a reduction of approximately 13%. These findings demonstrate that the advantages of effective visual context management translate into tangible gains in operational efficiency.

Critical weaknesses
  1. The evaluation focuses on smaller models, and the benefits for stronger or newer models remain unclear. The paper evaluates GPT-4.1-mini, Gemini-2.5-Flash, Qwen3.5-9B, and Ministral-3-8B, but the key 2×2 accuracy comparison is conducted only on GPT-4.1-mini. Although the four-model results support the effectiveness of the full system, it remains unclear whether limiting visual context provides similar benefits for models that are better at understanding multiple images and using context. Adding the key matched comparison on a stronger model, such as Qwen3.8-27B or Gemini-3.8-Flash, would help clarify the method’s scope.

  2. The practical value of keeping and revisiting older images has not been directly verified. The paper emphasizes that images remain accessible after leaving the active slots, but the current analysis does not clearly show how much this capability helps solve tasks. Table 2 reports that 7.4% of the 734 examples corrected by the method used PROMOTE, without distinguishing between viewing a new image for the first time and revisiting an image that had left the active slots. Moreover, the experiment that adds images to fixed reasoning traces mainly shows how context size grows; it does not establish whether revisiting older evidence improves answers. The authors should measure actual image revisits in the existing traces and provide case studies explaining how the retrieved images inform later decisions.

  3. The added benefit of active image selection over keeping only the two most recent images remains unclear. In Table 8, under the same two-image limit, the full method improves accuracy over Auto-Recent-2 by only 0.63 percentage points. The current evidence therefore does not establish that explicit selection through PROMOTE consistently outperforms a simple recency-based strategy. The results provide stronger support for limiting the number of images than for letting the model actively select visual evidence. The authors should repeat the paired comparison with Auto-Recent-2 under the same evaluation protocol and examine which tasks or examples clearly benefit from active selection. This would help establish the practical value of the additional selection mechanism.

Questions for authors

See weakness

5 comments

Thank you for recognizing the framework's design and controlled evaluation. Your comments raise three closely connected questions: whether a stronger model benefits from visual evidence management, whether persistent images are actually revisited, and whether explicit selection improves on a recency policy. Our new Qwen3.8-27B results show larger accuracy gains on the original matched collection, together with positive selection gains across all seven benchmarks. We address each question below.

Manuscript changes: Section 3.2 (pp. 7–8) adds the stronger-model matched comparison; Section 3.3 (p. 9) reports the independent selection repeat and actual revisits; Figure 6A (p. 10) shows how retrieved evidence guides a later decision.

W1. Larger accuracy gains with a stronger backbone

We followed your suggestion and evaluated Qwen3.8-27B with all four matched settings. On the same 1,260 questions used in the original study, Full's gain over Matched-Auto-All increases from 1.83 to 3.81 percentage points, and its gain over Auto-Recent-2 increases from 0.63 to 2.06 points:

Model on the original matched collectionFull accuracyFull − Matched-Auto-AllFull − Auto-Recent-2
GPT-4.1-mini, original study66.27%+1.83 pp+0.63 pp
Qwen3.8-27B, new study90.08%+3.81 pp+2.06 pp

The collection comprises V*Bench, MathVision-mini, and RealWorldQA, a subset of the seven-benchmark run below. Each row reports within-model contrasts under that study's configuration; the four settings share tools, compiler, budgets, and scoring within each study. In the new Qwen comparison, both dynamic slots use explicit model selection in Full, and original task images are common inputs across settings.

We also extended the four-setting comparison to all seven benchmarks, totaling 6,350 evaluation instances. Full reaches 89.87%, versus 86.94% for Matched-Auto-All, 87.69% for Matched-Append, and 88.41% for Auto-Recent-2; it ranks first on every benchmark and reduces total solver tokens by 15.39% relative to Matched-Auto-All. Vanilla accuracy is 86.13%.

These results show that the accuracy benefit becomes more pronounced in this stronger-model evaluation. They suggest that a more capable backbone can make better use of the freedom to select intermediate evidence: stronger reasoning and tool use can make explicit evidence management more valuable.

W2. Persistent images are revisited during actual task solving

Your distinction between first inspection and revisiting is central to this question. We therefore counted a revisit only when the same derived image was viewed, disappeared from the complete image input for at least one model request, and was subsequently displayed again. Across all Qwen3.8-27B Full trajectories, this occurs in 285 episodes, with 369 retrieval events.

Behavior in the seven-benchmark Full runObserved result
First inspection of an older artifact118 / 6,350 (1.86%)
Revisit after an interval of visual absence285 / 6,350 (4.49%)
Total revisit events369
Episodes inspecting multiple artifacts and revisiting earlier evidence256

The two inspection categories can occur within the same episode. Table 2's 7.4% refers to PROMOTE use among 734 rescued Qwen3.5-9B examples; the new statistics use all 6,350 Qwen3.8-27B Full episodes, including unsuccessful ones.

Revisiting is particularly frequent on V*Bench (16.75%) and HRBench-4K (11.00%). Among the 256 multi-artifact revisit episodes, 231 end with a correct answer, 24 with an incorrect answer, and one with an invalid answer. The observed return to previously evicted images establishes that persistent storage is actively used during benchmark solving: earlier evidence remains accessible while intervening requests use other views.

To show how retrieval changes a later decision, main-text Figure 6A (p. 10) presents a real Qwen3.8-27B trajectory on V*Bench item 21. The model first crops a striped sail twice. It then retrieves an earlier wide crop after it has left the visual input, locates the requested solid blue sail, and crops it to reveal the spider emblem. The recorded sequence shows the model using restored scene context to redirect its search and answer correctly.

W3. Selection gains persist across tasks and an independent repeat

The relevant test is to keep the visual capacity fixed and change which images are selected. With Qwen3.8-27B, Full improves over Auto-Recent-2 on every benchmark, achieving a pooled gain of 1.46 points. The largest gains occur on MathVision-mini, V*Bench, and HRBench-4K:

BenchmarkNAuto-Recent-2FullGain
HRBench-4K80092.00%94.00%+2.00 pp
V*Bench19194.76%96.86%+2.09 pp
WeMath testmini1,74095.40%95.92%+0.52 pp
MathVision-mini30487.83%90.46%+2.63 pp
MMStar1,50081.67%83.20%+1.53 pp
RealWorldQA76586.41%88.24%+1.83 pp
MMMU dev+validation1,05084.19%86.00%+1.81 pp
All seven, sample-weighted6,35088.41%89.87%+1.46 pp

HRBench comprises 200 base questions with four option orders. Gains are calculated from unrounded correct counts.

We then independently reran both settings on the original three benchmarks, regenerating model responses and tool trajectories for the same 1,260 questions. Full's advantage is +2.06 points in the first run and +2.14 in the repeat, with positive gains on all three benchmarks in both runs. Across all seven benchmarks, the paired comparison contains 124 rescues and 31 regressions. These consistent task-level and repeated gains support the practical value of choosing evidence at a fixed two-slot capacity.

Dear Reviewer, as the discussion period ends on November 18, we would appreciate hearing your thoughts on our rebuttal soon. This would give us time to address any remaining questions or run additional experiments if needed. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.