acceptodds
Under review as a conference paper at ICLR 2027

Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning

Abstract

Tool-augmented vision-language models increasingly "think with images": they call crop, zoom, or code tools and reason over the returned pixels. However, recent work using blind tests, gain decompositions, and attention analyses has shown that returned images contribute little, raising the question: if pixels do not carry the gain, what does? We hypothesize that the load-bearing signal is the structured text emitted before any returned pixel arrives: tool name, coordinates, target description, and intent. This textual scaffold encodes where to look and what to find. We introduce TextCall (call-but-no-return) to test this: it keeps the scaffold but replaces returned images with the text placeholder [Image output skipped]. Three studies support the hypothesis. (i) Non-necessity of returned pixels: across LoRA and full fine-tuning, TextCall matches or exceeds full thinking-with-images. (ii) Sufficiency of the scaffold: on matched training queries, scaffold-only input yields equivalent accuracy to returned-image input. (iii) Task-dependent scaffold content: decomposing the scaffold shows that its content helps where the task needs spatial reasoning, while on high-resolution search an empty tool-call skeleton already matches it. Together these results support the Tool-Call Scaffold Hypothesis: in current thinking-with-images distributions, the active signal is the structured text emitted at tool-call time; the returned image is a redundant carrier. TextCall preserves accuracy while reducing latency by 29–46% and eliminating tool-execution API calls. Our claims hold for current thinking-with-images benchmarks; constructing tasks where pixels are genuinely load-bearing remains an open direction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, visual reasoning, tool use, multimodal learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_J3ycOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper present TextCall, where the pixel output from the tool calls are omitted for think-with-image models. Experimental results demonstrate that the gains provided by returning such images are negligible, and even without any pixel information, the models achieve performance comparable or even better performance on selected benchmarks.

Critical strengths
  • The paper studies an interesting problem regarding think-with-image paradigm: what actually carries the gains. And the author's tool-call scaffold hypothesis is interesting.
  • This authors conducted experiments to support their hypothesis.
Critical weaknesses
  • The benchmark used in this work are not specifically for think-with-image, they are conventional multi-modal benchmarks focusing on different capabilties. It is not sure how much gain model can get from tool calls.
  • The base model is also pretty dated.
Questions for authors
  • How's Textcall perform on recent think-with-image benchmarks such as VisualToolBench?
  • For closed-source models such as GPT, Claude, Gemini, does TextCall still effective?
3 comments
AuthorsOpenReview ↗

To Reviewer J3yc

Thank you for the review, and for highlighting the question at the center of our paper: what actually carries the gains in thinking with images. The revision answers it more directly: it keeps the controlled experiment and replaces the Scaffold Hypothesis, which the skeleton-only ablation did not support, with tests of when returned pixels add value. We ran the experiments you asked for:

Your pointWhat we ranResult
Benchmarks not specific to TWI; VisualToolBenchVisualToolBench, N = 603Every arm at a 6–7% pass rate; inconclusive
Dated base modelMatched 65K full FT on Qwen3-VL-8BTextCall 79.42 vs TWI 79.27 (+0.15 pp [-0.98, +1.31])
Closed-source modelsGPT-6.1 Sol on V*, two runsReturned crops help: +5.8 / +4.2 pp

Benchmarks and VisualToolBench (Weakness 1, Question 1). We ran VisualToolBench (Qwen3-VL-8B, Python image tool, local Qwen3.6-35B judge; App. H, Table 13). Task-paired over N = 603, TextCall − TWI is ΔARS = -0.011 [-0.029, +0.007] (pass rate 6.14% vs 6.30%; Direct 6.97%). With every arm at a 6–7% pass rate, the benchmark is at floor for 7–8B models and cannot separate the arms, so we report it as inconclusive. On how much tools can help: on V*Bench, the standard high-resolution benchmark in TWI work, a returned crop that carries detail missing from a 224 px overview adds +39.8 to +46.6 pp (§3.5, Table 3).

Dated base model (Weakness 2). The result holds on Qwen3-VL-8B for the checkpoints we trained (full table in our reply to kqrD). Both arms use the same 65K data and training recipe and differ only in the 19,038 returned images. Qwen3-VL emits 0–1000 normalized boxes while DeepEyesV2 traces use pixel boxes, so for every Qwen3-VL evaluation the sandbox converts such boxes to pixels (§3.4). The conversion matters only for TWI, which needs the right region in its returned crop; TextCall never sees returned crops.

Closed-source models (Question 2). Closed models cannot be fine-tuned, so TextCall training cannot be applied; we instead compared returned crops with placeholders at inference. We added GPT-6.1 Sol on V* (N = 191, two independent runs; §3.5 and App. H, Table 12), using a 0–1000 crop function; we did not run Claude or Gemini in the rebuttal period.

GPT-6.1 SolRun 1Run 2
Direct (no tool)177 (92.7%)176 (92.1%)
Required crop + returned image187 (97.9%)185 (96.9%)
Required crop + placeholder176 (92.1%)177 (92.7%)

Returning the crop beats the placeholder in both runs (+5.8 pp, p = .0034; +4.2 pp, p = .0215) and also beats Direct. So for this frontier model, placeholders are not as effective as returned crops. We use this to bound our claim (§1, §3.5, §6), which covers the open 7–8B SFT checkpoints we trained.

We hope these additions address your concerns. If they do, we would be grateful if you would consider updating your score; we are glad to clarify anything further.

Thanks for your responses and conducted additional experiments on new benchmarks, new open-source models and new closed-source models. However, my concern is not resolved by the current results. It seems that actual pexiel tool call helps for closed sourced models, and current experiments evidence are purely based on small models, hence the conclusion in this paper is less supported by the current results.

AuthorsOpenReview ↗

Response to Reviewer J3yc (follow-up)

Thank you for the reply. We would like to clarify two points.

1. Whether returned pixels help depends on the information they add, not on model scale. Our 7–8B models also benefit strongly when the returned crop carries detail missing from the input. In §3.5 (Table 3), we inject an answer-free Turn-1 call that crops the ground-truth target on all 191 V*Bench items. With the overview downsampled to 224 px, returning the native-resolution crop instead of a placeholder raises Qwen3-VL-8B from 82/191 to 171/191 (+46.6 pp, TWI-trained) and from 93/191 to 169/191 (+39.8 pp, TextCall-trained). With the overview at native resolution, the same ground-truth crop adds only +3.7 / +4.5 pp over ordinary runs. The gain we observe for GPT-6.1 Sol (+5.8 / +4.2 pp, Table 12) falls in this same range. The closed-source model therefore follows the same pattern as our open models rather than a different one. Our claim is not that returned pixels are useless, but that their value depends on whether they add information the model does not already have.

2. Our goal is to understand where the gains of TWI come from. We keep training and evaluation matched and change only what a tool call returns (the crop or a placeholder), which directly tests whether returned pixels are necessary. This control requires training the model without image returns, which is not possible for closed-source models; for them we can only compare returns at inference, as we did for GPT-6.1 Sol. TextCall is the controlled condition of this design. On the two open backbones we study, removing returned images preserves accuracy (Qwen2.5-VL 71.23 vs. 69.84; Qwen3-VL 79.42 vs. 79.27) while reducing latency by 13–23%. We state this scope explicitly in §1 and §6.

We are happy to discuss any further questions.

Reviewer_JhDrOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the "thinking with images" paradigm in modern Vision-Language Models (VLMs) and hypothesizes that the visual outputs returned by the tool calls contribute little to the performance gains on VQA tasks. The paper argues that these gains are due to structured text generated before the tool execution, e.g., reasoning traces, tool-call instructions, etc. The paper refers to this as the "Tool-Call Scaffold Hypothesis". To this end, the paper proposes TextCall which replaces the visual output from a tool call (such as crop/zoom, etc.) with a text placeholder token, but retains the reasoning traces and tool-call instructions. The paper investigates this hypothesis through three studies: (a) if the visual outputs from tools-calls are necessary with training-time intervention, TextCall, (b) if same results can be observed in inference-only setup with Gemini-3-Flash, and (c) the structure is task-dependent. TextCall is evaluated across several VQA benchmarks and performance is shown to be comparable to full thinking-with-images approaches, but with reduced inference latency.

Critical strengths

S1. The paper proposes a simple intervention to the thinking-with-images paradigm that reduces inference latency without sacrificing performance on the evaluated benchmarks. The results suggest that returned tool-outputs may not be necessary in VQA tasks.

Critical weaknesses

W1. The main idea of the paper is not particularly surprising given prior work has already highlighted the limitations of visual reasoning in VLMs [1], VLMs' struggle with leveraging the available perceptual representations [2] and benefits of representing the visual tools as structured text [3]. Particularly, Janjua et al. [3] already show that raw pixel-level outputs from tools do not need to be passed to the reasoning models, as long as their information can be parsed through a textual structure. Therefore, the broader observation that pixel-level tool-outputs are non-essential to tool reasoning has already been explored. This work does not distinguish their contribution from prior work nor establishes the novelty of the proposed "Tool-Call Scaffold Hypothesis". Further, this paper takes a more restrictive approach by not supplementing the removed tool-outputs (crop/zoom) with additional perceptual information. The comparable performance shows that the results are likely confounded by underlying VLMs existing abilities.

[1] Fu, X., Hu, Y., Li, B., Feng, Y., Wang, H., Lin, X., … Krishna, R. (2024). BLINK: Multimodal Large Language Models Can See but Not Perceive. arXiv Preprint arXiv:2404. 12390.

[2] Fu, S., Bonnen, T., Guillory, D., & Darrell, T. (2025). Hidden in plain sight: VLMs overlook their visual representations. arXiv [Cs.CV]. Retrieved from http://arxiv.org/abs/2506.08008

[3] Janjua, M. K., Silva, H., Niu, D., & Rashidi, B. (2026, June). Don’t Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5165–5174.

W2. The proposed intervention, TextCall, is limited in its applicability. In tasks where visual information is required (such as tasks in BLINK benchmark [1]), the tool output cannot be discarded trivially. This is likely why this work's evaluation is limited to relatively simpler VQA tasks. Many VQA benchmarks can be solved to a considerable extent without additional visual evidence, as also highlighted by BLINK [1]. In Tab. 2, the authors show that question-only condition with Gemini-3-Flash already observes ~51% percentage points on DeepEyes V2 suggesting that performance is largely attributed to language priors. On perception-centric benchmarks such as BLINK, the reported performance remains limited in Tab. 1, and the comparable performance of both thinking-with-images and TextCall does not establish that returned visual information is unnecessary, particularly since both methods struggle. The experiments leave open whether TextCall is effective when tool outptus provide genuinely new perceptual information, instead of relying on crop/zoom operations whose effectiveness may stem from underlying VLM's strong language priors.

Questions for authors

None

4 comments
AuthorsOpenReview ↗

To Reviewer JhDr (1/2)

Thank you for the thoughtful review. Your W2 asks exactly the right question, and we ran the experiment it suggests: when the tool supplies information missing from the input, returned images are necessary (+40–47 pp). The revision now states this as the boundary of our finding.

W1. Novelty relative to prior work (BLINK; Fu et al., 2025; P²).

We agree that VLMs often under-use visual evidence (BLINK; Fu et al., 2025) and that P² already shows tool information can reach the model as text. We have also withdrawn the Scaffold Hypothesis, whose novelty you questioned (§1, §4, §6). This sharpens our question relative to prior work:

  • P² runs external experts (depth, optical flow, correspondence) that compute information the VLM does not have, and studies how to represent these outputs: it converts them into text at inference.
  • TextCall studies standard TWI crops, which repeat regions already in the input, and asks whether returning them matters at all. It removes them, without any substitute, in both training and evaluation, which a test-time probe cannot do.

You note that without a substitute, a tie may simply reflect the VLM's existing abilities. We agree, and the revision states this as the finding rather than leaving it as a confound: crops that repeat what the model already sees add nothing measurable after matched training (79.42 vs 79.27 on Qwen3-VL-8B), while crops that carry new information are necessary (W2a). P² is now discussed in §2 and App. I, and Fu et al. (2025) in §2.

W2. Is TextCall effective when tool outputs provide genuinely new perceptual information?

No, and the revision now shows this directly.

(a) New information: returned crops are necessary (§3.5, Table 3). On all 191 V*Bench items, we give the Qwen3-VL-8B models a fixed Turn-1 call that crops the ground-truth target, then return either the crop (Img) or the placeholder (Ph) on every call.

OverviewCheckpointImgPhΔ (95% CI)McNemar p
224 pxTWI-FT171/19182/191+46.6 pp [+38.7, +53.9]4.6×10⁻²³
224 pxTextCall-FT169/19193/191+39.8 pp [+32.5, +47.6]9.6×10⁻¹⁹
NativeTWI-FT174/191150/191+12.6 pp [+7.3, +17.8]1.9×10⁻⁵
NativeTextCall-FT175/191157/191+9.4 pp [+4.2, +14.7]9.1×10⁻⁴
  1. Missing detail. When the overview is downsampled to 224 px, the crop carries detail the input lacks, and placeholders cannot replace it.
  2. Native resolution. The gain is much smaller. Part of it comes from the injected prefix, which lowers the placeholder arm (150 / 157) below ordinary runs of the same models (165–168); against ordinary runs, the ground-truth crop adds +3.7 / +4.5 pp.
  3. TextCall can still use pixels. Given the same crops, TextCall-FT does as well as TWI-FT (175 vs 174 native; 169 vs 171 at 224 px), so training with placeholders does not remove the ability to read returned images.
AuthorsOpenReview ↗

To Reviewer JhDr (2/2)

(b) Why TWI and TextCall tie in ordinary V runs (§3.5; App. H, Table 11).* At native resolution, the model often finds the target from the overview by itself, so the returned crop repeats what it has already seen. To isolate the returned image, we freeze each checkpoint's own Turn-1 reasoning, code and crop, then continue with either images (Img) or placeholders (Ph) on every return (two saved prefixes, p1 and p2; JPEG V*, N = 191):

Checkpointp1: Img vs Ph (Δ pp, CI)p2: Img vs Ph (Δ pp, CI)
Qwen3-VL TWI165 vs 164 (+0.5 [-4.7, +5.8])168 vs 165 (+1.6 [-2.6, +5.8])
Qwen3-VL TextCall172 vs 167 (+2.6 [-2.1, +7.3])162 vs 163 (-0.5 [-6.3, +5.2])
DeepEyesV2-RL159 vs 158 (+0.5 [0.0, +1.6])156 vs 156 (0.0 [-1.6, +1.6])

No difference is detectable (all p ≥ .38). On the 57–59% of items where the Turn-1 crop finds the target, placeholders already reach 83.5–90.7%, and the returned crop adds +2.7 to +7.3 pp there. GPT-6.1 Sol, required to crop on every item, gains a similar +5.8 / +4.2 pp (Table 12); our finding covers the open 7–8B SFT checkpoints we trained.

(c) The 51% figure, BLINK and VisualToolBench.

  • The 51% was an error. Old Table 2 included 247 items without an input image. On the 753 image-bearing items, question-only falls to 37.7–40.6% and the image adds 25.5–29.0 pp (§4.1, Table 4; details in our reply to kqrD, Q4).
  • BLINK. We agree that a tie on BLINK cannot show that returned images are unnecessary: question-only is at chance there (37.8% vs 37.7%), and 7–8B crop agents reach only about 50%. We now report BLINK and VisualToolBench (6–7% pass rate; App. H) as inconclusive, not as support.

We hope this addresses your concerns, especially W2, which shaped the revised framing. If it does, we would be grateful if you would consider updating your score; we are glad to discuss further.

I thank the authors for their comments. Since the authors have withdrawn the "Tool-Call Scaffold Hypothesis", which was the entire claimed central contribution of the paper, this warrants a thorough rewriting of the paper and subsequent review rounds. Also, the new position is that the returned visual information maybe unnecessary when it is redundant with information already available in the input. This is much narrower empirical observation than the original hypothesis (which was withdrawn), and also perhaps not too surprising. So, what conceptual novelty remains beyond the fact that redundant information can be removed in certain settings?

I stress the rewrite and review again also because there are several references that contain hallucinated titles, authors, etc., see these two examples:

  1. Minsoo Kim et al. Diversity over frequency: Understanding tool-use collapse in multimodal reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), 2026. arXiv:2606.00096

  2. Haris Janjua et al. P2: Perceiving through text-converted expert outputs for multimodal reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026. arXiv:2604.12896

AuthorsOpenReview ↗

Response to Reviewer JhDr (follow-up)

Thank you for the follow-up and for checking our references.

References. You are right, and we apologize. The two entries had wrong titles and authors, although the arXiv IDs and venues were correct. The papers are Kim, Tan & Kim, Diversity Over Frequency: Rethinking Tool Use in Visual Chain-of-Thought Agents (ICML 2026, arXiv:2606.00096), and Janjua, Silva, Niu & Rashidi, Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs (CVPR 2026, arXiv:2604.12896). We then checked all 111 cited entries against arXiv and the venue pages (title and full author list) and corrected every remaining metadata error, including the titles of MMStar, CV-Bench and VisualToolBench. We also re-checked how the text describes Kim et al. and P² against the papers and made the wording more precise. The corrected PDF is uploaded.

What remains after withdrawing the hypothesis. The withdrawn part was our interpretation of why TextCall works. The central experiment, the matched SFT comparison (Table 1), is unchanged. The revision rewrites the abstract, introduction and conclusion to state what it shows, corrects the audit (Table 4), and adds Qwen3-VL-8B (§3.4) and the boundary experiments (§3.5) to bound the claim.

Why the finding is not obvious. The redundancy we report is not something one can assume in advance. Returned crops are native-resolution views, and TWI systems are trained to rely on them; o3-style agents and the RL methods built on DeepEyes treat these returned pixels as the mechanism of visual reasoning. Whether such crops are redundant on the standard TWI benchmarks (V*, HR-Bench) cannot be read off from inference-time perturbations, because a model trained with images is out of distribution when they are removed. It needs a model trained without them, which is the control we provide. With that control, removing returned crops costs nothing on two backbones (Qwen2.5-VL 71.23 vs 69.84; Qwen3-VL 79.42 vs 79.27). Under the same control, the crop becomes decisive once it carries detail the input lacks (+46.6 / +39.8 pp, Table 3).

What this gives the field.

  1. A practical test of whether a benchmark actually requires returned pixels: train a matched placeholder model and compare. By this test, V* and HR-Bench do not require them for the 7–8B models we train.
  2. Evidence that the gains attributed to "thinking with images" on these benchmarks do not depend on the returned crop after matched training.
  3. A cheaper training and inference recipe where returns are redundant (13–23% lower latency).

The claim in the revision is exactly what these controlled experiments establish. We would be glad to clarify any remaining concern.

Reviewer_kqrDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether the images returned by visual tools are actually necessary for "thinking with images" (TWI). The authors propose TextCall, a method that keeps the intermediate reasoning and tool-call code while replacing returned images with a fixed text placeholder during both training and inference. Using Qwen2.5-VL-7B, they show that TextCall matches or outperforms standard TWI under both LoRA and full SFT settings. Further analyses compare the information captured in tool-call text versus images and ablate the reasoning text and spatial code. The paper also reports lower inference latency because it avoids processing images repeatedly.

Critical strengths
  • An interesting and relevant research question. The paper challenges the common assumption that visual tool-use gains mainly come from looking at cropped images again. Understanding whether returned visual observations are truly needed is important for both interpreting TWI systems and building more efficient visual agents.

  • A well-designed, training-aligned ablation. Unlike approaches that only remove images at inference time, TextCall removes returned images during both training and evaluation. This avoids a train-test distribution mismatch. The matched SFT comparisons offer solid empirical evidence that returned pixels may be unnecessary in the tested settings.

Critical weaknesses
  • Limited practical value over the direct baseline. My main concern is that TextCall perform worse than the unmodified base model using direct inference. Across the core benchmarks, TextCall 65K reaches 71.23% average accuracy, compared to roughly 73.93% for Base Direct. While TextCall beats standard TWI SFT, it is unclear why anyone would use either method instead of simple direct inference. Moreover, the latency comparison in Table 4 leaves out Base Direct. This weakens the claim that TextCall is a practical deployment choice rather than just a cheaper version of an already underperforming TWI baseline.

  • Weak evidence for the proposed mechanism and limited testing in stronger tool-use settings. The paper credits the maintained performance to the tool-call scaffold, yet the skeleton-only ablation in Appendix C matches or beats the full scaffold on V*Bench and HR-4K. This implies that the base model's built-in visual reasoning ability, rather than useful tool-call content, may drive much of the performance. In addition, the main comparisons are limited to SFT, whereas the DeepEyes V2 checkpoint only beats Base Direct after RL. Because the matched RL experiments were unsuccessful, it remains unclear whether TextCall still works in stronger tool-trained models.

  • The related work could also better connect these findings to recent studies on visual tool-use mechanisms. Specifically, [1] examines how tool usage drops during RL even as accuracy improves, which closely relates to the RL behaviors shown in Appendix I. Furthermore, while the paper cites [2], their findings on internal capability gains versus tool-driven gains deserve a more direct discussion when interpreting the TextCall results.

[1] Diversity Over Frequency: Rethinking Tool Use in Visual Chain-of-Thought Agents, ICML26

[2] What Does Vision Tool-Use Reinforcement Learning Really Learn? ICML26

Questions for authors
  • Practical motivation: Since Base Direct beats TextCall on most core benchmarks, what is the practical benefit of deploying TextCall over direct inference? Could the authors include Base Direct in the accuracy-latency trade-off comparison under the exact same evaluation setup?

  • Generalization beyond SFT: Can the authors provide more evidence that removing returned pixels maintains performance in an RL-trained TWI model that genuinely benefits from tool use? This seems crucial since the official DeepEyes V2 RL checkpoint performs much better than the SFT versions.

  • Mechanism attribution: How do the authors explain the gap between the Tool-Call Scaffold Hypothesis and the skeleton-only results in Appendix C? How much of the maintained performance comes from meaningful reasoning and spatial code versus the base model's existing ability to answer directly?

  • Interpretation of Table 2: The question-only setup reaches a surprisingly high accuracy of 51.2% without seeing any visual input. Could the authors clarify the task breakdown and any language or answer-distribution biases behind this number? Specifically, how much of the 73.1% scaffold-only accuracy reflects actual visual information captured in the scaffold, rather than answers that can already be guessed from the question alone?

3 comments
AuthorsOpenReview ↗

To Reviewer kqrD (1/2)

Thank you for the detailed review. In short: we agree that Direct is the faster default when no agent loop is needed, we have withdrawn the Scaffold Hypothesis, and we have corrected old Table 2. The main controlled result is unchanged.

Practical motivation. Why deploy TextCall when Base Direct scores higher (73.93 vs 71.23)? Please add Base Direct to the latency comparison.

TextCall is not proposed as a replacement for Direct. The two answer different questions: Direct asks whether an agent loop is needed at all; TextCall asks whether, inside the loop, returned crops are what makes TWI work.

Why Direct leads on Qwen2.5-VL-7B: the reasoning format, not image removal. Asking the base model to reason before answering already lowers it from 73.93 to 69.22 (cf. Sprague et al., 2025). Single-turn CoT SFT lands in the same range (Free-CoT 71.62, Text-CoT 72.75), next to TextCall 71.23 and TWI 69.84. Removing returned images causes no drop.

On Qwen3-VL-8B the gap closes (matched 65K full FT; §3.4, Table 2):

Qwen3-VL-8BV*HR-4KHR-8KMMStarCV-2DCV-3DMean
Direct (letter, 0-shot)84.8276.6270.7563.9379.5592.7578.07
Direct (think, 0-shot)82.2078.5073.3867.8078.7992.3378.83
TWI (65K)86.3978.3873.7567.7379.6989.6779.27
TextCall (65K)87.9678.6273.3866.4079.8390.3379.42

TextCall − TWI = +0.15 pp [-0.98, +1.31]; no benchmark differs (all p ≥ .24). TextCall also matches or slightly exceeds 0-shot Direct (79.42 vs 78.07 letter and 78.83 think), so the accuracy cost of the agent loop seen on Qwen2.5-VL disappears on a stronger base. Direct remains faster.

Cost (V, N = 191, sequential, 2×H20-3e; §5, Table 6).* Direct is now included.

VariantCallsReturned visual tokensOutput tokensLatency (s)
Qwen2.5 Direct (letter / think)101 / 1230.66 / 1.07
Qwen2.5 TWI2.867554793.89
Qwen2.5 TextCall2.2403712.99 (-23%)
Qwen3 Direct (letter / think)101 / 1150.24 / 1.14
Qwen3 TWI2.374483323.08
Qwen3 TextCall2.0703002.68 (-13%)

Direct is 2.4–11× faster than TextCall. Inside the loop, TextCall saves 13–23% because it skips the prefill of returned images, uses fewer turns and writes shorter outputs.

Generalization beyond SFT. Does removing returned pixels hold for an RL-trained TWI model?

We agree that RL helps: on our six core benchmarks the official DeepEyesV2 checkpoints average 75.05 after RL vs 70.40 after SFT, above Base Direct (73.93; Table 1). A matched RL comparison of pixel removal was not possible: under GRPO our matched TWI run stopped calling tools altogether (0–0.5% tool calls by step 900; App. J, Table 15), the same collapse Kim et al. (2026) report. Two observations still bear on your question:

  • Official RL checkpoint. On DeepEyesV2-RL, continuing from its own Turn-1 calls, returning placeholders instead of images leaves V* unchanged (159 vs 158; 156 vs 156; §3.3, Table 11).
  • Our TextCall RL run. It never sees returned pixels and still calls tools on 75% of items, yet improved on high-resolution benchmarks (V* 78.01 → 81.68, HR-8K 67.50 → 70.62 at step 800) while degrading on CV-Bench (CV-3D 73.67 → 56.83).

Both fit MED (Ma et al., 2026): RL gains come mainly from intrinsic reasoning rather than tool outputs.

AuthorsOpenReview ↗

To Reviewer kqrD (2/2)

Mechanism attribution. How do the skeleton-only results fit the Scaffold Hypothesis, and how much comes from the base model's existing ability?

We agree. We have withdrawn the Scaffold Hypothesis, together with every claim that code "grounds" perception (Abstract, §1, §4, §6). Table 5 (§4.2) now places Skeleton-only next to the unmodified baselines:

Qwen2.5-VL-7BV*HR-4KCV-2DMean
Base Direct (letter, 0-shot)80.6371.2575.4575.78
Base Agent (untuned)62.8354.7570.5162.70
Full (reasoning + code, 9.5K LoRA)76.9671.5070.9373.13
Reasoning only67.0272.1259.8166.32
Code only64.9265.5055.0861.83
Skeleton only (empty text and code)79.0672.6263.2171.63

With empty text and code, Skeleton-only recovers from the untuned agent (62.70) to near Direct on V* (79.06 vs 80.63) and HR-4K (72.62 vs 71.25). So on these benchmarks most of the retained accuracy comes from the base model's existing ability, recovered by adapting to the multi-turn format, not from the content of the reasoning or code. Deleting only one part (Reasoning only 66.32, Code only 61.83) does worse than deleting both, because it leaves incoherent traces, such as a <think> that announces a crop followed by pass (§4.2). None of this affects the main result: Tables 1–2 keep <think> and <code> identical across TWI and TextCall and vary only the returned image.

Interpretation of old Table 2. Why does question-only reach 51.2%, and how much of the 73.1% reflects visual information in the scaffold?

Much less than old Table 2 suggested. It contained two errors, now corrected (§4.1, Table 4; App. B): 247 of its 1,000 items were text-only math and logic questions with no input image (question-only solves 92.3% of them), which produced most of the 51.2%, and the 79.4% cell came from an unpaired earlier run. We removed both, together with the claims built on them.

On all 753 image-bearing items (T = the trajectory's reasoning and code, without any tool output):

Answering modelQuestion only+ Image+ T+ Image + T
GPT-6.1 Sol40.6%66.1%68.3%67.9%
Gemini-3-Flash37.7%66.7%65.3%—
  • Priors are modest. Question-only reaches 37.7–40.6%; adding the image adds 25.5–29.0 pp.
  • Breakdown by answer type (GPT / Gemini, question-only): numeric answers 30.5 / 28.9% (n = 315), where the image adds 40–50 pp; free-form text 43.5 / 43.3% (n = 386); yes/no 80.8 / 50.0% (n = 52), the main answer-distribution bias.
  • T nearly matches the image (T − Image: +2.1 pp [-1.6, +5.7] for GPT, -1.3 pp [-5.05, +2.39] for Gemini) because the teacher wrote the trajectory after seeing the image. In 15.3% of traces the Turn-1 <think> already deduces the answer, and 42.6% write answer-identifying labels or values into code or comments (App. B).

Related work (Kim et al.; Ma et al., ICML 2026).

We now discuss Kim et al. (Diversity Over Frequency) next to our own RL collapse (§2; App. I, Table 14), and relate Ma et al. (MED) to our results in §1, §2 and App. I (Table 14). MED's finding that most post-RL gains come from intrinsic reasoning rather than tool execution, measured by test-time perturbation of fixed checkpoints, is consistent with ours; TextCall asks the same question while also removing returned images during training.

We hope these changes address your concerns. If they do, we would be grateful if you would consider updating your score; we are glad to clarify anything that remains.

Reviewer_kqrDOpenReview ↗

Response to Authors

I thank the authors for their response, additional experiments, strengthened related work, and clarifications. However, ultimately, I could not find the practical value that TextCall has compared to Direct, and withdrawing the scaffold hypothesis means that the central claim of the paper changes. Therefore, my assessment is that the paper should receive a fresh evaluation through resubmission, and it does not seem to reach the acceptance bar at this point.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers. The revision combines our training-aligned control with new backbone, resolution, shared-prefix and frontier-model experiments to show when visual tool agents need returned pixels and when they do not.

Main finding. Gains of thinking-with-images (TWI) agents are usually credited to "seeing" the crops their tools return. We test this directly. Two models are trained on the same 65,807 DeepEyesV2 traces with matched SFT and evaluation; they differ only in what a crop call returns: the cropped image (TWI) or the placeholder [Image output skipped] (TextCall). The intervention replaces 19,038 returned images in 8,511 traces.

  • Qwen2.5-VL-7B: TextCall 71.23 vs TWI 69.84 (+1.39 pp [+0.13, +2.65]).
  • Qwen3-VL-8B: TextCall 79.42 vs TWI 79.27 (+0.15 pp [-0.98, +1.31]).

Removing returned crops costs no measurable accuracy, saves 755 / 448 visual tokens per item and cuts latency by 13–23% (V*, Table 6). When no agent loop is needed, single-turn Direct remains 2.4–11× faster.

Test-time probes (Faithfulness, MED) perturb models that were trained with images; P² converts new expert outputs into text at inference. None of them trains a model without returned images, which is what this control does.

When returned pixels do matter (§3.5, V*Bench, N = 191). What decides it is whether the returned crop adds information the model does not already have.

  • New information. With a 224 px overview and a ground-truth crop cut from the native image, returning the crop adds +46.6 pp [+38.7, +53.9] (TWI-FT, 171 vs 82) and +39.8 pp [+32.5, +47.6] (TextCall-FT, 169 vs 93) (Table 3). Here placeholders fail.
  • Information the model already has. At native resolution, continuing each checkpoint from its own Turn-1 call, returning images instead of placeholders changes accuracy by -0.5 to +2.6 pp (Table 11). On the 57–59% of items where the model's own Turn-1 crop finds the target, placeholders already reach 83.5–90.7%, so the returned crop adds only +2.7 to +7.3 pp there. A ground-truth crop at native resolution gives a smaller gain (+12.6 / +9.4 pp; Table 3).
  • Frontier model. GPT-6.1 Sol, required to crop on every item, gains +5.8 / +4.2 pp from returned crops in two runs (p = .0034, .0215; Table 12). Our finding covers the open 7–8B SFT checkpoints we trained.

Withdrawn or corrected. The Scaffold Hypothesis; TextCall as a default method; and old Table 2, which mixed in 247 text-only items and one unpaired cell (now redone on 753 image-bearing items, Table 4).

Statistics. Item-paired bootstrap 95% CIs (B = 10,000); binary tests report exact McNemar p.

Revisions (in red). Abstract, §1 and §6 now state the narrowed claim. §3 adds Qwen3-VL-8B (§3.4) and the boundary experiments (§3.5). §4 corrects the trajectory audit (Table 4) and adds Skeleton-only and Base Direct to Table 5. §5 adds Direct to the latency table (Table 6). App. B and H–K add the audit details, boundary-experiment details (shared-prefix branch, GPT-6.1 Sol, coordinate audit, VisualToolBench and BLINK), related work, RL dynamics and coordinate shuffling.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.