- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper present TextCall, where the pixel output from the tool calls are omitted for think-with-image models. Experimental results demonstrate that the gains provided by returning such images are negligible, and even without any pixel information, the models achieve performance comparable or even better performance on selected benchmarks.
Critical strengths
- The paper studies an interesting problem regarding think-with-image paradigm: what actually carries the gains. And the author's tool-call scaffold hypothesis is interesting.
- This authors conducted experiments to support their hypothesis.
Critical weaknesses
- The benchmark used in this work are not specifically for think-with-image, they are conventional multi-modal benchmarks focusing on different capabilties. It is not sure how much gain model can get from tool calls.
- The base model is also pretty dated.
Questions for authors
- How's Textcall perform on recent think-with-image benchmarks such as VisualToolBench?
- For closed-source models such as GPT, Claude, Gemini, does TextCall still effective?
3 comments
Thanks for your responses and conducted additional experiments on new benchmarks, new open-source models and new closed-source models. However, my concern is not resolved by the current results. It seems that actual pexiel tool call helps for closed sourced models, and current experiments evidence are purely based on small models, hence the conclusion in this paper is less supported by the current results.
Response to Reviewer J3yc (follow-up)
Thank you for the reply. We would like to clarify two points.
1. Whether returned pixels help depends on the information they add, not on model scale. Our 7–8B models also benefit strongly when the returned crop carries detail missing from the input. In §3.5 (Table 3), we inject an answer-free Turn-1 call that crops the ground-truth target on all 191 V*Bench items. With the overview downsampled to 224 px, returning the native-resolution crop instead of a placeholder raises Qwen3-VL-8B from 82/191 to 171/191 (+46.6 pp, TWI-trained) and from 93/191 to 169/191 (+39.8 pp, TextCall-trained). With the overview at native resolution, the same ground-truth crop adds only +3.7 / +4.5 pp over ordinary runs. The gain we observe for GPT-6.1 Sol (+5.8 / +4.2 pp, Table 12) falls in this same range. The closed-source model therefore follows the same pattern as our open models rather than a different one. Our claim is not that returned pixels are useless, but that their value depends on whether they add information the model does not already have.
2. Our goal is to understand where the gains of TWI come from. We keep training and evaluation matched and change only what a tool call returns (the crop or a placeholder), which directly tests whether returned pixels are necessary. This control requires training the model without image returns, which is not possible for closed-source models; for them we can only compare returns at inference, as we did for GPT-6.1 Sol. TextCall is the controlled condition of this design. On the two open backbones we study, removing returned images preserves accuracy (Qwen2.5-VL 71.23 vs. 69.84; Qwen3-VL 79.42 vs. 79.27) while reducing latency by 13–23%. We state this scope explicitly in §1 and §6.
We are happy to discuss any further questions.
To Reviewer J3yc
Thank you for the review, and for highlighting the question at the center of our paper: what actually carries the gains in thinking with images. The revision answers it more directly: it keeps the controlled experiment and replaces the Scaffold Hypothesis, which the skeleton-only ablation did not support, with tests of when returned pixels add value. We ran the experiments you asked for:
TextCall79.42 vsTWI79.27 (+0.15 pp [-0.98, +1.31])Benchmarks and VisualToolBench (Weakness 1, Question 1). We ran VisualToolBench (Qwen3-VL-8B, Python image tool, local Qwen3.6-35B judge; App. H, Table 13). Task-paired over N = 603,
TextCall − TWIis ΔARS = -0.011 [-0.029, +0.007] (pass rate 6.14% vs 6.30%;Direct6.97%). With every arm at a 6–7% pass rate, the benchmark is at floor for 7–8B models and cannot separate the arms, so we report it as inconclusive. On how much tools can help: on V*Bench, the standard high-resolution benchmark in TWI work, a returned crop that carries detail missing from a 224 px overview adds +39.8 to +46.6 pp (§3.5, Table 3).Dated base model (Weakness 2). The result holds on Qwen3-VL-8B for the checkpoints we trained (full table in our reply to kqrD). Both arms use the same 65K data and training recipe and differ only in the 19,038 returned images. Qwen3-VL emits 0–1000 normalized boxes while DeepEyesV2 traces use pixel boxes, so for every Qwen3-VL evaluation the sandbox converts such boxes to pixels (§3.4). The conversion matters only for
TWI, which needs the right region in its returned crop;TextCallnever sees returned crops.Closed-source models (Question 2). Closed models cannot be fine-tuned, so
TextCalltraining cannot be applied; we instead compared returned crops with placeholders at inference. We added GPT-6.1 Sol on V* (N = 191, two independent runs; §3.5 and App. H, Table 12), using a 0–1000cropfunction; we did not run Claude or Gemini in the rebuttal period.Direct(no tool)Returning the crop beats the placeholder in both runs (+5.8 pp, p = .0034; +4.2 pp, p = .0215) and also beats
Direct. So for this frontier model, placeholders are not as effective as returned crops. We use this to bound our claim (§1, §3.5, §6), which covers the open 7–8B SFT checkpoints we trained.We hope these additions address your concerns. If they do, we would be grateful if you would consider updating your score; we are glad to clarify anything further.