acceptodds
Under review as a conference paper at ICLR 2027

With Tools or Because of Tools? Tracing Capability Gains in Multimodal Agents

Abstract

Multimodal agents increasingly rely on tools for image inspection and code execution, and their benchmark gains are often attributed to these interactions. Yet aggregate accuracy does not reveal whether tools expand the set of solvable problems or merely provide alternative paths to answers already within reach. We study DeepEyesV2, Thyme, and Qwen3-VL-8B-Instruct across 15 benchmarks, comparing each agent with its tool-free counterpart. We additionally train a tool-free multimodal reasoner from DeepEyesV2 data and edit tool-use trajectories to probe the roles of generated calls and returned results. Tool access yields no consistent performance advantage, while the trained reasoner matches or exceeds the two specialized agents on several benchmarks. Across DeepEyesV2 and Thyme, 93.2% and 96.0% of tool-enabled successes are reproduced by at least one non-tool reference. Some remaining successes persist after returned results are removed, whereas tool outputs judged novel, correct, and properly integrated often occur on problems already solvable without tools. The full tool-use loop yields clear gains in specific settings, but much of its apparent advantage remains attainable through non-tool reasoning, revealing a gap between tool-enabled success and tool-contributed capability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal agents, tool use, vision-language models, thinking with images, agent evaluation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_2J6iOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether multimodal agents actually gain new capabilities from tool use. The authors evaluate models across multiple image-based perception and reasoning benchmarks, comparing tool-enabled and tool-free performance. They also use a separately trained tool-free reasoner and several ablations to analyze the contributions of tool calls and returned results. The results show that most problems solved with tools can also be solved without them, and tool use provides little or inconsistent improvement, although it helps on some tasks.

Critical strengths
  1. Interesting findings that could motivate future work to better understand and analyze tool use, rather than simply reporting benchmark improvements. The paper raises an important question about whether tools actually provide new capabilities.
  2. Good experimental setup overall, with comparisons between tool-enabled and tool-free models, as well as different ablations to understand the actual contribution of tool use.
Critical weaknesses
  1. Limited generalizability of the findings - The experiments focus on small models and tasks that mainly involve visual reasoning and image manipulation. For many of these tasks, the required information is already present in the input, so models may be able to solve them without tools. This is different from search tasks, where tools retrieve information unavailable in the original input. The paper does not evaluate such settings or larger models, making it unclear how broadly the conclusions apply.
  2. The writing and organization need improvement - The paper reads more like a laundry list of experimental results, making it hard to identify the main takeaways. The authors should better organize the results around a few key insights and explain how each experiment supports these insights. In particular, it would be useful to better explain why tools help in some settings but not others, rather than presenting the results as separate observations.
Questions for authors
  1. Why does tool use provide so little improvement? This is counterintuitive for models specifically trained to use tools. Is it mainly because the tasks do not require tools, the models make unnecessary calls, or they fail to use returned results correctly? Can you provide an analysis of these failure modes to better explain the findings?
  2. Do stronger models, such as 30B or 100B models, show similar issues? Can you evaluate at least one larger model to see whether it makes fewer unnecessary calls and uses returned information more effectively?
  3. Could the findings be related to overfitting from task-specific SFT? Do the same issues appear in models like SenseNova-MARS, which uses a more general RL-only agent-training recipe without additional task-specific SFT?
  4. Given that models like DeepEyesV2 also support search, do the findings hold on benchmarks like MMSearch, HR-MMSearch, etc.? Would tools provide more substantial gains when retrieving external information rather than processing information already available in the input?
  5. How does the findings generalize to OOD settings/datasets? Can you evaluate on other such datasets or domains to see whether the findings generalize beyond the training distribution?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 2J6i (Part 1 of 3)

Thank you for the suggestions on generalization, failure modes, and presentation. We added the requested larger model, MARS, and external-information comparisons. We group the overlapping questions below and will reorganize the paper around the findings these experiments establish.

Unless otherwise noted, supplementary decoding experiments use three independent seeds.

W1, Q2 & Q5. Larger models and generalization across tasks and domains

We extend the comparison to Qwen3-VL-32B-Instruct on complete OCRBench and both HRBench resolutions:

Qwen3-VL-32B: complete benchmarks.

BenchmarkFullTFFull − TF
OCRBench (0–1000)832.7839.0−6.3
HRBench-4K (%)83.581.0+2.5
HRBench-8K (%)78.078.6−0.6

The 32B model gains on HR4K, but tool access does not improve OCR or HR8K. Multiple calls occur in 544/3000 OCR Full trajectories, versus 277/3000 for 8B. We also check how many Full-successful questions are solved by TF or our Pure-Text Reasoner (PTR):

Can the same Full-successful questions be solved without tools?

Evaluation scopeModelFull-correct recordsOwn TF (%)TF or PTR (%)
All 15 benchmarksDeepEyesV217,33585.093.2
All 15 benchmarksThyme16,93491.496.0
All 15 benchmarksQwen3-VL-8B18,27890.595.4
OCRBenchQwen3-VL-8B255894.198.8
OCRBenchThyme256096.999.1
OCRBenchQwen3-VL-32B249895.499.2

Each percentage uses the Full-correct records in its row as the denominator. TF or PTR means either reference answers the same question correctly.

For 32B on OCRBench, its own TF covers 95.4% of Full successes; adding PTR raises coverage to 99.2%. The 15-benchmark results span natural images, small visual details, OCR, charts, mathematics, and logic. PTR training examples do not overlap with the evaluation benchmarks.

For Q5, we also report cross-benchmark evaluation on TIR-Bench and O3-Bench from the paper:

Additional benchmarks reported in the paper: Full / TF (%).

ModelTIR-Bench (1215 questions)O3-Bench (345 questions)
DeepEyesV218.4 / 18.927.2 / 28.0
Thyme16.6 / 16.426.7 / 27.1

Full and TF differ by at most 0.8 pp across these two additional benchmarks. This extends the observed pattern beyond the main 15-benchmark suite to further tool-oriented and challenging visual tasks.

MARS (Q3) tests another agent-training recipe, while MMSearch (Q4) tests tasks that benefit from external evidence. Across the additional models and benchmarks, tool-enabled performance remains task-dependent rather than consistently better. On OCRBench, most Full successes are also reproduced by Tool-Free inference for Qwen3-VL-32B and MARS-8B.

AuthorsOpenReview ↗

Response to Reviewer 2J6i (Part 2 of 3)

W2 & Q1. Explain the mechanisms and organize the paper around the main findings

We agree that the paper should make its findings easier to follow. We will organize the results around three questions and their answers:

Research questionMain findingEvidence
What do tools add to performance and cost?Most Full successes have tool-free solutions; tool use often adds cost without consistent accuracy gains.Own-TF/PTR coverage and paired accuracy, token, and time comparisons.
Why do calls help, add nothing, or fail?Returns may supply useful information, verify an existing answer, or be ignored or misused.Process labels, full trajectories, and the fixed-prefix crop comparison.
Can strong reasoning be learned without tool-calling supervision?PTR provides a competitive tool-free training route.PTR exceeds both specialized agents on 10/15 benchmarks.

The trajectories distinguish the failure modes in your question. Redundant execution: in MathVerse #17, DeepEyes derives 27° before Python evaluates the same expression; TF also solves it. Unsuccessful localization: in HR4K #25, four crops miss a license plate, and DeepEyes eventually reads it from the original image. Helpful crop with a correct TF answer: in HR8K #78, Thyme reads “ONE WAY” after cropping, while its TF answer is also correct.

We connect these cases to aggregate HR statistics:

HRBench mechanisms: 800 inputs per model and resolution.

AgentResolutionWith ≥1 callWith ≥2 callsReturn ignored / misusedTF wrong → Full correctTF correct → Full wrong
DeepEyesV2HR-4K8001064176254
DeepEyesV2HR-8K8001044354449
ThymeHR-4K100002030
ThymeHR-8K109053129

Call and integration counts describe Full trajectories; answer changes compare Full and TF on the same input.

DeepEyes calls on every HR input, but returns are often ignored or misused; Thyme calls much less frequently and usually integrates its returns correctly. Call frequency alone does not explain the final score. For example, Thyme corrects 20 answers but loses 30 on 4K, and corrects 31 but loses 29 on 8K. Small net changes can therefore reflect both unnecessary interaction and gains offset by new errors.

Useful returns also coexist with alternative solutions: among informative, Full-correct trajectories, TF solves 80.0% for DeepEyes and 94.7% for Thyme. Under the fixed-prefix comparison, withholding a requested crop lowers Qwen3-VL-8B continuation accuracy by 2.9/3.2 points on HR4K/8K. A tool can help a particular attempt even when the same question has a successful tool-free solution.

Following your recommendation, we will revise the paper around these three findings, place representative trajectories beside the mechanism results, combine the accuracy–cost comparisons, and move detailed protocol variants to the appendix. The revised structure separates what tools add, why particular calls help or fail, and how competitive reasoning can be learned without tool calls.

AuthorsOpenReview ↗

Response to Reviewer 2J6i (Part 3 of 3)

Q3. Test MARS and the alternative RL-only agent-training recipe

We evaluated MARS-8B using its native crop interface. The MARS training description specifies RL-only agent post-training from an instruction-tuned base for its 8B and 32B models. This tests the alternative to task-specific agent SFT suggested in your question.

MARS-8B: OCRBench and HRBench results.

BenchmarkFullTFFull − TF
OCRBench (0–1000)811.7831.3−19.6
HRBench-4K (%)83.277.4+5.8
HRBench-8K (%)80.174.0+6.1

MARS gains 5.8/6.1 points on HR4K/8K, while Full scores 19.6 points lower on OCRBench. On Full-successful OCR records, its own TF covers 94.2%; TF or PTR covers 98.3%. Thus, high OCR overlap also appears with this RL-only recipe, while the HR gains demonstrate substantial task-dependent tool value. We will include the MARS experiments and cite MARS in the revised paper. The RL-only agent also shows task-dependent gains: tools help on HRBench but reduce accuracy on OCRBench.

Q4. Does search help more when it retrieves external information?

We use all 300 MMSearch questions in a fixed-corpus retrieval experiment. The corpus contains 298 deduplicated reference webpages, indexed by title, snippet, and page text without question or answer labels. The agent can issue up to three BM25 queries, each returning up to three documents of 4096 characters each. Query images are supplied when available.

MMSearch questions with a fixed local corpus: DeepEyesV2, 300 questions; QA-F1 × 100.

ConditionWhat the model receivesQA-F1
No retrieval (TF)Question and query image, when available9.1
Agent-generated queries + BM25Original inputs, then matches from the fixed local corpus22.0
Provided evidenceOriginal inputs and the corresponding document; no search41.9

The corresponding reference page is retrieved in 317/900 runs (35.2%). Directly supplying that page removes query selection and provides up to 12288 characters. We also compare relevant evidence with evidence reassigned from another question:

Evidence-swap control on 300 MMSearch questions: Qwen3-VL-8B; single run per condition; QA-F1 × 100.

Evidence supplied with the question and available query imageQA-F1
None15.1
Relevant cached evidence52.6
Evidence from another question6.2

Providing more knowledge and strengthening reasoning are different contributions. Our paper asks whether tool use extends reasoning capability beyond what the model can achieve from the original inputs. Across our visual and reasoning benchmarks, most Full-successful questions also have tool-free solutions. Search delivers clear gains by supplying task-relevant external information; these gains establish the value of information access, rather than by themselves establishing stronger reasoning.

Reviewer_PmAPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether using tools actually improves the capabilities of multimodal agents or it simply provides alternative paths to answers they could already produce without tools. To test this, the authors evaluate DeepEyesV2, Thyme, and Qwen3-VL-8B-Instruct across 15 benchmarks, comparing tool-enabled inference against the same models without tools. They also train a separate tool-free visual reasoner and introduce trajectory interventions that remove tool calls or returned results to examine their contributions to successful reasoning.

The main finding is that most tool-enabled successes are also achievable without tools, with the combined non-tool references reproducing 93.2%, 96.0%, and 95.4% of the three agents' successful answers. Tool use sometimes introduces as many errors as it fixes, although certain tasks show clear benefits, such as Qwen3-VL's 7.1-point improvement on HRBench-4K. The paper argues that success with tools enabled should not automatically be interpreted as success that was caused by tools.

Critical strengths
  1. There are well designed interventions that reveal the distinct roles of tool interactions for each model evaluated. So, the trajectory ablations provide concrete evidence that different tools contribute in fundamentally different ways. For example, Qwen3-VL on HRBench-4K, the 7.1-point improvement from tool use disappears when the returned crop is withheld, demonstrating the importance of additional visual evidence. Whereas, DeepEyesV2 retains much of its TreeBench improvement without the returned result. This contrast is informative because it distinguishes benefits associated with tool outputs from those that persist through the tool-generation process itself.
Critical weaknesses
  1. The central question and several methods substantially overlap with recent work. MED (Ma et al., 2026) already separates tool-induced gains and harms from intrinsic reasoning improvements, while TextCall (Shao et al., 2026) examines whether tool-call text contributes more than returned visual information. More importantly, Wang et al. (2026) already conducts causal audits at the policy, trajectory, and individual tool-step levels. Although this paper extends the analysis to code-executing agents and a broader benchmark suite, its main findings and methodological contributions appear largely incremental.

  2. The tool-free baselines are not fully controlled for prompting differences. Although Full and Tool-Free use identical checkpoints, Appendix B shows that Tool-Free receives additional reasoning instructions. For example, DeepEyesV2 is explicitly instructed to plan extensively and reason carefully without tools. Consequently, differences in performance may reflect changes in prompting and reasoning behavior, not just tool availability.

Questions for authors

1, Appendix E indicates that Qwen's trajectory ablations use independently generated tool-use histories. Could the authors instead compare an unmodified Full trajectory against the exact same trajectory with only its returned crop removed? This would clarify whether the 7.1-point HRBench-4K gain is attributable to the crop itself rather than differences in the generated trajectories.

  1. Qwen3-VL gains 7.1 points from tool use on HRBench-4K but performs slightly worse than Tool-Free on HRBench-8K (Table 1). Is there any intuition on why does cropping help on the 4K benchmark but not the higher-resolution setting?

  2. The Tool-Free experiments show that trained agents can solve many questions without executing tools. However, could their capabilities have been acquired through tool-use training?

9 comments
AuthorsOpenReview ↗

Response to Reviewer PmAP (Part 1 of 4)

Thank you for the questions about our contribution, prompt controls, crop removal, and training. We address these below, combining Q1 and Q2 because the controlled crop experiment helps explain the 4K/8K contrast.

Unless otherwise noted, supplementary decoding experiments use three independent seeds.

W1. What does this study add beyond MED, TextCall, and Wang et al.?

We thank the reviewer for highlighting these closely related studies. Our core experiments and initial manuscript had already been completed independently before the August 2026 releases of TextCall and Wang et al. We regard these as closely related concurrent efforts and distinguish their contributions from ours below. Our contribution is to answer three connected questions: Does tool use expand the observed set of solved questions? Does a useful return mean the problem requires a tool? Can a model learn competitive reasoning without tool-calling supervision? We link the questions an agent answers correctly, its use of individual tool returns, and our Pure-Text Reasoner (PTR) to distinguish tool-assisted performance from capability already attainable without tools.

Related workMain focusOur contribution
MEDTracks crop-and-zoom RL checkpoints, separating intrinsic learning from tool-induced gains and harms and diagnosing how each changes during training.Link tool-enabled successes to own-TF and PTR solutions. Trained without tool-call supervision, PTR surpasses both Full baselines on 10/15 benchmarks.
TextCallRetains tool-call text while replacing returned images with a placeholder; matched LoRA, full fine-tuning, and RL experiments test the roles of calling text and pixels.Test solutions without tool calls or their textual scaffold. PTR surpasses both Full baselines on 10/15 benchmarks without tool-call supervision.
Wang et al.Intervenes on tool-use policy, whole trajectories, and individual returns under fixed prefixes to separate observation effects from calling behavior and diagnose failures.Show that useful tool returns and correct tool-free solutions often coexist, linking process-level evidence to question-level coverage.

Across 15 benchmarks, each model's own TF run solves 85.0%, 91.4%, and 90.5% of the questions answered correctly by Full for DeepEyesV2, Thyme, and Qwen3-VL-8B, respectively. Including the independently trained PTR raises coverage to 93.2%, 96.0%, and 95.4%.

The result also holds when we focus specifically on useful tool interactions. Among informative calls followed by a correct Full answer, the same agent's TF run solves 80.0% of DeepEyesV2 questions and 94.7% of Thyme questions. Separately, the Qwen fixed-prefix experiment gives a local benefit of +2.9/+3.2 pp from returning the requested crop on HR4K/8K. The crop comparison measures help after a request has been made; Full/TF compares solving the problem from the beginning. A useful return in one attempt is therefore compatible with strong performance without tools.

PTR adds a constructive training result: it exceeds both specialized agents' Full scores on 10/15 benchmarks without learning tool calls (Q3). Our contribution is to distinguish success with tools from additional problem-solving capability, and to demonstrate a competitive learning route without tool-calling supervision.

AuthorsOpenReview ↗

Response to Reviewer PmAP (Part 2 of 4)

W2. Control the additional reasoning instructions in the Tool-Free prompt

Full uses the prompt from the original DeepEyesV2 code, including its Python interface and tool-use instructions. TF (same reasoning instructions) keeps Full's task objective, step-by-step reasoning request, and answer format; it is told that tools are unavailable, and tool-specific instructions are removed. It receives none of the original TF prompt's extra planning and reflection instructions. We retain Full's saved runs and rerun only TF.

DeepEyesV2: use the original code prompt for Full and the same reasoning instructions for TF.

BenchmarkFullOriginal TFTF (same reasoning instructions)Full − TF (same instructions)Full runs with Python
OCRBench (0–1000)816.7867.3835.3−18.62980/3000
MathVerse (Vision-dominant, 788 problems)47.845.248.2−0.42334/2364

OCR scores use the 0–1000 scale; MathVerse scores are percentages. The call counts refer to the unchanged Full runs. Neither TF setting executes tools.

We also apply this comparison to the fixed HRBench diagnostic set:

DeepEyesV2: the same prompt comparison on the fixed HRBench diagnostic set.

BenchmarkFull (%)Original TF (%)TF (same reasoning instructions, %)Full − TF (same instructions, pp)Full runs with Python
HRBench-4K67.062.762.3+4.7299/300
HRBench-8K58.759.761.7−3.0299/300

Fixed diagnostic set: 25 original questions × four option rotations per resolution. Full and TF use the same prompt design as in the OCR/MathVerse comparison.

TF scores higher on OCR, MathVerse, and HR8K; Full scores higher on HR4K. The fixed-prefix experiment below separately tests the contribution of a requested crop. Strong tool-free performance persists with Full's reasoning instructions, while the benefit of tool-enabled inference varies by task.

AuthorsOpenReview ↗

Response to Reviewer PmAP (Part 3 of 4)

Q1 & Q2. Does the requested crop help, and why do 4K and 8K behave differently?

We added the requested crop-removal experiment and examined the model's answers and calling behavior to understand the 4K/8K difference.

First, compare the same crop request with and without its result. We pause Qwen3-VL-8B after it asks for a crop but before it receives the result. One version receives the cropped image normally; the other receives only a sentence saying that the result is withheld. The original image, preceding reasoning, and selected coordinates stay unchanged. We then let both versions finish, allowing further calls.

Across 775 selected HRBench-4K and 770 HRBench-8K input records, including option rotations, withholding the crop reduces accuracy from 82.1% to 79.2% and from 78.2% to 75.0%, respectively, with three continuations per record in each condition.

The requested crop provides a modest benefit of 2.9/3.2 pp to these continuations. The overall Full/TF comparison also includes the model's decisions about where to crop and how to use what it sees. The larger Qwen3-VL-32B model shows the same overall direction as 8B:

Complete HRBench: 200 original questions × four option rotations per resolution.

Model4K Full / TF (%)Δ4K (pp)8K Full / TF (%)Δ8K (pp)
Qwen3-VL-8B84.0 / 76.9+7.174.6 / 75.8−1.2
Qwen3-VL-32B83.5 / 81.0+2.578.0 / 78.6−0.6

The trajectories reveal two different processes. Helpful crops direct attention to the right object or make overlapping details distinguishable. Harmful sequences repeatedly inspect an unhelpful region or lose the relation between objects seen in separate crops. The original 8B traces and additional 8B/32B runs illustrate these behaviors:

Model and questionWhat the trajectory showsObserved behavior and likely mechanism
8B, HR4K #48: person relative to a binTF selects the foreground photographer. Full makes one crop around the background person beside the bin and correctly answers “in front.”The two runs select different referents; the crop focuses on the intended person.
32B, HR4K #562: bicycle countOne crop lets Full distinguish three bicycles, including a partly hidden wheel/frame; TF counts four.Local inspection separates overlapping objects.
8B, HR8K #104: sign above a doorFull repeats the identical crop six times and ends without an answer; TF reads “ABRI PATAUD” correctly.Repeating an unchanged view fails to resolve the difficulty and exhausts the interaction budget.
8B, HR8K #556 (additional run): chair relative to bicyclesBoth crops show the requested objects. The chair's horizontal coordinates are left of the bicycles, but Full concludes “in front”; TF answers “left.”Finding both objects succeeds, but combining the separate views into a spatial relation fails.
32B, HR8K #175: flag colorThe first crop includes the rooftop flag. Later crops leave that region and show sky; Full reaches six calls without an answer. TF gives the correct colors.Follow-up localization loses a target already found.
Reviewer_gTyLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether multimodal agents improve because tools provide genuinely new information or because tools merely offer alternative routes to answers already solvable without tools. It evaluates DeepEyesV2, Thyme, and Qwen3-VL-8B-Instruct on 15 benchmarks covering visual understanding, OCR, chart reasoning, and mathematical reasoning. The authors compare Full and Tool Free settings, train a Pure-Text Reasoner, and perform No-Result, No-Call, and Empty-Call trajectory edits. They find no consistent overall advantage from tool access, observe that most Full successes are reproduced by non-tool references, and show that tools can both correct and introduce errors. A notable exception is Qwen3-VL-8B on HRBench-4K, where the returned crop appears to provide genuine benefit. The paper argues that “success with tools” should be distinguished from “success because of tools.

Critical strengths
  1. Question-level success-set analysis. The paper examines whether each Full success is also achieved by Tool-Free or Pure-Text references. This directly distinguishes aggregate accuracy improvements from genuine expansion of the solvable problem set.
  2. Detailed decomposition of tool interactions. The No-Result, No-Call, and Empty-Call experiments help separate the effects of call syntax, generated code, and returned results. The Qwen HRBench-4K case provides a concrete example, where returned visual evidence appears necessary for the observed gain
Critical weaknesses
  1. The non-tool references do not fully identify causal tool contribution. Full and Tool-Free use different prompts, interaction contexts, and independently sampled generation paths. Their differences may therefore reflect tools, prompt changes, altered reasoning trajectories, or sampling variability. If Tool-Free solves a question, this demonstrates an observed non-tool solution but does not prove that the tool was unnecessary within the corresponding Full trajectory. Trajectory editing also introduces placeholder messages that may affect model behavior. In addition, single decoding runs do not capture variance across random seeds.

  2. The Pure-Text Reasoner and process annotations raise external-validity concerns. The Pure-Text Reasoner is trained on the DeepEyesV2 source pool using additional SFT and GRPO, so it is not a matched ablation of the specialized agents. The paper does not clearly report whether this training data was deduplicated against the evaluation benchmarks, so possible overlap could inflate non-tool coverage. Process labels are primarily generated by a single vision-language judge; although the reported human agreement is encouraging, category-level agreement statistics and uncertainty estimates are not fully reported. Several Thyme categories also have very small sample sizes.

Questions for authors
  1. Are the SFT and GRPO examples disjoint from the 15 evaluation benchmarks at the image, question, or near-duplicate level? If not, can the coverage results be reported after deduplication?
  2. Were Full, Tool-Free, and trajectory-edit experiments repeated with multiple independent decoding seeds? Please report across-seed means and variances.
  3. What exact placeholder message is used in No-Result? Have alternative placeholders been tested to rule out effects caused by the specific inserted text?
  4. Can you provide category-level annotation agreement, confusion matrices, or Cohenʼs κ, together with the distribution of human-audit samples across models and benchmarks?
6 comments
AuthorsOpenReview ↗

Response to Reviewer gTyL (Part 1 of 3)

Thank you for highlighting the need to distinguish tool effects from prompting, generation variability, and annotation choices. We group the overlapping questions about experimental controls, the tool-free reference, and annotations below.

Unless otherwise noted, supplementary decoding experiments use three independent seeds.

W1 & Q3. Separate tool effects from prompting and test the replacement wording

We use two controls: retain the original Full prompt while removing the extra planning instructions from TF, and change only the response to an already-generated crop request. We also check how many of Full's successful answers TF can produce without tools.

1. Use Full's reasoning instructions for TF.

Full uses the prompt from the original DeepEyesV2 code, including its Python interface and tool-use instructions. TF (same reasoning instructions) keeps Full's task objective, step-by-step reasoning request, and answer format; it is told that tools are unavailable, and tool-specific instructions are removed. It receives none of the original TF prompt's extra planning and reflection instructions. We retain Full's saved runs and rerun only TF.

DeepEyesV2: use the original code prompt for Full and the same reasoning instructions for TF.

BenchmarkFullOriginal TFTF (same reasoning instructions)Full − TF (same instructions)Full runs with Python
OCRBench (0–1000)816.7867.3835.3−18.62980/3000
MathVerse (Vision-dominant, 788 problems)47.845.248.2−0.42334/2364

OCR scores use the 0–1000 scale; MathVerse scores are percentages. The call counts refer to the unchanged Full runs. Neither TF setting executes tools.

We also apply this comparison to the fixed HRBench diagnostic set:

DeepEyesV2: the same prompt comparison on the fixed HRBench diagnostic set.

BenchmarkFull (%)Original TF (%)TF (same reasoning instructions, %)Full − TF (same instructions, pp)Full runs with Python
HRBench-4K67.062.762.3+4.7299/300
HRBench-8K58.759.761.7−3.0299/300

Fixed diagnostic set: 25 original questions × four option rotations per resolution. Full and TF use the same prompt design as in the OCR/MathVerse comparison.

TF scores higher on OCR, MathVerse, and HR8K; Full scores higher on HR4K. The fixed-prefix experiment below separately tests the contribution of a requested crop. Strong tool-free performance persists with Full's reasoning instructions, while the benefit of tool-enabled inference varies by task.

Original No-Result protocols. DeepEyesV2's saved code-only HR traces continue after the generated code without an inserted tool-response message. Thyme's code_only branch likewise retains the code and appends no <sandbox_output> or replacement text. Qwen3-VL-8B's format_only branch instead uses this literal acknowledgement:

Tool response from image_zoom_in_tool: the tool was invoked, but its returned result is withheld in this setting.

The new Qwen fixed-prefix experiment reuses that acknowledgement after one selected crop request; the alternative-wording check below uses a different sentence. These are distinct protocols.

AuthorsOpenReview ↗

Response to Reviewer gTyL (Part 2 of 3)

2. Return or withhold a crop that the model has already requested. We let Qwen3-VL-8B generate a crop request, then pause before returning the cropped image. From that exact point, we run two versions: one receives the normal crop response, including the image and its accompanying message; the other receives only the following sentence:

Tool response from image_zoom_in_tool: the tool was invoked, but its returned result is withheld in this setting.

The original image, question, earlier reasoning, and crop coordinates are unchanged. Both versions then finish the problem, with further crop calls still available.

Qwen3-VL-8B: return the requested crop, or replace it with a text message.

BenchmarkAnswers per conditionCrop returned (%)Crop replaced with text (%)Difference (pp)
HRBench-4K232582.179.2+2.9
HRBench-8K231078.275.0+3.2

4K: 775 inputs; 8K: 770 inputs. Answer counts pool the repeated runs. Inputs include answer-option rotations. For each pair, the model sees the same original image and has made the same crop request; only the tool response changes.

Withholding the crop reduces continuation accuracy by 2.9 points on 4K and 3.2 on 8K. Both versions start from the same request and preceding conversation, so this comparison measures the returned crop's contribution to that attempt. Q2 reports the per-seed results.

To address your concern that the replacement message itself might disrupt the answer, we repeat the comparison with a second wording:

Tool response from image_zoom_in_tool: no observation content is provided.

This check uses 75 inputs per resolution, pairing the real-crop and text-replacement answers from the same request:

Repeat the crop-versus-text comparison with a second replacement sentence.

BenchmarkAnswers per conditionCrop returned (%)Crop replaced with text (%)Difference (pp)
HRBench-4K22580.076.0+4.0
HRBench-8K22583.178.2+4.9

75 inputs per resolution, yielding 225 answers per condition across repeats. Each pair starts from the same crop request.

Withholding the crop also lowers accuracy with this alternative message on the 75-input subsets. The direction of the result therefore holds under both tested wordings.

3. Check whether the same questions can also be answered without tools. Across 15 benchmarks, each model's own TF run solves 85.0%, 91.4%, and 90.5% of the questions answered correctly by Full for DeepEyesV2, Thyme, and Qwen3-VL-8B, respectively. Including our Pure-Text Reasoner (PTR) raises coverage to 93.2%, 96.0%, and 95.4%. For 32B on OCRBench, own-TF coverage is 95.4%.

The crop experiment tests whether a return helps after a particular request; the coverage comparison tests solving from the original input. Most Full-successful questions also have a correct answer from the same model without tools, even before adding PTR.

AuthorsOpenReview ↗

Response to Reviewer gTyL (Part 3 of 3)

W2, Q1 & Q4. Validate the tool-free reference, training data, and process annotations

1. PTR and training data. PTR starts from Qwen2.5-VL-7B-Instruct, using 53K SFT and 30K GRPO examples from the DeepEyesV2 source pool. We checked that these do not overlap with the evaluation benchmarks. Qwen2.5-VL-72B generates fresh SFT solutions from the original images and questions, without tools or access to the agent trajectories. PTR receives images; “Pure-Text” refers to its tool-free solution.

We separate own-TF coverage from coverage after adding PTR: 95.4% versus 99.2% for Qwen3-VL-32B on OCRBench. Most overlap is already present with own-TF. Full/TF compares the same trained agent; PTR supplies an independent reference trained without tool-calling supervision.

2. Process labels and human audit. Final-answer correctness determines coverage. Process labels assess information gain, content quality, and use of the return, mapped to tool roles. Two annotators independently reviewed a random 30% sample of tool-using trajectories, including the inputs, full interaction, and final answers, blind to automatic labels and each other's decisions.

Human–human agreement is 94%; GPT-4o and Qwen3-VL-30B-A3B role labels agree at 96%, and each agrees with the human audit at 93%. Appendix F gives counts, proportions, and 95% Wilson intervals for the full automatically annotated set: 28,144 DeepEyesV2 and 804 Thyme records. Thyme's informative proportion is 95.0% [92.3, 96.8] on OCRBench (363 called records) and 93.6% [87.3, 96.9] on HRBench-8K (109). These are category-frequency intervals, not category-level agreement within the human audit.

Among informative, Full-correct cases, TF solves 80.0%/94.7% for DeepEyes/Thyme. We will consolidate these details in the revised paper. PTR uses non-overlapping training data and adds an independent reference to own-TF; human-audit agreement is distinguished from automatic category frequencies.

Q2. Report multiple decoding seeds, means, and variances

We repeated Full/TF on complete OCRBench and the controlled crop-return interventions on HRBench with three independent decoding seeds. Weights, questions, prompts, and sampling settings stay fixed within each condition. The tables report means, sample SDs, and sample variances (denominator 3 − 1).

OCRBench (0–1000): mean ± sample SD.

ModelFullTFFull / TF sample variance
DeepEyesV2816.7 ± 10.3867.3 ± 23.4105.1 / 548.5
Qwen3-VL-8B852.7 ± 4.0844.3 ± 4.016.3 / 16.3
Thyme853.3 ± 6.0854.3 ± 9.936.3 / 97.4
Qwen3-VL-32B832.7 ± 8.6839.0 ± 3.074.3 / 9.0

SD is in score points; variance is in squared score points.

HRBench-4K scores from the three continuations are:

HRBench-4K: per-seed crop-versus-text results.

Tool responseRun 1Run 2Run 3MeanSDVariance
Real crop81.782.282.382.10.30.1
Withheld crop79.178.679.979.20.60.4

The crop improves accuracy in all three runs, averaging 2.9 points (question-clustered 95% interval [0.1, 5.7]). Resampling groups each original question's four option rotations together.

These repeats quantify decoding variability in OCRBench Full/TF and HR crop-return interventions; the HR repeats continue from fixed crop requests.

Reviewer_eAvDOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a study comparing tool use and reasoning to determine which is better at solving real-world benchmarks. This paper's main aim is to explore how tool interactions can become procedural rather than genuinely helpful. Sometimes, the agent performs extra tool-mediated steps, yet this added interaction does not necessarily improve the final reasoning outcome. Additionally, the method is trained to reason over the tool traces and tool-use trajectories to probe the roles of generated calls and returned results. This has shown improvements on some tasks, even over tool use. The benchmarks provide a thorough study on diverse scenarios.

Critical strengths

This paper presents important findings for a comparison between tool use and reasoning. In it, the authors study how much of an agent's tool-assisted success represents additional capability beyond what it can already solve without tools. The benchmarks provide a thorough study of this debate. Most of the findings are decisive and useful for the community developing LLMs that reason and use tools for solving multimodal tasks.

Critical weaknesses
  • Reasoning for benchmarks like maths is beneficial, but for other tasks we may need grounded information. So the author should provide a clear conclusion and organize as follows, with findings for all three parts: one: perception-only benchmarks (when is good and when is bad), two: reasoning, and three: tasks that require tools for grounding information.

  • The tool-use study is kind of blind and does not provide analysis or findings from multi-step tool use for tool-based vs. reasoning analysis. In Table 14, we see the tool use "crop", but the paper could also benefit from high-level analysis of how the model performs crop, which could have improved performance if done right.

  • The math benchmarks do not target the tool's contribution to math reasoning. How can we see the improvement contribution that could come from better perception and reasoning? Can you compare performance on the original image vs. using only the structured text info to separate perception from reasoning in small mathematical benchmarks? We just need to ensure that the claim requires a tighter match between the test and the observation.

  • It will be hard to compare without the important axes of time and compute. While direct model inference will use the least compute, reasoning and tool use will differ in the scale of compute cost and inference time. We need to examine that as well.

Questions for authors
  • What other tools can be used and developed to perform the study? Can new tools help outperform the reasoning cleanly?

  • Why do the authors not retrain for tool use further, just as the authors trained for reasoning, to see a fair comparison?

7 comments
AuthorsOpenReview ↗

Response to Reviewer eAvD (Part 1 of 4)

Thank you for the suggestions on tool contributions, interaction behavior, and cost.

Unless otherwise noted, supplementary decoding experiments use three independent seeds.

W1. Separate perception, reasoning, and external grounding

We distinguish three contributions: recognizing image details, performing calculations, and retrieving external facts.

1. Recognizing text and small objects in images. We compare Full and TF on complete OCRBench and HRBench:

OCRBench (0–1000): 1000 questions.

ModelFullTFFull − TF (score points)
Qwen3-VL-8B852.7844.3+8.4
DeepEyesV2816.7867.3−50.6
Thyme853.3854.3−1.0
Qwen3-VL-32B832.7839.0−6.3

Complete HRBench: 200 original questions × four option rotations per resolution.

Model4K Full / TF (%)Δ4K (pp)8K Full / TF (%)Δ8K (pp)
Qwen3-VL-8B84.0 / 76.9+7.174.6 / 75.8−1.2
Qwen3-VL-32B83.5 / 81.0+2.578.0 / 78.6−0.6

Tools show no consistent OCR advantage. On HR, both Qwen models gain on 4K but score slightly lower with tools on 8K. The trajectories include both useful crops that expose small details and redundant displays or confirmations. A crop enlarges part of the original image so the model can inspect small text or objects more easily. W2 examines these behaviors and isolates the benefit of receiving a crop after the same call.

2. Executing computations. Alongside the complete MathVerse results, we compare both agents on its 788 vision-dominant problems:

MathVerse: Full versus TF.

Problem setModelFull (%)TF (%)Full − TF (pp)
All versionsDeepEyesV251.848.6+3.2
All versionsThyme47.347.9−0.6
Vision-dominant (788 problems)DeepEyesV247.845.2+2.6
Vision-dominant (788 problems)Thyme44.644.1+0.5

In MathVerse #17, DeepEyesV2 derives the correct answer of 27° before invoking Python to verify (180 − 36 − 90) / 2. Its TF run also answers correctly. In some cases, as here, the tool only verifies an answer already obtained through the model's reasoning, without supplying a new calculation needed to reach it. W3 separately tests how presenting diagram information in text changes performance.

3. Acquiring external facts. MMSearch improves with agent-generated queries and local BM25 retrieval and improves further when the relevant document is supplied directly (Q1). Search expands the available evidence.

On the visual and reasoning benchmarks, most questions answered correctly by Full are also answered correctly by TF or our Pure-Text Reasoner (PTR) without tool execution.

AuthorsOpenReview ↗

Response to Reviewer eAvD (Part 2 of 4)

W2. Explain multi-step behavior and how the model uses crops

We examine where the model crops, what the crop shows, and how the model uses it. We then check whether TF answers the same question correctly.

Call sequences and use of returns. Each model–resolution pair below has 200 original questions × four option rotations, giving 800 evaluated inputs:

HRBench: call sequences and use of the returned observations.

ModelResolutionWith ≥1 callWith ≥2 callsRepeated same-source crop boxReturn ignored / misused
DeepEyesV2HR-4K8001068417
DeepEyesV2HR-8K8001040435
ThymeHR-4K100000
ThymeHR-8K109005

All counts are trajectories. Repetition means the same source image expression and crop coordinates appear more than once; the final column uses the saved process annotations.

DeepEyes calls on every input, yet many returns are ignored or misused. Thyme calls less often and generally uses its returns correctly. This distinguishes how often a tool is invoked from how its output contributes to the answer.

Three trajectories illustrate the mechanisms; Full and TF both answer correctly in all three:

Model and questionWhat happened before and after the callWhat the sequence shows
DeepEyesV2, HR4K #25: license plateFour crops miss the plate; the third repeats the first. After describing the failures, the model reads “RK 879” from the original image.Repeated localization adds work without supplying the final evidence.
DeepEyesV2, HR8K #250: parking signStates “Public Parking” before cropping and repeats it afterward.Confirmation of an existing answer.
Thyme, HR8K #78: road signFinds the small sign hard to read, crops it, then reads “ONE WAY.”Useful visual access, but a tool-free solution also succeeds.

More calls do not guarantee better answers: Qwen3-VL-32B makes multiple calls in 544/3000 OCR runs, yet Full scores below TF. The plate example illustrates how repeated localization can add steps without useful evidence.

Does the requested crop help the model answer? We let Qwen3-VL-8B choose a region to crop, then pause before it receives the result. We run the rest of the answer twice: once returning the crop normally, and once replacing that tool response with a short text saying the result is withheld. The original image, question, and everything the model has said up to the crop request stay the same. The model can still request further crops. This measures accuracy after that shared crop request; the Full/TF table in W1 measures solving from the start.

The comparison uses 775/770 selected input records on 4K/8K, including option rotations, with three continuations per condition. Under this fixed-prefix setting, withholding the requested crop reduces continuation accuracy by 2.9/3.2 pp on HRBench-4K/8K. Providing the crop helps the model answer slightly more accurately in this setting.

We also check whether TF solves questions with informative tool returns:

When a tool return is useful, can TF still answer the same question?

AgentSubsetRecordsSolved by own TF (%)
DeepEyesV2Informative calls764959.7
DeepEyesV2Informative calls with Full correct484580.0
ThymeInformative calls73279.4
ThymeInformative calls with Full correct58494.7

Even among informative, Full-correct trajectories, TF solves 80.0%/94.7% for DeepEyes/Thyme. We will present these process examples alongside the aggregate analysis. A tool return can help an ongoing attempt while the same question remains solvable without tools.

AuthorsOpenReview ↗

Response to Reviewer eAvD (Part 3 of 4)

W3. Compare original images with structured text on the same mathematical questions

We compare the official image and text-only versions of the same 788 MathVerse problems using Qwen3-VL-8B. Both conditions disable tools, with output limits of 4096 and 8192 tokens. The underlying problems and model weights stay fixed. The official text-only version states diagram information in words, reducing the need to interpret visual relations.

Same 788 MathVerse problems, Qwen3-VL-8B, both conditions tool-free.

Output limit (tokens)Image version (%)Text-only version (%)Image − Text (pp)
409658.261.6−3.4
819262.765.3−2.6

Image accuracy is 3.4/2.6 points below text accuracy at the two limits; a larger budget helps both representations. Making diagram relations explicit in text improves accuracy on the same problems, consistent with a perceptual bottleneck.

W4. Compare time and computational cost as well as accuracy

We complement the accuracy results with output volume (text + tool results) and elapsed time, grouped into the paper's four task families. Text includes generated reasoning and code; tool results include returned text and images. The table compares Full, TF, and PTR, alongside each agent's runtime relative to PTR.

Output tokens and runtime by task family.

Task familyDeepEyes tokens Full / TFThyme tokens Full / TFPTR tokensDeepEyes time / PTRThyme time / PTR
Real-world understanding1026.9 / 81.4242.8 / 150.1193.01.9×1.4×
OCR tasks419.4 / 89.5236.6 / 174.9138.73.1×2.1×
Charts588.4 / 197.1162.7 / 163.6143.02.1×1.7×
Reasoning659.4 / 848.8400.7 / 407.3406.43.0×1.4×

Tokens count generated text (including code) + tool results (returned text and images). Each entry averages the benchmark-level per-question means within its family, weighting benchmarks equally. Runtime columns are Full total wall-clock time / PTR time on four H20 GPUs; PTR = 1.0×.

Tool use increases output volume in several task families, while both agents incur longer runtimes than PTR across all four families. PTR remains competitive in output length across the four task families. The per-question measurements below show how this trade-off varies by model and benchmark; time includes image processing, tool execution, and subsequent model reasoning.

Mean end-to-end time per question, Full / TF.

ModelBenchmarkTime (s)
Qwen3-VL-8BOCRBench1.9 / 1.2
DeepEyesV2OCRBench2.6 / 0.5
ThymeOCRBench1.6 / 1.1
Qwen3-VL-32BOCRBench3.2 / 2.3
Qwen3-VL-8B (single run per input)HRBench-4K9.6 / 6.9
Qwen3-VL-8B (single run per input)HRBench-8K14.7 / 7.9
Qwen3-VL-32BHRBench-4K15.2 / 10.1
Qwen3-VL-32BHRBench-8K22.5 / 10.7
DeepEyesV2MathVerse vision-dominant3.7 / 4.6
ThymeMathVerse vision-dominant2.0 / 1.9

Time includes image processing, model generation and tool execution, excluding answer judging.

OCR tool use adds latency without a consistent score gain. On HRBench, Qwen3-VL-32B takes longer at both resolutions, gaining accuracy on 4K but scoring slightly lower on 8K. Tool use often adds tokens and time without a corresponding accuracy gain.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

Dear Reviewers and Area Chair,

Thank you for your careful reviews and constructive feedback. In response to your comments, we have conducted additional experiments and revised the manuscript as follows:

  • Prompt and statistical controls. We examined the effects of reasoning instructions, decoding variability, and statistical dependence, and further validated our process annotations. These additional controls support our original findings, with Tool-Free inference remaining competitive. (gTyL, PmAP)

  • Contributions of tool returns. Fixed-prefix interventions confirm that returned crops can benefit individual reasoning trajectories, even when the full tool loop does not outperform Tool-Free inference overall. We further find that most informative, successful tool-use cases also have correct tool-free solutions. This distinguishes the local value of tool observations from their contribution to additional solved questions. (eAvD, gTyL, PmAP)

  • Additional models. We extended the evaluation to Qwen3-VL-32B and MARS-8B. Both show high overlap between tool-enabled and tool-free successes on OCRBench, extending our findings across model scales and training recipes. (2J6i)

  • External information retrieval. Our MMSearch experiments show substantial gains when relevant external evidence is provided. These results distinguish improvements from acquiring missing facts from those arising through reasoning over the original inputs. (eAvD, 2J6i)

  • Tool-free learning. We clarified PTR's training procedure and data construction. Without tool-calling supervision, PTR exceeds both specialized agents on 10 of 15 benchmarks, demonstrating a competitive alternative training route. (gTyL, PmAP, 2J6i)

  • Interaction behavior and efficiency. We added analyses of redundant calls, unsuccessful tool interactions, and output-token costs. PTR also achieves competitive accuracy with lower measured runtime across all four task families, demonstrating the practical efficiency of this tool-free training route. (eAvD, 2J6i)

We hope these additional experiments and clarifications address the reviewers' concerns, and welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.