- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether multimodal agents actually gain new capabilities from tool use. The authors evaluate models across multiple image-based perception and reasoning benchmarks, comparing tool-enabled and tool-free performance. They also use a separately trained tool-free reasoner and several ablations to analyze the contributions of tool calls and returned results. The results show that most problems solved with tools can also be solved without them, and tool use provides little or inconsistent improvement, although it helps on some tasks.
Critical strengths
- Interesting findings that could motivate future work to better understand and analyze tool use, rather than simply reporting benchmark improvements. The paper raises an important question about whether tools actually provide new capabilities.
- Good experimental setup overall, with comparisons between tool-enabled and tool-free models, as well as different ablations to understand the actual contribution of tool use.
Critical weaknesses
- Limited generalizability of the findings - The experiments focus on small models and tasks that mainly involve visual reasoning and image manipulation. For many of these tasks, the required information is already present in the input, so models may be able to solve them without tools. This is different from search tasks, where tools retrieve information unavailable in the original input. The paper does not evaluate such settings or larger models, making it unclear how broadly the conclusions apply.
- The writing and organization need improvement - The paper reads more like a laundry list of experimental results, making it hard to identify the main takeaways. The authors should better organize the results around a few key insights and explain how each experiment supports these insights. In particular, it would be useful to better explain why tools help in some settings but not others, rather than presenting the results as separate observations.
Questions for authors
- Why does tool use provide so little improvement? This is counterintuitive for models specifically trained to use tools. Is it mainly because the tasks do not require tools, the models make unnecessary calls, or they fail to use returned results correctly? Can you provide an analysis of these failure modes to better explain the findings?
- Do stronger models, such as 30B or 100B models, show similar issues? Can you evaluate at least one larger model to see whether it makes fewer unnecessary calls and uses returned information more effectively?
- Could the findings be related to overfitting from task-specific SFT? Do the same issues appear in models like SenseNova-MARS, which uses a more general RL-only agent-training recipe without additional task-specific SFT?
- Given that models like DeepEyesV2 also support search, do the findings hold on benchmarks like MMSearch, HR-MMSearch, etc.? Would tools provide more substantial gains when retrieving external information rather than processing information already available in the input?
- How does the findings generalize to OOD settings/datasets? Can you evaluate on other such datasets or domains to see whether the findings generalize beyond the training distribution?
6 comments
Response to Reviewer 2J6i (Part 2 of 3)
W2 & Q1. Explain the mechanisms and organize the paper around the main findings
We agree that the paper should make its findings easier to follow. We will organize the results around three questions and their answers:
| Research question | Main finding | Evidence |
|---|---|---|
| What do tools add to performance and cost? | Most Full successes have tool-free solutions; tool use often adds cost without consistent accuracy gains. | Own-TF/PTR coverage and paired accuracy, token, and time comparisons. |
| Why do calls help, add nothing, or fail? | Returns may supply useful information, verify an existing answer, or be ignored or misused. | Process labels, full trajectories, and the fixed-prefix crop comparison. |
| Can strong reasoning be learned without tool-calling supervision? | PTR provides a competitive tool-free training route. | PTR exceeds both specialized agents on 10/15 benchmarks. |
The trajectories distinguish the failure modes in your question. Redundant execution: in MathVerse #17, DeepEyes derives 27° before Python evaluates the same expression; TF also solves it. Unsuccessful localization: in HR4K #25, four crops miss a license plate, and DeepEyes eventually reads it from the original image. Helpful crop with a correct TF answer: in HR8K #78, Thyme reads “ONE WAY” after cropping, while its TF answer is also correct.
We connect these cases to aggregate HR statistics:
HRBench mechanisms: 800 inputs per model and resolution.
| Agent | Resolution | With ≥1 call | With ≥2 calls | Return ignored / misused | TF wrong → Full correct | TF correct → Full wrong |
|---|---|---|---|---|---|---|
| DeepEyesV2 | HR-4K | 800 | 106 | 417 | 62 | 54 |
| DeepEyesV2 | HR-8K | 800 | 104 | 435 | 44 | 49 |
| Thyme | HR-4K | 100 | 0 | 0 | 20 | 30 |
| Thyme | HR-8K | 109 | 0 | 5 | 31 | 29 |
Call and integration counts describe Full trajectories; answer changes compare Full and TF on the same input.
DeepEyes calls on every HR input, but returns are often ignored or misused; Thyme calls much less frequently and usually integrates its returns correctly. Call frequency alone does not explain the final score. For example, Thyme corrects 20 answers but loses 30 on 4K, and corrects 31 but loses 29 on 8K. Small net changes can therefore reflect both unnecessary interaction and gains offset by new errors.
Useful returns also coexist with alternative solutions: among informative, Full-correct trajectories, TF solves 80.0% for DeepEyes and 94.7% for Thyme. Under the fixed-prefix comparison, withholding a requested crop lowers Qwen3-VL-8B continuation accuracy by 2.9/3.2 points on HR4K/8K. A tool can help a particular attempt even when the same question has a successful tool-free solution.
Following your recommendation, we will revise the paper around these three findings, place representative trajectories beside the mechanism results, combine the accuracy–cost comparisons, and move detailed protocol variants to the appendix. The revised structure separates what tools add, why particular calls help or fail, and how competitive reasoning can be learned without tool calls.
Response to Reviewer 2J6i (Part 3 of 3)
Q3. Test MARS and the alternative RL-only agent-training recipe
We evaluated MARS-8B using its native crop interface. The MARS training description specifies RL-only agent post-training from an instruction-tuned base for its 8B and 32B models. This tests the alternative to task-specific agent SFT suggested in your question.
MARS-8B: OCRBench and HRBench results.
| Benchmark | Full | TF | Full − TF |
|---|---|---|---|
| OCRBench (0–1000) | 811.7 | 831.3 | −19.6 |
| HRBench-4K (%) | 83.2 | 77.4 | +5.8 |
| HRBench-8K (%) | 80.1 | 74.0 | +6.1 |
MARS gains 5.8/6.1 points on HR4K/8K, while Full scores 19.6 points lower on OCRBench. On Full-successful OCR records, its own TF covers 94.2%; TF or PTR covers 98.3%. Thus, high OCR overlap also appears with this RL-only recipe, while the HR gains demonstrate substantial task-dependent tool value. We will include the MARS experiments and cite MARS in the revised paper. The RL-only agent also shows task-dependent gains: tools help on HRBench but reduce accuracy on OCRBench.
Q4. Does search help more when it retrieves external information?
We use all 300 MMSearch questions in a fixed-corpus retrieval experiment. The corpus contains 298 deduplicated reference webpages, indexed by title, snippet, and page text without question or answer labels. The agent can issue up to three BM25 queries, each returning up to three documents of 4096 characters each. Query images are supplied when available.
MMSearch questions with a fixed local corpus: DeepEyesV2, 300 questions; QA-F1 × 100.
| Condition | What the model receives | QA-F1 |
|---|---|---|
| No retrieval (TF) | Question and query image, when available | 9.1 |
| Agent-generated queries + BM25 | Original inputs, then matches from the fixed local corpus | 22.0 |
| Provided evidence | Original inputs and the corresponding document; no search | 41.9 |
The corresponding reference page is retrieved in 317/900 runs (35.2%). Directly supplying that page removes query selection and provides up to 12288 characters. We also compare relevant evidence with evidence reassigned from another question:
Evidence-swap control on 300 MMSearch questions: Qwen3-VL-8B; single run per condition; QA-F1 × 100.
| Evidence supplied with the question and available query image | QA-F1 |
|---|---|
| None | 15.1 |
| Relevant cached evidence | 52.6 |
| Evidence from another question | 6.2 |
Providing more knowledge and strengthening reasoning are different contributions. Our paper asks whether tool use extends reasoning capability beyond what the model can achieve from the original inputs. Across our visual and reasoning benchmarks, most Full-successful questions also have tool-free solutions. Search delivers clear gains by supplying task-relevant external information; these gains establish the value of information access, rather than by themselves establishing stronger reasoning.
Response to Reviewer 2J6i (Part 1 of 3)
Thank you for the suggestions on generalization, failure modes, and presentation. We added the requested larger model, MARS, and external-information comparisons. We group the overlapping questions below and will reorganize the paper around the findings these experiments establish.
Unless otherwise noted, supplementary decoding experiments use three independent seeds.
W1, Q2 & Q5. Larger models and generalization across tasks and domains
We extend the comparison to Qwen3-VL-32B-Instruct on complete OCRBench and both HRBench resolutions:
Qwen3-VL-32B: complete benchmarks.
The 32B model gains on HR4K, but tool access does not improve OCR or HR8K. Multiple calls occur in 544/3000 OCR Full trajectories, versus 277/3000 for 8B. We also check how many Full-successful questions are solved by TF or our Pure-Text Reasoner (PTR):
Can the same Full-successful questions be solved without tools?
Each percentage uses the Full-correct records in its row as the denominator. TF or PTR means either reference answers the same question correctly.
For 32B on OCRBench, its own TF covers 95.4% of Full successes; adding PTR raises coverage to 99.2%. The 15-benchmark results span natural images, small visual details, OCR, charts, mathematics, and logic. PTR training examples do not overlap with the evaluation benchmarks.
For Q5, we also report cross-benchmark evaluation on TIR-Bench and O3-Bench from the paper:
Additional benchmarks reported in the paper: Full / TF (%).
Full and TF differ by at most 0.8 pp across these two additional benchmarks. This extends the observed pattern beyond the main 15-benchmark suite to further tool-oriented and challenging visual tasks.
MARS (Q3) tests another agent-training recipe, while MMSearch (Q4) tests tasks that benefit from external evidence. Across the additional models and benchmarks, tool-enabled performance remains task-dependent rather than consistently better. On OCRBench, most Full successes are also reproduced by Tool-Free inference for Qwen3-VL-32B and MARS-8B.