acceptodds
Withdrawn from ICLR 2027

TIDE-Bench: Task-Aware and Diagnostic Evaluation of Tool-Integrated Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tool-integrated reasoning has emerged as a promising paradigm for enhancing large language models with external computation, retrieval, and execution capabilities. However, the field still lacks a high-quality and unified evaluation benchmark, and existing TIR evaluations remain limited in dataset quality, task diversity, diagnostic comprehensiveness, and evaluation efficiency. In this work, we introduce TIDE-Bench, a holistic and efficient benchmark for evaluating TIR methods, featuring three key advantages. First, it provides diverse task settings, combining widely used mathematical reasoning and knowledge-intensive QA tasks with two newly designed tasks—tool-grounded experimental design task and dynamic interactive task—to probe models’ abilities in complex tool invocation and multi-tool coordination. Second, TIDE-Bench adopts a comprehensive yet task-aware evaluation protocol, jointly measuring final answer quality, process reliability, tool-use efficiency, and inference cost across heterogeneous task settings. Third, TIDE-Bench constructs high-quality and discriminative evaluation sets by filtering low-discrimination instances from existing datasets, substantially reducing evaluation cost while focusing on more challenging samples. Extensive experiments on multiple foundation models and TIR methods reveal persistent bottlenecks in tool grounding, offering insights for future TIR research.

Primary area:
datasets and benchmarks
Keywords:
Tool-augmented reasoning, Benchmark, Large language models, Evaluation, Question answering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pRCMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents TIDE-Bench, a unified benchmark for Tool-Integrated Reasoning (TIR), arguing that existing TIR evaluations fall short in data quality, task diversity, diagnostic granularity, and efficiency. It advances TIR evaluation beyond answer accuracy toward whether tools are actually used, and introduces cost awareness, which is directly useful for the iterative development of TIR methods.

Critical strengths

The benchmark's greatest strength is that it adds evaluation of reasoning–answer consistency and tool supportedness, along with two new settings: ExpoDesign and Interaction. ExpoDesign and Interaction test capabilities that GSM8K/HotpotQA cannot cover: multi-tool coordination, goal specification, state tracking, and grounding user-facing replies in tool evidence.

Critical weaknesses

However, the weaknesses are also apparent. The selection of 500 instances may bias the entire benchmark toward samples that a specific model (Qwen3-32B) finds hard, rather than those that are genuinely difficult for other models. The evaluation relies heavily on a single LLM judge, which may compromise reliability: although the authors report human–judge agreement, there is no judge-reliability analysis for the cleaned math/QA accuracy that drives the main comparisons, nor any cross-judge sensitivity check. Finally, the tool-efficiency metric mean(q_i/c_i) is gameable—a single call maximizes it regardless of whether more calls would improve the answer—and it is not monotone in any cost-meaningful sense.

Questions for authors
  1. What is the difference between the post-cleaning Qwen3-32B results in Table 1 and the no-tool Qwen3-32B results in Table 3? Why are the numbers inconsistent between the two tables?
  2. Since the filtering selects the 500 items with the lowest no-tool performance for Qwen3-32B, how do you ensure that the retained sets remain discriminative for models stronger or weaker than Qwen3-32B?
  3. Could you report the per-model accuracy distributions on the filtered sets, and provide a manual audit of, e.g., the Omni-MATH items whose post-cleaning accuracy is 0.00 (are they actually solvable given the provided tools)? Could you also report the scores computed by metrics other than the LLM judge?
  4. Please justify the tool-efficiency metric, how do you prevent it from being trivially maximized by making a single call, and how should it be interpreted when quality is low but calls are few?
  5. Could you provide a table specifying the exact identity and version of the customer simulator model, so that readers can assess reproducibility?
Reviewer_YV2xOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces TIDE-Bench, a benchmark for evaluating Tool-Integrated Reasoning (TIR) in LLMs. It combines existing math and QA datasets with two new tasks: ExpoDesign for experiment design and Interaction to evaluate multi-turn customer service. The paper also introduces more metrics for evaluating the TIR quality.

Critical strengths
  1. The paper studies an important problem: how to evaluate whether LLMs can use tools effectively.

  2. They try to cover multiple domains and propose many different evaluation metrics.

Critical weaknesses
  1. The paper aims to evaluate TIR, but I think it does not clearly enough to show what is the tool-use ability and how it is different from general task-solving ability. For example, in the failed ExpoDesign case, the model writes incorrect Python code because it does not correctly know how to solve the mathematical problem. Does this mean the model has poor tool-use ability, or simply poor mathematical ability? I do not think all task failures should be considered failures of TIR. The authors don't clearly define what TIR ability means or explain how their benchmark distinguishes tool-use errors from other reasoning errors, which makes me confused.

  2. The authors argue that existing benchmarks mainly focus on final answers. However, I believe many previous works have already studied different stages of tool use, including planning, reasoning, and understanding and also evaluting tool using process, such as function-calling correctness, or intermediate milestones in multi-turn tool interactions. Therefore, I do not see what is fundamentally new about the proposed diagnostic metrics, especially given the very small set of tools.

  3. The overall benchmark design lacks a clear focus. The benchmark combines several very different tasks, but I do not understand why these particular tasks were selected. Although they all involve tools, this does not necessarily mean they evaluate the same TIR ability. It's a little bit random. Why are these tasks included these tasks, while other common tool-use tasks, such as coding tasks or enterprise tool calling tasks (e.g., EnterpriseBench), are excluded? The paper needs a clearer explanation of how these tasks fit together and what we can learn by evaluating them in one benchmark.

  4. I am also confused about the construction and evaluation of ExpoDesign, which is not sufficiently explained. The authors mention that research briefs are written by humans or generated by models, but do not sufficiently explain how they are generated or how their quality is checked. The LLM-as-a-judge evaluation is also not clear enough. Although the authors report human–LLM agreement, I would like to see more details about the scoring rubric, concrete scoring examples, and how the judge distinguishes experiment design errors, tool-use errors, and incorrect conclusions. The reliability of the judge for these open-ended reports is already particularly challenging. I am also concerned about possible data leakage. ExpoDesign tasks are derived from existing arXiv papers, and models are allowed to use Search. Although the authors remove paper titles, the research briefs may still contain specific descriptions that allow models to find the original papers, code, or experimental results. In that case, models may simply reproduce existing solutions rather than design experiments themselves. The authors should investigate further how to design this dataset more carefully. More generally, I think designing and validating ExpoDesign itself is already a big research problem that deserves a separate paper to discuss. Including it in this paper without sufficient discussion makes the contribution hard to justify and less convincing.

  5. The novelty of Interaction is also unclear. Interaction is essentially a simulated customer service task where models communicate with customers. I think this is very similar to existing agent benchmarks, like the τ-bench series, which already evaluate multi-turn interactions, tool use, and task completion. I do not see a clear difference between Interaction and these previous works.

Questions for authors

See weaknesses.

Reviewer_UjSyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Tool-Integrated Reasoning (TIR) is a paradigm that combines large language models with external computation, retrieval, and execution capabilities. While TIR has garnered significant attention, the field lacks high-quality unified evaluation benchmarks. Existing TIR evaluations suffer from limitations in dataset quality, task diversity, diagnostic comprehensiveness, and evaluation efficiency. The paper proposes TIDE-Bench, a comprehensive and efficient evaluation benchmark for TIR.

Critical strengths
  1. The proposed ExpoDesign and Interaction fill gaps in “complex tool coordination” and “deployment-oriented interaction scenarios” that existing benchmarks overlook.
  2. Unlike traditional evaluations focusing solely on final accuracy, TIDE-Bench introduces several evaluation dimensions, enabling the benchmark to not only determine “whether the model got it right” but also diagnose “how it got it right” and “whether the cost was justified”.
  3. The paper discusses the efficiency of tool invocation. This is a practically important dimension.
Critical weaknesses
  1. Both new tasks (ExpoDesign and Interaction) rely on LLM judges, which may introduce issues: the model preference bias, credibility of automated judgment. Appendix D.2 reports human-LLM consistency of only ~60-70%, meaning 30-40% of judgments disagree between humans and LLMs. The paper does not discuss this risk sufficiently.

  2. The work uses Qwen3-32B as the evaluator for data cleaning. This may introduce the model bias. The cleaning process may favor sample types that Qwen3-32B is good at solving, and other models may perform differently on the filtered dataset. The paper does not discuss this potential bias or its impact on benchmark generality.

  3. Evaluating general models (Qwen, GPT series) alongside TIR-specific fine-tuned models (SearchR1, Tool-Star) in the same benchmark poses issues. Their advantages may reflect training strategies rather than base model capabilities.

  4. The paper’s layout needs optimization. for example, excessive whitespace between Figure 1 and the body text, considerable blank space appears before and after the Related Work section (Section 2).

  5. The contributions are currently too lengthy and could benefit from consolidation: For example, Contribution 4 (systematic experiments) could reasonably be subsumed under Contribution 1 (the benchmark itself), since proposing a benchmark inherently entails extensive experiments and analysis.

  6. The beginning of the Method section devotes considerable space to discussing limitations, which is more appropriately placed in the Introduction or Appendix.

  7. A small error that should be corrected. The caption of Figure 2 contains the awkward expression “pipeline of pipeline of”

  8. In the experimental section, methods used for comparison should be properly cited with references.

Questions for authors

Please see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.