- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper presents TIDE-Bench, a unified benchmark for Tool-Integrated Reasoning (TIR), arguing that existing TIR evaluations fall short in data quality, task diversity, diagnostic granularity, and efficiency. It advances TIR evaluation beyond answer accuracy toward whether tools are actually used, and introduces cost awareness, which is directly useful for the iterative development of TIR methods.
Critical strengths
The benchmark's greatest strength is that it adds evaluation of reasoning–answer consistency and tool supportedness, along with two new settings: ExpoDesign and Interaction. ExpoDesign and Interaction test capabilities that GSM8K/HotpotQA cannot cover: multi-tool coordination, goal specification, state tracking, and grounding user-facing replies in tool evidence.
Critical weaknesses
However, the weaknesses are also apparent. The selection of 500 instances may bias the entire benchmark toward samples that a specific model (Qwen3-32B) finds hard, rather than those that are genuinely difficult for other models. The evaluation relies heavily on a single LLM judge, which may compromise reliability: although the authors report human–judge agreement, there is no judge-reliability analysis for the cleaned math/QA accuracy that drives the main comparisons, nor any cross-judge sensitivity check. Finally, the tool-efficiency metric mean(q_i/c_i) is gameable—a single call maximizes it regardless of whether more calls would improve the answer—and it is not monotone in any cost-meaningful sense.
Questions for authors
- What is the difference between the post-cleaning Qwen3-32B results in Table 1 and the no-tool Qwen3-32B results in Table 3? Why are the numbers inconsistent between the two tables?
- Since the filtering selects the 500 items with the lowest no-tool performance for Qwen3-32B, how do you ensure that the retained sets remain discriminative for models stronger or weaker than Qwen3-32B?
- Could you report the per-model accuracy distributions on the filtered sets, and provide a manual audit of, e.g., the Omni-MATH items whose post-cleaning accuracy is 0.00 (are they actually solvable given the provided tools)? Could you also report the scores computed by metrics other than the LLM judge?
- Please justify the tool-efficiency metric, how do you prevent it from being trivially maximized by making a single call, and how should it be interpreted when quality is low but calls are few?
- Could you provide a table specifying the exact identity and version of the customer simulator model, so that readers can assess reproducibility?