- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces ToolPlan-Bench, a diagnostic benchmark for identifying process flaws that task success rates may overlook in tool-calling agents. Built on τ-bench’s retail and airline domains, it defines seven failure categories and introduces ROWP, the proportion of successful trajectories flagged as containing planning flaws. Experiments on 990 trajectories from six models, supplemented by human annotation and intervention studies, report substantial ROWP across models and no statistically significant reductions from the tested planning prompts or execution-time guardrails.
Critical strengths
-
The paper examines execution issues that final task success rates may overlook, particularly incorrect tool calls, skipped preconditions, and redundant operations in successful trajectories. This perspective provides a useful complement to existing outcome-based evaluations.
-
The experimental evaluation is comprehensive. The authors evaluate six models across two executable domains, collecting 990 baseline trajectories, and conduct human annotation, rule sensitivity analyses, and two types of intervention experiments.
Critical weaknesses
-
Several error labels rely on a single reference action sequence, but deviating from that sequence does not necessarily violate task requirements. For example, independent operations may be reordered, necessary information may be obtained through alternative tools, and identical queries before and after a state change may serve a verification purpose. How are these cases handled? I am concerned that the current ROWP metric may conflate genuine defects, efficiency issues, and valid alternative execution paths. The rule perturbations in §4.4 do not directly assess false-positive rates in these cases. Moreover, human annotators follow the same definitions and consult the gold plan, so their agreement with the rule-based labels does not independently establish the validity of the definitions themselves.
-
The paper overinterprets statistically nonsignificant results. In several places, the absence of a statistically significant change in ROWP is interpreted as evidence that planning prompts or execution-time checks cannot effectively address the problem. For example, the pooled guardrail experiment for GPT-5.2 reports a ΔROWP of −0.126 with a 95% CI of [−0.255, 0.004]. This result supports uncertainty about the magnitude of the improvement, but does not establish that the intervention is ineffective.
-
As tasks involve more operations, the probability of observing at least one error increases even if the per-operation error rate remains constant. Therefore, an increase in ROWP alone does not establish that planning ability deteriorates with task complexity. Similarly, a mismatch between the first action and the reference sequence does not demonstrate that the model formed an incorrect plan from the outset. Controlling for trajectory length, reporting per-operation error rates, and analyzing recovery after divergence would provide stronger support for these conclusions.
Questions for authors
See Weakness
The author-reviewer discussion period will end soon. Please review and acknowledge the authors’ responses. If your concerns have been addressed, please reconsider your score.