acceptodds
Under review as a conference paper at ICLR 2027

ToolPlan-Bench: Diagnosing Hidden Planning Failures in Long-Horizon Tool-Calling Agents Beyond Success Rate

Abstract

Large language model (LLM) agents are increasingly deployed on long-horizon, tool-calling tasks, yet they are almost universally evaluated by a single black-box outcome: task success rate. We argue that success rate systematically hides planning failures. An agent can reach the correct final state through a flawed plan—taking redundant actions, executing operations out of order, skipping required preconditions, or stopping prematurely on easy instances—and still be scored as a success. We introduce ToolPlan-Bench, a diagnostic benchmark that decomposes agent behaviour into a structured taxonomy of seven planning-failure modes and measures the right-outcome-wrong-plan rate (ROWP): the fraction of successful trajectories that nonetheless carry a planning-failure label. Built on the real tasks, tools, and executable databases of -bench (retail and airline domains), ToolPlan-Bench evaluates six frontier models on 990 live agent trajectories. We find that ROWP is large and statistically robust—the bootstrap 95% lower bound exceeds 0.23 for every model, meaning at least one in four successful trajectories hides a planning flaw. ROWP rises monotonically with task complexity (from 0.43 to 0.57 as the number of required write operations grows), and 60% of trajectories diverge from the gold plan at the very first step. Five independent human annotators validate our rule-based labels (Cohen's between the human majority and the rule labels on the core label; on the structural dimensions underlying ROWP). Finally, we show that a natural intervention—prompting the agent to plan before acting—can significantly change success rate (in both directions across models) yet has no significant effect on ROWP for any model, demonstrating that the planning failures ToolPlan-Bench measures are robust to naive prompting and constitute an open challenge.

Primary area:
datasets and benchmarks
Keywords:
tool-calling agents, planning failures, diagnostic benchmark, agent evaluation, long-horizon planning, LLM agents, right-outcome-wrong-plan

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_hpdqOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ToolPlan-Bench, a diagnostic benchmark for identifying process flaws that task success rates may overlook in tool-calling agents. Built on τ-bench’s retail and airline domains, it defines seven failure categories and introduces ROWP, the proportion of successful trajectories flagged as containing planning flaws. Experiments on 990 trajectories from six models, supplemented by human annotation and intervention studies, report substantial ROWP across models and no statistically significant reductions from the tested planning prompts or execution-time guardrails.

Critical strengths
  1. The paper examines execution issues that final task success rates may overlook, particularly incorrect tool calls, skipped preconditions, and redundant operations in successful trajectories. This perspective provides a useful complement to existing outcome-based evaluations.

  2. The experimental evaluation is comprehensive. The authors evaluate six models across two executable domains, collecting 990 baseline trajectories, and conduct human annotation, rule sensitivity analyses, and two types of intervention experiments.

Critical weaknesses
  1. Several error labels rely on a single reference action sequence, but deviating from that sequence does not necessarily violate task requirements. For example, independent operations may be reordered, necessary information may be obtained through alternative tools, and identical queries before and after a state change may serve a verification purpose. How are these cases handled? I am concerned that the current ROWP metric may conflate genuine defects, efficiency issues, and valid alternative execution paths. The rule perturbations in §4.4 do not directly assess false-positive rates in these cases. Moreover, human annotators follow the same definitions and consult the gold plan, so their agreement with the rule-based labels does not independently establish the validity of the definitions themselves.

  2. The paper overinterprets statistically nonsignificant results. In several places, the absence of a statistically significant change in ROWP is interpreted as evidence that planning prompts or execution-time checks cannot effectively address the problem. For example, the pooled guardrail experiment for GPT-5.2 reports a ΔROWP of −0.126 with a 95% CI of [−0.255, 0.004]. This result supports uncertainty about the magnitude of the improvement, but does not establish that the intervention is ineffective.

  3. As tasks involve more operations, the probability of observing at least one error increases even if the per-operation error rate remains constant. Therefore, an increase in ROWP alone does not establish that planning ability deteriorates with task complexity. Similarly, a mismatch between the first action and the reference sequence does not demonstrate that the model formed an incorrect plan from the outset. Controlling for trajectory length, reporting per-operation error rates, and analyzing recovery after divergence would provide stronger support for these conclusions.

Questions for authors

See Weakness

Reviewer_A5bSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that success rates can mask underlying planning issues. To address this, the authors use a rule-based system to categorize failed trajectories on the τ-bench benchmark into seven distinct types. They introduce the metric "ROWP"—representing the proportion of successful trajectories where the outcome is correct but the process contains errors. Evaluating six different models, the study finds that at least one-quarter of successful trajectories fall into this category; the authors further note that these rule-based labels were verified by five human annotators.

Critical strengths

The paper is clearly written and easy to follow. The supplementary material is fairly complete (code, trajectories, and intermediate results), which makes the results checkable.

Critical weaknesses
  1. The human annotation is not real. The paper says five people validated the labels, but the supplement contains a script that simulates these five annotators. I re-ran it and the output exactly matches their submitted files.
  2. The rules are too strict. Basically, if the agent doesn't do exactly what the reference answer does, it gets flagged. For example, if it looks up the user with a different tool than the reference uses, that counts as a "missing precondition", even though it got the same info. The paper says only key steps are checked, but I don't see that anywhere in the code. Once these cases are excluded, the "one in four" claim no longer holds.
  3. This question has also been studied before (PAE, Cao et al., 2026), which the paper does not cite.
Questions for authors

The five annotator files in the supplement are identical to the output of generate_5_annotators.py. Were these annotators real people or simulated?

Reviewer_evbUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Current benchmarks mainly focus on the task success rate. However, even if the agent finishes the final task, the intermediate process may still have planning failures. Only using the task success rate may bury this. To mitigate this issue, it proposes a ToolPlan-Bench which is based on the \tau-bench. It has seven planning-failure modes, gold action sequence to diagnose the agent trajectories. It defines ROWP (right-Outcome-Wrong-Plan) to measure such failures. It also has human labelers and shows the alignment with them.

Critical strengths

1.The paper identifies an important limitation of existing agent evaluations, where task success rate may overlook planning failures in otherwise successful trajectories.

2.The proposed seven failure categories and ROWP metric provide a structured way to diagnose agent behavior beyond final task outcomes.

3.The paper evaluates six models across 990 trajectories and includes human validation of the rule-based labels.

Critical weaknesses

1.The evaluation compares agent trajectories against a single reference plan, although multiple valid execution paths may exist. Deviations from the gold plan do not necessarily indicate planning failures.

  1. The paper identifies seven planning failure modes, but the rationale for selecting these specific categories is unclear. How comprehensive is this taxonomy, and could it miss other important types of planning failures?

  2. Some failure categories, such as redundant calls or wrong arguments, may reflect inefficiency or tool-use errors rather than planning failures

  3. The benchmark only covers two domains from τ-bench in a single-turn setting, without interactive users. It is unclear how well the findings generalize to more complex, dynamic, or long-horizon agent tasks.

Questions for authors

1.The paper evaluates a plan-check prompt, which may not adequately address more complex failures that require execution-time monitoring or replanning. While this prompt does not significantly reduce ROWP, the results provide limited evidence about the effectiveness of other, more structured planning approaches.

and please also refer to weakness

Reviewer_mPF9OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper address the problem that task success rate hides plannning failures in tool agent tasks. The author proposes ToolPlan-Bench based on τ-bench that compares agent trajectory with golden action and assigns 7 planning-failure labels. Then the authors defines a headline metric ROWP to calculate the fraction of successful trajectories that carry at least one such failures. The results of six frontier models shows that (1) ROWP is robust (2) ROWP grows with planning complexity, (3)Models have distinct failure signatures. (4)Method such as a plan-before-acting prompt cannot fix ROWP and planning failures remains a open problem.

Critical strengths
  1. The problem is well-defined. Benchmarks such as τ-bench scores only the final success rate. Detecting intermediate failures such as planning failures is important and useful for community and researcher. It also clearly demonstrates the 7 planning-failure modes in tool agent tasks, which maybe helpful for future researcher to detect similair error taxonomy.

  2. Simple and explicit metric. The paper uses ROWP to systematically quantify successful trajectory contains planning failures. It is cheap to reproduce and provides a useful metric for process-level evaluation of tool-calling agents. According to experiment results it seems like ROWP is consistent across 6 frontier models.

Critical weaknesses
  1. My main concern is about novelty. Evaluating the intermediate process instead of final outcome is a well-established direction. For example, Agent GPA (arXiv:2510.08847) evaluates agents in Goal, Plan, and Action. A run that reaches its goal can therefore still be penalized for a poor or unfollowed plan, which is close to the paper's planning-failure taxonomy. AgentProcessBench (arXiv:2603.14465) collects human step-level labels (correct / neutral / incorrect) for 1,000 tool-use trajectories, including τ²-Bench. It also shows that successful trajectories still contain incorrect steps (8.0% of steps in successful τ²-Bench trajectories are labeled incorrect). None of these is discussed. Also, your main method to add 7 planning failure patterns with gold sequence upon τ-bench seems incremental.

  2. The construction of planning failures seems problematic. All planning failures seems to rely on one single golden sequence. Many tasks admit several valid paths. A trajecotry can diverge but remains correct. On successful trajectories, most labels flag a skipped read, a repeated call, or a failed write followed by a corrected one. These are inefficiencies or error recovery, yet the paper treats them as planning failures without evidence. The authors should report each label's rate on successful trajectories. They should also check whether successful runs that carry a failure label succeed less often when the task is rerun (pass^k) or made harder.

  3. Inter-annotator agreement on the core label is low (Fleiss κ = 0.58). The sample is also small (n = 48) and was selected using the rule labels, and κ is not reported on the successful trajectories that ROWP depends on.

  4. The readbility of intro seems weak. Please consider rephase them.

Questions for authors
  1. How does ToolPlan-Bench differ from existing process-level evaluations such as Agent GPA and AgentProcessBench?
  2. How do you handle tasks that admit more than one valid action sequence? The authors should report each label's rate on successful trajectories. They should also check whether successful runs that carry a failure label succeed less often when the task is rerun (pass^k) or made harder.
  3. Please address the weakness 3 about inter-annotator.

Comments on the paper

1 comment

The author-reviewer discussion period will end soon. Please review and acknowledge the authors’ responses. If your concerns have been addressed, please reconsider your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.