- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DeepPlanning-ACE, a benchmark for evaluating planning when user requirements are initially ambiguous, change during interaction, or include infeasible requests. Building on DeepPlanning’s travel and shopping tasks, it constructs verified interaction variants and combines an LLM-based user simulator with deterministic controls over requirement disclosure, event timing, and termination. Evaluation considers both the validity of the final plan and interaction behavior, including clarification, adaptation, and evidence-grounded explanations of infeasibility. Experiments with nine LLMs acting as tool-using agents reveal remaining difficulties in producing valid plans and handling evolving requirements. I consider handling evolving user requirements an important aspect of reliable long-horizon agent planning.
Critical strengths
- The benchmark addresses an important aspect of reliable long-horizon planning, handling user requirements that are initially ambiguous, change during execution, or cannot be satisfied. Its controlled interaction setting provides a useful environment for evaluating whether agents can maintain valid plans as requirements evolve.
- The evaluation covers nine LLMs across travel and shopping tasks and three interaction conditions. Assessing both final-plan validity and interaction behavior provides a broader view of agent performance and captures failures that final-outcome evaluation alone would overlook.
Critical weaknesses
- Some reported comparisons use different evaluation instances. In particular, the shopping Changing condition contains 91 cases for some models and 101 for others due to different generation batches. Although the paper acknowledges this limitation, differences in case composition make the model comparisons difficult to interpret. Evaluation on a shared set of instances would strengthen the empirical conclusions.
Questions for authors
- Could you provide results for all models on the same set of shopping Changing instances under a consistent evaluation protocol?
- What agent framework and time, turn, and token limits were used, and were these settings consistent across models and evaluation batches?
- In the Changing condition, could you compare success rates between episodes with one requirement revision and those with two successive revisions, where the agent must act on the updated requirement before the second revision occurs?
7 comments
Q2:
Thank you for highlighting the need to clarify the agent framework, inference limits, and their consistency across evaluations. In response to your comment, we revised Section 4.1 (Dataset and Inference Protocol) and the Reproducibility statement to make these details explicit.
Both domains use custom function-calling agent loops, as stated in Section 4.1. Specifically, the agent model receives the conversation history and domain-specific tool schemas and generates either tool calls or a user-facing response. The loop executes the requested tools, appends their results to the conversation, and invokes the model again. Travel tools query read-only databases, while shopping tools support product queries and operations on an isolated shopping cart. User-facing responses are handled by a separate user simulator constrained by the deterministic interaction controller. Section 3.3 (Deterministic Interaction Controller) and the appendix “Construction, Controller, and Verification Details” describe authorized disclosures, event ordering, and termination conditions. Completion requires disclosure of all requirements and events followed by a final <plan> or <final_cart> submission; shopping additionally requires a successful cart inspection after the latest modification.
As specified in Section 4.1, the default limits are 1,800 seconds per episode, up to 900 seconds per request, bounded by the remaining episode time, 400 agent-model calls, and eight user replies after the initial request. Tool replies and deterministic termination acknowledgements do not consume user turns. Each generation permits up to 16,384 output tokens, with no separate cumulative token budget.
Regarding consistency, these are shared defaults, but they do not describe every historical attempt. Some historical batches used extended timeouts in response to rate-limit or connection errors, and the assembled results include targeted recovery attempts. We now explicitly acknowledge this qualification in Section 4.1 and the Reproducibility statement. Section 4.2 (Evaluation Metrics) further clarifies that the main table is a retrospective assembled cohort rather than one uniformly executed rerun. The appendix “Main-Cohort Scoring and Execution Provenance” documents the recovery history. Accordingly, we acknowledge that execution differences limit exact replication and controlled comparisons across batches.
Q3:
Thank you for this suggestion. In response, we have added a comparison of single and double requirement revisions in Section 4.4, “Adapting to Successive Requirement Revisions,” and Table 2, with the detailed protocol in Appendix H.
We compare the same 65 shopping tasks across seven agents, using the DeepSeek user simulator. In the single-revision setting, the requirement changes directly from its initial value to the final target. In the double-revision setting, it changes first to an intermediate value and then to the same final target. Both settings share the initial requirement, final target, inventory, budget, and other task constraints.
Crucially, the second revision occurs only after the agent acts on the first updated requirement: it must successfully add a product satisfying the intermediate requirement after receiving the first revision, retain that product in the cart, and remove the superseded initial product. Verbal acknowledgments, failed additions, or actions performed before the first revision do not qualify. These conditions are independently checked by the evaluator.
The following table reports joint success, requiring both final-task correctness and process compliance. All 65 tasks remain in each denominator, including failures, incomplete episodes, and episodes that never reach the second revision.
| Agent | One revision | Two revisions | Difference (pp) | Paired 95% CI (pp) |
|---|---|---|---|---|
| DeepSeek-Flash | 37/65 (56.92%) | 32/65 (49.23%) | −7.69 | [−16.92, 1.54] |
| DeepSeek-V4-Pro | 37/65 (56.92%) | 37/65 (56.92%) | 0.00 | [−10.77, 10.77] |
| GLM-5.3 | 42/65 (64.62%) | 40/65 (61.54%) | −3.08 | [−13.85, 7.69] |
| GLM-5.3-Flash | 29/65 (44.62%) | 23/65 (35.38%) | −9.23 | [−20.00, 1.54] |
| Kimi-K3 | 35/65 (53.85%) | 36/65 (55.38%) | +1.54 | [−7.69, 12.31] |
| Qwen3.8-Flash | 30/65 (46.15%) | 32/65 (49.23%) | +3.08 | [−7.69, 13.85] |
| Qwen3.8-Max | 38/65 (58.46%) | 33/65 (50.77%) | −7.69 | [−16.92, 1.54] |
Differences are calculated as two revisions minus one revision, in percentage points. Confidence intervals use 10,000 paired-task bootstrap resamples. Four agents decline, two improve, and one remains unchanged. All intervals include zero, so the results do not establish a consistent success-rate penalty from adding a second revision.
W1 and Q1:
Thank you for highlighting this issue. We agree that using different generation batches can confound model performance with case composition, making comparisons difficult to interpret. Acknowledging this limitation alone was insufficient. Following your suggestion, we have revised the manuscript to compare all seven models on the same fixed set of 65 shopping Changing instances, replacing the previous comparison with model-dependent denominators of 91 or 101.
The shared set contains 36, 17, and 12 instances from difficulty Levels 1, 2, and 3, respectively. Each instance involves one requirement revision and satisfies the construction checks, including the availability of a valid provisional product and, where applicable, a full-cart budget witness. These dataset counts and the difficulty distribution are now specified in Section 4.1, “Dataset and Inference Protocol”.
All models are compared using the same task definitions, revision rules, and evaluation criteria. The revision is disclosed only after the agent successfully performs relevant tool-based work under the provisional requirement. Shopping success additionally requires replacing the provisional product, removing superseded products, and inspecting the cart after its final modification. The construction and interaction rules are detailed in Appendix F.2, “Controlled Intent Conditions”, with verification criteria in Appendix F.4, “Outcome and Process Verification”.
The updated results are reported separately for the two user simulators. Within each column, all seven models share the same simulator and the same 65 instances:
These results appear in the “Multi-product shopping” portion of Table 1, in the two C columns. The table caption explicitly specifies “65 single-revision tasks” for shopping Changing.
We use shopping-v3.1 for final-task validity and 3i-v3.9 for interaction-process verification. The reported metric is joint success, J=B×P, requiring both a valid final cart and successful completion of the required interaction checks. Every model–simulator cell retains the fixed denominator of 65; missing, failed, incomplete, or unscorable trajectories count as failures. This policy is stated in Section 4.2, “Evaluation Metrics” and the Table 1 caption.