acceptodds
Under review as a conference paper at ICLR 2027

DeepPlanning-ACE: Can Language Agents Ace Interactive Long-Horizon Planning?

Abstract

Long-horizon planning benchmarks typically assume that user requirements are complete, unambiguous, and fixed from the outset. Real-world planning, however, is inherently interactive: users disclose constraints gradually, clarify underspecified preferences, revise earlier decisions, and sometimes make requests for which no valid solution exists. We introduce DeepPlanning-ACE, named after capabilities essential to interactive planning: resolving Ambiguity, adapting to Change, and Explaining infeasibility. This also captures the benchmark's central question: can language agents truly ace deep planning as user intent evolves through interaction? DeepPlanning-ACE comprises bilingual episodes derived from verifiable base instances in travel planning and multi-product shopping. It preserves the original tasks and constructs ambiguous, changing, and infeasible variants when their defining conditions can be verified. To enable controlled and reproducible user simulation, the benchmark decouples semantic intent from linguistic realization. User requirements, intent changes, and infeasibility conditions are specified and verified in advance. A deterministic controller governs information disclosure, state transitions, turn limits, and termination, while a language model can select from authorized disclosures without modifying their canonical content. Evaluation covers both task completion and interaction quality along the ACE dimensions: clarifying ambiguous requirements, adapting to changing constraints, and explicitly explaining infeasibility. Experiments across nine language agents show that multi-turn interactive planning remains challenging for all evaluated models, revealing persistent limitations in maintaining valid plans, adapting to changing requirements, and explaining infeasibility with sufficient tool evidence.

Primary area:
datasets and benchmarks
Keywords:
Travel Planning, Long-horizon planning benchmarks, DeepPlanning, Agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_b1rgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DeepPlanning-ACE, a benchmark for evaluating planning when user requirements are initially ambiguous, change during interaction, or include infeasible requests. Building on DeepPlanning’s travel and shopping tasks, it constructs verified interaction variants and combines an LLM-based user simulator with deterministic controls over requirement disclosure, event timing, and termination. Evaluation considers both the validity of the final plan and interaction behavior, including clarification, adaptation, and evidence-grounded explanations of infeasibility. Experiments with nine LLMs acting as tool-using agents reveal remaining difficulties in producing valid plans and handling evolving requirements. I consider handling evolving user requirements an important aspect of reliable long-horizon agent planning.

Critical strengths
  1. The benchmark addresses an important aspect of reliable long-horizon planning, handling user requirements that are initially ambiguous, change during execution, or cannot be satisfied. Its controlled interaction setting provides a useful environment for evaluating whether agents can maintain valid plans as requirements evolve.
  2. The evaluation covers nine LLMs across travel and shopping tasks and three interaction conditions. Assessing both final-plan validity and interaction behavior provides a broader view of agent performance and captures failures that final-outcome evaluation alone would overlook.
Critical weaknesses
  1. Some reported comparisons use different evaluation instances. In particular, the shopping Changing condition contains 91 cases for some models and 101 for others due to different generation batches. Although the paper acknowledges this limitation, differences in case composition make the model comparisons difficult to interpret. Evaluation on a shared set of instances would strengthen the empirical conclusions.
Questions for authors
  1. Could you provide results for all models on the same set of shopping Changing instances under a consistent evaluation protocol?
  2. What agent framework and time, turn, and token limits were used, and were these settings consistent across models and evaluation batches?
  3. In the Changing condition, could you compare success rates between episodes with one requirement revision and those with two successive revisions, where the agent must act on the updated requirement before the second revision occurs?
7 comments
AuthorsOpenReview ↗

W1 and Q1:

Thank you for highlighting this issue. We agree that using different generation batches can confound model performance with case composition, making comparisons difficult to interpret. Acknowledging this limitation alone was insufficient. Following your suggestion, we have revised the manuscript to compare all seven models on the same fixed set of 65 shopping Changing instances, replacing the previous comparison with model-dependent denominators of 91 or 101.

The shared set contains 36, 17, and 12 instances from difficulty Levels 1, 2, and 3, respectively. Each instance involves one requirement revision and satisfies the construction checks, including the availability of a valid provisional product and, where applicable, a full-cart budget witness. These dataset counts and the difficulty distribution are now specified in Section 4.1, “Dataset and Inference Protocol”.

All models are compared using the same task definitions, revision rules, and evaluation criteria. The revision is disclosed only after the agent successfully performs relevant tool-based work under the provisional requirement. Shopping success additionally requires replacing the provisional product, removing superseded products, and inspecting the cart after its final modification. The construction and interaction rules are detailed in Appendix F.2, “Controlled Intent Conditions”, with verification criteria in Appendix F.4, “Outcome and Process Verification”.

The updated results are reported separately for the two user simulators. Within each column, all seven models share the same simulator and the same 65 instances:

ModelQwen3.8-Flash userDeepSeek-Flash user
DeepSeek-V4-Pro44/65 (67.7%)37/65 (56.9%)
GLM-5.3-Flash33/65 (50.8%)29/65 (44.6%)
Qwen3.8-Flash35/65 (53.8%)30/65 (46.2%)
DeepSeek-Flash36/65 (55.4%)37/65 (56.9%)
GLM-5.338/65 (58.5%)42/65 (64.6%)
Kimi-K335/65 (53.8%)35/65 (53.8%)
Qwen3.8-Max32/65 (49.2%)38/65 (58.5%)

These results appear in the “Multi-product shopping” portion of Table 1, in the two C columns. The table caption explicitly specifies “65 single-revision tasks” for shopping Changing.

We use shopping-v3.1 for final-task validity and 3i-v3.9 for interaction-process verification. The reported metric is joint success, J=B×PJ=B\times P, requiring both a valid final cart and successful completion of the required interaction checks. Every model–simulator cell retains the fixed denominator of 65; missing, failed, incomplete, or unscorable trajectories count as failures. This policy is stated in Section 4.2, “Evaluation Metrics” and the Table 1 caption.

Thank you for highlighting the need to clarify the agent framework, inference limits, and their consistency across evaluations. In response to your comment, we revised Section 4.1 (Dataset and Inference Protocol) and the Reproducibility statement to make these details explicit.

Both domains use custom function-calling agent loops, as stated in Section 4.1. Specifically, the agent model receives the conversation history and domain-specific tool schemas and generates either tool calls or a user-facing response. The loop executes the requested tools, appends their results to the conversation, and invokes the model again. Travel tools query read-only databases, while shopping tools support product queries and operations on an isolated shopping cart. User-facing responses are handled by a separate user simulator constrained by the deterministic interaction controller. Section 3.3 (Deterministic Interaction Controller) and the appendix “Construction, Controller, and Verification Details” describe authorized disclosures, event ordering, and termination conditions. Completion requires disclosure of all requirements and events followed by a final <plan> or <final_cart> submission; shopping additionally requires a successful cart inspection after the latest modification.

As specified in Section 4.1, the default limits are 1,800 seconds per episode, up to 900 seconds per request, bounded by the remaining episode time, 400 agent-model calls, and eight user replies after the initial request. Tool replies and deterministic termination acknowledgements do not consume user turns. Each generation permits up to 16,384 output tokens, with no separate cumulative token budget.

Regarding consistency, these are shared defaults, but they do not describe every historical attempt. Some historical batches used extended timeouts in response to rate-limit or connection errors, and the assembled results include targeted recovery attempts. We now explicitly acknowledge this qualification in Section 4.1 and the Reproducibility statement. Section 4.2 (Evaluation Metrics) further clarifies that the main table is a retrospective assembled cohort rather than one uniformly executed rerun. The appendix “Main-Cohort Scoring and Execution Provenance” documents the recovery history. Accordingly, we acknowledge that execution differences limit exact replication and controlled comparisons across batches.

Thank you for this suggestion. In response, we have added a comparison of single and double requirement revisions in Section 4.4, “Adapting to Successive Requirement Revisions,” and Table 2, with the detailed protocol in Appendix H.

We compare the same 65 shopping tasks across seven agents, using the DeepSeek user simulator. In the single-revision setting, the requirement changes directly from its initial value to the final target. In the double-revision setting, it changes first to an intermediate value and then to the same final target. Both settings share the initial requirement, final target, inventory, budget, and other task constraints.

Crucially, the second revision occurs only after the agent acts on the first updated requirement: it must successfully add a product satisfying the intermediate requirement after receiving the first revision, retain that product in the cart, and remove the superseded initial product. Verbal acknowledgments, failed additions, or actions performed before the first revision do not qualify. These conditions are independently checked by the evaluator.

The following table reports joint success, requiring both final-task correctness and process compliance. All 65 tasks remain in each denominator, including failures, incomplete episodes, and episodes that never reach the second revision.

AgentOne revisionTwo revisionsDifference (pp)Paired 95% CI (pp)
DeepSeek-Flash37/65 (56.92%)32/65 (49.23%)−7.69[−16.92, 1.54]
DeepSeek-V4-Pro37/65 (56.92%)37/65 (56.92%)0.00[−10.77, 10.77]
GLM-5.342/65 (64.62%)40/65 (61.54%)−3.08[−13.85, 7.69]
GLM-5.3-Flash29/65 (44.62%)23/65 (35.38%)−9.23[−20.00, 1.54]
Kimi-K335/65 (53.85%)36/65 (55.38%)+1.54[−7.69, 12.31]
Qwen3.8-Flash30/65 (46.15%)32/65 (49.23%)+3.08[−7.69, 13.85]
Qwen3.8-Max38/65 (58.46%)33/65 (50.77%)−7.69[−16.92, 1.54]

Differences are calculated as two revisions minus one revision, in percentage points. Confidence intervals use 10,000 paired-task bootstrap resamples. Four agents decline, two improve, and one remains unchanged. All intervals include zero, so the results do not establish a consistent success-rate penalty from adding a second revision.

Reviewer_LvvJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DeepPlanning-ACE, a benchmark for evaluating interactive planning under ambiguous requirements, changing user preferences, and infeasible requests. It extends travel planning and shopping tasks with controlled interaction scenarios and uses a rule-based controller to manage information disclosure and user revisions. Evaluation checks both the final plan and the interaction process. Experiments with nine agents show that travel performance is mainly limited by plan validity, while shopping tasks also expose failures in handling revisions and explaining infeasibility.

Critical strengths
  • This paper addresses a timely and important problem: the benchmark studies how agents handle unclear requests, changing requirements, and infeasible goals, all of which matter in practical planning.
  • The evaluation checks both outcomes and interaction processes. Beyond final-plan correctness, it checks whether agents ask clarification questions, respond to revisions, and support infeasibility explanations with tool evidence.
  • The evaluation uses a controlled and verifiable environment. Scenarios are checked against the underlying data, and a rule-based controller manages information disclosure and event order.
  • The paper also offers useful failure analysis. The experiments cover nine agents across travel and shopping tasks, identifying concrete failures such as invalid itineraries, incomplete cart updates, and unsupported infeasibility claims.
Critical weaknesses
  1. For the experiment design, there is limited discrimination of interaction capabilities in the primary metric. In travel ambiguity and change, base and joint success counts are identical; in shopping ambiguity, only 2 of 608 base-correct outcomes fail the process check. We can see that in these settings, joint success is almost entirely determined by final-plan validity, with process verification providing little additional discrimination. While this does not imply that interaction is unnecessary, the reported results do not sufficiently establish whether the primary metric distinguishes interaction capabilities from underlying planning competence. I wonder if the author could reporting per-model process success rates alongside the full breakdown of base and process outcomes?

  2. The design of the ACE scenarios may be oversimplified and template-based:

  • Ambiguity is only presented as an explicit choice between two options, such as “the 12th or the 13th.” Models can easily spot this and ask a question. Could the benchmark also support more free-form requests, where agents must identify missing or unclear details themselves?
  • The paper does not test over-clarification. For example, are the agents penalized for asking unnecessary questions?
  • Template coverage is narrow. Travel ambiguity only concerns departure dates, while changes only concern hotel star ratings, budgets, and room counts, etc. This undermines the ecological validity of the benchmark.
  • User messages use precomputed canonical text, which limits their naturalness. Their naturalness is also not validated by humans.
  1. Experiment reliability and counfounding factors
  • Timeouts make the results hard to interpret. MiMo-V2.6-Pro leaves 53/120 shopping ambiguity cases incomplete, including 45 with explicit time limits. Qwen3.8-Max times out on 64/120 Chinese travel normal cases, which partly explains why its ambiguity score exceeds its normal score. The results do not clearly separate planning and interaction failures from runtime limits.
  • The evaluation consitions are not fully consistent. For example, Shopping change uses different case sets across models (91 vs. 101 cases). The evaluation also mixes old and new trajectories, and one GLM run required a metadata correction.
  • Uncertainty is not reported. The paper can be strengthened by reporting repeated-run results, variance, or confidence intervals. This makes it easier to judge the stability of small score differences.
  • Some large performance gaps need deeper dives. For instance, Kimi-K3 scores 0% on Chinese travel ambiguity but 58.3% on English travel ambiguity. The paper may need further analysis and manual checks would help explain this gap, especially since the two languages use different cases.
  1. There can be potential user simulator bias. DeepSeek-Flash serves as both the user simulator and an evaluated agent, and it ranks first across all interactive shopping conditions. This raises the possibility that the simulator favors agents with a similar interaction style. An ablation using a different simulator would help test this concern.
  2. Missing details for reproducibility
  • The paper does not clearly report turn limits, time limits, sampling parameters, or tool-call budgets. Since many cases fail due to timeouts, these settings are especially important for interpreting and reproducing the results.
  • The reproducibility statement mentions model architecture and training procedures, which do not match this study. It also claims to report random seeds, but these details are missing from the provided paper. The statement should accurately reflect the experimental settings that are actually disclosed.
Questions for authors

See weaknesses.

13 comments

Thank you for this suggestion. In response, we have revised Section 4.2 (Evaluation Metrics) and added Table 7 in Appendix B.1 (Reporting Protocol) to report all seven models’ process success rates under both the Qwen3.8-Flash and DeepSeek-Flash user simulators. The complete process-rate results are reproduced below.

Process success measures whether an episode passes all required interaction checks, regardless of whether its final plan passes the base-task evaluator. Each rate is the number of process-passing episodes divided by the fixed number of generated episodes. CN and EN denote Chinese and English travel; S denotes shopping. A, C, and E denote ambiguity, change, and infeasibility, respectively. For each agent and user simulator, each condition contains 120 episodes, except travel E (55) and shopping C (65). Missing, failed, incomplete, and unscorable episodes remain in the denominators. Normal episodes have no interaction-process score and are excluded.

Process success rates (%) with the Qwen3.8-Flash user simulator

ModelCN-ACN-CCN-EEN-AEN-CEN-ES-AS-CS-E
DeepSeek-V4-Pro80.081.769.187.595.098.289.287.790.8
GLM-5.3-Flash97.596.781.897.586.774.595.875.469.2
Qwen3.8-Flash91.787.576.499.2100.096.495.878.581.7
DeepSeek-Flash95.894.283.678.380.090.996.778.561.7
GLM-5.396.794.292.793.395.890.990.884.663.3
Kimi-K383.397.567.390.091.781.889.273.878.3
Qwen3.8-Max98.395.072.798.399.287.370.063.185.8

Process success rates (%) with the DeepSeek-Flash user simulator

ModelCN-ACN-CCN-EEN-AEN-CEN-ES-AS-CS-E
DeepSeek-V4-Pro100.099.274.595.0100.089.194.283.163.3
GLM-5.3-Flash100.0100.085.5100.0100.092.795.076.992.5
Qwen3.8-Flash95.0100.078.274.2100.098.276.773.882.5
DeepSeek-Flash100.0100.092.799.2100.092.797.578.598.3
GLM-5.3100.0100.092.7100.0100.0100.097.584.680.8
Kimi-K393.393.367.3100.099.290.991.780.077.5
Qwen3.8-Max99.297.581.8100.099.290.978.376.990.8

These results show that process success varies across models and conditions under both simulators. For example, shopping-infeasibility process success ranges from 61.7% to 90.8% with Qwen users and from 63.3% to 98.3% with DeepSeek users. For Qwen3.8-Flash as the agent, English-travel ambiguity process success is 99.2% with Qwen users and 74.2% with DeepSeek users.

Locations of the manuscript revisions: Section 4.2, “Evaluation Metrics,” now explicitly points to the per-model process rates for both user simulators. Appendix B.1, “Reporting Protocol,” explains their scope and denominator treatment. Its new Table 7, “Process success rates (%) for all seven agents under both user simulators” (p. 18), contains all 126 rates reported above, including the Qwen-user results.

AuthorsOpenReview ↗

W2.1: Ambiguity is only presented as an explicit choice between two options, such as “the 12th or the 13th.” Models can easily spot this and ask a question. Could the benchmark also support more free-form requests, where agents must identify missing or unclear details themselves?

We agree that recognizing an explicit alternative differs from independently identifying missing information. Our clarification-form study addresses this distinction through Omit: the target requirement is removed entirely, without candidate answers or any indication that information is missing. Only a field-specific question before the first final submission unlocks the answer; generic requests, guesses, and turn exhaustion do not. This is reported in Section 4.5 and Table 3, with the protocol in Appendix C.

The study compares Hint (candidates plus a clarification instruction), Choice (candidates without that instruction), Omit (target field withheld), and Full (canonical value supplied initially). Other requirements and the final goal remain fixed. Three models evaluate ten shared cases per setting across Chinese travel, English travel, and shopping, yielding 360 records. Targets include party size, room count, and hotel star rating in travel, and brand, color, and season in shopping. Appendix C.3, Table 12 provides a concrete four-condition example.

Each entry below is a count out of 10. J denotes joint task-and-process success, including no-overclarification; B, base-task success; Q, targeted clarification; and F, runtime failure. Full reports B because elicitation is unnecessary.

SettingModelHint JChoice JOmit JFull BOmit QOmit F
Chinese travelDeepSeek-Flash334682
Chinese travelGLM-5.3125673
Chinese travelQwen3.8-Max424763
English travelDeepSeek-Flash633873
English travelGLM-5.3853853
English travelQwen3.8-Max8461073
ShoppingDeepSeek-Flash7207010
ShoppingGLM-5.38607010
ShoppingQwen3.8-Max523846

Omit records successful discovery of unannounced information gaps: targeted clarification occurs in 21/30 Chinese-travel, 19/30 English-travel, and 4/30 shopping episodes, with joint successes of 13/30, 12/30, and 3/30. The distinction matters: asking the right question does not guarantee a valid final plan. The pooled counts combine three models evaluated on ten shared cases per setting. Full results appear in Appendix C.5, Tables 13–15.

Removing the clarification instruction (Hint → Choice) changes pooled joint success from 8 to 7, 22 to 12, and 20 to 10 out of 30 in Chinese travel, English travel, and shopping, respectively. Removing the candidates and target field (Choice → Omit) changes it from 7 to 13, 12 to 12, and 10 to 3. Thus, omission does not produce a uniform decline. On the same base-task metric, Omit versus Full yields 14 versus 19, 15 versus 26, and 3 versus 22 successes, each out of 30. These comparisons are descriptive: execution conditions varied, and shopping Omit includes 26/30 runtime-failed records. Model-level paired differences and 95% bootstrap intervals are in Appendix C.6, Table 16.

The field-sensitivity study (Section 4.6, Table 4; Appendix A) provides complementary evidence by extending ambiguity beyond departure dates. Each ambiguity pair preserves the final goal and admits fully feasible interpretations with disjoint valid solutions. On 15 matched cases per condition, room-count/star-rating joint successes are 6/5 for DeepSeek-Flash, 8/12 for GLM-5.3, and 9/6 for Qwen3.8-Max. Shopping attribute/quantity successes are 10/12, 9/11, and 7/7, respectively. These paired counts show that field sensitivity varies across models: GLM-5.3 improves from room count to star rating, whereas the other two decline; quantity improves shopping results for two models. This study broadens field coverage while retaining explicit candidates; Omit specifically removes that cue. Omit therefore evaluates discovering an unannounced missing requirement, while vague expressions, ambiguous references, and multiple interacting omissions remain beyond these controlled tests.

AuthorsOpenReview ↗

W2.2: The paper does not test over-clarification. For example, are the agents penalized for asking unnecessary questions?

Thank you for this suggestion. In response to your comment, we have revised the manuscript to explicitly evaluate and penalize over-clarification in all ambiguous tasks. The revisions are documented in the following locations:

  • Section 3.2, “Controlled Intent Conditions” defines strict over-clarification as unnecessary requests for already known, delegated, tool-retrievable, or irrelevant information.
  • Section 3.4, “Outcome and Process Verification” specifies that such violations cause process success P=0P=0, yielding joint success J=BP=0J=BP=0, even when the final solution is correct.
  • Section 4.2 and the caption of Table 1 clarify that all reported main-experiment ambiguity scores include this penalty.
  • Appendix B.2, “What the Decomposition Establishes” reports its impact on the DeepSeek-user cohort while holding the trajectories and other scoring criteria fixed.

For completeness, the following table provides the corresponding results for every agent, task setting, and both user simulators. Entries show joint-success counts without → with the over-clarification check, out of 120 tasks per cell, using identical trajectories, base scores, and other process checks.

Task settingAgentQwen user simulatorDeepSeek user simulator
Chinese travelDeepSeek-V4-Pro74 → 7219 → 19
Chinese travelGLM-5.3-Flash57 → 5748 → 48
Chinese travelQwen3.8-Flash33 → 3320 → 19
Chinese travelDeepSeek-Flash71 → 7158 → 58
Chinese travelGLM-5.379 → 7966 → 66
Chinese travelKimi-K354 → 5466 → 65
Chinese travelQwen3.8-Max77 → 7781 → 81
English travelDeepSeek-V4-Pro87 → 8620 → 19
English travelGLM-5.3-Flash53 → 5361 → 61
English travelQwen3.8-Flash85 → 8564 → 47
English travelDeepSeek-Flash65 → 6159 → 58
English travelGLM-5.359 → 5963 → 63
English travelKimi-K379 → 7970 → 70
English travelQwen3.8-Max75 → 7469 → 69
ShoppingDeepSeek-V4-Pro78 → 7868 → 66
ShoppingGLM-5.3-Flash73 → 7269 → 68
ShoppingQwen3.8-Flash78 → 7856 → 47
ShoppingDeepSeek-Flash82 → 8286 → 84
ShoppingGLM-5.382 → 8278 → 78
ShoppingKimi-K367 → 6765 → 65
ShoppingQwen3.8-Max72 → 4981 → 65

Across 5,040 main-experiment observations, the check excludes 83 otherwise successful cases. All ambiguity scores in the revised main table incorporate this penalty.

Reviewer_hcGSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a simulatable framework for evaluating interactive planning tasks across the three ACE dimensions.

Critical strengths

The paper is well-written, and the proposed framework is timely and comprehensive. It enables the evaluation of both final plans and the trajectories leading to them. The authors also adopt the ACE dimensions, which capture three orthogonal and important capabilities involved in different aspects of planning.

Although the evaluation currently focuses only on travel planning and shopping tasks, I believe the framework is extensible and could be readily generalized to other domains.

The authors also evaluate a sufficient number of models, and the results are well-presented and easy to follow.

Critical weaknesses
  1. Limited Coverage of Situation
  • Although I believe the framework is extensible, I find that the situation settings used to evaluate the A/C/E dimensions are not broad. For example, in the ambiguous settings, the authors consider only one field for each task (departure date for travel planning and item attributes for shopping). Similarly, I find the infeasibility setting particularly interesting, but the authors only consider budget infeasibility in shopping tasks.

  • This limited coverage could be a significant limitation. According to the results from the paper, model performance differ between shopping and planning tasks. However, it is unclear whether these differences are due to the characteristics of tasks (shopping/planning) or from the specific fields selected for evaluation (departure date in travel planning versus item attributes in shopping). If the latter is the case, we might observe different patterns even across different fields within the same task domain. This raises concerns about whether the current benchmark sufficiently captures the broader capabilities required in each domain, or whether the observed results are overly influenced by the particular settings chosen by the authors.

  1. Novelty and Positioning
  • I agree that A/C/E captures several important capabilities required for interactive planning. However, I find that the individual concepts are not entirely new to the planning domain. For example, ambiguity has already been studied a lot in the context of multi-turn interactions between humans and LLMs, and handling changes to user requirements in travel planning domain has also been explored in related work (e.g., [1]).

  • The authors should more clearly position their contributions relative to existing work. In particular, it would be helpful to clarify what new aspects of interactive planning this framework enables to evaluate that previous benchmarks could not capture. (see "questions for authors part" for my specific questions)

[1] Oh et al., Flex-TravelPlanner: A Benchmark for Flexible Planning with Language Agents, 2025.

Questions for authors

Regarding point 1, 2 in Critical Weaknesses, my questions are:

  1. Limited Coverage of Situation How easily could additional settings be incorporated into each of the A/C/E dimensions? How much might the evaluation results change if different settings were introduced? If other researchers wanted to extend the benchmark by introducing additional settings, what would be the main bottlenecks or challenges?

  2. Novelty and Positioning What unique insights can we gain from this paper's results that other papers cannot capture? How do author position this paper among these related papers?

3 comments
AuthorsOpenReview ↗

W1 and Q1:

Thank you for highlighting this limitation. We agree that cross-domain differences may reflect both domain characteristics and the selected fields. We added the following experiments to examine this concern.

A: Field sensitivity. We compare room count versus hotel star rating on 15 matched English travel tasks, and item attributes versus purchase quantity on 15 matched shopping tasks and target items. Paired variants preserve the final goal and other requirements. Both interpretations must admit complete feasible solutions with disjoint valid solution sets, ensuring that clarification is necessary.

Joint success requires both final-task validity and successful clarification behavior. The following results are reported in Section 4.6, Table 4; construction details and full results appear in Appendix A.

ModelTravel: room count → star ratingShopping: attribute → quantity
DeepSeek-Flash40.0% → 33.3%66.7% → 80.0%
GLM-5.353.3% → 80.0%60.0% → 73.3%
Qwen3.8-Max60.0% → 40.0%46.7% → 46.7%

Changing the travel field improves GLM-5.3 by 26.7 percentage points but reduces Qwen3.8-Max by 20.0 points. Thus, field sensitivity can be substantial and differ in direction across models.

E: Mechanism sensitivity. On 15 matched English travel tasks, we compare unavailable hotel star–service combinations with infeasible fixed-price room-charge budgets. Each mechanism has a neighboring feasible control: an available service combination or a budget increased to the verified cost lower bound. These are independent backup requests; the feasible main itinerary remains active.

The following backup-handling results are reported in Section 4.6, Table 4; control definitions and full results appear in Appendix A.

ModelAttribute infeasibilityBudget infeasibility
DeepSeek-Flash100.0%53.3%
GLM-5.393.3%86.7%
Qwen3.8-Max100.0%73.3%

These are automatic backup-handling rates, requiring correct status, preceding tool evidence, consistent actions, and legal completion. They measure backup handling rather than full-task joint success or independently adjudicated explanation quality. Feasible-control handling is 93.3–100.0%. Together, A/E comprise 360 records across three models; each cell contains 15 episodes, including runtime failures.

C: Revision depth. We compare one versus two revisions on the same 65 shopping tasks across seven models, adding 455 double-revision observations and reusing 455 primary single-revision observations. Both versions preserve the initial requirement, final target, and other constraints. Each revision requires new valid work, and verification checks removal of superseded products, final replacement, and subsequent cart inspection.

The following results are reported in Section 4.4, Table 2; the selection and replay protocol appears in Appendix H.

ModelOne revision: joint successTwo revisions: joint success
DeepSeek-Flash56.92%49.23%
DeepSeek-V4-Pro56.92%56.92%
GLM-5.364.62%61.54%
GLM-5.3-Flash44.62%35.38%
Kimi-K353.85%55.38%
Qwen3.8-Flash46.15%49.23%
Qwen3.8-Max58.46%50.77%

Differences range from −9.23 to +3.08 percentage points; all paired 95% confidence intervals include zero, establishing neither a consistent penalty nor equivalence. All 65 pairs remain in each model's denominator, including failed and unreached-event episodes. This extends revision depth rather than systematically testing new C fields.

Extensibility and bottlenecks. Adding a setting requires an input transformation, disclosure/event rules, and process/outcome checks. The principal challenge is validating the setting:

  • A requires globally feasible interpretations with disjoint valid solution sets, without leaking the intended choice through defaults or tools.
  • C requires meaningful provisional work that triggers a revision and verification that superseded decisions and downstream consequences are repaired.
  • E requires a verifiable inventory-absence certificate or global cost lower bound, a neighboring feasible control, and preservation of the feasible main task.

Researchers must also prevent unintended changes to other constraints, validate simulator disclosure behavior, and adapt evidence checks and semantic review. Consequently, some extensions require substantial field-specific construction and validation (Appendix A.5, “Extending Coverage”).

These additions demonstrate setting-dependent variation but do not eliminate domain–field confounding. Small samples, wide uncertainty intervals, execution differences, and outstanding independent semantic review limit causal and ranking claims. We therefore restrict our conclusions to the evaluated settings.

AuthorsOpenReview ↗

W2 and Q2:

Thank you for these closely related questions. We agree that ambiguity resolution, requirement changes, and infeasibility handling have substantial precedents. Our contribution lies in how these interactions are constructed, controlled, and evaluated in tool-grounded, long-horizon planning with coupled constraints.

Positioning relative to prior work. Flex-TravelPlanner already studies sequential constraint introduction and constraint prioritization. Its constraint-adaptive experiment specifically evaluates constraint addition. ACE instead withdraws and replaces an active requirement after relevant tool-based work, requiring agents to repair decisions already made. In shopping, this includes removing superseded products, adding valid replacements, and inspecting the updated cart. This distinction concerns the evaluated protocol; we do not claim that adapting to changing requirements is itself new. (Oh et al., 2025)

ACE also evaluates clarification before commitment and evidence-grounded handling of an infeasible backup while completing a feasible main task. In A, agents must ask a targeted clarification question while avoiding unnecessary questions. In E, agents must establish the backup’s infeasibility using tool evidence, explain it, avoid unauthorized substitutes, and complete the remaining itinerary or cart. A deterministic controller governs canonical requirements, disclosure, and event timing, while separate verifiers assess complete-plan validity and interaction behavior. The contribution is this combined construction and evaluation protocol, rather than the individual A/C/E categories.

Additional insights from the results. The clearest finding is that final-plan correctness can conceal interaction failures. In the DeepSeek-user shopping E cohort, 555 outputs satisfy the feasible main task, but 86 fail the process requirements. Thus, 15.5% of otherwise correct outcomes would be accepted by final-task evaluation alone. Conversely, 234 episodes pass the E process checks but fail the main task. These results distinguish successful handling of an infeasible request from successful completion of the overall plan, demonstrating why both need to be evaluated.

The additional discrimination is also condition-dependent: all 248 base-correct shopping C outcomes pass the process checks in the same cohort. Our results therefore identify both where interaction verification reveals otherwise hidden failures and where it adds no filtering among correct plans. This provides a more specific insight than simply showing that multi-turn planning is difficult. These distinctions are not directly measured by Flex-TravelPlanner’s reported constraint-satisfaction evaluation. The decomposition localizes observed failures; it does not establish that planning and interaction are independent capabilities.

Locations in the manuscript. These points are explicitly discussed in the following places:

  • Section 1, final paragraph: states that the contribution is “this combined construction and evaluation protocol, rather than new intent categories.”
  • Section 2, first two paragraphs: positions ACE relative to Flex-TravelPlanner and prior interactive-agent benchmarks, explaining the distinctions in requirement replacement, infeasible backups, and complete-plan verification.
  • Sections 3.2–3.4: describe the controlled A/C/E conditions, deterministic interaction controller, and joint outcome–process evaluation.
  • Section 4.3, under “Correct final plans can mask interaction failures” and “The added discrimination is condition-dependent,” presents the empirical findings discussed above. Appendix B provides the detailed outcome–process decomposition.
AuthorsOpenReview ↗

Update on Tool Correction and Revised Experimental Results

We would like to inform you of an evaluation correction prompted by Reviewer b1rg’s comment. We audited the trajectory sources underlying all reported experiments and all 28 tables for the same restaurant-tool issue. Valid numeric coordinates could be rejected by string-only argument validation or fail equivalent-coordinate matching. The corrected implementation accepts numbers and strings and uses exact decimal normalization, without changing the database or introducing geographic tolerance.

This audit led to 436 new episode reruns: 368 in the primary evaluation and 68 in supplementary studies. Previously completed corrected-tool Kimi recoveries were retained. The new reruns were:

ExperimentModelChinese travelEnglish travel
PrimaryDeepSeek-Flash118160
PrimaryGLM-5.3104
PrimaryGLM-5.3-Flash38
PrimaryKimi-K3215
PrimaryQwen3.8-Flash3711
Clarification formDeepSeek-Flash1416
Field/mechanismDeepSeek-Flash035
User wordingDeepSeek-Flash12
Total185251

No additional affected episodes were identified for DeepSeek-V4-Pro or Qwen3.8-Max in the primary evaluation, or for GLM-5.3 or Qwen3.8-Max in the supplementary studies. Shopping and the shopping-only revision-depth comparison do not use this tool.

Affected episodes were rerun end to end with the common corrected tool, preserving original tasks, frozen databases, model settings, nominal budgets, and study-specific user policies. New outcomes replaced old ones regardless of score; unfinished episodes remained failures in the original denominators. Unaffected outcomes were retained. This was targeted recovery rather than a uniform rerun of all experiments.

We updated the main table, Sections 4.5–4.7, related appendices, and Appendix G’s recovery documentation. We recomputed derived statistics and confidence intervals and verified all 28 tables against source records. We provide the revised results below for your information.

Primary travel results (%). Each entry lists Normal / A / C / E, using task success for Normal and joint task-and-process success for A/C/E. Q-user and D-user denote Qwen3.8-Flash and DeepSeek-Flash user simulators. Each condition has 120 episodes except E, which has 55.

ModelTravelQ-user: N/A/C/ED-user: N/A/C/E
DeepSeek-FlashChinese41.7/59.2/59.2/54.542.5/47.5/48.3/47.3
DeepSeek-V4-ProChinese11.7/60.0/54.2/47.38.3/15.8/12.5/3.6
GLM-5.3Chinese47.5/65.8/44.2/58.248.3/53.3/48.3/47.3
GLM-5.3-FlashChinese32.5/47.5/27.5/38.226.7/40.0/31.7/30.9
Kimi-K3Chinese43.3/45.0/40.8/41.846.7/54.2/44.2/43.6
Qwen3.8-FlashChinese33.3/27.5/29.2/25.522.5/16.7/14.2/21.8
Qwen3.8-MaxChinese66.7/64.2/65.0/52.763.3/67.5/57.5/50.9
DeepSeek-FlashEnglish55.0/50.8/53.3/54.554.2/54.2/46.7/47.3
DeepSeek-V4-ProEnglish65.8/71.7/65.0/67.317.5/15.8/22.5/16.4
GLM-5.3English50.0/49.2/50.0/50.952.5/52.5/48.3/65.5
GLM-5.3-FlashEnglish54.2/44.2/40.8/30.947.5/50.8/45.8/52.7
Kimi-K3English61.7/65.8/55.8/50.967.5/60.0/59.2/56.4
Qwen3.8-FlashEnglish60.0/70.8/63.3/61.853.3/40.0/40.0/43.6
Qwen3.8-MaxEnglish62.5/61.7/67.5/56.460.8/57.5/65.0/60.0

Supplementary results (%). J = joint success; B = task success; H = request-handling success. Only DeepSeek-Flash required supplementary recovery; the other two models’ results are unchanged. Denominators are per model and condition: clarification 10, field/mechanism 15, wording 20.

Study / conditionMetricDeepSeek-FlashGLM-5.3Qwen3.8-Max
Clarification, Chinese: HintJ30.010.040.0
Clarification, Chinese: ChoiceJ20.020.020.0
Clarification, Chinese: OmitJ40.050.040.0
Clarification, Chinese: FullB50.060.070.0
Clarification, English: HintJ50.080.080.0
Clarification, English: ChoiceJ30.050.040.0
Clarification, English: OmitJ30.030.060.0
Clarification, English: FullB80.080.0100.0
Field A, English: room countJ53.353.360.0
Field A, English: star ratingJ53.380.040.0
Field E, English: attribute, feasibleH/J100.0/73.3100.0/80.093.3/80.0
Field E, English: attribute, infeasibleH/J100.0/46.793.3/66.7100.0/73.3
Field E, English: budget, feasibleH/J100.0/73.393.3/53.3100.0/73.3
Field E, English: budget, infeasibleH/J46.7/40.086.7/66.773.3/66.7
Wording: canonicalJ35.065.070.0
Wording: validated rewritingJ65.030.075.0

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.