- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces GroupTravelBench, a benchmark of 650 tasks in which an LLM agent plans trips for groups through multi-turn conversations with simulated users. The agent needs to uncover individual preferences, resolve disagreements, and produce an itinerary that balances group satisfaction and fairness. The benchmark combines travel tools backed by cached real-world data with rule-based outcome metrics and LLM-based assessments of the interaction. Experiments with nine models find that preference elicitation and planning remain difficult, with no model exceeding 12% plan validity. The authors also check evaluation stability, sensitivity to simulator choice, and agreement with human ratings.
Critical strengths
The benchmark addresses a meaningful gap: group travel planning requires agents to uncover different users’ preferences and negotiate conflicts, beyond producing a feasible itinerary for one person. This gives the benchmark a clear purpose and makes it a useful setting for studying coordination among users with competing needs.
The paper makes a substantial effort to validate the evaluation through repeated runs, simulator replacement, offline/online comparisons, and human assessment of judge scores (Section 5.3 and Appendix C). These checks increase confidence that the reported differences are reasonably stable within the proposed setting.
Critical weaknesses
The results do not clearly isolate the difficulty introduced by group planning. Appendix C.8 shows that missing intra city transportation dominates plan validity failures, which appears to be a general itinerary construction problem. The oracle preference experiment is useful, but it does not separate coordination failures from ordinary planning failures. A matched single-user control or an ablation of conflicting preferences would help. Without this separation, the low validity scores provide limited evidence about the group-specific capabilities that motivate the benchmark.
The evidence for realistic group coordination is limited. Section 3.3 deliberately restricts users from volunteering preferences, while the simulator audit in Appendix C.3 checks compliance with this prescribed behavior rather than similarity to human conversations. These controls make evaluation manageable, but may reward exhaustive questioning under a fixed protocol more than effective coordination in real groups. Validation with human participants, or sensitivity tests allowing more proactive user behavior, would strengthen confidence that the benchmark measures the intended capability.
The attainable performance on the generated tasks is unclear (Section 3.2). Task validation checks POI existence, preference consistency, and transportation reachability, but these checks do not establish that a complete itinerary satisfying the combined scheduling, budget, and group constraints exists. The paper provides neither reference itineraries nor estimates of achievable utility and fairness. Such evidence would help distinguish model limitations from difficult or incompatible task specifications.
Questions for authors
Clarification on task feasibility (Section 3.2): Were complete reference itineraries constructed or checked for the generated tasks? What evidence establishes that tasks admit plans passing all nine validity checks, and are known infeasible cases handled separately?
Clarification on the source of difficulty (Appendices C.2 and C.9): Missing intra-city transportation dominates validity failures, and validity remains low with oracle preferences. Do existing results show how much additional difficulty comes specifically from conflicting group preferences, beyond general itinerary construction? Please distinguish what the current evidence establishes from what would require a future controlled comparison.
10 comments
Rebuttal Part 2
We performed rule-based preference de-confliction across our 650 tasks (while keeping group size, destinations, trip duration, tool environment, and interaction budget strictly matched) to ensure that within every task, positive preferences and negative preferences never conflict across users. Note that this still does not remove all tension, because within a limited trip duration, even when only positive preferences exist, the agent must still weigh trade-offs when choosing which attractions and restaurants to schedule. The results are summarized below (note that DeepSeek-V4-Pro numbers differ slightly from the main paper because the official DeepSeek API now routes to the updated deepseek-v4-pro-0813 snapshot, which does not affect the conclusions):
| GU | GF | PV | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Orig+ Std. | Orig+ Oracle | Deconf+ Std. | Deconf+ Oracle | Orig+ Std. | Orig+ Oracle | Deconf+ Std. | Deconf+ Oracle | Orig+ Std. | Orig+ Oracle | Deconf+ Std. | Deconf+ Oracle |
| DeepSeek-V4-Pro | 10.68 | 12.67 | 11.11 | 12.72 | 56.84 | 63.21 | 59.49 | 65.19 | 11 | 12 | 14 | 15 |
| Qwen3.5-Plus | 8.63 | 11.31 | 9.39 | 11.40 | 44.90 | 52.11 | 46.24 | 53.59 | 12 | 16 | 13 | 12 |
| Qwen3.5-Flash | 6.97 | 8.95 | 7.84 | 9.58 | 41.90 | 42.72 | 42.73 | 45.57 | 1 | 1 | 2 | 1 |
In the table, Orig refers to the original tasks with conflicts, Std. refers to not providing full preferences upfront, Deconf refers to the de-conflicted tasks, and Oracle refers to supplying complete user preferences.
The results reveal two clear takeaways:
-
Applying
+Deconfalone or+Oraclealone raises both GU and GF, with+Oraclebringing a noticeably larger boost. This suggests that the primary bottleneck of current models does not lie in reconciling opposing positive and negative preferences across users, but rather in incomplete preference elicitation—which prevents planning under full information—and weak trade-off decisions over which attractions and restaurants to prioritize under limited time, which is precisely the challenge we aim to introduce through group coordination. -
Across all four configurations, PV barely changes, demonstrating that the difficulty brought by multi-user coordination does not affect PV: PV serves as a structural feasibility check, whereas GU and GF are what truly measure group coordination difficulty.
To summarize our findings from these analyses:
- Low PV relates to group planning mainly because the model faces a larger volume of preferences; even without inter-user conflicts, assembling a multi-day, multi-user itinerary leaves many opportunities for minor mistakes (with numerous preferences, many POIs, and long contexts, a single slip anywhere reduces PV to 0).
- GU and GF are specifically designed to evaluate multi-user coordination quality, and the difficulty introduced by group coordination is reflected in these two metrics, whereas PV remains a check on itinerary feasibility.
R to W2
The evidence for realistic group coordination is limited. Section 3.3 deliberately restricts users from volunteering preferences, while the simulator audit in Appendix C.3 checks compliance with this prescribed behavior rather than similarity to human conversations. These controls make evaluation manageable, but may reward exhaustive questioning under a fixed protocol more than effective coordination in real groups. Validation with human participants, or sensitivity tests allowing more proactive user behavior, would strengthen confidence that the benchmark measures the intended capability.
Thank you for this thoughtful point. We understand your concern that our behavioral constraints on simulated users may depart from real-world group coordination dynamics and might inadvertently reward exhaustive questioning. We clarify both points below.
Rebuttal Part 3
First, regarding why we restrict user simulator behaviors: as explained in Section 3.3, our main motivation is concern that unconstrained user behavior would confound how we attribute performance to the agent. Concretely:
- Disallowing users from proposing itinerary plans → ensures GU and GF are attributable to the agent;
- Disallowing users from volunteering preferences unprompted → ensures PC measures the agent's elicitation skill;
- Disallowing @-mentions among users or casual chitchat → ensures inter-user conflicts must be detected and mediated by the agent while keeping the group conversation convergent.
Alongside these rules, we preserved as many realistic behaviors as possible that do not compromise metric attribution:
- Each user's opening turn shares 1–3 preferences whose tone reflects their preference strength, and users may speak up proactively whenever a strong preference is violated;
- Users can proactively ask the agent factual questions, such as attraction opening hours or restaurant suggestions;
- Users can make concessions when asked by the agent.
Second, we would like to clarify that these rules do not reward exhaustive questioning:
- The interaction budget is strictly capped (15 / 20 / 25 turns depending on task difficulty), which precludes exhaustive probing.
- Whether each user is willing to compromise is fixed in advance—1,490 compromisable users (54.2%) vs. 1,258 non-compromisable users (45.8%)—and for compromisable users, the concession limit is capped at 2. Thus, a model cannot simply ask exhaustive questions to talk everyone into dropping all negative preferences; instead, it must selectively ask for compromises only when conflicts cannot otherwise be resolved.
To further examine the effect of these constraints and test whether our setup rewards exhaustive questioning, we ran a sensitivity study relaxing the user simulator rules: we allowed users to volunteer preferences proactively, speak up even when weak preferences are violated (originally allowed only for strong-preference violations), and make suggestions about itinerary arrangements, while still disallowing user-to-user @-mentions or chitchat so that conversations remain convergent. The results are shown below (note that DeepSeek-V4-Pro numbers differ slightly from the main paper because the official DeepSeek API now points to the updated deepseek-v4-pro-0813 snapshot, though this does not affect the experimental conclusions):
| Model | PC | GU | GF (%) | PV (%) | ||||
|---|---|---|---|---|---|---|---|---|
| Orig. | Relax | Orig. | Relax | Orig. | Relax | Orig. | Relax | |
| DeepSeek-V4-Pro | 32.00 | 37.00 | 10.68 | 10.77 | 56.84 | 55.41 | 11 | 10 |
| Qwen3.5-Plus | 30.00 | 36.00 | 8.63 | 9.46 | 44.90 | 43.88 | 12 | 13 |
| Qwen3.5-Flash | 21.00 | 27.00 | 6.97 | 7.13 | 41.90 | 38.50 | 1 | 1 |
From this comparison, we observe that:
- Because users express their preferences more proactively, PC increases, and knowing more preferences naturally enables the agent to achieve a higher GU.
- At the same time, GF drops slightly across models; inspecting the trajectories suggests that when users speak more—especially offering planning suggestions and emphasizing their own wishes—it becomes harder for the agent to balance everyone's demands equitably.
These findings show that relaxing user constraints indeed shifts the evaluation metrics (making it harder to attribute GU, GF, and PC purely to the agent), yet the resulting gain in GU remains modest. This indicates that our controlled protocol does not reward exhaustive questioning (otherwise GU would drop under the relaxed user setting), and that poor group coordination remains a bottleneck even under more proactive user dynamics. Nevertheless, we still prefer to retain these constraints (trading off some realism) so as to evaluate agent capabilities in a relatively clean environment (free from user-simulation confounders), while leaving better trade-offs between evaluability and realism as an interesting direction for future work.
Rebuttal Part 1
Thank you sincerely for your thoughtful feedback and for taking the time to review our manuscript. We greatly value the insightful questions you raised; below, we discuss each point in detail, and we commit to organizing these revisions and updating the manuscript PDF within the next few days (prior to the end of the discussion phase).
R to W1
Thank you very much for this perceptive question. We understand that your concern focuses on two aspects: low PV does not substantiate insufficient group coordination capability, and the difficulty specific to group coordination is not clearly separated and verified.
We address each aspect in detail below.
Clarifying the Role of PV vs. Group Coordination
We apologize that highlighting PV in our abstract and introduction may have caused ambiguity, and we would like to clarify:
Accordingly, we will revise the wording in the abstract and introduction to emphasize low GU and GF, which are the metrics that capture insufficient multi-user coordination.
Meanwhile, to support our point that handling more preferences, more preference conflicts, and longer contexts lowers PV, we broke down PV by the number of users N and trip length (in days):
These tables show a clear downward trend in PV as either the number of users or the number of travel days grows. Because more users and longer trips mean the agent must process more preferences, more conflicts among preferences, and longer contexts, this is one factor driving PV down. Yet overall, even on 2-user and 2-day trips, PV remains very low—indicating that in multi-user task settings, current models lack sufficient capability to produce spatio-temporally feasible itineraries in the first place. We therefore regard PV as measuring an agent's ability to generate spatio-temporally reachable plans under group travel scenarios, and we next show empirically that PV is largely independent of group coordination.
Disentangling Group Coordination Difficulty from General Planning
Below, we report an ablation on de-conflicted tasks paired with the oracle-preference setting to demonstrate that multi-user coordination capability impacts GU and GF, without materially affecting PV.