acceptodds
Under review as a conference paper at ICLR 2027

GroupTravelBench: Benchmarking LLM Agents on Multi-User, Multi-Turn Travel Planning

Abstract

Travel planning in the real world is overwhelmingly a group activity, yet existing LLM travel-planning benchmarks reduce it to a single user, where the field is approaching saturation. This single-user assumption sidesteps what makes group planning hard for an agent: discovering private preferences across multiple users, surfacing conflicts, and balancing utility against fairness. To bring the task back to its multi-user reality, we introduce GroupTravelBench, the first benchmark for multi-user, multi-turn travel planning. Built from real user profiles, Point-Of-Interest (POI) data, and ticket prices, it comprises 650 tasks across three difficulty levels, each running in a controlled synchronous group-chat sandbox with cached tool data for reproducible offline evaluation. Beyond the multi-step reasoning and tool use that single-user benchmarks already test, GroupTravelBench probes three group-specific capabilities: (i) elicitation of private preferences through multi-turn dialogue; (ii) coordination of inter-user conflicts via compromise or subgrouping; and (iii) planning that balances group utility against fairness. We pair this with a complementary evaluation framework combining rule-based outcome metrics and LLM-judge process metrics. Across a wide range of frontier models, even the strongest agents fall short on all four rule-based outcome metrics, with plan validity at most 12%, suggesting that group-level outcome quality is a key open challenge for LLM travel-planning agents. Our code and data are available at https://anonymous.4open.science/r/Group_Travelbench_Anonymous-2AD7.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Travel planning, Tool use

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_NrofOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GroupTravelBench, a benchmark of 650 tasks in which an LLM agent plans trips for groups through multi-turn conversations with simulated users. The agent needs to uncover individual preferences, resolve disagreements, and produce an itinerary that balances group satisfaction and fairness. The benchmark combines travel tools backed by cached real-world data with rule-based outcome metrics and LLM-based assessments of the interaction. Experiments with nine models find that preference elicitation and planning remain difficult, with no model exceeding 12% plan validity. The authors also check evaluation stability, sensitivity to simulator choice, and agreement with human ratings.

Critical strengths

The benchmark addresses a meaningful gap: group travel planning requires agents to uncover different users’ preferences and negotiate conflicts, beyond producing a feasible itinerary for one person. This gives the benchmark a clear purpose and makes it a useful setting for studying coordination among users with competing needs.

The paper makes a substantial effort to validate the evaluation through repeated runs, simulator replacement, offline/online comparisons, and human assessment of judge scores (Section 5.3 and Appendix C). These checks increase confidence that the reported differences are reasonably stable within the proposed setting.

Critical weaknesses

The results do not clearly isolate the difficulty introduced by group planning. Appendix C.8 shows that missing intra city transportation dominates plan validity failures, which appears to be a general itinerary construction problem. The oracle preference experiment is useful, but it does not separate coordination failures from ordinary planning failures. A matched single-user control or an ablation of conflicting preferences would help. Without this separation, the low validity scores provide limited evidence about the group-specific capabilities that motivate the benchmark.

The evidence for realistic group coordination is limited. Section 3.3 deliberately restricts users from volunteering preferences, while the simulator audit in Appendix C.3 checks compliance with this prescribed behavior rather than similarity to human conversations. These controls make evaluation manageable, but may reward exhaustive questioning under a fixed protocol more than effective coordination in real groups. Validation with human participants, or sensitivity tests allowing more proactive user behavior, would strengthen confidence that the benchmark measures the intended capability.

The attainable performance on the generated tasks is unclear (Section 3.2). Task validation checks POI existence, preference consistency, and transportation reachability, but these checks do not establish that a complete itinerary satisfying the combined scheduling, budget, and group constraints exists. The paper provides neither reference itineraries nor estimates of achievable utility and fairness. Such evidence would help distinguish model limitations from difficult or incompatible task specifications.

Questions for authors

Clarification on task feasibility (Section 3.2): Were complete reference itineraries constructed or checked for the generated tasks? What evidence establishes that tasks admit plans passing all nine validity checks, and are known infeasible cases handled separately?

Clarification on the source of difficulty (Appendices C.2 and C.9): Missing intra-city transportation dominates validity failures, and validity remains low with oracle preferences. Do existing results show how much additional difficulty comes specifically from conflicting group preferences, beyond general itinerary construction? Please distinguish what the current evidence establishes from what would require a future controlled comparison.

10 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Thank you sincerely for your thoughtful feedback and for taking the time to review our manuscript. We greatly value the insightful questions you raised; below, we discuss each point in detail, and we commit to organizing these revisions and updating the manuscript PDF within the next few days (prior to the end of the discussion phase).

R to W1

The results do not clearly isolate the difficulty introduced by group planning. Appendix C.8 shows that missing intra city transportation dominates plan validity failures, which appears to be a general itinerary construction problem. The oracle preference experiment is useful, but it does not separate coordination failures from ordinary planning failures. A matched single-user control or an ablation of conflicting preferences would help. Without this separation, the low validity scores provide limited evidence about the group-specific capabilities that motivate the benchmark.

Thank you very much for this perceptive question. We understand that your concern focuses on two aspects: low PV does not substantiate insufficient group coordination capability, and the difficulty specific to group coordination is not clearly separated and verified.

We address each aspect in detail below.

Clarifying the Role of PV vs. Group Coordination

We apologize that highlighting PV in our abstract and introduction may have caused ambiguity, and we would like to clarify:

  • PV is not a metric tailored to multi-user coordination. As you rightly noted, it applies to general itinerary construction and serves simply as a check on the spatio-temporal feasibility of an itinerary; we did not intend to imply that low PV stems from poor multi-user coordination.
  • Intuitively, however, in group travel planning an agent must cope with a larger set of preferences, resolve conflicting desires, and track a longer dialogue context. This can cause the agent to overlook spatio-temporal reachability constraints and fail PV—which can be viewed as extra complexity introduced by the group planning task setup itself, though fundamentally it reflects weakness in generating spatio-temporally reachable schedules rather than a lack of multi-user coordination skill.

Accordingly, we will revise the wording in the abstract and introduction to emphasize low GU and GF, which are the metrics that capture insufficient multi-user coordination.

Meanwhile, to support our point that handling more preferences, more preference conflicts, and longer contexts lowers PV, we broke down PV by the number of users NN and trip length (in days):

IndicatorN=2N=3N=4N=5N=6Total
Task Count90126105204125650
Mean PV (%)8.724.954.162.452.334.06
Indicator2-Day3-Day4-Day5-Day6-Day7-DayTotal
Task Count90120962037071650
Mean PV (%)11.795.912.932.031.221.254.06

These tables show a clear downward trend in PV as either the number of users or the number of travel days grows. Because more users and longer trips mean the agent must process more preferences, more conflicts among preferences, and longer contexts, this is one factor driving PV down. Yet overall, even on 2-user and 2-day trips, PV remains very low—indicating that in multi-user task settings, current models lack sufficient capability to produce spatio-temporally feasible itineraries in the first place. We therefore regard PV as measuring an agent's ability to generate spatio-temporally reachable plans under group travel scenarios, and we next show empirically that PV is largely independent of group coordination.

Disentangling Group Coordination Difficulty from General Planning

Below, we report an ablation on de-conflicted tasks paired with the oracle-preference setting to demonstrate that multi-user coordination capability impacts GU and GF, without materially affecting PV.

AuthorsOpenReview ↗

Rebuttal Part 2

We performed rule-based preference de-confliction across our 650 tasks (while keeping group size, destinations, trip duration, tool environment, and interaction budget strictly matched) to ensure that within every task, positive preferences and negative preferences never conflict across users. Note that this still does not remove all tension, because within a limited trip duration, even when only positive preferences exist, the agent must still weigh trade-offs when choosing which attractions and restaurants to schedule. The results are summarized below (note that DeepSeek-V4-Pro numbers differ slightly from the main paper because the official DeepSeek API now routes to the updated deepseek-v4-pro-0813 snapshot, which does not affect the conclusions):

GUGFPV
ModelOrig+ Std.Orig+ OracleDeconf+ Std.Deconf+ OracleOrig+ Std.Orig+ OracleDeconf+ Std.Deconf+ OracleOrig+ Std.Orig+ OracleDeconf+ Std.Deconf+ Oracle
DeepSeek-V4-Pro10.6812.6711.1112.7256.8463.2159.4965.1911121415
Qwen3.5-Plus8.6311.319.3911.4044.9052.1146.2453.5912161312
Qwen3.5-Flash6.978.957.849.5841.9042.7242.7345.571121

In the table, Orig refers to the original tasks with conflicts, Std. refers to not providing full preferences upfront, Deconf refers to the de-conflicted tasks, and Oracle refers to supplying complete user preferences.

The results reveal two clear takeaways:

  • Applying +Deconf alone or +Oracle alone raises both GU and GF, with +Oracle bringing a noticeably larger boost. This suggests that the primary bottleneck of current models does not lie in reconciling opposing positive and negative preferences across users, but rather in incomplete preference elicitation—which prevents planning under full information—and weak trade-off decisions over which attractions and restaurants to prioritize under limited time, which is precisely the challenge we aim to introduce through group coordination.

  • Across all four configurations, PV barely changes, demonstrating that the difficulty brought by multi-user coordination does not affect PV: PV serves as a structural feasibility check, whereas GU and GF are what truly measure group coordination difficulty.

To summarize our findings from these analyses:

  • Low PV relates to group planning mainly because the model faces a larger volume of preferences; even without inter-user conflicts, assembling a multi-day, multi-user itinerary leaves many opportunities for minor mistakes (with numerous preferences, many POIs, and long contexts, a single slip anywhere reduces PV to 0).
  • GU and GF are specifically designed to evaluate multi-user coordination quality, and the difficulty introduced by group coordination is reflected in these two metrics, whereas PV remains a check on itinerary feasibility.

R to W2

The evidence for realistic group coordination is limited. Section 3.3 deliberately restricts users from volunteering preferences, while the simulator audit in Appendix C.3 checks compliance with this prescribed behavior rather than similarity to human conversations. These controls make evaluation manageable, but may reward exhaustive questioning under a fixed protocol more than effective coordination in real groups. Validation with human participants, or sensitivity tests allowing more proactive user behavior, would strengthen confidence that the benchmark measures the intended capability.

Thank you for this thoughtful point. We understand your concern that our behavioral constraints on simulated users may depart from real-world group coordination dynamics and might inadvertently reward exhaustive questioning. We clarify both points below.

AuthorsOpenReview ↗

Rebuttal Part 3

First, regarding why we restrict user simulator behaviors: as explained in Section 3.3, our main motivation is concern that unconstrained user behavior would confound how we attribute performance to the agent. Concretely:

  • Disallowing users from proposing itinerary plans → ensures GU and GF are attributable to the agent;
  • Disallowing users from volunteering preferences unprompted → ensures PC measures the agent's elicitation skill;
  • Disallowing @-mentions among users or casual chitchat → ensures inter-user conflicts must be detected and mediated by the agent while keeping the group conversation convergent.

Alongside these rules, we preserved as many realistic behaviors as possible that do not compromise metric attribution:

  • Each user's opening turn shares 1–3 preferences whose tone reflects their preference strength, and users may speak up proactively whenever a strong preference is violated;
  • Users can proactively ask the agent factual questions, such as attraction opening hours or restaurant suggestions;
  • Users can make concessions when asked by the agent.

Second, we would like to clarify that these rules do not reward exhaustive questioning:

  • The interaction budget is strictly capped (15 / 20 / 25 turns depending on task difficulty), which precludes exhaustive probing.
  • Whether each user is willing to compromise is fixed in advance—1,490 compromisable users (54.2%) vs. 1,258 non-compromisable users (45.8%)—and for compromisable users, the concession limit is capped at 2. Thus, a model cannot simply ask exhaustive questions to talk everyone into dropping all negative preferences; instead, it must selectively ask for compromises only when conflicts cannot otherwise be resolved.

To further examine the effect of these constraints and test whether our setup rewards exhaustive questioning, we ran a sensitivity study relaxing the user simulator rules: we allowed users to volunteer preferences proactively, speak up even when weak preferences are violated (originally allowed only for strong-preference violations), and make suggestions about itinerary arrangements, while still disallowing user-to-user @-mentions or chitchat so that conversations remain convergent. The results are shown below (note that DeepSeek-V4-Pro numbers differ slightly from the main paper because the official DeepSeek API now points to the updated deepseek-v4-pro-0813 snapshot, though this does not affect the experimental conclusions):

ModelPCGUGF (%)PV (%)
Orig.RelaxOrig.RelaxOrig.RelaxOrig.Relax
DeepSeek-V4-Pro32.0037.0010.6810.7756.8455.411110
Qwen3.5-Plus30.0036.008.639.4644.9043.881213
Qwen3.5-Flash21.0027.006.977.1341.9038.5011

From this comparison, we observe that:

  • Because users express their preferences more proactively, PC increases, and knowing more preferences naturally enables the agent to achieve a higher GU.
  • At the same time, GF drops slightly across models; inspecting the trajectories suggests that when users speak more—especially offering planning suggestions and emphasizing their own wishes—it becomes harder for the agent to balance everyone's demands equitably.

These findings show that relaxing user constraints indeed shifts the evaluation metrics (making it harder to attribute GU, GF, and PC purely to the agent), yet the resulting gain in GU remains modest. This indicates that our controlled protocol does not reward exhaustive questioning (otherwise GU would drop under the relaxed user setting), and that poor group coordination remains a bottleneck even under more proactive user dynamics. Nevertheless, we still prefer to retain these constraints (trading off some realism) so as to evaluate agent capabilities in a relatively clean environment (free from user-simulation confounders), while leaving better trade-offs between evaluability and realism as an interesting direction for future work.

Reviewer_DUauOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

GroupTravelBench evaluates travel-planning agents that interact with multiple users whose preferences may conflict. It contains 650 tasks constructed from real travel resources and anonymized user-profile summaries, with controlled preference synthesis and simulated group conversations. Original, agent-inferred, and post-compromise preference tables support evaluation of elicitation and planning. The benchmark reports preference completeness, group utility, group fairness, and plan validity separately, alongside an LLM-based process score. Experiments cover nine models, repeated evaluations, oracle preference access, simulator replacements, comparisons with live tools, and human audits of user behavior and process judgments. The results show substantial remaining difficulty, particularly in producing structurally valid itineraries.

Critical strengths
  1. The task adds a meaningful interaction problem to travel planning. Discovering several users’ private preferences and tracking negotiated changes requires more than solving a larger single-user itinerary. The three-table representation is a useful design: it distinguishes what users want, what the agent has understood, and what users subsequently agree to. Compromise and subgrouping make the evaluation responsive to coordination behavior rather than simply counting conflicting constraints.
  2. The validation is stronger than a leaderboard alone. The oracle-preference experiment shows that planning remains difficult even when elicitation is removed, and the validity breakdown identifies missing local transportation as a dominant failure. The simulator audit, replacement experiments, and online/offline comparison address plausible sources of evaluation noise. I also appreciate that the paper explicitly distinguishes compliance with its controlled user contract from realism of unrestricted human conversation.
Critical weaknesses

The interpretation of Group Fairness needs more justification. Equation 3 compares raw utility totals across users, while Appendix A.1 gives credit for satisfied positive preferences and zero for respected negative preferences. Users can also have different numbers of positive items. Consequently, a lower utility can reflect the construction of a preference table rather than worse treatment. For example, fully satisfying one user’s single must-item and another user’s two must-items would produce utilities of 2 and 4, and hence GF = 50, despite satisfying both users completely. This is a property of the stated metric, not evidence that this exact case occurs frequently in the dataset. Raw-utility balance is a legitimate quantity to report, but Section 5.2 interprets low GF more broadly as an inability to treat users equitably. That interpretation needs a check that differences in preference-table scale are not driving the result. The concern does not erase the benchmark’s elicitation and coordination contributions.

Questions for authors

Are users’ attainable positive-utility scales comparable within a task under the synthesis procedure? If not, can you check on the existing plans whether normalizing for the available positive preference weight materially changes the GF conclusions or model ordering? This need not establish an optimal itinerary; the question is whether preference-table scale drives the fairness result.

4 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Thank you very much for your valuable time and insightful feedback; we are greatly encouraged by your positive assessment of our work. Below, we address your concerns first, and we commit to compiling these revisions and updating the paper PDF within the next few days (before the end of the discussion period).

R to W&Q

W: The interpretation of Group Fairness needs more justification. Equation 3 compares raw utility totals across users, while Appendix A.1 gives credit for satisfied positive preferences and zero for respected negative preferences. Users can also have different numbers of positive items. Consequently, a lower utility can reflect the construction of a preference table rather than worse treatmen...

Q: Are users’ attainable positive-utility scales comparable within a task under the synthesis procedure? If not, can you check on the existing plans whether normalizing for the available positive preference weight materially changes the GF conclusions or model ordering? This need not establish an optimal itinerary; the question is whether preference-table scale drives the fairness result.

This is a very thoughtful point, and we appreciate your careful questions and feedback. We fully understand your concern that GF may be constrained by preference-table size itself, which could make unnormalized GF naturally low and potentially affect our conclusion regarding models' insufficient ability to plan equitably. Below, we provide a detailed explanation.

Explanation of the Scoring Rules

Under our scoring rules, satisfying a positive preference earns positive points while respecting (not scheduling) a negative preference earns zero points. We designed the scoring this way for two reasons:

  • We cannot determine what positive score would be reasonable for not scheduling a negative preference. Assigning +2 for satisfying a strong preference and +1 for satisfying a weak preference is already an empirical setting; if we additionally rewarded avoiding negative preferences, we could not determine how many points would be reasonable. Penalizing only violations of negative preferences keeps utility scoring cleaner and more reasonable.
  • Giving rewards for not scheduling negative preferences could lead to repeated scoring. For example, every scheduled attraction not on a user's negative list would earn points, meaning longer trips would automatically accumulate more bonus points. We consider such a reward mechanism pathological.

Task Synthesis and Preference Scale

First, we would like to clarify how preference scale is controlled during task synthesis. In fact, our task synthesis pipeline incorporates soft controls that ensure preference scales are roughly comparable across users without forcing them to be strictly equal.

Specifically, we believe that in reality different individuals should naturally have different numbers of preferences. However, to balance preference scale, our prompts impose soft count guidelines—for instance, in each city, 2–4 attraction preferences in total across (must, reject, positive, negative), 2–4 dining preferences, at least one negative preference, and so on. Second, preference synthesis uses modes such as copy_minor and copy_moderate (see Section 3.2, Stage 3), meaning that one user's preferences (e.g., a father's) may be inherited from another's (e.g., a mother's) with minor adjustments, which also helps maintain scale comparability. Even when two group members' preferences are generated independently, the count guidelines in our prompts ensure that any differences remain well controlled.

To quantify the distribution of preference scales across users within a task (considering positive preferences only), we count all positive preferences for each user and take the difference between the maximum and minimum within the task as the range, where a larger range indicates a larger discrepancy in preference scale. The range distribution across all 650 tasks is:

Within-Task Positive-Preference RangeMeanMax≤2≤3≤4≤5
Share (650 tasks)2.64953.2%73.8%86.3%92.6%
  • As shown in the table, 92.6% of tasks have a range of at most 5. Intuitively, larger group sizes and more destination cities lead to a wider range, and we believe this distribution reflects the real-world heterogeneity of preference scales across different users.
  • The mean range is 2.64—that is, on average within a task, the user with the most positive preferences has 2.64 more items than the user with the fewest—while the average number of positive preferences per user is 13.34. Relatively speaking, we consider the difference in preference scale to be small, showing that user preference scales are well controlled and consistent with our synthesis design above.
AuthorsOpenReview ↗

Rebuttal Part 2

Estimating Attainable Utility and Normalizing GF

While the discussion above shows both the realistic heterogeneity and controllability of preference scale, one might still worry about how this heterogeneity affects GF. Below, we present our attainable-utility estimation and the resulting normalized GF scores.

Challenges and Approach for Utility Estimation

In practice, we cannot compute an upper bound on attainable utility solely from the preference table (i.e., directly multiplying positive preference counts by their corresponding weights). When computing individual utility, aside from must-level preferences on attractions, most preferences are category-level—such as attraction categories and food categories—which are credited multiple times during scoring (for instance, if User A prefers park-type attractions, scheduling both Park B and Park C yields 1 utility point each). Estimating attainable utility from the preference table alone (counting "prefers parks" once as +1) would be far smaller than the utility a user actually obtains.

The difficulty in estimation is that actual attainable utility depends on the number of attractions and restaurants scheduled each day. However, we do not impose a fixed daily schedule template on the agent (such as fixing two attractions and two meals per day); the agent is completely free to arrange the itinerary to maximize GU, so we do not know beforehand how many attractions or meals it will schedule.

Nevertheless, to estimate an attainable-utility upper bound for each user, we empirically assume a reference itinerary skeleton: three meals per day and a daily attraction count equal to each user's max_poi_per_day, establishing a candidate schedule template based on the cities and duration involved in each task. We then consider the best-case scenario: every activity slot takes the maximum score achievable under that user's preferences—for example, for User A, if must_eat exists, three meals yield at most 3×2 = 6 points, whereas if only pref_eat exists, they yield at most 3×1 = 3 points—corresponding to fully satisfying all of the user's preferences. Note that this estimate serves as an empirical upper bound: in practice, an agent cannot satisfy User A on every slot, so even if the daily number of attractions or meals varies, actual utility will almost never exceed this estimated bound.

Under this estimation scheme, we compute each user's attainable utility and divide the minimum by the maximum within each task—that is, characterizing GF under the idealized case where every user's preferences are fully satisfied. The statistics are as follows:

Mean min/maxMedian min/maxMin Within-Task min/max# Tasks with Identical Upper Bounds for All Users
0.9360.9520.68086

On average, the min/max ratio is 0.936; the closer this value is to 1, the more comparable attainable utilities are across users within a group. Meanwhile, the minimum within-task min/max ratio is 0.680, showing that there do exist a few tasks where the preference distribution itself leads to a lower GF ceiling, though this does not affect the positive overall statistical finding across the vast majority of tasks.

Normalized GF Results

Finally, we normalize each user's utility by this estimated attainable utility to compute GF-norm. The results for all 9 models in the main table are as follows:

ModelGF (Raw)RankGF-normRank
DeepSeek-V4-Pro54.22155.321
Qwen3.6-Max52.11252.952
Qwen3-235B-Th48.95349.463
Qwen3-30B-Th48.63448.924
GPT-5.148.14548.715
Qwen3-4B-Th47.57647.686
Qwen3.5-Plus44.93745.437
Qwen3.5-Flash41.85842.288
Qwen3-4B-It41.72941.999

As can be seen from the table, normalized GF-norm is slightly higher than raw GF, but because the difference is minimal, the model rankings remain unchanged.

In summary, based on the experiments and analyses above, we believe GF is a reliable metric for measuring fairness in group planning, and the conclusions and model rankings derived from it are sound. We will further elaborate on these analyses and report GF-norm in the revised paper.

We hope these responses resolve your concerns regarding the GF metric. Thank you once again for your valuable time and insightful questions, which have also deepened our understanding of GF! We will organize and update the paper PDF as soon as possible.

AuthorsOpenReview ↗

Looking forward to your reply!

Dear Reviewer Duau,

We hope this message finds you well.

Thank you again for your valuable comments. We have prepared detailed responses to each point and hope they address your concerns.

As the discussion deadline is approaching, we would be grateful if you could share any further feedback at your earliest convenience. Your insights are highly appreciated and will help us further improve the quality of our work.

Thank you very much for your time and effort. Please feel free to let us know if any additional clarification or additional details are needed.

Sincerely, The Authors

AuthorsOpenReview ↗

Follow-up on Our Rebuttal Before Discussion Closes

Dear Reviewer Duau,

Our rebuttal has been available since November 14, but we have not yet heard back from you. As the discussion period ends in about a day, we would be grateful for the chance to clarify anything that remains unclear.

Our rebuttal and the revised manuscript include additional experiments and analyses responding to your comments.

Could you please review our rebuttal and let us know whether it addresses your concerns and warrants reconsideration of your assessment?

Thank you for your time and consideration.

Best regards, The authors

Reviewer_v2cLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GroupTravelBench, a benchmark for multi-user, multi-turn travel planning with LLM agents. It contains 650 tasks synthesized using real user profiles, POIs, and transportation data. Through controlled group conversations with simulated users, it evaluates preference elicitation, conflict coordination, and group-aware planning. Key design elements include four preference-strength tiers and three tables tracking original, agent-inferred, and post-compromise preferences. Evaluation combines four rule-based outcome metrics with an LLM-based process score. Experiments with nine models reveal substantial limitations in preference elicitation and itinerary feasibility, with a maximum plan-validity rate of 12% under the standard setting. Additional experiments include oracle-preference access, simulator replacement, live-tool comparisons, and human validation.

Critical strengths
  1. The task setting is valuable, and the three-table design supports tracking interaction-dependent preference states. The benchmark combines hidden-preference elicitation, conflict negotiation, post-compromise state updates, and final planning. Separating original, agent-inferred, and effective preferences allows changes in user preferences and the agent’s understanding of those changes to be examined separately. This provides a useful foundation for studying multi-user interactive agents.

  2. The environment is substantial and supported by multiple reliability checks. The 650 tasks cover different group archetypes, destinations, and trip lengths, with tool caches and an explicit interaction protocol. Three independent trials, offline/online tool comparisons, simulator replacements, a human audit of user behavior, and human validation of judge scores strengthen confidence in results within the defined environment. The oracle-preference experiment also makes a useful attempt to separate elicitation failures from downstream difficulties.

Critical weaknesses
  1. The core group metrics do not sufficiently distinguish preference scale, concessions, and actual coordination quality. GF directly compares raw utilities without normalization for individual preference scale or attainable utility (Eq. 3). Users can have different numbers and total weights of preferences, so even fully satisfying each user need not produce a high GF. In addition, GU and GF are evaluated against post-compromise preferences: removing a rejection can improve the score without changing the final itinerary. This can capture agreement, but does not by itself establish that original needs are better accommodated or reveal whether concessions disproportionately burden particular members. Consequently, the interpretation of low fairness and coordination quality remains ambiguous. Analyses using individually normalized utilities, together with satisfaction relative to original preferences and the distribution of concessions, are needed.

  2. The experiments do not sufficiently separate multi-user coordination difficulty from generic itinerary-construction difficulty, and task-level feasible references are missing. Appendix C.9 identifies missing intra-city transportation and transportation-origin mismatches as the dominant validity errors; neither is specific to multi-user coordination. The oracle-preference experiment is useful, but removes only initial elicitation while retaining both conflict coordination and ordinary planning. It therefore cannot isolate their respective contributions. Group-size analyses are also confounded by trip duration and city count, as the authors acknowledge. Furthermore, task validation checks POI existence, preference consistency, and transport reachability, but does not provide reference itineraries passing all PV checks. Without matched conflict-free group controls and human- or solver-constructed feasible references, the maximum PV of 12% is difficult to interpret specifically as a multi-user planning capability gap.

Questions for authors
  1. Do the main model rankings and conclusions about insufficient fairness remain stable when fairness is computed using individually normalized utilities and scores are reported against both original and post-compromise preferences? How evenly are concessions distributed across users?

  2. Can the authors compare conflicting and conflict-free tasks while keeping group size, destinations, duration, tool environment, and interaction budget approximately matched? Combined with oracle-preference access, can this quantify the additional difficulty introduced by conflict coordination?

  3. On a representative subset, can the authors provide human- or solver-constructed reference itineraries that pass every PV check, together with task-level failure rates by error category and GU/GF evaluated only on valid itineraries? These results would clarify the source of low PV and whether the utility and fairness conclusions depend on infeasible plans.

9 comments
AuthorsOpenReview ↗

Rebuttal Part 1

We sincerely appreciate your thoughtful feedback and the time you dedicated to reviewing our work. We highly value your constructive suggestions; below, we address your concerns and questions first, and we commit to organizing these revisions and updating the paper PDF as soon as possible over the coming days (before the end of the discussion period).

R to W1,Q1

W1: The core group metrics do not sufficiently distinguish preference scale, concessions, and actual coordination quality. GF directly compares raw utilities without normalization for individual preference scale or attainable utility (Eq. 3). Users can have different numbers and total weights of preferences, so even fully satisfying each user need not produce a high GF. In addition, GU and GF are evaluated against post-compromise preferences: removing a rejection can improve the score without changing the final itinerary. This can capture agreement, but does not by itself establish that original needs are better accommodated or reveal whether concessions disproportionately burden particular members. Consequently, the interpretation of low fairness and coordination quality remains ambiguous. Analyses using individually normalized utilities, together with satisfaction relative to original preferences and the distribution of concessions, are needed.

Q1: Do the main model rankings and conclusions about insufficient fairness remain stable when fairness is computed using individually normalized utilities and scores are reported against both original and post-compromise preferences? How evenly are concessions distributed across users?

Thank you very much for these valuable questions. We understand that your main concerns are whether preference scale affects GF and whether the compromise mechanism affects GU and GF. We agree that conducting further experiments and analyses on these factors helps strengthen the reliability of our metrics. Our responses are as follows.

Impact of Preference Scale on GF

In theory, GF can indeed be bounded by differences in preference-table size, which might make unnormalized GF naturally low—meaning that even fully satisfying every user need not yield a high GF. In practice, however, our preference tables preserve realistic preference heterogeneity while also ensuring that preference scales remain roughly comparable across users.

First, real-world user preferences are inherently heterogeneous—for real users, preference scales across individuals are naturally different. Therefore, when synthesizing user preferences, we do not force preference scales to be strictly identical; instead, we apply soft controls (e.g., constraining total attraction preferences across must, reject, positive, and negative to 2–4 items) to keep preference-table scales comparable. For more details, please refer to the "Task Synthesis and Preference Scale" subsection in our response to Reviewer DUau.

At the same time, preference-table size does not fully determine attainable utility. When computing individual utility, aside from must-level attraction preferences, most preferences are category-level (such as attraction categories and food categories), which are counted multiple times during scoring (e.g., if User A prefers park attractions, scheduling both Park B and Park C awards 1 point of utility each). Thus, attainable utility depends on the concrete itinerary schedule. Accordingly, under an empirically assumed itinerary skeleton, we estimated each user's attainable utility assuming every user's preferences are maximally satisfied (for details, please refer to the "Challenges and Approach for Utility Estimation" subsection in our response to Reviewer DUau), and used this to normalize individual utilities. The results are shown below:

ModelGF (Raw)RankGF-normRank
DeepSeek-V4-Pro54.22155.321
Qwen3.6-Max52.11252.952
Qwen3-235B-Th48.95349.463
Qwen3-30B-Th48.63448.924
GPT-5.148.14548.715
Qwen3-4B-Th47.57647.686
Qwen3.5-Plus44.93745.437
Qwen3.5-Flash41.85842.288
Qwen3-4B-It41.72941.999

As seen from the results, normalized GF shows a slight increase over raw GF across all models, but the gains are very limited, and model rankings remain completely stable. This is primarily because within each task, users' attainable utilities differ very little (the mean within-task min/max ratio is 0.936). This indicates that GF is not driven by differences in preference scale: within each task, users' attainable utilities are comparable, confirming that our GF metric is accurate and that our finding of low fairness holds.

AuthorsOpenReview ↗

Rebuttal Part 2

Impact of Concessions on GU and GF

Your main concern here is that models might over-exploit the compromise mechanism (frequently eliciting concessions from multiple users or repeatedly burdening one user), leading to a high GU on a final plan that deviates substantially from users' original needs. We respond as follows.

First, by the design of our interaction protocol, the agent cannot abuse the compromise mechanism:

  • Whether a user can compromise is pre-specified (reflecting that in real life, some people are easygoing and willing to concede, whereas others are firm and unwilling to compromise).
  • Even compromisable users are capped at making at most 2 concessions, so the number of concessions per user can only take values in {0, 1, 2}.

Even in the worst case, one member in the group concedes twice while others do not concede at all (placing compromises on a single user). Hence, the agent cannot repeatedly exploit concessions to inflate its score (such as stripping away reject items for all users); we introduced the compromise mechanism primarily to give the agent a tool for mediating conflicts so as to achieve higher group utility.

Second, we analyzed the empirical distribution of concessions across trajectories:

  1. Across all 650 tasks (2,748 users in total), there are 1,490 compromisable users (54.2%) and 1,258 non-compromisable users (45.8%), a roughly balanced split.

  2. For each compromisable user, we computed the distribution of concession counts per model (as noted above, valid values are 0, 1, or 2; ≥3 occurs rarely due to prompt non-compliance):

Model0 times1 time2 times≥3 times
DeepSeek-V4-Pro79.2%19.2%1.5%0.022%
GPT-5.190.7%9.1%0.1%0
Qwen3.6-Max89.4%10.1%0.4%0
Qwen3.5-Plus86.8%12.6%0.6%0
Qwen3.5-Flash90.8%9.0%0.2%0
Qwen3-235B-Th90.0%9.7%0.3%0
Qwen3-30B-Th89.8%10.1%0.2%0
Qwen3-4B-Th93.4%6.5%0.1%0
Qwen3-4B-It90.3%9.6%0.1%0
All Models88.9%10.7%0.4%0.002%

As shown in the table, models severely under-utilize the compromise mechanism: even for DeepSeek-V4-Pro, which achieves the highest GU, 79.2% of compromisable users are never engaged for a concession, indicating that current models still lack strong conflict-handling and coordination abilities.

  1. Within each task, among compromisable users, we use the standard deviation of concession counts to measure how evenly concessions are distributed (a standard deviation closer to 0 indicates a more even distribution).

Since this requires at least 2 compromisable users to be well-defined, we compute it over trajectories that have at least 2 compromisable users and at least one accepted concession:

ModelAvg. Concessions per Compromisable UserAvg. Std. Dev.
DeepSeek-V4-Pro0.5040.471
GPT-5.10.3720.454
Qwen3.6-Max0.4100.468
Qwen3.5-Plus0.4150.478
Qwen3.5-Flash0.3830.466
Qwen3-235B-Th0.3800.468
Qwen3-30B-Th0.3610.462
Qwen3-4B-Th0.3480.456
Qwen3-4B-It0.4040.457
All Models0.4060.466

As shown, the mean concession count is 0.406 and the mean standard deviation is 0.466, meaning most within-group patterns look like {0, 1} or {0, 0, 1, 1}. The low standard deviation shows that concessions are evenly distributed (rather than piled onto a single user), while the low mean shows that models still under-utilize the compromise mechanism (yielding very few effective concessions).

AuthorsOpenReview ↗

Rebuttal Part 3

Because concessions are both infrequent and evenly distributed, the pre- and post-compromise preference tables differ very little. We therefore expect little difference in final metrics regardless of which preference table is used for scoring. The comparison below confirms this (eff uses post-compromise preferences; orig uses original preferences):

ModelGU effGU origGF effGF origGF-norm (orig)
DeepSeek-V4-Pro10.4710.4154.2253.9555.08
GPT-5.17.237.2248.1448.2548.90
Qwen3.6-Max9.679.6452.1152.1353.03
Qwen3.5-Plus8.648.6144.9344.8045.42
Qwen3.5-Flash6.996.9841.8541.9242.37
Qwen3-235B-Th7.357.3448.9549.0949.68
Qwen3-30B-Th6.236.2248.6348.6048.96
Qwen3-4B-Th5.735.7247.5747.5047.66
Qwen3-4B-It6.136.1241.7241.5241.82

Consistent with our inference, evaluating against original preferences yields nearly identical scores to evaluating against post-compromise preferences, and all model rankings remain unchanged.

In summary, these results demonstrate that in our evaluation framework, users' preference scales (total attainable utilities) are comparable, and the compromise mechanism is bounded and evenly distributed across users. Thus, GF and GU objectively reflect models' planning and coordination capabilities, reinforcing our conclusions regarding insufficient fairness and coordination quality.

Additionally, regarding whether GU and GF should be computed on original preferences: because current statistics reflect weak utilization of compromises by existing models, future models with stronger coordination skills may elicit more concessions, leading to slightly larger differences between final and original preference tables (though still bounded by our cap of 2 concessions per user). Even then, we still recommend evaluating on post-compromise preferences, as this captures a key characteristic of group travel—user preferences can evolve through negotiation. For example, if User A initially rejects visiting place X, but agrees to become neutral after learning that both B and C strongly want to go, group utility is more appropriately evaluated against the agreed post-compromise state. Evaluating against original preferences measures how well initial desires are met—which is certainly a valid auxiliary indicator to report—but in group travel planning and negotiation, we believe evaluation should capture such negotiated preference updates, which also reflects the model's coordination ability.

Reviewer_xbEkOpenReview ↗
Rating
4
Confidence
2(−1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces GroupTravelBench, a multi-user, multi-turn travel planning benchmark for LLM agents. It constructs 650 tasks across three levels, 22 social archetypes and 143 destination cities in China, grounds real 3718 user profiles 338K POIs and 253533-entry cache tool data to evaluate results. Each task runs in a controlled synchronous group-chat framework, where agent interacts with user simulators to elicit hidden preferences, coordinate conflicts (compromise and group splitting) to generate an itinerary. Performance is measured via four-tier hierarchy preferences in three coexisting tables (original, agent-inferred, post-compromise), across four rule based outcome metrics and an LLM process evaluation. 9 frontier and lightweight LLMs are used to benchmark results, revealing that the task is challenging for smaller instruction-tuned models, but even top models are limited by missing intra-city transport continuity.

Critical strengths
  1. The benchmark introduced in the paper is novel, reproducible and valuable for LLM agents on multi-user, multi-turn dynamics.
  2. The benchmark also introduced various methodology in detail, e.g. three coexisting preference tables, cached sandboxes without live API, and also designed comprehensive and solid experiment metrics for evaluation, e.g. feasibility check and simulator compliance auditing.
Critical weaknesses
  1. The related work section is comprehensive across travel-specific planning, but could be further strengthened by broader multi-agent negotiation and social choice benchmarks to contextualize how group consensus and compromise are handled.
  2. some user simulators operate under prompt constraints (like suggesting itinerary ordering) but these are real human dynamics, can probably extend to idealized dialogue partners.
  3. geographic scope and cultural influences are limited to domestic travel in China.
Questions for authors
  1. what would the Plan Validity scores be if using automated rule-based post processor to repair simple spatial continuity gaps intra-city?
  2. the reasoning for GPT-5.1 mentioned it being instruct mode but larger capacity, yet it still scored low on PV, how is that related?
6 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Thank you for taking the valuable time to review our paper, and we appreciate your positive recognition of our work as well as your constructive comments. Below, we first address your concerns and questions, and we commit to organizing these revisions and updating the paper PDF as soon as possible over the next few days (before the end of the discussion period).

R to W1

The related work section is comprehensive across travel-specific planning, but could be further strengthened by broader multi-agent negotiation and social choice benchmarks to contextualize how group consensus and compromise are handled.

Thank you very much for this suggestion. We agree with your view that incorporating broader work on multi-agent negotiation and social choice helps better contextualize our work. After reviewing the literature, we plan to add related work along the following lines:

  • Negotiation benchmarks (e.g., Deal or No Deal, CaSiNo, NegotiationArena, LLM-Stakeholders, Sotopia, MultiAgentBench).
  • Social choice and group recommendation, including recent LLM-based consensus work (Habermas Machine, Generative Social Choice).
  • Multi-party scheduling (Natural Plan).

Our reflections and how GroupTravelBench differs from these lines of work are summarized below:

  • In existing negotiation benchmarks, each agent usually represents one party, and success is the deal reached. In our setting, the agent is a neutral coordinator serving all users.
  • Social choice and group recommendation treat preferences as given inputs to aggregate. We require preferences to be elicited through dialogue, and we track how accepted compromises change the preference state with the three preference tables.
  • Existing consensus and negotiation tasks end in a decision or a statement. We require the outcome to become an executable itinerary that passes hard structural checks and is scored on both utility and fairness.

We will incorporate this content into the main text as Section 2.4.

R to W2

some user simulators operate under prompt constraints (like suggesting itinerary ordering) but these are real human dynamics, can probably extend to idealized dialogue partners.

Thank you very much for raising this question. We understand your concern that constraining certain behaviors of the user simulator may represent a somewhat idealized setup compared with real human users.

We acknowledge that our constraints on the user simulator may not fully replicate real-world group-chat behaviors. However, as noted in Section 3.3 of the paper, we restrict user behaviors primarily out of concern that unconstrained user actions could confound our evaluation of the agent. Specifically:

  • Prohibiting suggestions on itinerary planning → ensures that GU and GF are attributed to the agent;
  • Prohibiting volunteering preferences proactively → ensures that PC measures the agent's elicitation ability;
  • Prohibiting @-mentioning other users or chitchat → ensures that conflicts must be discovered and mediated by the agent, while also ensuring that the dialogue converges.

At the same time, we retained as many realistic behaviors as possible that do not interfere with metric attribution:

  • In the opening message, users reveal 1–3 preferences with tone matching preference strength, and they are allowed to proactively interject when a strong preference is violated;
  • Users can proactively ask the agent for factual information, such as attraction opening hours or restaurant recommendations;
  • In response to the agent's inquiries, users can make compromises.

Overall, our group-chat environment is designed more like an environment with potential user confounding factors removed, so that the relevant metrics can be cleanly attributed to the agent's behavior. Without constraints on the user simulator, group chats might fail to terminate properly (e.g., users chatting among themselves or discussing plans directly) and distort the final evaluation metrics. We also acknowledge that this trade-off sacrifices some real-world user behaviors, and future work could explore building more realistic group-chat environments with a better trade-off between evaluability and realism.

AuthorsOpenReview ↗

Rebuttal Part 2

R to W3

geographic scope and cultural influences are limited to domestic travel in China.

Thank you for pointing out the concern regarding geographic coverage and cultural influences. We agree on the importance of geographic scope and cultural diversity in benchmark evaluation. However, because we do not have access to travel-related data from other countries (such as POI data and user profile data, which may require partnerships with specific map providers like Google Maps), we are currently restricted to travel planning within China.

That said, we would also like to clarify that although our dataset is limited to domestic travel in China, the paper itself provides a clear task-construction pipeline, evaluation philosophy, and framework. This means that if members of the community have access to such data elsewhere, our setup can be easily extended to travel in other countries and regions. We will also seek broader collaborations to expand the geographic coverage and cultural diversity of the benchmark.

R to Q1

what would the Plan Validity scores be if using automated rule-based post processor to repair simple spatial continuity gaps intra-city?

Thank you for this question. We understand that you may want to know how many of the missing intra-city transportation errors are trivial gaps that can be easily fixed. Below, we first categorize the missing intra-city transportation cases, identify which ones can be repaired by rules, and then report the post-repair PV scores.

Regarding missing intra-city transportation, our current check logic requires an intra-city transport segment whenever two consecutive activities have different locations. Under this criterion, we analyzed all plans generated by the models in the main table and grouped these cases into the following categories:

  • A: The two activities correspond to the exact same place with slightly different names—e.g., Xiamen Haishang Yage Hotel vs. Xiamen Haishang Yage Hotel (Zhongshan Road Pedestrian Street Branch)—accounting for 7.7%. Such errors can be fixed via post-processing, and we believe this is more of a misjudgment by the checker (no transport is actually needed between them, and we will incorporate this fix into our PV checking rules).
  • B: Two consecutive activities are within walking distance and the next location is a specific POI—e.g., Confucius Temple (Fuzimiao) → Nanjing Impressions (Confucius Temple Branch)—accounting for 13.4%. The vast majority of these can be repaired via post-processing.
  • C: The next location is a placeholder anchored to the previous location—e.g., Wuxi Zoo → Light meal inside Wuxi Zoo—accounting for 13.3%. These cannot be repaired, because the destination is unspecified and cannot be patched by rules.
  • D: The two locations are far apart, accounting for 16.5%. These can be partially repaired automatically (depending on whether sufficient transit time is left between the two activities).
  • E: Necessary information for judgment is missing—e.g., an endpoint never appeared in any tool output, or it is a generic placeholder such as a local specialty restaurant—accounting for 49.1%. These cannot be repaired by rules, as there is no way to determine the origin and destination.

After repairing Category A, most of Category B, and the repairable part of Category D, the results are as follows:

ModelOriginal PVRankRepaired PVRank
Qwen3.5-Plus12.38112.771
DeepSeek-V4-Pro8.15211.362
Qwen3.6-Max7.3638.413
Qwen3-235B-Th3.3344.004
Qwen3.5-Flash1.3161.855
Qwen3-4B-It1.4951.696
Qwen3-30B-Th1.2371.547
GPT-5.11.0581.448
Qwen3-4B-Th0.2190.219

As can be seen, after post-processing repair, the models' overall rankings are largely unaffected, and PV shows only a modest increase. This is due to the multiple checks in PV: even if this particular gap is repaired, a plan still cannot earn a valid PV score if it fails other checks. This suggests that in group travel planning, current models may struggle to attend to every detail, and generating a spatio-temporally feasible plan remains challenging.

AuthorsOpenReview ↗

Rebuttal Part 3

R to Q2

the reasoning for GPT-5.1 mentioned it being instruct mode but larger capacity, yet it still scored low on PV, how is that related?

We sincerely apologize that our unclear explanation caused a misunderstanding. Here, we would like to distinguish between two things:

  • A: Being able to carry out multi-turn tool calls and interactions and produce a plan.

  • B: The validity of the generated plan (i.e., PV), for which A is a prerequisite.

When comparing GPT-5.1 with open-source instruct models, our intention was to show that smaller open-source instruct models cannot even complete A (falling into repetitive tool calls and failing to generate a parseable plan). In contrast, GPT-5.1 has larger capacity and stronger capability, so despite being an instruct model, it is able to complete Task A—yet this does not mean it can perform Task B well. Therefore, regarding your question, the two are actually unrelated; our point was solely about the relationship between an instruct model's capacity and completing A.

We hope our responses help resolve your questions. Once again, thank you for your valuable time and constructive comments, and we will organize these updates into the paper PDF as soon as possible.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2)

Dear Area Chair and Reviewers,

We sincerely thank you for your valuable time and insightful feedback. We are encouraged that our problem setting and core contributions have been recognized:

  • Extending travel planning to multi-user, multi-turn interaction with private preferences and conflicts, providing a novel and valuable task setting (xbEk, v2cL, DUau, Nrof);
  • Using three coexisting preference tables with compromise and subgrouping to deterministically track preference elicitation and state updates, offering a useful design for studying multi-user coordination (xbEk, v2cL, DUau);
  • Building a reproducible cached sandbox and evaluation framework verified by extensive reliability checks (xbEk, v2cL, DUau, Nrof).

The reviewers' initial questions mainly focused on metric reliability, attributing the difficulty introduced by multi-user travel planning, task solvability, user-simulator constraints, and broader related work.

To address these points, we conducted additional controlled experiments and analyses, and uploaded a revised paper PDF (revisions highlighted in blue) for your reference. Detailed point-by-point responses and experimental tables are provided in our individual replies to each reviewer. The table below summarizes our key updates and their locations.

Focus AreaReviewer(s)Revised / Added SectionSummary of Revisions and Key Conclusions
1. Preference scale & GF-normv2cL, DUau, Nrof§3.2, §4.1, §5.2, Tab. 1, App. A.1, C.17 (Tabs. 23–25)Explained scoring rationale; quantified within-task positive-preference range (mean 2.64) and attainable-utility min⁡/max⁡\min/\max ratio (mean 0.936), and reported normalized GF-norm across all 9 models—showing minimal score shifts and completely unchanged rankings, confirming that GF is not distorted by preference-table scale and reliably measures group planning fairness.
2. Concessions & eff vs. origv2cL§4.1, App. C.7 (Tab. 17), C.17 (Tab. 25)Reported concession counts (88.9% zero among compromisable users) and within-task evenness (mean 0.406, std 0.466), plus eff vs. orig scoring—showing nearly identical GU/GF and identical rankings.
3. Coordination vs. PV difficultyv2cL, NrofAbstract, §1, §4.1, §5.2, §6, App. C.2 (Tabs. 12–13)Clarified that GU/GF measure multi-user coordination while PV checks general feasibility; added PV breakdown by NN and duration, plus a matched 2×22\times 2 {Orig, Deconf} × {Std., Oracle} ablation on all 650 tasks—showing +Deconf and +Oracle raise GU/GF while PV barely changes, cleanly separating multi-user coordination bottlenecks (in GU/GF) from general spatio-temporal scheduling difficulty (in PV).
4. Task solvability & referencesv2cL, Nrof§3.2, §5.2, App. C.17 (Tab. 26)Built rule-based greedy reference plans (gu, leximin, gf) achieving PV=100%\text{PV}=100\% across all 650 tasks with substantially higher GU/GF (e.g., GU=20.64,GF=84.04%\text{GU}=20.64, \text{GF}=84.04\% under leximin), proving that every task admits valid solutions and that low agent scores stem from model capability gaps rather than infeasible task specifications.
5. PV errors, repair & valid plansxbEk, v2cL§5.2, App. C.9 (Tabs. 18–20)Added task-level PV error occurrence and sole-cause rates; categorized missing intra-city transport (A–E) and reported post-repair PV across all 9 models; and compared GU/GF on valid vs. invalid plans—showing modest PV gains with stable rankings after repair, confirming that low PV reflects genuine multi-constraint scheduling failures rather than trivial gaps, and does not confound GU/GF conclusions.
6. User-simulator sensitivityxbEk, Nrof§3.3, App. C.3 (Tab. 14)Clarified how simulator constraints ensure clean metric attribution without rewarding exhaustive questioning (under capped turns and ≤2\le 2 concessions), and added a relaxed-simulator study (Orig. vs. Relax)—showing higher PC but modest GU gains and slightly lower GF, confirming that our controlled protocol does not reward exhaustive probing
7. Broader related workxbEk§2.4Added Multi-Agent Negotiation and Group Decision Making, contrasting our setting with negotiation benchmarks, social choice / group recommendation, and multi-party scheduling.
AuthorsOpenReview ↗

General Response (2/2)

During the discussion period, we are grateful that Reviewers xbEk and Nrof confirmed that our responses and revisions have addressed their concerns. While Reviewers v2cL and DUau have not yet had a chance to post follow-up comments, their main concerns—such as metric reliability, difficulty attribution, and task solvability—closely overlap with those raised by xbEk and Nrof. We therefore hope that our responses and additional results satisfactorily address their concerns as well.

In particular, through a series of controlled experiments and analyses, we verified both the reliability of our evaluation metrics and the solvability of all benchmark tasks, and provided a simple rule-based greedy baseline showing that current models still face clear challenges in resolving inter-user conflicts and generating spatio-temporally feasible plans. Together, these results also confirm that our metrics can accurately disentangle and evaluate distinct aspects of an agent's capabilities—spanning preference elicitation (PC), multi-user conflict coordination and fairness (GU and GF), spatio-temporal planning feasibility (PV), and interaction process quality (LLM judge).

Once again, we sincerely thank the Area Chair and all reviewers for their time and valuable feedback, which have greatly improved the quality and rigor of our paper. We hope that our work provides a solid foundation for future research on multi-user interactive agents.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.