acceptodds
Under review as a conference paper at ICLR 2027

LODGE: A CONVERSATIONAL TRAVEL AGENT BENCHMARK WITH GROUNDED GOALS AND CONTROLLABLE DIFFICULTY

Abstract

Benchmarks for conversational search agents typically assume a highly specific target: the user knows exactly which flight to buy or which hotel to book. Real users usually arrive with underspecified requests. Evaluating such agents therefore requires tasks whose correct outcome is known, with difficulty that can be varied deliberately rather than incidentally. We present LODGE, a benchmark of 360 conversational accommodation-booking tasks, 300 of them a fixed evaluation split on which all results here are reported, each with a unique bestfitting target listing established by contrastive construction and confirmed by human review, over a shared catalog of synthetic listings. Difficulty is expressed as composable axes that vary one property of the task while the definition of success stays fixed. GEOFLEX varies whether the named destination is the target locality (PRECISE), an anchor to search beyond (NEARBY), or a region to refine into localities (BROAD); VAGUENESS varies how explicitly the same goal is disclosed. Across seven language agents, these axes separate capabilities that aggregate accuracy conflates: booking completion is nearly uninformative about task success, NEARBY is harder than BROAD for every agent despite supplying more locational information, and robustness to vagueness is independent of geographic competence. Small-scale supervised fine-tuning improves the difficult settings disproportionately, with results consistent with transfer to held-out countries. The dataset, user simulator, and evaluation harness will be released upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, benchmark, synthetic data, conversational search, geographic reasoning, user simulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PqEzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LODGE, a conversational benchmark for evaluating accommodation-booking agents across 360 tasks (with 300 forming the fixed evaluation split) over a shared catalog of 3,660 synthetic listings. By generating hidden user goals conditioned on a contrastive set of one target listing and eight partial-match alternatives—followed by human verification—each task has a uniquely correct booking outcome by construction, enabling deterministic evaluation without relying solely on LLM judges. The benchmark varies task difficulty along composable axes while keeping the definition of success fixed: GEOFLEX (varying whether the stated destination is the exact locality, a nearby anchor within 50 km, or a broad administrative region) and VAGUENESS (varying how readily the user simulator discloses preferences). Evaluations across seven open- and closed-weight agents reveal that high booking completion rates mask substantial wrong-locality failures, that searching outward from a plausible nearby anchor (NEARBY) is consistently harder for agents than refining a broad region (BROAD), and that small-scale LoRA fine-tuning can improve geographic adaptation on difficult splits.

Critical strengths
  1. Contrastive construction yields a scalable, judge-free ground truth for multi-turn item selection: A recurring challenge in interactive benchmarks is that tasks either lack a single ground-truth answer or rely heavily on subjective LLM judges. By synthesizing a target listing alongside K=8K=8 partial-match alternatives in geographically disjoint areas and conditioning goal generation on this contrastive set, LODGE makes the primary metric (Booked Target) completely mechanical while bounding human verification cost at K+1=9K+1=9 comparisons per task regardless of total catalog size.

  2. Controlled axes expose counter-intuitive failure modes within the evaluated setting: Separating Booked Any from Booked Target across the GEOFLEX levels clearly shows how models like Qwen3.8-27B achieve a 99.7% booking rate by confidently booking in the wrong city (43.3% wrong-locality rate on NEARBY). Moreover, the empirical finding in Table 2 that NEARBY is consistently harder than BROAD for all seven agents—because a plausible anchor returning partial matches lulls the agent into staying put, whereas an empty region search forces refinement—is an interesting behavioral observation.

Critical weaknesses
  1. Overly narrow problem scope with no empirical validation of broader generalizability: Despite framing the contribution around conversational search and travel agents broadly, the benchmark is restricted entirely to a single narrow subtask—selecting one accommodation listing using three basic tools (search_listings, get_listing_details, create_reservation). It omits the multi-step itinerary planning, cross-tool coordination, and budget trade-offs that define realistic travel agents; in fact, the authors explicitly concede in Appendix E (line 1346) that their contrastive single-target recipe cannot transfer to itinerary planning where no single item is correct. Even for general conversational search, there are no experiments extending the framework to other domains, and the primary axis (GEOFLEX) is tightly coupled to geographic coordinates/administrative regions and an artificially restricted search API (which only accepts exact city strings and lacks basic radius or regional search). As a result, low scores on NEARBY/BROAD heavily conflate search strategy with whether an LLM has memorized obscure world geography across 75 countries in its parametric weights (e.g., recalling from memory that Antsiranana is in the Diana region of Madagascar in Appendix A).

  2. Confounding simulator/termination artifacts and missing details in the SFT study: Several key failures appear driven by simulator or harness quirks rather than true agent deficits. In Appendix A (GPT-4.1 Mini transcript), when the agent reasonably asks the user for nearby area suggestions after an empty search, the user simulator itself guesses wrong non-target towns ("Nosy Be or Ambanja") and misleads the agent. In the L2 vagueness transcript (Appendix A, line 850), GPT-5 fails to book only because the simulator says "Before you submit, can you confirm the nightly rate...", and the episode terminates immediately after the agent answers that question. Penalizing the agent for following a "confirm before submitting" request is a termination artifact rather than a failure of preference elicitation. Finally, while the Reproducibility Statement (line 497) states that "Appendix D.5 gives the fine-tuning configuration and hardware" and Table 10 uses "150 curated demonstrations", Appendix D.5 omits all LoRA hyperparameters/hardware details, and it is unclear where 150 training tasks come from when Section 4.1 states that only 60 released tasks exist outside the 300-task evaluation split.

Questions for authors
  1. Generalizability beyond single-item accommodation search: Since the paper claims the axis framework and findings generalize to conversational search agents broadly, yet Appendix E notes the construction cannot transfer to multi-item planning and all 360 tasks are limited to lodging with an artificially restricted city-string search tool, can you provide any empirical evidence or concrete demonstration that the GEOFLEX/anchoring findings hold in other domains or with realistic search APIs that natively support spatial/regional queries?

  2. Disentangling geographic memorization from search strategy: In NEARBY and BROAD tasks, if an agent does not have the neighboring town names of a locality in one of the 75 countries memorized in its weights, and asking the user simulator leads to hallucinated/wrong town suggestions (as in Appendix A), how is the agent expected to succeed? How much of the NEARBY/BROAD failure rate is simply a lack of parametric geographic knowledge of smaller countries?

  3. Clarification on SFT data split and L2 termination artifacts: Could you clarify where the 150 training tasks in Table 10 come from (given 360−300=60360 - 300 = 60 non-evaluation tasks in the release), provide the missing LoRA configurations promised in Appendix D.5, and clarify what fraction of L2 "no-booking" failures are triggered by the simulator's "confirm before you submit" phrasing rather than genuine agent abstention?

1 comment
AuthorsOpenReview ↗

Follow-up for Reviewer PqEz

Thank you again for your review. In our responses above, we clarified the scope of the benchmark, added a place-lookup control and a recall probe for the geographic-knowledge concern, quantified how much of the L2 failure comes from the confirm-before-submit ending, and described the fine-tuning split and training settings. If you have any further questions or concerns, please let us know and we will respond promptly while the discussion is open.

Reviewer_UMa4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work presents LODGE to address the lack of travel-agent evaluation tasks that combine a checkable best booking with deliberately varied request difficulty. LODGE uses contrastive tasks with unique target listings, asks agents to infer a hidden goal through deterministic tools and a simulated user, and varies geographic specificity and goal disclosure while keeping the success criterion fixed. The experiments show that (i) booking any listing is much weaker evidence of success than booking the target; (ii) Nearby is harder than Broad for all seven agents; (iii) greater vagueness lowers target booking while largely preserving geographic search behavior; and (iv) LoRA fine-tuning yields larger gains on Nearby and Broad than on Precise.

Critical strengths
  • LODGE targets an important and underexplored evaluation setting. Real travel requests often require preference elicitation and geographic disambiguation before a commitment can be made. Unlike many travel-agent tasks that grade plans or rely on judge-defined satisfaction, LODGE constructs multi-turn booking tasks with a uniquely correct listing in a shared catalog, so the primary outcome can be read from the environment rather than assigned by a judge.

  • Its goal-preserving difficulty design produces substantive diagnostics. GeoFlex and Vagueness change task properties without changing the success criterion. Across seven agents, this design reveals nontrivial results: booking completion is not a proxy for success, Nearby is consistently harder than Broad, and vagueness robustness separates from geographic search behavior. These findings give the benchmark value beyond another leaderboard.

Critical weaknesses
  • Construction and external validity need more evidence. Listings, goals, and the uniqueness test depend heavily on the generator and its satisfaction predicate. The paper says every goal was human-reviewed, but it does not report reviewer counts, qualifications, edit/rejection rates, agreement, or an independent audit rate. The synthetic catalog and fixed simulator persona also limit how far absolute scores should be read as predictions for real travel search.

  • Some controls and uncertainty estimates are missing. Locality-only search can conflate search policy with parametric geographic knowledge; the authors themselves identify a near-ceiling issue in Precise for many tasks. A single-shot retrieval or random control, sensitivity analysis for turn/stall/result limits, and task-clustered confidence intervals or paired tests for model contrasts would make the central comparisons easier to interpret.

Questions for authors
  • Please quantify the human-review process for hidden goals, including reviewer count and qualifications, failure rates, edits/rejections, agreement, and any independent audit of the generator's satisfaction predicate.

  • How much of the Nearby/Broad failure reflects a lack of parametric geographic knowledge rather than poor search expansion or refinement? A locality-recall control or a gazetteer-tool condition would help isolate these factors, as would a single-shot retrieval baseline with the fully revealed goal.

2 comments
AuthorsOpenReview ↗

Follow-up for Reviewer UMa4

Thank you again for your review. In our responses above, we added details on the human-review process and the independent audit, the place-lookup, single-shot and recall controls, the turn-cap and stall-rule sensitivity checks, and task-clustered confidence intervals for the main comparisons. If you have any further questions or concerns, please let us know and we will respond promptly while the discussion is open.

Thanks for your responses, which address my concerns. I have no more concerns, and I will maintain my positive score.

Reviewer_rJKeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LODGE, a benchmark for conversational accommodation booking in which agents must clarify an underspecified request, search for suitable locations, and book the correct listing. It contains 360 tasks over a synthetic catalog, with a unique target established through contrastive construction and human review. The benchmark varies geographic flexibility and preference disclosure to examine different sources of difficulty, while scoring success by whether the target is booked. Experiments with seven agents on 300 evaluation tasks show that completing a booking often masks incorrect choices, and that searching beyond a named locality is particularly difficult. A small-scale fine-tuning study reports improvements concentrated in the harder geographic settings.

Critical strengths

The benchmark provides a clear, verifiable outcome for an underspecified conversational task. Contrastive construction and human review establish a unique target listing, allowing booking success to be scored without an LLM judge. This makes the main result easier to interpret and exposes failures that booking-completion rates would miss.

The controlled variants provide useful diagnostic evidence. The vagueness variants and locality-cue ladder hold the target and catalog fixed while changing what information the agent can obtain. These comparisons help distinguish difficulty finding the right locality from difficulty choosing a listing, making the benchmark more informative than an aggregate success score alone.

Critical weaknesses

The geographic comparisons do not fully support the claimed isolation of difficulty (Section 3.1; Table 2). Unlike the vagueness variants, GeoFlex levels use different tasks and geographic areas. Balancing task counts does not establish that locality familiarity, candidate distributions, and other sources of difficulty are comparable. The locality-only search interface also makes success depend on recalling alternative place names without geographic lookup tools. Consequently, the Nearby–Broad gap does not by itself establish that misleading anchors cause the difference. Stronger controls are needed because this interpretation is a central finding.

The vagueness results may partly reflect simulator and termination behavior rather than preference elicitation (Section 4.2; Appendices A and E). The paper’s L2 example shows an agent retrieving the target but failing to book after a conditional request to confirm costs. This tests interpretation of booking instructions as well as discovery of hidden preferences. All experiments use one simulator, and the turn cap and stall rules are not varied. These choices may materially affect abandonment rates and model rankings, weakening the claim that the axis cleanly isolates elicitation ability. Simulator replacement and termination-rule sensitivity checks would help establish that the findings hold beyond the chosen interaction protocol.

Questions for authors

Clarification on geographic controls: Beyond equal task counts, which properties are balanced across GeoFlex levels—for example, locality familiarity, candidate density, and listing ambiguity? What existing evidence supports attributing the Nearby–Broad gap specifically to anchoring?

Clarification on the vagueness manipulation: Does L2 change only preference disclosure, or also the wording and timing of booking authorization? In the example where the target is retrieved but booking fails after a conditional cost-confirmation request, how is this distinguished from an elicitation failure?

Clarification on termination effects: Can the existing logs show how much of the additional L2 failure comes from the turn cap, stall rule, or simulator termination after expressing booking intent? Have any existing checks established whether the reported rankings are stable to these settings? A broader simulator or parameter sweep would be a suggestion for future validation, rather than a requirement for the rebuttal.

2 comments
AuthorsOpenReview ↗

Follow-up for Reviewer rJKe

Thank you again for your review. In our responses above, we added a place-lookup control for the geographic confound, a closer account of how L2 differs from L1, and termination-rule and simulator-replacement checks. If you have any further questions or concerns, please let us know and we will respond promptly while the discussion is open.

Thanks for the responses. The new results clear some of my previous concerns, and I will raise my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.