- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces LODGE, a conversational benchmark for evaluating accommodation-booking agents across 360 tasks (with 300 forming the fixed evaluation split) over a shared catalog of 3,660 synthetic listings. By generating hidden user goals conditioned on a contrastive set of one target listing and eight partial-match alternatives—followed by human verification—each task has a uniquely correct booking outcome by construction, enabling deterministic evaluation without relying solely on LLM judges. The benchmark varies task difficulty along composable axes while keeping the definition of success fixed: GEOFLEX (varying whether the stated destination is the exact locality, a nearby anchor within 50 km, or a broad administrative region) and VAGUENESS (varying how readily the user simulator discloses preferences). Evaluations across seven open- and closed-weight agents reveal that high booking completion rates mask substantial wrong-locality failures, that searching outward from a plausible nearby anchor (NEARBY) is consistently harder for agents than refining a broad region (BROAD), and that small-scale LoRA fine-tuning can improve geographic adaptation on difficult splits.
Critical strengths
-
Contrastive construction yields a scalable, judge-free ground truth for multi-turn item selection: A recurring challenge in interactive benchmarks is that tasks either lack a single ground-truth answer or rely heavily on subjective LLM judges. By synthesizing a target listing alongside partial-match alternatives in geographically disjoint areas and conditioning goal generation on this contrastive set, LODGE makes the primary metric (
Booked Target) completely mechanical while bounding human verification cost at comparisons per task regardless of total catalog size. -
Controlled axes expose counter-intuitive failure modes within the evaluated setting: Separating
Booked AnyfromBooked Targetacross the GEOFLEX levels clearly shows how models like Qwen3.8-27B achieve a 99.7% booking rate by confidently booking in the wrong city (43.3% wrong-locality rate on NEARBY). Moreover, the empirical finding in Table 2 that NEARBY is consistently harder than BROAD for all seven agents—because a plausible anchor returning partial matches lulls the agent into staying put, whereas an empty region search forces refinement—is an interesting behavioral observation.
Critical weaknesses
-
Overly narrow problem scope with no empirical validation of broader generalizability: Despite framing the contribution around conversational search and travel agents broadly, the benchmark is restricted entirely to a single narrow subtask—selecting one accommodation listing using three basic tools (
search_listings,get_listing_details,create_reservation). It omits the multi-step itinerary planning, cross-tool coordination, and budget trade-offs that define realistic travel agents; in fact, the authors explicitly concede in Appendix E (line 1346) that their contrastive single-target recipe cannot transfer to itinerary planning where no single item is correct. Even for general conversational search, there are no experiments extending the framework to other domains, and the primary axis (GEOFLEX) is tightly coupled to geographic coordinates/administrative regions and an artificially restricted search API (which only accepts exact city strings and lacks basic radius or regional search). As a result, low scores on NEARBY/BROAD heavily conflate search strategy with whether an LLM has memorized obscure world geography across 75 countries in its parametric weights (e.g., recalling from memory that Antsiranana is in the Diana region of Madagascar in Appendix A). -
Confounding simulator/termination artifacts and missing details in the SFT study: Several key failures appear driven by simulator or harness quirks rather than true agent deficits. In Appendix A (GPT-4.1 Mini transcript), when the agent reasonably asks the user for nearby area suggestions after an empty search, the user simulator itself guesses wrong non-target towns ("Nosy Be or Ambanja") and misleads the agent. In the L2 vagueness transcript (Appendix A, line 850), GPT-5 fails to book only because the simulator says "Before you submit, can you confirm the nightly rate...", and the episode terminates immediately after the agent answers that question. Penalizing the agent for following a "confirm before submitting" request is a termination artifact rather than a failure of preference elicitation. Finally, while the Reproducibility Statement (line 497) states that "Appendix D.5 gives the fine-tuning configuration and hardware" and Table 10 uses "150 curated demonstrations", Appendix D.5 omits all LoRA hyperparameters/hardware details, and it is unclear where 150 training tasks come from when Section 4.1 states that only 60 released tasks exist outside the 300-task evaluation split.
Questions for authors
-
Generalizability beyond single-item accommodation search: Since the paper claims the axis framework and findings generalize to conversational search agents broadly, yet Appendix E notes the construction cannot transfer to multi-item planning and all 360 tasks are limited to lodging with an artificially restricted city-string search tool, can you provide any empirical evidence or concrete demonstration that the GEOFLEX/anchoring findings hold in other domains or with realistic search APIs that natively support spatial/regional queries?
-
Disentangling geographic memorization from search strategy: In NEARBY and BROAD tasks, if an agent does not have the neighboring town names of a locality in one of the 75 countries memorized in its weights, and asking the user simulator leads to hallucinated/wrong town suggestions (as in Appendix A), how is the agent expected to succeed? How much of the NEARBY/BROAD failure rate is simply a lack of parametric geographic knowledge of smaller countries?
-
Clarification on SFT data split and L2 termination artifacts: Could you clarify where the 150 training tasks in Table 10 come from (given non-evaluation tasks in the release), provide the missing LoRA configurations promised in Appendix D.5, and clarify what fraction of L2 "no-booking" failures are triggered by the simulator's "confirm before you submit" phrasing rather than genuine agent abstention?
Follow-up for Reviewer PqEz
Thank you again for your review. In our responses above, we clarified the scope of the benchmark, added a place-lookup control and a recall probe for the geographic-knowledge concern, quantified how much of the L2 failure comes from the confirm-before-submit ending, and described the fine-tuning split and training settings. If you have any further questions or concerns, please let us know and we will respond promptly while the discussion is open.