acceptodds
Under review as a conference paper at ICLR 2027

MobiLens: A Human-Centered Benchmark for Personalized Mobility Agents

Abstract

Large language models have enabled mobility agents to interpret complex travel intents, use navigation tools, and tailor route and destination recommendations to individual users' needs, preferences, and contexts. However, existing benchmarks provide limited support for jointly assessing behavioral alignment and broader human-centered outcomes under a common evaluation protocol. To address this gap, we introduce MobiLens, a human-centered benchmark built from real-world user queries, pre-query user context, and observed mobility decisions. Inspired by naturalistic decision-making theory, our framework evaluates mobility agents across five human-centered outcome dimensions: personalization, diversity, fairness, explainability, and robustness, alongside process-level measures of tool-use success and operational efficiency. We benchmark 11 agent configurations spanning reasoning, memory augmentation, multi-agent coordination, and recommendation-oriented designs under a shared offline navigation-tool replay protocol. Our results reveal an execution–alignment gap: most agents reliably execute tools and produce format-compliant answers, yet show limited alignment with users' observed route and destination choices. Performance also varies across user groups and mobility scenarios, underscoring the need to evaluate behavioral alignment beyond aggregate execution success. Code and benchmark resources are available at https://anonymous.4open.science/r/MobiLens-F917.

Primary area:
datasets and benchmarks
Keywords:
Personalized Mobility Agents, Human-Centered Evaluation, Agent Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_a8xdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MobiLens, a human-centered benchmark for evaluating personalized mobility agents using real-world navigation data. It evaluates 11 agent configurations across personalization, diversity, fairness, explainability, robustness, tool-use success, and efficiency under an offline navigation environment. Experiments across three LLM backbones reveal an execution–alignment gap: agents can reliably use tools and produce valid answers, yet still align poorly with users’ actual choices.

Critical strengths
  1. The paper provides a carefully engineered, behavior-grounded benchmark with real-world mobility data, and a reproducible offline tool environment.
  2. The benchmark evaluates diversity, fairness, explainability, robustness, tool-use success, and efficiency, having rich metrics. This broader evaluation reveals important trade-offs between different base models and methods.
Critical weaknesses
  1. The paper mainly integrates existing agent frameworks, memory mechanisms, and evaluation dimensions into a unified mobility benchmark, without proposing a new agent architecture, learning algorithm, or personalization mechanism, may fall to limited conceptual and methodological novelty.
  2. For a benchmark paper whose main contribution is evaluation, the empirical section itself should be a central strength. However, the main text largely revolves around one aggregate table, with only one model, limited visualization or deeper diagnostic analysis.
  3. The dataset contains 1,608 episodes collected from a single navigation platform and consists of Chinese-language interactions. This relatively narrow setting makes it unclear whether the conclusions generalize to other regions, platforms, languages, or mobility systems.
  4. The user’s eventual route, transit plan, or POI choice is treated as the behavioral reference for Hit@K, but that observed choice is not necessarily the only suitable or preferred recommendation.
Questions for authors
  1. How sensitive are the personalization conclusions to the assumption that the single observed post-query decision is the appropriate reference?
  2. Since completeness and persuasiveness are central to the claimed human-centered evaluation, how robust are the LLM-judged explainability results to the choice of evaluator?
6 comments
AuthorsOpenReview ↗

Response to Reviewer a8xd [1/3]

We sincerely thank you for recognizing the careful construction of our benchmark, its grounding in real-world mobility behavior, and its reproducible tool environment, as well as the breadth of our evaluation and the trade-offs it reveals. We address your concerns below.

W1: The paper mainly integrates existing agent frameworks, memory mechanisms, and evaluation dimensions into a unified mobility benchmark...

Response: Thank you for this comment. MobiLens contributes a multidimensional evaluation protocol for personalized mobility agents. We link real-world requests and trip context to users’ pre-query profiles, behavioral histories, and observed post-query decisions. Across POI search, driving, and public transit, we use those decisions as references for recommendation ranking and evaluate behavioral alignment alongside task execution, diversity, fairness, explainability, and robustness.

To clarify the differences between MobiLens and existing benchmarks, the following comparison covers scoring references, agent configurations, and how each benchmark evaluates performance.

Evaluation dimensionMobiLensMobilityBenchMapSatisfyBench
Outcome scoringHit@K against observed user choicesRoute validity and constraint satisfaction using expert-defined tool workflowsSatisfaction of explicit and behavior-supported implicit factors
Agent configurations11 configurations across 4 categories (reasoning, memory-augmented, multi-agent, recommendation)ReAct and Plan-and-ExecuteReAct with a simulated user
Behavioral history accessPre-query behavioral records—Tool access and a separate profile-and-history experiment
Memory strategy comparisonPaired ReAct: last-k, summary, and retrieval vs. no history——
DiversityLabel variation and matched-option distinctness——
FairnessTask-standardized Hit@3 gaps and worst-group Hit@3——
RobustnessTime-of-day stability and hard-query retention——
Performance breakdownsPOI, driving, and transitRoute-planning task typeService domain and implicit-factor source

A dash indicates that the published protocol does not report a dedicated evaluation for that dimension.

We constructed 1,608 evaluation episodes from real navigation logs and reserved post-query behavior for scoring. We evaluated 11 agent configurations in a shared offline replay environment covering 19 navigation tools. We standardized navigation interfaces and task instructions while preserving each framework’s planning and memory mechanisms. Within each backbone, we paired the last-k, summary, and retrieval ReAct variants with the same no-memory baseline on identical episodes.

The evaluation across three LLM backbones yielded three main findings.

  1. Execution reliability and behavioral alignment can diverge. With Qwen3.7-Max, most configurations achieved tool invocation success rates above 94%, while the highest Hit@3 was 56.8%.

  2. Framework performance varies by mobility task. With Qwen3.7-Max, iAgent achieved the highest Hit@3 in POI search, while CAMEL led in driving and public transit. In our implementations, iAgent uses memory to rerank candidates, and CAMEL uses multi-round coordination between two agents.

  3. Memory organization and task relevance matter. Summarization provides more consistent gains over no-memory ReAct; history benefits driving and transit more consistently than POI selection.

W2: For a benchmark paper whose main contribution is evaluation, the empirical section itself should be a central strength...

Response: Thank you for this comment. Table 1 summarizes the results for Qwen3.7-Max, and Tables 2 and 3 in Appendix D.1 report the full evaluation of 11 agent configurations across three LLM backbones. We will add a concise comparison of task-level Hit@3 across all three backbones to the main text.

We also analyzed GPT-5.4 iAgent’s tool traces on 450 POI requests. The tool results contained a candidate matching the user’s observed choice in 212 requests, and the final top three matched that choice in 140. Thus, in at least 72 requests (16.0%), a matching candidate appeared in the tool results, yet none of the final top three matched the observed choice. We will include this analysis in the main text to show where matches were lost between tool retrieval and final recommendation.

AuthorsOpenReview ↗

Response to Reviewer a8xd [2/3]

W3: The dataset contains 1,608 episodes collected from a single navigation platform and consists of Chinese-language interactions...

Response: Thank you for your comment. MobiLens covers a range of mobility needs and user decision contexts: POI recommendation, driving, and public transit, and links real user requests with pre-query histories, request contexts, and subsequent mobility decisions. These episodes therefore capture varied request intents, travel conditions, and decision trade-offs, providing meaningful scenario diversity for evaluating personalized mobility agents.

Building on this data, our contribution is MobiLens, the first human-centered benchmark built from real-world user queries, pre-query user context, and observed mobility decisions. It provides an evaluation framework covering five human-centered outcome dimensions together with process-level measures, benchmarks 11 agent configurations across three LLM backbones under a shared offline navigation-tool replay protocol, and releases the benchmark episodes, tool-response cache, evaluation harness, and evaluation protocol.

Because releasing these artifacts is itself a key contribution,  other institutions can use the framework to construct comparable evaluations once suitable local data become available, and our shared offline replay protocol makes such extensions practical. Our work thus offers a concrete methodology for extending personalized mobility evaluation to additional settings and for supporting future cross-region research.

W4: The user’s eventual route, transit plan, or POI choice is treated as the behavioral reference for Hit@K, but that observed choice is not necessarily the only suitable or preferred recommendation.

Response: Thank you for this comment. A mobility request can have several suitable recommendations. MobiLens uses the user’s observed choice as a behavioral reference. Hit@K measures whether that choice appears among the agent’s top KK recommendations.

Using observed choices as a behavioral reference is consistent with prior work on implicit feedback, where clicks and purchases serve as signals of user preferences (Hu et al., 2008). In mobility applications, Barnes et al. (2024) evaluate predicted routes against recorded driving trajectories in Google Maps, and Baidu Maps’ Hydra uses clicks on displayed travel options as implicit feedback (Liu et al., 2019).

We will clarify in the revised metric definition that a Hit@K miss means the observed choice was absent from the top KK. Other recommendations may still be suitable for the request.

References

Hu, Y., Koren, Y., & Volinsky, C. (2008). Collaborative filtering for implicit feedback datasets. In 2008 Eighth IEEE International Conference on Data Mining (pp. 263–272). IEEE.

Barnes, M., Abueg, M., Lange, O. F., Deeds, M., Trader, J., Molitor, D., Wulfmeier, M., & O’Banion, S. (2024). Massively scalable inverse reinforcement learning in Google Maps. In The Twelfth International Conference on Learning Representations (ICLR).

Liu, H., Tong, Y., Zhang, P., Lu, X., Duan, J., & Xiong, H. (2019). Hydra: A personalized and context-aware multi-modal transportation recommendation system. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 2314–2324).

AuthorsOpenReview ↗

Response to Reviewer a8xd [3/3]

Q1: How sensitive are the personalization conclusions to the assumption that the single observed post-query decision is the appropriate reference?

Response: Thank you for raising this question. As discussed in W4, the observed post-query choice serves as our behavioral reference. To test how sensitive the estimated benefit of summary memory is to this reference, we rescored the same Qwen3.7-Max ReAct outputs on 150 randomly sampled requests, with 50 each for POI search, driving, and public transit. The expanded rule retained the observed choice and also credited alternatives that experts judged acceptable based on the request, context, and tool evidence.

Scoring ruleNo-memory ReAct Hit@3Summary-memory ReAct Hit@3Difference
Observed choice only52.0%60.7%+8.7 pp
Observed choice or an acceptable alternative78.0%84.0%+6.0 pp

On this sample, the estimated Hit@3 advantage of summary memory remained positive under both rules, decreasing from 8.7 to 6.0 percentage points when acceptable alternatives were credited (change of −2.7 pp, paired bootstrap 95% CI [−6.5, +0.9] pp).

Q2: Since completeness and persuasiveness are central to the claimed human-centered evaluation, how robust are the LLM-judged explainability results to the choice of evaluator?

Response: Thank you for this comment. We conducted an additional human evaluation of 100 explanation outputs. Three annotators independently rated all outputs on Faithfulness (FS), Completeness (CS), and Persuasiveness (PS) using the same 0–5 rubrics as GLM-5.2. Following a leave-one-annotator-out reference design (Calderon et al., 2025), we compared GLM-5.2 against the held-out annotator in each round, and measured agreement with Kendall's τ-b and mean absolute error (MAE).

The results show that GLM-5.2's scores closely match the human reference, with MAE of 0.31, 0.23, and 0.20 for FS, CS, and PS, respectively. Paired bootstrap resampling shows no significant MAE difference for FS and CS, and a slightly lower MAE for GLM-5.2 on PS; Kendall's τ-b indicates positive rank associations with human ratings across all three dimensions. These results support using GLM-5.2 to assist explanation-quality evaluation under shared scoring rubrics, and indicate that the LLM-judged explainability results are robust to the choice of evaluator. We will include the human evaluation protocol and results in the revised paper.

References

Calderon, N., Reichart, R., & Dror, R. (2025). The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 16051–16081). Association for Computational Linguistics.

Kendall, M. G. (1945). The treatment of ties in ranking problems. Biometrika, 33(3), 239–251.

Thank you again for taking the time to review our paper. We hope the additional analyses address your concerns and clarify MobiLens’s contribution to evaluating personalized mobility agents.

Reviewer_qFcPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MobiLens is a benchmark for personalised mobility agents built from the interaction logs of a commercial Chinese navigation platform. It contains 1,608 episodes (450 POI, 686 driving, 472 transit) from 1,458 users, each pairing a real query with de-identified user attributes, roughly 400 pre-query behavioural records from a three-month window, and request context; the evaluation reference is the user's observed post-query decision (POI: LLM-aligned from subsequent behaviour; driving: the trajectory taken; transit: the clicked plan). Agents interact with 19 navigation tools served from a frozen response cache. The framework scores personalisation (Hit@1/3 against the observed decision), diversity (label and behaviour-matching diversity), fairness (one-minus-max-gap of task-standardised Hit@3 across gender, age, vehicle ownership, city tier and consumption level), explainability (faithfulness / completeness / persuasiveness from a GLM-5.2 judge), and robustness (temporal stability and hard-query retention), plus tool-invocation success, delivery rate and latency. Eleven agent frameworks (ReAct variants, Plan-and-Execute, Letta, MetaGPT, CAMEL, InteRecAgent, iAgent, MACRec) are run on three backbones at temperature 0. The paper reports an "execution–alignment gap": tool success and delivery exceed 95% for most configurations while the best Hit@3 is 56.8, and alignment varies across user groups and task types.

Critical strengths
  • Behaviour-grounded references from real logs, evaluated under a shared, reproducible protocol. Using what users actually did afterwards rather than LLM-judged correctness is the right instinct and is rarer than it should be. The frozen replay cache, deterministic matching rules, and the two reference baselines (the production system's own responses in D.2 and each agent's last-round tool candidates in D.3) make the numbers interpretable in a way most agent benchmarks are not. Eleven frameworks across three backbones under one integration protocol is a substantial and useful comparison.

  • Breadth of evaluation, statistical care, and openness. Eleven agent frameworks across three backbones are compared under one integration protocol, and the appendix analysis (paired McNemar with BH-FDR correction, deterministic failure decomposition, user-level bootstrap, worst-group absolute levels) is satisfactory. The paper is clear and easy to follow, and code and artifacts are provided for review with a stated public release.

Critical weaknesses
  • The ceiling of Hit@K against observed behaviour is unknown, so the "execution–alignment gap" is a framing rather than a finding. The reference is a single observed choice already shaped by what the production system showed (transit GT is the clicked index among presented plans). The paper's own tables suggest the metric is near a label-imposed ceiling: three frontier backbones land within 2–3.5 points of each other (D.7); the production system matches or beats agents on POI (28.1 vs. median 25.3; D.2, absent from main text); no-memory ReAct reaches 48.1 against a best of 56.8, so ~85% of alignment needs no history, and on GPT-5.4 two memory variants are at or below no-memory; last-round tool candidates alone score 34–46 (D.3). The baselines that would settle this are missing — a habit baseline, a supervised next-POI model on the same histories, and the size of the "persona-required" set mentioned in E.5. Two further gaps in the same metric: 390 of 686 driving episodes come from a second window with no production response and score 25–35 points lower than the rest (D.2), pooled without comment; and memory gains come entirely from driving/transit while POI declines (Table 3, D.6), which Section 4.4.2 does not report.

  • Most human-centered dimensions do not discriminate, and the fairness statistics are inflated. Across 33 cells, gender fairness spans 94.6–99.6, region 89.6–99.1, consumption 84.8–96.6, explanation scores mostly 86–99, temporal robustness 74–95, behaviour-matching diversity 41–51. The explanation judge is a single model with no second judge or human calibration, and correlates with Hit@K at ρ = 0.04 / 0.20. Diversity is never motivated for commute-style requests. Pooled fairness tests (L432–437, D.9) treat the same episodes across 11 frameworks as independent and report q < 10⁻⁷; the 60+ POI claim rests on 23 episodes and the 55–59 bin has 11 classifiable POI episodes. A gap in behavioural predictability is not shown to be service inequity. No uncertainty is reported for Hit@K, so framework rankings (e.g. CAMEL's 0.1–0.6 point leads) are inside noise and, per D.5, do not replicate across backbones.

  • Main-text results mostly restate Table 1; the failure analysis (D.10) and reference comparisons (D.2/D.3) stay in the appendix. More broadly, the benchmark tests many dimensions but its takeaways are already well established: no single configuration dominates, memory augmentation helps some settings and not others, and stronger models make fewer tool-use errors. The paper does not yet surface what is new about mobility specifically.

  • [Minor] Positioning. MobilityBench and MapSatisfyBench are cited but what MobiLens shares with them — platform, query pool, tool suite, cache is unclear. The next-POI prediction literature the POI task belongs to (AgentMove, NAACL 2025; LLMMove; LLM-Mob; DeepMove/STAN) is not mentioned, nor are MapEval (ICML 2025) or Singh et al.'s TravelPlanner+ (EMNLP 2024 Industry).

Questions for authors

Clarification

  • What do MobiLens, MobilityBench and MapSatisfyBench share — platform, raw query pool, tool suite, response cache? Were any of the 11,929 raw queries or 19 tools used in those benchmarks?
  • How many of the 1,608 episodes are in the "strictly persona-required" set referenced in E.5? What Hit@3 does a habit baseline (most frequent route / POI from the user's history) achieve, and what does a trained next-POI model achieve on the 450 POI episodes?
  • Why do the 390 second-window driving episodes have no production response, how were they selected, and why do agents score 25–35 points lower on them? Please report driving results by source.
  • Will the ~400 pre-query behavioural records per episode be released? Appendix B says full behavioural histories are not released and that released episodes are restricted to "the behavioral window linked to each query" — if agents cannot be given the same history, the personalisation results are not reproducible.
  • How is "consumption level" derived — self-reported, inferred from in-app spending, or from POI price tiers — and how reliable is it? The same for vehicle ownership, given that non-car owners have driving-navigation episodes.
  • Does the main-text Hit@K "metadata matching" for POI (L239) involve anything beyond the five-level string waterfall in Appendix C? How often does each waterfall level fire, and was any level validated against human judgement?
  • Was the explanation judge compared against any second judge or human ratings? What is the rationale for letting refusals receive high explanation scores?
  • What is the mean number of delivered candidates per configuration? A configuration emitting one candidate has Hit@3 = Hit@1, which affects comparability across frameworks and sets LD to zero.
  • GPT-5.4 iAgent has the lowest TISR in the table (82.5) and the best Hit@1 of its block (38.9). How is this consistent with the execution–alignment framing?
  • In D.6, the retrieval-variant nets "(+19, +3, +7)" follow a Qwen / DeepSeek / GPT ordering, but the GPT cells listed as "(−6, +27, +3)" give retrieval +3. Which ordering is correct?

Suggestions:

  • Move a condensed failure analysis (D.10) and the D.2/D.3 reference comparisons into the main text, and report Table 12 as a four-bucket breakdown with denominators rather than the reasoning-residual share only.
  • Quantify which task types or scene categories favour which agent paradigms, so the paper's design guidance rests on more than the single Longde County example (L420–426).
  • Add the next-POI and map-agent literature and TravelPlanner+ to related work, and state explicitly what is new relative to MobilityBench and MapSatisfyBench.
  • Expand TISR at first use (L207) and define consumption level where it is introduced.
7 comments
AuthorsOpenReview ↗

Response to Reviewer qFcP [1/7]

We sincerely appreciate your thorough review and your recognition of our behavior-grounded references, shared evaluation protocol, and systematic comparison across frameworks and backbones. We take your concerns seriously and address them in detail below.

W1: The ceiling of Hit@K against observed behaviour is unknown, so the "execution–alignment gap" is a framing rather than a finding...

Response: Thanks for the comments. Users’ actual choices provide observable behavioral evidence for evaluating personalized recommendations. In the absence of explicit preference feedback, recommendation research commonly uses interactions such as clicks and purchases as indirect signals of user preferences (Hu et al., 2008). MobiLens follows a similar approach by using users’ subsequent mobility choices as evaluation references to measure the alignment between recommendations and observed behavior.

Regarding your concern that the metric may already be close to a label-imposed ceiling, we examined the complete tool traces of GPT-5.4 iAgent for each of the same 450 POI requests. For 212 requests, a candidate matching the user’s observed choice appeared in the tool results, while the final top-three recommendations matched the observed choice for 140 requests. Thus, for 16.0% of all requests, a matching candidate was already present in the tool results but did not appear in the final top three. For these cases, both the observed labels and the tool traces are fixed, and the gap occurs specifically between candidate retrieval and final recommendation. This identifies quantifiable room for improvement for this configuration using its existing tool results.

To address the missing baselines you identified, we have added a user-habit baseline, next-POI prediction, and statistics on the size of the "persona-required" subset. We will report the experimental results in our responses to the specific questions below.

We also address we clarify that Appendix D.2 contained an outdated description: the recoverable production candidate records had already been restored before the experiments in our response to Q3.

Regarding the decline on POI tasks, additional history or profile information may distract the model from the current request and route constraints when its use is not effectively guided, causing performance fluctuations or degradation. The lower POI scores relative to driving and transit suggest that POI selection is more challenging in this setting. Although simple memory strategies may fail to improve POI performance, iAgent achieves the highest POI Hit@3 across all three backbones. Its targeted retrieval and memory-based reranking provide a structured way to connect historical preferences with the current task, suggesting that effective use of personal information matters more than simply adding it.

Reference

Hu, Y., Koren, Y., & Volinsky, C. (2008). Collaborative filtering for implicit feedback datasets. In 2008 Eighth IEEE International Conference on Data Mining (pp. 263–272). IEEE.

AuthorsOpenReview ↗

Response to Reviewer qFcP [2/7]

W2: Most human-centered dimensions do not discriminate, and the fairness statistics are inflated...

Response: Thank you for your comments. We added a human evaluation of 100 outputs, with three annotators using the same 0–5 rubrics as GLM-5.2 for Faithfulness, Completeness, and Persuasiveness. Following the leave-one-out reference design, each round compared GLM-5.2 and one held-out annotator against the mean scores of the other two annotators, with results averaged across three rounds (Calderon et al., 2025). We used Kendall’s τ-b to account for tied scores when assessing ranking agreement(Kendall, 1945), and MAE to measure differences on the original scale.

The results show that GLM-5.2's scores closely match the human reference: the MAE differences are small. For Faithfulness and Completeness, not statistically distinguishable from zero under paired bootstrap resampling. For Persuasiveness, GLM-5.2 is slightly closer to the human reference. Its scores also show positive rank associations with human ratings across all three dimensions. These results support using GLM-5.2 to assist explanation-quality evaluation under shared scoring rubrics. We will include the human evaluation protocol and results in the revised paper.

For commuting, users’ habitual preferences influence route choices, while the weather, traffic conditions, and arrival deadlines on a particular day may also change those choices. For example, a user who usually commutes by bus may switch to the subway when rain causes congestion and arriving on time is important, because the subway is expected to be faster and its arrival time more predictable. When both options meet the current needs, presenting bus and subway options side by side, with estimated travel times and numbers of transfers, can help users weigh the trade-offs and see that their preferred transportation modes have been considered.

The pooled fairness tests treated repeated evaluations of the same requests across the 11 frameworks as independent observations. In the revision, we will remove these pooled tests and their results while retaining the task-standardized group comparisons in Hit@3. We will also remove explanations of age-group differences that rely on request-composition statistics from age groups with small sample sizes. We sincerely thank you for your careful reading and constructive suggestions.

Regarding uncertainty in framework comparisons, we conducted three repeated runs of all 11 configurations using Qwen3.7-Max, on fixed subsets of 100 episodes per task. The table reports mean ± standard deviation across runs, in percent.

FrameworkPOI Hit@1POI Hit@3Driving Hit@1Driving Hit@3Transit Hit@1Transit Hit@3Overall Hit@1Overall Hit@3
ReAct without memory16.7±1.523.7±2.535.7±2.348.0±0.043.0±2.067.7±3.131.8±0.846.4±1.6
Plan-and-Execute14.7±2.518.3±3.132.7±1.244.0±1.742.3±3.268.0±4.429.9±2.243.4±2.9
ReAct + last-k16.0±1.024.3±2.337.3±0.648.7±0.644.7±3.169.3±4.532.7±1.247.4±2.3
ReAct + summary15.7±0.622.3±2.345.0±0.056.3±0.648.3±4.279.3±4.936.3±1.252.7±2.3
ReAct + retrieval17.0±1.024.3±0.636.0±1.048.0±1.044.3±1.568.3±3.232.4±0.546.9±1.6
Letta16.7±1.523.3±3.243.3±2.556.0±1.747.7±3.178.7±3.135.9±2.152.7±2.6
MetaGPT17.0±3.524.7±2.938.3±1.551.3±1.250.0±3.670.3±3.235.1±2.248.8±2.2
CAMEL18.7±1.226.0±3.648.7±1.560.7±0.653.7±1.582.0±2.040.3±0.656.2±1.8
InteRecAgent18.0±1.723.3±2.342.3±4.255.7±3.849.3±4.574.7±4.636.6±3.351.2±3.6
iAgent19.7±1.526.0±3.641.7±1.552.0±1.746.0±2.670.0±3.635.8±1.849.3±2.8
MACRec18.3±2.122.7±1.548.0±1.758.7±0.653.3±1.585.0±1.739.9±0.555.4±0.8

The repeated-run averages preserve the task-specific patterns: iAgent achieves the highest POI Hit@1 and shares the highest POI Hit@3 with CAMEL, while CAMEL leads driving on both metrics and achieves the highest transit Hit@1. MACRec leads transit Hit@3. These results reinforce the observed POI strength of iAgent and route-navigation strength of CAMEL.

Reference

Calderon, N., Reichart, R., & Dror, R. (2025). The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 16051–16081). Association for Computational Linguistics.

Kendall, M. G. (1945). The treatment of ties in ranking problems. Biometrika, 33(3), 239–251.

AuthorsOpenReview ↗

Response to Reviewer qFcP [3/7]

W3: Main-text results mostly restate Table 1; the failure analysis (D.10) and reference comparisons (D.2/D.3) stay in the appendix...

Response: Thank you for this comment. To clarify where mobility agents miss users’ observed choices and when behavioral history helps, we will bring the key reference comparisons, failure analyses, and memory results from the appendix into the main text. We would also like to clarify the following specific insights these results provide for personalized mobility agents.

  1. The failure decomposition and the reference comparison jointly localize the gap. The D.10 classification shows that tool-side failures are more prevalent in POI tasks. This task-dependent pattern explains why POI selection remains the weakest scenario: the bottleneck lies not only in final ranking, but also in whether the right tools are invoked and whether candidate retrieval succeeds. Meanwhile, D.3 replaces the delivered answers with each configuration's final tool candidates, demonstrates that the agent's candidate selection consistently improves overall performance. The two analyses separate two sources of error: failures that originate before selection and gains that originate at selection, thereby clarifying where future agent designs should intervene.

  2. Successful tool execution does not ensure that recommendations match users’ actual choices. Under Qwen3.7-Max, most frameworks achieve tool invocation success rates above 94%, while the highest Hit@3 is 56.8%. This shows that execution reliability and alignment with user behavior are distinct capability dimensions.

  3. Framework advantages depend on the structure of the mobility task. iAgent achieves the highest POI Hit@3 across all three backbones, while CAMEL performs strongly on driving tasks and achieves the highest transit Hit@3 with Qwen3.7-Max. These patterns reflect how the frameworks support different task requirements.

    POI selection involves two steps: whether the target location is retrieved as a candidate, and whether it can then be ranked among the top three. iAgent's two-stage design of candidate retrieval followed by memory-based reranking aligns with this structure: its memory layer extracts frequently visited places from history to guide targeted retrieval, while the reranking stage demonstrates strong ordering-correction capability, promoting relevant candidates that the tool initially places lower in the results. For driving, the key shortfall lies not in execution but in the orchestration layer, specifically whether route planning is correctly initiated, and this is precisely the gap that CAMEL's dual-role, multi-round collaboration structure addresses. The planning request is repeatedly reaffirmed across dialogue rounds, thereby suppressing drift in tool orchestration and providing a mechanism for more consistently invoking the route-planning tools required by the task.

  4. How memory is organized affects its benefits. Among the ReAct memory strategies compared, summarized memory provides more consistent improvements over the no-memory baseline. This highlights the importance of how historical information is organized for use in personalized mobility decisions.

Reviewer_dBQsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MobiLens tests whether travel agents make recommendations that match users' real choices. It contains 1,608 episodes with user context, past travel records, and recorded decisions. Tests of 11 agent configurations across three language models show that successful tool use often does not lead to recommendations that match those decisions.

Critical strengths
  • The benchmark uses recorded user decisions, rather than asking another language model to judge whether recommendations match user behavior. A human audit checks 100 POI links. (Section 3.2; Appendix A.)
  • A fixed cache of 19 navigation tools makes comparisons repeatable without changing traffic or API results. (Section 3.3.)
  • The analysis covers recommendation quality, group differences, explanations, and efficiency. Results by task and user group reveal differences hidden by overall scores. (Sections 4.4–4.8; Appendix D.)
Critical weaknesses
  1. Matching a past choice is an incomplete measure of personalization. Section 3.2 uses later routes and clicks as references, and Appendix D.2 notes that transit choices come from options shown by the production system. These labels reflect the options users saw and can miss other suitable recommendations; a user study or multiple acceptable answers would strengthen the interpretation.
  2. The data cover a narrow setting. All episodes come from one Chinese navigation platform over 19 collection days. The paper acknowledges this limit, but the results do not establish performance across other regions, languages, or longer periods.
  3. Explanation scores need human validation. One model, GLM-5.2, judges all explanations. The 100-episode human audit validates POI links, not explanation scores, so it does not establish whether these scores reflect user understanding or preference.
  4. The robustness comparisons do not isolate causes. Temporal and query robustness compare different naturally occurring episodes. Task-type standardization helps, but remaining differences between groups can still affect scores; paired tests with changed wording or context would provide stronger evidence.
Questions for authors

see weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer dBQs [1/2]

We sincerely thank you for recognizing the value of our references based on recorded user decisions, the reproducibility of our evaluation environment, and our analysis across tasks and user groups. We appreciate your constructive suggestions and address your concerns below.

Q1: Matching a past choice is an incomplete measure of personalization. Section 3.2 uses later routes and clicks as references, ...

Response: Thank you for your comment. Recorded user choices are used as behavioral evidence in recommendation and mobility research. Hu et al. (2008) model preferences from implicit feedback, Barnes et al. (2024) evaluate routes against recorded trajectories, and Hydra learns transportation preferences from implicit feedback (Liu et al., 2019). MobiLens follows this practice by using observed mobility decisions as references: they provide concrete evidence of what users chose for a given request and context.

In practical route recommendation, a single request often admits several feasible options. For example, one transit route may be a few minutes faster but require a transfer, whereas another may take slightly longer but provide a direct connection. How a user weighs travel time against transfers is often not explicitly stated in the query. The recorded decision therefore provides concrete evidence for assessing whether an agent ranks the option the user actually selected among its top recommendations.

We appreciate the suggestion of a user study. A future study with the navigation platform could ask users which other feasible options they would accept for a sample of real trip requests. Their feedback could provide additional references for evaluation and inform improvements to recommendation ranking.

References

Hu, Y., Koren, Y., & Volinsky, C. (2008). Collaborative filtering for implicit feedback datasets. In 2008 Eighth IEEE International Conference on Data Mining (pp. 263–272). IEEE.

Barnes, M., Abueg, M., Lange, O. F., Deeds, M., Trader, J., Molitor, D., Wulfmeier, M., & O’Banion, S. (2024). Massively scalable inverse reinforcement learning in Google Maps. In The Twelfth International Conference on Learning Representations (ICLR).

Liu, H., Tong, Y., Zhang, P., Lu, X., Duan, J., & Xiong, H. (2019). Hydra: A personalized and context-aware multi-modal transportation recommendation system. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 2314–2324). Association for Computing Machinery.

Q2: The data cover a narrow setting. All episodes come from one Chinese navigation platform over 19 collection days...

Response: Thank you for this comment. MobiLens covers a range of mobility needs and user decision contexts: POI recommendation, driving, and public transit, and links real user requests with pre-query histories, request contexts, and subsequent mobility decisions. These episodes therefore capture varied request intents, travel conditions, and decision trade-offs, providing meaningful scenario diversity for evaluating personalized mobility agents.

Building on this data, our contribution is MobiLens, the first human-centered benchmark built from real-world user queries, pre-query user context, and observed mobility decisions. It provides an evaluation framework covering five human-centered outcome dimensions together with process-level measures, benchmarks 11 agent configurations across three LLM backbones under a shared offline navigation-tool replay protocol, and releases the benchmark episodes, tool-response cache, evaluation harness, and evaluation protocol.

Because releasing these artifacts is itself a key contribution, other institutions can use the framework to construct comparable evaluations once suitable local data become available, and our shared offline replay protocol makes such extensions practical. Our work thus offers a concrete methodology for extending personalized mobility evaluation to additional settings and for supporting future cross-region research.

We thank you for this suggestion. We will add this discussion to the revised paper and describe how other researchers can extend the framework to additional regions and languages.

AuthorsOpenReview ↗

Response to Reviewer dBQs [2/2]

Q3: Explanation scores need human validation. One model, GLM-5.2, judges all explanations. The 100-episode human audit...

Response: Thank you for your comment. We conducted an additional human evaluation of 100 explanation outputs. Three annotators independently rated all outputs on Faithfulness, Completeness, and Persuasiveness using the same 0–5 rubrics as GLM-5.2. Following a leave-one-annotator-out reference design (Calderon et al., 2025), we compared GLM-5.2 against the held-out annotator in each round, and measured agreement with Kendall's τ-b and mean absolute error (MAE).

The results show that GLM-5.2's scores closely match the human reference, with MAE of 0.31, 0.23, and 0.20 for FS, CS, and PS, respectively. Paired bootstrap resampling shows no significant MAE difference for FS and CS, and a slightly lower MAE for GLM-5.2 on PS; Kendall's τ-b indicates positive rank associations with human ratings across all three dimensions. These results support using GLM-5.2 to assist explanation-quality evaluation under shared scoring rubrics. We will include the human evaluation protocol and results in the revised paper.

References

Calderon, N., Reichart, R., & Dror, R. (2025). The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 16051–16081). Association for Computational Linguistics.

Kendall, M. G. (1945). The treatment of ties in ranking problems. Biometrika, 33(3), 239–251.

Q4: The robustness comparisons do not isolate causes. Temporal and query robustness compare different naturally occurring episodes...

Response: Thank you for this comment. Our original analyses assess overall performance across temporal conditions and levels of query clarity in naturally occurring episodes. To provide complementary evidence at the instance level, we added paired analyses using Qwen3.7-Max, examining wording changes within the same episode and temporal differences between similar trips.

For query robustness, we evaluated four ReAct configurations using Qwen3.7-Max on the same 100 episodes. We paraphrased each query while preserving its meaning and keeping the episode context and observed user choice fixed.

FrameworkOriginal Hit@3 (%)Paraphrased Hit@3 (%)Change (pp)95% CI (pp)
ReAct without memory49.048.0−1.0[−8.0, +5.0]
ReAct + last-k51.051.00.0[−5.0, +5.0]
ReAct + summary55.050.0−5.0[−10.0, −1.0]
ReAct + retrieval52.050.0−2.0[−8.0, +3.0]

Hit@3 changed by at most 2 percentage points for three configurations, with confidence intervals including zero. ReAct with summarized memory declined by 5 points, suggesting some sensitivity to query wording.

For temporal robustness, since offline records cannot provide the real road conditions at another moment for the same request and the corresponding user choices, we additionally conducted paired comparisons between semantically similar off-peak and peak requests.

FrameworkOff-peak Hit@3 (%)Peak-hour Hit@3 (%)Change (pp)
ReAct without memory45.536.4−9.1
ReAct + last-k45.534.1−11.4
ReAct + summary40.936.4−4.5
ReAct + retrieval43.234.1−9.1

All four configurations scored lower on peak-hour requests, with Hit@3 declines ranging from 4.5 to 11.4 percentage points. Since each pair consists of two real requests, the trips and users may still differ within a pair. We treat this result as a supplement to the original time-of-day group analysis.

We sincerely thank you once again for the time and effort you invested in reviewing our paper, as well as for your constructive feedback and suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.