- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces MobiLens, a human-centered benchmark for evaluating personalized mobility agents using real-world navigation data. It evaluates 11 agent configurations across personalization, diversity, fairness, explainability, robustness, tool-use success, and efficiency under an offline navigation environment. Experiments across three LLM backbones reveal an execution–alignment gap: agents can reliably use tools and produce valid answers, yet still align poorly with users’ actual choices.
Critical strengths
- The paper provides a carefully engineered, behavior-grounded benchmark with real-world mobility data, and a reproducible offline tool environment.
- The benchmark evaluates diversity, fairness, explainability, robustness, tool-use success, and efficiency, having rich metrics. This broader evaluation reveals important trade-offs between different base models and methods.
Critical weaknesses
- The paper mainly integrates existing agent frameworks, memory mechanisms, and evaluation dimensions into a unified mobility benchmark, without proposing a new agent architecture, learning algorithm, or personalization mechanism, may fall to limited conceptual and methodological novelty.
- For a benchmark paper whose main contribution is evaluation, the empirical section itself should be a central strength. However, the main text largely revolves around one aggregate table, with only one model, limited visualization or deeper diagnostic analysis.
- The dataset contains 1,608 episodes collected from a single navigation platform and consists of Chinese-language interactions. This relatively narrow setting makes it unclear whether the conclusions generalize to other regions, platforms, languages, or mobility systems.
- The user’s eventual route, transit plan, or POI choice is treated as the behavioral reference for Hit@K, but that observed choice is not necessarily the only suitable or preferred recommendation.
Questions for authors
- How sensitive are the personalization conclusions to the assumption that the single observed post-query decision is the appropriate reference?
- Since completeness and persuasiveness are central to the claimed human-centered evaluation, how robust are the LLM-judged explainability results to the choice of evaluator?
6 comments
Response to Reviewer a8xd [2/3]
W3: The dataset contains 1,608 episodes collected from a single navigation platform and consists of Chinese-language interactions...
Response: Thank you for your comment. MobiLens covers a range of mobility needs and user decision contexts: POI recommendation, driving, and public transit, and links real user requests with pre-query histories, request contexts, and subsequent mobility decisions. These episodes therefore capture varied request intents, travel conditions, and decision trade-offs, providing meaningful scenario diversity for evaluating personalized mobility agents.
Building on this data, our contribution is MobiLens, the first human-centered benchmark built from real-world user queries, pre-query user context, and observed mobility decisions. It provides an evaluation framework covering five human-centered outcome dimensions together with process-level measures, benchmarks 11 agent configurations across three LLM backbones under a shared offline navigation-tool replay protocol, and releases the benchmark episodes, tool-response cache, evaluation harness, and evaluation protocol.
Because releasing these artifacts is itself a key contribution, other institutions can use the framework to construct comparable evaluations once suitable local data become available, and our shared offline replay protocol makes such extensions practical. Our work thus offers a concrete methodology for extending personalized mobility evaluation to additional settings and for supporting future cross-region research.
W4: The user’s eventual route, transit plan, or POI choice is treated as the behavioral reference for Hit@K, but that observed choice is not necessarily the only suitable or preferred recommendation.
Response: Thank you for this comment. A mobility request can have several suitable recommendations. MobiLens uses the user’s observed choice as a behavioral reference. Hit@K measures whether that choice appears among the agent’s top recommendations.
Using observed choices as a behavioral reference is consistent with prior work on implicit feedback, where clicks and purchases serve as signals of user preferences (Hu et al., 2008). In mobility applications, Barnes et al. (2024) evaluate predicted routes against recorded driving trajectories in Google Maps, and Baidu Maps’ Hydra uses clicks on displayed travel options as implicit feedback (Liu et al., 2019).
We will clarify in the revised metric definition that a Hit@K miss means the observed choice was absent from the top . Other recommendations may still be suitable for the request.
References
Hu, Y., Koren, Y., & Volinsky, C. (2008). Collaborative filtering for implicit feedback datasets. In 2008 Eighth IEEE International Conference on Data Mining (pp. 263–272). IEEE.
Barnes, M., Abueg, M., Lange, O. F., Deeds, M., Trader, J., Molitor, D., Wulfmeier, M., & O’Banion, S. (2024). Massively scalable inverse reinforcement learning in Google Maps. In The Twelfth International Conference on Learning Representations (ICLR).
Liu, H., Tong, Y., Zhang, P., Lu, X., Duan, J., & Xiong, H. (2019). Hydra: A personalized and context-aware multi-modal transportation recommendation system. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 2314–2324).
Response to Reviewer a8xd [3/3]
Q1: How sensitive are the personalization conclusions to the assumption that the single observed post-query decision is the appropriate reference?
Response: Thank you for raising this question. As discussed in W4, the observed post-query choice serves as our behavioral reference. To test how sensitive the estimated benefit of summary memory is to this reference, we rescored the same Qwen3.7-Max ReAct outputs on 150 randomly sampled requests, with 50 each for POI search, driving, and public transit. The expanded rule retained the observed choice and also credited alternatives that experts judged acceptable based on the request, context, and tool evidence.
| Scoring rule | No-memory ReAct Hit@3 | Summary-memory ReAct Hit@3 | Difference |
|---|---|---|---|
| Observed choice only | 52.0% | 60.7% | +8.7 pp |
| Observed choice or an acceptable alternative | 78.0% | 84.0% | +6.0 pp |
On this sample, the estimated Hit@3 advantage of summary memory remained positive under both rules, decreasing from 8.7 to 6.0 percentage points when acceptable alternatives were credited (change of −2.7 pp, paired bootstrap 95% CI [−6.5, +0.9] pp).
Q2: Since completeness and persuasiveness are central to the claimed human-centered evaluation, how robust are the LLM-judged explainability results to the choice of evaluator?
Response: Thank you for this comment. We conducted an additional human evaluation of 100 explanation outputs. Three annotators independently rated all outputs on Faithfulness (FS), Completeness (CS), and Persuasiveness (PS) using the same 0–5 rubrics as GLM-5.2. Following a leave-one-annotator-out reference design (Calderon et al., 2025), we compared GLM-5.2 against the held-out annotator in each round, and measured agreement with Kendall's τ-b and mean absolute error (MAE).
The results show that GLM-5.2's scores closely match the human reference, with MAE of 0.31, 0.23, and 0.20 for FS, CS, and PS, respectively. Paired bootstrap resampling shows no significant MAE difference for FS and CS, and a slightly lower MAE for GLM-5.2 on PS; Kendall's τ-b indicates positive rank associations with human ratings across all three dimensions. These results support using GLM-5.2 to assist explanation-quality evaluation under shared scoring rubrics, and indicate that the LLM-judged explainability results are robust to the choice of evaluator. We will include the human evaluation protocol and results in the revised paper.
References
Calderon, N., Reichart, R., & Dror, R. (2025). The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 16051–16081). Association for Computational Linguistics.
Kendall, M. G. (1945). The treatment of ties in ranking problems. Biometrika, 33(3), 239–251.
Thank you again for taking the time to review our paper. We hope the additional analyses address your concerns and clarify MobiLens’s contribution to evaluating personalized mobility agents.
Response to Reviewer a8xd [1/3]
We sincerely thank you for recognizing the careful construction of our benchmark, its grounding in real-world mobility behavior, and its reproducible tool environment, as well as the breadth of our evaluation and the trade-offs it reveals. We address your concerns below.
Response: Thank you for this comment. MobiLens contributes a multidimensional evaluation protocol for personalized mobility agents. We link real-world requests and trip context to users’ pre-query profiles, behavioral histories, and observed post-query decisions. Across POI search, driving, and public transit, we use those decisions as references for recommendation ranking and evaluate behavioral alignment alongside task execution, diversity, fairness, explainability, and robustness.
To clarify the differences between MobiLens and existing benchmarks, the following comparison covers scoring references, agent configurations, and how each benchmark evaluates performance.
A dash indicates that the published protocol does not report a dedicated evaluation for that dimension.
We constructed 1,608 evaluation episodes from real navigation logs and reserved post-query behavior for scoring. We evaluated 11 agent configurations in a shared offline replay environment covering 19 navigation tools. We standardized navigation interfaces and task instructions while preserving each framework’s planning and memory mechanisms. Within each backbone, we paired the last-k, summary, and retrieval ReAct variants with the same no-memory baseline on identical episodes.
The evaluation across three LLM backbones yielded three main findings.
Execution reliability and behavioral alignment can diverge. With Qwen3.7-Max, most configurations achieved tool invocation success rates above 94%, while the highest Hit@3 was 56.8%.
Framework performance varies by mobility task. With Qwen3.7-Max, iAgent achieved the highest Hit@3 in POI search, while CAMEL led in driving and public transit. In our implementations, iAgent uses memory to rerank candidates, and CAMEL uses multi-round coordination between two agents.
Memory organization and task relevance matter. Summarization provides more consistent gains over no-memory ReAct; history benefits driving and transit more consistently than POI selection.
Response: Thank you for this comment. Table 1 summarizes the results for Qwen3.7-Max, and Tables 2 and 3 in Appendix D.1 report the full evaluation of 11 agent configurations across three LLM backbones. We will add a concise comparison of task-level Hit@3 across all three backbones to the main text.
We also analyzed GPT-5.4 iAgent’s tool traces on 450 POI requests. The tool results contained a candidate matching the user’s observed choice in 212 requests, and the final top three matched that choice in 140. Thus, in at least 72 requests (16.0%), a matching candidate appeared in the tool results, yet none of the final top three matched the observed choice. We will include this analysis in the main text to show where matches were lost between tool retrieval and final recommendation.