acceptodds
Under review as a conference paper at ICLR 2027

MobiFLow: Datasets and Benchmarks for Federated Learning in Real-world Mobility

Abstract

Recent advances in Federated Learning (FL) have established it as a promising paradigm for real-world domains where privacy, regulatory constraints, or data ownership concerns prevent centralized data collection, such as mobility. Despite growing interests in tasks including traffic or passenger flow prediction, the empirical evaluation of FL strategies in the mobility domain remains limited. Due to privacy concerns, existing studies often rely on proprietary datasets, hindering reproducibility, while publicly available mobility datasets are rarely designed for federated settings. To address these limitations, we introduce MobiFLow, an open-source federated dataset suite for mobility applications built upon natural partitions, i.e., splits derived from intrinsic dataset properties. The suite comprises seven datasets spanning diverse mobility tasks, modalities, and scales, covering both cross-device and cross-silo FL scenarios. MobiFLow is complemented by standardized preprocessing, baseline models, benchmarking utilities, and practical evaluation guidelines to support reproducible research. Its modular design allows researchers to seamlessly integrate new datasets and federated learning strategies within a unified framework. We provide baseline results using widely adopted FL strategies and show that realistic mobility heterogeneity makes collaboration far from uniformly beneficial: local training is often surprisingly strong, pooled training can be competitive when client distributions are relatively homogeneous, and no general-purpose FL method consistently dominates across datasets, underscoring the need for realistic benchmarks and mobility-aware FL methods. By releasing datasets, code, and benchmarking tools, MobiFLow aims to promote reproducible research and establish a common ground for the systematic evaluation and fair comparison of FL methods in mobility.

Primary area:
datasets and benchmarks
Keywords:
federated learning, benchmark, datasets, mobility

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WvYaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MobiFLow, an open-source benchmark suite for federated learning in real-world mobility applications. To address the lack of standardized and reproducible evaluation protocols in this domain, MobiFLow integrates seven mobility datasets covering diverse tasks, modalities, and scales, with natural client partitions designed to reflect realistic data heterogeneity in both cross-device and cross-silo FL settings. The benchmark provides standardized preprocessing pipelines, baseline models, evaluation protocols, and implementations of seven representative FL strategies.

Critical strengths

S1. This paper addresses an important practical challenge in federated learning for mobility applications, where the lack of standardized and publicly available benchmarks limits reproducibility and fair comparisons. Establishing a realistic evaluation framework for this domain is valuable and relevant to the FL research community.

S2. The proposed MobiFLow benchmark integrates seven real-world mobility datasets covering diverse tasks, data modalities, and both cross-device and cross-silo FL settings. Its natural client partitions, standardized preprocessing pipelines, and unified evaluation protocols provide a useful foundation for reproducible and systematic research.

S3. The authors conduct experiments comparing multiple representative FL strategies against local and centralized training baselines, along with additional sensitivity and resource analyses.

Critical weaknesses

W1. The related work section overlooks established federated learning frameworks such as FATE and FederatedScope.

W2. Several inconsistencies appear across the main paper and appendix. 1) FedCrawdad is evaluated using Macro-F1 in Table 1 but Haversine distance in Appendix A.3, while the reported client counts for FedMelbourne and FedSUNT differ between Tables 1 and 2. 2) The reported GPU-hour statistics across different tables appear difficult to reconcile.

W3. The explanations for the performance differences among local, pooled, and federated training remain largely speculative. More targeted analyses and empirical evidence are needed to substantiate the authors' interpretations.

W4. Presentation could be improved. The text in several figures and tables is too small, making them difficult to read and interpret. Moreover, many important content and experimental results are relegated to the appendix, making it difficult for readers to follow and fully understand the main findings.

W5. The experiments are conducted on a single computing node, without explicitly modeling network latency, bandwidth variability, device availability, or privacy-preserving mechanisms.

W6. A central claim of the paper is that natural client partitions better reflect realistic data ownership and deployment conditions. However, some partitions are constructed from geographical or operational grouping rather than actual independent data owners.

Questions for authors

Q1. Why did the authors choose to implement FL strategies from scratch rather than build upon established frameworks such as FATE or FederatedScope? How does MobiFLow differ from these frameworks in terms of data preprocessing, communication mechanisms, and other key functionalities? Furthermore, can MobiFLow be readily integrated with existing FL frameworks, and what additional effort would such integration require?

Q2. Many well-established datasets are widely used for the mobility tasks considered in this paper. What criteria guided the selection of the seven datasets, and why were more commonly used task-specific benchmarks not adapted to federated settings such as PeMS, Geolife, T-Drive, TaxiNYC, etc.?

Reviewer_BT2FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript introduces an open-source FL dataset suite (MobiFLow). It targets mobility applications (e.g., Urban road traffic, Multimodal personal mobility, Taxi mobility traces etc.). It is based on 6 existing datasets + 1 newly introduced). It also includes standardized preprocessing and baseline models.

Critical strengths

Benchmarking FL and mobility models on real-life dataset is a challenging task. Mostly because mobility datasets are privacy-sensitive, hence many studies do not share data. Having one such benchmarking suit would be beneficial for the community.

Critical weaknesses
  1. No naive baselines or state-of-the-art methods. The forecasting tasks lack naive baselines (e.g., simple historical-average and seasonal-naive baselines), so it is unclear how much any model actually learns. On the other side, all seven FL strategies date from 2017–2021, with predictive models mostly LSTM/RNN variants. The AI and ICLR community have developed much more advanced methods (including FL for distribution shifts and drifts). The main finding is that local training wins under distribution shifts and drifts, yet no personalized, clustered or shift-aware method is evaluated
  2. Statistical tests are not clear. Paired t-tests assume independent, approximately normal differences. Please state the unit of pairing, the sample size, the number of seeds, and whether the assumptions were checked. With three expanding-window folds, n = 3 and the folds share training data. The split protocol also varies across datasets without explanation.
  3. Unsupported claims and writing inconsistency.
  • 3.1 - Section 4.4 states that pooled training beats local training on cross-silo FedSUNT because the data are near-IID. However, Figure 2 ranks Pooled 8th and Local 5th there.
  • 3.2 - FedCrawdad: Table 1 lists classification with Accuracy/F1 and an RNN. Appendix A.3 describes coordinate regression evaluated with EDS and an LSTM with self-attention.
  • 3.3 - FedCollecty: The main text says GPS traces and an LSTM. The appendix uses only inertial sensors and a CNN+LSTM.
  1. FL platform integration. Many researchers utilize existing FL platforms. Please show how MobiFLow plugs into established frameworks.
Questions for authors

See Weaknesses

Reviewer_rmxrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors argue that FL strategies are not properly evaluated in the mobility domain and introduce MobiFlow, a python library that includes a set of seven datasets together with baseline models and evaluation metrics as well as existing FL strategies with benchmarks.

Critical strengths

The project can be useful to researchers on the domain.

Critical weaknesses
  1. The main weakness of this work is its novelty. This is a nice engineering project that can be useful to the community but I do not see anything novel on the submission. Notably, the authors add seven datasets on the library but only one of them is new (FedSmartMaceio) and includes data from only 12 clients (the smallest number of clients in the library). Similarly, the authors included multiple algorithms but none of them is their contribution.

  2. The selected parameters for the experimental results are not properly explained/justified.

Secondary issues:

  1. The font size on Table 1 is very small and unreadable on a printed paper.
Questions for authors

I took a careful look at the paper but can you please clarify the novelty of this work?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.