- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MobiFLow, an open-source benchmark suite for federated learning in real-world mobility applications. To address the lack of standardized and reproducible evaluation protocols in this domain, MobiFLow integrates seven mobility datasets covering diverse tasks, modalities, and scales, with natural client partitions designed to reflect realistic data heterogeneity in both cross-device and cross-silo FL settings. The benchmark provides standardized preprocessing pipelines, baseline models, evaluation protocols, and implementations of seven representative FL strategies.
Critical strengths
S1. This paper addresses an important practical challenge in federated learning for mobility applications, where the lack of standardized and publicly available benchmarks limits reproducibility and fair comparisons. Establishing a realistic evaluation framework for this domain is valuable and relevant to the FL research community.
S2. The proposed MobiFLow benchmark integrates seven real-world mobility datasets covering diverse tasks, data modalities, and both cross-device and cross-silo FL settings. Its natural client partitions, standardized preprocessing pipelines, and unified evaluation protocols provide a useful foundation for reproducible and systematic research.
S3. The authors conduct experiments comparing multiple representative FL strategies against local and centralized training baselines, along with additional sensitivity and resource analyses.
Critical weaknesses
W1. The related work section overlooks established federated learning frameworks such as FATE and FederatedScope.
W2. Several inconsistencies appear across the main paper and appendix. 1) FedCrawdad is evaluated using Macro-F1 in Table 1 but Haversine distance in Appendix A.3, while the reported client counts for FedMelbourne and FedSUNT differ between Tables 1 and 2. 2) The reported GPU-hour statistics across different tables appear difficult to reconcile.
W3. The explanations for the performance differences among local, pooled, and federated training remain largely speculative. More targeted analyses and empirical evidence are needed to substantiate the authors' interpretations.
W4. Presentation could be improved. The text in several figures and tables is too small, making them difficult to read and interpret. Moreover, many important content and experimental results are relegated to the appendix, making it difficult for readers to follow and fully understand the main findings.
W5. The experiments are conducted on a single computing node, without explicitly modeling network latency, bandwidth variability, device availability, or privacy-preserving mechanisms.
W6. A central claim of the paper is that natural client partitions better reflect realistic data ownership and deployment conditions. However, some partitions are constructed from geographical or operational grouping rather than actual independent data owners.
Questions for authors
Q1. Why did the authors choose to implement FL strategies from scratch rather than build upon established frameworks such as FATE or FederatedScope? How does MobiFLow differ from these frameworks in terms of data preprocessing, communication mechanisms, and other key functionalities? Furthermore, can MobiFLow be readily integrated with existing FL frameworks, and what additional effort would such integration require?
Q2. Many well-established datasets are widely used for the mobility tasks considered in this paper. What criteria guided the selection of the seven datasets, and why were more commonly used task-specific benchmarks not adapted to federated settings such as PeMS, Geolife, T-Drive, TaxiNYC, etc.?