- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces NetSpace-Bench, an application-oriented benchmark for mobile traffic generation and forecasting, with a particular focus on preserving spatial heterogeneity and assessing its value for downstream decision-making. The authors introduce the Spatial Dispersion Ratio (SDR) to assess whether relative differences across regions are preserved, and SAD* to interpret spatial allocation error relative to a uniform-allocation baseline. The PENTAD framework further evaluates five dimensions: total traffic volume, temporal distribution, spatial allocation, region–time interactions, and variability across windows. The downstream evaluation covers five tasks: capacity planning, energy saving, resource allocation, congestion warning, and synthetic data utility.
The benchmark evaluates eight generative models and four forecasting models on spatially held-out regions in three cities. Across the 24 generative-model evaluations, the authors report pronounced spatial homogenization in 18 cases and energy-saving performance that is essentially no better than taking no action in 19 cases. Validation includes cross-city comparisons, correlation analyses between metrics and decision utility, controlled perturbation experiments matched for RMSE, and controlled comparisons that hold the mean-field structure fixed while adjusting variability across windows.
Critical strengths
-
SDR and SAD* capture two complementary properties that aggregate error metrics do not explicitly distinguish: a model may preserve substantial differences across regions yet assign demand to the wrong locations. Using uniform allocation as a reference also makes spatial allocation errors measured in terms of geographic transport distance easier to interpret. Table 8 provides a concrete example: GeoKNN has an SDR of 0.415 for Baltimore but a SAD* of 1.939, indicating that it retains some regional variation while allocating traffic less accurately than uniform allocation. This distinction has practical value because an informative spatial evaluation must consider both the magnitude of regional differences and their correspondence to geographic locations.
-
The mean-field decomposition provides a clear, interpretable structure for the evaluation dimensions. Across all 15 city–perturbation combinations, the RMSE-matched experiments identify the dimension targeted by each perturbation, providing a useful test of the metrics’ diagnostic specificity. The matched controls in Appendix further show that identical mean structures can still yield different downstream costs in the tasks considered. These findings support evaluating variability separately from mean-field properties.
Critical weaknesses
1、The mathematical definitions of the core metrics remain incomplete, limiting independent assessment of the evaluation results. Section 3.2 describes DLD primarily at a conceptual level: it uses a one-dimensional Wasserstein distance to compare distributions of fluctuations across windows after normalization by a mean-based scale. However, Appendix A.6, cited as providing the full definition, does not give an explicit formula for DLD. It therefore remains unclear how the fluctuations are constructed, whether normalization uses the observed means or the respective means of the observed and generated data, and how the distances for individual region–time cells are aggregated. These choices affect whether the metric successfully removes the influence of scale error and how regions with different traffic levels are weighted in the final score.
Furthermore, CLE is defined only for strictly positive joint distributions, whereas the problem formulation allows nonnegative traffic values. The paper does not explain how the metric applies when the mean field contains zero entries. Because DLD and CLE are used directly in model comparisons, analyses of metric complementarity, and controlled experiments, these gaps in their definitions limit the ability to interpret the core results and assess their applicability.
2、The results in G4 lack a comparison of the various models with the always-alert policy, making it difficult to assess the incremental decision-making value of the model outputs. Furthermore, the sensitivity of the applied research results to the selected operating points is unclear. Appendix A.14 explicitly states that some generative models achieve regret values of approximately 0.090 in the congestion warning task because they almost always issue an alert. Under the current loss specification, always issuing an alert incurs approximately 9% of the cost of never issuing one. Low regret in this task therefore does not, by itself, demonstrate that a model can effectively identify periods of congestion, nor does it establish additional decision value over the simple strategy of always issuing an alert.
This issue directly affects the interpretation of the reported reversals in model rankings: better warning scores for models with larger errors may primarily reflect a cost setting that favors frequent alerts. The paper also acknowledges in Section 5 that each task is evaluated at only one representative operating point. For a benchmark whose central contribution concerns practical decision value, the current experiments do not sufficiently establish the range of conditions under which the observed model advantages and rankings hold. Direct comparisons with the corresponding simple strategies and targeted sensitivity analyses of key cost parameters would materially affect my confidence in these application findings.
Questions for authors
1、What is the exact mathematical definition of DLD? Please specify how fluctuations across windows are constructed, which means are used for normalization, and how the Wasserstein distances for individual region–time cells are aggregated. How are cells with zero or near-zero means handled? These details are necessary to understand whether DLD is independent of scale error and how much weight low-traffic regions receive in the evaluation.
2、Are the observed and generated joint distributions used to compute CLE in the experiments always strictly positive? If the mean field contains zero entries, how is CLE defined? If smoothing or another form of regularization is required, do these choices materially affect the assessment of interaction structure or the model rankings?
3、In G4, do the individual models offer a clear cost advantage over the “always alert” strategy? Do the main reported ranking reversals and the relationship between warning utility and conventional error metrics persist when the ratio of missed-alarm to false-alarm costs is varied within a reasonable range? These results would help determine how much of the current performance reflects useful warning information and how much is attributable to the specific cost setting.
12 comments
Incomplete mathematical definitions of core metrics(W1, Q1, and Q2)
Rigorous and complete mathematical definitions can eliminate the ambiguity of key indicators in our paper. The revised Appendix A.6 now states all formulas explicitly. Specifically, DLD compares the distributions of relative fluctuations across windows, and CLE compares the mean fields after positive regularization and marginal balancing
Here is the number of regions and the number of within-window positions. and are the respective window-averaged fields, is the empirical law of relative fluctuations at region and position , is the jointly active cell set, and is the regularized mean-field representative with uniform regional and time marginals. The following definitions specify all quantities and their use in Appendices A.6.1 and A.6.2. And we answer each related question your raised as follows.
- DLD
- How fluctuations across windows are constructed: For each region and relative time position , we collect observations across windows and define and . The empirical laws are and , where and are the observed and modeled window counts and denotes a point mass. The implemented ratios and yield the same Wasserstein distance because subtracting one from both distributions preserves that distance.
- Which means are used for normalization: Each field uses its own cross-window mean, and . This gives both fluctuation distributions zero mean and removes positive multiplicative level differences at each included cell.
- How the distances of individual region-time cells are aggregated: DLD is the arithmetic mean of the cellwise Wasserstein distances over . With equal window counts, as in our experiments, each distance is , with the two sequences sorted independently. Each included cell receives weight . We compute city scores separately and average the three city scores for the main model comparison.
- How cells with zero or near-zero means are handled: We define with . The threshold determines eligibility, and each included series is divided by its own mean. We report active-cell coverage and the fraction of cells with different observed and modeled active status. An empty active set gives an undefined DLD value. Table 9 reports complete coverage for 36 model-city comparisons and coverage of 99.9802%, 77.9705%, and 99.9433% for KG-Diff, MobiFM, and PatchTST in BM, respectively. Across thresholds from to , the maximum score change is . This stability supports interpreting the score through the relative fluctuations of the included cells, with coverage specifying the extent of each comparison.
- Whether DLD is independent of scale error, and how much weight low-traffic regions receive: For positive cellwise factors and that remain constant across windows, replacing by and by preserves their respective normalized fluctuations. DLD is therefore invariant to these multiplicative scale changes whenever the active set is preserved. All included cells have equal weight, including low-traffic cells. With complete coverage, every region has total weight . With partial coverage, a region's weight is proportional to its number of included time positions. Read alongside LBE and coverage, DLD helps distinguish discrepancies in traffic level, relative fluctuation shape, and the set of represented cells. This gives researchers specific properties to examine when improving a model.
- CLE
- Whether the observed and generated joint distributions in our experiments are strictly positive: All observed mean fields are strictly positive. The modeled mean fields contain seven zero entries for KG-Diff in BM, 3,886 for MobiFM in BM, and ten for PatchTST in BM. The reported CLE calculation applies the same positive regularization to every mean field, so the probability matrices supplied to marginal balancing are strictly positive throughout the evaluation.
- How CLE is defined when the mean field contains zero entries: Let , , and be the average entry of . For , define with . For , set . Then , where contains probability matrices with row sums and column sums . Sinkhorn balancing gives the unique solution. CLE is the total-variation distance between the two resulting representatives, as defined above. This construction retains every region and time position. For active fields, the uniform regularization mass is , independent of traffic units.
- Whether this choice materially affects the assessment of interaction structure or the model rankings: Table 8 evaluates the 36 model-city comparisons at . Every city's model ordering is unchanged, with Spearman correlation equal to 1 relative to the default throughout this six-order-of-magnitude range. The maximum absolute score change is through and over the complete range. All 72 observed and modeled representatives reach the stated numerical tolerance within 61 iterations. The unchanged within-city rankings support consistent model selection when regional and temporal interaction structure is the property of interest. Researchers can compare this structural fidelity using a common treatment of zero entries, with the preferred ordering stable across the tested regularization choices.
Thank you for the careful and constructive review. We agree that the paper should give detailed definitions explicitly and support its application findings with policy comparisons and sensitivity analyses, and doing so will substantially improve its readability and quality. We have revised the paper accordingly.