acceptodds
Under review as a conference paper at ICLR 2027

Spatial Heterogeneity That Matters: An Application-Oriented Benchmark for Mobile Traffic Modeling

Abstract

Mobile traffic modeling is important for understanding spatial urban functions and corresponding human activity patterns, particularly the spatial heterogeneity. But it can be ignored by evaluations focusing on citywide error-based metrics aggregated across spatial and temporal dimensions, and capturing aggregate scale alone does not establish usefulness for region-level network management and planning within cities. To support these practical applications, we introduce NETSPACE-BENCH, an application-oriented benchmark that identifies modeling weaknesses of existing methods and guides the further direction by jointly evaluating modeling quality, especially the spatial context-related heterogeneity and decision utility. We achieve these through spatial reference and dispersion checks, five complementary evaluation perspectives, and five operational decision tasks. Specifically, our evaluation across 12 models and three cities reveals that 18 of 24 eligible main generation runs retain less than 5% of observed region-mean dispersion and 19 yield energy policies approximately no better than inaction. These findings motivate explicit evaluation of spatial heterogeneity for practical applications, and the operational value of controllability through spatial context remains an open question.

Primary area:
datasets and benchmarks
Keywords:
spatial intelligence, computer network

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YT9zOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces NetSpace-Bench, an application-oriented benchmark for mobile traffic generation and forecasting, with a particular focus on preserving spatial heterogeneity and assessing its value for downstream decision-making. The authors introduce the Spatial Dispersion Ratio (SDR) to assess whether relative differences across regions are preserved, and SAD* to interpret spatial allocation error relative to a uniform-allocation baseline. The PENTAD framework further evaluates five dimensions: total traffic volume, temporal distribution, spatial allocation, region–time interactions, and variability across windows. The downstream evaluation covers five tasks: capacity planning, energy saving, resource allocation, congestion warning, and synthetic data utility.

The benchmark evaluates eight generative models and four forecasting models on spatially held-out regions in three cities. Across the 24 generative-model evaluations, the authors report pronounced spatial homogenization in 18 cases and energy-saving performance that is essentially no better than taking no action in 19 cases. Validation includes cross-city comparisons, correlation analyses between metrics and decision utility, controlled perturbation experiments matched for RMSE, and controlled comparisons that hold the mean-field structure fixed while adjusting variability across windows.

Critical strengths
  1. SDR and SAD* capture two complementary properties that aggregate error metrics do not explicitly distinguish: a model may preserve substantial differences across regions yet assign demand to the wrong locations. Using uniform allocation as a reference also makes spatial allocation errors measured in terms of geographic transport distance easier to interpret. Table 8 provides a concrete example: GeoKNN has an SDR of 0.415 for Baltimore but a SAD* of 1.939, indicating that it retains some regional variation while allocating traffic less accurately than uniform allocation. This distinction has practical value because an informative spatial evaluation must consider both the magnitude of regional differences and their correspondence to geographic locations.

  2. The mean-field decomposition provides a clear, interpretable structure for the evaluation dimensions. Across all 15 city–perturbation combinations, the RMSE-matched experiments identify the dimension targeted by each perturbation, providing a useful test of the metrics’ diagnostic specificity. The matched controls in Appendix further show that identical mean structures can still yield different downstream costs in the tasks considered. These findings support evaluating variability separately from mean-field properties.

Critical weaknesses

1、The mathematical definitions of the core metrics remain incomplete, limiting independent assessment of the evaluation results. Section 3.2 describes DLD primarily at a conceptual level: it uses a one-dimensional Wasserstein distance to compare distributions of fluctuations across windows after normalization by a mean-based scale. However, Appendix A.6, cited as providing the full definition, does not give an explicit formula for DLD. It therefore remains unclear how the fluctuations are constructed, whether normalization uses the observed means or the respective means of the observed and generated data, and how the distances for individual region–time cells are aggregated. These choices affect whether the metric successfully removes the influence of scale error and how regions with different traffic levels are weighted in the final score.

Furthermore, CLE is defined only for strictly positive joint distributions, whereas the problem formulation allows nonnegative traffic values. The paper does not explain how the metric applies when the mean field contains zero entries. Because DLD and CLE are used directly in model comparisons, analyses of metric complementarity, and controlled experiments, these gaps in their definitions limit the ability to interpret the core results and assess their applicability.

2、The results in G4 lack a comparison of the various models with the always-alert policy, making it difficult to assess the incremental decision-making value of the model outputs. Furthermore, the sensitivity of the applied research results to the selected operating points is unclear. Appendix A.14 explicitly states that some generative models achieve regret values of approximately 0.090 in the congestion warning task because they almost always issue an alert. Under the current loss specification, always issuing an alert incurs approximately 9% of the cost of never issuing one. Low regret in this task therefore does not, by itself, demonstrate that a model can effectively identify periods of congestion, nor does it establish additional decision value over the simple strategy of always issuing an alert.

This issue directly affects the interpretation of the reported reversals in model rankings: better warning scores for models with larger errors may primarily reflect a cost setting that favors frequent alerts. The paper also acknowledges in Section 5 that each task is evaluated at only one representative operating point. For a benchmark whose central contribution concerns practical decision value, the current experiments do not sufficiently establish the range of conditions under which the observed model advantages and rankings hold. Direct comparisons with the corresponding simple strategies and targeted sensitivity analyses of key cost parameters would materially affect my confidence in these application findings.

Questions for authors

1、What is the exact mathematical definition of DLD? Please specify how fluctuations across windows are constructed, which means are used for normalization, and how the Wasserstein distances for individual region–time cells are aggregated. How are cells with zero or near-zero means handled? These details are necessary to understand whether DLD is independent of scale error and how much weight low-traffic regions receive in the evaluation.

2、Are the observed and generated joint distributions used to compute CLE in the experiments always strictly positive? If the mean field contains zero entries, how is CLE defined? If smoothing or another form of regularization is required, do these choices materially affect the assessment of interaction structure or the model rankings?

3、In G4, do the individual models offer a clear cost advantage over the “always alert” strategy? Do the main reported ranking reversals and the relationship between warning utility and conventional error metrics persist when the ratio of missed-alarm to false-alarm costs is varied within a reasonable range? These results would help determine how much of the current performance reflects useful warning information and how much is attributable to the specific cost setting.

12 comments

Thank you for the careful and constructive review. We agree that the paper should give detailed definitions explicitly and support its application findings with policy comparisons and sensitivity analyses, and doing so will substantially improve its readability and quality. We have revised the paper accordingly.

AuthorsOpenReview ↗

Incomplete mathematical definitions of core metrics(W1, Q1, and Q2)

Rigorous and complete mathematical definitions can eliminate the ambiguity of key indicators in our paper. The revised Appendix A.6 now states all formulas explicitly. Specifically, DLD compares the distributions of relative fluctuations across windows, and CLE compares the mean fields after positive regularization and marginal balancing

DLD⁡(X^,X)=1∣Aε∣∑(r,t)∈AεW1(ν^rt,νrt),\operatorname{DLD}(\widehat X,X)=\frac{1}{|\mathcal A_\varepsilon|}\sum_{(r,t)\in\mathcal A_\varepsilon}W_1(\widehat\nu_{rt},\nu_{rt}),

CLE⁡δ(X^,X)=12∑r,t∣[Bδ(M^)]rt−[Bδ(M)]rt∣.\operatorname{CLE}_\delta(\widehat X,X)=\frac12\sum_{r,t}\left|[B_\delta(\widehat M)]_{rt}-[B_\delta(M)]_{rt}\right|.

Here RR is the number of regions and HH the number of within-window positions. MM and M^\widehat M are the respective window-averaged fields, νrt\nu_{rt} is the empirical law of relative fluctuations at region rr and position tt, Aε\mathcal A_\varepsilon is the jointly active cell set, and Bδ(M)B_\delta(M) is the regularized mean-field representative with uniform regional and time marginals. The following definitions specify all quantities and their use in Appendices A.6.1 and A.6.2. And we answer each related question your raised as follows.

  • DLD
    • How fluctuations across windows are constructed: For each region rr and relative time position tt, we collect observations across windows and define Uwrt=(Xwrt−μrt)/μrtU_{wrt}=(X_{wrt}-\mu_{rt})/\mu_{rt} and U^wrt=(X^wrt−μ^rt)/μ^rt\widehat U_{wrt}=(\widehat X_{wrt}-\widehat\mu_{rt})/\widehat\mu_{rt}. The empirical laws are νrt=W−1∑wδUwrt\nu_{rt}=W^{-1}\sum_w\delta_{U_{wrt}} and ν^rt=W^−1∑wδU^wrt\widehat\nu_{rt}=\widehat W^{-1}\sum_w\delta_{\widehat U_{wrt}}, where WW and W^\widehat W are the observed and modeled window counts and δu\delta_u denotes a point mass. The implemented ratios X/μX/\mu and X^/μ^\widehat X/\widehat\mu yield the same Wasserstein distance because subtracting one from both distributions preserves that distance.
    • Which means are used for normalization: Each field uses its own cross-window mean, μrt=W−1∑wXwrt\mu_{rt}=W^{-1}\sum_wX_{wrt} and μ^rt=W^−1∑wX^wrt\widehat\mu_{rt}=\widehat W^{-1}\sum_w\widehat X_{wrt}. This gives both fluctuation distributions zero mean and removes positive multiplicative level differences at each included cell.
    • How the distances of individual region-time cells are aggregated: DLD is the arithmetic mean of the cellwise Wasserstein distances over Aε\mathcal A_\varepsilon. With equal window counts, as in our experiments, each distance is drt=W−1∑i=1W∣U(i)rt−U^(i)rt∣d_{rt}=W^{-1}\sum_{i=1}^W|U_{(i)rt}-\widehat U_{(i)rt}|, with the two sequences sorted independently. Each included cell receives weight 1/∣Aε∣1/|\mathcal A_\varepsilon|. We compute city scores separately and average the three city scores for the main model comparison.
    • How cells with zero or near-zero means are handled: We define Aε={(r,t)∣μrt>ε, μ^rt>ε}\mathcal A_\varepsilon=\{(r,t)\mid\mu_{rt}>\varepsilon,\ \widehat\mu_{rt}>\varepsilon\} with ε=10−12\varepsilon=10^{-12}. The threshold determines eligibility, and each included series is divided by its own mean. We report active-cell coverage γ=∣Aε∣/(RH)\gamma=|\mathcal A_\varepsilon|/(RH) and the fraction of cells with different observed and modeled active status. An empty active set gives an undefined DLD value. Table 9 reports complete coverage for 36 model-city comparisons and coverage of 99.9802%, 77.9705%, and 99.9433% for KG-Diff, MobiFM, and PatchTST in BM, respectively. Across thresholds from 10−1510^{-15} to 10−910^{-9}, the maximum score change is 2.78×10−172.78\times10^{-17}. This stability supports interpreting the score through the relative fluctuations of the included cells, with coverage specifying the extent of each comparison.
    • Whether DLD is independent of scale error, and how much weight low-traffic regions receive: For positive cellwise factors arta_{rt} and brtb_{rt} that remain constant across windows, replacing XwrtX_{wrt} by artXwrta_{rt}X_{wrt} and X^wrt\widehat X_{wrt} by brtX^wrtb_{rt}\widehat X_{wrt} preserves their respective normalized fluctuations. DLD is therefore invariant to these multiplicative scale changes whenever the active set is preserved. All included cells have equal weight, including low-traffic cells. With complete coverage, every region has total weight 1/R1/R. With partial coverage, a region's weight is proportional to its number of included time positions. Read alongside LBE and coverage, DLD helps distinguish discrepancies in traffic level, relative fluctuation shape, and the set of represented cells. This gives researchers specific properties to examine when improving a model.
  • CLE
    • Whether the observed and generated joint distributions in our experiments are strictly positive: All observed mean fields are strictly positive. The modeled mean fields contain seven zero entries for KG-Diff in BM, 3,886 for MobiFM in BM, and ten for PatchTST in BM. The reported CLE calculation applies the same positive regularization to every mean field, so the probability matrices supplied to marginal balancing are strictly positive throughout the evaluation.
    • How CLE is defined when the mean field contains zero entries: Let Mrt=W−1∑wXwrtM_{rt}=W^{-1}\sum_wX_{wrt}, Jrt=1/(RH)J_{rt}=1/(RH), and mˉ(M)\bar m(M) be the average entry of MM. For mˉ(M)>10−12\bar m(M)>10^{-12}, define Qδ(M)=(M/∑r,tMrt+δJ)/(1+δ)Q_\delta(M)=(M/\sum_{r,t}M_{rt}+\delta J)/(1+\delta) with δ=10−9\delta=10^{-9}. For mˉ(M)≤10−12\bar m(M)\le10^{-12}, set Qδ(M)=JQ_\delta(M)=J. Then Bδ(M)=arg⁡min⁡B∈UR,HDKL(B∥Qδ(M))B_\delta(M)=\arg\min_{B\in\mathcal U_{R,H}}D_{\mathrm{KL}}(B\|Q_\delta(M)), where UR,H\mathcal U_{R,H} contains probability matrices with row sums 1/R1/R and column sums 1/H1/H. Sinkhorn balancing gives the unique solution. CLE is the total-variation distance between the two resulting representatives, as defined above. This construction retains every region and time position. For active fields, the uniform regularization mass is δ/(1+δ)\delta/(1+\delta), independent of traffic units.
    • Whether this choice materially affects the assessment of interaction structure or the model rankings: Table 8 evaluates the 36 model-city comparisons at δ∈{10−12,10−10,10−9,10−8,10−6}\delta\in\{10^{-12},10^{-10},10^{-9},10^{-8},10^{-6}\}. Every city's model ordering is unchanged, with Spearman correlation equal to 1 relative to the default throughout this six-order-of-magnitude range. The maximum absolute score change is 3.963×10−53.963\times10^{-5} through δ=10−8\delta=10^{-8} and 4.096×10−34.096\times10^{-3} over the complete range. All 72 observed and modeled representatives reach the stated numerical tolerance within 61 iterations. The unchanged within-city rankings support consistent model selection when regional and temporal interaction structure is the property of interest. Researchers can compare this structural fidelity using a common treatment of zero entries, with the preferred ordering stable across the tested regularization choices.
Reviewer_aLYjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a novel benchmark for mobile traffic generation and forecasting, and does evaluate it extensively. It also provides a new error metric called Spatial Dispersion Ratio SDR to quantify how well the observed variation in traffic is maintained in the model outputs as well as evaluates the benchmark with multiple other performance indicators.

Critical strengths

The paper is very well written and it includes extensive experiments and evaluation, via benchmarking 12 mobile traffic models, eight for generation and four for forecasting. The paper presents also novel error metrics and shows that the presented benchmark is able to address more individual evaluation points than any previous benchmark alone and in addition evaluates the models’ cross-window variation and regional collapse that no other benchmark does.

Critical weaknesses

The discussion on how the benchmark could be used and how it would benefit research in practice is largely missing, so the motivation of the work is quite vague. But actually nothing very critical.

Questions for authors

Could you add a clarification on the explicit gap on the prior benchmarks that your method is addressing? Could you also present the definitions of the error metrics already in the introduction to make them more meaningful? The term PENTAD is not really explained. I guess it is a set of multi perspective quality indicators, but in the text it appears only in the figure, described that we develop PENTAD and in appendix, but in the main text it is not defined at all.

7 comments
AuthorsOpenReview ↗

Benchmark Motivation and Practical Use

Thank you for encouraging us to explain how the benchmark can support subsequent research. We have added a brief statement of the benefit for subsequent research within the existing Introduction and a dedicated Motivation section after Preliminaries and before the method section. That section describes its use in model development and selection. These revisions connect the benchmark's measurements to the research decisions they can inform.

During model development, researchers can first examine conventional errors alongside the structural diagnostics. SDR indicates whether regional variation is retained, SAD* quantifies spatial misallocation relative to the error of uniform allocation, and the complementary metrics separate volume, temporal shares, coupling, and fluctuations. A researcher can then formulate a targeted change and use a controlled ablation to test whether the intended property improves. The perturbation comparison demonstrates the diagnostic step. Similar overall errors can correspond to different structural changes, and a larger regional group error can arise from volume scaling even when regional shares remain unchanged.

During model selection, the operational tasks compare the decisions induced by candidate outputs under the relevant objective and costs. LastDay and PatchTST illustrate this use, with their advantages differing between capacity or energy decisions and congestion recognition. The added policy comparisons and sensitivity analyses show how the benchmark identifies both the benefit over simple strategies and the conditions under which it occurs. Researchers can diagnose what needs improvement, test whether an intervention addresses it, and select outputs that support the intended network application.

AuthorsOpenReview ↗

Clarifying the Gap in Prior Benchmarks

We are grateful for this suggestion. The revised explanation states the gap relative to existing spatial evaluation. ZoomDiff evaluates time-averaged grid demand with Spatial RMSE, and STContext compares performance across regional strata. These measurements reveal spatial accuracy and variation in performance across groups. The gap addressed by NetSpace-Bench is the need for a common, interpretable connection between those errors, the traffic structure retained by model outputs, and their additional value for network decisions.

Our protocol makes that connection through complementary diagnostics and explicit references. SDR measures the retained amount of relative regional variation. SAD* compares geographic demand allocation with uniform shares, while PENTAD additionally separates total volume, temporal allocation, regional temporal coupling, and cross-window fluctuations. The operational tasks then evaluate what happens when those outputs are used for capacity planning, energy saving, allocation, congestion warnings, and synthetic-data utility. This allows researchers to determine both what a numerical improvement represents and whether it supports the intended use.

Three findings motivate this evaluation need. SeasonalMean assigns the same profile to all regions while achieving the lowest generation RMSE in the three cities. The matched perturbation comparison shows how similar overall errors can mask different structural changes. The policy and operating analyses further identify benefits relative to simple strategies and the conditions under which model rankings change. The Introduction, dedicated Motivation section, related-work comparison, and expanded table now state this contribution before the detailed methodology.

AuthorsOpenReview ↗

Introducing Metric Definitions Earlier

Thank you for recommending that the metric definitions appear earlier. The Introduction now explains the metrics before presenting the numerical findings. It defines MAE as the mean absolute deviation over evaluated entries and RMSE as the square root of the mean squared deviation. It then introduces the structural metrics through their comparison objects and reference values, so readers can interpret the reported scores.

SDR is the modeled coefficient of variation of regional mean traffic divided by its observed counterpart. A value near zero indicates strongly reduced regional variation, while a value near one indicates comparable relative dispersion. SAD is the geographic Wasserstein distance between regional demand shares. SAD* divides that distance by the allocation error of uniform shares, so a value below one indicates improvement over uniform allocation. Combining these two diagnostics distinguishes retaining the amount of regional variation from allocating demand to the appropriate locations.

The Introduction also defines LBE as relative absolute total-volume error and TPD as Jensen-Shannon divergence between normalized temporal shares. CLE compares regularized mean fields after balancing their spatial and temporal marginals, and DLD compares cross-window fluctuations normalized by the respective local means. These definitions explain what each score contributes when interpreting the model comparisons. The metric section and appendix provide the full equations, aggregation rules, zero-value conventions, and numerical details. Readers can therefore interpret the main findings using the introductory definitions and consult the full specification for implementation details.

Reviewer_qYh3OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

NETSPACE-BENCH evaluates mobile network traffic generation and forecasting through regional dispersion, spatial allocation, five diagnostic perspectives, and five downstream proxy tasks. SDR measures preservation of variation in regional mean demand, while SAD* normalizes spatial allocation error against a uniform-allocation reference. Experiments compare eight generation methods and four forecasting methods across three cities, with controlled perturbations and downstream cost analyses.

Critical strengths
  1. The framework connects modeling errors to regional structure and application costs in an interpretable way. Distinguishing loss of regional variation from incorrect spatial allocation is practically useful; SeasonalMean's low RMSE despite absent regional variation illustrates why aggregate error alone can be insufficient for model selection.

  2. The documented protocols, processed data, configurations, and archived outputs could form a reusable evaluation resource. RMSE-matched perturbations and matched-mean-field controls provide useful diagnostic checks, and the paper appropriately acknowledges the limits of its metrics and offline proxy tasks.

Critical weaknesses

The incremental contribution over existing spatial and grouped evaluation is insufficiently established. Regional MAE/RMSE already penalize assigning identical values to different regions. ZoomDiff uses Spatial RMSE on time-averaged grid demand; STContext uses regional stratification in a related mobility domain. These are not equivalent to the full proposed protocol, but require a clearer account of its added value. Figure 5(a) shows equal RMSE but different overload outcomes for one region; the MAEs also differ (34.5 versus 105.6), so it does not show that SDR/SAD* outperform a conventional diagnostic combination. Table 10 supports specificity of factor-designed perturbations, but lacks comparison with grouped errors and existing spatial checks. A benchmark need not introduce a new predictive model; its significance instead depends on credible additional insight and a clearly specified data/protocol contribution.

Questions for authors
  1. Contribution clarification: What specific diagnostic or model-selection capability is gained over region/demand-stratified errors plus existing spatial metrics? Which data are newly curated or collected, and what is added relative to the upstream resources?

  2. Baseline validity: What evidence establishes reasonable convergence and scale fidelity for representative learned generators, given their extreme LBE? Please clarify method-specific adaptation/tuning and whether author-code comparisons or repeated runs support the central findings.

10 comments

Thank you for the careful and constructive review. You raised three points: (1) the added value of our metrics over grouped errors and existing spatial metrics, (2) our data and protocol contribution relative to the upstream resources, and (3) the validity of the learned baselines, including their convergence, output scale, and method-specific adaptation. We agree that the paper should explain these points more clearly, and doing so will substantially improve its readability and quality. We have revised the paper accordingly.

AuthorsOpenReview ↗

Added value of our metrics over grouped errors and existing spatial metrics (Weakness and Q1)

We agree that the current paper does not clearly explain how our evaluation differs from existing practice, including grouped errors and spatial metrics, and what it adds beyond them. It also does not clearly show the limitations of these existing metrics or how our benchmark provides a more comprehensive evaluation. To address this, we do a summary including ZoomDiff, STContext and etc, in Table 3 shown as follows.

Table 3. Spatial evaluation functions of grouped errors and existing spatial checks.

Metric and sourceDirect assessmentGrouped errorsDispersion RetentionSpatial MisallocationUniform-allocation reference
Groupwise RMSE / SMAPE (STContext)Accuracy within central and non-central regions✓✗✗✗
Spatial RMSE (ZoomDiff)Error in regional mean demand✗✗✗✗
Regional R² (MCDiff)Fit of regional mean demand✗✗✗✗
S-JSD (STK-Diff)Agreement of pooled traffic-value distributions✗✗✗✗
Ours, grouped errors with SDR and SAD*Group-specific accuracy and explicit spatial diagnostics✓✓✓✓

A check mark means that the metric directly evaluates the aspect listed in that column. Dispersion retention is measured by SDR, the ratio of modeled to observed coefficients of variation of regional mean traffic. SAD measures spatial misallocation as the Wasserstein distance between modeled and observed regional demand shares, using geographic distances as transport costs. SAD* normalizes this error by the corresponding error of uniform allocation. Grouped errors are retained alongside these diagnostics. The table concerns spatial evaluation only.

  • Regarding regional stratification of STContext, our diagnostics extend the analysis beyond reporting RMSE and SMAPE for predefined central and non-central areas. Grouped errors reveal where accuracy differs across the selected groups. SDR additionally quantifies how much regional differentiation is retained, while SAD* measures where demand shares are misplaced and compares that error with uniform allocation. In Table 15 of revised version, we retain the same principle of evaluating errors within region groups and report low-, middle-, and high-demand groups separately.
  • Regarding Spatial RMSE of ZoomDiff, our spatial evaluation goes beyond the magnitude of errors in regional mean demand. Spatial RMSE can increase when all regional demands are multiplied by the same factor even though their relative shares remain unchanged. SDR and SAD* distinguish preserved spatial structure under volume scaling from a change in regional demand allocation.
  • Regarding Regional R2 of MCDiff, our paired SDR and SAD* diagnostics go beyond a single goodness-of-fit score for regional means. Regional R² measures residual error relative to observed variation. SDR quantifies dispersion retention, while SAD* measures spatial misallocation relative to the uniform-allocation reference.
  • Regarding S-JSD of STK-Diff, our spatial evaluation goes beyond matching the pooled distribution of traffic values. S-JSD compares distributions after flattening the regional traffic matrix, so exchanging complete regional records preserves that distribution. SDR measures regional differentiation, and SAD* retains regional locations when assessing misplaced demand shares.

Additionally, our added value also lies in directly using these diagnostics together to distinguish different types of error. Grouped errors and regional fit scores show the magnitude and location of discrepancies, but do not always distinguish errors in total volume from changes in relative spatial structure. LBE measures total-volume error, SDR measures dispersion retention, and SAD* measures spatial misallocation relative to the uniform-allocation reference. Table 15 illustrates this distinction: volume scaling and spatial reallocation produce similar Spatial RMSE values, but different responses in LBE, SDR, and SAD*. Together, these diagnostics help researchers identify which properties of the model outputs need improvement and guide further analysis. Last but not least, while these metrics remain useful for measuring accuracy and locating large errors, SDR and SAD* are computed over all regions without choosing groups or thresholds, and they are not affected by total traffic volume. They therefore give an objective, scale-free assessment of how much regional variation a model retains and how well it allocates demand across regions, which can be compared directly across models and cities. We have added these clarification to the revised paper.

AuthorsOpenReview ↗

Clarifying the Role of Figure 6

As for the missleading presentation of conventional diagnostic combination shown in Figure 5(a) (now it is figure 6(a)), we apologize for not explaining the role of this figure clearly. The original caption reported the matched RMSE and the missed-overload counts, but did not report the candidates' MAE values. The main text cited the figure in the paragraph on spatial collapse. In the revised manuscript, the figure is cited in Section 5.4, which concerns decision evaluation. Figure 6 is a single-region illustration and does not involve SDR or SAD*, which are defined across regions. Its purpose is to show why the benchmark evaluates the decisions induced by model outputs in addition to their errors.

In Figure 6(a), both candidates have RMSE 111.6. The left candidate has the lower MAE, 34.5 compared with 105.6 for the right candidate, but misses five overload hours, whereas the right candidate misses none. RMSE and MAE together would therefore favor the candidate that hides the overloads. An overload warning depends on whether demand exceeds the threshold at the relevant hours, and a candidate can obtain a smaller average error while understating the peaks that trigger the warning. This outcome is captured by measures aligned with the decision, such as the congestion miss cost and the warning task, which is why the benchmark evaluates decisions directly and lets the operational objective inform model selection.

The revised Figure 6 places the error values and the missed overload counts side by side, with the annotations "Lower MAE, more missed overloads" and "Higher MAE, no missed overloads", and the caption now identifies overload detection as the property illustrated. The added value of SDR and SAD* over grouped errors and existing spatial metrics is shown by Tables 3 and 15 of the revised manuscript rather than by this figure.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions for the Area Chair

We thank the reviewers for their careful evaluation. Their reviews identify several strengths of the work. Reviewers qYh3 and YT9z both note the practical value of distinguishing the loss of regional variation from incorrect spatial allocation, and both describe the framework as interpretable: it connects modeling errors to regional structure and application costs in an interpretable way (qYh3), and its decomposition provides a clear, interpretable structure for the evaluation dimensions (YT9z). Both also regard the RMSE-matched perturbations and the matched controls as useful checks of the diagnostics. Reviewer aLYj finds the paper very well written with extensive experiments and evaluation, and Reviewer qYh3 notes that the documented protocols, data, configurations, and outputs could form a reusable evaluation resource. The main concerns and our revisions are summarized below.

  1. Added value over grouped errors and existing spatial metrics (qYh3, aLYj). Table 3 compares the diagnostic functions of existing metrics, and the new Table 15 evaluates grouped errors and existing spatial checks under all five RMSE-matched perturbations. After the response, Reviewer qYh3 wrote that these tables clarify how the proposed diagnostics relate to grouped errors and existing spatial metrics and that the revised discussion of Figure 6 resolves my concern about that example.

  2. Data and protocol contribution (qYh3). Appendix A.16 and Figure 8 separate the upstream inputs from our contributions: the processed datasets, model adapters, evaluation code, and benchmark protocol.

  3. Baseline validity (qYh3). Appendix A.4 documents each method's implementation source, inputs, configuration, and checkpoint rule, and Table 7 reports the validation-objective reductions. We corrected ZoomDiff's region-to-grid mapping so that it preserves regional traffic totals and updated its results. Each learned model is trained with three random seeds in each city. With these updates, 15 of the 24 generation evaluations retain less than 5% of the observed regional dispersion.

  4. Metric definitions (YT9z, aLYj). Appendix A.6 now gives explicit formulas for DLD and CLE, including the treatment of zero values, and Tables 8 and 9 report the sensitivity to the regularization and the coverage of excluded cells. Our response additionally reports that excluding the lowest-traffic cells leaves the top-ranked model unchanged. The Introduction now introduces PENTAD and all metrics before the findings.

  5. Decision value and operating points (YT9z). Appendices A.17 and A.18 add comparisons with always-alert and never-alert policies and sweeps over cost ratios, congestion quantiles, and the operating points of the other tasks. Section 5.4 now reports how the correlation between RMSE and warning regret changes sign across cost ratios.

  6. Motivation and practical use (aLYj). A new Motivation section describes how the benchmark supports model development and selection.

On novelty. Reviewer qYh3 deferred the assessment of novelty to reviewers closer to this area. On this point, the review of Reviewer aLYj, who reports familiarity with this topic and its relevant literature, describes the work as a novel benchmark that presents also novel error metrics and evaluates the models' cross-window variation and regional collapse that no other benchmark does. The review of Reviewer YT9z states that SDR and SAD* capture two complementary properties that aggregate error metrics do not explicitly distinguish. We hope these assessments are helpful to the Area Chair in evaluating the novelty of the work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.