acceptodds
Under review as a conference paper at ICLR 2027

CrossCityBench: A Diagnostic Benchmark for Cross-City Spatio-Temporal Generalization

Abstract

Cross-city spatio-temporal forecasting asks whether source-city information improves prediction when a target city provides limited observations and follows a different distribution. Existing evidence is difficult to compare because studies vary source-target pairs, target-data budgets, forecasting protocols, and reported metrics. CrossCityBench evaluates target-only learning, alignment, meta-learning, pre-training, knowledge distillation, and federated learning under matched target information and forecasting conditions. It first tests whether source access improves on the strongest target-only reference, then examines whether the recorded ordering persists across computational cost, missing-data sensitivity, city-pair sensitivity, Spatio-Temporal Pattern Bank (STPB) alignment, and federated collaboration cost. Under two core tasks and a matched three-day target-information protocol, D2MHyper and CrossST are the only evaluated transfer methods whose recorded average Mean Absolute Error (MAE) is below the strongest target-only value on both tasks. Accuracy and sensitivity produce different empirical orderings, so in-domain error alone is insufficient for selection. STPB is an optional prototype-alignment diagnostic, not an explanation of learned representations. The resulting guide ties each candidate to measured evidence and remains restricted to the evaluated traffic settings. CrossCityBench provides a falsifiable protocol for making transfer claims comparable and diagnostically traceable. Code, configurations, and data-processing resources are available at https://anonymous.4open.science/r/CrossCityBench-87CB.

Primary area:
datasets and benchmarks
Keywords:
Cross-City Traffic Forecasting, Spatio-Temporal Generalization, Transfer Learning, Benchmarking, Domain Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MfP1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper address inconsistent evaluation protocols and the lock of comprehensive assessment in cross-city ST transfer learning. To solve these issues, a standardized benchmark with 23 models acorss 3 learning paradigms is introduced. Results show that transfer learning does not always outperform target-only training, with CrossST and D2MHyper demonstrating the strongest performance across the two core tasks but different robustness trade-offs.

Critical strengths
  1. This paper pointed out three legitimate challenges that the ST transfer learning is facing: inconsistent evaluation protocels, unclear benefits of transfer learning, and accuracy-only evaluations. A standardized benchmark was proposed to address the challenges above.
  2. The key findings will be beneficial to this area. 1) transfer learning is not always beneficial, 2) no universally superior method exists
Critical weaknesses
  1. The novelty of this paper is limited, although it is submitted as a benchmark and dataset paper. It does not introduce any new datasets but just a benchmark framework built on existing public spatio-temporal dataset such as PeMS03, PeMS08, PsME-BAY and METR-LA.
  2. As a benchmark paper, several important baselines are omitted for comparison such as AdaRNN[1], DSAT-Net[2], TPB[3], TransGTR[4], and TSJT[5]

[1] Du, Y., Wang, J., Feng, W., Pan, S., Qin, T., Xu, R., & Wang, C. (2021). AdaRNN: Adaptive Learning and Forecasting of Time Series. Proceedings of the 30th ACM International Conference on Information & Knowledge Management, 402–411. [2] Tang, Y., Qu, A., Chow, A. H. F., Lam, W. H. K., Wong, S. C., & Ma, W. (2022). Domain Adversarial Spatial-Temporal Network: A Transferable Framework for Short-term Traffic Forecasting across Cities. Proceedings of the 31st ACM International Conference on Information & Knowledge Management, 1905–1915. [3] Liu, Z., Zheng, G., & Yu, Y. (2023). Cross-city Few-Shot Traffic Forecasting via Traffic Pattern Bank. Proceedings of the 32nd ACM International Conference on Information & Knowledge Management, 1451–1460. [4] Jin, Y., Chen, K., & Yang, Q. (2023). Transferable Graph Structure Learning for Graph-based Traffic Forecasting Across Cities. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 1032–1043. [5] Zhang, Y., Wang, X., Yu, X., Sun, Z., Wang, K., & Wang, Y. (2025). Drawing Informative Gradients from Sources: A One-stage Transfer Learning Framework for Cross-city Spatiotemporal Forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, 39(1), 1147–1155.

Questions for authors

Refer to the weaknessess above

Reviewer_7wRpOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CrossCityBench, a diagnostic framework to evaluate cross-city spatio-temporal forecasting methods The main motivation (and contribution) of this paper is to evaluate a bunch of such methods under a controlled comparison --- dataset pairs, normalization, horizon, pre-processing, etc. The evaluation revealed some interesting findings --- only a few (CrossST, D2MHyper) methods compare favorably against the best target-only methods. The authors also analyze some understudied properties, such as sensitivity to missing data, compute requirements, and federated learning.

Critical strengths
  1. The attempt to unify cross-city spatio-temporal forecasting under a comparable framework is meaningful and of impact. Throughout the years, spatio-temporal forecasting (especially cross-city spatio-temporal forecasting) is a field that lack standardized and controlled benchmarks, and thus I appreciate the authors' efforts in trying to construct one.

  2. The findings are of practical value in this field. I personally work in this field of cross-city spatio-temporal forecasting, and thus find the findings of this paper of value --- most methods for cross-city spatio-temporal forecasting did not outperform the best target-only baselines. This finding calls for more rigorous evaluations in this field, which, in my opinion, is very much appreciated.

Critical weaknesses
  1. The evaluated settings are very limited. I understand that the amount of compute required is considerable, but as a benchmark & evaluation paper, this is what people would expect. The authors only evaluate on a very constrained setting --- 3 days of target data, two source-target pairs. This makes the evaluation (and the findings) rather weak. I would recommend the authors experiment on LargeST (https://proceedings.neurips.cc/paper_files/paper/2023/file/ee57cd73a76bd927ffca3dda1dc3b9d4-Paper-Datasets_and_Benchmarks.pdf), which lie exactly in the PeMS system and are much larger and more diverse.

  2. The factor of "heterogeneity" is not well evaluated and analyzed. The authors clearly recognized "heterogeneity" as an important factor -- in Figure 1 (disparity) and in Section 3 (structural heterogeneity), but no experiments are done to analyze how heterogeneity played a role in the results. This is crucial because in transfer learning/domain generalization, the "distance" between domains & tasks is the key. However, the authors never propose a metric to define the heterogeneity or analyze how heterogeneity played a role. I consider it an overclaim of this paper.

  3. The categorization of existing methods did not transfer to meaningful observations/analyses/conclusions. Because the authors categorize existing methods into "distillation/pre-training/alignment", I would expect that the authors would show some paradigm-level diagnostics, such as "what kind of methods perform better on some scenarios but not on others". However, no such observations are given -- this is even more disappointing because in Table 4, pre-training methods perform the best, while in Table 5, alignment methods perform the best. The authors could have attributed the difference to some paradigm-level differences, but I did not see such efforts. Another evidence to support my confusion is that, intra-category variance goes beyond inter-category variance. For example, in Table 4, the worst pre-training method very much underperforms other paradigms. This makes the categorization of existing methods kind of bland and uninformative --- neither did it transfer to practical insights, nor did it explain the predictive accuracy.

  4. Figure 3 is confusing. From my understanding, STG-NCDE is a competitive target-only method, which warrants it to get a position in Figure 3 (as a competitive target-only baseline), but no --- only DyHSL exists in Figure 3. It becomes more confusing as the authors have clearly evaluated STG-NCDE, but did not include it in Figure 3. This makes it very confusing for readers to understand the key information of Figure 3.

  5. Section 4.3.2 is very confusing. In Figure 4(a), the authors deliver a result that DyHSL achieves more accurate results under missing data --- which is counter-intuitive and the authors attributed it to "fluctuations", which is confusing --- if it were indeed fluctuation, the authors could have tested it multiple times and obtain results that are reliably free of "fluctuations" --- yet they did not. Moreover, since DyHSL is a target-only baseline, Figure 4(b) becomes even more confusing --- how can a “target-only” baseline achieve different results under "city-pair contrast"? Did you alter the target city? If so, the authors should not have compared the results under the same figure --- the basis is not comparable. If the authors did not alter the target city, then how DyHSL changed in terms of predictive performance (in Figure 4(b))? This contradiction looks really confusing to me.

Questions for authors

Aside from addressing the "weaknesses", I do not see any ways in which the authors can materially affect my assessment.

Reviewer_3EhROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CrossCityBench compares cross-city traffic forecasting implementations under a stated common target-information budget. The suite includes ten target-only models, ten centralized transfer models, and three federated models evaluated under a separate protocol. The two core transfers concern traffic flow and traffic speed. Additional analyses examine resource costs, missingness, city-pair changes, an optional prototype-alignment score, and federated communication. The paper's main empirical message is that transfer utility and the preferred implementation depend on the task and operating constraint. It also proposes execution records and admission rules intended to make future comparisons traceable.

Critical strengths
  1. Including ten target-only candidates makes the first comparison useful. A transfer method is tested against the strongest recorded target-only result rather than merely against other transfer methods or a weak reference. The explicit inclusion of validation in the scarce-target budget is also the right design choice. If enforced by the execution pipeline, this would provide a meaningful common basis for comparing otherwise incompatible learning procedures.

  2. The paper is careful about several interpretation boundaries that matter here. Federated results are separated from centralized comparisons, uploaded payload is not presented as total communication, and dataset catalogues are distinguished from executed evaluations. The missingness and city-pair results are not claimed to identify causal mechanisms. These choices keep much of the discussion appropriately tied to the measured settings.

Critical weaknesses
  1. I could not reconcile the released execution paths with the comparison contract. Section 4.1 includes fitting and checkpoint selection within three days of target observations and fits normalization statistics on the training partition. But the anonymous repository's D2MHyper training entry point hardcodes seven target training days. Its splitting helper takes validation from a separate 10% portion of the target series outside those training days. The released STG-NCDE loader computes normalization statistics before splitting the complete series into training, validation, and test partitions. These are differences in information access, which at the first place I thought was just optimization preferences. If these paths produced Tables 4 and 5, the advertised matched comparison does not hold. If separate wrappers or another revision produced the tables, the paper needs to bind the reported results to those paths and their actual split records. More protocol declarations in Sections J and K do not resolve this mismatch. Because comparability is the benchmark's main contribution, I need this connection established before relying on its rankings.

  2. The headline finding does not match what Tables 4 and 5 report. DAGN has average MAE 18.36 on flow, below the strongest target-only value of 18.62, and 3.33 on speed, below 3.82. It therefore satisfies the same fixed-execution criterion used to identify D2MHyper and CrossST. The assertion that only those two implementations improve on target-only learning across both tasks, repeated in the abstract and Sections 4.2 and 5.1, excludes a qualifying third method. Again I thought it was barely a request to declare DAGN's small flow advantage statistically significant but It turns out to be a mismatch with the paper's own descriptive predicate. The authors should reconcile the underlying results and regenerate the summary from the complete model-level table.

Questions for authors
  1. Which code revision, invocation, and split files generated Tables 4 and 5? In particular, where is the implementation that keeps both target fitting and validation inside three days and fits normalization only on the declared training partition? An existing execution record could resolve the first concern. I am asking for the provenance of the reported runs, not a new tuning study during rebuttal.

  2. Are DAGN's average MAEs of 18.36 and 3.33 correct? If so, what additional prespecified rule excludes it from the cross-task utility finding? If there is no such rule, please correct the finding consistently wherever it appears.

10 comments
Area_Chair_1qQkOpenReview ↗

Rebuttal Follow-up

Dear Reviewer 3EhR,

As the Author-Reviewer Discussion period approaches its deadline, could you please take a moment to review the authors' rebuttal?

Thank you for your time and effort!

I literally checked the supplementary files myself and found that the original training commands, checkpoints, and predictions are listed as unknown for all 40 rows. The original submission presented Tables 4 and 5 as a fair comparison under a shared three-day budget, but the authors have yet to establish that the runs behind those tables followed that protocol. The ten new runs demonstrate a narrower comparison that excludes both D2MHyper and STG-NCDE. That is a substantive change in the evidence, and it leaves the original submission’s main claim unsupported. I keep my original rating of 2

AuthorsOpenReview ↗

Interpreting the Historical Run Ledger and the Independent CrossCityBench Results

We appreciate the reviewer’s examination of the supplementary files. The comment moves from 40 archival “unknown” entries to a conclusion about the entire paper by merging three distinct objects: historical numerical results, newly executed matched comparisons, and diagnostics computed from recorded data.

1. What the 40 “unknown” entries mean. In docs/ORIGINAL_ROW_EXECUTION_LINKS.csv, “unknown” identifies unavailable original training commands, checkpoints, and saved predictions. It records artifact availability. Questions about target-history access, preprocessing, and training schedules require execution-level information. The ledger also maps all 40 model-task rows and 480 metric cells to Tables 4 and 5, enabling exact numerical recomputation. An archival status field contains no measurement of historical budget usage. The reviewer’s inference exceeds what that file records.

2. What the three-day protocol actually specifies. Section 4.1 of the uploaded manuscript defines 864 target observations, including 691 for fitting and 173 for validation, fitting-only normalization, preprocessing within the allowance, and aligned forecast origins. It separates native defaults from standardized interfaces. D2MHyper’s seven-day native default and the STG-NCDE preprocessing order are described explicitly. A model’s source-code default and the configuration of a particular execution are separate facts. The current adapters expose the three-day setting for direct inspection.

3. What the ten new executions demonstrate. AdaRNN, TPB, TransGTR, TSJT, and a newly trained target-only Graph WaveNet were evaluated on both flow and speed. The ten runs follow the stated target allowance, fitting-only scalers, and fixed test origins. The run index identifies configurations, metrics, and artifact hashes. The ZIP contains inputs, indices, and checks, while large weights and predictions are described in a separate artifact archive. This is a controlled five-method comparison across two tasks with a specified experimental scope. The original 40-row matrix is separate. Its two questioned models fall outside this defined method set, which leaves the executed five-method comparison intact.

4. What happens when D2MHyper and STG-NCDE are removed. Main-text Table 7 and Appendix M.12 directly examine four inclusion conditions in the original matrix. With both rows excluded, the strongest retained target-only references are 19.16 flow MAE and 3.82 speed MAE. DAGN records 18.36 and 3.33, while CrossST records 14.67 and 3.21. Both remain below the respective references in the archived matrix. This directly tests whether the recorded preference depends entirely on those two rows. The conclusion is conditional on the archived measurements and reproducible from supplied files.

5. Why the broader diagnostic findings remain assessable. CrossCityBench asks how reference choice, cross-city heterogeneity, missingness, and operating cost affect model selection. For instance, 100 flow/speed reference combinations are enumerated using archived CSV inputs and executable scripts. Those calculations describe the sensitivity of recorded rankings to comparator selection. The new ten-run block separately tests a matched-setting comparison. The validity of a conditional numerical calculation and the reproducibility of a new execution must be judged from their own inputs and procedures. The ledger’s unknown fields supply no direct counterexample to either analysis.

6. Where the updated paper presents these results. Section 4.1 explains the information contract and comparison layers. Table 7 brings the key exclusion result into the main text. Appendices M.7, M.8, M.11, and M.12 document implementation checks, the controlled executions, comparator enumeration, and row-selection sensitivity. The updated manuscript is already uploaded.

In summary, the reviewer’s conclusion extends beyond the issue identified in the supplementary ledger. The cited fields establish archival availability, while the manuscript also contains separately executed experiments and independently computable diagnostics. It records which historical run artifacts are available and enables exact correspondence with the original tables. The new executions address a controlled comparison under a specified target allowance. The diagnostic analyses quantify comparator dependence and inclusion sensitivity from disclosed inputs. These are distinct scientific results with distinct assessment criteria. We ask the reviewer and the Area Chair to evaluate each conclusion against its corresponding materials in the uploaded manuscript, and we welcome a specific technical question about any remaining point.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Request for Joint Consideration of Rebuttal Progress and Outstanding Review Questions

Dear Program Chairs, Senior Area Chairs, and Area Chairs,

We sincerely appreciate your time, guidance, and thoughtful consideration of CrossCityBench (Submission #15068). We would like to bring the latest developments in our rebuttal discussion to your attention and respectfully invite your joint consideration of the remaining questions.

Throughout the discussion, we have provided detailed, point-by-point responses to all three reviewers, supported by additional experiments, quantitative analyses, updated manuscript sections, and supplementary materials. Several important developments have emerged from these exchanges.

First, Reviewer 7wRp has explicitly recognized the practical significance of CrossCityBench, the substantial improvements in methodological coverage, and the expanded analysis of cross-city heterogeneity. The reviewer has also updated the overall assessment. Our subsequent responses further explained how the main manuscript presents the benchmark framework, unified evaluation protocol, core comparisons, and principal findings, while the appendices provide detailed supporting analyses.

Second, Reviewer 3EhR has explicitly accepted the newly completed five-method comparison under the shared three-day target-information budget. The latest discussion has focused on the treatment of zero-valued observations in METR-LA. We have provided a detailed quantitative response using two evaluation conventions applied to identical forecasts, including exact label populations and the resulting changes in model rankings. This analysis directly illustrates the importance of evaluation conventions in cross-city model selection.

Third, Reviewer MfP1's concerns regarding benchmark novelty and baseline coverage have received comprehensive technical responses. We have addressed all five requested publications, completed additional comparisons involving four transfer methods and a newly trained target-only control, and presented the systematic analysis of 100 reference combinations. We sincerely appreciate the Area Chair's earlier invitation to continue this discussion. Further feedback from this reviewer would be especially valuable for clarifying how these completed analyses inform the original assessment.

These developments demonstrate substantial progress in addressing the reviewers' technical questions and strengthening the scientific presentation of CrossCityBench. The updated manuscript and supplementary materials have already been uploaded, with the relevant experimental results and implementation details available for examination.

As the review process advances, we respectfully invite the Program Chairs, Senior Area Chairs, and Area Chairs to consider the complete rebuttal record and help facilitate further discussion of any specific outstanding concerns. We would particularly appreciate consideration of the reviewers' updated perspectives, the technical questions that have received substantive responses, and the scientific contributions established by the expanded evaluations.

We sincerely hope that the complete revised manuscript, the detailed rebuttal, and the progress made during the discussion will receive careful consideration in the overall assessment. We remain fully available to respond promptly to any further technical questions.

Thank you very much for your time, guidance, and thoughtful consideration.

Sincerely,
The Authors

AuthorsOpenReview ↗

Concluding Statement on CrossCityBench's Scientific Contributions and Comprehensive Assessment

Dear Program Chairs, Senior Area Chairs, and Area Chairs,

We appreciate your careful consideration of CrossCityBench (Submission #15068). We highlight the research contribution, quantitative findings, and rebuttal progress.

We respectfully request a holistic scientific assessment of the complete manuscript and discussion record, beyond a simple arithmetic average of reviewer ratings.

1. The central innovation is diagnostic evaluation of cross-city transfer utility.

CrossCityBench addresses a fundamental question: when does source-city information improve forecasting over a strong target-only alternative under comparable information access? It connects information budgets, reference selection, heterogeneity, missingness, forecasting horizons, costs, and federated communication to model selection. This structured evaluation goes beyond a leaderboard by identifying the conditions under which transfer is valuable.

The benchmark covers 27 implementations across target-only and diverse transfer mechanisms. The main text presents the common protocol, core comparisons, and diagnostic findings. Appendices supply detailed analyses within page constraints.

2. The findings materially change how cross-city results should be interpreted.

Across 100 combinations of target-only flow and speed references, between three and ten transfer implementations qualify on both tasks. Eighteen combinations make all ten qualify, while the strongest task-specific references retain D2MHyper, DAGN, and CrossST in the archived comparison. Thus, reference choice changes apparent transfer utility even with fixed predictive entries.

Model preferences also vary across task, horizon, and operational requirement. For instance, CrossST leads average flow MAE, while FGITrans achieves 8.12% lower 60-minute flow MAE at 6.42 times the peak memory. In the separate federated setting, FedGTP uploads 89.83% less than FedCTPM with an accuracy trade-off. These findings inform deployment decisions.

3. Our rebuttal supplies concrete experiments and precise methodological answers.

We submitted detailed responses and revised materials. At the reviewers' request, AdaRNN, TPB, TransGTR, and TSJT were evaluated alongside a newly trained Graph WaveNet target-only control. The ten new executions use 864 target observations, split into 691 fitting and 173 validation observations, with fitting-only normalization and aligned test origins.

We corrected the joint-transfer membership to include DAGN and supplied reproducible arithmetic correspondence for 40 historical model-task rows and 480 metric cells. The manuscript distinguishes archived calculations from new controlled executions.

The METR-LA discussion yielded a further important diagnostic: identical forecasts produce different rankings under all-label and positive-label scoring. Under the established zero-masked benchmark convention, Graph WaveNet leads the new five-method speed comparison. All-label scoring remains a sensitivity analysis. Observation handling can reverse the apparent benefit of transfer, reinforcing the benchmark's scientific motivation.

4. The discussion record documents meaningful scientific progress.

Reviewer 7wRp recognized the practical value of CrossCityBench and the improvements in heterogeneity analysis and methodological coverage. Reviewer 3EhR explicitly accepted the separate five-method experiment under the shared three-day target-information budget. We addressed METR-LA zeros using paired scoring and established conventions. For Reviewer MfP1, we addressed all five requested publications, provided four new baseline comparisons, and explained benchmark novelty through the 100-reference analysis.

These developments add substantive information beyond the initial ratings. The chairs can assess each result and remaining question on its scientific merits.

5. Request for an independent assessment of originality and impact.

CrossCityBench provides a reusable framework for asking which source information, target domain, evaluation convention, and operating condition make cross-city transfer useful. Its importance lies in the methodological framework and in findings that change practical model-selection conclusions.

We respectfully ask the Program Chairs, Senior Area Chairs, and Area Chairs to evaluate the benchmark's originality, scientific utility, revised manuscript, and complete rebuttal on their merits. We hope the decision reflects the complete scientific record, beyond the arithmetic mean of numerical ratings.

Thank you for your time and thoughtful consideration.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.