- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Ride-Hailing, a dataset consisting of four years of ride-hailing marketplace observations across 200 areas at half-hourly resolution. It contains nine anonymized targets and weather, holiday, and event covariates. RideBench evaluates 32 methods on one-week (short-term) and eight-week (long-term) forecasting. The paper also combines overall and scenario-specific errors in the benchmark and introduces daily directional accuracy, daily correlation to capture long-term demand evolution. Additionally, they also use first-week accuracy within the longer horizon evaluations. Further analyses examine covariate access, training-area count, historical context, and training objectives.
Critical strengths
-
The released domain-specific dataset contains four years of data across 200 areas, nine targets, and multiple exogenous categories. This offer a substantial resource for studying heterogeneous marketplace forecasting. The evaluation suite combines novel ideas of evaluating disturbance periods and combining eight-week forecasts with near-term and temporal-aggregation metrics. This could expose behavior obscured by aggregate error alone. The proposed evaluation design is a valuable part of the benchmark contribution, alongside the dataset itself.
-
The 32 methods span seasonal baselines, channel-independent and multivariate models, exogenous-aware methods, and foundation models which is a broad empirical coverage. The training/testing splits and training-only normalization are appropriately used. Figure 5's comparisons with and without covariates and Figure 6's fixed-target-area scaling study is especially useful as it provides evidence about external information usage and pooled training, beyond simply reporting a leaderboard.
Critical weaknesses
-
The central dataset is however not sufficiently specified and validated. Section 3.1 describes synthesis, target transformations, weather aggregation, and event-impact encoding. However there is not much explaination about these procedures, and there is no quantitative checking about which original relationships are preserved or removed. The Table 4 is useful as it gives target identifiers and numeric types, but there is no coarse semantic descriptions or characterization of relationships among the nine targets. Figure 3 illustrates plausible patterns, yet leaves the weather-level construction, gain calculations, and heterogeneity representations insufficiently defined. For comparison, LargeST (NeurIPS 2023) documents source and selection procedures and explicitly describes its regional and temporal analyses [1]. Comparable, documentation would make RideBench's content more interpretable and scientific. Additionally, it is not clear what the future weather and event-impact labels signify. Clarifying these points would help strengthen the case about the real forecasting problem this paper claims to represent.
-
The long-horizon motivation and conclusions need further discussion. ProbTS (NeurIPS 2024) already evaluates point and distributional forecasts across short and long horizons [2], and TIME (ICML 2026) includes MASE/CRPS evaluation and motivates tasks through operational requirements [3]. The relevant distinction may be along the lines of the temporal behavior RideBench evaluates, rather than a general absence of long-horizon or probabilistic benchmarking. This needs further clarification. Another concern is that daily averaging retains weekday–weekend seasonality, so the introduced daily Acc./Corr. metrics do not isolate broad trend. Weekly or seasonally adjusted evaluation would help test that interpretation. I also am not convinced by the argument of "unavoidable trade-off" arising from different winners from each metric. TiDE leads both daily metrics and both full-horizon and first-week RMSE in Table 3.
References
[1] Liu et al. LargeST: A Benchmark Dataset for Large-Scale Traffic Forecasting. NeurIPS 2023, Datasets and Benchmarks. Published paper.
[2] Zhang et al. ProbTS: Benchmarking Point and Distributional Forecasting across Diverse Prediction Horizons. NeurIPS 2024, Datasets and Benchmarks. Proceedings.
[3] Qiao et al. It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarks. ICML 2026. Proceedings.
Questions for authors
-
The paper would benefit from stating what precisely was synthesized in the dataset and what is being transformed. Also what evidence supports preservation of forecasting-relevant structure? Can you provide descriptions of the nine targets and characterize their dependencies? Since multivariate models are already evaluated, per-target results and cross-target relationship analysis would clarify which dependencies are observed and whether they benefit forecasting.
-
How are event-impact (scores and categories) determined? This can help explicitly distinguish an operational forecast from forecasting conditional on oracle covariates.
-
In Figure 3, how are weather levels defined, which targets and normalization produce the plotted values, and how are the non-holiday baseline and percentage annotations computed? What does each heterogeneity point represent, and what transformation produces its x/y coordinates? Please also specify the benchmark's metric aggregation across targets, areas, and overlapping windows, and the categorical adapter used for each model.
-
What planning decision motivates eight weeks of half-hourly output and does the claimed trend difficulty persist under weekly aggregation or after removing weekly seasonality? A focused analysis of existing predictions, with uncertainty accounting for overlapping origins, would help establish this.
Additionally, a separate fine-tuning study of the foundation models, reporting adaptation data and compute alongside zero-shot performance, would be informative.
11 comments
Rebuttal Part 2
W1. The central dataset is however not sufficiently specified and validated. ... Clarifying these points would help strengthen the case about the real forecasting problem this paper claims to represent.
W2. The long-horizon motivation and conclusions need further discussion. ... This needs further clarification. Another concern is that daily averaging retains weekday–weekend seasonality, so the introduced daily Acc./Corr. metrics do not isolate broad trend. Weekly or seasonally adjusted evaluation would help test that interpretation. I also am not convinced by the argument of "unavoidable trade-off" arising from different winners from each metric. TiDE leads both daily metrics and both full-horizon and first-week RMSE in Table 3.
Thank you for your detailed feedback. We understand that your concerns mainly involve two aspects: the specification and validation of the dataset, and the motivation and interpretation of the long-horizon benchmark. We briefly address these below and provide the detailed responses in Q1–Q5.
(1) Dataset specification and validation. The original description did not sufficiently explain the processing of real operational records, the meaning of the transformed targets, or the evidence for retaining forecasting-relevant properties. We therefore clarify why the dataset is described as synthesized, characterize the relationships among the nine released targets, and provide comparisons between the original and processed data. We also explain the Figure 3 calculations and distinguish the benchmark's weather and activity annotations from the inputs used in production. These points are addressed in Q1–Q3.
(2) Benchmark motivation and interpretation. The eight-week, half-hourly task comes from the need to support both advance planning and near-term scheduling. Accordingly, its evaluation combines full-horizon errors, daily demand changes, and first-week accuracy. The daily metrics capture weekday–weekend and holiday-stage changes that matter for these decisions. Our distinction from ProbTS and TIME lies in combining these requirements with heterogeneous multi-area and exogenous-aware ride-hailing evaluation; we will clarify this positioning in the related work. Regarding the “unavoidable trade-off,” our intended point is that choosing among the evaluated models still requires balancing overall accuracy, first-week accuracy, and day-level trend quality. TiDE's strong RMSE results are valuable, but the operational requirement involves all three dimensions and the complementary metrics within each dimension. For example, the original Table 3 reports:
| Model | Overall: WMAPE / MAE / RMSE | Daily Acc. / Corr. | First week: WMAPE / MAE / RMSE |
|---|---|---|---|
| ModernTCN | 0.1262 / 4.816 / 14.34 | 0.6089 / 0.3273 | 0.1193 / 4.587 / 14.28 |
| TiDE | 0.1328 / 5.069 / 14.32 | 0.6137 / 0.3715 | 0.1194 / 4.582 / 13.99 |
| Chronos-2 (zero-shot) | 0.1374 / 5.270 / 15.07 | 0.5905 / 0.1702 | 0.1140 / 4.382 / 14.27 |
For operational evaluation, we place particular emphasis on WMAPE, which is listed first in our error tables. It normalizes each target's accumulated absolute error by its accumulated observed magnitude and then averages across targets, allowing us to assess relative error across targets of different scales. MAE measures absolute error on the released scale, while RMSE gives greater weight to large deviations. Thus, TiDE's lower RMSE shows an advantage under a large-error-sensitive criterion, while its overall WMAPE of 0.1328 remains above ModernTCN's 0.1262, and its first-week WMAPE of 0.1194 remains above zero-shot Chronos-2's 0.1140.
Our objective is a model that performs strongly across all three dimensions and their individual metrics. In the original comparison, TiDE's trend and RMSE strengths coexist with gaps in overall and near-term WMAPE. This is the practical model-selection trade-off we intended to emphasize. Our additional eight-week experiment in Q5 shows that fine-tuning Chronos-2 improves all three dimensions simultaneously and achieves a stronger balance, although the best results across individual metrics are still shared by different models. We will incorporate this finding and clarify the interpretation in the revised paper.
The operational motivation is detailed in Q4, and Q5 reports our additional foundation-model fine-tuning experiment.
We will incorporate these explanations and results into the revised manuscript.
Rebuttal Part 3
Q1. The paper would benefit from stating what precisely was synthesized in the dataset and what is being transformed. Also what evidence supports preservation of forecasting-relevant structure? Can you provide descriptions of the nine targets and characterize their dependencies? Since multivariate models are already evaluated, per-target results and cross-target relationship analysis would clarify which dependencies are observed and whether they benefit forecasting.
Thank you for your question. The nine targets originate from endogenous market indicators of a real ride-hailing platform and undergo anonymization, desensitization, aggregation, and other processing. After processing, the released targets should not be interpreted as the original business indicators with their original semantics. This is why we describe Ride-Hailing as a synthesized dataset. We are sorry that, to protect sensitive information, we do not disclose more detailed indicator definitions or the complete processing pipeline.
(i) Preservation of forecasting-relevant properties. The transformed data retain characteristics relevant to time-series forecasting research. Figure 3 illustrates multi-scale periodicity, responses to different exogenous factors, and heterogeneity. To further quantify their preservation, we apply the same aggregation procedures to the original and processed data and compare the resulting profiles using correlation (Corr.) and agreement in adjacent-change directions (Acc.).
| Periodic profile | Corr. | Acc. |
|---|---|---|
| Daily | 0.9982 | 1.0000 |
| Weekly | 0.9978 | 0.9970 |
| Yearly | 0.9285 | 0.9423 |
The daily and weekly profile correlations exceed 0.997, and the yearly correlation is 0.9285. Thus, the multi-scale periodic profiles remain highly consistent after processing. The corresponding exogenous-response comparisons are:
| Exogenous-response profile | Corr. | Acc. |
|---|---|---|
| Weather disturbance | 0.9762 | 1.0000 |
| Holiday effect | 0.9954 | 0.9730 |
| Event type 1 | 0.9979 | 0.9787 |
| Event type 2 | 0.9920 | 0.9787 |
For weather, we compare response gains across disturbance levels. For holidays, we compare individual public holidays' response-gain curves and average the consistency scores. Correlations across the four response profiles range from 0.9762 to 0.9979. In addition, the proportion of areas with decreasing holiday demand is similar before and after processing (6.50% afterward), retaining regional differences in response direction.
We also rank the 32 models by their average rank across evaluation metrics on each dataset and obtain a Spearman correlation of 0.8132 between the two rankings. Together, these comparisons support the preservation of the examined forecasting profiles and broadly consistent relative model performance.
(ii) Relationships among the nine targets. We further compute the Pearson correlation matrix of the released targets using all 10,512,000 training records, pooling the 200 areas:
| Target | endo_1 | endo_2 | endo_3 | endo_4 | endo_5 | endo_6 | endo_7 | endo_8 | endo_9 |
|---|---|---|---|---|---|---|---|---|---|
endo_1 | 1.000 | 0.534 | 0.746 | 0.808 | 0.841 | 0.965 | 0.585 | 0.849 | 0.783 |
endo_2 | 0.534 | 1.000 | 0.431 | 0.507 | 0.761 | 0.436 | 0.432 | 0.789 | 0.390 |
endo_3 | 0.746 | 0.431 | 1.000 | 0.790 | 0.596 | 0.718 | 0.865 | 0.627 | 0.954 |
endo_4 | 0.808 | 0.507 | 0.790 | 1.000 | 0.680 | 0.775 | 0.709 | 0.699 | 0.793 |
endo_5 | 0.841 | 0.761 | 0.596 | 0.680 | 1.000 | 0.786 | 0.493 | 0.935 | 0.605 |
endo_6 | 0.965 | 0.436 | 0.718 | 0.775 | 0.786 | 1.000 | 0.560 | 0.754 | 0.776 |
endo_7 | 0.585 | 0.432 | 0.865 | 0.709 | 0.493 | 0.560 | 1.000 | 0.518 | 0.817 |
endo_8 | 0.849 | 0.789 | 0.627 | 0.699 | 0.935 | 0.754 | 0.518 | 1.000 | 0.621 |
endo_9 | 0.783 | 0.390 | 0.954 | 0.793 | 0.605 | 0.776 | 0.817 | 0.621 | 1.000 |
The off-diagonal coefficients range from 0.390 to 0.965. In particular, endo_1–endo_6, endo_3–endo_9, and endo_5–endo_8 have correlations of 0.965, 0.954, and 0.935. Thus, the observed correlations support close relationships among the released targets. We will add the correlation heatmap to the revised main text.
Rebuttal Part 1
Thank you for your very detailed and thoughtful review. Your assessment of both the strengths and weaknesses has been valuable in helping us examine the work more carefully, and we welcome the opportunity to discuss these points in depth.
Thank you for recognizing these contributions. We provide a large-scale dataset derived from real industrial operations together with a comprehensive benchmark. The forecasting challenges motivating this work arise directly from our production needs and also offer valuable research problems for the academic community.
(1) Dataset. We understand the importance of high-quality datasets. We therefore chose to release a dataset for the ride-hailing setting, where market-driven demand brings together multi-scale periodicity, diverse covariate effects, and heterogeneity across areas. These are challenges that forecasting methods must address in real applications. Releasing this dataset also requires protecting sensitive information. We therefore need to anonymize and transform the data to balance research value with sensitive-information protection. The original submission described the source and processing too briefly. In the revision, we will expand this description and provide target-relationship analyses and quantitative comparisons between the original and processed data, as detailed in the responses below.
(2) Benchmark. We also understand the importance of a well-specified benchmark and have incorporated operational considerations into its design. For example, business evaluation requires errors to reflect the magnitude of deviations on the original scale. We therefore use training-only normalization and inverse-transform predictions before calculating test errors, so MAE and RMSE are expressed in the units of the released targets. We also include target-wise WMAPE to assess errors relative to each target's scale. Beyond the metrics, joint modeling across heterogeneous areas and the choice of exogenous covariates reflect the spatial diversity and external influences encountered in production. Similarly, evaluating first-week accuracy together with daily trajectory consistency over eight weeks reflects the need to support both near-term scheduling and advance planning.
Thank you again for recognizing the value of this work. In the responses below, we explain the dataset construction and evaluation choices in greater detail, within the constraints of sensitive-information protection, and provide additional evidence to address your concerns. We hope these clarifications and results help resolve the questions raised in your review.