- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SHR, a graph deep learning framework that leverages variations across hierarchies as multi-task regularization. Specifically, a training framework is used to optimize a shared spatiotemporal forecasting model across multiple structurally diverse hierarchies. In addition, an efficient recursive procedure is introduced to incorporate forecasts from every aggregation level while enforcing the corresponding coherence constraints. Experimental results demonstrate the effectiveness of the proposed methods.
Critical strengths
-
The empirical analysis examines several relevant aspects of the proposed mechanism.
-
The paper is easy to follow.
Critical weaknesses
The methodological novelty is limited. The forecasting architecture is adapted from HiGP, and forecast averaging across multiple hierarchies has been explored previously. The paper should more clearly establish the contribution of shared multi-hierarchy training beyond structural data augmentation and forecast combination.
The experimental comparisons leave several important issues unresolved:
- The main results average predictions across 20 hierarchies for SHR, whereas the fixed-hierarchy baseline uses one structure. Although the appendix includes evaluations without averaging, the headline comparisons do not clearly separate training regularization from additional inference computation.
- The scalability experiments primarily compare the proposed operator with dense projection-based reconciliation. Comparisons with implementations that exploit tree structure or sparse operations are needed.
- The inductive experiment compares SHR only with a fixed-hierarchy model on one GLA spatial split, providing limited evidence of its advantage over other transferable forecasting models.
Questions for authors
-
How much of the improvement in Table 1 is retained when SHR uses a single inference hierarchy? Could you report comparisons under identical reconciliation and inference settings to isolate the contribution of multi-hierarchy training?
-
The improvements reported in Table 1 over the baselines appear marginal, making it difficult to rule out the effect of random experimental variation. Could you report results averaged over multiple runs with different random seeds, together with standard deviations or statistical significance tests?
-
The main architecture figure is somewhat difficult to interpret, and the interactions among the different modules are not fully clear. Could the authors provide a more detailed walkthrough of the pipeline, clarifying the role of each component and how they interact within the overall framework?
-
Please evaluate the robustness of SHR under more challenging conditions. For example, does the method remain effective in long-term forecasting settings (e.g., T = 96, 192, 336, and 720)? Such experiments would help demonstrate the robustness and generalization ability of the proposed method.
6 comments
Weakness 4: The inductive experiment compares SHR only with a fixed-hierarchy model on one GLA spatial split, providing limited evidence of its advantage over other transferable forecasting models.
The inductive experiment complements the other experiments presented in the paper, which provide extensive evidence of the effectiveness of the methods. The purpose of this experiment is to isolate the effect of multi-hierarchy training when the model is applied to unseen series, which is why we compare SHR with the same model trained on a fixed hierarchy. HiGP cannot be applied directly, as it relies on node embeddings that do not transfer to unseen graphs (as explained in Sec. 5). The original inductive results use a single region of LargeST, Greater Los Angeles (GLA), split into subgraphs. To strengthen this evidence, we added to Tab.4 a harder inductive protocol that uses the three geographically disjoint regions of LargeST: the model is trained on GLA and the Greater Bay Area and tested on San Diego, a region never seen during training. SHR (rand.) achieves the lowest error in both settings (Tab. 4) and at all horizons (Tab. 10), supporting the benefit of our approach. We thank the reviewer for this suggestion.
Question 2: The improvements reported in Table 1 over the baselines appear marginal, making it difficult to rule out the effect of random experimental variation. Could you report results averaged over multiple runs with different random seeds, together with standard deviations or statistical significance tests?
Table 1 already provides standard deviations over multiple runs with different random seeds, as do all other results in the paper. We now state it also in the caption of Tab. 1.
Question 3: The main architecture figure is somewhat difficult to interpret [...] Could the authors provide a more detailed walkthrough of the pipeline, clarifying the role of each component and how they interact within the overall framework?
We thank the reviewer for the suggestion. We have replaced Figure 1 with a more detailed walkthrough of the pipeline (see Fig.1), which separates the training and inference steps and shows the role of each component. At each training step, a hierarchy is sampled, the shared forecaster predicts all its nodes, the recursive operator makes the predictions coherent, and the loss is computed at every level. At inference, the same model is applied under several sampled hierarchies, and the coherent bottom-level forecasts are averaged. The same steps are described in Sec. 3.1 and at the beginning of Sec. 3.
Question 4: Please evaluate the robustness of SHR under more challenging conditions. For example, does the method remain effective in long-term forecasting settings (e.g., T = 96, 192, 336, and 720)? Such experiments would help demonstrate the robustness and generalization ability of the proposed method.
We did not consider long-term horizons because our method relies on spatial hierarchies, whose main benefit is to capture dependencies among the series, and these are most relevant at short horizons. For example, in traffic, long-horizon forecasts are largely driven by daily and weekly patterns, which reduces the advantage of exploiting a graph structure. For this reason, we follow the standard horizons of the spatiotemporal forecasting literature for that domain (Sec. 5). A natural way to address long-term forecasting would be to use temporal hierarchies, which aggregate the series over increasingly coarser time scales, and we mention this as an interesting extension of our work in Sec. 6. Regarding the robustness of our method, the paper already includes extensive experiments, including the inductive setting on unseen graphs (Sec. 5 and App. C.4), ablations (Tab. 2 and 3), and sensitivity analyses (App. C.2) that probe the effectiveness of the approach under many different scenarios.
We thank the reviewer for the feedback. We have addressed the points raised in the review, adding new experiments and extending the discussion where needed. We kindly ask the reviewer to reconsider the score in light of these clarifications, and we are happy to continue the discussion if any aspect remains unclear.
[1] Cini A. et al., Graph-based time series clustering for end-to-end hierarchical forecasting, ICML 2024
[2] Rangapuram S. et al., End-to-End Learning of Coherent Probabilistic Forecasts for Hierarchical Time Series, ICML 2021
[3] Hyndman R. J. et al., Fast computation of reconciled forecasts for hierarchical and grouped time series, Computational Statistics & Data Analysis 2016
Thank you for your response. While some of my concerns have been addressed, several key issues remain unresolved, including the need for stronger evidence of the proposed method’s advantages over other transferable forecasting models and an evaluation of SHR’s robustness under more challenging conditions. After considering the authors’ overall response, I have decided to maintain my original score.
We thank the reviewer for appreciating our empirical analysis and for finding the paper easy to follow.
The reference architecture stems from HiGP: please note that we are not interested in proposing a specific architecture, but rather a general framework, which we instantiate with this architecture. In fact, relying on an existing architecture allows us to isolate the effect of training on multiple hierarchies from architectural choices. As for “forecast averaging”, it is a very broad idea, and the main difference between our approach and existing methods is in how the forecasts being combined are obtained. In our case, forecasts are obtained from the same model by processing different hierarchical views of the same input data: this is a novel approach as we discuss in the paper.
Our main contribution is introducing multi-hierarchy training and inference as a method for regularizing hierarchical spatiotemporal models and, as such, obtaining more accurate predictions in this context. This is a clear, distinct, and novel contribution with respect to prior works, as we discuss in the paper (Sec. 4). (1) Unlike data augmentation, this is not just about training on more data: the hierarchy has a strong impact not only on prediction targets but also on how the input is processed, as it defines the aggregates, the paths through which information propagates, and the coherence constraints. (2) As discussed above, while combining predictions obtained by different means is a standard approach to improve forecasting accuracy, the novelty here is how these forecasts are obtained. We have revised the introduction to make these aspects clearer.
In SHR, the prediction is defined as the expected forecast over the distribution of hierarchies, which we estimate by averaging over sampled hierarchies (Eq. 6); which is why we report it in the main results. As you correctly point out, the appendix already included many analyses showing the impact of the regularization even without inference-time averaging. We have now moved one of these to the main body of the paper (Fig. 5), which compares, under identical reconciliation and inference settings, a model trained on a single fixed hierarchy with models trained on increasingly many and diverse hierarchies, all evaluated on the same hierarchies without averaging.
To further quantify how much of the improvement is retained without averaging, we ran an additional experiment in Tab.8 (App. C.1), where SHR is evaluated without inference-time averaging on two different fixed hierarchies. Results show gains in performance even when using a single completely random hierarchy from the training distribution. We expect even better results to be possible with better heuristics to choose the inference hierarchy (i.e., by choosing the hierarchy that performs best on the validation set). This provides further convincing evidence that performance improvements do not stem from inference-time averaging alone.
We thank the reviewer for pointing this out. We hope these additional results fully address the concern, and we are happy to continue the discussion.
Our scalability experiment (Fig. 6) compares our approach against the one used in popular deep hierarchical forecasting models [1,2]. This involves computing the inverse of a matrix that is typically dense even if the initial matrices are sparse. More efficient implementations exist that exploit the tree structure or the sparsity of the hierarchy [3]. These implementations reduce the cost of projection-based reconciliation, but they still require solving a least-squares problem over the entire hierarchy. Our operator, instead, relies only on local parent-child operations, without any matrix inversions. We revised the discussion in Sec. 5, clarifying that the computational cost we refer to is that of standard dense implementations, and we are looking into adding another baseline in the revision.