- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies the problem of reconciling predictions with hierarchical dependencies. For example, total sales in a region must be the sum of the sales for each store in the region, but predicting all of them individually will almost certainly violate that assumption. The paper claims that a baseline solution does not have calibrated prediction intervals due to (co)variance estimation error. The paper develops an alternative method, and contains experiments on many datasets, where the paper's method outperforms baselines on several metrics.
Critical strengths
None
Critical weaknesses
The paper is extremely hard to understand. Many critical concepts are not explained when they are first used, and some are not explained at all. Examples include:
- Theorem 1: none of the left-hand-side quantities are defined or explained.
- Corollary 1: and are not defined or explained.
- The paper claims that separating the "predictive sample" from the "point forecast" is an important part of the method, but does not explain why this is useful.
- Section 3.5: it is not clear how the point forecaster is constructed. The text presents four options (but Table 1 has six?) at a high level, but the details later in the section are not connected to any of the options.
- Eq. 4: is not explained.
- The terms "anchor" and "centering" are used repeatedly but not explained.
- Line 44: W is not defined.
- Line 46: "node wise variance ratio" is not explained.
- Line 55: H and S not defined.
These make it impossible to learn how the method works and judge what is actually proved in the theorems.
The experiments also have several issues:
- One of the stated motivations is that MinT prediction intervals do not always have nominal coverage. However, the experiments only report coverage in an ablation (Table 3) but not in the main results of Table 2. The reported coverage is also relative to MinT, while the absolute coverage of MinT is not reported. As a result, the claim that negative relative coverage is closer to nominal could be false.
- The metrics listed in Section 4 do not fully match those reported in Table 2. The metrics should also be cited.
- Section 4 claims 72 methods are evaluated, but Table 2 only contains 12. The text claims the method list is in the Appendix A.2, but that does not contain 72 methods.
Minor points:
- Figure 1 text is too small.
- Line 207: the citation makes no sense. The text talks about "information patterns", but cites the original transformer paper.
Questions for authors
No questions.
Thank you for the review and for the specific list. Most items are symbols used before they are defined, which we will fix. The paper is densely packed and assumes some familiarity with hierarchical forecasting, and we're sorry that made it hard to follow.
Definitions.
Why separate the samples from the point forecast.
In practice (say sales/demand forecasting), planners use a point forecast for replenishment and quantiles for safety stock, and both have to add up.
Sec. 3.5 and Table 1. There are four information patterns: a per-leaf MLP, attention among leaves (LeafAttn), attention among all nodes in three masking variants (FullAttn, TreeAttn, LeafTreeAttn), and hierarchy routing (RouteAttn). That gives the six rows. Every head outputs a leaf correction δ, and the point forecast is S(b^+δ). The text after Table 1 details RouteAttn, the head used in Table 2. Each aggregate's discrepancy ga (its direct forecast minus the anchor sum) is split among its descendant leaves by softmax weights that sum to one (Eqs. 14-15). We will add a one-line summary per head.
Coverage. Fair point. Absolute 90% coverage, averaged over the 29 datasets:
CovGap in Table 3 is ∣coverage−0.90∣ per dataset and origin, so a negative difference from MinT means closer to 90%, from above or below. The table also shows a limitation plainly. Every residual-based method under-covers, ours included, because errors on the training data understate errors on new data. MinT's Total coverage is below 0.80 on 28 of 29 datasets, and CoFlow narrows the gap without closing it. Coverage alone isn't the target either: HINT covers best but has 1.7 times MinT's CRPS. This table goes into the main text.
Metrics and the 72 methods. Table 2 shows the four metrics. MASE, coverage gap and energy score are in Tables 3, 4 and 15. We will gather all seven in one table with citations (Gneiting & Raftery 2007 for CRPS, interval and energy scores, Hyndman & Koehler 2006 for MASE, Davydenko & Fildes 2013 for relMAE). The 72 methods are 13 baselines (8 standard reconcilers including the unreconciled base, 3 probabilistic MinT variants, 2 HINT configurations) and 59 CoFlow variants, which make up the factor ablations. Table 2 shows ten baselines plus the two final compositions, and the other three are in Table 14. A.2 doesn't list all 72, so we will add the full list.
Minor. We will enlarge the text in Figure 1. The Vaswani et al. citation belongs where attention is introduced, not on "information patterns", and we will move it.