- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Retail forecasters that rank well on aggregate error still show structured bias once deployed, and the paper argues the cause is a third error class, distinct from temporal-structural error and from distribution shift: the statistical functional a loss implicitly estimates (MSE the mean, MAE the median, pinball a quantile) does not match label-side distributional pathology. Each series is binned on skewness, within-series CV and zero-inflation ratio, test series are grouped into four regimes (Benign/ZIR/Skew/CV), and the proposed Regime Bias Vector is the norm of the per-regime mass-ratio biases centred on the global bias. Evaluation runs on an introduced benchmark, RetailShiftBench (31,213 daily SKU series of 89 steps, built without trend breaks), plus M5, with XGBoost and TFT backbones, 13 loss objectives, and a three-stage attribution from pooled training to random-label cells to pathology-defined cells. The headline: WMAPE sits in a narrow 0.427–0.518 band while RBV spans 0.078–0.953, and regime-aware training moves XGB-MSE RBV 0.243 to 0.041 while XGB-Huber goes up 240.3%.
Critical strengths
- The three-stage attribution is the right experiment, and the random-label control is the part most papers of this kind skip. Cells matched in count and size but with permuted label semantics are nearly inert on M5 (), which is what separates "granularity helped" from "pathology semantics helped."
- Table 12 decodes a mean and a median from a single forward pass of one frozen Chronos-Bolt checkpoint, global bias / RBV 0.381 versus / RBV 0.789, so capacity, architecture and training are held exactly fixed and the gap is the estimand alone. The one place where a central claim and the data behind it line up.
Critical weaknesses
- RBV, the metric everything rests on, is defined three incompatible ways. §4.2 uses overlapping regimes centred on and asserts an identity that is false under overlap; Eq. (2) centres on a weighted mean; Algorithm 1 demands a mutually exclusive partition and takes an unweighted norm. The implementation is the third, since the unweighted form reproduces Tables 2, 11 and 12 to , so a reader who codes RBV from the body text gets different numbers. The abstract's "intrinsic floor plus training-attributable excess" decomposition then dies on the paper's own p. 29: ql60 pooled RBV 0.11 against a floor of 0.77, an excess of .
- The two main tables are not the same runs, and the paper never says so. Table 3's note defines its "global" column as Table 2. XGBoost matches, TFT does not: TFT-MAE WMAPE is 0.4776 in Table 2 and 0.4475 in Table 3, a gap of 0.0301, more than twice the 1.3% effects reported as findings. The giveaway is the rule heuristic, decomposition-invariant by the same note, whose RBV reads 0.033 in Table 3, precisely Table 2's rule-heuristic bias. §D.1 still claims "paired train/test splits," so the TFT half of the headline result is unverifiable.
- The loss$\to$pathology narrative the paper is built on is denied by Table 2, repeatedly, and there is no statistical apparatus to arbitrate:
- "MSE suffers under ZIR ... MAE performs poorly on Skew ... Huber and Tweedie degrade under CV" fails all three ways. Largest is almost always CV (XGB-MAE: Skew vs CV ), and Huber, the one named CV-fragile, has its smallest bias on CV ( vs ZIR ).
- The introduction's "MSE over-predicts sparse regimes, MAE under-predicts them (Tab. 2)" is false on the whole TFT half: MSE ZIR , Tweedie , Poisson , all under-prediction. §D.1(iii) concedes the sign is preprocessing-dependent. Page 1 states it flat.
- §B.3 claims "all cell-level biases are significant at the 1% level," but Table 5 carries only pooled and RBV intervals and no p-value appears anywhere. The abstract's "3 seeds" meets Table 10's "Single seed (42)," Tables 2 and 3 have no uncertainty at all, and TFT-Tweedie's (absolute 0.008) is reported as a directional finding against seed std of 0.002–0.007.
Questions for authors
- Which definition of RBV produced the reported numbers: §4.2 (overlapping regimes, centred on ), Eq. (2) (centred on the weighted mean ), or Algorithm 1 (single-valued regime map, unweighted )? Relatedly, please either prove or withdraw the §4.2 claim that holds exactly under arbitrary regime overlap.
- Table 2 and Table 3 disagree on the TFT global baseline: RBV 0.338 / 0.953 / 0.486 / 0.908 (Tab. 2) versus 0.335 / 0.914 / 0.459 / 0.873 (Tab. 3), and WMAPE 0.4934 / 0.4776 / 0.4798 / 0.4757 versus 0.4675 / 0.4475 / 0.4603 / 0.4514, despite §D.1's statement that the global and decomposed evaluations use paired splits. Which set is correct, and were the two tables produced by the same runs?
- In Table 3 the rule heuristic's RBV is 0.033 while Table 2 gives 0.027; 0.033 is exactly Table 2's rule-heuristic Bias(All). Was a bias value transcribed into the RBV column, and if so which other cells of Table 3 are affected?
- Given §E.1's own statement that "Pooled training can instead push RBV below the floor (ql60: 0.11 0.77)", in what sense is a floor, and in what sense is the abstract's decomposition into a floor plus a training-attributable excess well defined when that excess is ?
Response to Reviewer bEij
We thank the reviewer for the careful reading and concrete questions. Definition consistency. Algorithm 1 is the single RBV definition and produced every reported number: four overlapping marginal groups, a series counted once per flag, centered once at Bglobal, unweighted ℓ2. Per-regime training uses only the disjoint three-tier cells (27 nominal, 14 used on RSB; 22 on M5). §4.2 states this two-layer structure; the identity suggesting an alternative reading was withdrawn. RBV∗ is an oracle dispersion reference, not a lower bound.
Table 2/3 and the Rule column. The submitted TFT cells mixed two epoch budgets (
max_epoch2 and 10) and an earlier date protocol (training ended 2025-07-30, evaluation began 2025-08-04), so they were not aligned with XGB's fixed-origin split. We audited every cell and regenerated all TFT cells in one batch (max_epoch=50, training labels ending before the 2025-08-03 origin, targets 2025-08-04..08-10, 3 seeds). The corrected run is canonical; old and new are never mixed. The Rule reference is 0.022 (submitted 0.033 was the pooled bias). These corrections preserve the regime-bias pattern, the direction of the decomposition effect for mean-type losses, and the model-class moderation. The one reading sharpened rather than reversed is the TFT feature channel: under the corrected protocol it partially reduces RBV (0.688→0.414) instead of worsening it.Loss–pathology narrative. We withdraw the one-to-one pairing. Under the unified protocol the largest absolute bias is the CV regime in 11 of 12 model–loss cells (the exception is the data-driven Huber threshold, whose largest bias is ZIR; 11/12 count from Tab. 2). The direction is a loss-family fingerprint: median-type and robust losses are negative in all 18 pathological cells; mean-type losses are positive in all nine raw-scale XGB cells and negative in all nine normalized TFT cells (§6.2; App. D.1(ii)). The introduction no longer states the MSE/MAE sparse-regime direction without the protocol qualifier.
Seed variance and significance. All entries follow the 3-seed protocol of the General response, including the controlled comparison, the label-normalization ablation, and both Huber protocols; non-averaged entries are labeled there; TFT feature/control arms are MSE-only. XGB gaps exceed seed dispersion by an order of magnitude: pooled-to-regime MSE 0.226±0.004→0.030±0.002, Tweedie 0.164±0.005→0.024±0.002, MAE 0.683±0.002→0.911±0.003, Huber (data-driven) 0.081±0.002→0.540±0.003 (Tab. 10); the feature arms behave the same way (0.030±0.017, 0.029±0.001, 0.900±0.006, 0.243±0.002). TFT-MSE dispersion is larger (0.688±0.160→0.292±0.103; features mean 0.414, Tab. 9); every TFT contrast has ∣t∣ below the 5% critical value 4.30 (Tab. 11), so all TFT contrasts are reported as directional. The earlier “1% significance” claim is replaced by seed variability.
Tweedie (TFT). The TFT Tweedie/Poisson arms use a non-negative softplus-scaled target (
center=False), so the deviance is defined; what remains is an estimand difference, not an implementation defect, so absolute values are not comparable to XGB's raw-scale Tweedie and only within-protocol contrasts are reported.Huber protocol. The protocols are separated. On RSB, fixed δ=5.68 gives ori 0.206, features 0.026, split 0.035; data-driven δ (holdout q75) gives 0.081, 0.243, 0.540 (Tab. 12, θ^ row; features in App. E). On M5 the same flip reproduces: fixed 0.364→0.070, data-driven 0.245→0.372. The mechanism is δ-estimation fragility under zero-inflated data: a holdout residual quantile is pulled down by abundant easy zero/low-demand samples, pushing Huber toward a median-like estimator and blunting its response to regime information.
Normalization. On RSB-MAE softplus drives bias-RBV to 0.085 while ∣E−0.5∣ worsens to 0.279; linear z-score restores 0.781 with ∣E−0.5∣=0.142 (Tab. 12, T row). Softplus rewrites the effective estimand; the residual after linear restoration is per-series reweighting, not estimand drift.
Directional signatures. The signs hold across seeds and controls but are configuration-dependent: feature conditioning, decomposition and the Huber-δ protocol can reverse them. The sign pattern is the fingerprint; the magnitude is configuration-dependent.