- Rating
- 2(+1)
- Confidence
- 3(+1)
- Informativeness
- 2(+1)
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper studies performance degradation in spatiotemporal kriging reconstruction under level shifts. It analyzes the link between shifts and model failure, especially when neural models underperform simple historical baselines, and proposes KRIN, which improves robustness by removing and restoring each node’s recent mean. Experiments across models and datasets, including matched comparisons of observation-node locations, show reduced prediction errors for most models.
Critical strengths
- The evaluation is systematic and conclusions are cautious. It compares models under normal and level-shift conditions, distinguishes simplified architectures from public implementations, examines observation-node locations through matched comparisons, and reports cases where KRIN fails to beat strong baselines. This strengthens the findings’ credibility.
Critical weaknesses
- Limited novelty. KRIN removes each node’s historical-window mean and restores it after prediction, closely resembling RevIN [1], which removes and restores instance statistics, and NLinear [2], which subtracts and restores the last observed value.
- Gains over existing centering methods appear limited. Table 23 suggests that most improvements stem from generic level centering rather than KRIN’s specific recent-window mean design.
[1] Kim, T., Kim, J., Tae, Y., Park, C., Choi, J.-H., & Choo, J. Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift. ICLR 2022.
[2] Zeng, A., Chen, M., Zhang, L., & Xu, Q. Are Transformers Effective for Time Series Forecasting? AAAI 2023, 37(9), 11121–11128.
Questions for authors
- Table 4 shows that KRIN nearly eliminates the additional baseline-normalized outside-observation penalty caused by local shifts. However, this may follow from its equivariance and synthetic shift construction. How much of this result is an empirical finding beyond the mathematical structure?
- Why does KRIN substantially outperform SNaive-7+CM on some simplified architectures, while the corresponding public implementations still underperform this baseline?
3 comments
I thank the authors for their patient and detailed responses. I learned several new things from them, and they have addressed my questions. However, the paper still has some shortcomings, and I remain concerned about its readability, as I still find parts of it difficult to follow. Nevertheless, in light of the authors’ clarifications, I have decided to raise my score.
Thank you for your careful reading and thoughtful feedback, and for raising your score in light of our clarifications. We are glad that our responses addressed your questions.
We take your remaining concern about readability seriously and would like to use the rest of the discussion period to act on it. If you are able to spare a moment, we would be very grateful if you could point us to one or two sections or passages that you still found difficult to follow. Even a brief pointer would help us direct the revision to where it matters most for readers.
We fully understand if time does not permit, and we will continue improving the paper's organisation and clarity in any case. Thank you again for your generous engagement with our work.
e thank the reviewer for these questions.
W1. Contribution beyond RevIN and NLinear.
KRIN removes and restores a historical mean without window scaling or a learned affine; it adds no new operator. Our contribution concerns failures against admissible baselines and level sources. A statistic using current values must come from observed nodes. Table 66 compares sources on the same backbones without added parameters. In Chicago's settled local-shift test, SPIN-s with an observed-node level matches KRIN inside the shifted region (29.7 vs 30.1 MAE), but outside has 4.9 times KRIN's MAE (149.8 vs 30.5), above the uncentred model (102.8). The outside ratio is also 4.9 in Subway (627.5 vs 127.2).
W2. Does the gain come mainly from centring?
Yes (§5.1). The former “full RevIN” row used unbiased standard deviation plus 1e-5 without a learned affine. It is relabelled and RevIN as published is added (Tables 23, 62–63). Across SPIN-s and IGNNK-R, with 3 training seeds × 10 mask seeds per cell, paired intervals favour the mean in 6/16 comparisons and RevIN in 5; 4 include one. In IGNNK-R Airports 2020–21, 5/30 evaluations from two training seeds meet the extreme criterion: non-finite MAE or MAE above three times the KRIN median. All five are finite; excluding them gives RevIN MAE 8.86 versus 8.58 for the mean. Maximum shortfalls (mean / RevIN) are 11.1% / 11.4% on SPIN-s and 7.0% / 10.2% on IGNNK-R excluding that cell; last-value removal gives 15.2% on SPIN-s and 17.6% on IGNNK-R. These results establish neither equivalence nor overall superiority; the mean is a parameter-free default.
Q1. What does Table 4 show beyond mathematical cancellation?
The near-zero centred change is expected: the shift starts 14 days before evaluation, so the full history shares it. Exact cancellation holds for additive offsets in preprocessed space; raw-count multiplication differs slightly after log1p. We treat this as an implementation check (§7.2). We also measure:
For onset, we fix the normalisation reference at α times the no-shift SNaive-7 MAE rather than the contemporaneous SNaive-7 error used in Table 4. During transition, the weekly reference still includes pre-shift observations. The fixed reference prevents that baseline's transition error from rescaling the placement contrast. R is the reduction in excess penalty, not absolute MAE recovery (means, 95% crossed-bootstrap intervals):
Mean R is negative in all four pairs in days 1–7, positive for IGNNK-R and near zero for SPIN-s in days 8–14, and positive with intervals above zero in all four in days 15–28. Original checkpoints were not retained. Retraining with the same code, settings and seeds reproduces Table 4 for IGNNK-R to numerical precision, but not SPIN-s exactly; both original SPIN-s estimates fall within the rerun intervals. Negative R does not imply higher IGNNK-R MAE: Chicago error falls by 37% inside and 12% outside, widening the gap. SPIN-s outside instead has higher first-week centred MAE (439.2 vs 370.4; Table 66).
Q2. Why does SPIN-s outperform SPIN-R against +CM?
The cause remains unresolved. Only SPIN-s and SATCN-s (2/6 centred simplified models) beat +CM on Chicago/Subway Shock at r = 0.5, versus 6/6 on Bikeshare and 5/6 on Airports (Table 16).
Both share inputs, preprocessing, KRIN, masks, checkpoint selection and final-target masked MSE. SPIN-s uses a GRU and attention across observed nodes (hidden 64, batch 32); SPIN-R uses four sparse spatiotemporal attention layers and learned node embeddings (hidden 32, batch 4). Both omit original SPIN's layer-wise and whole-window losses. These controls do not isolate an architectural cause (Tables 25, 69; §7.5).