- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes HiRAF, a retrieval-augmented forecasting method that uses a frozen time-series foundation model’s hidden states to retrieve historical segments with useful future continuations. It averages the retrieved continuations after adjusting their level and scale, then blends them with the backbone forecast using a validation-fitted scalar. Across six datasets, HiRAF improves both evaluated backbones, while a bucketed index reduces online retrieval overhead; the central claim is that forecasting-model representations provide more predictively relevant retrieval keys than raw-series similarity or separate embeddings.
Critical strengths
- The design is simple and easy to understand. The method reuses the forecasting model’s hidden states without updating model weights or requiring a separate embedding model in the self-retrieval configuration.
- The authors report consistent forecasting improvements. Both Chronos-Bolt and Timer-S1 improve across all six datasets, with approximately 6–7% reductions in their respective mean MSEs.
- Layer-wise results show increasing alignment with future compatibility in deeper layers, providing empirical support for the proposed retrieval key.
- The paper does a good job on scaling to larger databases for retrieval. It evaluates approximate retrieval against exact search and reports end-to-end timing on a five-million-entry database.
Critical weaknesses
- The reported 7% MSE improvement over TS-RAG compares different backbones, Timer-S1 for HiRAF and Chronos-Bolt for TS-RAG. With the same backbone, the gain is approximately 3.6%, so the headline partly reflects backbone differences.
- I think there are important relevant baselines. The controlled comparison includes instance-normalized raw contexts and one embedding model, but omits strong alternatives such as endpoint-anchored L2 or retrieval using predicted futures.
- No confidence intervals or significance tests are reported.
Questions for authors
- Can you emphasize the same-backbone improvement over TS-RAG and compare runtime with both methods using the same approximate-search implementation?
- I am curious about isolating the effect. Can you separately report individual-neighbor future error, averaged retrieval-forecast error, and complementarity with backbone errors to establish why hidden-state retrieval helps?
- Do the improvements remain statistically reliable under temporal block-bootstrap evaluation, different index seeds, and matched baseline comparisons at longer horizons?
7 comments
W1 — The 7% also involves a backbone change
We accept this. Retrieval-augmented forecasting methods of this kind are a correction applied to a backbone forecast — which may differ from RAG in this respect — so a more appropriate measure is the improvement each method achieves over its own backbone. Taking the three rows of Table 1 that share the Chronos-Bolt backbone:
| ETTh1 | ETTh2 | ETTm1 | ETTm2 | Weather | Electricity | Mean | |
|---|---|---|---|---|---|---|---|
| Chronos-Bolt (backbone) | .3615 | .2516 | .3107 | .1486 | .2006 | .1126 | .2309 |
| TS-RAG | .3556 | .2451 | .2904 | .1465 | .1905 | .1114 | .2232 |
| HiRAF (Chronos-Bolt) | .3450 | .2465 | .2739 | .1405 | .1743 | .1104 | .2151 |
On the same backbone, TS-RAG improves on the backbone by 3.3% and HiRAF by 6.8%, a factor of 2.05×. On the Timer-S1 backbone (Table 2) HiRAF improves on the backbone by 6.2% and TS-RAG's retrieval key, run through the same pipeline, by 2.8%, a factor of 2.25× — the ratio is close on the two backbones, which makes it a more stable measure than a direct comparison of the absolute values. The 3.6% you compute is the same three rows read as HiRAF against TS-RAG (.2232 → .2151); the two readings are consistent.
We will revise the abstract and introduction so that the same-backbone numbers are stated first and the cross-backbone 7% is reported as a secondary remark.
W2 — Stronger baselines: endpoint-anchored L2, and retrieval by predicted futures
We added both. In each case the only thing that changes is the vector handed to the search: same backbone, same retrieval database, same re-alignment and aggregation, same scalar fusion.
(a) Retrieval by the backbone's predicted futures. Six datasets, Timer-S1 self-retrieval, k = 50, three normalisations of the forecast tried and the strongest reported per dataset. The hidden-state key wins on all six, and the best forecast key retains only 13%–56% of HiRAF's gain over the backbone. Working well takes two things: the retrieved forecast has to be accurate on its own, and it has to offer a correction where the backbone is wrong. This key has only the first: it is the backbone's forecast itself, so retrieving with it groups together the windows the backbone predicts alike, and the mean true future of that group is what training already aligns the forecast to, leaving almost no systematic error to remove. On electricity it retrieves a more accurate forecast than ours (.1407 vs .1445) and keeps only 56% of the gain.
(b) Endpoint-anchored L2. Six datasets × two backbones = 12 cells, k = 10, exact search; the hidden-state key is Timer-S1's in both halves.
| retrieval key | mean fused MSE | vs backbone |
|---|---|---|
| backbone (no retrieval) | .2259 | — |
| instance-normalised raw context (the control used in the paper) | .2191 | −3.03% |
| Chronos-T5 embedding (TS-RAG's key) | .2178 | −3.60% |
| endpoint-anchored L2 | .2157 | −4.51% |
| hidden state (HiRAF) | .2141 | −5.22% |
Endpoint-anchored L2 is one common form of prediction-oriented retrieval: it matches on where the context ends rather than on the shape of the context as a whole. It is a strong baseline, which agrees with our hypothesis that a retrieval key has to carry what is relevant to the continuation. It also beats the separate embedding model TS-RAG retrieves with (−4.51% against −3.60%), although that embedding comes from a pretrained time-series model and this one is a hand-written distance. But the continuation is not always governed by proximity to the end of the context, and where it is not, the hidden-state key is the better one, since the backbone has already computed whatever is predictively relevant for that series. Overall the hidden-state key is ahead: −5.22% against −4.51%. We will add endpoint-anchored L2 to the key comparison in the revision.
W3 — No confidence intervals or significance tests
You are right, and we have replaced the procedure. The test in the submission resampled (channel, window) rows independently, but test windows slide with stride 1, so adjacent windows share L + H − 1 = 575 steps; against a moving-block bootstrap this understated the width of the confidence interval by a factor of 5–12, and a Newey–West HAC estimator (bandwidth 576) agrees with the block bootstrap. The revision reports moving-block bootstrap intervals (10,000 resamples), with a channel-clustered variant and Politis–Romano subsampling as block-length-independent cross-checks. Intervals are reported at the widest block length b satisfying b/n ≤ 0.1, since beyond that ratio each resample covers nearly the whole series and the interval narrows spuriously — in the direction that would favour us.
HiRAF against its own backbone, H = 64, all six datasets and both backbones:
| dataset | backbone | HiRAF | backbone | ΔMSE% | 95% CI of the difference | p |
|---|---|---|---|---|---|---|
| ETTh1 | Timer-S1 | .3172 | .3242 | −2.17% | [−1.12e−2, −2.38e−3] | 0.0036 |
| ETTh1 | Chronos-Bolt | .3441 | .3615 | −4.82% | [−2.44e−2, −9.81e−3] | <1e−4 |
| ETTh2 | Timer-S1 | .2410 | .2483 | −2.94% | [−1.18e−2, −3.66e−3] | <1e−4 |
| ETTh2 | Chronos-Bolt | .2449 | .2516 | −2.65% | [−1.23e−2, −2.61e−3] | <1e−4 |
| ETTm1 | Timer-S1 | .2646 | .2948 | −10.27% | [−4.12e−2, −2.19e−2] | <1e−4 |
| ETTm1 | Chronos-Bolt | .2794 | .3107 | −10.08% | [−4.53e−2, −2.08e−2] | <1e−4 |
| ETTm2 | Timer-S1 | .1505 | .1609 | −6.44% | [−1.42e−2, −7.22e−3] | <1e−4 |
| ETTm2 | Chronos-Bolt | .1436 | .1486 | −3.35% | [−6.62e−3, −3.69e−3] | <1e−4 |
| weather | Timer-S1 | .1749 | .1799 | −2.82% | [−8.25e−3, −1.79e−3] | 0.0016 |
| weather | Chronos-Bolt | .1860 | .2006 | −7.29% | [−2.01e−2, −8.75e−3] | <1e−4 |
| electricity | Timer-S1 | .1136 | .1170 | −2.95% | [−4.23e−3, −2.76e−3] | <1e−4 |
| electricity | Chronos-Bolt | .1098 | .1126 | −2.52% | [−3.78e−3, −2.12e−3] | <1e−4 |
All 12 cells are significant.
We thank you for these valuable comments. We have run the additional experiments you asked for, and respond to each point below. We hope these resolve your concerns, and we look forward to your reply; if you have any further questions, we will answer them promptly.