acceptodds
Under review as a conference paper at ICLR 2027

Rethinking RAG for Time Series: Does Similarity Really Lead to Predictive Utility?

Abstract

Historical patterns beyond the input context can provide useful information for time-series forecasting. Retrieval-augmented forecasting seeks to exploit such information by retrieving historical segment candidates whose contexts are similar to the query context. This raises a fundamental question: does similarity reliably indicate the predictive utility of a historical candidate? We examine it from three perspectives: (1) how prevalent useful historical candidates are, (2) whether similarity-based retrieval can identify them, and (3) whether their predictive utility remains consistent across augmentation interfaces. Across 24 experimental settings, our empirical results reveal several important findings. Useful candidates are common: 38.32–56.23% of candidates improve the forecast, with the best candidates yielding gains of 24.56–71.01%. Surprisingly, similarity-based retrieval underperforms random sampling in 83.33% of retrieval comparisons. Predictive utility is also strongly interface-dependent: in 55.56% of cross-interface transfers, historical candidates selected as optimal under the source interface yield negative gains under the target interface. Our findings suggest that similarity alone is insufficient for retrieval, and future retrieval methods should better account for which historical candidates are actually useful under the chosen augmentation interface.

Primary area:
learning on time series and dynamical systems
Keywords:
time series forecasting, retrieval-augmented forecasting, time series foundation models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Y9psOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether time series similarity based retrieval is a useful metric for time series retrieval. The paper has two conclusions: (i) although similarity-based retrieval is useful and benefits the downstream model, there are cases where random retrieval is equally beneficial. (ii) the retrieval should be domain-aware in that the utility function of retrieving certain series in a domain should depend on the domain itself. While getting to that conclusion, the paper establishes a useful analysis framework. My main criticism with respect to this is that benchmark (LTSF benchmark) set is limited in diversity and the paper would benefit greatly from a more encompassing benchmark set.

Critical strengths
  1. The paper investigates the utility of similarity-based selection, which is quite important in time where there are lots of retrieval methods in time series forecasting.
  2. Overall, I find the analytical framework in this paper valuable.
Critical weaknesses
  1. The conclusion of this paper depends on LTSF datasets, which are quite limited in diversity. Especially, exchange dataset is known to be near random walk, and hence it is obvious to me that random selection will yield competitive if not better results. Similarly, ETT datasets are known to be highly periodic.
Questions for authors

I suggest authors add Gift-eval or Monash datasets in the analysis. In the current form, experiments are my main concern and addressing this issue will reflect in a change in my assessment.

1 comment
AuthorsOpenReview ↗

Response to the Review [1/1]

Thank you for asking us to test across a wider range of domains and series characteristics.

1. Do RQ1–RQ3 hold on broader benchmarks?

We evaluated 43 dataset entries across GIFT-Eval (24 datasets, with 209 sampled time series) and Monash (19 datasets, with 100 sampled time series). The two benchmarks cover a wide range of domains and sampling frequencies (see Tables 14–17 of Appendix E, pp. 20–25). Specifically,

  • RQ1. Useful historical candidates remained common across both benchmarks. Oracle@1000 reduced MSE in 127/129 dataset–interface settings (71/72 on GIFT-Eval and 56/57 on Monash). The median proportions of useful candidates were 47.2% and 44.3%, respectively.
  • RQ2. Similarity-based retrieval remained weakly correlated with predictive utility. Same-pool Random@10 exceeded the Best@10 of the best-in-hindsight retriever (r∗r^*) in 61/129 settings (36/72 on GIFT-Eval and 25/57 on Monash). The median retrieval–utility Spearman correlations were +0.006 and +0.017 on GIFT-Eval and Monash, respectively, with an overall median of +0.011 across all 43 datasets. The median oracle gaps were 15.97 and 19.03 percentage points on GIFT-Eval and Monash, respectively.
  • RQ3. Predictive utility remained interface-dependent. Useful-set Jaccard overlap (JJ) was within three percentage points of its independence reference in all 129/129 interface-pair comparisons. The median utility-rank Spearman correlations (ρU\rho_U) were +0.036 on GIFT-Eval and +0.016 on Monash. Furthermore, source-oracle transfers resulted in negative gains in 170/258 directional comparisons (91/144 on GIFT-Eval and 79/114 on Monash).

Overall, these results demonstrate that the three main findings persist across datasets covering a substantially wider range of domains and sampling frequencies. The weak correlation between similarity and predictive utility, together with the interface dependence of candidate utility, is therefore not limited to the periodic ETT datasets or the Exchange dataset in our original evaluation.

Reviewer_wrJoOpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates a fundamental question in retrieval-augmented time series forecasting: Does similarity between historical analogs and the current input translate into improved predictions? Its objective is to examine the design assumptions underlying existing methods and provide empirical insights. The authors define candidate-level “predictive utility” and investigate three questions: availability, retrievability, and interface dependence. Through experiments, the paper examines these questions and presents its findings. However, the current version does not provide sufficiently robust evidence to support its conclusions, given the limited experimental settings and datasets.

Critical strengths

The research question is clear and has community relevance. Retrieval-augmented forecasting often uses similarity as a proxy metric for correlation. Direct anlysis can help explain why retrieving similar history does not improve the prediction.

Critical weaknesses
  1. The effectiveness of similarity-based retrieval has already been discussed in prior work, such as SARAF, and the conclusions offer limited additional insight to the community. The candidate-level diagnostics and cross-interface comparisons are potentially valuable, but the paper offers limited insight into when similarity is reliable, why it fails, or which conditions determine transferability across interfaces.

  2. The main experiments use a single configuration of context length, forecasting horizon, candidate-pool size, and limited datasets. The conclusions may not generalize to other configurations or datasets.

Questions for authors
  1. Can controlled experiments varying stationarity and noise clarify when similarity reliably predicts utility? Synthetic data could complement the real-world evaluation.
  2. Are the main conclusions robust to changes in context length, forecasting horizon, candidate-pool size, and retrieval budget? Do they also hold on datasets beyond the six evaluated datasets?
  3. What are the architecture, weight sources, and training procedures of the Mid interfaces?
  4. Only one backbone, Chronos, is used for the main experiments, while Moirai is used only in RQ1 on ETTh1. Could author provide other backbones to show the generality of the conclusions?
2 comments
AuthorsOpenReview ↗

Response to the Review [1/2]

Thank you for your questions about our diagnostics and when they apply.

1. When similarity reliably predicts utility (Question 1 and Weakness 1).

We varied stationarity and noise using widely used perturbations, including Gaussian white noise, AR(1) noise, sparse spikes, and frequency drift, to examine how similarity predicts utility. The results show that the correlation weakens as the level of white noise increases and varies across perturbations and interfaces (see Table 30 of Appendix I, p. 34). Specifically, we applied one perturbation at a time to a sinusoid with a linear trend, while keeping the query forecast origin and 128 candidate start points fixed. Raw-Cos tracked utility for Cov and Pre under clean or light-noise conditions. The table reports Spearman ρ between Raw-Cos scores and candidate-level predictive utility.

Synthetic conditionCov ρ\rhoPre ρ\rho
Clean0.8400.775
Gaussian white noise, σ=0.01\sigma = 0.010.8600.765
Gaussian white noise, σ=0.30\sigma = 0.300.1460.447
Gaussian white noise, σ=0.60\sigma = 0.60−0.0430.245
AR(1) noise, σ=0.60\sigma = 0.600.4790.884
Sparse spikes, σ=0.30\sigma = 0.300.711−0.406
Frequency drift, r=0.15r = 0.150.170−0.424

For Pre, spikes and drift reversed the relationship at the tested levels. The relationship remained strong under AR(1) noise at σ=0.60\sigma = 0.60. With Chronos-2 held fixed on the clean series, Cov/Pre had a utility-rank Spearman ρU\rho_U of +0.718, and source-oracle transfers increased MSE in both directions for Cov/Mid and Pre/Mid. For Cov/Pre, transfers reduced MSE in both directions under sparse spikes (σ=0.05\sigma = 0.05), but increased MSE in both directions under strong white noise (σ=0.60\sigma = 0.60) and frequency drift (r=0.15,ρU=−0.800r = 0.15, \rho_U = −0.800) (see Table 31 of Appendix I, pp. 34–36).


2. Robustness to various configurations and datasets (Question 2 and Weakness 2).

We added experiments covering all the configurations and datasets requested by the reviewer, and the results demonstrate that the main conclusions are robust (see Tables 22–23 of Appendices G.1–G.2, pp. 27–30). Specifically,

  • Context length, forecasting horizon, candidate-pool size, and retrieval budget. On ETTh1 and Weather, we varied context length (256/512/1,024), forecasting horizon (32/64/128), and pool size (1,000/2,000/3,000) in separate sweeps.
    • Across 30 context/horizon settings, the maximum ρ\rho for the Best@10-selected retrievers was +0.025, and the maximum ρU\rho_U was +0.116. JJ ranged from 20.41% to 37.70% and was at or below its independence reference in 23/30 comparisons.
    • In the pool-size sweep, the maximum ρ\rho for the Best@10-selected retrievers and the maximum ρU\rho_U were +0.024 and +0.105, respectively, and JJ changed by at most about 0.15 percentage points per matched pair.
    • Across retrieval budgets K∈{1,5,10,20,50}K \in \{1, 5, 10, 20, 50\} on the six original datasets, the full-pool ρ\rho for the Best@KK-selected retrievers peaked at +0.076 (see Table 24 of Appendix G.3, p. 30).
  • Additional 43 datasets. Across 24 GIFT-Eval datasets and 19 Monash datasets, Oracle@1000 reduced MSE in 127/129 settings. The median and maximum ρ\rho values for the Best@10-selected retrievers were +0.011 and +0.366, respectively, while the corresponding values for ρU\rho_U were +0.023 and +0.168. Across the tested configurations and benchmarks, similarity remained an imperfect guide to predictive utility, and utility rankings showed limited agreement across augmentation interfaces (see Tables 14–17 of Appendix E, pp. 20–25).

3. Mid architecture and training (Question 3).

Bolt-Mid uses the released Chronos-Bolt weights and TS-RAG Adaptive Retrieval Mixer (ARM) weights. C2-Mid applies a shared ARM to each of the four Chronos-2 forecast tokens. With the backbone and forecasting head frozen, we trained the ARM on the released TS-RAG pairs for 10,000 AdamW updates (batch size 256, learning rate 3×10−43 \times 10^{-4}). Moirai applies an ARM to each forecast patch and trains the mixer using its native likelihood loss. TimesFM-3 and Toto-2.0 each use a separately trained ARM with the forecasting weights frozen (see Table 21 of Appendix F, p. 26).

AuthorsOpenReview ↗

Response to the Review [2/2]

4. Other backbones (Question 4).

We evaluated three additional backbones, namely Moirai, TimesFM-3, and Toto-2.0, on ETTh1 and Weather. All 18 settings showed positive Oracle@1000 gains, demonstrating the generality of the conclusions (see Tables 18–20 of Appendix F, pp. 26–27). Specifically,

  • RQ2. The largest ρ for the Best@10-selected retriever was +0.155, and same-pool Random@10 exceeded its Best@10 in 11/18 settings.
  • RQ3. The maximum ρ_U was +0.153, and J was at or below its independence reference in 13/18 interface pairs.

The main RQ2 and RQ3 findings remained consistent across the additional backbones.


5. Contribution beyond SARAF (Weakness 1).

Thank you for the comment. We believe this work makes valuable contributions beyond existing studies (e.g., SARAF).

  • We ask how a historical candidate affects forecast error, whereas SARAF asks whether similar historical contexts have similar continuations. We define candidate-level predictive utility as the reduction in forecast error achieved by supplying a candidate through a specified augmentation interface, relative to the same forecaster without retrieval. This complements SARAF’s motivating analysis of whether similar historical contexts have similar continuations.

  • Our work highlights this issue through comprehensive experiments, and the results demonstrate that it cannot be ignored. Although the community has recognized that similarity does not necessarily translate into improved forecasting, this issue has often been overlooked in existing studies. To our knowledge, no prior study has directly compared similarity-based retrieval methods with random retrieval. Our work addresses this gap through comprehensive experiments and raises this important question, providing a direction for future research.

Reviewer_owUXOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper analyzes whether similar historical patches are indeed helpful in retrieval-augmented time series forecasting. It reports three findings: 1) useful candidates are common, 2) similarity-based retrieval underperforms random sampling, and 3) predictive utility is strongly interface-dependent.

Critical strengths
  1. The research questions are meaningful and make sense. The paper challenges a common implicit assumption in retrieval-augmented time series forecasting, “a candidate is useful if it is similar”, which has not been directly examined before. Given the growing volume of work in this direction, this kind of verification is necessary.

  2. The experiments are well controlled. The paper fixes the query set and the candidate pool, and compares backbones, interfaces and retrievers under the same setting.

  3. The conclusion of RQ3 is surprising. It shows that different augmentation interfaces lead to different “usefulness” of the same candidate. This challenges another common implicit assumption, that a patch useful in one way will also be useful in another.

Critical weaknesses
  1. The candidate pool consists of 1000 randomly sampled patches. Considering the dataset lengths, this covers roughly 20% of Exchange, 12% of ETTh and under 5% of ETTm. This is important because the results depend on the proportion of “useful” patches in the pool, especially in the comparison with random sampling from the same pool.

  2. The experimental setting differs from actual retrieval-augmented time series forecasting. Existing works usually aggregate the top-k candidates and use them together, whereas this work uses only a single candidate. This is understandable for isolating the effect, but a verification under the top-k setting is also needed.

  3. The claims are too general and too strong compared to the scope the experiments actually cover. The abstract and introduction discuss retrieval methods for time series in general, but the experiments are limited to frozen TSFMs. For methods that learn how to use the given historical candidates, such as RAFT or TimeRAF, the model itself is optimized under similarity-based retrieval, which may lead to higher utility for more similar patches. The paper MUST either restrict its claims or extend the experiments with supporting evidence. This is the main weakness that makes me lean toward rejection.

Questions for authors
  1. Will be the conclusion different if the pool size increase?

  2. Will be the conclusion different if the experiment uses top-k candidates instead of a single candidate?

2 comments
AuthorsOpenReview ↗

Response to the Review [1/2]

Thank you for highlighting candidate-pool coverage, joint Top-K forecasting, and the need to test learned systems.

1. Candidate-pool size (Question 1 and Weakness 1).

On ETTh1 and Weather, we fixed 1,000 test queries per variable and used three seeds to construct pools of 1,000, 2,000, and 3,000 historical candidates. Experimental results demonstrate that the conclusions of all three RQs persisted as the pool size increased (see Table 22 of Appendix G.1, pp. 27–28). Specifically:

  • RQ1. The useful-candidate proportion remained between 35.14% and 53.05%, varying by at most about 0.15 percentage points within a matched setting.
  • RQ2. Same-pool Random@10 exceeded the highest Best@10 among the six retrievers in 50/54 comparisons. For the retriever with the highest Best@10 in each setting, retrieval–utility Spearman ρ\rho ranged from −0.019 to +0.024.
  • RQ3. Within a matched interface pair, the Jaccard overlap JJ between positive-utility candidate sets changed by at most about 0.15 percentage points, and the utility-rank Spearman ρU\rho_U remained low (0.005–0.105).

2. Joint Top-K forecasting (Question 2 and Weakness 2).

Our newly added experiment demonstrates that retrieval similarity and predictive utility remained weakly correlated in joint forecasts. On the full ETTh1 test set, we formed groups of 10 from the highest-ranked training histories according to Raw-Cos and used each group for one joint forecast. The mean within-query Spearman ρ\rho between group similarity and predictive utility was −0.0368, −0.0385, and +0.0205 for C2-Cov, C2-Pre, and C2-Mid, respectively, and +0.0144 for Bolt-Mid (see Table 25 of Appendix H.1, p. 30).

The detailed joint-forecasting results are summarized below.

SystemNo retrievalGroup 1Best group (rank)Group 100ρ\rho
C2-Cov0.37710.46390.3706 (89)0.3726−0.0368
C2-Pre0.37710.38950.3504 (99)0.3508−0.0385
C2-Mid0.37710.37290.3713 (5)0.3729+0.0205
Bolt-Mid0.36150.35610.3551 (5)0.3562+0.0144

All values are MSE. Group 1 contains the ten highest-ranked candidates, while Group 100 contains candidates ranked 991–1,000. Best group is selected in hindsight from all 100 groups.

AuthorsOpenReview ↗

Response to the Review [2/2]

3. Learned systems and scope (Weakness 3).

Although RAFT and TimeRAF can benefit from learning how to use historical candidates, our newly added experiments demonstrate that similarity is still weakly correlated with utility (see Appendices H.2–H.4, pp. 30–33). Specifically,

  • For RAFT:
    • Pearson retrieval improved average MSE despite weak within-query correlation with utility. On the full ETTh1 test split, Pearson Top-20 per scale reduced MSE from 0.3836 under random sampling to 0.3693. Despite this average gain, the mean within-query Spearman ρ\rho between group similarity and group utility was only +0.150 (see Table 26 of Appendix H.2, pp. 31–32).
    • Pearson retrieval’s advantage varied with the training and weighting setups. When both checkpoints used random sampling at test time, the variant trained on random candidates with equal weights achieved lower MSE than the original Pearson-weighted model (0.3724 vs. 0.3836). The same variant also favored random selection over Pearson Top-20 when its checkpoint and weights were held fixed (0.3724 vs. 0.3745) (see Table 27 of Appendix H.3, p. 32).
    • RAFT’s retrieval scores offered limited gains with frozen Chronos-2. To assess the retrieval scores separately from RAFT’s forecaster, we used them to rank candidates for Chronos-2 on the six original datasets. In this comparison, same-pool Random@10 exceeded the Best@10 of RAFT’s best scale in 14/18 settings, with retrieval–utility ρ\rho ranging from −0.010 to +0.048 (see Table 29 of Appendix H.4, p. 33).
    • Hindsight selection revealed room for improvement with the same forecaster. With the original RAFT checkpoint fixed, selecting the best of 100 joint groups in hindsight reduced MSE to 0.3513, compared with 0.3693 under native Pearson retrieval (see Table 28 of Appendix H.3, p. 33).
  • For TimeRAF:
    • The learned retrieval scores were nearly uncorrelated with group utility. In our local ETTh1 reproduction, the mean within-query Spearman ρ\rho was −0.0012. In the same experiment, learned retrieval and random sampling also produced nearly identical MSE values (0.3658 vs. 0.3657) (see Table 26 of Appendix H.2, pp. 31–32).
    • The retrained TimeRAF scorer also offered limited gains with frozen Chronos-2. When we used its scores to rank candidates for Chronos-2, same-pool Random@10 exceeded the Best@10 in 17/18 settings (see Table 29 of Appendix H.4, p. 33).

The joint-group forecasting results provide an additional comparison between native retrieval and random sampling.

SystemNo retrievalNative retrievalRandom jointGroup MSE spanρ\rho
RAFT0.38250.36930.38360.0133+0.1502
TimeRAF0.36400.36580.36570.0001−0.0012

Group MSE span is the maximum minus minimum MSE across the 100 joint groups.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.