acceptodds
Under review as a conference paper at ICLR 2027

Which Histories Matter for Time Series Forecasting? Learning Predictive Relevance with Future Supervision

Abstract

Retrieval-augmented time-series forecasting typically selects historical examples by similarity between observed pasts, although similar pasts can evolve differently. We propose Predictive Relevance Retrieval (PRR), which uses realized future compatibility as privileged supervision to learn a retrieval function that remains strictly past-only at inference. PRR combines Pearson retrieval with a futuresupervised predictive representation to expand candidate support, then reranks the union using statistical and learned pair relations. Across six datasets and four long horizons, PRR improves Pearson retrieval in 23 of 24 conditions, reducing AnalogFutureMSE by 26.7% on average. A candidate-budget-matched variant, PRR-B100, retains nearly the same retrieval improvement while using at most 100 candidates at inference, showing that the gain is not explained simply by a larger candidate pool. We then connect the retriever to five frozen forecasting backbones using validation-calibrated trust. Downstream effects are heterogeneous, and calibration primarily reduces harmful retrieval use rather than making improved retrieval universally beneficial. These results show that predictive relevance and forecast utility are empirically distinct objectives.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series Forecasting, Retrieval-Augmented Forecasting, Predictive Relevance, Future Supervision, Historical Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_T6PJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Predictive Relevance Retrieval (PRR), which uses historical future outcomes as training supervision to learn which past time-series segments are useful for forecasting, while relying only on observed pasts at inference. PRR combines similarity-based and learned retrieval with a reranker, improving retrieval quality over Pearson correlation across six datasets. However, these improvements yield modest, dataset-dependent forecasting gains, highlighting that better historical analogs do not necessarily improve a strong forecasting model. Some other minor concerns I have regard the lack of multiple seeds, and the positioning of the theoretical section. I understand it is required to introduce the setup and the notations,

Critical strengths
  • I think the research question is clear. The authors are trying to distinguish similarity between historical pasts from compatibility of their future trajectories.
  • Empirical results show consistent gains over Pearson with retrieval helping in 23/24 settings, with a 26.7% average reduction in analog future MSE.
  • The paper shows that improved individual analog quality does not necessarily translate into better forecasting, which I believe is an important finding for future retrieval-based forecasting work.
  • The authors have run some ablations and report negative findings as well for transparency.
  • Codebase is provided.
Critical weaknesses
  • The main weakness is there is a limited advantage over a simple baseline. Training-free anchored L2 beats PRR in 17/24 retrieval settings and achieves greater average downstream improvement. This undermines the importance of the method.
  • The reported gains are relatively small. Mean forecasting improvement is 1.41%, with a 0.46% median and degradation in 26/120 settings.
  • Full PRR uses a single seed, and headline results lack confidence intervals. The multi-seed evidence mainly concerns PRR-Stat.
  • I am unsure about the role of the theoretical section because the theory provides standard conditional-expectation characterizations without guarantees for the implemented learning objective.
Questions for authors
  • First of all, given that training-free anchored L2 outperforms PRR in 17/24 retrieval settings and achieves larger average downstream gains, when does PRR justify its additional training and inference cost?
  • Can you isolate the benefits of learned candidate generation and embedding-based reranking by evaluating each independently under fixed candidate support and comparable model capacity?
  • How stable are the full-PRR results across training seeds? I believe a statistical significance analysis can make the results more reliable.
  • Can you quantify whether the gap between retrieval quality and forecasting gains arises primarily from correlated neighbor errors, lack of complementarity with the direct forecaster, or something else?
  • Finally, do the Candidate Prior and Shuffled Future findings extend to full PRR at the main forecasting horizons?
Reviewer_E1zoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the candidate selection problem in retrieval-augmented time series forecasting, arguing that standard past-window Pearson correlation does not necessarily retrieve historical windows whose future trajectories match the query's future evolution. The authors propose Predictive Relevance Retrieval (PRR), which uses endpoint-anchored future trajectory MSE (d(q,i) = 1/H‖yq - yi‖22) on historical training data as privileged supervision while keeping test-time retrieval strictly past-only. Specifically, PRR trains a dual 1D-CNN encoder via KL divergence against a future-similarity teacher distribution, forms a candidate pool by unioning the Top-100 Pearson matches and Top-100 learned-embedding matches (CPRR(q) = C_MP(q) ∪ C_Memb(q)), and trains a two-layer residual MLP reranker over 7 handcrafted past statistics plus embedding differences using a listwise cross-entropy and pairwise softplus loss. The authors also decompose population relevance into a query-independent candidate prior G*(oi) and query-specific compatibility H*(oq, oi).

Empirically, the paper evaluates PRR across six standard multivariate benchmarks (ETTh1, Weather, Electricity, Traffic, Exchange, Solar) at four horizons (H ∈ {96, 192, 336, 720}, lookback L=96) under a same-channel retrieval constraint. In the main text (Table 1), PRR reduces AnalogFutureMSE relative to Pearson correlation in 23 of 24 conditions by 26.7% on average, and a budget-matched inference variant (PRR-B100) retains a 26.6% reduction. For downstream forecasting (Table 2), the uniform mean of the Top-10 retrieved futures is blended with five frozen dataset-specific backbones (PatchTST, iTransformer, TimeMixer, Seg-MoE, DLinear) via a single per-condition validation-selected scalar β ∈ [0, 0.20], achieving a +1.41% mean (+0.46% median) MSE improvement across 120 conditions while degrading slightly on MAE (-0.19%, Appendix F.4). Crucially, however, stress tests buried in Appendix G (Tables 19, 22, 23) show that a training-free, zero-parameter Anchored L2 baseline (subtracting the last observed value from the past window before computing Euclidean distance) beats full PRR in 17 of 24 retrieval conditions (median PRR gain -4.28%), beats Anchored-PRR (A-PRR) in 18 of 24 conditions, and outperforms full PRR on downstream forecasting across the 120 backbone conditions (+1.85% vs. +1.41% mean MSE gain; 64 wins, 10 ties, 46 losses for Anchored L2).

Critical strengths

Disciplined Diagnostic Separation of Retrieval Relevance, Candidate Priors, and Downstream Utility: The paper's strongest contribution is methodological and diagnostic rather than algorithmic. By holding five downstream forecasting backbones frozen, enforcing identical chronological query manifests, and introducing controlled ablations (Candidate Prior and Shuffled Future), the paper cleanly demonstrates that (a) lowering AnalogFutureMSE of individual neighbors does not automatically improve the ensemble forecast of a strong parametric model (the relevance–utility gap, highlighted on ETTh1 where retrieval MSE drops by 25–35% yet downstream forecasting degrades in 14/20 conditions), and (b) apparent retrieval gains on datasets like Weather and Exchange are heavily driven by query-independent candidate priors rather than query-specific matching.

Critical weaknesses

The Main-Text Baseline (Pearson) Is Metric-Mismatched to the Target, and Training-Free Anchored L2 Beats PRR and A-PRR in the Majority of Conditions (Appendix G.3, G.6, G.7): The headline claim in the Abstract and Table 1—that PRR improves retrieval by 26.7% over Pearson in 23/24 conditions—is largely an artifact of mismatching the baseline similarity metric and the evaluation metric. AnalogFutureMSE evaluates endpoint-anchored, fixed-scale Euclidean distance yt(h) = xtrain-normt+h - xtrain-normt, whereas Pearson correlation normalizes each window by its local standard deviation and is invariant to amplitude/scale. When the past retrieval rule is matched to the future evaluation geometry via a trivial, zero-parameter Anchored L2 rule (Appendix G.3, Table 19), training-free Anchored L2 reduces AnalogFutureMSE over Pearson by 26.25% (winning 24/24 conditions) and outperforms full PRR in 17 of 24 conditions (with PRR suffering a median degradation of -4.28% relative to Anchored L2, losing 0/4 on Traffic and 0/4 on Solar). Even when Pearson is replaced by Anchored L2 inside the PRR pipeline (A-PRR, Table 23), the learned support expansion and reranker still lose to standalone training-free Anchored L2 in 18 of 24 conditions (median gain -2.39%). Furthermore, on downstream forecasting across all 120 conditions (Table 22), training-free Anchored L2 outperforms full PRR (64W/10T/46L, +1.85% vs. +1.41% mean MSE reduction). Relegating these results to pages 24–26 while framing the main paper around beating Pearson substantially overstates the effectiveness of the proposed learned retriever. Limited Algorithmic Novelty, Per-Dataset/Per-Horizon Training Overhead, Weak Downstream Validation, and Misalignment with the Foundation Model / LLM Roadmap: Novelty: Training a past-window dual encoder using future trajectory similarity as supervision is already central to prior retrieval-augmented time series frameworks such as TS-RAG (Ning et al., NeurIPS 2025), which pretrains a contrastive retriever aligning lookback windows via future horizon similarity, as well as privileged future distillation in TimeKD (Liu et al., 2025). Propositions 1 and 2 in Appendix A are standard textbook properties of conditional expectation (𝔼[U ∣ O] minimizing L2 risk and orthogonal decomposition via iterated expectation). Validation & Real-World Applicability: PRR trains a separate CNN dual encoder and MLP reranker from scratch for every individual dataset and horizon (D, H) (24 separate trained stacks), all evaluated with a single random seed (seed 0) in Tables 1–2 without variance estimates. Downstream forecasting gains over Direct backbones are marginal (+1.41% mean, +0.46% median MSE, and a -0.19% degradation in MAE in Appendix F.4), with most of the MSE gain coming from boosting the simplest linear baseline (DLinear, +3.29%) while PatchTST degrades on average (-0.07%). Moreover, in 59 of 120 conditions, the validation-selected blending weight β hits the hard upper bound of the hand-chosen grid (β = 0.20, Table 9). Foundation Model / LLM Roadmap Alignment: Modern time-series foundation models (Chronos, TimesFM, Moirai, Timer, TS-RAG) and LLM RAG architectures rely on universal zero-shot retrievers and in-context / cross-attention conditioning that dynamically fuse retrieved trajectories and covariates at the token/patch level. Training dataset-and-horizon-specific univariate retrievers and combining them via a static dataset-wide scalar linear blend ŷA = (1-β)ŷD + β ŷR on small non-foundation models does not advance foundation-model-era forecasting or solve real-world non-stationary forecasting problems (where PRR fails on regime-shifting series like ETTh1).

Questions for authors

Clarification / Core Assessment — Why does future-supervised learning degrade Anchored L2 in 18 of 24 conditions (Table 23)? In Appendix G.7, A-PRR includes the full Anchored-L2 Top-100 candidates in its union and initializes its residual reranker directly on top of the standardized Anchored-L2 score sPRR(q,i) = sL2(q,i) + αr Δθ(fqi). Since setting αr = 0 exactly recovers standalone Anchored L2, why does training the future-supervised encoder and listwise reranker degrade test AnalogFutureMSE relative to standalone Anchored L2 in 18 out of 24 conditions (including 0/4 wins on Electricity, Traffic, and Solar)? Does this indicate severe overfitting of the future-supervised reranker under temporal distribution shift between the training period and the test period? Clarification / Core Assessment — Comparison against learned retrievers (e.g., TS-RAG) and Time Series Foundation Models: Why were learned retrieval baselines that also supervise or align representations with future trajectories (such as the contrastive retriever and Adaptive Retrieval Mixer in TS-RAG, Ning et al., 2025) omitted from the matched retrieval and downstream benchmarks, and how does PRR perform when paired with zero-shot time-series foundation models (e.g., Chronos, TimesFM, Moirai) rather than dataset-specific models? Clarification / Core Assessment — Single-seed evaluation and MAE degradation: All full-PRR results in Tables 1 and 2 use a single seed (seed 0, Appendix C.3). Given that the median downstream MSE gain is only +0.46% and MAE degrades by -0.19% (winning only 56/120 conditions, Appendix F.4), how many of the 120 downstream conditions show statistically significant improvements over Direct across multiple training seeds, and what happens if PRR's training objective d(q,i) is defined with L1 distance to match MAE?

Reviewer_YtyVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a retrieval-augmented time series forecasting algorithm called Predictive Relevance Retrieval (PRR). It uses future compatibility as supervision to learn a retrieval function but only uses past information in inference. Specifically, PRR combines Pearson correlation-based retrieval with a learned predictive representation to identify relevant historical examples. A fusion reranker is subsequently employed to select the most relevant candidates.

To improve the overall forecasting performance, the retrieved historical futures are averaged to construct a retrieval-based forecast, which is linearly combined with the predictions of existing forecasting models. The combination weight is calibrated using validation data.

Experimental results on six datasets and four forecasting horizons demonstrate that PRR can effectively retrieve historical examples with similar future trajectories. However, the improvement in retrieval does not lead to better forecasting performance. When combined with five existing forecasting models, including PatchTST, iTransformer, TimeMixer, Seg-MoE, and DLinear, PRR achieves an average MSE reduction of only 1.41% across 120 experimental settings. Among these settings, PRR improves forecasting performance in 79 cases, produces identical results in 15 cases, and degrades performance in 26 cases.

Critical strengths
  1. It is a well-motivated paper, and the proposed idea is interesting.
  2. The design of future supervision is natural and interesting.
  3. Generally extensive experiments.
Critical weaknesses
  1. The effectiveness of the proposed PRR for the overall forecasting is limited.
  2. The naïve anchored L2 method achieves quite good performance, which shows that the improvement of PRR is quite limited. The strong performance of the simple Anchored L2 baseline weakens the contribution of PRR.
Questions for authors
  1. Why do you choose different encoders for query and history. Please provide additional justification. Both query and history are from the same time series, and the future-distance supervision is symmetric with respect to the query and historical examples. Therefore, using a shared-weight encoder seems a more natural and parameter-efficient design.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.