Hidden States Are Better Keys for Retrieval-Augmented Time Series Forecasting

Time series foundation models (TSFMs) deliver high-quality zero-shot forecasts on a target dataset. Unlike a conventional dataset-specific model, however, a TSFM can use only the part of the target series' history that fits in its input window, while the history outside the window is rich in periodic structure and statistical regularities. Retrieval-augmented forecasting can exploit that history without training the model, but existing methods compute similarity on raw series or with a separate embedding model: they measure how closely a historical segment's context matches in shape, whereas what enters the forecast is the segment's continuation, so the quantity the key measures is misaligned with the quantity the forecast needs. We argue that a retrieval key should measure predictive similarity, and that the hidden state of a frozen TSFM has exactly this property: used as the key, it retrieves continuations whose error is 14% lower than with the raw series and 17% lower than with a separate embedding model. Building on this, we propose HiRAF (Hidden-state Retrieval-Augmented Forecasting), which reads the hidden state from the forward pass that produces the forecast, retrieves the most similar segments from the target series' history with it, and uses their continuations to correct the backbone's forecast. No model is trained; the key is a by-product of the forward pass that produces the forecast, so no separate embedding model is needed, and with a bucketed index each retrieval scans only a small fraction of the history. On six benchmark datasets HiRAF achieves better zero-shot accuracy than existing TSFMs and 7% lower MSE than the retrieval-augmented method TS-RAG; with an index of five million entries, its prediction time is 1.05× that of the backbone alone, against 8.17× for TS-RAG.