- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Predictive Relevance Retrieval (PRR), which uses historical future outcomes as training supervision to learn which past time-series segments are useful for forecasting, while relying only on observed pasts at inference. PRR combines similarity-based and learned retrieval with a reranker, improving retrieval quality over Pearson correlation across six datasets. However, these improvements yield modest, dataset-dependent forecasting gains, highlighting that better historical analogs do not necessarily improve a strong forecasting model. Some other minor concerns I have regard the lack of multiple seeds, and the positioning of the theoretical section. I understand it is required to introduce the setup and the notations,
Critical strengths
- I think the research question is clear. The authors are trying to distinguish similarity between historical pasts from compatibility of their future trajectories.
- Empirical results show consistent gains over Pearson with retrieval helping in 23/24 settings, with a 26.7% average reduction in analog future MSE.
- The paper shows that improved individual analog quality does not necessarily translate into better forecasting, which I believe is an important finding for future retrieval-based forecasting work.
- The authors have run some ablations and report negative findings as well for transparency.
- Codebase is provided.
Critical weaknesses
- The main weakness is there is a limited advantage over a simple baseline. Training-free anchored L2 beats PRR in 17/24 retrieval settings and achieves greater average downstream improvement. This undermines the importance of the method.
- The reported gains are relatively small. Mean forecasting improvement is 1.41%, with a 0.46% median and degradation in 26/120 settings.
- Full PRR uses a single seed, and headline results lack confidence intervals. The multi-seed evidence mainly concerns PRR-Stat.
- I am unsure about the role of the theoretical section because the theory provides standard conditional-expectation characterizations without guarantees for the implemented learning objective.
Questions for authors
- First of all, given that training-free anchored L2 outperforms PRR in 17/24 retrieval settings and achieves larger average downstream gains, when does PRR justify its additional training and inference cost?
- Can you isolate the benefits of learned candidate generation and embedding-based reranking by evaluating each independently under fixed candidate support and comparable model capacity?
- How stable are the full-PRR results across training seeds? I believe a statistical significance analysis can make the results more reliable.
- Can you quantify whether the gap between retrieval quality and forecasting gains arises primarily from correlated neighbor errors, lack of complementarity with the direct forecaster, or something else?
- Finally, do the Candidate Prior and Shuffled Future findings extend to full PRR at the main forecasting horizons?