- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper first shows that for RAG-based TS forecasters, such as TS-RAG and Cross-RAG on a frozen Chronos-Bolt backbone, a retrieved continuation that beats the base forecast often exists, but the final forecast rarely makes use of it. To address this issue, the authors propose ROAR, which is a refiner for TS forecasting which works in the prediction space. On seven TS benchmarks, ROAR reports the best average MSE and also improves several RAG methods and two other backbones.
Critical strengths
[1] Diagnostic study (Appendix A) is the most useful part of the paper to me.
The section includes controls that are often missing in this TS research area (e.g., split-horizon check for regression to the mean (Table 7) and the shuffled-neighbor control (Table 5)). It also reports the results that do not fit the authors' story, for example that Cross-RAG's retrieved candidates are no better than random draws and that no observable feature separates queries where retrieval helps from those where it does not. This makes me think that other researchers working on RAG-based methods using TSFMs would find it useful.
[2] The refiner is cheap and transfers across hosts and backbones.
Stage I adds only about 160 trainable parameters, and the same module improves three RAG hosts (Fig. 3) and two other backbones (Table 3). The backbone transfer result is more convincing to me than the main table, since the gains there are larger than the gaps between methods in Table 2.
Critical weaknesses
[1] Main mechanism (opportunity-aware weighting) is never isolated.
The ablation study shown in Table 4 removes Stage I as a whole, so the scorer, gate, aligned candidates and convex refinement all disappear together. Thus, the gain only tells us that a gated mixture of aligned candidates helps. It does not show that the opportunity term helps compared with plain weighting (beta = 0), or that the gate loss matters (lambda_gate is never set to 0 in Fig. 5). Simple non-learned refiners are also missing.
This is important as Eq. 13 already shows that simply combining the base and candidate predictions can improve the performance, even w/o considering the opportunity. Also, results in Appendix A.4 shows that the features used by the gate do not provide much information about even when retrieval is actually helpful. Therefore, I believe the improvement may come mainly from combining 20 aligned predictions, and not from the proposed mechanism.
[2] The gains are small, reported without uncertainty, and Stage II doubles inference cost.
While results are averaged over 5 runs, standard deviations or significance tests are not provided. The gap to Cross-RAG is 0.193 vs. 0.187 average MSE, and Cross-RAG is better than ROAR's own base in Table 4, which is why I believe the gains are small. Stage II improves the average MSE only by 0.0018, but Table 11 shows that it almost doubles the latency, FLOPs, and number of parameters, as the frozen reference branch needs to run together with the updated branch. Also, with , the paper’s own analysis shows that Stage II can improve only about 12% of the remaining error. For these reasons, I am concerned whether Stage II is worth the additional cost compared to simply averaging two separately trained modules.
The evaluation is also narrow.
- It uses a single horizon (H = 64),
- four of the seven datasets are ETT variants,
- and there are no probabilistic metrics even though the method shifts quantiles.
The sweeps in Fig. 5 are run on the test splits, so if K and lambda_ref were chosen this way, the zero-shot protocol is weakened.
[Smaller issues]
The main text says retrieval uses the TS-RAG subset of the Chronos corpus, but Appendix A.1 says each variable retrieves from its own training split, and this changes how zero-shot the setting is. PFRP and RATL are adapted to zero-shot without describing what was changed. There are also a few broken figure and equation references.
Questions for authors
Q1) Could you report the performance of ROAR with , = 0, and a fixed-gate uniform-average refiner over the same 20 aligned candidates? If beta = 0 performs within noise of the full model, the framing of the paper would need to change, and my rating depends mostly on this.
Q2) What are the standard deviations over the 5 runs for Tables 2 and 4, including the reproduced baselines? This would tell me whether the gaps to Cross-RAG are larger than seed variance.
Q3) Which retrieval database is used at test time in the main experiments, the Chronos-corpus subset or each target dataset's training split? Also, were K, lambda_ref and beta selected on the evaluation datasets? Both affect how zero-shot the setting really is.
Q4) Does reference anchoring in Stage II add anything beyond a cost-matched baseline, such as an ensemble of two forecasting modules or Stage I on top of a module fine-tuned with plain MSE? Given the 2x inference cost, I would like to see this comparison.
I thank the authors for their additional experiments during the rebuttal period. However, I still have some concerns regarding the generalizability of the proposed method and the contribution of the opportunity-aware mechanism. Although the new results have addressed some of my questions, I am not fully convinced that the proposed method gives sufficient value. I will maintain my current score, but I will discuss the paper with other reviewers and consider their reviews & opinions.