- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TRACE corrects a frozen forecaster after the fact by retrieving its own past errors. It keeps a memory of validation residuals and finds neighbors with a descriptor built from the context and, optionally, the base forecast. It shrinks the averaged residual where neighbors disagree. A dial (beta) can also mix in part of the neighbors’ trajectories, interpolating between error retrieval and analog (future) retrieval. A validation gate then picks, per setting, among this retrieval correction and simpler ones: a seasonal blend, an overlap blend, a static or rolling bias, or no change. Across backbones and datasets, it improves on uncorrected forecasts and on RAFT, SARAF and analog retrieval applied to the same fixed forecasts.
Update: I see no rebuttal from the authors, as the deadline is less than 12 hours. Therefore, I vote reject for this paper.
Critical strengths
-
The question is well framed. The paper separates case relevance from what the forecast already captures, and nests analog and error retrieval in one dial
-
Evaluations seem comprehensive to me
Critical weaknesses
-
The main weakness is that it is hard to tell where the gain comes from. TRACE picks, per setting, among identity, a seasonal blend, an overlap blend of earlier forecasts, static and rolling bias, and retrieval. In the main comparison, non-retrieval candidates are selected in a lot of long-term cells (Section 4.2). The rolling bias also uses test labels once their targets have passed, which the static-memory baselines never see. The paper itself says Table 8 is descriptive, so the key ablation is missing: the same selector without the retrieval candidates. Without it, the results mostly show that validation-selected post-processing helps, not that retrieving errors does
-
I’m not convinced the paper’s own experiments support its main message. When everything else is held fixed, ANALOG-KNN actually has lower average error than RESIDUAL-KNN on six of the seven long-term datasets and on all four PEMS datasets. And when TRACE does choose retrieval, it nearly always keeps some of the analog drift: beta > 0 in 435 settings, out of 449 where retrieval is selected. If I read this correctly, pure error retrieval is chosen in at most 14 settings. The 15.6% headline also leans heavily on weak, lightly trained backbones. For the stronger backbones (TimeXer, TimeMixer, PAttn and PatchTST), the long-term MSE gain is much less, and TimeMixer actually gets worse on PEMS
Questions for authors
-
What do the results look like with the same selector but no retrieval candidates?
-
How often is the causal rolling bias selected in the main comparison? What changes if candidates that use test-period labels are removed?
-
Section 4.2 suggests beta > 0 in 435 of 449 retrieval selections. Where is pure error retrieval (beta = 0) actually chosen, and which configuration produced Table 1?
Dear Reviewer QrAr,
W1 & Q1: Ablation Study
Removing retrieval and rerunning the same selector raised mean test MAE by 2.00% for long-term forecasting, 23.01% for traffic, and 3.43% for electricity prices on the settings with paired test results.
We removed both retrieval candidates (historical errors alone and historical errors with trajectory drift) and repeated validation selection across all 585 settings. We kept the base forecasts, remaining correction options, two chronological validation folds, scoring rule, and tie-breaking rule fixed.
Each row compares mean MAE on the same settings. For the long-term MSE metric reported in Table 1, removal raised mean error from 0.40585 to 0.42229 (+4.05%). In 134 of the 260 paired long-term settings, both MSE and MAE were unchanged.
Removing retrieval left the validation winner unchanged in 136 settings and forced a switch in 449. Test results cover all 136 unchanged settings and 219 of the 449 replacements, giving 355 paired comparisons. The largest observed increase in MAE was on traffic. The effect across the full benchmark remains open because test errors are unavailable for the other 230 replacements.
Q2: Rolling bias
TL;DR: Rolling bias was selected in 9 of the 585 settings (1.54%), all long-term forecasting settings. Removing rolling bias leaves most predictions unchanged and in the remaining nine settings the selector chooses the next highest ranked option on validation.
Under the policy reported in the main comparison, rolling bias was selected in 9 of the 364 long-term settings, 0 of the 156 short-term traffic settings, and 0 of the 65 electricity-price settings. The nine selections occurred on ETTh2 (3), ETTm2 (3), Traffic (2), and Weather (1). Seven were at the longest horizon, 720 steps. The 56 rolling-bias selections in Appendix Table 8 come from a separate experiment with a different candidate pool.
With the same validation scores and tie-breaking rule, removing rolling bias can change the selected forecast only in those nine settings.
W2 & Q3: Error Retrieval, Trajectory Drift, and Configuration
The two methods share the same frozen base forecasts, although their neighbor descriptors and tuning grids differ. We tested the transfer rules using the same eight neighbors and weights on German DLinear/24. With correction strength tuned on the same validation grid, test MSE was 0.4978 for historical errors, 0.4920 for aligned trajectories, and 0.4829 for errors plus half the trajectory drift. Aligned trajectories had lower MSE than historical errors (0.4920 versus 0.4978), but higher MAE (0.4461 versus 0.4445). The result supports the framework’s ability to combine the two: the intermediate rule had the lowest error. Across the main benchmark, validation selected an intermediate drift weight in 299 of 449 retrieval selections.
TL;DR: TRACE selected pure error retrieval in seven long-term forecasting settings. Table 1 reports full TRACE. In settings where validation selected retrieval, it also determined β.
The table shows the breakdown of the 14 settings by dataset, backbone, and forecast horizon, where pure error retrieval is chosen.
Table 1 reports
p_v2_folded, which allows all four drift weights at every horizon. The same selector chooses the correction and its hyperparameters in each setting. It evaluates each candidate on two consecutive validation periods, computes its error relative to the uncorrected forecast for each reported metric, and selects the candidate with the lowest worst-case ratio across metrics and periods. The uncorrected forecast wins ties. The selected configuration is then fixed for test evaluation. For example, on the same 24-hour German electricity-price task, this rule selected β = 0 for DLinear, 0.5 for PatchTST, and 1 for TimeXer.