acceptodds
Under review as a conference paper at ICLR 2027

Target Fidelity in Time-Series Forecast Evaluation: A Controlled Study of Forecast Combinations

Abstract

Persistent recording disturbances can make forecasting benchmarks reward agreement with altered observations. Evaluating robustness therefore requires distinguishing this agreement from fidelity to the original recorded trajectory. The distinction is difficult to measure when input repair, predictor training, and forecast combination change together. We introduce a controlled protocol that applies disturbances across the forecast boundary while retaining the pre-injection future as an observed reference. Forecasting heads are trained with equal individual losses and frozen before combination fitting, allowing the same prediction to be scored against retained-source and altered-observation targets under matched repair conditions. In a post-hoc analysis with causal repair, changing only the scoring target reverses comparisons between fixed forecasts, including reversals whose directions persist across training seeds. Oracle access to retained source inputs improves source-target accuracy while increasing altered-target error. These results motivate pairing robustness claims with an explicit scoring target and the available repair information.

Primary area:
learning on time series and dynamical systems
Keywords:
time-series forecasting, forecast evaluation, target fidelity, data corruption, robustness evaluation, forecast combinations

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_E6VdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, we study the problem of choosing a scoring target for time series robustness evaluation: when a recorded perturbation crosses the prediction boundary and continues into the future, the prediction can be measured against either the altered observation or the original trajectory retained before the perturbation, which are different goals. To this end, we propose a controlled two-objective protocol: we inject artificial perturbations such as zero and horizontal shifts into the real dataset, while keeping the future before the perturbation as a reference, so that the same prediction can score the two targets separately. To separate the effects of scoring target, input repair, and prediction combination, we first train and freeze a multi-head prediction library, compare multiple combination rules on it, and compare no repair, causal repair, and oracle repair using pre-perturbation input. Post-hoc analysis shows that only changing the scoring target will reverse the ranking between the combination rules. The oracle repair reduces the error relative to the original trajectory, but increases the error relative to the changed observation. However, none of the incremental comparisons of the preset quality characteristics with the reliability control had a significant effect. Accordingly, the authors suggest that reporting robustness results should include the scoring objective and available repair information; The end-to-end model comparison and natural audit view analysis are also presented.

Critical strengths

The authors provide code, configuration, and reproduction instructions in the supplementary material so that others can replicate the experiment.

Critical weaknesses
  1. The core conclusion is almost obvious. As long as the perturbation is correlated with the prediction difference, the standard answer ranking will change. It is also expected that the oracle fix makes the source error drop and the error increase after the change. The experiment was only about how many times the number reversal occurred, and did not bring new insight.

  2. All four comparisons the authors had set out to test had no effect, and the main conclusion now is that the analysis done after reading the results is limited.

  3. It has limited guiding significance for practical evaluation. The perturbation is artificial. There is little difference between a causal fix and no fix; what really makes a difference is an oracle fix that is not available in the real world.

Questions for authors
  1. Besides the fact that changing targets changes rankings, what else did the rank-reversal experiment reveal that we hadn't expected?

  2. The authors also say that how common such perturbations are in real data is another question. Given a real benchmark, how should a researcher decide which target to score?

  3. Causal repair performs almost identically to no repair, and the only substantial difference comes from the oracle repair, which is unavailable in practice. What, then, does the recommendation to "state the repair access" imply for practical evaluation?

Reviewer_zce2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is extremely hard to follow, and appears to be research conducted by AI, with little to no human input. My high level understanding reading this paper is that it concerns evaluation in forecasting models, specifically in settings where the targets are missing (all zeros).

Critical strengths

I believe this paper is AI generated, and I do not see any strengths.

Critical weaknesses

The paper is extremely hard to follow.

Let's take the problem formulation in Section 3 as example:

  • Line 126: "Let X∈RX \in R ... be a retained observed input and future." -- here retained is not a standard terminology in time series forecasting, that I am aware of.
  • Line 127: "A prescribed transformation produces" -- what is a prescribed transformation?
  • Line 128: "A forecasting rule ff receives ... input-side audit summaries, and the repair tensor permitted by condition" -- None of these are standard terms that make sense to a time series forecasting practitioner.
  • Equation 1 is misformated: El(⋅)\mathbb{E}l(\cdot) instead of E[l(⋅)]\mathbb{E}\left[l(\cdot)\right], which a simple human review should have caught.

Similarly in Section 4:

  • Line 156: "The bank has seven internal heads: direct linear, trend-seasonal decomposition, patch encoding, temporal convolution, variate mixing, spectral-periodic, and audit residual."-- It is unclear to me what the bank is or it's motivation.
  • Line 159: "We minimize the equally weighted mean of their individual 0.7MAE + 0.3MSE losses." -- This weird combination of MAE and MSE would not pass any human scrutiny.
Questions for authors

N/A

Reviewer_RUk2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how the choice of evaluation target affects comparisons of time-series forecasting methods when perturbations introduced during data recording persist into the forecasting period. It proposes a controlled, dual-target evaluation protocol that uses a shared, fixed bank of forecasting models to score the same set of predictions against both the retained pre-perturbation observations and the perturbed observations. The experiments show that changing the scoring target can reverse the rankings of different forecast combinations. They also show that using the retained original inputs can improve fidelity to the original data while potentially reducing agreement with the perturbed observations.

Critical strengths

The paper addresses an interesting question concerning the balance between accuracy and dynamic responsiveness in long-term time-series forecasting, showing that the relative performance of different methods is not consistent across these two objectives. This offers useful insights into time-series forecasting. In addition, the authors propose a controlled experimental approach to investigate this question and report comprehensive, credible results.

Critical weaknesses
  1. The paper devotes substantial space to the seven forecasting heads, AIF, the gating mechanism, and the reliability controller, without giving comparable attention to its central findings. This makes the paper difficult to follow.

  2. In my view, the text in the experimental sections need not repeat information already presented in the tables, as it does in Section 5.3. The discussion should focus more on the conclusions drawn from the results.

  3. The controlled study covers only two forecast horizons, 96 and 336 steps, and the 336-step extension does not include one of the datasets. Although the separate end-to-end comparison covers four forecast horizons, the paper has not demonstrated whether its central findings regarding sensitivity to the evaluation target also hold at horizons of 192 and 720 steps.

Questions for authors
  1. What about other types of perturbations beyond those involving zero values?

  2. What is the rationale for the weighted combination 0.7 MAE + 0.3 MSE on line 159?

  3. Section 4.2 provides very detailed experimental configurations, but restricting the evaluation to a forecast horizon of 96 steps is not common practice. Could the authors explain the rationale for this design choice?

  4. Line 221 states, “Eleven of 45 banks reached the epoch cap.” Does this mean that all the remaining training runs were terminated by early stopping? If so, could the models in these 11 cases have been insufficiently trained?

  5. Are the “Reversals” and “Winner changes” results in Table 1 statistically significant?

  6. The “Positive” and “Holm” results in Table 4 appear inconsistent. Please provide the raw p-values.

  7. Why does Section 6 use a different experimental configuration from Section 4? I would appreciate an explanation from the authors; without clarification, the results appear to have been cherry-picked.

The research question addressed in this paper has potential. I hope the authors can address the questions above, and I would be happy to raise my score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.