- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
In this paper, we study the problem of choosing a scoring target for time series robustness evaluation: when a recorded perturbation crosses the prediction boundary and continues into the future, the prediction can be measured against either the altered observation or the original trajectory retained before the perturbation, which are different goals. To this end, we propose a controlled two-objective protocol: we inject artificial perturbations such as zero and horizontal shifts into the real dataset, while keeping the future before the perturbation as a reference, so that the same prediction can score the two targets separately. To separate the effects of scoring target, input repair, and prediction combination, we first train and freeze a multi-head prediction library, compare multiple combination rules on it, and compare no repair, causal repair, and oracle repair using pre-perturbation input. Post-hoc analysis shows that only changing the scoring target will reverse the ranking between the combination rules. The oracle repair reduces the error relative to the original trajectory, but increases the error relative to the changed observation. However, none of the incremental comparisons of the preset quality characteristics with the reliability control had a significant effect. Accordingly, the authors suggest that reporting robustness results should include the scoring objective and available repair information; The end-to-end model comparison and natural audit view analysis are also presented.
Critical strengths
The authors provide code, configuration, and reproduction instructions in the supplementary material so that others can replicate the experiment.
Critical weaknesses
-
The core conclusion is almost obvious. As long as the perturbation is correlated with the prediction difference, the standard answer ranking will change. It is also expected that the oracle fix makes the source error drop and the error increase after the change. The experiment was only about how many times the number reversal occurred, and did not bring new insight.
-
All four comparisons the authors had set out to test had no effect, and the main conclusion now is that the analysis done after reading the results is limited.
-
It has limited guiding significance for practical evaluation. The perturbation is artificial. There is little difference between a causal fix and no fix; what really makes a difference is an oracle fix that is not available in the real world.
Questions for authors
-
Besides the fact that changing targets changes rankings, what else did the rank-reversal experiment reveal that we hadn't expected?
-
The authors also say that how common such perturbations are in real data is another question. Given a real benchmark, how should a researcher decide which target to score?
-
Causal repair performs almost identically to no repair, and the only substantial difference comes from the oracle repair, which is unavailable in practice. What, then, does the recommendation to "state the repair access" imply for practical evaluation?