- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Time series forecasting models can be explained by identifying which past values drive the predictions. However, recent works generate a single importance scores vector to explain the whole predictions, which can be too broad. The paper argues that different forecast steps should depend on different past values. Therefore, the work proposes a new method called Horizon-Resolved eXplanation (HRX), which adds the horizon axis to the explanation so that every forecast step has its own importance attribution map. From this perspective, a new evaluation protocol is proposed to measure how much a single forecast step changes when the most important features are removed. The new study also shows that step-wise dependence is low-dimensional. Extensive experiments with a setting of 56K runs across 26 backbones and 15 benchmarks show that improvement of the method comes from the horizon axis rather than any particular estimator.
Critical strengths
The paper proposes a clear, simple but novel idea on explaining time series forecasters that each forecast has H values so each step should have its own importance map. Table 1 can show how the work is different from previous works while Figure1 can show the motivation for HRX.
The method is simple and efficient with Gradient of a single forecast step. This help the method still be practical in real applications.
The paper is well written and easy to follow with the definition of each term clearly defined before going to the formula.
The experiments are comprehensive with 26 backbone and 15 benchmarks with 56K runs to show comparison.
The results show that explanations with the horizon axis can have better quality than previous work with only a single explanation.
The analysis on Time vs. channel axis, and how many maps a forecast needs in Section Analysis can show how Horizon should contribute.
Critical weaknesses
The behaviour is expected that if we explain each time step prediction independently, we should see different importance maps. However, the objective of explanation is normally for the whole period, then the features drive that seem enough in some applications. It is great if the paper can show an explanation on each forecast time step and can contribute to the whole explanation for the whole steps at once.
In reality, the motivation of HRX from the user's perspective is not clear. A practitioner or a developer may need explanation for each forecast step, but what this brings them more value than a single vector for all steps is not clear.
Explanations appear hard to read for users. The predictions can be interpreted by humans easier with segments or see the whole period instead of checking individual time points and then checking each explanation. For example, in Figure 1, some users can see the output as a trend or a segment with temporal relationship and ask why past segments or trends can lead to this rather than look at each time step.
The evaluation estimates the shared vector and estimates g_own, g_shuffled, and g_margin. However, as I understand, all need a horizon axis to estimate AUC, and the shared vector also depends on the attribution map of each forecast step. However, compared to previous works, this evaluation can not be used since there are no horizon steps in the setting and the output of HRX is not the same format as others. The paper should consider a shared vector as a single vector to compare or use other metrics to compare to other methods that return a single vector.
The motivation is good but methodology of HRX seems to apply existing gradient-based methods to explain each forecast step. The method should also take the forecast steps around one step to see the relationship and improve the method.
How the method can adapt with time-frequency transformation and segment-based explanation method since the segments can be more interpretable than time points.
How does the explanation for each forecast step cooperate together to explain the whole prediction period? Since each attribution map can explain well in each forecast step but when combined together, the faithfulness or correctness of explanation can be no better than the previous method under the whole period. These comparisons should be considered to show effectiveness of HRX in practice.
The experiments should consider experiment on foundation models which can finetune rather than only zero-shot setting since this can have different behaviour.
Questions for authors
Can the authors explain why users need an explanation for each forecast step rather than a single vector?
Can the authors explain how users can interpret explanations?
Can the authors discuss about the metrics and how the metric can compare between their method with previous ones?
Can the authors explain why they choose gradient only for estimator instead of other methods in time series explanation?
Can the authors show how effective of the method when the explanation presentation is segments or other interpretable form?
Can the authors show how explanation of each forecast step can combine together to explain the forecast and show it better than previous work?
3 comments
Thanks the authors for the rebuttal. This almost address my concerns. I keep my positive score.
Thank you for your review and constructive feedback!
We look forward to the continued discussion with the other reviewers and AC during the post-rebuttal phase.
Please feel free to let me know if you have any further questions or concerns.
If you feel that your concerns have been adequately addressed, I would sincerely appreciate your consideration of an updated score.