- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes an algorithm for auditing whether a given attribution is more informative than a random attribution. Attributions that fail this test are considered uninformative. The approach is class-conditional, allowing informativeness to be assessed for specific target classes. Passing the test does not establish that an attribution is meaningful or plausible in an absolute sense; it only indicates that it is preferred over a random baseline. The evaluation uses several datasets, predominantly synthetic ones, with Integrated Gradients and Feature Ablation as reference attribution methods. The paper addresses an important problem in time-series attribution and presents a promising approach. However, the limited evaluation, insufficient justification of the decision threshold, and lack of analysis of performance differences across datasets and classes restrict the conclusions that can be drawn about its generalizability and practical applicability.
Critical strengths
Mathematical details are presented detailed and complete.
Critical weaknesses
- The evaluation does not include sota network architectures. This limits the evidence that the approach generalizes across models and attribution methods. A broader comparison is needed to understand how the choice of attribution method affects the assessment.
- The threshold used to classify an attribution as informative lacks sufficient statistical and empirical justification. The paper does not adequately examine sensitivity to this choice, variation across datasets, or whether a common threshold can be applied reliably across various settings.
- The figures suggest that the approach succeeds only for certain datasets and classes, but the evaluation does not sufficiently explain these differences. It remains unclear which properties determine success or failure and how a practitioner could identify suitable settings in advance.
- The evaluation relies heavily on synthetic and controlled datasets which are useful for systematic analysis, but the small number of real-world datasets provides limited evidence of effectiveness in practical applications.
Questions for authors
None aside from above
9 comments
Q1/W1: The evaluation does not include sota network architectures
Q1/W1: The evaluation does not include sota network architectures.
We thank the reviewer for this suggestion. We add three public datasets, CricketX and UWaveGestureLibraryX from the UCR archive and EpilepticSeizure, an EEG dataset used by [B], and three strong time-series classifiers. Mantis is a recent foundation model for time-series classification that outperforms existing foundation models across four dataset collections [C], and we fine-tune it on each dataset. HYDRA is faster and more accurate than the most accurate dictionary methods on 112 UCR datasets [D], and an independent benchmark finds its combination with MultiROCKET among the two best classifiers [E]. ModernTCN is a recent pure convolution architecture for general time-series analysis [F]. Table 1 gives the test accuracy of each model. We explain Mantis and ModernTCN with IG and HYDRA with FA, because the HYDRA transform is not differentiable. We audit each model with five perturbation baselines, including zeros, NULL zeros, Gaussian noise, NULL noise and CF train, where CF train is fitted on the train split. We use each baseline with both directions, MoRF and LeRF. A class passes when the explanation is better than the random in this class by at least ε = 0.005. Tables 2 to 4 count the classes that pass. First, the main failure in our paper also appears in these models. With zeros or Gaussian noise, the audit over all test series usually passes, but at least one class often fails. Second, adding NULL to the zeros baseline fixes some of these class failures, while NULL with noise and CF train fix only a few, and many classes still fail with them. Each fix therefore helps only in some settings, so every configuration needs its own audit.
Table 1: test accuracy / macro-F1
| dataset | Mantis | HYDRA | ModernTCN |
|---|---|---|---|
| CricketX | 0.795 / 0.797 | 0.795 / 0.793 | 0.710 / 0.702 |
| EpilepticSeizure | 0.814 / 0.812 | 0.791 / 0.792 | 0.754 / 0.751 |
| UWaveGestureLibraryX | 0.857 / 0.856 | 0.851 / 0.846 | 0.738 / 0.718 |
Table 2: CricketX, classes that pass (of 12)
| model | Dir. | zeros | NULL zeros | noise | NULL noise | CF train |
|---|---|---|---|---|---|---|
| Mantis | MoRF | 11 | 11 | 11 | 11 | 3 |
| Mantis | LeRF | 11 | 11 | 11 | 12 | 10 |
| HYDRA | MoRF | 8 | 12 | 7 | 0 | 4 |
| HYDRA | LeRF | 11 | 7 | 9 | 1 | 6 |
| ModernTCN | MoRF | 12 | 12 | 12 | 12 | 6 |
| ModernTCN | LeRF | 12 | 12 | 12 | 12 | 10 |
Table 3: EpilepticSeizure, classes that pass (of 5)
| model | Dir. | zeros | NULL zeros | noise | NULL noise | CF train |
|---|---|---|---|---|---|---|
| Mantis | MoRF | 3 | 4 | 3 | 2 | 3 |
| Mantis | LeRF | 4 | 4 | 1 | 1 | 3 |
| HYDRA | MoRF | 3 | 3 | 0 | 0 | 2 |
| HYDRA | LeRF | 5 | 5 | 1 | 2 | 5 |
| ModernTCN | MoRF | 4 | 4 | 1 | 1 | 3 |
| ModernTCN | LeRF | 3 | 3 | 1 | 3 | 3 |
Table 4: UWaveGestureLibraryX, classes that pass (of 8)
| model | Dir. | zeros | NULL zeros | noise | NULL noise | CF train |
|---|---|---|---|---|---|---|
| Mantis | MoRF | 5 | 6 | 4 | 2 | 3 |
| Mantis | LeRF | 6 | 6 | 5 | 5 | 4 |
| HYDRA | MoRF | 2 | 2 | 2 | 0 | 1 |
| HYDRA | LeRF | 6 | 6 | 7 | 0 | 6 |
| ModernTCN | MoRF | 8 | 8 | 8 | 8 | 7 |
| ModernTCN | LeRF | 5 | 8 | 6 | 8 | 1 |
Reference:
[B] Fan H, Zhang F, Wang R, Huang X, Li Z. Semi-supervised time series classification by temporal relation prediction. InICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2021 Jun 6 (pp. 3545-3549). IEEE.
[C] Vasilii Feofanov, Songkang Wen, Shifeng Xie, Simon Roschmann, Marius Alonso, Hongbo Guo, Romain Ilbert, Malik Tiomoko, Quentin Bouniot, Zeynep Akata, Lujia Pan, Jianfeng Zhang, Ievgen Redko. Mantis: Lightweight Foundation Model for Time Series Classification. Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:30692-30719, 2026.
[D] Dempster, Angus, Daniel F. Schmidt and Geoffrey I. Webb. “Hydra: competing convolutional kernels for fast and accurate time series classification.” Data Mining and Knowledge Discovery 37 (2022): 1779-1805.
[E] Middlehurst, Matthew, Patrick Schäfer and A. Bagnall. “Bake off redux: a review and experimental evaluation of recent time series classification algorithms.” Data Mining and Knowledge Discovery 38 (2023): 1958 - 2031.
[F] Luo, Donghao and Xue Wang. “ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis.” International Conference on Learning Representations (2024).
Q2/W2: The threshold used to classify an attribution as informative lacks sufficient statistical (1/3)
Q2/W2: The threshold used to classify an attribution as informative lacks sufficient statistical and empirical justification.
We thank the reviewer for this point. We support ε = 0.005 with two tests. The first test shows that ε is larger than the change that the standard deviation of different random ranking, so a Healthy label does not come from a lucky random draw. The second test shows that our conclusions do not change when ε changes.
First, we choose ε so that a label cannot come from a lucky random draw. Our audit draws one random ranking for each test series, from a manual seed. For a fixed model, this random ranking is the only random part of the audit, so ε must be larger than the change in the gap that a different random draw can cause. To measure this change, we redraw the random ranking with 10 seeds, m = 0, …, 9, where draw 0 is the one used in the audit. The GT map does not change, so we score it only once. For each draw m, we compute the same gap as in Equation 3:
Δ_m = S(GT) − S(rand_m),
where S is the true-class confidence averaged over the 11 retention ratios. For MoRF we flip the sign, so a positive Δ_m always means that GT beats random. Each draw then gets its own label, which is Healthy if Δ_m ≥ ε, Inverted if Δ_m ≤ −ε, and Blind otherwise.
The test covers 108 settings, which come from 6 datasets × 3 architectures × 2 directions × 3 scopes. The datasets are FPH, RPH, FPH-FD, RPH-FD, MC-1v5 and PP, and the architectures are Linear, CNN and RNN. The two directions are LeRF, which replaces the least important points with the baseline first, and MoRF, which replaces the most important points first. The three scopes are all test examples, the examples of class 0 and the examples of class 1. All settings use the base models with the background and seed 0.
Tables 5 to 7 report, for each setting, the gap of the audit's own draw (Δ₀), the mean of the 10 gaps (Δ̄) and their standard deviation (σ). The standard deviation is at most 0.0047, so ε = 0.005 is larger than the spread between random draws in every setting. A gap of at least ε therefore does not come from a lucky draw. Indeed, the audit label does not change in any of the 108 settings, for every single draw.
Paper Summarisation
We thank the reviewer for the constructive review and for recognizing that our paper addresses an important problem in time-series attribution with a promising approach. We apologize that our unclear writing may have caused confusion about what the audit test is. Our audit tests the faithfulness evaluator rather than the attribution itself. To clarify this difference, the table below compares our audit with testing an attribution against random, which is similar to ROAR [A] (Hooker et al., 2019).
Reference:
[A] Hooker, S., Erhan, D., Kindermans, P.-J., & Kim, B. (2019). A benchmark for interpretability methods in deep neural networks. In Advances in Neural Information Processing Systems (Vol. 32).