- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The general task the authors solve in the paper is the following: Given class labels for multivariate time series samples that broadly classify the samples into faults and normal operation, they want to identify channels and time steps that distinguish the faulty samples from the normal ones, aiding root cause analysis.
Understanding which channels and time steps discriminate the classes from each other can be a valuable hint to solve this, therefore the authors propose a hybrid convolutional-transformer architecture for multivariate time series classification with builtin channel level and temporal interpretability.
They first embed each channel with a dedicated 1D-CNN. The resulting per-channel embeddings are used as tokens and a cls token is appended to it. They then expand the resulting tensor to exist S (num channels) times and append the original multivariate time series to it. They project the resulting tokens with a shared linear layer into a d-dimensional embedding space and add a learnable positional encoding before passing it to a transformer encoder layer. the final cls token is passed to the classifier. The idea is, that with this setting, the cls token can allocate attention weights to each univariate encoding individually, and to the raw time series tokens across channels, which enables interpretation of the cls attention weights as feature importance indicators. They argue that attention from cls to cnn tokens reveals channel importance and attention from cls to raw time step tokens reveals temporal importance. They train the model with input dropout to simulate missing or corrupted channels, by either dropping the raw input or the cnn embeddings of a randomly selected subset of samples.
They test their architecture on one dataset from the UEA Time Series Classification archive and augment it with synthesized faulty classes (synthetic stuck value in channel 1 for two different time ranges, adding a class label for both). They show that the channel with the fault is the one with the highest importance (attention) when classifying the fault class correctly and that the attention on the raw data tokens peaks where the fault is inserted. The second dataset is from actual field vehicles and contains a real fault due to component issues. They show that the method correctly points at the pressure 2 channel.
The problem that the authors investigate is an important one and solving it with one architecture that is inherently interpretable is interesting. Despite that, the work is not comparing against a single baseline and the proposed architecture misses proper ablations of the individual components and algorithmic decisions (individual embeddings vs shared ones, CNN vs MLPs, input dropout vs no input dropout vs the proposed structured input dropout, ...), which makes it difficult to assess the actual benefit. Additionally, I think this line of research overlaps with research in the area of time series anomaly detection and localization. (1) Contrasting against sota TSAD approaches (e.g. alora-T) and (2) evaluation on localization benchmarks like SMD or SWaT would strengthen the paper significantly. Lastly, I think a comparison against common interpretability methods like SHAP would improve the manuscript further. The code and dataset are not accessible. That said, I recommend a reject and think that another round of polishing is necessary to finalize the manuscript.
Critical strengths
- Given fault labels the method can be trained end-to-end and comes with inherent interpretability.
Critical weaknesses
- It's hard to assess the actual benefit of the proposed method because evaluations on standard benchmarks and against proper baselines are missing.
- The architecture and algorithmic decisions are not properly ablated.
Questions for authors
- In the methods section you describe a reshaping to M = 1 + S.F + L. Is each sequence element in M a token and embedded individually to d dimensions?
- Do you plan to release the second dataset?
- You state that the input level dropout during training is essential. Do you have any experiments that back this, that maybe not have made it into the manuscript?