- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper intorduces Context in Time-series benchmark, bridging the gap of lack of context in time sereis benchmark in anomaly detection and imputation tasks. In addition, the authors construct a langugage-time-series alignment dataset for training to improve the understanding of both textual and time-series inputs.
Critical strengths
-
This paper extend Context in Time-series benchmark to include anomaly detection and imputation tasks.
-
The authors construct a langugage-time-series alignment dataset for training to improve the understanding of both textual and time-series inputs.
-
The authors evaluate multiple general-purpose LLM and time-series language models on the proposed benchmark.
Critical weaknesses
-
The dataset cannot be accessed in the provided huggingface link. Without the documented dataset, the experimental results are not reproducible.
-
The benchmark are synthetic, and I am concerned that the LLM models achieving good performance on this synthetic dataset is still robust on the real-world dataset. The authors could evaluate on a small real-world dataset or run some robustness test, such as under noisy or event condition, which will strength the generalizability.
-
In the paper, the authors cliam that CiT-Bench evaluates models' ability to integrate textual context with numerical time-series observations. In the experiment, the authors should evaluate the same LLMs under three input settings: (1) time-series only, (2) textual context only, and (3) both time series and textual context (which is done in the experiment). This aims to verify whether the model can achieve the similar performance only relying on a single modality. If so, the motivation of pairing text with time series becomes weaker.
-
In table 2, the number of queries (400) is larger than number of scenes (100 or 200) for three types of tasks. In the paper, the authors do not provide any details about the data split. I am somehow concerned about the potential data leakage if the training data and test data share the same scene and time series, especially given that the number of queries exceeds the number of scenes.
Questions for authors
Q1. Could the authors clarify whether all queries associated with the same scene are kept together whenever training, validation, or test partitions are constructed?
Q2: Could the authors provide the text-only and time-series-only evaluation to verify the necessity of integrating two modalities?