- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces READ-Bench which is a benchmark for historical-case retrieval across 12 time-series diagnostic datasets spanning five domains. The angle of the benchmark focuses on shared faults/event types rather than pure time-series behavior. The authors then compare classical distances, symbolic methods, pretrained time-series embeddings, fusion strategies, and rerankers to provide insights on performance.
Critical strengths
-
The paper is clearly structured, and the progression from benchmark construction to base retrieval, fusion, and reranking makes the experiments easy to follow. I also liked the diagrams and well-desgined figures of the paper.
-
The benchmark is useful in a sense that by defining relevance by shared fault type rather than signal shape, it better reflects the goal of retrieving diagnostically relevant historical cases (not just similar time-series behavior).
Critical weaknesses
-
The practical assumptions behind label-aware reranking need more analysis. Particularly, the strongest results rely on a Gaussian-process reranker that uses labels from a small set of candidates. The paper should make the label budget and selection procedure explicit and examine how performance changes as the number and quality of resolved cases vary.
-
Additionally, given the use of labels from a small set of candidates by the GPC reranker, I am not sure how fair is it to compare it with the other language-model rerankers method. Perhaps, these methods also get to train with a few labels to provide more fairness and see where the gap in performance comes from, supervision or the actual method.
-
All embedders are forced into a channel-independent, mean-pooled, resampled setup, which can discard the channel-localized evidence that matters for diagnosis. The claims that pretrained representations offer “no advantage” and that LLMs provide “almost no benefit” are therefore much broader than this setup supports (I'd avoid such a broader claims).
Questions for authors
- Could you compare them under more comparable supervision settings, for example by providing the language models with the same labeled examples?
- How many labeled candidates does the GPC reranker use per query, and how are these candidates selected? Could you evaluate how its performance changes with the number and quality of available labels?