acceptodds
Under review as a conference paper at ICLR 2027

READ-Bench: Benchmarking Historical Instance Retrieval for Time-Series Diagnosis

Abstract

Time-series diagnostic systems rarely rely on retrieving relevant historical cases, and when they do, retrieval is typically evaluated only indirectly through downstream prediction. We introduce **READ-Bench**, a benchmark for historical-case retrieval across 12 diagnostic datasets, centered on multivariate time series, that defines relevance by shared fault or event type rather than signal shape, so that visually different traces of the same fault count as relevant while similar-looking traces of different faults do not. Organizing existing datasets into queries, corpora, and explicit relevance judgments, READ-Bench is, to our knowledge, the broadest testbed to date for historical-case retrieval across diagnostic domains. Analogous to retrieval-augmented generation, we treat retrieval as a base retriever followed by a reranker, evaluating classical distances, symbolic retrievers, self-supervised and foundation-model embedders, and their fusion for search, and label-aware and language-model rerankers for reranking, under one protocol that varies supervision, normal-series pollution, and corpus scale with significance testing across datasets. Under a common channel-independent retrieval interface, pretrained representations offer no statistically detectable advantage over strong classical and symbolic baselines for search alone. The decisive factor is instead a small amount of resolved-case supervision at reranking, namely a Gaussian-process reranker that propagates a few neighbor labels in the embedding space and improves rankings far more than swapping among more sophisticated unsupervised representations or language-model reasoning, a gain that holds under corpus pollution and at full corpus scale. Guided by these findings, we introduce normal-residual scoring, which ranks each window by its departure from normal operation, and build a system that fuses a normal-residual-scored foundation-model embedder with a dynamic time warping leg by reciprocal-rank fusion, then reranks with the label-aware Gaussian-process reranker. This system improves NDCG@10 over its own search stage on all 12 datasets, by from the reranking step alone and by over the strongest single base retriever applied uniformly across datasets.

Primary area:
learning on time series and dynamical systems
Keywords:
time-series retrieval, retrieval benchmark, time-series diagnosis, reranking, time-series foundation models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sVVXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces READ-Bench which is a benchmark for historical-case retrieval across 12 time-series diagnostic datasets spanning five domains. The angle of the benchmark focuses on shared faults/event types rather than pure time-series behavior. The authors then compare classical distances, symbolic methods, pretrained time-series embeddings, fusion strategies, and rerankers to provide insights on performance.

Critical strengths
  • The paper is clearly structured, and the progression from benchmark construction to base retrieval, fusion, and reranking makes the experiments easy to follow. I also liked the diagrams and well-desgined figures of the paper.

  • The benchmark is useful in a sense that by defining relevance by shared fault type rather than signal shape, it better reflects the goal of retrieving diagnostically relevant historical cases (not just similar time-series behavior).

Critical weaknesses
  • The practical assumptions behind label-aware reranking need more analysis. Particularly, the strongest results rely on a Gaussian-process reranker that uses labels from a small set of candidates. The paper should make the label budget and selection procedure explicit and examine how performance changes as the number and quality of resolved cases vary.

  • Additionally, given the use of labels from a small set of candidates by the GPC reranker, I am not sure how fair is it to compare it with the other language-model rerankers method. Perhaps, these methods also get to train with a few labels to provide more fairness and see where the gap in performance comes from, supervision or the actual method.

  • All embedders are forced into a channel-independent, mean-pooled, resampled setup, which can discard the channel-localized evidence that matters for diagnosis. The claims that pretrained representations offer “no advantage” and that LLMs provide “almost no benefit” are therefore much broader than this setup supports (I'd avoid such a broader claims).

Questions for authors
  • Could you compare them under more comparable supervision settings, for example by providing the language models with the same labeled examples?
  • How many labeled candidates does the GPC reranker use per query, and how are these candidates selected? Could you evaluate how its performance changes with the number and quality of available labels?
Reviewer_7Wc8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces READ-Bench, a benchmark for retrieving historical cases for time-series diagnosis, built from 12 datasets with relevance defined by shared fault type rather than signal shape. It benchmarks classical distances, symbolic retrievers, foundation-model embedders, fusion, and label-aware and language-model rerankers under one protocol with significance testing, finding that base retrievers are indistinguishable while label-aware reranking and a proposed normal-residual scoring give the main gains. A composed system outperforms the best single retriever.

Critical strengths

S1. A useful and carefully constructed benchmark. Defining relevance by fault type rather than shape, covering five diagnostic domains with heterogeneous channel counts, lengths and class imbalance, enforcing coarse-grained train/test splits, controlling normal-window pollution, and testing significance across datasets together make this a comprehensive testbed for diagnostic time-series retrieval.

S2. Clear, well-supported negative and positive findings that are noteworthy: frozen time-series foundation-model embeddings do not beat DTW or SAX-BM25 for retrieval. Language-model rerankers add essentially nothing. The simple normal-residual scoring is a cheap, encoder-agnostic improvement that is robust to pollution. These are actionable for practitioners building retrieval-augmented diagnosis systems.

Critical weaknesses

W1. The "small amount of resolved-case supervision" claim in the abstract overstates what the label-aware rerankers use. GPC, Majority-Vote and QPurity are given the fault labels of every item in the top-20 pool, which in practice means a fully labeled corpus. With labels on all candidates, the reranker amounts to predicting the query's class from its labeled neighbours and promoting same-class items, so the gain is expected and is nearly matched by Majority-Vote in Table 13 on CHARM+NR and TiRex+NR. No experiment varies the fraction of labeled corpus items, which is the setting the text motivates "some resolved incidents have confirmed labels" (line 296). The abstract and Section 4.3 should be reframed accordingly, and a label-fraction sweep is needed to support the "small amount" wording.

W2. The proposed system is a modest contribution relative to the benchmark. It is a composition of existing pieces, e.g., NR is essentially a local mean-centering of embeddings, RRF fusion with DTW, and a GPC/kNN reranker. The best configuration is selected on the same 12 datasets it is evaluated on, and a stronger reranked single pool is not the one fused in the final system without explanation. The paper would be stronger if positioned purely as a benchmark and analysis paper with the system as a reference baseline.

Questions for authors

Q1. It is better to add a sweep over the fraction of labeled corpus items, e.g., 5%, 10%, 25%, 100%, for GPC and Majority-Vote. How quickly does the reranking gain disappear?

Q2. Were NR's k=30, the GPC settings, and the fusion leg chosen on the same 12 datasets reported in Table 1? If so, it is better to state this and, if possible, report a leave-one-dataset-out selection.

Q3. For the FM embedders, windows are mean-pooled over channels before encoding (Section 3.3), which discards channel identity on datasets with hundreds of channels. How much of the "no advantage of pretrained representations" conclusion (line 473) depends on this channel-independent interface rather than on the encoders themselves?

Q4. Why is CHARM rather than MantisV2, which is the strongest NR base retriever, used in the final system, and what does MantisV2+NR | DTW-I + GPC achieve?

Reviewer_Fq3rOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper reformulates 12 existing time-series datasets as historical-case retrieval tasks, where relevance is determined by shared fault or event labels. It compares various retrieval methods and proposes a pipeline combining normal-residual scoring, DTW-based fusion, and Gaussian-process reranking. The authors claim that pretrained representations offer limited advantages over conventional methods, while incorporating a small amount of label information substantially improves retrieval performance. The experimental results generally support the effectiveness of label-aware reranking. I find the broad empirical comparison useful, but the benchmark's fundamental task definition is questionable. Since relevance is entirely determined by class-label agreement, it is unclear how the task meaningfully differs from conventional classification followed by class-based ranking. Furthermore, sharing the same fault label does not necessarily imply genuine diagnostic relevance, which limits the practical significance of the reported improvements.

Critical strengths

S1. Meaningful Problem Motivation

The paper addresses historical-case retrieval for time-series diagnosis, an important but relatively underexplored problem. Evaluating retrieval quality independently of downstream diagnostic performance is a meaningful direction.

S2. Broad and Diverse Benchmark Construction

READ-Bench integrates 12 datasets spanning multiple diagnostic domains, with substantial variation in channel counts, sequence lengths, and fault categories. This provides a useful foundation for evaluating retrieval methods across heterogeneous time-series settings.

S3. Comprehensive Empirical Evaluation

The paper systematically compares classical distances, symbolic methods, foundation-model representations, fusion strategies, and reranking approaches. The analysis of supervision, normal-series pollution, and retrieval components provides useful empirical insights.

Critical weaknesses

W1. Insufficient Distinction from Time-Series Classification

The benchmark defines relevance entirely through fault-class equality. Consequently, a supervised classifier can predict the query's fault category and rank historical instances according to their known class labels. It remains unclear what distinct retrieval capability the proposed benchmark evaluates beyond conventional fault-type classification.

W2. Limited Validity of Diagnostic Relevance

Sharing the same fault label does not necessarily imply that two historical cases are equally useful for diagnosing a new incident. However, the benchmark does not account for operating conditions, incident metadata, verified root causes, or intervention outcomes that may determine the actual relevance of historical cases. Therefore, the current relevance definition does not sufficiently capture the diagnostic utility of retrieved historical cases.

W3. Insufficient Justification for Fault-Specific Retrieval

The proposed normal-residual scoring assumes that subtracting nearby normal embeddings reveals fault-relevant information, but this assumption is not sufficiently validated. Furthermore, channel-independent embedding aggregation may discard important multivariate dependencies and channel identities. Consequently, the paper does not sufficiently establish whether the proposed representations preserve the information necessary to distinguish different fault mechanisms.

W4. Limitations in Experimental Validity and Generalization

Several major conclusions are not adequately supported by the experimental design. In particular, LLM rerankers are instructed to assess waveform similarity, whereas the benchmark defines relevance through shared fault types. Furthermore, the practical supervision budget and the scalability of the complete retrieval pipeline are not sufficiently evaluated. These limitations make it difficult to generalize the reported findings to diagnostic retrieval settings beyond the specific experimental conditions considered.

W5. Limited Reproducibility and Benchmark Accessibility

The authors state that the curated corpora, query sets, relevance judgments, and implementation code will be released upon acceptance. However, these artifacts are not available in the provided submission materials.*Given that benchmark construction is a central contribution, the lack of accessible artifacts limits independent verification and reproducibility of the benchmark construction process and reported experimental results.

Questions for authors
  1. If relevance is defined solely by matching fault-class labels, why not solve this as a standard time-series classification problem? The proposed retrieval task appears to be a reformulation of an existing classification problem.

  2. Why are LLM rerankers instructed to evaluate waveform similarity when similar waveforms do not necessarily indicate the same fault type? This seems inconsistent with the paper's definition of diagnostic relevance.

  3. Do the labels across all 12 datasets represent verified underlying fault causes, or merely observable symptoms and event categories? If the latter, how does the benchmark establish genuine diagnostic relevance beyond class-label matching?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.