acceptodds
Under review as a conference paper at ICLR 2027

TS-Haystack: A Multi-Task Retrieval Benchmark for Long-Context Physiological Time Series

Abstract

Time Series Language Models (TSLMs) promise natural-language question answering over real-world temporal data, but their ability to retrieve and relate events in long time-series remains largely untested. We introduce TS-Haystack, a needle-in-a-haystack benchmark of ten event-grounded question-answering tasks over long physiological recordings, with contexts from 100 seconds to 9 hours, spanning direct retrieval, temporal understanding, multi-step retrieval, and contextual anomaly detection. Using this benchmark, we find that existing TSLMs exhibit performance degradation on long-context tasks: accuracy declines with context length, models that directly tokenize time-series without compression run out of memory at every context length, and other TSLMs collapse toward near-zero accuracy in interval retrieval tasks when increasing time-series length. These findings align with existing literature on text and multi-modal long context retrieval. We further contextualize TSLM performances with an oracle baseline and signal-blind baseline, and isolate their failure modes with an Agentic Retrieval for Time Series (ARTS) approach, a Large Language Model (LLM) agent with signal-specific classifiers as tools. Our findings uncover new failure modes in TSLMs at long context and show potential for architectures that further decouple local signal perception from global retrieval. Our code and datasets will be made publicly available upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
time-series, time series benchmark, physiological signals, medical AI, time series language models, long context retrieval, ecg, sleep psg

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5Jq1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors address a fundamental question: whether temporal language models possess the ability to identify transient events—or exhibit other cognitive capabilities—within long-duration physiological recordings. To investigate this, they developed an evaluation benchmark called TS-HAYSTACK. Furthermore, they introduce a model named ARTS. Their experimental results indicate that, across the various configurations evaluated, the models generally demonstrate limited temporal localization capabilities; additionally, some models appear to be influenced by a "prior" regarding the correct answer, which is inadvertently leaked within the model's output prompts.

Critical strengths
  1. TS-HAYSTACK adapts the classic "needle-in-a-haystack" paradigm into a long-context event retrieval and question-answering task for multi-channel physiological recordings, and provides an initial validation of its effectiveness. This is a bold yet logical approach.
  2. During the construction process, the authors distinguish between semi-synthetic activity segments and complete, authentic sleep or ECG data segments. Furthermore, by reusing existing annotations, they have established clear, well-defined objectives for the TS-HAYSTACK framework.
  3. The authors have conducted extensive experiments, with a comprehensive appendix that provides further details on the work. The inclusion of signal-agnostic baselines and "annotation-aware" (or "oracle") baselines offers valuable benchmarks, allowing readers to understand the appropriate points of comparison.
Critical weaknesses
  1. Some explanations are stronger than the evidence provided. The high performance of the baseline—which uses ground-truth annotated timelines—confirms that the task can indeed be solved based on such annotations. However, given that the baseline employs GPT-5.4 while the temporal language models primarily use Qwen3.5, it remains unclear whether the performance gap stems from differences in signal perception/representation or from inherent disparities in the models' overall capabilities.
  2. The effects of context length and sample difficulty have not been fully disentangled. Since evaluation samples of varying lengths are generated separately, an increase in length—particularly in naturally sampled sleep or ECG data—may coincide with changes in the number of events or the nature of the required answers. While Figure 5 offers some analysis, the current curves primarily illustrate performance variations across different length settings; they do not isolate the specific impact of increasing context length itself.
  3. The independent contribution of the iterative tools within ARTS remains ambiguous. During the pre-scanning phase, ARTS utilizes a specialized classifier to process the entire recording and provides a predicted event timeline to the language model. Although the authors present some supporting evidence, they have not isolated the incremental gains provided by subsequent tool calls compared to answering questions directly based on the initial timeline.
  4. In the "Related Work" section, the authors categorize HEARTS under the heading of "short-sequence multiple-choice classification." In reality, HEARTS encompasses long physiological sequences, event localization, and output formats that extend beyond multiple-choice questions. For instance, Table 1 in the original HEARTS paper explicitly indicates that the framework handles sequence lengths ranging from 60 to 1 million. This characterization risks creating a misleading impression regarding both the current work and existing related research.
Questions for authors
  1. Could the authors provide a baseline comparison that uses the same Qwen3.5-9B backbone and takes the ground-truth annotated timeline as input?
  2. For representative tasks such as event localization, could the authors conduct a small-scale controlled experiment—keeping the target events, the number of events, and the questions constant, while varying only the amount of irrelevant background information—to analyze the impact of sequence length?
  3. Could the authors include a baseline that employs the same classifier, timeline prediction mechanism, and language model, but prohibits any subsequent tool calls?
  4. Could the authors provide a more explicit clarification regarding the fundamental differences between HEARTS and TS-HAYSTACK?
2 comments
AuthorsOpenReview ↗

Response to reviewer 5Jq1

We thank the reviewer for a detailed and constructive review, and for recognizing the significant effort in the benchmark's construction and validation.

[W1] and [Q1]: Qwen3.5-9B Oracle.

We run Qwen3.5-9B with the same protocol and labels as the existing GPT 5.4 Oracle.

Summary:

DatasetFlamingo (9B)ITFormer (9B)Oracle Qwen3.5-9BOracle GPT-5.4
Capture240.290.280.890.97
ECG (LTAF)0.340.330.880.96
Sleep0.310.300.920.93

Detailed Oracle results (Qwen 3.5-9B / GPT 5.4, per task and dataset):

TaskCapture24ECG (LTAF)Sleep
Existence1.00 / 1.000.97 / 1.001.00 / 1.00
Localization1.00 / 1.000.86 / 0.990.99 / 1.00
Counting0.96 / 0.970.87 / 0.980.89 / 0.96
Ordering1.00 / 1.000.95 / 1.000.99 / 0.99
State query0.15 / 0.820.98 / 1.000.95 / 0.91
Antecedent1.00 / 0.940.97 / 1.000.86 / 0.90
Comparison0.94 / 0.960.79 / 0.900.90 / 0.91
Multi-hop0.96 / 0.990.50 / 0.840.69 / 0.73
Anomaly detection0.97 / 1.001.00 / 1.000.99 / 1.00
Anomaly localization0.97 / 0.990.94 / 0.940.98 / 0.99

This suggests that the gap to the TSLMs stems mainly from signal perception and representation rather than from the capability of the language model.

[W2] and [Q2]: Marginalization of length.

We run the controlled experiment for Capture24, where our semi-synthetic design allows. In ECG and Sleep datasets, we do not find enough unmodified signals that satisfy the condition for a clean ablation.

ModelMethod100s15min1h
BaselineSignal-blind0.0000.0000.000
BaselineOracle (Qwen3.5-9B)1.0001.0001.000
TSLMOpenTSLM-Flamingo (Qwen3.5-9B)0.0900.0030.000
Qwen3.5-9BReAct coding agent0.1200.0300.007
Qwen3.5-9BARTS (pre-pass only, no tools)0.2270.1030.077
Qwen3.5-9BARTS (no pre-pass, tools only)0.1230.0770.020
Qwen3.5-9BARTS (both)0.2400.1530.107

We discuss these results in our response to reviewer GGWe and our revised manuscript. We additionally provide a new characterization of performance degradation beyond context length alone (see updated Figure 5 and Appendices C.4. and C.5.)

[W3] and [Q3]: Agentic loop ablation

We perform the ablation and discuss results in depth in our commentary on review cLBD. We find that tool improvements largely depend on the dataset, tool quality and usage on the part of LLMs. On Capture24, tools improve the model’s accuracy (except for Qwen 3.5-2B). In ECG and Sleep these tools are rarely used by the model. We further isolate this difference with a randomized re-querying experiment that shows classification improvements beyond the pre-pass in Capture24 only.

[W4] and [Q4]: HEARTS.

We thank the reviewer for noting this. HEARTS covers long physiological sequences in several tasks (around 26% of tasks above 100k datapoints, including the use of Capture24 (Appendix A.2.1 on their paper). HEARTS’s significant contribution tests several healthcare signals and a variety of tasks, and is complementary to TS-Haystack. Of the long context (>100k datapoints) tasks, a mix of whole signal classification, event localization and statistical calculations questions are posed. TS-Haystack is complementary, as it stress-tests retrieval and relation of events (localization, multi-hop, comparison) at long contexts, and while overlapping, provides a new difficulty gradient for models to be evaluated on. The related works section is updated in the revised manuscript.

We thank the reviewer for their constructive review. We have updated the manuscript with the Qwen 3.5-9B Oracle results, tool use ablation, a more detailed discussion isolating context length and signal complexity as drivers of degradation and updated related works. We are happy to run further analysis during the discussion period and would be thankful if the review would revisit their score during this period.

Dear Reviewer 5Jq1,

Thank you for your constructive feedback during the rebuttal period. In response, we added a backbone-matched Qwen3.5-9B Oracle, a length-controlled ablation, an agentic-loop ablation and a revised description of HEARTS in the updated manuscript. Together, these better isolate what drives long-context retrieval performance in time series and where current architectures fail. As today is the final day of the discussion period, we would be grateful to hear whether these responses address your concerns.

Reviewer_cLBDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TS-Haystack for long-context retrieval over physiological time series. It evaluates whether TSLMs can retrieve, localize, and relate sparse events. The main finding is that current TSLMs degrade substantially with context length, especially on localization-style tasks, while oracle and agentic baselines suggest that fine-grained temporal grounding is a major bottleneck.

Critical strengths

S1. The problem is important as many real time series applications are natural long-horizon.

S2. The benchmark design is thoughtful and considers several differences between text and time series events.

S3. The authors provide several useful experimental design, for example, the oracle and signal-blind baselines, and the localization experiments.

Critical weaknesses

W1. The authors should provide more explanations about why long-context temporal grounding fails. I would also suggest the authors do latent-budget or compression-ratio ablations.

W2. In order to attribute the degradation, event count, event density, or sequence complexity should also be considered instead of attributing the degradation to context length.

W3. Why does ARTS uses specific classifiers and an exhaustive pre-pass over the signal? I think in this way the gain can not show that agentic retrieval is better.

W4. A pre-pass-only ablation is also needed.

Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Response to reviewer cLBD

We thank the reviewer for the concise review and the concrete suggestions.

[W1 & W2]: Long context failures and attribution of task accuracy degradation

We ablate and discuss long-context temporal grounding and task difficulty in the additional page of our revised manuscript and appendix.

Concretely, we include an ablation that marginalizes the effects on length on performance degradation at matched difficulty (Figure 6), nuance performance failure drivers in our discussion and update Figure 5 (right) to show the correlation of dataset task degradation next to events per context length, as a new insightful view. We also detail our experiments and results in Appendix C.3, C.4, and C.5.

We run an experiment (see Reviewer GGWe) that keeps the same target event and question and grows only the surrounding recording to better understand the role of long context in model failure. Our semi-synthetic design allows for this ablation in Capture24. On Capture24, OpenTSLM-Flamingo localizes the same 300 targets at 0.09 at 100 s and rarely at 15 min or 1 h. LLMs degrade more, while starting from a higher 100s baseline, aligning with localization results showcased in the existing paper’s experiments.

We further improve our diagnosis of temporal retrieval failure by running a Qwen 3.5 9B Oracle baseline, showcasing a high ceiling for label aggregation by LLMs and locate the failure mode in precise retrieval, which TSLMs, despite integrating the signal natively, do not restore. ARTS, as an experimental diagnostic baseline, improves though remains imperfect and may serve as a naive achievable baseline for future architectures.

[W3] and [W4]: ARTS’s no tools ablation

We agree that ARTS's gains alone do not show that agentic retrieval is better. We propose ARTS as a diagnostic baseline to better diagnose the failure modes of LLMs and TSLMs in TS-Haystack by decoupling local perception from LLM aggregation.

We ablate tool use gains below, and show a similar decomposition in the controlled needle experiment in reviewer GGWe. The classifier pre-pass is the main accuracy driver, and also an efficiency choice (one sweep of inexpensive local classifiers gives the LLM a global view in a single prompt, instead of spending many tool calls to find where to look).

DatasetSettingQwen3.5 2BQwen3.5 9BQwen3.5 35B-A3Bgpt-oss 120BDeepSeek V3.2Gemini 3.8 FlashMedian calls
Capture24Pre-pass0.2970.2200.2170.2160.2080.229–
+ tools0.1980.2680.2510.2430.2790.2669
ECGPre-pass0.2190.4810.4960.4730.4650.501–
+ tools0.2610.4430.4330.476†0.4850.507†1
SleepPre-pass0.2160.355†0.355†0.310†0.3310.359–
+ tools0.2600.3470.3400.2990.346†0.366†1

†: not significant

Results indicate that tool use improvements depend on the dataset and the classification Macro-F1 gains (if any) of reclassifying. In Capture24, models use a median of 9 calls and show improvements in performance (except for Qwen 3.5-2B). In ECG and Sleep, models rarely use the tool (median of one tool call before providing a final answer). Changes in results in these rows are mixed. We find that models use the classifier to confirm the pre-pass, yet in the case of Sleep and ECG reinspection does not yield improved classification.

We ablate this by scoring each classifier against the ground-truth annotations on six long recordings per dataset, comparing the labels from the fixed pre-pass grid with those from 60 randomly placed re-queries per recording (imitating LLM queries).

ClassifierPre-pass grid macro-F1Re-queried ranges macro-F1
Capture240.430.58
ECG rhythm0.820.82
ECG beats0.780.78
Sleep stages0.710.71
Sleep arousals0.280.27

Re-querying improves classification only on Capture24. On Sleep and ECG, it returns mostly the same labels as the pre-pass, so additional tool calls add no new information (details in updated version, Appendix C.14, Table 17).

We add these results in the updated version of the manuscript. We thank the reviewer, as these ablations have helped to better contextualize results in TS-Haystack.

We hope these results have further shown the value of our benchmark and strengthened the interpretation of results. We would be grateful if the reviewer would revisit their score and are happy to run additional analysis during this period.

Thank you for your response. I still have some concerns. The role of compression is unclear and the new experiment is only on Capture24. Is compression the main reason? Any evidence like budget or compression-ratio analyses? For the new experiments, can we conclude that most of gains are from the classifier pre-pass? If so, maybe the authors should consider more when introducing ARTS as evidence for agentic retrieval.

AuthorsOpenReview ↗

Role of compression [1 / 3]

We thank the reviewer for their question and patience as we ran these additional experiments.

On ARTS:

The main benefit comes from the classifier pre-pass and agentic reclassification helps when the classifier improves upon the naive pre-pass window selection. ARTS main objective in TS-Haystack as part of the experimental design is to decompose retrieval and aggregation and provide an achievable baseline from the signal alone for future TSLMs. We will nuance this point in our revision.

On compression:

We isolate in the OpenTSLM-Flamingo and ITFormer architectures the bottleneck for localization, linked to compression. While this analysis goes beyond TS-Haystack as a benchmark, we hope it provides a deeper understanding of where the models fail and will include it in our Appendix.

Summary. In both TSLMs, the information needed for localization is lost in the module that compresses the encoder tokens into a short, fixed set of tokens for the language model: Flamingo's Perceiver resampler and ITFormer's fusion module. Increasing the number of output tokens (64 → 128 in Flamingo; 25 → 50 → 100 in ITFormer) does not seem to recover performance, nor training the same module directly on a localization objective. We generalize this experiment to a pre-trained time-series encoder (Chronos-2 [1]) to confirm this is not an artifact of training on TS-Haystack alone, and find similar results. We attribute the loss to resampling, as these modules pool through learned queries that are not anchored to positions in time, so a short event is averaged with the rest of the recording.

We detail our analysis step by step below.

We thank the reviewer for their engagement. We hope this provides further insights into OpenTSLM and ITFormer's failure modes, which despite the benefit of compression to run at long context for classification, are hindered at localizing events at such lengths. While our scope with TS-Haystack as a benchmark is not that of architectural research, we will update our manuscript to detail our findings on this bottleneck more precisely.

References

[1] Abdul Fatir Ansari, Oleksandr Shchur, Jaris Küken, Andreas Auer, Boran Han, Pedro Mercado, Syama Sundar Rangapuram, Huibin Shen, Lorenzo Stella, Xiyuan Zhang, et al. Chronos-2: From univariate to universal forecasting. arXiv Preprints, 2025. doi: 10.48550/arXiv.2510.15821.

Reviewer_GGWeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces TS-Haystack, a benchmark covering ten event-grounded question-answering tasks over long physiological recordings from activity monitoring, sleep, and ECG datasets. It evaluates time-series language models alongside a signal-blind baseline, an annotation-based oracle, and ARTS, which combines specialized signal classifiers with an LLM orchestrator. The results reveal substantial difficulties in temporal localization and retrieval as recording length increases.

Critical strengths

The benchmark addresses a practically relevant capability: retrieving and relating brief events within hours of physiological recordings. Its coverage of multiple domains and tasks makes it a potentially useful evaluation resource. The signal-blind baseline is particularly valuable because it reveals that some apparently strong results can be achieved without accessing the signal.

Critical weaknesses

The experiments do not sufficiently distinguish basic task failure from degradation caused by longer temporal context.

Several TSLMs already perform poorly at the shortest evaluated contexts. At 100 seconds, localization accuracy is only 0.11/0.09 on Capture24 and 0.04/0.05 on ECG for ITFormer/Flamingo (Tables 2 and 4). Their subsequent decline toward zero offers limited evidence about how an established retrieval capability deteriorates with length. Meanwhile, some higher scores remain below the signal-blind baseline: two-hour ECG ordering accuracy is 0.75/0.74, compared with 0.79 for QA-only.

ARTS and the oracle provide useful comparisons, but they change the perception pipeline or both the input representation and language-model backbone. Their performance gaps therefore do not isolate the source of failure.

The benchmark’s diagnostic contribution would be substantially stronger with demonstrated short-context competence and a matched evaluation that preserves the question, target events, and necessary local context while extending the surrounding recording. As presented, the experiments establish that the tested systems struggle more clearly than they establish what specifically fails because of longer temporal context.

Questions for authors
  1. Does the degradation persist when the same question and target events are evaluated across increasing context lengths, starting from a setting with reliable short-context performance?

  2. Can a same-backbone annotation oracle and a specialized event detector followed by deterministic querying clarify the respective contributions of perception and retrieval failures?

  3. The reproducibility statement contains unresolved references (“Section ??” and “Appendix ??”), and the promised closed-form random baselines appear to be missing.

2 comments
AuthorsOpenReview ↗

Response to reviewer GGWe

We thank the reviewer for their comments, and for noting the usefulness of our contextualization of results. We agree that results showed both that TSLMs and LLMs struggle with the task itself and degrade with context length, with difficult attribution to each due to varying tasks in the real benchmark, run the proposed analysis and include them in our revision.

[W1]: Difficulty-matched length ablation.

We build a set of 300 matched-context localization tasks from Capture24. The same experiment for Sleep and ECG is difficult, as finding appropriate windows that do not shift the target answer in a non semi-synthetic approach do not yield sufficient tasks. For each task, we insert a needle once into a fully annotated 1 h background (at most 30% sleep, no annotation gap longer than 3 s) and cut windows of 100 s, 15 min and 1 h around it, with the needle at a uniformly random position in each window. The question and target needle (taken from the benchmark localization template) remain fixed as irrelevant context grows. Trained TSLMs are evaluated zero-shot. We include an additional baseline “coding agent”, with a simple harness with the signal as an external file, and a python sandbox with numpy, seaborn and matplotlib.

ModelMethod100s15min1h
BaselineSignal-blind0.0000.0000.000
BaselineOracle (Qwen3.5-9B)1.0001.0001.000
TSLMOpenTSLM-Flamingo (Qwen3.5-9B)0.0900.0030.000
Qwen3.5-9BReAct coding agent0.1200.0300.007
Qwen3.5-9BARTS (pre-pass only, no tools)0.2270.1030.077
Qwen3.5-9BARTS (no pre-pass, tools only)0.1230.0770.020
Qwen3.5-9BARTS (both)0.2400.1530.107

LLMs show clear degradation with length, sustaining a certain performance at longest lengths with the help of specialized classifiers. The use of classifiers as tools for agentic reinspection also shows improvement in this ablation; we further discuss their utility in our rebuttal for [cLBD]. TSLMs, while degrading in accuracy (9% → 0%), begin from a lower baseline than ARTS or other LLM formulations. Results suggest that increasing context does play a role in LLM and TSLM performance. We nonetheless deepen our characterization of model failures by showcasing additional difficulty factors present in natural signals in the revised manuscript (revised Figure 5 and discussion, details in Appendix C.4 and C.5).

We sincerely thank the review for the suggestion, as it has strengthened our diagnostic contribution, and incorporate these results in our updated results and discussion.

[W2]: Qwen 3.5-9B Oracle and SQL retrieval

We evaluate a deterministic querying pipeline with SQL over the real labels (SQL labels) and classifier predictions (SQL classifier), along with a backbone-matched oracle. We also run a base Qwen 3.5-9B as Oracle (full results table in our response to reviewer 5Jq1).

DatasetSQL, labelsOracle GPT-5.4Oracle Qwen3.5-9BARTS Qwen3.5-9BSQL, classifier
Capture241.000.970.890.270.18
ECG (LTAF)1.000.960.880.440.38
Sleep1.000.930.920.350.34

Oracle: Oracle Qwen3.5-9B scores lower than Oracle GPT 5.4 yet remains capable (above 88%) of recovering the answer from labels in all datasets. This suggests that the gap in LLMs and TSLMs lie mainly in perception and temporal grounding rather than in the language model.

Programmatic pipeline: “SQL, labels” confirm the pipeline is well calibrated and questions well posed. The “SQL, classifier” shows lower performance, as it answers from imperfect classified segments and may either answer wrongly, or not provide an answer at all if a pre-required label is not classified (e.g. antecedent needs to anchor on one label to answer the targeted class).

[Q3]: reproducibility statement and random baselines.

We thank the reviewer for spotting this and fix it in the revised manuscript. We report the closed-form random baselines and majority-class reference. Full table and method shown in Appendix C.6.1 in the posted revision.

DatasetRandom (closed form)Majority answerSignal-blindFlamingo (9B)
Capture240.180.240.250.29
ECG (LTAF)0.210.280.340.34
Sleep0.220.240.320.31

On tasks that can be answered from label and transition statistics (ordering, state query, antecedent), the signal-blind model is above the random baseline, (e.g. 0.74 against 0.20 on Sleep antecedent).

We hope these experiments address the reviewer's concerns. The experiments have improved our interpretation of results and conclusions and hope they further show the value of TS-Haystack as a benchmark for future models to evaluate. We would be grateful if the reviewer would consider revisiting the score, and are happy to run further analyses during the discussion period.

Dear Reviewer GGWe,

Thank you for your engagement during the rebuttal period and for your constructive feedback. Following your suggestions, we ran a matched-context localization ablation that keeps the question and target event fixed while only the surrounding recording grows. We also added a same-backbone Qwen3.5-9B Oracle and a deterministic SQL pipeline over both labels and classifier predictions and deep dive into TSLM failure modes in our thread with reviewer cLBD.

Together, these help separate perception failures from retrieval failures and from degradation with context length, which was your central concern. We have also fixed the unresolved references and added the closed-form random and majority-class baselines. As today is the final day of the discussion period, we would be grateful to hear whether these additions address your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.