- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors address a fundamental question: whether temporal language models possess the ability to identify transient events—or exhibit other cognitive capabilities—within long-duration physiological recordings. To investigate this, they developed an evaluation benchmark called TS-HAYSTACK. Furthermore, they introduce a model named ARTS. Their experimental results indicate that, across the various configurations evaluated, the models generally demonstrate limited temporal localization capabilities; additionally, some models appear to be influenced by a "prior" regarding the correct answer, which is inadvertently leaked within the model's output prompts.
Critical strengths
- TS-HAYSTACK adapts the classic "needle-in-a-haystack" paradigm into a long-context event retrieval and question-answering task for multi-channel physiological recordings, and provides an initial validation of its effectiveness. This is a bold yet logical approach.
- During the construction process, the authors distinguish between semi-synthetic activity segments and complete, authentic sleep or ECG data segments. Furthermore, by reusing existing annotations, they have established clear, well-defined objectives for the TS-HAYSTACK framework.
- The authors have conducted extensive experiments, with a comprehensive appendix that provides further details on the work. The inclusion of signal-agnostic baselines and "annotation-aware" (or "oracle") baselines offers valuable benchmarks, allowing readers to understand the appropriate points of comparison.
Critical weaknesses
- Some explanations are stronger than the evidence provided. The high performance of the baseline—which uses ground-truth annotated timelines—confirms that the task can indeed be solved based on such annotations. However, given that the baseline employs GPT-5.4 while the temporal language models primarily use Qwen3.5, it remains unclear whether the performance gap stems from differences in signal perception/representation or from inherent disparities in the models' overall capabilities.
- The effects of context length and sample difficulty have not been fully disentangled. Since evaluation samples of varying lengths are generated separately, an increase in length—particularly in naturally sampled sleep or ECG data—may coincide with changes in the number of events or the nature of the required answers. While Figure 5 offers some analysis, the current curves primarily illustrate performance variations across different length settings; they do not isolate the specific impact of increasing context length itself.
- The independent contribution of the iterative tools within ARTS remains ambiguous. During the pre-scanning phase, ARTS utilizes a specialized classifier to process the entire recording and provides a predicted event timeline to the language model. Although the authors present some supporting evidence, they have not isolated the incremental gains provided by subsequent tool calls compared to answering questions directly based on the initial timeline.
- In the "Related Work" section, the authors categorize HEARTS under the heading of "short-sequence multiple-choice classification." In reality, HEARTS encompasses long physiological sequences, event localization, and output formats that extend beyond multiple-choice questions. For instance, Table 1 in the original HEARTS paper explicitly indicates that the framework handles sequence lengths ranging from 60 to 1 million. This characterization risks creating a misleading impression regarding both the current work and existing related research.
Questions for authors
- Could the authors provide a baseline comparison that uses the same Qwen3.5-9B backbone and takes the ground-truth annotated timeline as input?
- For representative tasks such as event localization, could the authors conduct a small-scale controlled experiment—keeping the target events, the number of events, and the questions constant, while varying only the amount of irrelevant background information—to analyze the impact of sequence length?
- Could the authors include a baseline that employs the same classifier, timeline prediction mechanism, and language model, but prohibits any subsequent tool calls?
- Could the authors provide a more explicit clarification regarding the fundamental differences between HEARTS and TS-HAYSTACK?
2 comments
Dear Reviewer 5Jq1,
Thank you for your constructive feedback during the rebuttal period. In response, we added a backbone-matched Qwen3.5-9B Oracle, a length-controlled ablation, an agentic-loop ablation and a revised description of HEARTS in the updated manuscript. Together, these better isolate what drives long-context retrieval performance in time series and where current architectures fail. As today is the final day of the discussion period, we would be grateful to hear whether these responses address your concerns.
Response to reviewer 5Jq1
We thank the reviewer for a detailed and constructive review, and for recognizing the significant effort in the benchmark's construction and validation.
[W1] and [Q1]: Qwen3.5-9B Oracle.
We run Qwen3.5-9B with the same protocol and labels as the existing GPT 5.4 Oracle.
Summary:
Detailed Oracle results (Qwen 3.5-9B / GPT 5.4, per task and dataset):
This suggests that the gap to the TSLMs stems mainly from signal perception and representation rather than from the capability of the language model.
[W2] and [Q2]: Marginalization of length.
We run the controlled experiment for Capture24, where our semi-synthetic design allows. In ECG and Sleep datasets, we do not find enough unmodified signals that satisfy the condition for a clean ablation.
We discuss these results in our response to reviewer GGWe and our revised manuscript. We additionally provide a new characterization of performance degradation beyond context length alone (see updated Figure 5 and Appendices C.4. and C.5.)
[W3] and [Q3]: Agentic loop ablation
We perform the ablation and discuss results in depth in our commentary on review cLBD. We find that tool improvements largely depend on the dataset, tool quality and usage on the part of LLMs. On Capture24, tools improve the model’s accuracy (except for Qwen 3.5-2B). In ECG and Sleep these tools are rarely used by the model. We further isolate this difference with a randomized re-querying experiment that shows classification improvements beyond the pre-pass in Capture24 only.
[W4] and [Q4]: HEARTS.
We thank the reviewer for noting this. HEARTS covers long physiological sequences in several tasks (around 26% of tasks above 100k datapoints, including the use of Capture24 (Appendix A.2.1 on their paper). HEARTS’s significant contribution tests several healthcare signals and a variety of tasks, and is complementary to TS-Haystack. Of the long context (>100k datapoints) tasks, a mix of whole signal classification, event localization and statistical calculations questions are posed. TS-Haystack is complementary, as it stress-tests retrieval and relation of events (localization, multi-hop, comparison) at long contexts, and while overlapping, provides a new difficulty gradient for models to be evaluated on. The related works section is updated in the revised manuscript.
We thank the reviewer for their constructive review. We have updated the manuscript with the Qwen 3.5-9B Oracle results, tool use ablation, a more detailed discussion isolating context length and signal complexity as drivers of degradation and updated related works. We are happy to run further analysis during the discussion period and would be thankful if the review would revisit their score during this period.