acceptodds
Under review as a conference paper at ICLR 2027

Bridging the Modality Gap between Time Series and Language for Cognitive Reasoning

Abstract

Time series analysis plays a central role in understanding dynamic systems and supporting forecasting, early warning, and decision-making. Recent studies have begun integrating time series with large language models (LLMs), yet most existing tasks remain focused on fixed-output prediction or state modeling, leaving the language understanding and reasoning capabilities of LLMs underexplored. Meanwhile, textualization-based approaches often fragment compact temporal structures and introduce substantial context overhead, while existing benchmarks provide limited reasoning depth and semantic diversity. To address these limitations, we introduce ***TSCognition***, a multimodal benchmark for comprehensive time series reasoning. It integrates real-world time series and textual information from ***15*** public sources and contains approximately ***41K*** question-answer pairs spanning five reasoning tasks: **Decoding**, **Grounding**, **Inferring**, **Extrapolating**, and **Acting**. Based on this benchmark, we further propose ***TSAlign***, a unified framework that directly aligns time series representations with the language representation space, enabling efficient and structured interaction between temporal signals and LLMs. Extensive experiments show that TSAlign consistently outperforms existing LLM-, VLM-, and time series QA-based methods on both TSCognition and the public TimerBed benchmark, while substantially reducing computational overhead. These results highlight representation alignment as an effective direction for scalable time series–language reasoning.

Primary area:
learning on time series and dynamical systems
Keywords:
Modality Gap, Time Series Reasoning, Large Language Models for Time Series

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UbVyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TSCognition, a benchmark of about 41K multiple choice QA samples built from 15 public sources spanning 8 domains and five reasoning task categories, and TSAlign, a framework that encodes multivariate time series into patch level representations, aligns them with principal directions of the LLM embedding matrix via attention and a gated residual, and fuses variables through a second gating mechanism. TSAlign is trained on 80% of TSCognition and reported to outperform LLM, VLM, and time series QA baselines on TSCognition, and to achieve the best overall performance on the external TimerBed benchmark for pattern analysis across six datasets.

Critical strengths

The dataset construction is serious work: 15 public sources, 8 domains, multivariate series of several hundred to about a thousand steps, and a train evaluation split with an out of distribution slice, assembled with a documented cleaning and quality control pipeline. Reporting results on TimerBed, a benchmark the authors did not create, is the single most credible piece of empirical evidence in the paper. The ablations over representation granularity, alignment strategy, and fusion strategy are systematic and easy to follow, and the efficiency analysis is complete about what is being compared.

Critical weaknesses

First, a factual error in the novelty claim. Section 3.3 states that TSCognition is, to the authors' knowledge, the first multimodal dataset for time series unifying multi-dimensional data, long-horizon dynamics, and cognitively structured reasoning tasks. This cannot be squared with the paper's own bibliography. MTBench (Chen et al., 2025), cited in this submission, is explicitly a multimodal time series benchmark for temporal reasoning and question answering across many datasets. TsRBench (Yu et al., 2026), also cited, is a multi-task, multi-modal time series reasoning benchmark for generalist models with compositional questions. TimeOmni-1 (Guan et al., 2025), again cited, already targets complex reasoning over time series. Either the claim rests on a bespoke reading of cognitively structured that the paper never defines against these predecessors, or it is simply wrong. A first-of-kind claim that the authors' own reference list refutes is not a wording issue; it is the load bearing sentence of the contribution, and it is false as written.

Second, the experimental labeling is misleading. The paper reports that under the zero-shot setting TSAlign-3B surpasses baselines on the majority of tasks. TSAlign contains a trained patch encoder, a projection layer, and two gating mechanisms; nothing about it can be evaluated zero-shot in the sense that term has in this literature, namely no task specific training. What the paper calls zero-shot for its own method can only mean trained on the 32K sample TSCognition train split but not separately tuned per task category, while the same table pits it against GPT-5.1, which is genuinely zero-shot and has seen none of these 41K items. Presenting a model fine tuned on 32K in-distribution samples and a no-training frontier model under one zero-shot label, then advertising that TSAlign-7B beats GPT-5.1 by 17.7% on average, compares training data against model capability and licenses none of the abstract's conclusions about what LLMs can do with time series. This is not loose language; it is a category error in the central comparison.

Third, the statistical language is not supported by the protocol. The paper says significantly outperforming for the TimerBed results and uses similar phrasing for the TSCognition margins, yet every experiment in the submission runs a single random seed (2026, per the appendix), with no error bars, no repeated runs, and no significance test anywhere in the main text or tables. Significant is a statistical claim, and there is no statistics behind it. The paper's two headline empirical sentences, beating GPT-5.1 and significantly outperforming traditional time series models, are both stated in a form the evidence does not support.

Fourth, the evaluation on TSCognition is circular in the place it matters most. The claim that TSAlign outperforms LLM, VLM, and time series QA baselines is established on a benchmark the authors built and on which their method is fine tuned, in a multiple choice format where every distractor is constructed from local patterns and common misinterpretations, which is precisely the artifact distribution a scorer trained on 32K such items learns to detect without reasoning about the series. No open ended generation, free form justification, or calibration check appears, so the paper's core verb, reasoning, is never tested in a form that requires it.

Fifth, the method is standard parts, and one named component misdescribes itself. Principal semantic directions are the principal components of the token embedding matrix, and the submission's own cited literature on representation geometry establishes that top principal components of embedding matrices encode common components and frequency artifacts rather than interpretable semantics; calling this subspace semantic is a mischaracterization of a linear algebra object the authors themselves cite. Patch encoding is PatchTST, the attention plus gated residual is a standard adapter, and variable fusion is a weighted average. The efficiency wins are measured against text serialization and per variable line plots, representations nobody productionizes for multivariate series QA, so the 16.8x and 3.0x figures do not speak to any real deployment decision. Finally, the reference list is dense with self citations from one research line, which effectively de anonymizes the submission.

Questions for authors
  1. Given that MTBench, TsRBench, and TimeOmni-1 are all cited in this submission and all precede it, what is the precise, checkable definition of cognitively structured reasoning under which the first-of-kind claim in Section 3.3 is true?
  2. What are the TSCognition numbers when all baselines are fine tuned on the same 32K training split, and what happens to the 17.7% average margin over GPT-5.1 once the comparison is restated as fine-tuned versus zero-shot rather than implied capability?
  3. Can you provide multi-seed runs with variance for the main tables, or withdraw the word significantly from claims that currently have no statistical test behind them?
Reviewer_U3nBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TSCognition, a multimodal benchmark for time series reasoning, covering five cognitive reasoning tasks across diverse real-world domains. Building on this, the authors propose TSAlign, which bridges the modality gap between time series and language through patch-level encoding, semantic alignment, and multivariate fusion. Extensive experiments on TSCognition and TimerBed demonstrate the effectiveness and efficiency of the proposed method compared with existing baselines.

Critical strengths

The paper is overall well-structured and relatively complet. It introduces both a new benchmark, TSCognition, and a corresponding alignment framework, TSAlign, to address the challenges of time series reasoning from both data and methodology perspectives. The authors also conduct systematic experiments, including comparisons across different models and comprehensive ablation studies, to validate the effectiveness of the proposed approach. Overall, this work provides a valuable contribution to time series reasoning with LLMs.

Critical weaknesses

1. Outdated positioning of TSCognition. The limitations of existing benchmarks discussed in Section 3.2 appear outdated and overstated. (1) Multivariate data: EngineMT-QA and ChatTS already support multivariate time series. (2) Sequence length: EngineMT-QA (600), TSQA (64–512), and ChatTS (~64–1500) go well beyond the claimed 50–125 steps. (3) Task complexity: MMTS-Bench and TimeSeriesExam already include complex reasoning tasks. (4) Domain diversity: Time-MQA, ChatTS, and MMTS-Bench cover diverse real-world and synthetic domains. These prior works should be properly acknowledged to justify the novelty of TSCognition.

2. Limited methodological novelty and insufficient support for high-level alignment. TSAlign largely follows the established paradigm of projecting time series features into an LLM, as explored in ChatTS, ITFormer, and Towards Time-Series Reasoning with LLMs (2409.11376). While PCA, gated residual injection, and multivariate fusion provide incremental modifications, the experiments do not convincingly demonstrate that these components achieve high-level semantic alignment rather than simply improving low-level feature representation. The authors should better clarify the unique methodological contribution and provide stronger evidence for their claims.

Questions for authors

1. Semantic validity of PCA-based alignment. PCA maximizes embedding variance rather than semantic interpretability. How can the authors demonstrate that the extracted principal directions capture meaningful and transferable semantic knowledge for time series reasoning?

2. Insufficient evidence for high-level semantic alignment. Although the authors conduct ablations on different alignment strategies, these results do not directly support the claim that TSAlign improves high-level semantic understanding rather than low-level feature representation. Could the authors provide more direct evidence?

3. Effectiveness of gated multivariate fusion. The authors argue that sigmoid gating avoids the competitive normalization of softmax. However, the final fusion still normalizes the weights across variables. What fundamentally explains the substantial performance gap between the two strategies in Table 6?

Reviewer_hNHFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces TSCognition, which covers five distinct cognitive tasks and integrates time-series data with contextual information.

Critical strengths
  1. An efficient architecture for an important task The authors address a key limitation of LLM-based time-series models: their narrow focus on individual downstream tasks, often at the expense of broader analysis and understanding. They propose TSAlign, which combines patch-based time-series encoding with an LLM. This approach reduces the required context length, enabling more efficient processing and a smaller memory footprint than text-based and vision-based representations of time series.
  2. A comprehensive new benchmark TSCognition contains question–answer pairs derived from 15 source datasets spanning eight domains. It covers several reasoning categories, with an emphasis on multivariate and long-context time series, making it a valuable contribution to the community.
Critical weaknesses

Weaknesses

  1. Exclusive reliance on multiple-choice questions The multiple-choice format may allow models to arrive at correct answers without fully reasoning about the underlying time series. A model could instead compare the available options and select the most plausible interpretation. Including open-ended questions, such as those requiring numerical answers, would provide a more informative assessment of the model’s reasoning capabilities. Given that the authors motivate their work by emphasizing the need for deeper understanding, tasks requiring free-form natural-language responses could also be appropriate.
Questions for authors

Questions

  1. How does the backbone LLM distinguish between the channels in a multivariate time series?
  2. Can the authors provide an experiment in which TSAlign generates an answer rather than selecting from predefined options? Such an experiment would provide stronger evidence for the claimed reasoning capabilities.
  3. To what extent does TSAlign’s time-series encoding preserve fine-grained details of the input signal? Can the original signal be reconstructed from this representation without substantial error? A case study examining this would help clarify what information the encoding retains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.