acceptodds
Under review as a conference paper at ICLR 2027

Thoth: Mid-Training Bridges LLMs to Time Series Understanding

Abstract

Large Language Models (LLMs) have demonstrated remarkable success in general-purpose reasoning. However, they still struggle to understand and reason about time series data, which limits their effectiveness in decision-making scenarios that depend on temporal dynamics. In this paper, we propose Thoth, the first family of mid-trained LLMs with general-purpose time series understanding capabilities. As a pivotal intermediate stage, mid-training achieves task- and domain-agnostic alignment between time series and natural language, for which we construct Book-of-Thoth, a high-quality, time-series-centric mid-training corpus. Book-of-Thoth enables both time-series-to-text and text-to-time-series generation, equipping LLMs with a foundational grasp of temporal patterns. To better evaluate advanced reasoning capabilities, we further present KnoTS, a novel benchmark of knowledge-intensive time series understanding, designed for joint reasoning over temporal patterns and domain knowledge. Extensive experiments demonstrate that mid-training with Book-of-Thoth enables Thoth to significantly outperform its base model and advanced LLMs across a range of time series question answering benchmarks. Moreover, Thoth exhibits superior capabilities when fine-tuned under data scarcity, underscoring the effectiveness of mid-training for time series understanding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, mid-training, time series understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hDAwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • Instead of adapting LLMs to time series through task-specific fine-tuning, the authors explore mid-training, an intermediate step between pre-training and post-training, to give LLMs a more general understanding of time series.

  • They take pretrained Qwen 3 models and train them on Book-of-Thoth, a dataset containing time series paired with natural language descriptions. The training focuses on both describing time series in text and generating time series from text.

  • Mid-training substantially improves time-series understanding. For example, trend recognition accuracy improves for the 8B model, although performance gains vary across tasks.

  • When fine-tuned on some of downstream training data, the mid-trained models generally perform better than directly fine-tuning the original LLMs, suggesting that mid-training helps models learn new time-series tasks with less data.

Critical strengths

S1. This work explores mid-training as a middle ground between general-purpose pre-training and task-specific fine-tuning, allowing LLMs to develop a broader understanding of time series before adapting to specific downstream tasks.

S2. The authors introduce Book-of-Thoth, a large-scale dataset of time-series–text pairs that supports learning in both directions (time series to text and text to time series). The automated data generation process also makes the approach relatively scalable.

S3. The experimental results show substantial improvements over the original Qwen3 models, particularly in recognizing temporal patterns. The evaluation across multiple benchmarks also helps demonstrate the usefulness of mid-training beyond a single task.

S4. The downstream fine-tuning experiments are particularly interesting, as Thoth generally outperforms the original Qwen3 model when both are fine-tuned using only five percent of the available training data. This provides further evidence that mid-training can help models adapt to downstream tasks with limited labeled data.

Critical weaknesses

W1. While the paper shows clear improvements over the original Qwen3 models, it would benefit from a more direct comparison with other training approaches under similar computational budgets. Without this, it is difficult to tell how much of the improvement comes specifically from mid-training rather than simply training the models on additional time-series data.

W2. Although the results are promising, the improvements are not consistent across all tasks, particularly classification. Additionally, the proposed KnoTS benchmark is relatively small and relies on LLM-based evaluation without reporting agreement with human evaluators. Further experiments on more diverse real-world datasets, along with human validation of the evaluation scores, would help strengthen the paper's broader claims about general-purpose time-series understanding.

Questions for authors

Q1. Could the authors provide a comparison between mid-training and direct task-specific fine-tuning under similar computational budgets? This would help clarify how much of the improvement comes specifically from mid-training rather than additional exposure to time-series data.

Q2. Since Book-of-Thoth relies heavily on synthetic time-series data, could the authors comment on how well the learned capabilities generalize to temporal patterns that differ from those seen during mid-training? Although the paper includes real-world evaluations, further out-of-distribution experiments would help support the broader generalization claims.

Q3. Given that KnoTS relies on LLM-based evaluation, have the authors considered comparing the evaluation scores against human judgments? This would help establish how reliable the reported improvements in reasoning and decision-making are.

Q4. Could the authors provide some insight into why the improvements are less consistent across certain downstream tasks, particularly classification? It would be interesting to understand whether this reflects limitations in the training data or the mid-training approach itself.

Q5. For the KnoTS benchmark, have the authors considered evaluating how much the model's answers depend on the actual time-series input? For example, removing or changing the time series while keeping the question and domain context fixed could help establish whether the model is genuinely reasoning over temporal information.

Reviewer_SxQKOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Thoth, a family of LLMs "mid-trained" for time series understanding tasks. The authors construct Book-of-Thoth, a synthetic time series and text corpus containing both time-series-to-text and text-to-time-series training pairs, along with general-purpose training data and time series knowledge samples. They subsequently used this dataset to mid-train Qwen3-8B and Qwen3-30B-A3B models without any modifications to their underlying architectures. The authors also introduce KnoTS, a benchmark for evaluating time series reasoning using real-world data across six domains. They evaluate Thoth on ChatTime, Time-MQA, and KnoTS, showing improvements over the original Qwen3 models and several existing LLMs baselines. They further show that Thoth performs better than the base model when fine-tuned using only 5% of the downstream training data.

Critical strengths
  • I find the idea of using an intermediate training stage in LLMs to better learn general time series characteristics interesting, even if not entirely novel, since most existing approaches focus on individual downstream tasks. The performance improvements over the base Qwen3 models are substantial in several settings. For example, Thoth-8B improves trend accuracy from 0.490 to 0.973 and volatility accuracy from 0.337 to 0.750 on ChatTime. Also, the results after fine-tuning with only 5% of the downstream data are also promising, and provide useful evidence that the additional training on time series-text paired data helps with the adaptation of a model to time series reasoning tasks.

  • I find KnoTS to be a useful contribution of the paper, since most existing time series QA benchmarks focus on relatively simple tasks revolving around time series characteristics, whereas KnoTS attempts to evaluate reasoning over real-world time series together with domain-specific knowledge. KnoTS covers six domains and uses human-written rubrics for evaluation. Also, the improvement on KnoTS with Thoth-8B compared to Qwen3-8B, and Thoth-30B-A3B compared to its base model, is reasonably encouraging.

Critical weaknesses
  • My biggest concern with the paper is that it does not clearly establish why mid-training is necessary. The authors train Thoth on Book-of-Thoth and compare it against the base Qwen3 models. But there is no comparison against directly fine-tuning the base models on the same data under a similar training budget. I feel this is important because the experiments show that additional training on Book-of-Thoth improves performance, but do not justify that having a separate "mid-training" stage is better than conventional fine-tuning. Since this is one of the main claims of the paper, I feel this comparison is missing. Further, the methodology is not explained in sufficient detail. For example, in Section 4.2, I could not gauge very well how numerical time series are represented and tokenized for Qwen3, and how the text-to-time-series generation task is implemented. I feel explaining these clearly is important for understanding what the model actually learns during mid-training.

  • I am also not fully convinced by the evaluation of knowledge-intensive time series reasoning on KnoTS. The benchmark has only 242 questions across six domains, and the evaluation relies on Gemini-3.5-Pro as the judge. Since this model was never publicly released, it is almost impossible to reproduce these results for most researchers. The paper also does not report how closely these scores agree with human evaluations, even on a subset of samples, or with another (more widely available) LLM as a judge. Further, since the questions include fairly detailed domain-specific background information, it is unclear how much of the performance comes from actually understanding the time series rather than reasoning primarily from the textual context. On KnoTS, Thoth-30B-A3B achieves 0.30 binary accuracy, compared to 0.35 for GPT-4o-mini, and both achieve 0.73 partial accuracy. Thus, I feel more evidence is needed before we can conclude that Thoth has better time series reasoning capabilities, since current results point more towards reasoning on textual context as a major contributor to the performance observed.

Questions for authors
  • The main motivation of the paper is that mid-training is an important intermediate stage for time series understanding. However, as I mentioned in the first weakness, the experiments mainly compare Thoth against the original Qwen3 models, and not their fine-tunedversions. Could the authors clarify why the improvements should be attributed specifically to mid-training, rather than simply to additional training on Book-of-Thoth? Any comparison against directly fine-tuning on the same dataset with a similar training budget would help gauge this more clearly.

  • For KnoTS, the authors use Gemini-3.5-Pro to evaluate model responses based on human-written rubrics. Was any human evaluation conducted to verify these scores, or even a subset of them? I am particularly interested in seeing how often the LLM judge agrees with human evaluations.

  • While I believe KnoTS is an interesting benchmark, I am not sure how much of the performance comes from understanding the actual time series, as I explicitly raised in weakness 2. Since the questions already provide domain-specific background information, could the authors clarify if they evaluated model performance using only the textual information, without the time series context?

  • Appendix C.2 shows that mid-training reduces general-domain performance, which is somewhat expected. For example, MMLU-Redux drops from 79.50 to 76.06, while CommonSenseQA drops from 78.46 to 75.35. Since the paper claims to preserve general-purpose LLM capabilities, could the authors shed more light on how they interpret these results and whether they can be considered an acceptable trade-off? Also, can the authors add an experiment comparing a fine-tuned Qwen model against Thoth on both general-domain performance and time series understanding? If Thoth beats the fine-tuned base Qwen model in general-domain performance, that might provide stronger motivation for the mid-training question I raised in Weakness 1.

  • The authors mention using Gemini-3.5-Pro for evaluating KnoTS and the open-ended Time-MQA tasks. However, since the model was never publicly released, could the authors clarify which exact model checkpoint was used and how they accessed it? If this was a privately available model, how do the authors expect other researchers to reproduce the benchmark results? Could the authors verify the results using a publicly accessible model and subsequently share those results?

Reviewer_gv5DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper first proposes Thoth, a series of mid-trained LLMs with general-purpose time series understanding capabilities. Following Troth, the paper then proposes Book-of-Thoth mid-training corpus. For evaluation, the paper finally presents KnoTS benchmark for time series understanding and reasoning. Extensive experiments show the performance upgrade to some extent.

Critical strengths
  • The motivation of the paper is reasonable. The content is complete, including the proposed method plus the proposed benchmark.
  • The idea of mid-training is interesting, and the main content is easy to follow.
  • The provided experimental results are leading in the given benchmark datasets and experimental settings.
Critical weaknesses
  • Closed-source baselines only contain small models like GPT-4o-mini, Grok-4.1-Fast, and Gemini-3-Flash. GPT-5.2 that is used for generating the data is not included in the baselines.

  • In Table 1, the performance of Qwen3-8B and Qwen3-30B on Time-MQA Classification and Judgement is concerning.

  • According to Appendix B, Time-MQA uses only 50 test items per task, so several reported leads amount to 2–3 questions in Table 1.

Questions for authors
  • How are numeric series serialized into text (precision, scaling, delimiters), and what is the length distribution of the series in Book-of-Thoth?
  • Who annotated KnoTS, how many annotators were involved, and was inter-annotator or judge–human agreement measured? Will Book-of-Thoth and KnoTS be released?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.