acceptodds
Under review as a conference paper at ICLR 2027

Dissecting LLMs for Time Series Forecasting: When They Help and Where They Struggle?

Abstract

Large language models (LLMs) have been introduced to time series forecasting (TSF) to incorporate contextual knowledge beyond numerical signals. However, existing studies question whether LLMs provide genuine benefits, often reporting comparable performance without LLMs. We show that such conclusions stem from limited evaluation settings and do not hold at scale. We conduct a large-scale study of LLM-based TSF (LLM4TSF) across 8 billion observations, 17 forecasting scenarios, 4 horizons, multiple alignment strategies, and both in-domain and out-of-domain settings. Our results demonstrate that LLM4TS indeed improves forecasting performance, with especially large gains in cross-domain generalization. Pre-alignment outperforming post-alignment in over 90% of tasks. Both pretrained knowledge and model architecture of LLMs contribute and play complementary roles: pretraining is critical under distribution shifts, while architecture excels at modeling complex temporal dynamics. Moreover, under large-scale mixed distributions, a fully intact LLM becomes indispensable, as confirmed by token-level routing analysis and prompt-based improvements. Overall, our findings overturn prior negative assessments, establish clear conditions under which LLMs are not only useful, and provide guidance for effective model design.

Primary area:
learning on time series and dynamical systems
Keywords:
Time series forecasting,Large language model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ADQdOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors present a detailed analysis for leveraging LLMs for time series forecasting, including the way of aligning LLMs with TS data, the scenarios where LLMs can be better leveraged for TS data, and the reason why LLMs can be helpful for TSF. They also provide the guidelines to apply LLMs to TSF.

Critical strengths
  1. The writing is quite clear.
  2. The experiments are comprehensive.
  3. The investigated problem is interesting.
Critical weaknesses
  1. I am challenging the motivation of this work. The authors raise a question 'are LLMs tryly indispensable for TSF'. However, they only investigate two kinds of alignment methods to leverage LLMs for TSF. It is noted that there are still other ways to leverage LLMs like Prompt4Cast [1]. I think the authors should narrow down their claims.

[1] PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecasting

  1. I agree the point whether LLMs are necessary for TS data. However, why the authors just compare two kinds of LLM alignment methods? I think it is more reasonable to include non-LLM methods, particularly those TS foundation models. In the current experiments, it is more like the authors aim to investigate which alignment method is better, instead of whether using LLMs is necessary.

  2. I am wondering whether the input prompts have large influence on the performance and whether the authors have simple experiments to discuss this effect?

  3. The authors used GPT-2 as the LLM backbone. I understand this is for efficient experiments. However, GPT-2 is too small to fully exploit the power of LLMs, and I also doubt whether using GPT-2 is useful to derive the correct conclusion.

  4. As an investigation work, I think only numerical comparisons are insufficient to provide an effective solution. The intermediate results can be helpful.

  5. Another concern is regarding the parameters. From two aspects, one is on the LLM parameters, and another is on the parameters of the non-LLM part. The first issue is related to my point 4, and I am wondering whether the conclusion still holds with different sizes of LLMs, or, whether different sizes of LLMs have different influences? Regarding the second issue, given you have two alignment methods which might bring different trainable parameters for the non-LLM part, I am wondering whether the authors control this? in my understanding, the reason why the pre-alignment has better performance is probably because this method has additional module as Figure 1, which might affect the performance.

Additionally, I am open to raise my score if my concerns could be addressed

Questions for authors

See above and I also have another minor question: Why did the authors choose to investigate the two kinds of LLM alignment methods instead of comparing with existing LLM driven baselines and non-LLM baselines?

Reviewer_jjYjOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates when and why pretrained LLMs help in time series forecasting (TSF). Moving beyond single-dataset evaluations, the authors train models across 55 datasets and evaluate both in-domain and on 7 unseen out-of-domain (OOD) datasets under pre-alignment and post-alignment strategies. Through ablations, statistical regime analysis (shifting and transition), and a routing mechanism, the study argues that while LLMs are often unnecessary on stationary in-domain series, intact pretrained LLMs provide gains under distribution shifts and OOD generalization.

Critical strengths

(1) Scaling the ablation study to 62 datasets (~8B observations) across both in-domain and zero-shot OOD regimes provides valuable new evidence, showing that multi-source training unlocks cross-domain benefits that single-dataset studies miss.

(2) Connecting the usefulness of pretrained LLMs to the degree of distribution shifting in the time series offers a potential explanation for when LLMs actually help versus when simple numerical encoders are enough.

Critical weaknesses

(1) The main results in Table 2 do not fully support overturning prior skeptical findings. On standard in-domain benchmarks with low shifting (ETTh1, ETTm1, Traffic, ECL), completely removing the LLM (w/o LLM) still matches or outperforms the full pretrained LLM. Even on datasets where the pretrained LLM wins, the margins are often tiny (e.g., MAE differences of 0.002–0.008 on NN, Covid, ETTm2, and AQShunyi), and unlike Appendix Tables 15–16, the key ablation experiments in Table 2 rely on a single fixed seed without error bars, confidence intervals, or statistical significance tests.

(2) Scaling from GPT-2 to Qwen-3 models (0.6B, 1.7B) degrades forecasting accuracy as model size grows, which contradicts the claim that pretrained language "world knowledge" drives the forecasting gains and instead suggests the frozen LLM mainly acts as a fixed regularizing feature map under distribution shifts.

(3) The theoretical proofs are tautological and add little explanatory value.

(4) The learned “Router” feels over-engineered. It requires an explicit Target Routing Ratio = 0.5 penalty during training, which forces low-shifting tokens through the LLM unnecessarily. In fact, Appendix Tables 5 and 6 show that simply thresholding the Shifting metric directly (> 0.2 to pass, < 0.1 to skip) beats the learned router on 16/17 and 13/17 datasets, raising the question of why a learned router is needed at all over a simple statistical rule.

Questions for authors

no more questions

Reviewer_F4KDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper systematically investigates when and why LLMs help time series forecasting. The authors compare pre-alignment and post-alignment strategies and evaluate their framework on 62 datasets across diverse domains, including out-of-distribution settings. Through benchmark comparisons, ablation studies, and routing analyses, they find that cross-dataset training improves generalization, pre-alignment generally performs better, and LLM benefits depend on pretrained knowledge and time-series characteristics. The paper further explores token-level routing to selectively apply LLMs to informative time-series patches, providing empirical insights into where LLM-based forecasting helps and struggles.

Critical strengths
  1. The paper directly investigates whether LLMs genuinely contribute to time series forecasting, addressing the concern that similar performance can sometimes be achieved without pretrained LLMs. This question is important for understanding the actual value and limitations of incorporating LLMs into forecasting systems.
  2. The paper evaluates numerous real-world datasets across multiple forecasting horizons and both in-distribution and out-of-distribution settings. Beyond performance comparisons, it examines the roles of pretrained knowledge, model architecture, time-series statistical properties, and token-level routing, with additional ablations and routing interventions. These analyses provide more informative insights into when and why LLMs help, making the paper valuable beyond simply proposing another forecasting method.
Critical weaknesses
  1. The paper identifies distribution shifts and transitions as factors associated with LLM benefits. However, statistical properties in real-world time series may be correlated, making their individual effects difficult to isolate. Although synthetic experiments provide supporting evidence, the extent to which these findings generalize across domains and forecasting settings remains unclear.
Questions for authors
  1. How did the authors distinguish the effects of distribution shifts and transitions on LLM forecasting performance, given that these characteristics may occur together in real-world time series?
  2. What evidence supports the generalizability of these findings across different domains and forecasting settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.