acceptodds
Under review as a conference paper at ICLR 2027

What Does Language-Model Pretraining Contribute to Time-Series Understanding

Abstract

Our contribution is a controlled empirical attribution analysis of the role of language-model pretraining in time-series understanding across decision targets and output interfaces. Across ten base language models, we compare frozen pretrained backbones with architecture-matched random controls under matched inputs, feature extraction, and learned-readout training. In controlled forecast- ing, pretraining improves process-family routing in all ten models, with gains of 7.5-31.4 percentage points, but the effect reverses when supervision targets the future-optimal expert; a five-model expected-risk control preserves this reversal. Affine forecasting heads instead reduce error in nine of ten models. On four real-data domains, task-specific readouts improve correct updating after evidence edits by 14.80 points on average, while preservation of unchanged facts decreases by 7.59 points. Under native constrained decoding, models retain the pre-edit answer in 88 of 100 evaluations conditioned on an initially correct target. Readout controls show that the measured contribution depends on how temporal features are accessed. Together, the analysis shows that pretraining supplies reusable temporal features, but its benefit is decision- and interface-dependent rather than a uniform gain in time-series understanding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language-model pretraining, time-series representation learning, pretraining attribution, representation probing, time-series forecasting, evidence-based question answering

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Zq7VOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether language-model pretraining contributes to time-series understanding by comparing frozen pretrained models with their randomly initialized versions. The experiments cover multiple aspects of time series understanding: structural recognition, expert routing, numerical forecasting, and question answering. The authors find that pretraining improves process-family routing and numerical forecasting but hurts future-optimal expert selection. On real-data question-answering tasks, pretraining improves updating after evidence edits but reduces preservation of unchanged facts. The authors conclude that pretraining benefits depend strongly on the task and readout interface.

Critical strengths

Controlled comparisons: Comparing pretrained and randomly initialized backbones across ten models, with matched readouts and additional controls, provides useful evidence about the effects of pretraining for the specific task the paper studies which is time series analysis.

Critical weaknesses

This paper seems to be completely written by AI. It is very difficult to read and understand. The contributions and positioning with respect to prior work are unclear to me. Inadequate effort has been made to describe why this question is interesting, or why the analysis is useful. I have concerns about the quality of this paper for ICLR audience: not sure whether people can read and understand this paper well enough to learn something new and build upon it. The paper needs a substantial rewriting to make it readable and suitable for ICLR audience (or any audience for that matter).

Weak absolute performance undermines the conclusions: Despite the reported improvements in updating, the paper's pretrained task-specific readouts achieve only 48.29% accuracy, comparable to the 48.31% majority baseline, with just 8.40% joint success. The 25.74% updating success is also close to the 25% expected from independent random guesses on binary questions. Question-conditioned readouts and native decoding perform even worse. Consequently, the evidence for meaningful time-series understanding remains weak.

Limited generalizability: The forecasting conclusions rely primarily on simple synthetic processes, a fixed expert bank, and small training and test sets. The real-data evaluation contains only 42 test groups and uses artificial constant-offset edits. It remains unclear whether the conclusions generalize to realistic time-series tasks or different experimental choices.

Questions for authors
  1. Does the reversal in expert-routing performance persist with different expert banks, larger training datasets, and more realistic time-series distributions?

  2. Given that aggregate question-answering accuracy is near or below trivial baselines, what evidence supports interpreting the improved updating scores as meaningful temporal understanding rather than just increased prediction variability?

  3. Why is the question considered by the authors interesting? What do their findings teach us and how are these useful to the community? Are any of their findings surprising, or are they expected?

Reviewer_YCAkOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper investigates the role of language-model pretraining in time-series understanding through controlled comparisons between frozen pretrained backbones and randomly initialized counterparts. The study combines several established evaluation approaches to examine three aspects: structural recognition and expert routing, numerical forecasting with learned readouts, and evidence-sensitive question answering. Experiments are conducted across ten language models, synthetic time-series data, and four real-world domains.

Critical strengths

The paper presents an extensive set of experiments across multiple language models, tasks, and evaluation settings, providing a broad empirical assessment of the effects of language-model pretraining on time-series understanding. The comparisons between pretrained and randomly initialized backbones, along with additional control experiments, help isolate the effects of pretraining under different settings. The paper also reports both positive and negative results, offering a relatively comprehensive view of the observed performance variations.

Critical weaknesses
  1. Limited novelty in the core contributions. The Introduction presents a controlled empirical attribution analysis across structural recognition, numerical prediction, and evidence-sensitive question answering as the primary contribution. However, the main technical components are largely established in prior work, including pretrained-versus-random backbone comparisons (Tan et al., 2024; Roger et al., 2026), linear probing and readout-based evaluations (Schumacher et al., 2026), feature-based expert selection (FFORMS; Talagala et al., 2023), and evidence-editing evaluations of answer consistency (CLIR-Bench; Nie et al., 2026). The paper mainly combines these existing techniques under a common experimental framework, without introducing a substantially new methodology or evaluation paradigm.

  2. Writing and presentation quality. The overall writing quality falls substantially short of the standards expected for a top-tier conference publication. The manuscript lacks a clear and compelling research narrative, with an Introduction that fails to articulate the research gap and distinctive contributions effectively. The writing is excessively repetitive, verbose, and frequently dominated by defensive qualifications rather than concise scientific arguments. These are not merely stylistic issues; they substantially undermine the clarity, coherence, and persuasiveness of the paper, making it difficult to identify its central scientific message.

[1] Tan, M., Merrill, M. A., Gupta, V., Althoff, T., & Hartvigsen, T. (2024). Are Language Models Actually Useful for Time Series Forecasting? Advances in Neural Information Processing Systems (NeurIPS), 37, 60162–60191. https://doi.org/10.52202/079017-1922

[2] Roger, A., Humane, P., Tai, Z., Legate, G., Mircea, A., Feofanov, V., & Rish, I. (2026). LLM Pretraining Shapes a Generalizable Manifold: Insights into Cross-Modal Transfer to Time Series. arXiv preprint arXiv:2605.20449. https://arxiv.org/abs/2605.20449

[3] Schumacher, D., Nourbakhsh, E., Slavin, R., & Rios, A. (2026). Prompting Underestimates LLM Capability for Time Series Classification. arXiv preprint arXiv:2601.03464. https://arxiv.org/abs/2601.03464

[4] Talagala, T. S., Hyndman, R. J., & Athanasopoulos, G. (2023). Meta-learning How to Forecast Time Series. Journal of Forecasting, 42(6), 1476–1501. https://doi.org/10.1002/for.2963

[5] Nie, F., Liu, E. B., Zhu, Y., Yan, L., Fan, W., & Han, J. (2026). CLIR-Bench: Benchmarking Multimodal Question Answering over Irregular Clinical Time Series. arXiv preprint arXiv:2607.09880. https://arxiv.org/abs/2607.09880

Questions for authors

What is the genuinely novel contribution of this work? Does the paper introduce any new methodology or provide fundamentally new scientific insights beyond existing studies, or is its contribution primarily a collection of experiments and descriptive analyses using established techniques? Please clarify what distinguishes this work from a comprehensive empirical report.

Reviewer_Sk2pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the contribution of LM pretraining to time series understanding by comparing frozen pretrained models with architecture matched random controls. It evaluates expert routing and numerical forecasting on synthetic data, alongside question answering under controlled edits to real observations using different readout interfaces. The results suggest that pretraining improves certain structural and forecasting tasks, but its benefits depend on the supervision target and output interface.

Critical strengths

This paper provides a controlled evaluation of pretraining effects. The matched pretrained vs. random comparison provides a framework for isolating the contribution of pretrained weights. In particular, the reversal in routing performance under different supervision targets shows a distinction between recognizing temporal structure and making effective downstream decisions.

This paper also provides an evaluation of evidence sensitivity beyond aggregate accuracy. The paired original and edited observations test whether models correctly update answers when relevant evidence changes while preserving answers to unchanged facts. This design helps distinguish updating and preservation behaviors that may not be captured by aggregate accuracy alone.

Critical weaknesses

The routing reversal does not establish a corresponding difference in downstream forecasting quality. The evaluation measures agreement with the future optimal expert rather than the forecasting loss of the selected expert, although different routing errors can have very different costs. Moreover, the primitive to composition distribution shift and lack of in distribution controls make it unclear whether the reversal reflects a general limitation of pretrained representations or the specific experimental setup.

The real data experiments provide limited evidence of reliable evidence grounded question answering. The updating gain is confined to task specific readouts, whose overall accuracy remains near a simple baseline and whose preservation of unchanged facts deteriorates. The question conditioned scorer fails to update correctly, while native decoding performs similarly with and without numerical evidence, making it difficult to attribute the reported behaviors to meaningful use of the time series.

Questions for authors
  1. Can the authors report the actual forecasting MSE or regret of the experts selected by the pretrained and random routers? The observed reversal in expert selection accuracy does not necessarily imply a corresponding difference in downstream forecasting quality.
  2. Does the routing reversal persist on held out primitive processes without the composition shift? Reporting training and held out accuracy, together with agreement between process family and future optimal labels, would help distinguish the effect of supervision from overfitting.
  3. Since the task specific heads are trained on offset edited examples, does their updating advantage generalize to unseen edit patterns? Comparisons against simple paired answer baselines would also help establish whether the observed gains reflect evidence sensitive answering.
  4. What forecasting error do naive baselines achieve, and how are pretrained and random features normalized before affine head training? These details are helpful for interpreting the reported numerical forecasting improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.