acceptodds
Under review as a conference paper at ICLR 2027

Language Pretraining Shapes a Generalizable Manifold: Insights into Cross-Modal Transfer to Time Series

Abstract

Can language-pretrained transformers become effective time-series forecasters, and if so why? In this paper, we show that cross-modal transfer arises because language pretraining preconditions time series training with a reusable manifold. A linear probe on frozen LLM states decodes realistic time-series trajectories without paired supervision, and retrieval in this projected space yields competitive forecasts, showing that structure and dynamics exist before finetuning. Pretrained initialization also improves optimization, producing coherent gradients and a highly anisotropic loss landscape unlike random initialization. Finetuning then acts as low-dimensional alignment, reusing existing directions rather than learning temporal primitives from scratch, as evidenced by low-rank updates, subspace alignment, and shared features for periodicity and trend. Together, these results support a geometric account of LLM-to-time-series transfer: language pretraining builds the manifold, and finetuning projects numerical dynamics onto task-relevant directions.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
foundation models, transfer learning, time series forecasting

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_QFq3OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides an interesting geometric perspective on how language pretraining helps with cross-modal transfer to time-series forecasting. The paper suggests that language-pretrained models may contain reusable sequential structures, which provides an informative explanation beyond architecture design. The main conclusion is supported by comprehensive analysis of representation geometry, initialization, and cross-modal feature reuse. The presentation is easy to follow, and the paper provides useful insights into the mechanisms underlying cross-modal transfer from language to time series.

Critical strengths
  1. The paper studies an important question in cross-modal transfer for time-series forecasting from an interesting perspective. It suggests that language pretraining may provide a reusable representation geometry for time-series forecasting, offering a different perspective on why language-pretrained models can work for time series forecasting.
  2. The experimental design is comprehensive and supports the main claim of the paper well. The proposed transfer mechanism is examined from multiple perspectives, with convincing experimental results and visualizations.
  3. The presentation is very clear. The motivation, experimental design and conclusions are well connected across sections and form a clear story. The figures are also informative and help illustrate the main findings.
Critical weaknesses

There are a few minor reference and presentation issues. Some references could be updated form arXiv versions to their published versions. The Phi-4-mini technical report reference appears to contain a formatting error. In addition, Figures 4 and 5 would be clearer if the font size were larger.

Questions for authors

In Section 3.2, the retrieval experiment provides evidence of shared temporal patterns between the projected text trajectories and real time series. I am curious whether the forecasting performance comes from a meaningful temporal relationship between the first and second halves of the selected trajectory, or mainly from retrieving a similar prefix. For example, if another candidate with a similarly matched prefix were selected, would its second half provide comparable forecasting performance? Alternatively, if this temporal relationship were broken by randomly shuffling the second halves across candidates, would the performance drop significantly?

Reviewer_CEMgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper explores why language-pretrained transformers can generalize to time-series forecasting. The authors learn a linear projection of frozen language-model representations using unpaired real-world time-series windows, then evaluate how well these representations support retrieval-based forecasting. They also compare pretrained and randomly initialized models under full fine-tuning, IO-only adaptation, and LoRA. To better understand the underlying mechanisms, they examine gradient alignment, effective rank, synthetic trajectories, crosscoder features, and several preliminary component ablations. The results reveal interesting differences in how pretrained and randomly initialized models learn and represent time-series data, lending some support to the idea of a shared sequential manifold. However, the retrieval-based evaluation setup needs further validation, and the evidence is not yet strong enough to fully support the proposed mechanistic explanation.

Critical strengths
  1. I like that the paper tries to explain why language models can work for time-series forecasting, rather than just focusing on prediction accuracy. The authors compare pretrained and randomly initialized models from two model families under different fine-tuning settings. The results suggest that pretraining affects how models learn, even when their final forecasting performance is similar. I found the convergence curves and gradient-alignment results particularly useful.

  2. The paper also includes some good control experiments. Appendix A.1 compares the learned representations with Gaussian noise, AR(1), phase-randomized, and time-shuffled baselines. It also tests held-out text and repeats the experiment with Phi-4. These tests provide stronger evidence that the probe is picking up temporal patterns in pretrained features. I also appreciate the periodic and non-periodic ablations in Appendix F, which are a useful first step toward understanding why the approach works.

Critical weaknesses
  1. The retrieval-based forecasting experiment may have data leakage. In Appendix A.2 (p. 13, lines 685–695), the authors normalize each query using all 512 time steps, even though only the first 256 steps are used to find similar sequences. This means the normalization uses information from the future that the model should not have access to. It can also affect which sequence is retrieved, so this is more than just a scaling issue. The baseline forecasts, meanwhile, use only the observed data in its original scale.I think the authors should normalize using only the observed 256 steps and explicitly clarify the data separation. The time-series matching bank used to fit the probe and the query windows are both drawn from the LOOP_SEATTLE validation split, but their separation by series and time is unclear. A simple nearest-neighbor baseline using real time-series data would also help show whether the language-model features actually add value. Another concern is that Eq. (3) trains the probe using real time-series targets. Therefore, the claim in Sec. 3.3 and the conclusion that the model recovers these patterns without any time-series supervision seems too strong. The additional control experiments do not fully address these concerns.

  2. The results show faster adaptation, but the explanation for why it happens is less convincing. Sec. 4.1 reports that different initialization strategies often reach similar performance once convergence is reached at step 4096. However, Tables 4–6 only compare their numerical results at step 64. This makes it difficult to judge how much of the benefit remains after longer training. The LoRA results and the covariance/CKA analysis are interesting, but they do not directly show that reusing learned temporal patterns is what causes faster adaptation. Appendix F takes a useful step toward testing this, but the language experiment removes a group of components that includes an entire MLP. The authors compare the combined effect with individual attention heads, without reporting what happens when the MLP is removed alone or when similarly sized groups of components are removed. As a result, the drop in performance could simply reflect general damage to the language model. The limitations section acknowledges this issue, but the abstract makes a stronger claim than the evidence currently supports.

Questions for authors
  1. Could the authors confirm whether the full-window normalization in Appendix A.2 was used in the actual experiments? It would be helpful to see the retrieval results when normalization uses only the observed prefix. I would also like to know how the probe bank and query windows are separated across series and time, and how a simple nearest-neighbor baseline using real time-series data performs under the same split.

  2. Could the authors provide numerical results at convergence for h=1 and h=64, broken down by dataset and including variation across seeds? This would help complement the existing learning curves. For reproducibility, please also clarify the exact data splits, all three random seeds (Table 3 only lists 420), and the learning rates and LoRA settings used for each model. Table 3 mentions an evaluation extraction script, but I could not find any code in the submission. Would it be possible to share an anonymous implementation, along with the scripts and configurations needed to reproduce the main results?

  3. In Sec. 4.1, how are gradients passed through the discrete-CDF inversion when training with quantile loss? Is interpolation, a differentiable approximation, or another method used? I would also like to clarify how the MASE denominator is calculated. Is it based on historical observations? Appendix B.2 defines it using H, which is described as the forecast horizon. As written, the formula does not seem to work for h=1 when m>=1.

  4. For the language ablation in Appendix F, could the authors report the effect of removing only MLP L1, as well as results from removing comparable sets of components? These additional tests would help clarify whether the observed effect comes from the proposed transfer of periodic patterns or simply from disrupting the language model more generally.

Reviewer_tihdOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I really appreciate the effort of the submission, and thanks for the interesting read. As I understand it, the paper asks why language-pretrained transformers can become good time-series forecasters. You propose that language pretraining builds a reusable sequential manifold, and that finetuning on time series only needs to align numerical dynamics onto it, not learn temporal structure from scratch. You test this in three steps:

  • First, a linear probe on frozen Qwen3 states, trained without paired data, decodes into realistic time-series trajectories, and retrieval in this space gives forecasts that beat simple baselines on mean MSE.
  • Second, you finetune Qwen3, Phi-4-mini, and a protein-pretrained control (ProGen2) on binned time series. Language-pretrained models converge much faster than random initialization, with LoRA matching full finetuning, coherent gradients from step one, and less representational collapse, while ProGen2 behaves more like random init on these measures.
  • Third, you compare the representations the two initializations end up with (including crosscoder features and circuit ablations) and argue that finetuning reuses pretrained directions while random init invents its own.

Overall, I think this is an interesting study to me, and I am currently borderline leaning towards reject on it, mainly because of the concerns below.

Critical strengths

(S1) I like the topic. Evidence on LLMs for time series is contradictory, and I appreciate that you ask why transfer happens, not just whether it does.

(S2) Section 3 convinced me most. The 2×2 ablation shows that neither pretrained weights nor meaningful text is enough on its own, and the probe generalizes to held-out text. The appendix controls (matched noise, AR(1), phase-randomized, and time-shuffled surrogates) all fail, so I don’t think it’s an artifact.

(S3) I believe the speedup: it holds across two LLM families, four regimes, and many metrics, and I appreciate your honest Limitations section.

Critical weaknesses

(W1) This is my main concern, since it targets the central claim. The paper shows that pretrained LMs optimize faster than random init, which I find convincing. But the title and abstract go further and attribute this to language pretraining building a reusable manifold. Any well-trained network with well-conditioned weights could produce the same speedup, and I could not find a weight-conditioning analysis or a shuffled-weight or spectrum-matched init control to rule this out. Could you add these controls, or narrow the claim to “pretrained LMs optimize faster and carry decodable temporal structure”?

(W2) I find ProGen2 too weak as the only non-language control. It is a single model, and I could not find how the 1024 bins were mapped into its vocabulary. Its results are also hard to reconcile: it is ahead of scratch on CRPS at step 64, yet it shows no gradient alignment and a scratch-like rank collapse. A few more non-language pretrained models would make the comparison much stronger.

(W3) I think the mechanistic evidence in Section 4 is observational. Gradient alignment, rank dynamics, and CKA describe what happens during training, but I don’t see an intervention showing that a language-built manifold causes the speedup. Since both inits end at similar performance, I read the result as early-training sample efficiency, and I believe that is how the claims should be scoped.

Questions for authors

(Q1) (W1) If you shuffle Qwen3’s weights within each matrix (keeping the weight statistics, destroying learned structure) or use a random init matched to its singular value spectra, how much of the speedup and the gradient alignment remains? This would show whether the benefit comes from language-built structure or from generic good initialization.

(Q2) (W2) How did you map the bins into ProGen2’s much smaller vocabulary, and can you add more non-language pretrained models (e.g., vision or code)? I would like to understand how they fit together.

(Q3) (W3) Is there an intervention showing the manifold causes the speedup, for example ablating or transplanting the directions found by the probe and measuring convergence speed and gradient alignment? Also, since both inits end similarly, have you trained longer to check whether any gap persists?

Comments on the paper

1 comment
Area_Chair_7BJYOpenReview ↗

Reminder

Dear Reviewers CEMg and QFq3,

Thank you for your efforts so far. This is a friendly reminder to review the authors' rebuttal and respond to their comments as soon as possible, before the discussion deadline (November 18 AoE). Please also indicate whether your concerns have been addressed. Thank you for your time and cooperation!

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.