acceptodds
Under review as a conference paper at ICLR 2027

CRITICAL-LAYER CONTEXT ALIGNMENT FOR LARGE LANGUAGE MODEL-BASED TIME-SERIES FORECASTING

Abstract

Large language models (LLMs) can forecast time series with their pretrained weights frozen. Existing approaches make numerical inputs compatible with LLMs, but leave open where to supply context and how to train for autoregressive forecasting. Similar local patterns can occur in different window states and at different times. Feeding predicted segments back into the input window also changes the history used for subsequent predictions. To use the overall state of the historical window and the corresponding time information when processing local patches, and to better adapt training to autoregressive forecasting, we propose CLCA (Critical-Layer Context Alignment). A window summary adjusts input patch embeddings, while adapters in selected layers of the fully frozen backbone combine timestamp representations with the current series state. For long-term forecasting, we feed predictions back during training and supervise subsequent segments across different forecast lengths, so that training covers the input conditions encountered during inference. Experiments on eight long-term benchmarks show overall state-of-the-art performance among the compared LLM-based and non-LLM methods under the reported protocols. Ablations show that combining the two context pathways and supervising predictions from windows containing earlier model outputs both improve forecasting performance. With adapter counts matched, adaptation in the final six layers performs best among the tested placements. Short-term, zero-shot, and alternative-backbone experiments further support the applicability of the context-adaptation design.

Primary area:
learning on time series and dynamical systems
Keywords:
Time series forecasting, Large language models, Context adaptation, Autoregressive forecasting, Parameter-efficient adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bFLAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes CLCA, a framework for LLM-based time-series forecasting that combines window-level context conditioning, layer-wise context adapters, and multi-step autoregressive training. The model uses numerical window statistics and timestamp representations to provide contextual information to a frozen language-model backbone, with adapters placed in the final six layers. During training, predicted segments are fed back into the input window so that later predictions are supervised under conditions closer to autoregressive inference. The method is evaluated on long-term forecasting, M4 short-term forecasting, and M3/M4 zero-shot transfer, with additional studies on adapter placement, component ablations, lightweight backbone replacements, and alternative pretrained language models.

Critical strengths

The paper is generally well written, and the method and training procedure are described clearly.

The supplementary material is also detailed and provides useful implementation information. The experimental evaluation is fairly broad, covering eight long-term forecasting datasets, M4 forecasting, zero-shot transfer, adapter placement, component ablations, and different backbone choices. The method also achieves competitive results on several datasets.

Critical weaknesses

My main concern is the limited methodological novelty. CLCA is largely built on AutoTimes and Deep TPC, together with existing ideas such as adapter placement, window-level conditioning, and multi-step autoregressive training. The paper introduces a new combination and a patch-state update pathway, but it is not clear to me that these incremental changes are sufficiently novel. In particular, the paper directly builds on Deep TPC but does not include a direct comparison with it.

The role of the pretrained LLM is also not sufficiently demonstrated. On ETTh1, removing the LLM backbone leads to almost no performance drop, while on ECL a lightweight attention module reaches the same average MSE. These replacement models are not perfectly controlled, since they still use LLM-generated timestamp embeddings and modify the adapter structure, but the results still weaken the claim that a large pretrained language backbone is necessary.

The evidence for the critical-layer claim is also limited. The placement study is only conducted on ETTh1 and ETTm1, and the differences between the final six layers and alternative placements are very small. No statistical significance test is provided, and it is unclear whether the same pattern holds across different backbones.

The baseline comparison could also be stronger. Recent autoregressive forecasting methods such as Timer-XL and Time-MoE are discussed in the related work but are not included in the main experiments. There is also a potential inconsistency in the evaluation protocol, since CLCA and CALF use 720-step targets to construct test samples, while some reproduced baselines use targets corresponding to each forecast horizon.

Finally, some of the reported gains are quite small, and the contribution of multi-step supervision is not fully separated from the additional computation it requires. The paper also does not systematically report training cost, inference latency, or memory usage, even though long-horizon forecasting requires multiple forward passes through the full 774M backbone.

Questions for authors

What is the key methodological novelty beyond combining AutoTimes, Deep TPC, and standard adapter mechanisms?

Why are the final six layers considered critical when the improvements over other placements are relatively small? Does this finding generalize to other datasets and backbones?

Why does removing the pretrained LLM backbone produce almost identical performance on ETTh1? What is the actual contribution of LLM pretraining?

Can the authors compare against stronger recent forecasting methods under identical test origins and evaluation protocols?

Can the authors provide compute-matched training ablations and report training/inference costs?

Reviewer_GcESOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CLCA studies conditioning a frozen LLM forecaster on numerical state and temporal context while training it for recursive multi-horizon use. It combines context construction, gated fusion and patch-level adapters in the final six layers, and detached prediction roll-in with supervision at several horizons.

Critical strengths
  1. The motivating issues are credible. Forecasts can benefit from information about temporal state, and recursive deployment creates a mismatch with short-horizon training. Context injection and training with model-generated histories respond directly to these issues.
  2. Several controls are informative. Figure 3 holds the number of adapters fixed across placements; Table 4 tests conditioning and training choices; Table 5 includes lighter backbone replacements. These experiments should be credited rather than described as missing.
Critical weaknesses
  1. The evidence does not yet clearly establish the new design principle beyond existing context injection. The paper explicitly builds on Deep TPC; numerical context, a direct patch path, and late-layer placement are genuine additions.
  2. Figure 3 supports late placement among the tested six-adapter arrangements, but does not show which addition explains the advance over the inherited conditioning design. Table 5 also shows near ties with much simpler backbones on some settings.
  3. The exposure-bias explanation is not isolated from the benefit of additional future-target supervision. The 96-only comparison in Table 4 changes both roll-in and the supervised horizons. The reported improvement therefore supports the combined training recipe, but cannot by itself show that model-generated histories reduce recursive error accumulation.
Questions for authors
  1. With context information, adapter placement, and training held fixed, what gain is attributable specifically to the added patch-context path over fusion-token-only conditioning?
  2. Under identical future-target supervision, does prediction-history roll-in improve performance over observed-history roll-in?
Reviewer_WV5zOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CLCA adapts a frozen pretrained language model for time-series forecasting through two context pathways: a numerical window summary adjusts input patch embeddings, while gated adapters in the final six backbone layers incorporate timestamp representations and the current series state. All backbone layers remain active and frozen. A shared 96-step predictor is trained across multiple forecast horizons by feeding detached predictions back into the input window, recomputing context, and supervising subsequent segments. Experiments cover eight long-term benchmarks, short-term forecasting, cross-dataset transfer, adapter placement, and lightweight backbone replacements; improvements vary by dataset and metric.

Critical strengths
  1. The adapter-placement study controls the number of adapters across early, middle, evenly spaced, random, and late-layer configurations. Reporting three-seed means and standard deviations provides useful empirical evidence about where context adaptation is effective.

  2. The evaluation covers eight long-term forecasting datasets, short-term forecasting, cross-dataset transfer, alternative backbones, and component ablations. The complete method achieves meaningful improvements in some settings, such as ETTh2, providing empirical support for the overall approach even though the gains vary across datasets.

Critical weaknesses
  1. Lightweight alternatives already approach or match the full model, leaving the necessity of the pretrained LLM backbone insufficiently established. The lack of training and inference cost comparisons also makes it unclear whether the small gains justify the additional computation.

  2. The distinctive contribution of critical-layer adaptation is insufficiently established: the final six layers offer only small gains over competitive placements, with evidence limited to two datasets and one backbone and no systematic study of adapter counts. The method primarily combines existing context-adaptation and autoregressive-training ideas, and the current evidence does not clearly establish a substantial methodological advance attributable to layer placement. Using the model’s own predictions as subsequent training inputs is essentially autoregressive rollout without teacher forcing, so this training component also offers limited novelty.

Questions for authors
  1. Given that lightweight alternatives approach or match CLCA on ETTh1 and Electricity, what evidence establishes the added value of the pretrained LLM backbone? Please compare training and inference costs with these alternatives under matched conditions and clarify whether the accuracy gains justify the additional computation.

  2. What substantive novelty does CLCA offer beyond existing context-adaptation mechanisms and autoregressive rollout without teacher forcing? Given the small differences between competitive layer placements, please provide evidence that isolates the contribution of late-layer placement, including its robustness across backbones and datasets and the rationale for using exactly six adapters.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.