acceptodds
Under review as a conference paper at ICLR 2027

LG-Transformer: LLM-Guided Discovery of Boosted Novel Transformer Models for Time-Series Modeling

Abstract

Transformer-based models have achieved strong results in long-term time-series forecasting, yet their architectures are still largely designed manually, leaving a vast combinatorial design space underexplored. To address this issue, we present LG-Transformer, an LLM-guided neural architecture search framework that automatically discovers high-performing Transformer variants for time-series modeling. Our method combines a pretrained large language model controller, with a weight-sharing supernet to enable context-aware architecture generation and efficient candidate evaluation. The search space spans key encoder design choices, including network depth, attention configuration, normalization placement, activation function, feed-forward expansion ratio, and residual operators. After search, the selected architecture is retrained from scratch, yielding a standard forecasting model with no additional inference-time overhead. Experiments on widely used multivariate forecasting benchmarks, including ETT, Electricity, Traffic, Weather, and ILI, show that the architectures discovered by LG-Transformer consistently outperform strong manually designed baselines and achieve highly competitive results across diverse prediction settings. These results demonstrate that LLM-guided architecture search is an effective and practical route to better time-series Transformers, while lowering the barrier for small industrial practitioners and startups to adopt large-model-driven solutions.

Primary area:
learning on time series and dynamical systems
Keywords:
Time-Series Forecasting, Neural Architecture Search, Large Language Models, Transformer Architecture, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_TbB8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LG-Transformer, an LLM-guided neural architecture search (NAS) framework for time series forecasting. The method employs a pretrained LLM as an autoregressive controller to generate Transformer encoder architectures from a structured search space. A weight-sharing supernet is used for efficient candidate evaluation, and shortlisted architectures are subsequently retrained from scratch to select the final forecasting model. The search space covers encoder depth, normalization placement, attention configurations, activation functions, FFN expansion ratios, and residual-path operators. Experiments on eight multivariate forecasting benchmarks show that LG-Transformer achieves competitive performance against manually designed forecasting models, obtaining the best MSE in 29 out of 32 settings. The paper also provides ablation studies on framework components, controller architectures, optimization objectives, and input-window robustness.

Critical strengths
  1. A practically motivated automated architecture search framework. The paper addresses the relevant problem of reducing manual architecture engineering for time series forecasting. The combination of an architecture controller, a weight-sharing supernet, and a final architecture retraining stage forms a reasonable search-to-deployment pipeline. The final forecasting model does not require the LLM controller during inference, which is a practical advantage.
  2. Competitive forecasting results across multiple benchmarks. The discovered architectures achieve competitive results across several widely used forecasting datasets and prediction horizons. The evaluation covers datasets with different dimensionalities and temporal resolutions. The paper also provides comparisons among different controller backbones and framework variants, offering some evidence of the effectiveness of the proposed pipeline.
Critical weaknesses
  1. Insufficient comparison with NAS methods and inadequate validation of the core LLM-guided search contribution. My primary concern is the lack of comparisons with established neural architecture search strategies. Although the paper proposes an LLM-guided NAS framework, the main experiments primarily compare the final discovered architectures against manually designed forecasting models rather than alternative architecture search methods. The proposed search space mainly covers architectural configurations and local operator choices within a predefined Transformer encoder template. While these choices constitute a valid architecture search space, it remains unclear whether a pretrained LLM is necessary to explore it effectively. Although Table 2 compares different controller backbones, the paper does not provide comparisons against standard search strategies such as random search, evolutionary search, or Bayesian optimization under matched search spaces and computational budgets. Consequently, it is difficult to determine whether the reported improvements originate from the LLM-guided controller, the search space itself, or the exploration of a sufficiently large number of candidate architectures. This substantially weakens the evidence supporting the paper's central methodological contribution.
  2. Relatively outdated forecasting baselines and insufficient clarity regarding baseline result sources. The forecasting baselines are relatively outdated, with Time-LLM (2024) being the most recent included method. The absence of more recent forecasting models limits the strength of the empirical comparisons, particularly given the paper's claims regarding competitive or state-of-the-art forecasting performance. Additionally, the reported PatchTST results appear consistent with those in its original publication, whereas the reported Time-LLM results appear slightly worse than the corresponding published results. The paper does not clearly explain the sources of these baseline results or distinguish independently reproduced results from those taken from previous publications. Clarifying the baseline result sources and corresponding reproduction protocols would help establish the reliability and fairness of the reported comparisons.
  3. Insufficient evidence for the stability of the reported forecasting improvements. Although LG-Transformer achieves the best MSE in 29 out of 32 forecasting settings, its improvements over the strongest competing methods are often relatively small. In several settings, the absolute MSE differences are only around 0.002–0.003. The experiments report a single random seed, without providing results from multiple independent runs or measures of variability. It is therefore unclear whether these small improvements are stable across different runs. This concern is particularly relevant for an NAS framework, where randomness can arise from both architecture search and the subsequent training of the selected architectures. Without repeated experiments, the consistency of the reported performance advantage is not sufficiently established.
  4. Limited and inconclusive ablation studies. The main framework and controller ablations are conducted only on ETTh1 and ETTm1 with prediction horizon 96. This limited evaluation makes it difficult to assess whether the conclusions generalize across datasets with different characteristics and longer forecasting horizons. Moreover, several ablation variants exhibit only marginal performance differences. For instance, replacing the MSE-based controller reward with an MAE-based reward results in differences of only 0–0.002 across the four reported metrics. The performance differences among several controller backbones are also relatively small. Given the absence of multiple-seed evaluations, these results provide limited evidence that the individual design choices consistently improve forecasting performance. In particular, the experiments do not convincingly establish the necessity or distinctive advantage of employing a pretrained LLM controller.
Questions for authors

Q1. Comparison with standard NAS methods. Could the authors compare the proposed LLM-guided controller with established architecture search strategies, such as random search, evolutionary search, and Bayesian optimization, under the same search space, candidate evaluation procedure, and computational budget? Such comparisons are essential for determining whether the pretrained LLM provides a meaningful advantage over conventional architecture search methods. Reporting search-performance curves against the number of evaluated candidates would also be informative.

Q2. Baseline result sources and reproduction protocols. The reported PatchTST results appear consistent with those in its original publication, whereas the Time-LLM results appear slightly worse than the corresponding published results. Could the authors clarify which baseline results were independently reproduced and which were taken from existing publications, and explain the evaluation protocols used to obtain them?

Q3. Statistical reliability of the main forecasting results. Given that the improvements over the strongest baselines are relatively small in many forecasting settings, could the authors report results from multiple independent runs, including means and standard deviations? Ideally, these experiments should account for variability in both the architecture search process and the final training of selected architectures, rather than only retraining a single fixed architecture.

Q4. Generality and reliability of the ablation conclusions. Could the authors extend the ablation studies to additional datasets and forecasting horizons? Given the small differences between several variants, multiple-seed evaluations would also help determine whether the reported improvements are consistent and meaningful.

Q5. Computational cost of architecture search. Could the authors report the total search cost, including GPU-hours and the number of evaluated architectures? A comparison of search efficiency against non-LLM NAS strategies would help establish whether the additional computational cost of the LLM controller is justified by improved search quality.

Reviewer_Bw2JOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes LG-Transformer, a NAS framework for time-series forecasting that uses a pretrained LLM (Qwen2.5-1.5B-Instruct) as an RL controller over a discrete Transformer encoder search space (depth, LayerNorm placement, heads, attention scaling, activation, FFN ratio, and Conv1D operators on the residual paths). Candidates are scored with inherited weights from a weight-sharing supernet, and the top ones are retrained from scratch, so the deployed model has no extra inference cost.

Critical strengths

The pipeline is clean and practical: the search space is a compact set of 14 discrete decisions, the reward comes from a weight-sharing supernet, and the final model is retrained from scratch so nothing from the search remains at inference time. Adding Conv1D operators on the residual paths is a sensible time-series-specific extension of the usual NAS cell space.

Critical weaknesses
  1. After reading the paper, I couldn't tell what model was actually built. The base encoder template is never described: Sec. 3.1 and App. B.1 only list the 14 searchable choices, but say nothing about tokenization (patching or not, patch length), channel independence, input normalization, or the prediction head, and these usually matter more than the block-level choices being searched. The final searched architectures are also never reported, so "Ours" in Table 1 is an unknown model and I can't separate what the search contributed from what the fixed template already provides.

  2. Asking for more baselines is usually a weak critique, but the central claim here is that searched architectures beat hand-designed ones, and the hand-designed set is dated. The strongest non-LLM baseline is PatchTST, and many patching-based forecasters have been published since. iTransformer is cited in the introduction but not compared. Beating this set by 1 to 3% MSE does not support the claim as stated.

  3. There is no analysis of what the search finds. The introduction motivates the work by saying the best architecture varies across datasets, but the paper never shows which depth, normalization layout, or residual operators were selected for each dataset, whether they differ, or why. Without this, the contribution reads as a single number per setting rather than any insight about forecasting architectures.

  4. Table 2 suggests the LLM controller barely matters. The LSTM controller gets 0.371/0.293 MSE vs. 0.367/0.287 for Qwen2.5-1.5B, and the 72B model gives 0.366/0.286, all on a single seed. Also, the LLM never generates anything: Eq. 4 uses the final-token hidden state as a feature for step-specific linear heads, so it's a frozen-prompt feature extractor rather than a "context-aware generator". The "LLM-guided" framing isn't supported by the paper's own evidence.

Questions for authors
  1. (Clarification) Where exactly is the Conv1D applied on the residual path: on the skip branch (Conv(x) + Attn(x)) or on the sublayer output (x + Conv(Attn(x)))? Eq. 2 only lists the operator set.

For other points, please refer to the weaknesses.

Reviewer_5VkzOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes LG-Transformer, an LLM-guided neural architecture search framework for time-series forecasting. A pretrained LLM controller generates Transformer architectures, while a weight-sharing supernet efficiently evaluates candidates using validation performance. The best architectures are then retrained from scratch and deployed without additional inference overhead. Experiments on eight multivariate forecasting benchmarks show that the searched architectures consistently outperform strong manually designed baselines.

Critical strengths
  1. Effective combination of LLM-based search and weight sharing. The framework makes a large Transformer design space practical to explore while keeping candidate evaluation efficient.

  2. Strong empirical performance with no inference-time overhead. The discovered architectures consistently outperform strong forecasting baselines, while the LLM controller and supernet are discarded after search.

Critical weaknesses
  1. The evaluation lacks direct NAS baselines. Most comparisons are against forecasting models rather than competing architecture-search methods, making it difficult to assess whether the proposed LLM-guided search is actually better than conventional NAS.

  2. The search space is relatively hand-crafted. The LLM only selects among predefined discrete choices, so much of the final performance may come from the manually designed search space rather than from the reasoning capability of the LLM controller.

  3. The reported gains over strong baselines are often modest, and the baseline selection is not sufficiently up-to-date. Although LG-Transformer wins many settings, improvements over PatchTST and Time-LLM are frequently small, while several more recent and competitive forecasting methods are not included.

Plz ref:

[1] SE-LLM: Semantic-Enhanced Time-Series Forecasting via Large Language Models. ICLR 2026

[2] TimeSAF: Towards LLM-Guided Semantic Asynchronous Fusion for Time Series Forecasting. ACL 2026

[3] Context-Alignment: Activating and Enhancing LLMs Capabilities in Time Series. ICLR 2025

Questions for authors

Why is the same encoder block configuration repeated across all layers? Have you considered searching heterogeneous layer-wise architectures, where different layers use different operations or configurations?

How much of the improvement comes from using an LLM controller itself, rather than from the predefined search space and the NAS procedure?

1 comment

Since the author did not give any reply, I think the author has given up the submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.