- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LG-Transformer, an LLM-guided neural architecture search (NAS) framework for time series forecasting. The method employs a pretrained LLM as an autoregressive controller to generate Transformer encoder architectures from a structured search space. A weight-sharing supernet is used for efficient candidate evaluation, and shortlisted architectures are subsequently retrained from scratch to select the final forecasting model. The search space covers encoder depth, normalization placement, attention configurations, activation functions, FFN expansion ratios, and residual-path operators. Experiments on eight multivariate forecasting benchmarks show that LG-Transformer achieves competitive performance against manually designed forecasting models, obtaining the best MSE in 29 out of 32 settings. The paper also provides ablation studies on framework components, controller architectures, optimization objectives, and input-window robustness.
Critical strengths
- A practically motivated automated architecture search framework. The paper addresses the relevant problem of reducing manual architecture engineering for time series forecasting. The combination of an architecture controller, a weight-sharing supernet, and a final architecture retraining stage forms a reasonable search-to-deployment pipeline. The final forecasting model does not require the LLM controller during inference, which is a practical advantage.
- Competitive forecasting results across multiple benchmarks. The discovered architectures achieve competitive results across several widely used forecasting datasets and prediction horizons. The evaluation covers datasets with different dimensionalities and temporal resolutions. The paper also provides comparisons among different controller backbones and framework variants, offering some evidence of the effectiveness of the proposed pipeline.
Critical weaknesses
- Insufficient comparison with NAS methods and inadequate validation of the core LLM-guided search contribution. My primary concern is the lack of comparisons with established neural architecture search strategies. Although the paper proposes an LLM-guided NAS framework, the main experiments primarily compare the final discovered architectures against manually designed forecasting models rather than alternative architecture search methods. The proposed search space mainly covers architectural configurations and local operator choices within a predefined Transformer encoder template. While these choices constitute a valid architecture search space, it remains unclear whether a pretrained LLM is necessary to explore it effectively. Although Table 2 compares different controller backbones, the paper does not provide comparisons against standard search strategies such as random search, evolutionary search, or Bayesian optimization under matched search spaces and computational budgets. Consequently, it is difficult to determine whether the reported improvements originate from the LLM-guided controller, the search space itself, or the exploration of a sufficiently large number of candidate architectures. This substantially weakens the evidence supporting the paper's central methodological contribution.
- Relatively outdated forecasting baselines and insufficient clarity regarding baseline result sources. The forecasting baselines are relatively outdated, with Time-LLM (2024) being the most recent included method. The absence of more recent forecasting models limits the strength of the empirical comparisons, particularly given the paper's claims regarding competitive or state-of-the-art forecasting performance. Additionally, the reported PatchTST results appear consistent with those in its original publication, whereas the reported Time-LLM results appear slightly worse than the corresponding published results. The paper does not clearly explain the sources of these baseline results or distinguish independently reproduced results from those taken from previous publications. Clarifying the baseline result sources and corresponding reproduction protocols would help establish the reliability and fairness of the reported comparisons.
- Insufficient evidence for the stability of the reported forecasting improvements. Although LG-Transformer achieves the best MSE in 29 out of 32 forecasting settings, its improvements over the strongest competing methods are often relatively small. In several settings, the absolute MSE differences are only around 0.002–0.003. The experiments report a single random seed, without providing results from multiple independent runs or measures of variability. It is therefore unclear whether these small improvements are stable across different runs. This concern is particularly relevant for an NAS framework, where randomness can arise from both architecture search and the subsequent training of the selected architectures. Without repeated experiments, the consistency of the reported performance advantage is not sufficiently established.
- Limited and inconclusive ablation studies. The main framework and controller ablations are conducted only on ETTh1 and ETTm1 with prediction horizon 96. This limited evaluation makes it difficult to assess whether the conclusions generalize across datasets with different characteristics and longer forecasting horizons. Moreover, several ablation variants exhibit only marginal performance differences. For instance, replacing the MSE-based controller reward with an MAE-based reward results in differences of only 0–0.002 across the four reported metrics. The performance differences among several controller backbones are also relatively small. Given the absence of multiple-seed evaluations, these results provide limited evidence that the individual design choices consistently improve forecasting performance. In particular, the experiments do not convincingly establish the necessity or distinctive advantage of employing a pretrained LLM controller.
Questions for authors
Q1. Comparison with standard NAS methods. Could the authors compare the proposed LLM-guided controller with established architecture search strategies, such as random search, evolutionary search, and Bayesian optimization, under the same search space, candidate evaluation procedure, and computational budget? Such comparisons are essential for determining whether the pretrained LLM provides a meaningful advantage over conventional architecture search methods. Reporting search-performance curves against the number of evaluated candidates would also be informative.
Q2. Baseline result sources and reproduction protocols. The reported PatchTST results appear consistent with those in its original publication, whereas the Time-LLM results appear slightly worse than the corresponding published results. Could the authors clarify which baseline results were independently reproduced and which were taken from existing publications, and explain the evaluation protocols used to obtain them?
Q3. Statistical reliability of the main forecasting results. Given that the improvements over the strongest baselines are relatively small in many forecasting settings, could the authors report results from multiple independent runs, including means and standard deviations? Ideally, these experiments should account for variability in both the architecture search process and the final training of selected architectures, rather than only retraining a single fixed architecture.
Q4. Generality and reliability of the ablation conclusions. Could the authors extend the ablation studies to additional datasets and forecasting horizons? Given the small differences between several variants, multiple-seed evaluations would also help determine whether the reported improvements are consistent and meaningful.
Q5. Computational cost of architecture search. Could the authors report the total search cost, including GPU-hours and the number of evaluated architectures? A comparison of search efficiency against non-LLM NAS strategies would help establish whether the additional computational cost of the LLM controller is justified by improved search quality.
Since the author did not give any reply, I think the author has given up the submission.