acceptodds
Withdrawn from ICLR 2027

TiNAS: Adapting Time Series Forecasting Architectures with Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The architecture best suited to a forecasting task can vary with its temporal dynamics, variable dependencies, and forecast horizon. Nevertheless, most forecasting models adapt to new tasks by optimizing parameters within a fixed architecture, leaving their internal structure unchanged. We introduce TiNAS, a task-feedback-driven framework that extends forecasting adaptation from parameter estimation to architecture optimization. It preserves the PatchTST representation pathway while searching its encoder at a fine granularity, including normalization placement, attention configuration, local temporal operators, feed-forward design, and depth. An autoregressive LLM controller generates these decisions sequentially by conditioning on the forecasting task, preceding architectural choices, and observed performance, and learns its search policy from validation MSE rewards. A weight-sharing supernet and progressive architecture derivation strategy enable efficient exploration, after which the highest-ranked architectures are independently retrained from scratch and selected by validation performance. Across eight forecasting benchmarks, TiNAS achieves the best average MSE and MAE on four datasets and improves the average MSE of PatchTST on seven. Controlled ablations further confirm the complementary roles of the search space, controller, and search strategy. These results show that task-specific structural adaptation can substantially extend the capability of an established forecasting backbone.

Primary area:
learning on time series and dynamical systems
Keywords:
time series forecasting, neural architecture search, large language models, task-adaptive architectures, Transformers

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ye9MOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TiNAS, an LLM-guided neural architecture search framework for time series forecasting. Instead of using a fixed model structure, TiNAS searches for task-specific architectures within the PatchTST encoder, including attention configurations, feed-forward layers, normalization placement, local convolutional operators, and network depth.

An autoregressive LLM controller generates architecture decisions based on task descriptions, previous choices, and validation feedback. The controller is optimized through reinforcement learning, while a weight-sharing supernet and progressive candidate selection are used to improve search efficiency. The selected architectures are subsequently retrained from scratch.

Experiments on eight forecasting datasets show that TiNAS generally improves upon PatchTST and achieves competitive performance against other forecasting models. The authors also conduct ablation studies on the search space, controller, and search strategy.

Critical strengths
  1. A meaningful research problem and a well-designed framework. Adapting forecasting architectures to different datasets and prediction horizons is a worthwhile research direction. The paper presents a reasonably complete framework that integrates fine-grained architecture search, performance feedback, weight sharing, and progressive candidate verification.

  2. Comprehensive experiments with useful observations. The evaluation covers multiple forecasting datasets and prediction horizons. The results generally improve upon the original PatchTST, and the ablation studies provide evidence for the contributions of different controller and search components. The observation that preferred architectures vary across tasks is also useful.

Critical weaknesses
  1. The advantage of using an LLM controller is not well established. The authors compare TiNAS with uniform random sampling, but this is a rather limited baseline for neural architecture search. Comparisons with established methods, such as Bayesian optimization, evolutionary algorithms, or conventional RL controllers, are missing. Without these comparisons, it is difficult to tell whether the LLM offers any real advantage over simpler optimization methods.

  2. What does the LLM actually learn about the forecasting tasks? As described in Appendix A.2, the controller mainly uses basic task information, previous architecture choices, and validation results. It does not explicitly consider important statistical characteristics of the time series. This raises a question about whether the LLM really learns how different data characteristics relate to architectural choices, or whether most of the improvement simply comes from reward-driven search.

  3. The evaluation is restricted to PatchTST. Although TiNAS is presented as a framework for task-adaptive architecture search, all architectural modifications are made within the PatchTST encoder. Other forecasting models are included only as baselines. I would like to see whether the proposed controller can also work with other architectures, such as iTransformer. Otherwise, the generality of the approach remains limited.

  4. The performance gains are not particularly impressive. TiNAS achieves competitive results, but the improvement over the strongest forecasting baseline is quite small. Some of the search-space ablations also show only marginal gains. Given the complexity of the proposed framework, I was expecting more substantial or consistent improvements.

  5. The search cost is another concern. TiNAS requires a separate architecture search for each dataset and forecasting horizon. While the authors provide some efficiency measurements, a complete comparison of search costs against simpler NAS methods is still missing. Also, can the learned search policy be reused for new tasks, or does the entire search process need to start over? This matters for the practical usefulness of the framework.

Questions for authors
  1. Why is a pretrained LLM necessary for the architecture controller? Could the authors compare TiNAS with conventional NAS optimizers under the same search space and evaluation budget? This would help establish whether the LLM provides benefits beyond standard search algorithms.

  2. What role does the LLM's pretrained knowledge play? Could the authors provide evidence that the controller learns useful relationships between time series characteristics and architectural choices, rather than simply improving through reinforcement learning and validation feedback?

  3. How general is the proposed method? Can the controller be applied to other forecasting architectures? Even a limited experiment would help clarify whether the approach generalizes beyond PatchTST.

  4. Is the additional search cost justified by the performance improvement? Could the authors report the complete search cost, including final retraining, and compare the accuracy–cost trade-off with simpler optimization methods?

Reviewer_HQLxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TiNAS, a framework in which an LLM searches for PatchTST encoder architectures suited to individual forecasting tasks. The patch embedding and prediction head are left as they are, while the LLM is given the task of selecting the architecture rather than doing the forecasting itself. The search space is made up of 14 architectural decisions that cover normalization placement, attention configuration, local temporal operations, FFN design and encoder depth. An LLM controller is trained on rewards based on validation MSE, with task information, earlier architectural choices and the performance history all fed in. The framework brings together weight-sharing evaluation, progressive candidate selection and independent retraining. As for validation, TiNAS is run on eight datasets and four prediction horizons, where it reports the lowest overall average MSE and MAE of the compared models and improves on PatchTST’s average MSE for six datasets. Component-wise ablation studies and cross-horizon architecture evaluations are also carried out to look at the effects of the search design and of task-specific architecture selection.

Critical strengths
  1. TiNAS restricts its architectural search to the encoder while preserving the PatchTST patch embedding, channel-independent modeling, and prediction-head design. Table 5 specifies a PatchTST-style reference configuration, and Table 3 compares this Fixed reference with the Full search space. Beyond aggregate benchmark performance, Table 7 reports the selected configurations for all 32 dataset–horizon settings, while Figure 3 evaluates the four horizon-specific architectures at all four target horizons on each of ETTm1 and ETTh1. The configuration selected for the target horizon achieves the lowest MSE among the four evaluated configurations in six of eight settings. These analyses provide inspectable evidence of horizon-dependent architectural preferences within the examined settings.

  2. The default search protocol separates shared-weight candidate screening from the evaluation of independently retrained finalists. Section 3.3 describes the use of training data for supernet optimization and finalist retraining, while validation data guide controller updates, candidate ranking, and early stopping. Appendix A.3 specifies that the three shortlisted architectures are retrained from scratch and ranked by validation MSE, validation MAE, and inherited derivation rank. Under the stated search-and-selection protocol, the selected architecture and checkpoint are then evaluated once on the test set. This reduces reliance on shared-weight scores for final selection, although those scores still influence candidate shortlisting.

Critical weaknesses
  1. The evidence that the reported gains are stable and competitive is thin. The difference in overall average MSE between TiNAS (0.319) and iTransformer (0.320) is small, but the variability across independent search and retraining runs is not reported, making it difficult to assess the reproducibility of this difference. Appendix A.1 describes Fixed as using the PatchTST-style reference configuration, yet its average MSE differs from that of PatchTST in Table 1 on all three overlapping datasets. On ECL, for example, Fixed already reaches 0.189 against PatchTST’s 0.205, so 0.016 of the 0.018 improvement that the paper highlights as notable is obtained before any search takes place and only 0.002 is attributable to the search itself, while on ETTm1 the direction reverses (0.392 vs. 0.387). To relate the Fixed–Full gains in Table 3 to the improvements over PatchTST in Table 1, the sources of the PatchTST and Fixed results and any differences in implementation, model configuration, training, or tuning conditions should be clarified. Finally, comparisons with additional relevant forecasting baselines such as TimeMixer, TimeMixer++, ModernTCN, and SOFTS are absent, limiting the assessment of competitiveness against relevant alternatives.

  2. The added benefit of the LLM controller and of the full search space has yet to be firmly established. The controller ablation studies and the Uniform Sampling comparison do lend support to some of the design choices. Even so, every one of the controller ablation variants keeps the LLM, and Uniform Sampling is not a search method trained on validation feedback. Thus, the extra value of the LLM, as against a trainable non-LLM controller given the same feedback and evaluation budget, is hard to assess. As for Table 10, it shows Fixed Attention, No Local and Fixed FFN reporting the same average MSE and MAE as Full on Weather at the reported precision, and Fixed FFN also matches Full on ETTm1 (0.385/0.399), as does w/o LoRA Updates in Table 2. The largest gains cited for the controller and the search strategy are likewise concentrated on Exchange-720, where the variants in Tables 8 and 11 span 0.865 to 1.071 MSE and w/o Periodic Derivation (0.865) is in fact marginally better than Full + Retraining (0.868); a spread of this size on a single setting is itself a sign of noise. This need not mean that these components are redundant, but repeated experiments are called for to find out how consistently the full search space and the controller add anything over the restricted alternatives.

Questions for authors

The following are clarification requests, and the answers would affect my assessment.

  1. Appendix A.1 describes Fixed in Table 3 as using the PatchTST-style reference configuration in Table 5, yet its average MSE differs from that of PatchTST in Table 1 on ETTm1, ECL, and Weather. Could you clarify the relationship between these two baselines, including the sources of their results and any differences in implementation, model configuration, training, or tuning conditions? Please also clarify which conditions are matched in the Fixed–Full comparison in Table 3.

  2. Table 4 replaces the LLM controller only with uniform sampling. Are there any results that set TiNAS against a trainable non-LLM controller, such as a small reinforcement-learning or evolutionary controller, using the same search space, task information, validation feedback, and the same budgets for candidate evaluation, selection and retraining?

  3. How many independent runs underlie the main results in Table 1 and the ablations in Tables 2 to 4? The only variability reported is for wall-clock timing in Table 12. Please distinguish repeated retraining of a chosen architecture from re-running the search itself, and in either case report the mean and the variability for the key comparisons.

Reviewer_BfvgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the architecture search problem inside PatchTST macro structure. The authors proposed a search space and used LLM as a controller to search among the space. The designed architecture is compared with previous supervised time-series models on the Informer datasets.

Critical strengths

None

Critical weaknesses

1. The paper's intellectual contribution is not clarified, and the solution is trivial

While neural architecture approaches have been deployed on time-series forecastor design, as also mentioned in line 41, the authors didn't highlight their drawbacks and why TiNAS is better than them. Furthermore, the searching space is within PatchTST macro architecture without a justification (according to Table 1, patchTST is not even the best performed baseline), which also introduced great limitations on the coverage of search space. Also, the proposed solutions are all commonly adapted by neural architecture search or LLM-guided architecture design.

2. The experiment results are marginal, not persuasive, and not highlighting the search complexity

As shown in Table 1, the designed structure remains marginal within iTransformer (almost half of the datasets iTransformer is the best), which is a baseline two years ago, with no comparisons to time-series foundation models. Also, the authors didn't provide any comparisons to the structures designed by existing neural architecture search strategies or LLM-guided forecastor designing methods.

Furthermore, Table 3 and 9 shows that "fixed" already reached .189 / .275 on ECL, which is close to the final .187 / .271 performance. Given that table 1 patchTST on ECL is .205 / .290, a majority of gain is from the re-training of PatchTST rather than the performed NAS. Ablation gains in table 11 is also marginal.

Both points are strongly determining my recommendation.

Questions for authors

Please see the weaknesses, especially on the novelty concern and baseline request. I also have the following questions:

  1. C.6 reports the running time but under a fixed configuration. What if inputs and output length varies?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.