- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TiNAS, an LLM-guided neural architecture search framework for time series forecasting. Instead of using a fixed model structure, TiNAS searches for task-specific architectures within the PatchTST encoder, including attention configurations, feed-forward layers, normalization placement, local convolutional operators, and network depth.
An autoregressive LLM controller generates architecture decisions based on task descriptions, previous choices, and validation feedback. The controller is optimized through reinforcement learning, while a weight-sharing supernet and progressive candidate selection are used to improve search efficiency. The selected architectures are subsequently retrained from scratch.
Experiments on eight forecasting datasets show that TiNAS generally improves upon PatchTST and achieves competitive performance against other forecasting models. The authors also conduct ablation studies on the search space, controller, and search strategy.
Critical strengths
-
A meaningful research problem and a well-designed framework. Adapting forecasting architectures to different datasets and prediction horizons is a worthwhile research direction. The paper presents a reasonably complete framework that integrates fine-grained architecture search, performance feedback, weight sharing, and progressive candidate verification.
-
Comprehensive experiments with useful observations. The evaluation covers multiple forecasting datasets and prediction horizons. The results generally improve upon the original PatchTST, and the ablation studies provide evidence for the contributions of different controller and search components. The observation that preferred architectures vary across tasks is also useful.
Critical weaknesses
-
The advantage of using an LLM controller is not well established. The authors compare TiNAS with uniform random sampling, but this is a rather limited baseline for neural architecture search. Comparisons with established methods, such as Bayesian optimization, evolutionary algorithms, or conventional RL controllers, are missing. Without these comparisons, it is difficult to tell whether the LLM offers any real advantage over simpler optimization methods.
-
What does the LLM actually learn about the forecasting tasks? As described in Appendix A.2, the controller mainly uses basic task information, previous architecture choices, and validation results. It does not explicitly consider important statistical characteristics of the time series. This raises a question about whether the LLM really learns how different data characteristics relate to architectural choices, or whether most of the improvement simply comes from reward-driven search.
-
The evaluation is restricted to PatchTST. Although TiNAS is presented as a framework for task-adaptive architecture search, all architectural modifications are made within the PatchTST encoder. Other forecasting models are included only as baselines. I would like to see whether the proposed controller can also work with other architectures, such as iTransformer. Otherwise, the generality of the approach remains limited.
-
The performance gains are not particularly impressive. TiNAS achieves competitive results, but the improvement over the strongest forecasting baseline is quite small. Some of the search-space ablations also show only marginal gains. Given the complexity of the proposed framework, I was expecting more substantial or consistent improvements.
-
The search cost is another concern. TiNAS requires a separate architecture search for each dataset and forecasting horizon. While the authors provide some efficiency measurements, a complete comparison of search costs against simpler NAS methods is still missing. Also, can the learned search policy be reused for new tasks, or does the entire search process need to start over? This matters for the practical usefulness of the framework.
Questions for authors
-
Why is a pretrained LLM necessary for the architecture controller? Could the authors compare TiNAS with conventional NAS optimizers under the same search space and evaluation budget? This would help establish whether the LLM provides benefits beyond standard search algorithms.
-
What role does the LLM's pretrained knowledge play? Could the authors provide evidence that the controller learns useful relationships between time series characteristics and architectural choices, rather than simply improving through reinforcement learning and validation feedback?
-
How general is the proposed method? Can the controller be applied to other forecasting architectures? Even a limited experiment would help clarify whether the approach generalizes beyond PatchTST.
-
Is the additional search cost justified by the performance improvement? Could the authors report the complete search cost, including final retraining, and compare the accuracy–cost trade-off with simpler optimization methods?