- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents RankTS, a component-level framework for time-series forecasting architecture composition and selection. RankTS first constructs a pool of forecasting components and composes them into candidate architectures represented as directed acyclic graphs (DAGs). It then employs task-local evolution to discover effective architectures on historical time-series tasks. By jointly encoding time-series characteristics and architecture structures, RankTS learns architecture preferences from pairwise historical evaluations, enabling series-conditioned architecture recommendations. Experiments on five benchmark datasets across four forecasting horizons show that RankTS achieves the best forecasting performance in more than half of the evaluated settings, while requiring only three candidate model fits on each target series.
Critical strengths
-
Novel perspective on forecasting architecture selection. The proposed framework goes beyond conventional model selection and ensembling by exploring component-level architecture composition and recommendation. This offers an interesting and relatively underexplored direction for adapting forecasting architectures to heterogeneous time series.
-
Effective task-local architecture evolution. The task-local evolution mechanism iteratively generates new candidate architectures based on retained promising ones, enabling the discovery of high-performing architectures tailored to different historical tasks. The experimental results provide evidence of its effectiveness in identifying competitive candidates.
Critical weaknesses
-
Limited cross-dataset transferability. The results in Figure 7 suggest that the ranker's transferability across datasets remains limited. This raises concerns about the framework's generalization to new datasets, particularly when dataset-specific offline architecture search and ranker training are required. Incorporating cross-dataset adaptation or meta-learning techniques could improve the ranker's generalization and enhance the practical applicability of RankTS by reducing the need for repeated offline training.
-
Insufficient clarity regarding baseline comparisons and evaluation protocols. The library-selection baselines in Table 1 require further clarification. Specifically, how many complete-model configurations are included in the library, and which models and hyperparameter configurations are considered? Moreover, how does the overall computational efficiency of RankTS compare with library-selection approaches, given that the latter do not require the costly offline task-local evolution process? A more comprehensive comparison that accounts for both offline and target-side computational costs would strengthen the experimental evaluation. In addition, several important evaluation details, such as baseline configurations, the definition of retrieval precision, and the historical-mean ranking baseline, are primarily presented in the appendix. Including these essential details in the main text would improve clarity and reproducibility.
Questions for authors
-
How could RankTS be extended to multivariate time-series forecasting?
-
Could the authors provide concrete examples or visualizations of the forecasting architectures represented by DAGs? Additionally, what advantages does the DAG-based representation offer over a simpler sequential architecture, particularly in terms of architectural expressiveness and forecasting performance?
-
What specific tasks are used during the offline task-local evolution stage? Are these tasks constructed from the same five benchmark datasets used in the main experiments, or are additional datasets involved?
5 comments
Part 2/3
W2. Baseline comparisons and evaluation protocols (continued)
3. Offline and target-side costs
RankTS incurs substantial offline acquisition cost. Each historical task evaluates 450 candidates, with the following average cumulative evaluation times.
| Dataset | Cumulative candidate-evaluation time per task (min) |
|---|---|
| Weather | 37.94 |
| London | 40.02 |
| Wind | 38.19 |
| KDD | 37.81 |
| Electricity | 38.88 |
At 150 historical tasks per dataset, these values correspond to approximately 94.5–100.1 cumulative candidate-evaluation hours per dataset. They are summed evaluation times, rather than end-to-end elapsed times under parallel execution, and exclude ranker training and other pipeline overhead.
New queries reuse the archive and ranker without repeating evolution. The measured online breakdown is as follows.
| Dataset | Budget | Features + recommendation (s) | Training + validation (s) | Selected-model test inference (s) |
|---|---|---|---|---|
| Weather | K=1 | 0.013 | 119.70 | 1.335 |
| Weather | K=3 | 0.013 | 358.22 | 1.017 |
| Electricity | K=1 | 0.049 | 55.08 | 0.723 |
| Electricity | K=3 | 0.049 | 181.12 | 1.032 |
Library selection also has an offline cost. Evaluating 13 configurations on 150 historical series requires 1,950 fits per dataset, compared with RankTS's nominal 67,500 candidate attempts. These counts do not directly determine runtime because model costs differ.
For N queries sharing an archive, average cost includes offline acquisition and ranker training divided by N, plus feature extraction, recommendation, target fitting, and inference. Thus, the benefit of reuse depends on deployment volume. The current timings do not establish an overall cost advantage over library selection; a complete comparison also requires library-acquisition and selector-training timings under matched execution conditions.
4. Essential evaluation definitions
We will make the following definitions explicit in the main text.
-
Retrieval Precision@K. For each query, relevance means membership among the 100 lowest-test-MSE architectures in the shared finite candidate set. Precision@K is the fraction of the recommended K candidates in this reference set. Test outcomes define retrospective evaluation only; they do not train the ranker or select the deployed predictor.
-
Historical-mean architecture ranking. Architectures are ordered by their mean within-task historical validation percentile. This differs from Hist, which selects complete models using centered log validation losses.
-
Target-side selection. The final predictor is selected by validation MSE within the prescribed shortlist before test evaluation.
Q1. Extension to multivariate forecasting
Two extensions are possible.
-
Channel-independent forecasting. Apply RankTS separately to each channel and concatenate the forecasts into a multivariate output. Alternatively, aggregate architecture scores across channels to choose a shared architecture, while forecasting each channel separately. This reuses the current pipeline but does not explicitly model cross-channel dependencies.
-
Joint multivariate forecasting. Preserve both time and variable axes throughout the operators, use descriptors that characterize cross-channel relationships, and produce a multivariate prediction head. Existing variable-attention and graph operators provide relevant building blocks, subject to compatible multivariate interfaces and training implementations.
These are proposed extensions. The reported experiments fit individual univariate series and do not establish multivariate forecasting performance.
Part 3/3
Q2. Concrete architectures and the advantages of DAGs
1. An actual recommended architecture
For Wind series T248, seed 0, the second-ranked architecture decomposes the input using a moving average, processes the smooth and residual branches separately, and fuses them before forecasting. (Figure 12 in the paper)
flowchart TD
X["Input series X"]
N0["n0: Moving Average<br/>kernel_size = 25"]
N1["n1: Identity<br/>input merge: X - n0"]
N2["n2: Temporal Linear<br/>bias = true"]
N3["n3: Temporal Linear<br/>bias = true"]
N4["n4: Temporal Mixer<br/>input merge: n2 + n3<br/>expansion = 1"]
H["Linear Forecasting Head"]
Y["Forecast"]
X --> N0
X --> N1
N0 --> N1
N1 --> N2
N0 --> N3
N2 --> N4
N3 --> N4
N4 --> H
H --> Y
We will add concrete architecture visualizations, including operator settings, connections, merges, and prediction heads, together with a chain example illustrating that simple architectures remain valid recommendations.
2. Architectural expressiveness
A sequential architecture is a special case of a DAG. DAGs additionally support branches, skip connections, and intermediate-result reuse, enabling separate processing and fusion of complementary temporal information.
| Capability | Strict sequential architecture | DAG architecture |
|---|---|---|
| Sequential processing | Supported | Supported |
| Parallel branches on the same input | Not directly in a strict chain | Supported |
| Separate trend/residual processing and fusion | Requires branches | Directly represented |
| Skip connections and reuse of intermediate results | Not supported by a strict chain | Supported |
3. Forecasting performance
We compared error distributions for sequential and branched architectures. The table reports mean MSE and its 25th and 75th percentiles.
| Dataset | Architecture | Mean | Q25 | Q75 |
|---|---|---|---|---|
| Weather | Sequential | 1.3461 | 1.1910 | 1.3864 |
| Weather | Branched DAG | 8.7062 | 1.1855 | 1.2284 |
| London | Sequential | 1.2213 | 1.1702 | 1.2366 |
| London | Branched DAG | 11.1944 | 1.1693 | 1.2176 |
| Wind | Sequential | 2.7590 | 0.3729 | 0.5665 |
| Wind | Branched DAG | 1.3579 | 0.3597 | 0.4104 |
| KDD | Sequential | 0.5459 | 0.4906 | 0.5878 |
| KDD | Branched DAG | 1.4674 | 0.4829 | 0.5640 |
| Electricity | Sequential | 0.2868 | 0.2065 | 0.2599 |
| Electricity | Branched DAG | 1.4472 | 0.2064 | 0.2355 |
-
Branched DAGs have lower Q25 and Q75 on all five datasets, showing that competitive branched architectures are available.
-
Mean MSE is higher on four datasets, with Wind the exception. This indicates sensitivity to high-error candidates and prevents a blanket claim that branching improves performance.
-
This is a descriptive comparison, not an isolated branching ablation. Establishing a causal forecasting advantage requires matching operator choices, capacity, and evaluation budgets.
Q3. Tasks used for offline task-local evolution
The main offline stage uses historical series from the same five benchmark datasets; no additional datasets are involved.
-
Task definition. Each task is one historical univariate forecasting series. In the main H96 acquisition setup, input length and forecasting horizon are both 96.
-
Offline acquisition. For each dataset, 150 historical series are used. Candidates are fitted on each series' training segment and evaluated on its validation segment; test losses do not guide evolution.
-
Online evaluation. A disjoint pool of 17 series per dataset is reserved for queries. The primary evaluated cohort retains 16 Wind queries and 17 for each other dataset.
The main protocol builds the archive and ranker separately for each dataset. The additional multi-source transfer experiment is a separate evaluation using pooled source datasets and a held-out target dataset.
Part 1/3
W1. Cross-dataset transferability
We agree that the current cross-dataset results do not establish reliable generalization to arbitrary new domains. The original experiment transfers a ranker trained on up to 150 historical series from one dataset. Limited source diversity may contribute to weak transfer, but we have not isolated its causal effect.
We additionally trained on multiple source datasets and evaluated on a held-out dataset at H96. The table reports test MSE.
Multi-source transfer is competitive with some baselines, but remains below within-dataset RankTS.
Ranker transfer and archive transfer must be distinguished. The original cross-dataset experiment still uses target historical archives. Eliminating target-side offline searches requires both source-only ranker training and source-only archive construction; the new results establish this only if both conditions hold.
Adaptation remains a promising extension. Pooled historical training, domain-conditioned ranking, or meta-learning could improve transfer, but their effectiveness requires further evaluation.
W2. Baseline comparisons and evaluation protocols
1. Complete-model library
The library contains 13 configurations from 11 families: DLinear, PatchTST, iTransformer, TimesNet, TimeMixer, FEDformer, Autoformer, Informer, Transformer, Nonstationary Transformer, and LightTS.
iTransformer contributes three configurations. The tuples (d_model, n_heads, e_layers, d_ff) are (32,4,1,64), (64,4,2,128), and (128,8,2,256).
Each other family contributes one configuration. Other configurable deep baselines use (64,4,2,128) where applicable; PatchTST uses patch length 16 and stride 8.
Shared protocol. Predictors use the same data splits, sampled-window limits, and optimization budget within each comparison.
2. Library-selection policies
Hist selects the configuration with the lowest mean historical utility loss, using within-task median-centered log validation losses.
Select predicts these losses using a multi-output random forest trained on historical series descriptors.
CM3+V fits Select's top three configurations and selects the target-validation winner.
Hist and Select each fit one target predictor. CM3+V and RankTS Top3+V share a three-candidate target-fitting budget, although this does not equalize runtime or offline cost. We will make the library contents and selection rules explicit in the main text.