- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes FLAG, a framework for multivariate time series forecasting that unifies (i) compact modeling of lagged cross-variable relations and (ii) input-adaptive information exchange within a shared representation. The core "factor-lag" operator connects variables through shared low-rank read/write bases U, V in R^{Nxd} and a full, lag-specific factor transition B_l in R^{rxr}, giving the interaction A_l = V B_l U^T with self-terms removed; messages are aggregated over valid patch offsets into a relational context C^(0). A "context-adaptive group communication" module then performs N->K->N soft-group aggregation/redistribution whose weights are recomputed per input/patch/channel, refining C^(0) through a residual update. Parameters and compute grow linearly in the number of variables N for fixed structural hyperparameters (2Nr + Lc r^2 factor params, up to LNK membership params). The authors evaluate on eight benchmarks (Weather, Traffic, Electricity, Solar-Energy, four ETT) with T=96 and H in {96,192,336,720} against eleven baselines, plus a five-dataset component ablation, lag-setting comparisons, factor-transition visualizations, and a look-back study.
Critical strengths
- Compact, parameter-efficient lag-relation operator. Representing per-offset interactions via shared read/write bases plus full r x r lag transitions avoids O(N^2) variable-pair matrices while retaining lag specificity, and the zero-initialization lets the model fall back to a channel-independent backbone — enabling a clean ablation.
- Unified "construct-then-adapt" design that lets the same relational context both determine aggregation weights and receive updates, tying relation modeling to input-adaptive use under one objective.
Critical weaknesses
- Marginal, inconsistent empirical gains without significance evidence. FLAG is best only on Weather (avg MSE) and second-best elsewhere by thin margins; on Traffic it is worse than SOFTS (0.424 vs 0.415) and on Electricity worse than FACT (0.167 vs 0.160). Results are averaged over only three seeds with no standard deviations or significance tests, so several deltas (e.g., Weather 0.237 vs GTR 0.241) are plausibly within seed noise. The Solar-Energy MAE advantage (0.209 vs 0.248, ~15.7%) is anomalously large and inconsistent with MSE, where FLAG is only second — a metric-consistency concern. This directly undercuts the headline claim.
- The ablations do not cleanly support the design. On Traffic, factor-lag alone worsens MSE (0.444 -> 0.454); on Weather, adding group communication worsens MSE (0.146 -> 0.150). Contributions are non-additive and dataset-specific, so the value of the "unified" framework is mostly shown on Traffic/Electricity. With no theoretical grounding or identifiability analysis, the central claim that combining relations with adaptive use is beneficial is under-supported.
Questions for authors
- Please report per-seed variance / confidence intervals and significance for the main results. Are the headline deltas (Weather, Traffic) statistically significant?
- Why is the Solar-Energy MAE improvement so large relative to the small MSE gain? Please confirm the metric/target computation is identical across baselines.
- How are r, K, and L chosen, and how sensitive is performance to them? A sensitivity study is missing.
- Given factor-lag alone hurts on Traffic and group communication hurts on Weather, how do you substantiate a unified contribution rather than a per-dataset selection effect?
- Provide runtime/memory comparisons to support the linear-N efficiency claim, and evidence that the adaptive aggregation weights and memberships actually vary meaningfully across inputs.
3 comments
Response to Q1, Q2, Q3
Response to Q1
We quantified variation across training seeds for all 32 forecasting tasks on the eight datasets. For seeds 2021, 2022, and 2023, we first averaged the unrounded results over the four horizons within each seed. We then computed the sample variance and a two-sided 95% Student t confidence interval for the mean (2 degrees of freedom). Table 1 and Appendix K, Table 6 report means and standard deviations. The table below gives the variances and intervals for the four-horizon averages under the selected configurations.
| Dataset | Mean MSE | MSE variance (×10⁻⁶) | MSE 95% CI | Mean MAE | MAE variance (×10⁻⁶) | MAE 95% CI |
|---|---|---|---|---|---|---|
| ETTh1 | 0.4334 | 0.391 | [0.4318, 0.4349] | 0.4255 | 0.429 | [0.4239, 0.4271] |
| ETTh2 | 0.3604 | 2.174 | [0.3568, 0.3641] | 0.3848 | 0.133 | [0.3839, 0.3857] |
| ETTm1 | 0.3742 | 0.695 | [0.3721, 0.3763] | 0.3803 | 0.376 | [0.3788, 0.3819] |
| ETTm2 | 0.2750 | 0.491 | [0.2733, 0.2767] | 0.3169 | 0.831 | [0.3146, 0.3192] |
| Weather | 0.2409 | 2.070 | [0.2373, 0.2444] | 0.2616 | 0.557 | [0.2597, 0.2634] |
| Solar-Energy | 0.1881 | 1.387 | [0.1852, 0.1910] | 0.2082 | 3.004 | [0.2039, 0.2125] |
| Electricity | 0.1664 | 0.941 | [0.1640, 0.1688] | 0.2616 | 5.305 | [0.2558, 0.2673] |
| Traffic | 0.4228 | 0.385 | [0.4212, 0.4243] | 0.2774 | 0.963 | [0.2749, 0.2798] |
For the comparisons raised by the reviewer, we treated each reported baseline score as a fixed reference and applied a two-sided exact sign test to the three FLAG seed results. On Weather, 2 of 3 MSEs were below GTR’s 0.241 (p=1.00). On Traffic, all 3 were above SOFTS’s 0.415 (p=0.25). For Weather MAE, all 3 were below TimeFilter’s 0.269 (p=0.25). With three seeds, 0.25 is the smallest attainable two-sided p-value for this test.
In Table 1, FLAG achieves the lowest MAE averaged over the four forecast horizons on six of eight datasets. On the other two, the gap to the best MAE is small, at 0.006 on Electricity (0.262 vs. 0.256) and 0.009 on Traffic (0.277 vs. 0.268). FLAG also achieves the lowest average MSE and MAE on ETTh2, ETTm1, and Solar-Energy. Taken together, these results show that the same architecture forecasts well across datasets with different temporal patterns and widely varying numbers of variables.
Response to Q2
FLAG has the lowest Solar-Energy error on both metrics in revised Table 1. Using the displayed three-decimal values, its average MSE of 0.188 is 5.1% lower than the joint runners-up FACT and TQNet (0.198). Its average MAE of 0.208 is 10.7% lower than the runner-up DUET (0.233). These percentages use different reference models. Against DUET on both metrics, FLAG reduces MSE by 20.7% and MAE by 10.7%. The apparent disparity therefore depends on the comparator.
| Model | Avg MSE | Avg MAE |
|---|---|---|
| FLAG | 0.188 | 0.208 |
| FACT | 0.198 | 0.260 |
| TQNet | 0.198 | 0.258 |
| DUET | 0.237 | 0.233 |
FLAG computes MSE and MAE on the same standardized predictions and targets across test windows, forecast steps, and variables. MAE averages absolute errors, whereas MSE averages squared errors and gives larger errors more weight.
We compare baselines at a common input length of 96 and forecast horizons of 96, 192, 336, and 720. TimeFilter scores come from Table 8 of its paper, DUET scores from the authors’ fixed-input-length results, and the remaining baselines from recorded runs at the same forecasting settings. Each source uses its own evaluation implementation.
Response to Q3
r is the number of latent factors, K the number of soft groups, and L the communication depth. We compared a limited set of candidate configurations and chose the best-performing candidate in that comparison for each dataset, using performance across all four horizons (Appendix A, Table 3). In the main experiments, we fixed the lag-bank size at 12 for every dataset. This covers all patch offsets from 0 through 11.
| Dataset | Variables | r | K | L |
|---|---|---|---|---|
| ETTh1 / ETTh2 / ETTm1 / ETTm2 | 7 | 4 | 2 | 2 |
| Weather | 21 | 16 | 2 | 4 |
| Solar-Energy | 137 | 16 | 2 | 2 |
| Electricity | 321 | 16 | 2 | 1 |
| Traffic | 862 | 16 | 1 | 2 |
Appendix B, Figure 7 varies r, K, and L one at a time on five datasets at H=96, averaging each setting over three seeds. On Traffic, increasing r from 4 to 12 lowers MSE from 0.408 to 0.388, while r=32 raises it to 0.427. All five tested K values on ETTm1 give 0.308, and K=1 and K=2 both give 0.389 on Traffic. Communication depth also has a useful range. On Traffic, L=3 gives 0.387 and L=5 gives 0.402.
We did not tune the lag-bank size in the main experiments. The separate lag ablation (Section 4.3, Figure 4) shows that adjusting it can improve individual tasks. On Solar-Energy at H=336, reducing the bank from 12 to 6 lowers three-seed mean MSE from 0.196 to 0.193.
Response to Q4, Q5
Response to Q4
FLAG separates the modeling of cross-variable relations from their use in forecasting. All eight datasets in the main experiments use the same architecture. The factor–lag module (D) first constructs the relational context, and the group communication module (G) uses that context adaptively for the current input. Figure 3 in Section 4.2 reports mean test MSE over three seeds at H=96. On Traffic, adding D alone raises MSE from 0.443 for Base to 0.452. G alone gives 0.390, and the full model gives 0.389. On Weather, D alone gives 0.154 and the full model gives 0.153. On ETTm1 and Electricity, the full model ties D alone to three decimal places. Across all four datasets, the same D→G architecture improves on Base and matches or slightly improves on the better single module at the reported precision. These results support our aim of using one architecture across forecasting tasks with different temporal patterns and numbers of variables.
Response to Q5
With r, K, L, lag count, window lengths, and feature widths fixed, the factor relations use parameters and group memberships use at most . Operations run through factors and groups, avoiding a dense N-by-N variable-pair matrix (Section 3.5). Section 4.5, Figure 6 and Appendix J, Figure 11 report six models on the same H100 at T=H=96, batch size 8, and FP32. Training time is averaged over one complete epoch, memory is peak allocated training memory, and inference latency is measured after warm-up.
| Model | ETTm1 train (ms/iter) | ETTm1 infer (ms/batch) | ETTm1 peak (MiB) | Traffic train (ms/iter) | Traffic infer (ms/batch) | Traffic peak (MiB) |
|---|---|---|---|---|---|---|
| FLAG | 8.47 | 1.32 | 124.08 | 23.25 | 8.08 | 2,519.81 |
| SOFTS | 6.32 | 0.88 | 68.69 | 12.74 | 4.12 | 654.60 |
| TimeFilter | 7.26 | 1.05 | 109.94 | 24.99 | 9.21 | 2,578.71 |
| DUET | 17.06 | 3.49 | 126.15 | 76.66 | 23.63 | 6,025.89 |
| TimeMixer | 18.47 | 3.58 | 82.86 | 53.96 | 15.30 | 5,744.55 |
| PatchTST | 9.27 | 1.16 | 179.83 | 81.28 | 37.62 | 6,452.53 |
From ETTm1 (7 variables) to Traffic (862), FLAG’s training time per iteration increases 2.75-fold, inference latency 6.14-fold, and peak training memory 20.31-fold. SOFTS remains lighter on both. On Traffic, FLAG uses 69.7% less training time per iteration and 58.2% less peak memory than DUET, with 65.8% lower inference latency and reference MSE of 0.389 versus 0.407. The two measurements describe practical costs under the respective dataset configurations. The linear order follows from the fixed-parameter analysis.
The input-adaptive quantity is the aggregation weight α. Group memberships π are learned soft assignments shared across windows. Each window’s messages determine how much each variable contributes within its groups, so α varies with the input, patch, and message channel (Section 3.4). We froze three H=96 checkpoints per dataset and kept each window’s message values fixed while replacing α with either mean validation-window weights or weights from another test window. At fixed coordinates, weights varied across windows in every tested layer and seed. Both replacements raised MSE for every seed on all four datasets (Appendix F, Table 5). This directly tests the predictive value of matching weights to the current window.
| Dataset | Input-matched α | Validation-mean α | Window-mismatched α |
|---|---|---|---|
| Weather | 0.153 | 0.179 | 0.185 |
| Solar-Energy | 0.171 | 0.255 | 0.251 |
| Electricity | 0.138 | 0.279 | 0.301 |
| Traffic | 0.389 | 0.724 | 0.743 |
Response to W1, W2
Thank you for your insightful feedback. We respond to each point below.
Response to W1
Revised Table 1 averages over four forecast horizons for each dataset and reports the mean and standard deviation for FLAG over three fixed seeds. Model rankings vary by dataset and metric. FLAG ranks first in 9 of 16 comparisons, including ties. Its average MSE on Traffic is 0.423 versus 0.415 for SOFTS, and on Electricity it is 0.166 versus 0.160 for FACT. On Weather, FLAG has the lowest average MAE (0.262). Its average MSE of 0.241 is higher than TimeFilter’s 0.239 and matches the reviewer’s GTR reference of 0.241 at three decimals. The overall ranking supports FLAG’s performance across tasks. Q1 reports uncertainty and test results, and Q2 addresses Solar-Energy.
Response to W2
FLAG connects two stages of forecasting. Factor-Lag (D) learns lagged relations shared across windows. Group Communication (G) uses the current window to determine which messages receive more weight. To test how these stages work together, we held the encoder, forecasting head, and training settings fixed within each dataset at T=H=96 and compared Base, D alone, G alone, and the full model (Section 4.2, Figure 3). The table reports mean test MSE over three seeds.
Traffic illustrates why the two stages should be evaluated together. D alone gives an MSE of 0.452, above Base at 0.443. When G uses D’s relations in the current window, the full model reaches 0.389, slightly below G alone at 0.390. On Weather, D alone already reaches 0.154, and adding G gives 0.153. The same D→G design improves on Base across all four datasets, from 7 variables on ETTm1 to 862 on Traffic. The contribution of each stage varies, while their combination allows one architecture to accommodate datasets with different temporal patterns and variable counts. A unique physical interpretation of individual latent factors would require additional identifiability assumptions.