- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether the benefits of data-derived attention masks in multivariate forecasting arise from meaningful channel relationships or merely from attention sparsity. The authors construct three relational masks using partial correlation, Pearson correlation, and lead–lag communities, and evaluate them in an inverted Transformer.
The evaluation compares the masked model against an all-ones control and degree-preserving rewired masks. Experiments show consistent improvements on six road networks, while the rewiring results suggest that the estimated relational structure contributes beyond sparsity alone. The main contribution is an empirical evaluation protocol rather than a new forecasting architecture.
Critical strengths
-
The paper addresses an important distinction between the benefits of meaningful relational structure and those of sparsity-induced regularization. This question has implications beyond time-series forecasting.
-
Using the same implementation for FULL and ONES avoids an identified implementation confound, while degree-preserving rewiring provides a useful comparison beyond conventional dense-versus-sparse ablations. The consistent results across the tested road networks are encouraging.
Critical weaknesses
-
Incomplete reproducibility of the main experimental evidence. The masks and construction procedure used in the first launch were not retained, and the authors cannot verify whether their estimation excluded the test split. Consequently, the reported 22-cell aggregate relies partly on unverifiable mask constructions. Although the released-mask experiments provide supporting evidence, they cover only four networks and two forecasting horizons. This limits confidence in the broader conclusions.
-
The experimental controls do not fully support some interpretations. Degree-preserving rewiring also changes community structure and other graph properties. Therefore, FULL outperforming RW establishes an advantage over the selected randomization baseline, but does not isolate the contribution of individual edge identities. Similarly, the cross-launch experiments involve differences in hardware, mask construction, or batch size, making it difficult to establish the additional necessity of same-launch execution beyond careful configuration matching.
Questions for authors
-
Can the authors reproduce the main findings using the released mask builder across all forecasting horizons? If this is infeasible, which headline conclusions remain supported when the analysis is restricted to experiments with verifiable mask provenance?
-
Would FULL retain its advantage under alternative randomization controls that preserve more of the original graph topology?
-
Can the authors clarify how random seeds and computational environments are matched across experimental arms, and provide statistical analysis accounting for dependence between horizons on the same network?
Re-run with the released masks, a topology-preserving null, and network-level tests
Thank you. We agree with both weaknesses and re-ran the study to answer Q1 and Q2. All new results below come from a re-run after submission, whose pass/fail criteria were written down before any run. New material in the revised PDF is in blue.
Q1 / W1. We trained all 24 cells (6 networks × 4 horizons) again, one launch per cell and seed, with explicit seeds and every mask's checksum logged (Section 4.6, Table 5). The four PEMS districts use the released masks themselves. For METR-LA and PEMS-BAY the original mask files could not be regenerated, so we rebuilt them from the public data on the training split only.
W2 / Q2. Agreed: unrestricted rewiring also destroys community structure. We added a null that keeps it: block-preserving rewiring keeps every degree and the number of edges between each pair of lead–lag communities. It keeps 57–98% of the masked model's gain over all-ones. So the supported statement is about which communities a sensor attends to, more than about individual edges, and the revision words it that way (Section 4.6 and Table 8). With five unrestricted draws per cell, the masked model is below every draw in 23 of 24 cells.
Q3. Seeds and environments. In the submitted launches the scripts passed no seed flag, and repetitions were paired by index. In the re-run, every arm of a cell and seed uses the same explicit seed (Section 4.6).
Q3. Dependence between horizons. We now test at the network level: 6/6 networks better, exact sign and Wilcoxon p=0.031 (the smallest value six networks allow), in both the submitted results and the re-run (Sections 3.3 and 4.1).
W2. Same-launch execution. Agreed: our launches differ in machine and sometimes in mask build or batch size (Section 4.4), and we do not separate these. What the data support is that pairing runs across launches, as we did, moves the estimate by several points; we do not claim the launch itself is the cause. The practical advice is to train the controls under the same machine, build and batch size, and one launch is the simplest way to do that. Section 4.4 and the abstract now say this.