acceptodds
Under review as a conference paper at ICLR 2027

Which Channels, or How Many? Matched Controls for Data-Derived Attention Masks in Multivariate Forecasting

Abstract

Forecasters that mix channels with attention are often given a fixed relational mask estimated from data, and the benefit is usually reported against the unmasked model. Such comparisons change more than the graph: implementations, training runs and machines differ, and a mask also makes attention sparse, so a gain does not show that the estimated edges matter. We study this question for three masks estimated from road-sensor series (partial correlation, Pearson correlation and lead–lag communities, assigned to separate heads of an inverted Transformer), using two controls trained in the same launch as the masked model: an all-ones mask that leaves every parameter and setting unchanged, and a rewiring of each relation that keeps every sensor's degree. We find that the masks lower test error against the all-ones control in all 22 dataset–horizon cells on six road networks, by 13% on average, and that the gain grows with the forecast horizon on every network; on three non-road series they do not help. We find that degree-preserving rewiring keeps only the smaller part of the gain, so most of it requires the estimated edges rather than sparsity alone. We also find that pairing runs from different launches moves the estimated effect by several points, more than the spread between seeds, which is why the controls must be trained together. We release the masks of the rewiring study, estimated without the test split, with their builder, a per-run configuration table that names each run's mask file, and the analysis code.

Primary area:
learning on time series and dynamical systems
Keywords:
multivariate time series forecasting, attention masks, channel dependence, graph priors, traffic forecasting, controlled evaluation, degree-preserving rewiring, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_x43yOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the benefits of data-derived attention masks in multivariate forecasting arise from meaningful channel relationships or merely from attention sparsity. The authors construct three relational masks using partial correlation, Pearson correlation, and lead–lag communities, and evaluate them in an inverted Transformer.

The evaluation compares the masked model against an all-ones control and degree-preserving rewired masks. Experiments show consistent improvements on six road networks, while the rewiring results suggest that the estimated relational structure contributes beyond sparsity alone. The main contribution is an empirical evaluation protocol rather than a new forecasting architecture.

Critical strengths
  1. The paper addresses an important distinction between the benefits of meaningful relational structure and those of sparsity-induced regularization. This question has implications beyond time-series forecasting.

  2. Using the same implementation for FULL and ONES avoids an identified implementation confound, while degree-preserving rewiring provides a useful comparison beyond conventional dense-versus-sparse ablations. The consistent results across the tested road networks are encouraging.

Critical weaknesses
  1. Incomplete reproducibility of the main experimental evidence. The masks and construction procedure used in the first launch were not retained, and the authors cannot verify whether their estimation excluded the test split. Consequently, the reported 22-cell aggregate relies partly on unverifiable mask constructions. Although the released-mask experiments provide supporting evidence, they cover only four networks and two forecasting horizons. This limits confidence in the broader conclusions.

  2. The experimental controls do not fully support some interpretations. Degree-preserving rewiring also changes community structure and other graph properties. Therefore, FULL outperforming RW establishes an advantage over the selected randomization baseline, but does not isolate the contribution of individual edge identities. Similarly, the cross-launch experiments involve differences in hardware, mask construction, or batch size, making it difficult to establish the additional necessity of same-launch execution beyond careful configuration matching.

Questions for authors
  1. Can the authors reproduce the main findings using the released mask builder across all forecasting horizons? If this is infeasible, which headline conclusions remain supported when the analysis is restricted to experiments with verifiable mask provenance?

  2. Would FULL retain its advantage under alternative randomization controls that preserve more of the original graph topology?

  3. Can the authors clarify how random seeds and computational environments are matched across experimental arms, and provide statistical analysis accounting for dependence between horizons on the same network?

1 comment
AuthorsOpenReview ↗

Re-run with the released masks, a topology-preserving null, and network-level tests

Thank you. We agree with both weaknesses and re-ran the study to answer Q1 and Q2. All new results below come from a re-run after submission, whose pass/fail criteria were written down before any run. New material in the revised PDF is in blue.

Q1 / W1. We trained all 24 cells (6 networks × 4 horizons) again, one launch per cell and seed, with explicit seeds and every mask's checksum logged (Section 4.6, Table 5). The four PEMS districts use the released masks themselves. For METR-LA and PEMS-BAY the original mask files could not be regenerated, so we rebuilt them from the public data on the training split only.

  • The masked model beats all-ones in 24/24 cells, by 10.6% on average (range 0.2–22.9%); 6/6 networks, p=0.031p = 0.031.
  • In the 8 cells of the original rewiring launch, the re-run agrees with the submitted gains within 1.7 points.
  • METR-LA does not replicate in size: 0.2–4.8%, and no interval excludes zero (submitted first launch: 9.0–21.8%). The revision says this in the abstract and Section 4.6, and withdraws the METR-LA effect.

W2 / Q2. Agreed: unrestricted rewiring also destroys community structure. We added a null that keeps it: block-preserving rewiring keeps every degree and the number of edges between each pair of lead–lag communities. It keeps 57–98% of the masked model's gain over all-ones. So the supported statement is about which communities a sensor attends to, more than about individual edges, and the revision words it that way (Section 4.6 and Table 8). With five unrestricted draws per cell, the masked model is below every draw in 23 of 24 cells.

Q3. Seeds and environments. In the submitted launches the scripts passed no seed flag, and repetitions were paired by index. In the re-run, every arm of a cell and seed uses the same explicit seed (Section 4.6).

Q3. Dependence between horizons. We now test at the network level: 6/6 networks better, exact sign and Wilcoxon p=0.031p = 0.031 (the smallest value six networks allow), in both the submitted results and the re-run (Sections 3.3 and 4.1).

W2. Same-launch execution. Agreed: our launches differ in machine and sometimes in mask build or batch size (Section 4.4), and we do not separate these. What the data support is that pairing runs across launches, as we did, moves the estimate by several points; we do not claim the launch itself is the cause. The practical advice is to train the controls under the same machine, build and batch size, and one launch is the simplest way to do that. Section 4.4 and the abstract now say this.

Reviewer_7Cm5OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The submission examines the effect of fixed data-dependent attention masking in an inverted transformer forecasting traffic data. The channels of the time series, processed as separate tokens in the inverted transformer setup, pass information around via masked attention where three specific masks are derived from statistics of the channels. The point of comparison is an unrestricted all-to-all attention, and the central observation is that the masking boosts performance on traffic forecasting but not on data related to stocks or solar observations.

Critical strengths

That the masking helps on traffic data but not everything tested is an interesting topic of investigation, and the authors approach it with a reasonable setup.

Critical weaknesses

The empirics in this work are not ready for publication. There are numerous disclaimers in the text about data management mishaps that call the bulk of the results into question. Repeatedly, it's noted that results are shown for which there is no record of the precise configuration used. Given the small scale of the models studied (~1.6M parameters), it is hard to think of a reason other than time pressure from the submission deadline that the problematic results could not have been rechecked. The transparency is commendable, but it remains that the paper's takeaways are not adequately supported.

Additionally, the central phenomenon -- that attention masks derived from the raw time series can outperform unrestricted learned attention -- is underexplored. Are all three derived masks equally important for the effect? How sensitive is the effect to the effective threshold used in each mask? Are the fixed masks necessary across all depths? Please see the questions for authors for some ideas for how to shed more light onto the origins of the effect.

Questions for authors
  • The same fixed masks are repeated across all three layers, and multiple times within each layer. There are several variations worth exploring to more substantially illuminate the effect, especially given how quick the training runs should be. First, to check that the masking is truly beneficial across all three layers, compare to masking up to layer ll and then unmask/ONES for the remaining L−lL-l layers. Second, in addition to the fixed degree scrambled masks (which is a good check), what's the effect of scrambling one of the three at a time? It would be interesting if, for example, the performance boost is primarily sensitive to the lead-lag communities and much less so to the other two. Third, the definitions of the relations used to define the masks involve selecting a threshold or degree k -- what's the sensitivity to these choices? Finally, it seems reasonable that the effect would vanish in the limit of infinite data -- can you show that the delta grows as the dataset shrinks (by training on eg 1/2, 1/4, and 1/8) of the data?

  • How do the masked and unmasked models perform on the longer horizons when repeating the shortest horizon in an autoregressive rollout? (ie use the H=12 model to forecast out to H=24, 48, and 96). Is the effect still present?

1 comment
AuthorsOpenReview ↗

We re-ran everything and ran each ablation you proposed

You are right that the runs are cheap enough to re-check, so we re-ran the study. All new results below come from a re-run after submission, whose pass/fail criteria were written down before any run. New material in the revised PDF is in blue.

W1. The re-check. All 24 cells were trained again with explicit seeds and every mask's checksum logged (Section 4.6, Table 5). The masked model beats all-ones in 24/24 cells, by 10.6% on average; the 8 cells with released masks agree with the submission within 1.7 points. One network does not hold up: on METR-LA the gain is only 0.2–4.8%, with no interval excluding zero (submitted first launch: 9.0–21.8%). The revision says so in the abstract.

Your ablations (four PEMS districts, H=12H = 12 and 96, three seeds; "keeps" = share of the masked model's gain over all-ones; Section 4.6 and Appendix F):

  • Masks in the first ll layers only: layer 1 alone keeps 14–72% of the gain, and layers 1–2 keep 70–99%, so the third layer adds little in some cells and up to 30% in others.
  • One relation scrambled at a time: scrambling Pearson loses 2–23% of the gain; scrambling partial correlation or lead–lag alone loses almost nothing, so the remaining relations cover for them.
  • Sensitivity to kk: the gain depends on kk (k=8k = 8 keeps 54–124%, k=32k = 32 keeps 75–160%).
  • Less training data: we expected the gain to grow as data shrink. It does not: the pattern is monotone in only 1 of 4 cells, and at H=96H = 96 on PEMS08 the gain falls from 17.3% to −0.6% with 1/8 of the data.
  • Autoregressive rollout: rolled out from the H=12H = 12 models, the masked model keeps lower error at 24, 48 and 96 steps (8.3–16.8%, 12.3–22.4% and 14.2–21.8% on PEMS and PEMS-BAY). On METR-LA the effect vanishes: 1.2%, 0.2% and −2.5%.

The results that go against us are reported in the revision as plainly as the others.

Reviewer_MZkJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates if fixed relational masks estimated from data can be helpful to improve multivariate time series forecasting, after controlling for implementation, launch, and sparsity. It considers three types of masks estimated from the training data: partial correlation, Pearson correlation, and lead lag communities. These masks are assigned to different heads of an inverted Transformer. The masked model (FULL) is compared with two controls trained in the same launch: an all-ones mask using the same masked implementation (ONES), and degree-preserving rewiring of each relation (RW).

The main findings are: (i) On six road networks, FULL performs better than ONES in all 22 dataset–horizon settings, with an average error reduction of 12.9%. The improvement becomes larger for longer horizons; (ii) In the rewiring launch, FULL performs better than RW, and RW performs better than ONES, which suggests that most of the improvement comes from the estimated relational edges; (iii) On three non-road datasets, the masks do not show consistent improvements; (iv) Comparing results across different launches can change the estimated improvement by up to 4.6 percentage points, which is comparable to the improvement at short horizons.

Critical strengths

The matched control design is clear and novel. FULL and ONES use the same masked attention implementation, and only the mask buffer is changed within the same launch, whereas the degree of each sensor is unchanged. This helps separate the effect of edge identity from sparsity, providing a stricter comparison than the common approach, which usually compares a masked model with an unmasked model.

The analysis of launch effects is also useful beyond this paper. The results demonstrate that the performance of the same model can vary across different launches by 0.4–4.1 times for different seeds. Also, comparing results from different launches can change the estimated improvement by up to 4.6 points. This improvement is comparable to the improvement at short horizons. I belive that this shows an important issue in the evaluation of forecasting models.

Critical weaknesses

The main results are only supported by (8/22) verifiable settings, and one important code check does not actually work. Only the rewiring launch has the masks available (PEMS03/04/07/08, H=12/96). But for METR-LA and PEMS-BAY, the masks are not provided. The other 14 settings are from earlier launches, where the masks and the code used to build them were not saved, so it is hard to verify whether the test data was excluded when constructing the masks.

Two important control experiments are also missing, and the rewiring results only use three random draws. The authors do not test using the union mask for all attention heads or trying a balanced/rotated head allocation. So it is not clear whether the 3/3/2 allocation is actually needed or better than using the same mask for all heads. For the rewiring experiment, there are only 3 draws for each setting. Therefore, I believe that the claim that the identity effect is larger than the variation between draws in all 8 settings is not that convincing.

Questions for authors
  1. In compute_stats.py, the assertion paired_runs_on_superseded_arrays == 0 seems to always pass, since build_manifest.py sets superseded_array_used = "no" for every run, and it is never changed to "yes". What does this assertion actually check? Also, how can you verify that the masks from the first launch did not use the test split?

  2. Can you test the explanation for the larger gains at longer horizons using the existing logs, without retraining? For example, you could group sensors based on the autocorrelation of their recent history and check whether FULL improves more over ONES for sensors with lower autocorrelation. If this is not possible, I suggest presenting the horizon explanation as a hypothesis in the Limitations section.

1 comment
AuthorsOpenReview ↗

You found a real error (Q1); the re-run, head controls, more draws and the sensor split

Thank you for reading the code this closely. Q1 finds a real error, so we start there. All new results below come from a re-run after submission, whose pass/fail criteria were written down before any run. New material in the revised PDF is in blue.

Q1. You are right: the assertion checks a constant and verifies nothing, and the supplement wrongly described that field as a per-run record. The real evidence was a separate check that covers only the rewiring launch: all 12 rewired draws regenerate byte-exactly from the released arrays, and none matches a superseded one. For the first launch no such check exists, so we cannot verify that its masks excluded the test split. The revised code computes the field from the regeneration check (96 masked runs verified, 98 not verifiable) and the check can now fail; the PDF says so in Section 3.3 and Appendix B. The supplement's README still has the old sentence, and we will correct it.

W1. We re-ran all 24 cells with every mask's checksum logged. The PEMS districts use the released arrays; for METR-LA and PEMS-BAY the masks were rebuilt from the public data on the training split. The masked model beats all-ones in 24/24 cells, by 10.6% on average; 6/6 networks, p=0.031p = 0.031. METR-LA does not replicate in size (0.2–4.8%, no interval excluding zero), and the revision says so in the abstract and Section 4.6.

W2. Union mask and head allocation (four PEMS districts, H=12H = 12 and 96, three seeds; Section 4.6 and Table 8):

  • The union of the three masks on every head is worse than separate heads in 8/8 cells, by 1.5–10.3%.
  • The two rotations of the 3/3/2 allocation stay within 1% of it in only 5 of 8 cells, so our criterion for "allocation does not matter" is not met.
  • A balanced 2/2/2 allocation at six heads keeps a gain of 4.2–21.5% over its own six-head all-ones control; 3/3/2 at eight heads gives 4.3–22.9%.

W2. Rewiring draws. With five draws per cell, the masked model is below every draw in 23 of 24 cells.

Q2. The re-run keeps per-sensor errors, so we ran your split. It is post hoc, as it was not among the pre-written criteria. Sensors whose training series has below-median autocorrelation at lag HH gain more in 21 of 24 cells. This supports the explanation, but the revision still presents it as a hypothesis in Limitations, as you suggest (Section 5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.