- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how positional encoding (PE) shapes Transformer forecasters when temporal dependence changes across input windows.
The paper reads softmax attention with an additive positional bias as a Bayesian posterior over the attended key. The content score acts as a log-prior and the PE bias as a log temporal-evidence likelihood. Under a weakly stationary Gaussian process and a simplified one-layer Transformer, it derives a population-level ELBO. The KL term compares the PE-induced lag profile with a Gaussian prior built from the autocovariance. For a mixture of stationary regimes, it shows that the best static PE cannot exceed a regime-wise oracle.
Motivated by this, the paper proposes Bayesian Time-aware Attention. BTA adds an input- and head-specific bias over a fixed bank of RBF and periodic kernels. A small MLP router sets from lag-averaged content scores and their DFT magnitude. Experiments insert BTA into Informer, PatchTST, Timer-XL and TimeBridge on eight LTSF benchmarks. They also include a synthetic two-regime GP study, ablations, case studies and an efficiency analysis.
Critical strengths
S1. Well-posed question and sound motivation. The paper asks explicitly how PE interacts with window-level non-stationarity. I think this question is rarely directly isolated in the LTSF. The Bayesian reading of positional bias is a well-chosen entry point. The paper build a clear path to solve this question. First, it reads attention as a Bayesian posterior: the content score is a prior over keys, and the positional bias is the temporal evidence. Second, it uses this view to measure how well a positional bias matches the temporal dependence of the data. Third, it shows that one shared, static bias must compromise when windows come from different regimes. It leads naturally to an input-adaptive bias, and the proposed solution is technically reasonable. Finally, the method mixes local and periodic temporal kernels, and a router sets their weights for each window and head. Different windows and heads can thus emphasize local, periodic or long-range dependence without regime labels. I think this question and motivation is very sound.
S2. Well structured paper and a simple and clear design. The paper is easy to follow. The method is simple and direct: the kernel bank is fixed, only router weights are learned, and only attention logits change. The router reads lag-averaged content attention scores together with their Fourier magnitude spectrum, a cheap summary of each window's temporal structure. The overview figure makes the whole pipeline easy to follow. And the paper give a clear and straight shared vocabulary for PEs, maps several biases to likelihood assumptions over lag. This makes PE designs easy to compare and shows clearly what is distinctive about the method.
S3. Broad experimental coverage. The paper tested on four backbones with different tokenization and attention designs, across eight benchmarks. The backbones cover point-wise (Informer), patch-wise (PatchTST, TimeBridge) and pretrained (Timer-XL) designs. Datasets and settings use public benchmarks, cover different sampling rates, dimensions and degrees of stationarity. The method improves every backbone it is added to. On TimeBridge, it is also compared with ten baselines, including recent non-stationary methods such as PULSE, DTAF and DUET, (although I thought TimeBirdge is strong enough as a LTSF method). Ablations, case studies and a controlled synthetic study with known regimes support the analysis.. The case study shows how the router picks different kernel mixtures for a weekly-periodic window and a locally dependent window.
Critical weaknesses
W1. The Bayesian view holds for any additive logit term, because the bias can take any value. It is a reinterpretation, so it does not by itself motivate the kernel mixture or the router. The ELBO analysis then uses a frozen-attention forecaster that is linear in the input, with a rank-one operator. The content pathway of attention is removed, and the paper does not show how this bound relates to the models actually trained with MSE or time–frequency MAE losses. The corollary only shows that one static PE cannot beat a regime-wise oracle. It gives no condition under which the gap is strict, and it does not show that an input-adaptive PE without regime labels can approach the oracle. The words "proving" and "strictly suboptimal" in the Introduction are therefore stronger than the appendix proof. In addition, by the paper's own definition, RoPE and absolute PE are not static, yet these are the native PEs of the backbones. So the main experiments do not test the static-versus-adaptive comparison that the theory analyzes. The synthetic ELBO study has a similar gap. Static relative biases get a posterior covariance equal to the small constant ε_q times the identity, so the KL term is dominated by log ε_q, which does not depend on forecast quality. The reported ELBO gap comes mostly from this term. Finally, the supplementary code adds design choices that the method section does not describe. Kernels are normalized over lags, a learnable per-head slope scales the bias, and the router uses a temperature. The periodic kernels are hard-coded per dataset rather than found by Fourier analysis. For TimeBridge, the bias is applied only to one PatchSampling layer, where queries are mixed by a convolution, so the lag between query and key has no clear temporal meaning.
W2. Closely related work is missing. BAM (ICLR 2026) reads PE as a Bayesian prior inside the attention softmax. DAPE (NeurIPS 2024) computes input- and head-specific positional biases with an MLP over attention scores. These two paper I think need to cite and discuss. Autoformer and Pathformer also use Fourier-based, input-adaptive lag or scale selection. The experiments also do not compare the method with static lag biases (ALiBi, T5, KERPLE, FIRE) or adaptive alternatives (DAPE) on the same backbones. This is the comparison that would isolate the value of input adaptivity. Moreover, the baselines are not always strong or consistent. Finally, the training protocol favors the method. In the released code, the method uses different learning rates, batch sizes and patience from the baseline, and its router is tuned per horizon. Results on TimeBridge come from a single seed, and the gains there are small. The parameter counts in the efficiency table seem not match the configurations.
Questions for authors
On theory and method design (W1)
- Beyond reparametrization, what testable prediction or design constraint follows from the Bayesian view? Why does it lead to a kernel mixture with a router, rather than to another form of adaptive bias? I think this is my main concern in theory design in this paper.
- Under what conditions on the regime autocovariances is the gap in the corollary strict? Under a regime-identifiability assumption, can an input-adaptive PE without regime labels approach the regime-wise oracle?
- How does the ELBO analysis, which uses a frozen-attention linear forecaster, relate to the models actually trained with MSE or time–frequency MAE losses? How do the conclusions transfer to backbones whose native PE is absolute, rotary or absent, since these are not static under your definition?
- In the synthetic ELBO study, how sensitive are the results to ε_q? Please split the KL into its mean and log-determinant terms, and explain how the calibration constants a and c are fitted. and also clarify which BTA test MSE is correct, Table 5 or Table 6.
- Where exactly is BTA inserted in each backbone? For TimeBridge, why only the PatchSampling layer? How should the lag between query and key be read when queries are mixed by a convolution? What happens if BTA is applied to Integrated Attention?
On related work and experiments (W2)
-
How do you position BTA relative to BAM, DAPE and PENGUIN? Can you compare BTA, on the same backbones, with static lag biases (ALiBi, T5, KERPLE, FIRE) and with DAPE?
-
Can you show that the baselines and BTA are trained under the same setting? And report multi-seed results with standard deviations, especially for TimeBridge.
If the authors can show that the baselines and BTA are trained under the same exp setting and report multi-seed results, add the key missing references, and include comparisons with static lag biases and DAPE, I would be happy to raise my score.