acceptodds
Under review as a conference paper at ICLR 2027

Beyond Static Positional Encoding: Bayesian Time-Aware Attention for Non-Stationary Long-term Time Series Forecasting

Abstract

Transformers have demonstrated strong performance in long-term time series forecasting (LTSF), yet modeling non-stationary temporal dependencies remains challenging. In this paper, we interpret self-attention as a Bayesian posterior over attended positions, combining a content-based prior induced by scaled dot-product similarity with a temporal-evidence likelihood induced by positional encoding. This formulation provides a probabilistic interpretation of how positional encoding shapes the temporal dependencies captured by self-attention. Building on this perspective, we derive a population-level evidence lower bound (ELBO) that characterizes the role of positional encoding in forecasting weakly stationary time series. Our analysis further highlights a limitation of static positional encoding: when temporal dependence structures vary across input windows, a fixed temporal bias must accommodate heterogeneous dynamics and can therefore be suboptimal for non-stationary forecasting. Motivated by this analysis, we propose Bayesian Time-Aware Attention (BTA), which employs a mixture-of-experts (MoE) router to adaptively combine temporal kernels and generate input-conditioned, head-specific positional biases. Extensive experiments on standard LTSF benchmarks demonstrate that BTA consistently improves forecasting performance across multiple Transformer backbones and achieves competitive overall performance with modest computational overhead and stable training.

Primary area:
learning on time series and dynamical systems
Keywords:
Long-term time-series forecasting, Non-stationarity, Adaptive positional encoding, Bayesian theory, Mixture-of-experts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XjGvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how positional encoding (PE) shapes Transformer forecasters when temporal dependence changes across input windows.

The paper reads softmax attention with an additive positional bias as a Bayesian posterior over the attended key. The content score acts as a log-prior and the PE bias as a log temporal-evidence likelihood. Under a weakly stationary Gaussian process and a simplified one-layer Transformer, it derives a population-level ELBO. The KL term compares the PE-induced lag profile with a Gaussian prior built from the autocovariance. For a mixture of stationary regimes, it shows that the best static PE cannot exceed a regime-wise oracle.

Motivated by this, the paper proposes Bayesian Time-aware Attention. BTA adds an input- and head-specific bias over a fixed bank of RBF and periodic kernels. A small MLP router sets from lag-averaged content scores and their DFT magnitude. Experiments insert BTA into Informer, PatchTST, Timer-XL and TimeBridge on eight LTSF benchmarks. They also include a synthetic two-regime GP study, ablations, case studies and an efficiency analysis.

Critical strengths

S1. Well-posed question and sound motivation. The paper asks explicitly how PE interacts with window-level non-stationarity. I think this question is rarely directly isolated in the LTSF. The Bayesian reading of positional bias is a well-chosen entry point. The paper build a clear path to solve this question. First, it reads attention as a Bayesian posterior: the content score is a prior over keys, and the positional bias is the temporal evidence. Second, it uses this view to measure how well a positional bias matches the temporal dependence of the data. Third, it shows that one shared, static bias must compromise when windows come from different regimes. It leads naturally to an input-adaptive bias, and the proposed solution is technically reasonable. Finally, the method mixes local and periodic temporal kernels, and a router sets their weights for each window and head. Different windows and heads can thus emphasize local, periodic or long-range dependence without regime labels. I think this question and motivation is very sound.

S2. Well structured paper and a simple and clear design. The paper is easy to follow. The method is simple and direct: the kernel bank is fixed, only router weights are learned, and only attention logits change. The router reads lag-averaged content attention scores together with their Fourier magnitude spectrum, a cheap summary of each window's temporal structure. The overview figure makes the whole pipeline easy to follow. And the paper give a clear and straight shared vocabulary for PEs, maps several biases to likelihood assumptions over lag. This makes PE designs easy to compare and shows clearly what is distinctive about the method.

S3. Broad experimental coverage. The paper tested on four backbones with different tokenization and attention designs, across eight benchmarks. The backbones cover point-wise (Informer), patch-wise (PatchTST, TimeBridge) and pretrained (Timer-XL) designs. Datasets and settings use public benchmarks, cover different sampling rates, dimensions and degrees of stationarity. The method improves every backbone it is added to. On TimeBridge, it is also compared with ten baselines, including recent non-stationary methods such as PULSE, DTAF and DUET, (although I thought TimeBirdge is strong enough as a LTSF method). Ablations, case studies and a controlled synthetic study with known regimes support the analysis.. The case study shows how the router picks different kernel mixtures for a weekly-periodic window and a locally dependent window.

Critical weaknesses

W1. The Bayesian view holds for any additive logit term, because the bias can take any value. It is a reinterpretation, so it does not by itself motivate the kernel mixture or the router. The ELBO analysis then uses a frozen-attention forecaster that is linear in the input, with a rank-one operator. The content pathway of attention is removed, and the paper does not show how this bound relates to the models actually trained with MSE or time–frequency MAE losses. The corollary only shows that one static PE cannot beat a regime-wise oracle. It gives no condition under which the gap is strict, and it does not show that an input-adaptive PE without regime labels can approach the oracle. The words "proving" and "strictly suboptimal" in the Introduction are therefore stronger than the appendix proof. In addition, by the paper's own definition, RoPE and absolute PE are not static, yet these are the native PEs of the backbones. So the main experiments do not test the static-versus-adaptive comparison that the theory analyzes. The synthetic ELBO study has a similar gap. Static relative biases get a posterior covariance equal to the small constant ε_q times the identity, so the KL term is dominated by log ε_q, which does not depend on forecast quality. The reported ELBO gap comes mostly from this term. Finally, the supplementary code adds design choices that the method section does not describe. Kernels are normalized over lags, a learnable per-head slope scales the bias, and the router uses a temperature. The periodic kernels are hard-coded per dataset rather than found by Fourier analysis. For TimeBridge, the bias is applied only to one PatchSampling layer, where queries are mixed by a convolution, so the lag between query and key has no clear temporal meaning.

W2. Closely related work is missing. BAM (ICLR 2026) reads PE as a Bayesian prior inside the attention softmax. DAPE (NeurIPS 2024) computes input- and head-specific positional biases with an MLP over attention scores. These two paper I think need to cite and discuss. Autoformer and Pathformer also use Fourier-based, input-adaptive lag or scale selection. The experiments also do not compare the method with static lag biases (ALiBi, T5, KERPLE, FIRE) or adaptive alternatives (DAPE) on the same backbones. This is the comparison that would isolate the value of input adaptivity. Moreover, the baselines are not always strong or consistent. Finally, the training protocol favors the method. In the released code, the method uses different learning rates, batch sizes and patience from the baseline, and its router is tuned per horizon. Results on TimeBridge come from a single seed, and the gains there are small. The parameter counts in the efficiency table seem not match the configurations.

Questions for authors

On theory and method design (W1)

  1. Beyond reparametrization, what testable prediction or design constraint follows from the Bayesian view? Why does it lead to a kernel mixture with a router, rather than to another form of adaptive bias? I think this is my main concern in theory design in this paper.
  2. Under what conditions on the regime autocovariances is the gap in the corollary strict? Under a regime-identifiability assumption, can an input-adaptive PE without regime labels approach the regime-wise oracle?
  3. How does the ELBO analysis, which uses a frozen-attention linear forecaster, relate to the models actually trained with MSE or time–frequency MAE losses? How do the conclusions transfer to backbones whose native PE is absolute, rotary or absent, since these are not static under your definition?
  4. In the synthetic ELBO study, how sensitive are the results to ε_q? Please split the KL into its mean and log-determinant terms, and explain how the calibration constants a and c are fitted. and also clarify which BTA test MSE is correct, Table 5 or Table 6.
  5. Where exactly is BTA inserted in each backbone? For TimeBridge, why only the PatchSampling layer? How should the lag between query and key be read when queries are mixed by a convolution? What happens if BTA is applied to Integrated Attention?

On related work and experiments (W2)

  1. How do you position BTA relative to BAM, DAPE and PENGUIN? Can you compare BTA, on the same backbones, with static lag biases (ALiBi, T5, KERPLE, FIRE) and with DAPE?

  2. Can you show that the baselines and BTA are trained under the same setting? And report multi-seed results with standard deviations, especially for TimeBridge.

If the authors can show that the baselines and BTA are trained under the same exp setting and report multi-seed results, add the key missing references, and include comparisons with static lag biases and DAPE, I would be happy to raise my score.

Reviewer_D8dSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper reads position-aware self-attention as a Bayesian posterior over key positions, with the content softmax as prior and the additive positional score as a log-likelihood of latent "temporal evidence" (Proposition 1). For a simplified single-layer Transformer it derives a population-level ELBO under a frozen-attention Gaussian likelihood (Theorem 2; Appendix D) and, modelling non-stationarity as a mixture of weakly stationary regimes, bounds the best static positional encoding (PE) by a regime-wise oracle (Corollary 3). The proposed module, Bayesian Time-aware Attention (BTA), adds to the logits a per-input, per-head bias bij=log⁡∑cωcexp⁡ψ(c)(∣i−j∣)b_{ij}=\log\sum_c\omega_c\exp\psi^{(c)}(|i-j|) over a fixed bank of 5 RBF and 6 periodic lag kernels; the weights come from a Top-kk MLP router fed with the lag-averaged content scores and their DFT magnitude (Sec. 4; Appendix J). Only the router is learned. Experiments add BTA to Informer and PatchTST (input 96) and Timer-XL (input 672), reporting average MSE reductions of 9.70%, 10.35% and 5.70% on four datasets (Tables 1, 10; three seeds in Table 11), and to TimeBridge at T=720T=720 against ten baselines on eight datasets, reporting a 2.30% average MSE reduction and best MSE on 7 of 8 datasets (Tables 2, 12). Ablations (Tables 3, 14), a two-regime synthetic study with an estimated ELBO (Appendix F) and a cost analysis (Table 13) complete the evaluation.

Critical strengths

A cheap, backbone-agnostic module with consistent multi-seed gains over vanilla attention (Tables 10, 11, 13). BTA improves 95 of 96 backbone–dataset–horizon–metric cells in Table 10, and the three-seed standard deviations in Table 11 (mostly ≤0.01\le0.01) are much smaller than the gaps for Informer and PatchTST (e.g. PatchTST ETTh1/96: 0.462±0.005→0.366±0.0040.462\pm0.005\to0.366\pm0.004), at about 0.31M extra FLOPs and 14.4K extra parameters on TimeBridge/ETTh1 (Appendix O). Some gains survive a cross-paper comparison with published vanilla numbers at the same lookback (PatchTST ETTh1 at input 96 is 0.469 in iTransformer's Table 10; BTA reaches 0.425).

The ablations separate the kernel bank from the input-adaptive routing (Tables 3, 14). On Weather/Informer, static Local+Periodic kernels give 6.00% MSE reduction, routing on the TE profile 10.65%, and adding the Fourier spectrum 14.05%; on Solar/TimeBridge the sequence is 2.02% → 4.17% → 5.65%. The adaptive component therefore contributes beyond the added priors, albeit in two fixed-seed settings with the static mixture's weight rule unspecified.

Critical weaknesses

The theoretical contribution does not support the claim made for it (Sec. 1; Sec. 3.2; Appendices E, F.3).

Corollary 3 (Eq. 5 / Eq. 62) is the inequality sup⁡ϕ∈Φstatic∑rπrLr(ϕ)≤∑rπrsup⁡ϕr∈ΦLr(ϕr)\sup_{\phi\in\Phi_{\text{static}}}\sum_r\pi_r L_r(\phi)\le\sum_r\pi_r\sup_{\phi_r\in\Phi}L_r(\phi_r), which holds for any subclass of Φ\Phi because the right-hand side is a regime-labelled oracle. No theoretical result compares the best static PE with the best regime-agnostic adaptive PE (the only such comparison is the empirical one in Appendix F), and the strictness condition in Appendix E is a conditional restatement for which no concrete process is exhibited. Theorem 2 bounds a constructed latent-variable model whose operator Mϕ,zM_{\phi,z} (Eq. 34) is fixed given zz; under a static PE the variational posterior is input-independent (Eq. 58), so that model's predictor loses all input dependence, whereas the simplified Transformer of Appendix A still adapts through qi⊤kjq_i^\top k_j (Eq. 18). The static-PE limitation is thus a property of the frozen-attention construction, not of Transformers.

The synthetic evidence in Table 6 inherits this. With σ2=1\sigma^2=1 and L=4L=4, the "expected log likelihood" column equals −2log⁡(2π)−2⋅MSE-2\log(2\pi)-2\cdot\mathrm{MSE} to within 4×10−44\times10^{-4} for RPE, ALiBi and BTA, so it is an affine function of test MSE spanning only 0.09 nats; the ranking is set by the KL column (348.1, 271.2, 51.4, 46.7). For every lag-only bias, which includes RPE and ALiBi as implemented (Fig. 7) and BTA (Eq. 7 depends only on ∣i−j∣|i-j| for a given input and head), the lag profile is identical across queries, so by Eqs. 57–58 the posterior covariance is exactly ϵqIK\epsilon_q I_K and all three pay the same 12(−Klog⁡ϵq)≈147\tfrac12(-K\log\epsilon_q)\approx147 nats in the log-determinant term. The 225–301 nat gaps between BTA and ALiBi/RPE therefore come from the Mahalanobis term in Eq. 59, which measures how closely the affinely calibrated profile tracks the regime-specific autocovariance mean γr\gamma_r; a shared profile cannot match two different γr\gamma_r while an input-dependent one can, so the term rewards adaptivity to the prior's mean, not forecasting accuracy, and its scale is set by K0−1K_0^{-1} and the per-method calibration constants. Appendix F.3 itself states that these values "are not a direct numerical evaluation" of Theorem 2, yet Secs. 3.2 and 6 present the ELBO as characterising the role of PE. The synthetic setup also gives BTA an expert bank containing the generating kernels (Eqs. 63–64; Appendix F.2) and an extra validation sweep over router width and Top-kk.

The headline comparison (Tables 2, 12) does not isolate adaptive from static PE and rests on unresolved baseline provenance, an inaccurate protocol statement, single seeds and possibly unequal tuning.

(a) BTA is compared only with each backbone's native attention and PE. TimeBridge has no positional encoding (Table 8), so TimeBridge+BTA versus TimeBridge conflates adding any relative bias with making it adaptive; there is no matched real-data comparison against ALiBi, T5 bias, KERPLE, a PENGUIN-style periodic bias, a learned per-lag bias or DAPE, and whether the "Static" weights in Tables 3 and 14 are prescribed or learned is not stated.

(b) Tables 2 and 12 state "Sequence length T=720T=720 for all models", but the iTransformer and DLinear columns of Table 12 match entry for entry the TimeBridge paper's Table 1 (per-horizon values in its Table 9), whose baselines used an input length searched over {96,192,336,512,720}\{96,192,336,512,720\}; the PatchTST ETTh1 entries (0.370/0.413/0.422/0.447) match the published PatchTST/64 values at L=512L=512 and the DLinear entries (0.375/0.405/0.439/0.472) DLinear's published L=336L=336 values. In Table 10, the vanilla Timer-XL rows for ETTh1, ETTm1 and Weather equal the Timer-XL paper's Table 12 in all 24 entries (a per-dataset model with rolling forecasting, whereas Appendix J.5 describes checkpoint fine-tuning), while the vanilla Solar row (avg 0.218) is worse than the published 0.198, so BTA's 11.80% Solar gain is about 2.5% in a cross-paper comparison and BTA is worse than the published vanilla at H=96H=96 (0.166 vs 0.162). The vanilla PatchTST Solar entries equal iTransformer's Table 10 exactly, whereas vanilla PatchTST on ETTh1 (0.516) is far worse than iTransformer's 0.469 at the same input length. Which rows were re-run and which were taken from other papers, and under what protocol, must be clarified.

(c) Against the TimeBridge paper's own T=720T=720 results, the vanilla TimeBridge numbers in Table 12 are worse on 7 of 8 datasets; TimeBridge+BTA then improves average MSE by about 1.3% rather than 2.30% (computed as in the paper), ties on ETTm2 and is worse on Electricity (0.152 vs 0.149). Margins over the paper's own vanilla TimeBridge are 0.002–0.006 on five datasets, although BTA is lower in 31 of 32 horizon-level MSE cells. No multi-seed means or standard deviations are reported for either model (Table 11 covers only the other backbones), while the TimeBridge paper's three-seed standard deviations are 0.002–0.018, and Appendix N discloses that the selected ETTh1 k=1k=1 and Solar k=3k=3 runs used a different batch size and number of patch-downsampling layers from the other sensitivity runs, without stating whether the Table 2 configurations match vanilla TimeBridge's.

Questions for authors
  1. Please state the provenance of every vanilla and baseline row in Tables 1, 2, 10 and 12 (re-run versus taken from a published table), reconcile Table 2 with Table 12 for PatchTST and TimeMixer, and explain why vanilla PatchTST on ETTh1 (0.516) and vanilla Timer-XL on Solar (0.218) are worse than published values at the same lookback while other rows match published tables exactly. Please also give the patch length, stride and padding behind the 4 PatchTST patches in Table 9.

  2. For Tables 2 and 12, were batch size, number of patch-downsampling layers, learning rate and loss identical for TimeBridge and TimeBridge+BTA, and was vanilla TimeBridge re-tuned with the same budget? Please report mean and standard deviation over three seeds for both, prioritising the datasets with margins of 0.006 or less.

  3. Are the "Static" mixture weights in Tables 3 and 14 fixed by hand or learned? The most informative control is a learned, input-independent mixture over the same kernel bank; one standard static bias (ALiBi or T5) on TimeBridge and PatchTST would further test whether the gain comes from adaptation rather than from adding a positional prior. Please also position BTA relative to DAPE, PENGUIN, Pathformer, StretchTime and Kairos.

  4. Appendix J.3 applies softmax over the retained Top-kk logits, so for k=1k=1 the selected weight is identically one and the forecasting loss gives no gradient to the router; yet k=1k=1 is the preferred ETTh1 setting (Sec. 5.5). How is the router trained in that case?

  5. Is a strict advantage for regime-agnostic adaptive PE claimed, and can you exhibit a concrete two-regime process for which the gap in Eq. 5 is strict? Please reconcile the BTA MSE values 0.1788 and 0.1869, report Table 6 with the KL split into log-determinant and Mahalanobis terms and with ϵq\epsilon_q, ϵ0\epsilon_0 varied over {10−2,10−1,1}\{10^{-2},10^{-1},1\}, and show results when the kernel bank excludes the generating kernels.

  6. How does the Fourier analysis of Appendix J.2 yield 3-day and 7-day periods for ETTm1, Solar and Weather (Figs. 2, 8, 9), whose dominant cycle is diurnal, and how is a 7-day kernel interpreted on Solar's 5-day input window? How is the lag-wise TE profile computed for Informer's ProbSparse attention, which does not form the full QK⊤QK^\top, and for TimeBridge's Cointegrated Attention, which attends across variates?

Reviewer_mbePOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper interprets position-aware attention as combining a content prior with temporal evidence. It derives an ELBO for a simplified, frozen-attention Gaussian forecasting model and compares a shared static mechanism with regime-specific oracle solutions. BTA aggregates content scores by lag and routes a bank of local and periodic kernels using this profile and its Fourier features, producing input- and head-dependent attention biases. Experiments cover four Transformer backbones and eight forecasting benchmarks, with component ablations, synthetic examples, and multi-seeds results for three backbones.

Critical strengths
  1. The proposed module has practical value across architectures. BTA improves several Transformers with different tokenization and positional mechanisms, and the three-seed results strengthen this evidence. Its small measured overhead on TimeBridge supports its usefulness as a reusable forecasting component.

  2. The structured design makes temporal modeling easier to investigate. Local and periodic kernel mixtures expose an understandable temporal preference, and the paper combines this design with theoretical analysis, controlled synthetic examples, and component ablations. The appendix also makes important approximations explicit.

Critical weaknesses
  1. The Bayesian analysis does not yet explain the learned algorithm sufficiently. Proposition 1's factorization applies to general additive biases. The frozen-operator ELBO and regime-specific oracle comparison motivate adaptation, but do not derive this kernel bank/router or connect it to MSE training. I appreciate that the paper identifies the oracle and the separate synthetic perturbation model. Even with these qualifications, the missing link is consequential: why should observable histories identify the useful regimes, and why should this router realize their advantage? This limits the significance of the theory as a central contribution.

  2. The adaptation assumptions and distinctive modeling benefit need stronger justification. Existing content attention and RoPE already adapt to inputs, and PAREformer and Non-stationary Transformers already pursue adaptive positional/attention mechanisms. BTA's specific routing construction is distinct, but the paper has not sufficiently explained which changing dependencies it particularly benefits beyond these mechanisms. Forecasting improvements and weight visualizations support usefulness more directly than this mechanism-level interpretation.

Questions for authors
  1. How are the static ablation weights chosen and trained? With comparable validation budgets, how much benefit remains against learned, input-independent weights over the same bank and a closely related adaptive module on the same backbone?

  2. Tables 2 and 12 both specify input length 720 but report different reference results, e.g., PatchTST/ETTh1 MSE 0.516 versus 0.413. BTA's synthetic MSE is 0.1788 in Table 5 and 0.1869 in Table 6. Appendix J specifies 11 kernels, whereas Figure 11 labels 13. Could authors map these results to their exact configurations and runs?

  3. Which implementation is meant by “RPE” attributed to Liutkus et al. (2021)? How is each affine calibration fitted, what training information does it use, and how sensitive is the ELBO ranking to calibration and perturbation choices? Clarifying these links and providing consistent, comparable results could strengthen the case for acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.