acceptodds
Withdrawn from ICLR 2027

Temporal Operator Attention: Recovering Common Operators in Time-Series Analysis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

16 views

Abstract

Real-world time-series applications repeatedly exhibit five common structural regimes: seasonal extrapolation, trend/anomaly separation, cross-channel demixing, regime-shift alignment, and transient recovery. Each requires a signed or zero-sum linear map over time, not a convex combination of the sequence's own values, and some further require this map to adapt to the input. Linear and MLP-based mixers, despite far fewer parameters, often match or outperform attention-based forecasters on exactly this structure, because they mix directly along the temporal dimension via a signed, position-indexed operator, rather than one derived from content-based similarity and confined to a probability simplex. We prove that softmax attention's row-stochastic kernel where nonnegative weights summing to one cannot realize any of the five recurring regimes from a single head, tracing the gap to this primitive-level mismatch rather than model scale. We propose Temporal Operator Attention (TOA), which restores this signed, position-indexed operator to the attention pathway itself: learnable dense residual operators wrap the score matrix, supplying signed degrees of freedom, while a gated variant further makes the operator input-adaptive. Stochastic Operator Regularization (SOR) controls overfitting from this added capacity by regularizing only the learned offsets, preserving identity-based signal transport. Across Autoformer, PatchTST, iTransformer, and DUET, TOA consistently improves long- and short-term forecasting, anomaly detection, and classification, with the largest gains where signed temporal mixing matters most.

Primary area:
learning on time series and dynamical systems
Keywords:
Time-series Analysis, Attention Mechanisms, Transformers

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CwSqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Softmax attention mixes tokens with a row-stochastic matrix, so each output is a convex combination of the values. The paper argues that this rules out the signed operators common in time-series analysis and proposes TOA:

O=σ(AS1) S2V,A=HWQ⊤WKH⊤,S1=I+M1,S2=I+M2,O = \sigma(A S_1)\, S_2 V, \qquad A = H W_Q^\top W_K H^\top, \qquad S_1 = I + M_1, \quad S_2 = I + M_2,

where M1,M2∈RN×NM_1, M_2 \in \mathbb{R}^{N \times N} are learned, σ\sigma is softmax, ReLU or a softplus-gated ReLU, and M1,M2M_1, M_2 are trained with random dropout (SOR). TOA is inserted into PatchTST, iTransformer, Autoformer and DUET and evaluated on over 40 datasets.

Critical strengths
  1. The question is well posed: whether simplex-constrained attention is a poor fit for signed temporal operators.
  2. TOA is a drop-in replacement, tested on four backbones and four tasks (long- and short-term forecasting, classification, anomaly detection).
  3. Code is anonymized and available, and the paper reports a case where TOA loses (ETTh2 in the multi-seed rerun).
Critical weaknesses
  1. Proposition 1 holds by construction and covers no model in the paper. A single softmax head with identity values outputs PvPv with every row of PP in the simplex, so it cannot equal a signed operator; nothing beyond this is proved. Sec. 3.1 grants that heads combined through WOW_O and an MLP can produce signed maps, and every backbone in the experiments has both.

  2. The added operators are known, and the paper's own ablations show they do not explain the gains. S2S_2 is a learned, input-independent N×NN \times N token-mixing matrix, as in the Random Synthesizer (Tay et al., 2021; not cited) and the token-mixing layer of TSMixer. Differential Transformer, which already gives signed attention weights, is cited but not compared. In UEA classification, TOA-Softmax, which only adds S1S_1 and S2S_2, lowers PatchTST accuracy from 65.62% to 64.80% and loses on 17 of 28 datasets (Table 3); accuracy rises only after softmax is replaced by ReLU. Rank-16 factorizations of M1M_1 and M2M_2 match the dense ones (Table 13), which contradicts Sec. 2, where HyperMLP is rejected for being low-rank.

  3. The forecasting improvements are below the noise level the paper itself measures. Forecasting and anomaly results come from one seed (App. H), and each backbone is compared with the best of three TOA variants. On PatchTST, the best variant improves average MSE by at most 0.003 on Weather, ECL, ETT, and Traffic (Table 2), while the five-seed rerun shows standard deviations up to 0.015 (App. I), and in that rerun softmax wins on ETTh2 at horizons 96 and 720. DLinear, TSMixer, and TimeMixer, whose advantage over Transformers motivates the paper, appear in no table. TOA-ReLU and TOA-Gated also add 79% (PatchTST) and 96% (iTransformer) to training step time (Table 12).

Questions for authors

See weaknesses.

Reviewer_ExhyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the limitations of simplex constrained softmax attention in representing signed or zero sum operators associated with 5 time series regimes. It proves an exact realization limitation for single layer single head attention with scalar values. TOA introduces learnable residual mixing matrices around the attention activation, with softmax, ReLU, and gated variants, together with SOR. Experiments across 4 time series backbones report improvements on synthetic tasks, forecasting, classification and anomaly detection.

Critical strengths

The paper provides an operator based motivation for revisiting attention in time series modeling. It connects 5 representative time series regimes to signed or zero sum transformations and uses this perspective to examine the limitations of softmax attention. This framing offers a connection between classical signal processing operations and the proposed architectural design.

Critical weaknesses
  1. The empirical gains are modest, like the improvements in forecasting are small. And the best performing variant varies across backbones and tasks, with no consistently superior TOA variant. With three variants and no clear selection principle, it is difficult to determine whether TOA provides reliable improvements without selecting the best performing configuration for each setting.
  2. The theoretical analysis does not establish the claimed realization capability of TOA. The positive results show that the post mixer S2 can represent target operators, but do not establish that the complete TOA layer can realize them. Under the same all input criterion, the ReLU and gated variants also cannot exactly realize general nonzero linear operators.
Questions for authors
  1. How is the choice among the 3 TOA variants determined without using test results? Does this selection preserve the reported forecasting improvements across backbones?
  2. Figs. 3–4 visualize normalized attention weights for the softmax baseline but raw query–key scores for TOA. Do the actual mixing operators applied to the values recover the target signed operators in these synthetic experiments?
  3. What experimental evidence distinguishes the contribution of signed mixing from the effects of additional model capacity, activation changes and SOR?
  4. The realization proofs establish the expressivity of S2 alone. How do these results extend to the complete TOA layer under the all input criterion of Proposition 1, particularly for the ReLU and gated variants?
Reviewer_R7yXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper diagnoses why linear/MLP mixers often match or beat Transformer forecasters: five recurring time-series regimes (seasonal extrapolation, trend/anomaly separation, cross-channel demixing, regime-shift alignment, transient recovery) require signed or zero-sum linear maps over time, whereas a softmax attention head's row-stochastic kernel (nonnegative, unit row sums) cannot express any of them (Proposition 1, with per-regime non-realizability lemmas in Appendix D). The proposed Temporal Operator Attention (TOA) wraps the score matrix with learnable residual dense operators S1 = I + M1 (pre-activation) and S2 = I + M2 (pre-value-aggregation), initialized near identity; three variants (TOA-Softmax, TOA-ReLU, TOA-Gated) isolate row normalization and input-adaptivity. Stochastic Operator Regularization (SOR) applies inverted dropout with a uniformly sampled rate to the offsets Mi only. Synthetic bottleneck experiments (2-layer, single-head PatchTST) show softmax failing on the signed-polarity variants (MSE 0.61/0.11) where the TOA variants succeed (≈0.02/0.001). Real-data experiments integrate TOA into Autoformer, PatchTST, iTransformer, and DUET across long-term forecasting (9 datasets), PEMS short-term forecasting, 28 UEA classification datasets, and 5 anomaly-detection benchmarks. Average rank improves consistently across these settings, and a SOR ablation shows gains concentrated on complex datasets.

Critical strengths
  1. A crisp, correct primitive-level diagnosis with convincing synthetic verification. The paper ties five classical regimes to a single structural property (non-simplex target operators), and the controlled single-head bottleneck shows softmax failing exactly where sign flips are required while TOA succeeds. This gives a concrete, checkable explanation of why linear mixers are competitive on exactly this structure.
  2. Breadth of empirical validation. Four architecturally distinct backbones × four task families × 40+ datasets with consistent average-rank improvement is a wide evidence base for a drop-in mixing-layer modification, and the per-backbone variant reporting is honest rather than cherry-picked.
Critical weaknesses
  • The theoretical contribution is thin relative to its framing. Proposition 1 is essentially the observation that softmax rows lie in the simplex, plus a scaling argument. The paper's own caveat (deep stacks can express signed operators through WO and the subsequent MLPs) removes the expressivity stakes. What remains is an informal statistical-efficiency claim that is never measured (no sample-complexity or depth-efficiency comparison). As presented, the theory rationalizes rather than predicts.
  • The natural control is missing, so the mechanism attribution is not established. The obvious baseline, adding a parallel signed dense temporal mixer (DLinear/TSMixer-style branch) to an unchanged attention layer, or replacing attention with such a mixer entirely, is absent. Without it, the paper cannot distinguish "attention needed operator repair" from "any pathway with signed dense temporal weights helps". The TOA-Softmax ablation's weak showing further suggests that the gains track normalization removal rather than the operator-residual design, and the paper's own classification section says as much. Effect sizes on real data (often around 1-2% relative MSE on the long-term forecasting benchmarks) make this attribution question decisive.
Questions for authors

1.Can you add the parallel-branch baseline (frozen attention + additive learned dense temporal mixer, parameter-matched) on a representative subset? If it matches TOA, the contribution is the diagnosis, not the mechanism. 2.Given that no single variant dominates and the best variant varies by backbone/dataset, how should a practitioner choose? Is there any held-out selection procedure, and were variant choices on benchmarks made using test results? 3.What are the parameter/FLOP/latency overheads of the dense N×N operators per head at typical N (especially Traffic/ECL-scale N), beyond the low-rank discussion in Appendix K? 4.Reporting per-dataset multi-seed error bars for the main tables (even for a subset) would materially strengthen the empirical claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.