- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Softmax attention mixes tokens with a row-stochastic matrix, so each output is a convex combination of the values. The paper argues that this rules out the signed operators common in time-series analysis and proposes TOA:
where are learned, is softmax, ReLU or a softplus-gated ReLU, and are trained with random dropout (SOR). TOA is inserted into PatchTST, iTransformer, Autoformer and DUET and evaluated on over 40 datasets.
Critical strengths
- The question is well posed: whether simplex-constrained attention is a poor fit for signed temporal operators.
- TOA is a drop-in replacement, tested on four backbones and four tasks (long- and short-term forecasting, classification, anomaly detection).
- Code is anonymized and available, and the paper reports a case where TOA loses (ETTh2 in the multi-seed rerun).
Critical weaknesses
-
Proposition 1 holds by construction and covers no model in the paper. A single softmax head with identity values outputs with every row of in the simplex, so it cannot equal a signed operator; nothing beyond this is proved. Sec. 3.1 grants that heads combined through and an MLP can produce signed maps, and every backbone in the experiments has both.
-
The added operators are known, and the paper's own ablations show they do not explain the gains. is a learned, input-independent token-mixing matrix, as in the Random Synthesizer (Tay et al., 2021; not cited) and the token-mixing layer of TSMixer. Differential Transformer, which already gives signed attention weights, is cited but not compared. In UEA classification, TOA-Softmax, which only adds and , lowers PatchTST accuracy from 65.62% to 64.80% and loses on 17 of 28 datasets (Table 3); accuracy rises only after softmax is replaced by ReLU. Rank-16 factorizations of and match the dense ones (Table 13), which contradicts Sec. 2, where HyperMLP is rejected for being low-rank.
-
The forecasting improvements are below the noise level the paper itself measures. Forecasting and anomaly results come from one seed (App. H), and each backbone is compared with the best of three TOA variants. On PatchTST, the best variant improves average MSE by at most 0.003 on Weather, ECL, ETT, and Traffic (Table 2), while the five-seed rerun shows standard deviations up to 0.015 (App. I), and in that rerun softmax wins on ETTh2 at horizons 96 and 720. DLinear, TSMixer, and TimeMixer, whose advantage over Transformers motivates the paper, appear in no table. TOA-ReLU and TOA-Gated also add 79% (PatchTST) and 96% (iTransformer) to training step time (Table 12).
Questions for authors
See weaknesses.