- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the internal attention mechanisms of time-series foundation models to understand their forecasting behavior on controlled synthetic signals. Focusing primarily on Chronos, the authors analyze how attention patterns evolve across encoder layers when processing Jacobi elliptic functions, cosine signals of varying frequencies, autoregressive iterative sequences, and the logistic map. They identify four recurring structural attention modes: locality, periodicity, special-point anchoring, and value equivalence.
To quantify these patterns, the authors employ two locality metrics, Expectation Over Distance (EOD) and Main Diagonal Strength (MDS), and introduce a Periodicity Strength (PS) metric based on the autocorrelation of diagonal-band attention weights. They examine the relationship between these metrics and forecasting performance using Spearman rank correlations and encoder-layer pruning experiments. Their findings suggest that early encoder blocks predominantly capture value-based relationships, whereas later blocks increasingly exhibit local and periodic attention patterns. In particular, block 5 of Chronos-small often shows strong locality or periodicity and contributes substantially to forecasting accuracy. For several signal families, stronger structural attention patterns are associated with lower forecasting error, although this relationship does not consistently hold for iterative sequences.
The authors further extend their analysis to Chronos-base, Moirai-1, and Moirai-2, observing that locality persists across architectures while periodic attention patterns are less pronounced in patch-based models. Overall, the paper provides a quantitative characterization of structural modes in time-series attention and investigates their relationship with forecasting performance.
Critical strengths
-
The paper addresses an important research question regarding how time-series foundation models internally process temporal information and why their forecasting performance varies across different signal characteristics. By using controlled synthetic signals, including Jacobi elliptic functions, autoregressive iterative sequences, and the logistic map, the authors systematically investigate attention patterns under varying periodicity, locality, and chaotic behavior. The identification of distinct structural modes, such as equal-value attention, special-point anchoring, locality, and periodicity, provides useful insights into how temporal attention patterns evolve across encoder layers.
-
Rather than relying solely on qualitative attention visualizations, the authors employ Expectation Over Distance (EOD) and Main Diagonal Strength (MDS) to quantify locality and introduce a Periodicity Strength (PS) metric to characterize recurring temporal patterns. They further investigate the relationship between these metrics and forecasting accuracy through Spearman rank correlations and encoder-layer pruning experiments. In particular, identifying block 5 of Chronos-small as exhibiting strong locality or periodicity alongside substantial forecasting improvements provides an interesting empirical observation. Additionally, the positional-bias ablations help distinguish architectural effects from input-dependent attention patterns, while experiments on Chronos-base and Moirai extend the analysis beyond a single model configuration.
Critical weaknesses
-
The paper assumes that recurring peaks in the autocorrelation of the diagonal-band attention profile reliably indicate meaningful periodicity in the model's attention. However, the proposed Periodicity Strength (PS) metric is not systematically validated against alternative structured attention patterns. In particular, the uncentered autocorrelation may be influenced by background attention concentration, while the number of overlapping terms decreases with increasing lag. Furthermore, the fixed peak-prominence threshold of 0.15 is not supported by a sensitivity analysis. Although the authors demonstrate that PS responds to periodic synthetic signals, they do not establish its robustness against nonperiodic attention patterns, noise, or varying sequence lengths. Evaluating the metric on controlled attention matrices with known periodic and nonperiodic structures would strengthen its validity.
-
The central interpretation assumes that locality and periodicity in attention matrices reflect temporal information that the model functionally uses for forecasting. However, attention weights alone do not determine the information transmitted through a Transformer layer, since the attention output also depends on the value representations and subsequent transformations. Consequently, identifying strong periodic or local attention does not necessarily imply that the corresponding temporal information contributes to the forecast. Although the authors acknowledge that their reported correlations are not causal, they do not directly examine whether the identified structural modes correspond to forecasting-relevant information in the encoder representations. Probing the recoverability of frequency, period, or phase information from intermediate representations would provide stronger support for this interpretation.
-
The authors average attention weights across eight heads before calculating locality and periodicity metrics. This assumes that the resulting matrix faithfully captures the structural patterns represented by individual heads. However, different heads may specialize in distinct temporal relationships, such as locality, periodicity, or extrema detection. Averaging these heterogeneous patterns may obscure head-level specialization or alter the measured periodicity strength. Although the appendix visualizes individual heads, the paper does not systematically compare head-wise metrics with those computed from averaged attention. A head-level quantitative analysis would help establish whether the reported layerwise trends reflect consistent structural organization or the contribution of a small subset of specialized heads.
-
The authors demonstrate that periodic attention patterns can align with the true period of the input signal and examine their association with forecasting MAE. However, they do not directly investigate whether the detected periodicity is preserved in the model's predictions. A model may identify the correct temporal period within its attention matrix while generating forecasts with incorrect frequency, phase, or amplitude. Therefore, the relationship between attention-derived periodicity and the actual temporal fidelity of the forecast remains unclear. Comparing the input period, attention-derived period, and forecast period, alongside frequency and phase errors, would provide a more direct assessment of whether the identified structural modes are reflected in forecasting behavior.
Questions for authors
- Periodicity metric validation: How robust are the PS results to the fixed prominence threshold (0.15), context length, and structured nonperiodic attention patterns? Have the authors evaluated potential false-positive periodicity detections?
- Forecasting-relevant representations: Beyond the reported correlations and layer-pruning experiments, is there evidence that the identified attention structures encode forecasting-relevant temporal information, such as recoverable frequency or phase features?
- Attention-head averaging: Appendix Figure 5 visualizes individual heads, but do the reported EOD, MDS, and PS trends persist when these metrics are computed separately for each head rather than from their average?
- Forecast periodicity: When the attention-derived period matches the input period, does the generated forecast also preserve that period? Are there cases where strong attention periodicity coincides with substantial frequency or phase errors?
- Comparison with SAE-based interpretability: Beyond the distinction discussed in Section 1, what additional mechanistic insights does the proposed attention-based framework establish relative to the SAE-based Chronos analysis of Mishra (2026)?
- Tokenization effects: Given the architectural and pretraining differences between Chronos and Moirai, what evidence isolates patch tokenization as the explanation for weaker periodic attention patterns?
4 comments
W2/Q2
We agree that attention weights alone do not show what the encoder represents. Here, we examined the encoder representations directly with a representational similarity analysis. For each series and block, we computed the token-by-token similarity of the hidden states (cosine similarity of the block outputs). We regressed it on candidate similarity structures over the same tokens:
- equal value
- equal slope
- temporal locality (where tokens closer in time are similar)
- a position-only template (the similarity every input of that length shares, average on 20 unrelated inputs)
- phase
We include the initial input embeddings as a baseline and a randomly initialised model as a control. We report grouped contributions and overlap statistics rather than individual coefficients, since value, slope and phase overlap strongly on smooth cycles. The variance inflation factors in the table reach 5.9 for phase in Elliptic (period), so we report their grouped contribution rather than individual coefficients. Where phase overlaps little with value and slope (Iterative (lag), VIF ≈ 1.1), the phase structure itself grows with depth (standardised coefficient 0.05 to 0.23 by block 6. The individual coefficients and the grouped contributions are shown in figure R3_W2_1 and figure R3_W2_3 respectively.
Figure R3_W2_3 shows that the structure seen in attention is mirrored in the hidden states, increasingly with depth. At the input embeddings and in the early blocks, similarity is organised by value (equal-value, equal slope, and phase group) explains about 0.76 of the ranked variance). At block 4 and on, the contribution of this group falls, while position in time (temporal locality and the position template, as a group) grows. In the same architecture with random weights, these candidates explain essentially none of the representational similarity at any block (R^2 = 0.004), so the organization by value, slope, locality and phase is not imposed by the tokenizer or the architecture.
Revision: The analysis shows that this information is organised in the representations where the attention modes appear. A decoding probe of phase and frequency would be a natural complement, which we will mention as future work. We will add this analysis and its figures to the appendix.
W3/Q3
We measure EOD, MDS, and PS on at the head level for Chronos-small. We recomputed all three metrics for each of the 8 heads in every block and every series, and repeated the Table 1 analysis per head.
In figure R3_W3_1, we compute Spearman correlation and find that heads agree with the average most at block 5 and least at block 6. Head 8 tracks the average in every block while head 7 shows no periodicity in most families at any block.
Figure R3_W3_2 shows the head-level analysis with forecast accuracy (Table 1)
-
Locality (EOD, MDS): Since EOD and MDS are linear, EOD and MDS of the averaged matrix are exactly the mean of the 8 heads' values, so averaging the heads does not create or hide a locality trend that the heads do not carry on average. Per head, the block-5 relation between MDS and forecast error keeps the sign of the averaged value in every head for Elliptic (k), Elliptic (period) and Iterative (decay), and in 5–7 of 8 heads for Iterative (lag) and Logistic. In the strongest case, Elliptic (k), MDS correlates with error at ρ = −0.9 in nearly every head.
-
Periodicity (PS): PS is not linear, and here the heads specialize. In each family, 2–5 heads carry no periodicity. Where a head does carry it, it usually agrees in sign with the averaged value (4–6 of 8 heads).
Revisions: We will add the per-head figure and this discussion to the appendix. We will add a discussion on specialized-heads for locality and periodicity.
W4/Q4
We test whether the block-5 attention's detected lag, the input lag, and the forecast lag match. We measure lag using the same autocorrelation method as finding the lag in the attention matrix. We measure the phase of a series by finding the minimum shift where the shifted series matches itself. We measure amplitude by subtracting the maximum value and the minimum value.
Figure R3_W4_1 shows our results. For every series whose block-5 attention identifies a period, the 64-step forecast reproduces the target's frequency, phase, and amplitude. We find no case of a correctly identified attention period followed by a wrong-frequency forecast. We note that the results were restricted to a 64-step forecast since Chronos degrades beyond this horizon length. For this reason, we cannot find the forecast period for a series whose number of steps per cycle is greater than 32.
We note that the attention lag can read a few steps short of long cycles. The reason is because at the presence of the local band on the main diagonal, PS’s autocorrelation pairs the attention around the main diagonal with the periodic peak, which pulls its maximum to a few steps earlier. We show this analytically in the appendix (section F).
Revision: As noted in W1/Q1, the detected lag at the attention matrix does not always match the input lag, and here we explained why. We modified our statement that "The detected period aligns with the true period of the input signal..." to "The detected period closely tracks the true period of the input signal." and also add "For long cycles it reads slightly short because the local band on the main diagonal pulls the autocorrelation peak towards shorter lag (Appendix F)" in the periodicity strength section.
W5/Q5
We thank the reviewer and will correct our description of Mishra (2026) in Section 1: that work does relate SAE features to forecast accuracy through feature ablations. The two approaches analyze different objects. Mishra studies what is represented in each token’s internal activations. We study how tokens exchange information through the encoder’s context-to-context attention. This offers several complementary insights:
- Structure between pairs of time steps. A local band of a given width, or an off-diagonal band at a fixed lag, describes which time steps attend to which. This can be compared directly with the input’s timescales, such as its period, which a feature defined on a single token does not directly express.
- Position vs content. In T5, the relative position bias is computed in block 1 and shared by all later blocks. Our layer pruning shows that the collapse when block 1 is removed comes from losing this bias. Attention lets us separate the structure the model imposes through position from the structure it builds from the input.
- Controlled stimuli. Each family varies one parameter, so we can show where each mode appears across blocks and how it changes with that parameter. Value-based modes dominate early blocks; locality and off-diagonal bands concentrate in later blocks.
- Comparison across architectures. The same metrics apply unchanged to Moirai-1 (patch embedding) and Moirai-2 (causal attention), without training a new dictionary for each model and layer.
We see the approaches as complementary: SAEs identify what a block represents, while attention shows how information is routed across time.
W6/Q6
Chronos and Moirai differ not only in tokenization but also in architecture, masking and training data, and our comparison between the two models cannot separate these factors.
To test the role of tokenization directly, we ran the experiment the reviewer suggests within a single model. Moirai-1.1-R-small accepts patch sizes P = 8, 16, 32, 64, 128, so the architecture and masking stay fixed and only the patching changes. We used the same signal throughout: a cosine^2 series with 32, 64, 128 or 256 steps per cycle in a 1024 step context. Changing P then changes only how many tokens cover one cycle (tokens per cycle = cycles/P). We measured block-wise attention between context tokens with the same metrics as in the paper.
Our results shown in figure R2_W3_1 show that periodicity changes with patch size for a fixed given signal. We discover a regime of 2-8 tokens per cycle where periodicity strength (PS) exceeds the noise floor. Around 8 tokens per cycle and more, PS begins to drop towards the noise floor. Furthermore, once a patch covers a full cycle or more, PS is at or below the floor in 5 of 6 runs (the exception is at exactly one token per cycle), and below one token per cycle it is zero or at the floor in every run. Within one model, therefore, the choice of patch size affects whether periodic attention can be measured.
Locality does not change with patch size. Attention is more local than uniform at every patch size, with no consistent trend in P. Patch size therefore does not explain the partial locality link we observe in Moirai.
Revision: We will add this experiment and revise our claims. While we cannot rule out architectural and masking differences as a cause of Moirai's weaker periodicity, the patch size alone affects periodicity for the same signal, so tokenization is one contributing factor. The partial locality link in Moirai is not explained by patch size and may reflect other differences between the models. The framework transfers most directly to models with pointwise tokenization, and its application to patch-based models requires accounting for patch size relative to the signal's period.
We thank the reviewer for their comments and address each below:
W1/Q1
We thank the reviewer for pointing out our equation 3 reflects "uncentered autocorrelation." We remark that that autocorrelation is actually centered in our code and computations. We will make the correction to equation 3 in the text.
Here we describe additional experiments that test the validity of the periodicity strength metric. As per the reviewer's suggestion, we test periodicity strengths on several nonperiodic attention patterns (segmented blocks, local band only, anchor points, attention sink, noise) while considering the effects of the prominence thresholds, sequence length, and noise. Figure R3_W1_1 shows our gallery and the associated band profile, as well as the detected lag (compared to the true lag of 13 if one exists). We find that while our metric correctly identifies most nonperiodic patterns to have PS=0, false positive cases and limitations on the detected lag can be identified.
False Positive cases:
Limitations: There are some cases where the lag is not correctly identified. One example is when the values on the local band dominate the periodic off-diagonal bands. In figure R3_W1_3, this is reflected in ratio (band height/local gain) <<1. The figure shows the probability of finding the correct lag over the ratio of (band height/local gain). A smaller prominence threshold can sometimes recover the information, whereas a larger prominence threshold like 0.3 can fail at regimes where (band height/local gain) >> 1. Another example is when the lag appears after n/2 (n is the size of the matrix) as we compute the band profile up to floor(n/2) (Figure R3_W1_4).
Lastly, we show the periodicity calculations for the synthetic series for different choices of prominence thresholds in figure R3_W1_5. We note that the periodicity ranking within a family is roughly unchanged. In particular, we show that the results from table 1 are generally unchanged with the choice of prominence threshold. It is only at prominence threshold 0.3 that the correlation weakens or the sign flips (Table R3_W1_6)
Revision