acceptodds
Under review as a conference paper at ICLR 2027

Structural Modes of Context-to-Context Attention in Chronos: A Mechanistic Autopsy of a Time Series Foundation Model

Abstract

Foundation models for time series forecasting are trained on vast, high-dimensional real-world data and synthetic corpora, yet they can fail on zero-shot forecasting of sequences generated by simple, deterministic functions. In this work, we investigate the internal mechanism behind this shortcoming of the original Chronos, a time series foundation model whose pointwise tokenization makes its attention directly interpretable. Following techniques in explainable AI, we probe Chronos with the controlled stimuli of Jacobi elliptic functions, auto-regressive iterative series, and the logistic map as inputs, where we vary the parameters that modulate periodicity, locality, and chaoticity. We extract the average attention matrices of each encoder block. These matrices resolve into a small set of canonical structural modes. Early blocks are dominated by value-based modes, in which tokens with equal values attend to each other and all tokens attend to extrema, while later blocks concentrate attention on local neighborhoods and off-diagonal bands. We quantify these modes with locality and periodicity metrics defined on the attention matrices, and we relate the behavior of the encoder to forecast accuracy through rank correlations and layer pruning. For several stimulus families, the forecast accuracy improves when the encoder resolves the input into at least one interpretable structural mode of strong locality or clear periodicity, and degrades when it resolves neither, as for near-aperiodic or chaotic signals. We also address how our analysis can be extended to other time series foundation models to diagnose how models encode using other architectural strategies such as patch embedding and causal masking. This suggests a promising direction for interpreting how deep learning architectures structure data processing.

Primary area:
interpretability and explainable AI
Keywords:
foundation models, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XbPVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the internal attention mechanisms of time-series foundation models to understand their forecasting behavior on controlled synthetic signals. Focusing primarily on Chronos, the authors analyze how attention patterns evolve across encoder layers when processing Jacobi elliptic functions, cosine signals of varying frequencies, autoregressive iterative sequences, and the logistic map. They identify four recurring structural attention modes: locality, periodicity, special-point anchoring, and value equivalence.

To quantify these patterns, the authors employ two locality metrics, Expectation Over Distance (EOD) and Main Diagonal Strength (MDS), and introduce a Periodicity Strength (PS) metric based on the autocorrelation of diagonal-band attention weights. They examine the relationship between these metrics and forecasting performance using Spearman rank correlations and encoder-layer pruning experiments. Their findings suggest that early encoder blocks predominantly capture value-based relationships, whereas later blocks increasingly exhibit local and periodic attention patterns. In particular, block 5 of Chronos-small often shows strong locality or periodicity and contributes substantially to forecasting accuracy. For several signal families, stronger structural attention patterns are associated with lower forecasting error, although this relationship does not consistently hold for iterative sequences.

The authors further extend their analysis to Chronos-base, Moirai-1, and Moirai-2, observing that locality persists across architectures while periodic attention patterns are less pronounced in patch-based models. Overall, the paper provides a quantitative characterization of structural modes in time-series attention and investigates their relationship with forecasting performance.

Critical strengths
  1. The paper addresses an important research question regarding how time-series foundation models internally process temporal information and why their forecasting performance varies across different signal characteristics. By using controlled synthetic signals, including Jacobi elliptic functions, autoregressive iterative sequences, and the logistic map, the authors systematically investigate attention patterns under varying periodicity, locality, and chaotic behavior. The identification of distinct structural modes, such as equal-value attention, special-point anchoring, locality, and periodicity, provides useful insights into how temporal attention patterns evolve across encoder layers.

  2. Rather than relying solely on qualitative attention visualizations, the authors employ Expectation Over Distance (EOD) and Main Diagonal Strength (MDS) to quantify locality and introduce a Periodicity Strength (PS) metric to characterize recurring temporal patterns. They further investigate the relationship between these metrics and forecasting accuracy through Spearman rank correlations and encoder-layer pruning experiments. In particular, identifying block 5 of Chronos-small as exhibiting strong locality or periodicity alongside substantial forecasting improvements provides an interesting empirical observation. Additionally, the positional-bias ablations help distinguish architectural effects from input-dependent attention patterns, while experiments on Chronos-base and Moirai extend the analysis beyond a single model configuration.

Critical weaknesses
  1. The paper assumes that recurring peaks in the autocorrelation of the diagonal-band attention profile reliably indicate meaningful periodicity in the model's attention. However, the proposed Periodicity Strength (PS) metric is not systematically validated against alternative structured attention patterns. In particular, the uncentered autocorrelation may be influenced by background attention concentration, while the number of overlapping terms decreases with increasing lag. Furthermore, the fixed peak-prominence threshold of 0.15 is not supported by a sensitivity analysis. Although the authors demonstrate that PS responds to periodic synthetic signals, they do not establish its robustness against nonperiodic attention patterns, noise, or varying sequence lengths. Evaluating the metric on controlled attention matrices with known periodic and nonperiodic structures would strengthen its validity.

  2. The central interpretation assumes that locality and periodicity in attention matrices reflect temporal information that the model functionally uses for forecasting. However, attention weights alone do not determine the information transmitted through a Transformer layer, since the attention output also depends on the value representations and subsequent transformations. Consequently, identifying strong periodic or local attention does not necessarily imply that the corresponding temporal information contributes to the forecast. Although the authors acknowledge that their reported correlations are not causal, they do not directly examine whether the identified structural modes correspond to forecasting-relevant information in the encoder representations. Probing the recoverability of frequency, period, or phase information from intermediate representations would provide stronger support for this interpretation.

  3. The authors average attention weights across eight heads before calculating locality and periodicity metrics. This assumes that the resulting matrix faithfully captures the structural patterns represented by individual heads. However, different heads may specialize in distinct temporal relationships, such as locality, periodicity, or extrema detection. Averaging these heterogeneous patterns may obscure head-level specialization or alter the measured periodicity strength. Although the appendix visualizes individual heads, the paper does not systematically compare head-wise metrics with those computed from averaged attention. A head-level quantitative analysis would help establish whether the reported layerwise trends reflect consistent structural organization or the contribution of a small subset of specialized heads.

  4. The authors demonstrate that periodic attention patterns can align with the true period of the input signal and examine their association with forecasting MAE. However, they do not directly investigate whether the detected periodicity is preserved in the model's predictions. A model may identify the correct temporal period within its attention matrix while generating forecasts with incorrect frequency, phase, or amplitude. Therefore, the relationship between attention-derived periodicity and the actual temporal fidelity of the forecast remains unclear. Comparing the input period, attention-derived period, and forecast period, alongside frequency and phase errors, would provide a more direct assessment of whether the identified structural modes are reflected in forecasting behavior.

Questions for authors
  1. Periodicity metric validation: How robust are the PS results to the fixed prominence threshold (0.15), context length, and structured nonperiodic attention patterns? Have the authors evaluated potential false-positive periodicity detections?
  2. Forecasting-relevant representations: Beyond the reported correlations and layer-pruning experiments, is there evidence that the identified attention structures encode forecasting-relevant temporal information, such as recoverable frequency or phase features?
  3. Attention-head averaging: Appendix Figure 5 visualizes individual heads, but do the reported EOD, MDS, and PS trends persist when these metrics are computed separately for each head rather than from their average?
  4. Forecast periodicity: When the attention-derived period matches the input period, does the generated forecast also preserve that period? Are there cases where strong attention periodicity coincides with substantial frequency or phase errors?
  5. Comparison with SAE-based interpretability: Beyond the distinction discussed in Section 1, what additional mechanistic insights does the proposed attention-based framework establish relative to the SAE-based Chronos analysis of Mishra (2026)?
  6. Tokenization effects: Given the architectural and pretraining differences between Chronos and Moirai, what evidence isolates patch tokenization as the explanation for weaker periodic attention patterns?
4 comments

We thank the reviewer for their comments and address each below:

W1/Q1

We thank the reviewer for pointing out our equation 3 reflects "uncentered autocorrelation." We remark that that autocorrelation is actually centered in our code and computations. We will make the correction to equation 3 in the text.

Here we describe additional experiments that test the validity of the periodicity strength metric. As per the reviewer's suggestion, we test periodicity strengths on several nonperiodic attention patterns (segmented blocks, local band only, anchor points, attention sink, noise) while considering the effects of the prominence thresholds, sequence length, and noise. Figure R3_W1_1 shows our gallery and the associated band profile, as well as the detected lag (compared to the true lag of 13 if one exists). We find that while our metric correctly identifies most nonperiodic patterns to have PS=0, false positive cases and limitations on the detected lag can be identified.

False Positive cases:

  • We find that PS can be triggered (PS>0) if there is a local band plus a single off-diagonal band if the off-diagonal band is far enough from the local band for PS to register the off-diagonal as a peak. An example of this is at row 1, column 4 of figure R3_W1_1. This is because the metric only looks for two peaks to identify a lag.
  • Noise can also trigger PS. Figure R3_W1_2 shows the computed PS for different levels of Dirichlet noise, prominence thresholds, and sequence length (n).

Limitations: There are some cases where the lag is not correctly identified. One example is when the values on the local band dominate the periodic off-diagonal bands. In figure R3_W1_3, this is reflected in ratio (band height/local gain) <<1. The figure shows the probability of finding the correct lag over the ratio of (band height/local gain). A smaller prominence threshold can sometimes recover the information, whereas a larger prominence threshold like 0.3 can fail at regimes where (band height/local gain) >> 1. Another example is when the lag appears after n/2 (n is the size of the matrix) as we compute the band profile up to floor(n/2) (Figure R3_W1_4).

Lastly, we show the periodicity calculations for the synthetic series for different choices of prominence thresholds in figure R3_W1_5. We note that the periodicity ranking within a family is roughly unchanged. In particular, we show that the results from table 1 are generally unchanged with the choice of prominence threshold. It is only at prominence threshold 0.3 that the correlation weakens or the sign flips (Table R3_W1_6)

Revision

  • We have revised our work with the corrected equation 3 to reflect our computations and added the noise floor to the periodicity profiles in Figure 3.
  • On the detection of the correct lag, we modified our statement that "The detected period aligns with the true period of the input signal," which we discuss further in [W4/Q1].
  • We will add the threshold analysis and the synthetic validation to the appendix of the main text where we will state the limitations and the false positive cases of PS.

W2/Q2

We agree that attention weights alone do not show what the encoder represents. Here, we examined the encoder representations directly with a representational similarity analysis. For each series and block, we computed the token-by-token similarity of the hidden states (cosine similarity of the block outputs). We regressed it on candidate similarity structures over the same tokens:

  • equal value
  • equal slope
  • temporal locality (where tokens closer in time are similar)
  • a position-only template (the similarity every input of that length shares, average on 20 unrelated inputs)
  • phase

We include the initial input embeddings as a baseline and a randomly initialised model as a control. We report grouped contributions and overlap statistics rather than individual coefficients, since value, slope and phase overlap strongly on smooth cycles. The variance inflation factors in the table reach 5.9 for phase in Elliptic (period), so we report their grouped contribution rather than individual coefficients. Where phase overlaps little with value and slope (Iterative (lag), VIF ≈ 1.1), the phase structure itself grows with depth (standardised coefficient 0.05 to 0.23 by block 6. The individual coefficients and the grouped contributions are shown in figure R3_W2_1 and figure R3_W2_3 respectively.

Figure R3_W2_3 shows that the structure seen in attention is mirrored in the hidden states, increasingly with depth. At the input embeddings and in the early blocks, similarity is organised by value (equal-value, equal slope, and phase group) explains about 0.76 of the ranked variance). At block 4 and on, the contribution of this group falls, while position in time (temporal locality and the position template, as a group) grows. In the same architecture with random weights, these candidates explain essentially none of the representational similarity at any block (R^2 = 0.004), so the organization by value, slope, locality and phase is not imposed by the tokenizer or the architecture.

Revision: The analysis shows that this information is organised in the representations where the attention modes appear. A decoding probe of phase and frequency would be a natural complement, which we will mention as future work. We will add this analysis and its figures to the appendix.

W3/Q3

We measure EOD, MDS, and PS on at the head level for Chronos-small. We recomputed all three metrics for each of the 8 heads in every block and every series, and repeated the Table 1 analysis per head.

In figure R3_W3_1, we compute Spearman correlation and find that heads agree with the average most at block 5 and least at block 6. Head 8 tracks the average in every block while head 7 shows no periodicity in most families at any block.

Figure R3_W3_2 shows the head-level analysis with forecast accuracy (Table 1)

  • Locality (EOD, MDS): Since EOD and MDS are linear, EOD and MDS of the averaged matrix are exactly the mean of the 8 heads' values, so averaging the heads does not create or hide a locality trend that the heads do not carry on average. Per head, the block-5 relation between MDS and forecast error keeps the sign of the averaged value in every head for Elliptic (k), Elliptic (period) and Iterative (decay), and in 5–7 of 8 heads for Iterative (lag) and Logistic. In the strongest case, Elliptic (k), MDS correlates with error at ρ = −0.9 in nearly every head.

  • Periodicity (PS): PS is not linear, and here the heads specialize. In each family, 2–5 heads carry no periodicity. Where a head does carry it, it usually agrees in sign with the averaged value (4–6 of 8 heads).

Revisions: We will add the per-head figure and this discussion to the appendix. We will add a discussion on specialized-heads for locality and periodicity.

W4/Q4

We test whether the block-5 attention's detected lag, the input lag, and the forecast lag match. We measure lag using the same autocorrelation method as finding the lag in the attention matrix. We measure the phase of a series by finding the minimum shift where the shifted series matches itself. We measure amplitude by subtracting the maximum value and the minimum value.

Figure R3_W4_1 shows our results. For every series whose block-5 attention identifies a period, the 64-step forecast reproduces the target's frequency, phase, and amplitude. We find no case of a correctly identified attention period followed by a wrong-frequency forecast. We note that the results were restricted to a 64-step forecast since Chronos degrades beyond this horizon length. For this reason, we cannot find the forecast period for a series whose number of steps per cycle is greater than 32.

We note that the attention lag can read a few steps short of long cycles. The reason is because at the presence of the local band on the main diagonal, PS’s autocorrelation pairs the attention around the main diagonal with the periodic peak, which pulls its maximum to a few steps earlier. We show this analytically in the appendix (section F).

Revision: As noted in W1/Q1, the detected lag at the attention matrix does not always match the input lag, and here we explained why. We modified our statement that "The detected period aligns with the true period of the input signal..." to "The detected period closely tracks the true period of the input signal." and also add "For long cycles it reads slightly short because the local band on the main diagonal pulls the autocorrelation peak towards shorter lag (Appendix F)" in the periodicity strength section.

W5/Q5

We thank the reviewer and will correct our description of Mishra (2026) in Section 1: that work does relate SAE features to forecast accuracy through feature ablations. The two approaches analyze different objects. Mishra studies what is represented in each token’s internal activations. We study how tokens exchange information through the encoder’s context-to-context attention. This offers several complementary insights:

  • Structure between pairs of time steps. A local band of a given width, or an off-diagonal band at a fixed lag, describes which time steps attend to which. This can be compared directly with the input’s timescales, such as its period, which a feature defined on a single token does not directly express.
  • Position vs content. In T5, the relative position bias is computed in block 1 and shared by all later blocks. Our layer pruning shows that the collapse when block 1 is removed comes from losing this bias. Attention lets us separate the structure the model imposes through position from the structure it builds from the input.
  • Controlled stimuli. Each family varies one parameter, so we can show where each mode appears across blocks and how it changes with that parameter. Value-based modes dominate early blocks; locality and off-diagonal bands concentrate in later blocks.
  • Comparison across architectures. The same metrics apply unchanged to Moirai-1 (patch embedding) and Moirai-2 (causal attention), without training a new dictionary for each model and layer.

We see the approaches as complementary: SAEs identify what a block represents, while attention shows how information is routed across time.

W6/Q6

Chronos and Moirai differ not only in tokenization but also in architecture, masking and training data, and our comparison between the two models cannot separate these factors.

To test the role of tokenization directly, we ran the experiment the reviewer suggests within a single model. Moirai-1.1-R-small accepts patch sizes P = 8, 16, 32, 64, 128, so the architecture and masking stay fixed and only the patching changes. We used the same signal throughout: a cosine^2 series with 32, 64, 128 or 256 steps per cycle in a 1024 step context. Changing P then changes only how many tokens cover one cycle (tokens per cycle = cycles/P). We measured block-wise attention between context tokens with the same metrics as in the paper.

Our results shown in figure R2_W3_1 show that periodicity changes with patch size for a fixed given signal. We discover a regime of 2-8 tokens per cycle where periodicity strength (PS) exceeds the noise floor. Around 8 tokens per cycle and more, PS begins to drop towards the noise floor. Furthermore, once a patch covers a full cycle or more, PS is at or below the floor in 5 of 6 runs (the exception is at exactly one token per cycle), and below one token per cycle it is zero or at the floor in every run. Within one model, therefore, the choice of patch size affects whether periodic attention can be measured.

Locality does not change with patch size. Attention is more local than uniform at every patch size, with no consistent trend in P. Patch size therefore does not explain the partial locality link we observe in Moirai.

Revision: We will add this experiment and revise our claims. While we cannot rule out architectural and masking differences as a cause of Moirai's weaker periodicity, the patch size alone affects periodicity for the same signal, so tokenization is one contributing factor. The partial locality link in Moirai is not explained by patch size and may reflect other differences between the models. The framework transfers most directly to models with pointwise tokenization, and its application to patch-based models requires accounting for patch size relative to the signal's period.

Reviewer_DqPtOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Thanks for the interesting submission; I enjoyed reading it. As I understand it, the paper opens the original Chronos model to ask why it forecasts some simple signals well and others badly. You feed it controlled synthetic signals (Jacobi elliptic functions, cosines, an iterative series, and the logistic map), average each encoder block’s attention, and measure locality and periodicity with simple metrics compared to a uniform baseline. I believe the main finding is that early blocks show value-based modes, while later blocks show locality and, for periodic inputs, bands one period apart. Block pruning and rank correlations suggest that forecasts are better when the encoder shows strong locality or clear periodicity, except for the iterative series. You also repeat parts of the analysis on Chronos-base, Moirai-1, and Moirai-2, where I understand periodicity is blurrier, and finally, you suggest that pointwise tokenization is what lets Chronos represent periodicity.

Critical strengths

(S1) I like the setup and framework. Using controlled signals with Chronos’s design makes the attention maps easy to read, and I appreciate the simple locality and periodicity metrics with clear uniform baselines.

(S2) I appreciate how careful the paper is. The pruning and position-bias ablation study is careful, and the extra appendices and the released code make the work thorough and easy to check.

Critical weaknesses

(W1) My main concern is that the paper links attention structure to the error of a truncated encoder, not to the failure it sets out to explain. The correlations are computed on the model cut after block 5. For the full model, the link is gone for the Jacobi family and goes the opposite way for the iterative decay series. The paper offers two possible reasons why the last block can hurt, but tests neither, even though the abstract frames the study as explaining this shortcoming.

(W2) I am not sure the link between attention modes and accuracy is strong enough yet. I could not find a test that combines “strong locality or clear periodicity,” and I did not see a threshold for what counts as strong. I also noticed that the abstract’s “neither mode” case (chaotic signals) does not match Sec 4.1 or 4.3, where I read that locality can be high.

(W3) I like the framework, but I think its transfer to other models is weaker than the paper suggests. Sec 2 says the analysis relies on pointwise tokenization, and in Moirai-1 and Moirai-2 periodicity is blurred, and the locality link only partly holds. The paper attributes this to tokenization, but the models also differ in architecture and masking, so this is not isolated.

(W4) Please provide small checks on real or noisier real-world series, since all experiments use synthetic formulas and I could not tell whether the locality and periodicity link holds with noise, trend, or several seasonalities. I understand that controlled stimuli are the point, so even a few real series would help show the modes matter beyond clean formulas.

Questions for authors

(Q1) (W1) Could you share the attention-error analysis for the full model, and test why the last block hurts, maybe by decoding from block 5 versus block 6 on the hardest Jacobi cases?

(Q2) (W2) Could you explain what counts as “strong locality” or “clear periodicity,” and test the two together? I would also love to hear how you analyze the chaotic case, where locality looks high.

(Q3) (W3) Could you check whether locality and periodicity change with patch size in Moirai-1 for the same signal? This would help show whether tokenization is the reason Chronos differs.

3 comments

We thank the reviewer for their comments and address each below:

W1/Q1

We compute the Spearman correlation between block structure (locality/periodicity) and the full model forecast error. Figure R2_W1_1 compares the results to the correlation between block structure and the error of the truncated model in table 5. We observe that for block 5, the relations mostly carry over while all skill correlations keep the same sign in every family.

  • For the Jacobi elliptic, block 5 locality correlates with forecast error at -0.9 for both the full model and the truncated model.
  • For the iterative series (decay), we get the opposite sign correlating with the raw MAE of the full model, however, skill still reads the same sign. The consistent result at block 5 suggest that it maintains its relevance for the model forecast.

On the performance of block 6, we show that decoding from block 5 produces a better forecast than decoding from block 6 for the hardest Jacobi elliptic series. Block 6 is either beneficial or neutral for the other series. To test whether this comes from block 6's attention pattern, we swap block 6’s attention heads with the attention heads from block 5 (figure R2_W1_2). The swap reduces the full model's error and the reduction is largest for the less extreme cases, where the forecast now follows the true decline after the peak instead of drifting upward. However, in no case does the swapped model reproduce the next cycle, which the block-5 decode does. This suggests that the harm of block 6 is not due to its lack of structure.

Figure R2_W1_3 shows that extending the number of cycles helps forecast the hardest Jacobi cases. This suggests block 6 is detrimental when the context does not establish the periodic mode at block 5. With one visible cycle, block 5 attention can't represent the period. When a second cycle is visible, block 5 picks up the period and block 6 is neutral or helps, without carrying the periodicity in its own attention.

Revision: We will add the full model error to block structure correlation table, the experiments on block 6, and revise the discussion on block 6. These additional experiments support our hypothesis that block 6 “applies an inductive bias that is counterproductive when the structural mode is not well established.” We did not test the hypothesis that the block “plays a different functional role for the decoder,” and we will state this. Adding more cycles also introduces more context, however, we did find that increasing context alone on the single cycle Jacobi does not improve the forecast. Furthermore, since the smooth Jacobi cases are not harmed by block 6, the sharpness of the harder Jacobi series may also contribute.

W2/Q2

In the paper, we compute the uniform baseline, that is locality and periodicity on a uniform attention matrix. This baseline distinguishes any activity in the attention matrix and serves as a null reference. Here, we define some new thresholds to distinguish strong locality and clear periodicity.

  • For locality, we compute the constant-input level, which is locality for a constant series. Since position information can contribute to locality (as shown in figure 3), a constant-input only has position information, and any locality above this threshold indicates that the content is meaningfully contributing to locality.
  • For periodicity, we use the noise floor since noisy matrices can trigger periodicity strength (figure R3_W1_2). See XbPV W1 for details.

Using these thresholds, we distinguish our synthetic series into strong/weak locality and clear/unclear periodicity. Figure R2_W2_1 shows how the computed locality and periodicity for our synthetic series compare against these thresholds.

These thresholds for strong locality and clear periodicity resolves the apparent inconsistency with the chaotic regime. For chaotic r (3.7–4.0), block-5 PS is at or below the noise floor, while periodic r (3.0–3.6) is above it. Chaotic MDS (0.53–0.55) is high relative to uniform attention, which is what Sec. 4.1/4.3 described, but it equals the constant-input level. Chaotic inputs therefore show no periodicity and no locality beyond the architecture's own. This is what "neither mode" in the abstract refers to, and these series have the highest errors. See figure R2_W2_2 for the details.

Figure R2_W2_3 shows the median skill (forecast error relative to a persistence forecast) for series in each combination of strong/weak locality and clear/unclear periodicity at block 5. We note that some cells contain only a few series. Series with unclear periodicity at block 5 forecast worse than series with clear periodicity (median skill 0.59 vs 0.05). This is not only a difference between families, as it occurs within elliptic (period), iterative (decay) and logistic (Table R2_W2_4). In iterative (lag) the order is reversed, consistent with our caveat in the paper that the relationship does not hold for the iterative series.

Locality is nevertheless relevant to the truncated model. Among series with unclear periodicity and strong locality, decoding after block 5 forecasts better than the full model (median skill 0.27 vs 0.63), whereas with clear periodicity and strong locality the full model is better (0.25 vs 0.06, Table R2_W2_5). This is the block-6 effect discussed in W1: block 6 is detrimental when the periodic mode is not established at block 5. When periodicity is not resolved, the encoder falls back on local attention, which the position bias largely provides, and a forecast decoded from this local mode can still be reasonable over a short horizon. We note that this difference comes mostly from Elliptic (k). We therefore read locality as the encoder's default mode rather than as a marker of an accurate forecast.

Revision: In light of these results, we have narrowed our claim on the link between structural modes and forecast accuracy, and added the 2 × 2 results as a table (Table 2, 10) and a corresponding discussion. In the abstract, "the forecast accuracy improves when the encoder resolves the input into at least one interpretable structural mode of strong locality or clear periodicity, and degrades when it resolves neither" now reads: "forecast accuracy is better when the later blocks resolve clear periodicity, and worse when they do not, as for near-aperiodic or chaotic signals. Strong locality, by contrast, is not by itself associated with accurate forecasts." We have revised the corresponding statements in the introduction, the discussion, and the conclusion. We also added the constant-input level and the noise floor to the Chronos-small locality and periodicity profiles (Figure 3). These thresholds were introduced for the Chronos-small analysis in Section 4.4; we will extend them to Chronos-base and Moirai in the final version.

W3/Q3

Chronos and Moirai differ not only in tokenization but also in architecture, masking and training data, and our comparison between the two models cannot separate these factors.

To test the role of tokenization directly, we ran the experiment the reviewer suggests within a single model. Moirai-1.1-R-small accepts patch sizes P = 8, 16, 32, 64, 128, so the architecture and masking stay fixed and only the patching changes. We used the same signal throughout: a cosine^2 series with 32, 64, 128 or 256 steps per cycle in a 1024 step context. Changing P then changes only how many tokens cover one cycle (tokens per cycle = cycles/P). We measured block-wise attention between context tokens with the same metrics as in the paper.

Our results shown in figure R2_W3_1 show that periodicity changes with patch size for a fixed given signal. We discover a regime of 2-8 tokens per cycle where periodicity strength (PS) exceeds the noise floor. Around 8 tokens per cycle and more, PS begins to drop towards the noise floor. Furthermore, once a patch covers a full cycle or more, PS is at or below the floor in 5 of 6 runs (the exception is at exactly one token per cycle), and below one token per cycle it is zero or at the floor in every run. Within one model, therefore, the choice of patch size affects whether periodic attention can be measured.

Locality does not change with patch size. Attention is more local than uniform at every patch size, with no consistent trend in P. Patch size therefore does not explain the partial locality link we observe in Moirai.

Revision: We will add this experiment and revise our claims. While we cannot rule out architectural and masking differences as a cause of Moirai's weaker periodicity, the patch size alone affects periodicity for the same signal, so tokenization is one contributing factor. The partial locality link in Moirai is not explained by patch size and may reflect other differences between the models. The framework transfers most directly to models with pointwise tokenization, and its application to patch-based models requires accounting for patch size relative to the signal's period.

W4

We provide some small checks on real data and other perturbed series with Chronos-small. We compare periodicity strength (PS) with a noise floor (described in W2), and locality (MDS) with uniform attention and with the constant-input level (described in W2)

Real series: In every series with a daily cycle (Australian electricity, Traffic, 25 M4 hourly series), block-5 attention is periodic 3-13 times greater than the noise floor.S&P 500 shows no periodicity in any block, and Exchange rate only a weak peak (below the floor). Locality is present in all series, at or moderately above the constant input level. Where block-5 attention picks up the daily period, the forecast reproduces it; for the series without periodicity, the forecast is flat. (Figure R2_W4_1, R2_W4_2)

Noise and trend: Starting from the paper's 4-cycle cosine series, we add noise and trend. Block-5 periodicity persists for moderate trends and for added noise up to half the signal's standard deviation. Periodicity disappears only when the noise is as large as the signal or the trend dominates. Both noise and trend are found to increase locality though never above the constant input level. (Figurre R2_W4_3)

Several seasonalities: Adding faster cycles (8 and 32 per context) to the 4-cycle base series weakens block-5 periodicity but never below the noise floor. The autocorrelation profile shows that the detected lag stays tied to the base cycle. (Figure R2_W4_4)

In all cases the modes behave as in the controlled stimuli: block 5 develops periodic attention when the series has a visible cycle and none when it does not. We note that the link between these modes and forecast accuracy is established in the controlled families (Table 1), where one factor varies at a time. With a few real series, which differ in many respects, such a correlation would not be interpretable. The checks here address the prerequisite: that the modes themselves appear in real and perturbed series. We will add these addition checks to the appendix.

Reviewer_ZqVpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The work addresses the still-limited understanding of how time series foundation models operate internally. To fill this gap, attention matrices are extracted and analyzed for structural patterns, locality, and periodicity. Generally, both locality and periodicity decline slightly but spike just before the last layer. This is connected to the first layers focusing mainly on value-based token mixing due to the inductive biases of the T5 architecture. The last (output) layer behaves very differently again, which is further supported by block ablation studies.

Critical strengths
  1. The work considers an open question (How do TSFMs work?) using a new (simple) methodology.
  2. The investigation is conducted extensively with various intuitive time series and is well visualized.
Critical weaknesses
  1. It is not convincing why a focus on the somewhat dated Chronos 1 model, in the small variant (l. 118), and averaging over heads (l. 202) are reasonable. The work later shows in the rather disconnected Appendix that these limitations are not absolutely necessary (e.g., Moirai-2 is patch-based).
  2. The work is fully qualitative. E.g., the locality measures are computed only on the hand-selected data instances, not on large pretraining or benchmarking time series datasets. It is uncertain whether the insights also hold for those.
Questions for authors
  1. What can explain the fact that here, late layers (e.g., in Chronos-small layer 5) are highly local (cf. Fig. 3), while the opposite is typically the case for language and image modeling?
  2. Why are no foundation models after 2024 mentioned? A lot has happened since, and internal representations have likely changed.
  3. Could the authors comment on how this work adds to the existing literature on time series foundation model internals? Below, I give a minimal set of works currently missing from the introduction (there is no related work section).

Minor opportunities for improvement (no request for responses to these):

  • Why do the models in Fig. 1 "fail" with k→1k \to 1? Isn't the (generally underspecified) task of forecasting just impossible with no patterns in the provided context to exploit? And the context provided of the Jacobi elliptic curves is very little.
  • Fig. 1 is also only partly a vector image.
  • The upper row in Fig. 39 does not seem to show Jacobi elliptic curves.
  • It is odd that Sec. 4 is named "Results" when Sec. 2 already contains results. Also, Sec. 4.1 contains not only results but also the method/experimental setup.
  • l. 231, "an artifact of the model": Which model is meant? The hypothesis was not clear to me.
  • l. 251: The more extreme case than uniform rows of XX would be if attention went only to entries the furthest away.
  • The math in Sec. 3.3 is hard to follow and would benefit from more rigorous definitions using formulas.
  • The paper would benefit from using standard machine learning notation, making vectors and matrices bold.
  • Terms like "figure 1" should be capitalized.
  • l. 628 (references): "opneAI" -> "OpenAI"; Generally, a lot of references lack an indication of the venue or are only preprints.

References:

  • Atharva Pandey, Abhilash Neog, and Gautam Jajoo. On the internal semantics of time-series foundation models. In NeurIPS Workshop on Recent Advances on Time Series Foundation Models (BERT2S), 2025.
  • Divo, Felix, Kraus, Maurice, Härle, Ruben, Ahrend, Patrick, & Kersting, Kristian. (2026). Finding the zeitgeist in time series foundation models. In 1st ICLR Workshop on Time Series in the Age of Large Models.
  • Alexander Chemeris, Ming Jin, and Randall Balestriero. Aionoscope: Debugging latent-state accessibility in time-series representations. arXiv preprint arXiv:2607.00956, 2026.
  • Yu, Annan, Maddix, Danielle C., Han, Boran, Zhang, Xiyuan, Ansari, Abdul Fatir, Shchur, Oleksandr, Faloutsos, Christos, Wilson, Andrew Gordon, Mahoney, Michael W., & Wang, Bernie. (2026). Understanding the implicit biases of design choices for time series foundation models. In The Fourteenth International Conference on Learning Representations.
  • Yu, Annan, Maddix, Danielle C., Han, Boran, Zhang, Xiyuan, Ansari, Abdul Fatir, Shchur, Oleksandr, Faloutsos, Christos, Wilson, Andrew Gordon, Mahoney, Michael W., & Wang, Bernie. (2026). Understanding transformers for time series: Rank structure, flow-of-ranks, and compressibility. In The Fourteenth International Conference on Learning Representations.
2 comments

We thank the reviewer for their comments and address each below:

W1

Why Chronos-1. Our analysis reads attention directly as a relation between time steps. Chronos-1 tokenizes pointwise, i.e. each token is one time step, so an n × n attention matrix is a map of which past time steps each step attends to. In the appendix section, we extend our work towards patch-embedding models and decoder-only models. We agree that the Moirai results in the appendix are not well connected to the main text, and we will revise our claims about transferability. The metrics can be computed on any attention matrix, but the interpretation does not transfer automatically. In particular, for patch-embedding models, the patch size relative to the signal's period must be chosen carefully to obtain the same interpretability. (See response to Dqpt W3 and figure R2_W3_1 for details). Periodicity exceeds the noise floor only when a cycle spans about 2–8 tokens and is at or below the floor when one patch covers a full cycle or more.

Why the small variant. Chronos-small has 6 encoder blocks with 8 heads each. Our layer-pruning analysis removes and adds blocks and reruns the forecast for every series in every family, which is computationally inexpensive at this size. The small number of blocks also keeps the block-by-block development of the modes easy to follow. Larger variants share the same architecture and tokenization, and extending the analysis to them is a natural next step. We report the results for a larger model (Chronos-base) in the appendix section.

Averaging over heads. We recomputed EOD, MDS and PS for each of the 8 heads in every block and every series for Chronos-small, and repeated the Table 1 analysis per head (details in XbPV W3). Figure R3_W3_1 shows the Spearman correlation and find that heads agree with the average most at block 5 and least at block 6. Head 8 tracks the average in every block while head 7 shows no periodicity in most families at any block. Figure R3_W3_2 shows the head-level analysis with forecast accuracy (Table 1)

  • Locality: EOD and MDS are linear in the attention matrix, so the value of the head-averaged matrix is exactly the mean of the 8 heads' values; averaging cannot create a locality trend that the heads do not carry on average. Per head, the block-5 relation between MDS and forecast error keeps the sign of the averaged value in every head for Elliptic (k), Elliptic (period) and Iterative (decay), and in 5–7 of 8 heads for Iterative (lag) and Logistic.
  • Periodicity: In each family, 2–5 heads carry no periodicity. Where a head does carry it, it usually agrees in sign with the averaged value (4–6 of 8 heads). Heads agree with the average most at block 5, the block Table 1 uses.

The head-averaged matrix therefore represents the locality results faithfully, while periodicity is carried by a subset of heads. We will add the per-head analysis to the appendix.

Revision: We will add the per-head analysis to the appendix, and connect the Moirai experiments to the main text in a section on other architectures. There, we will state that the framework applies most directly to pointwise-tokenized models and that patch-based models require accounting for patch size relative to the signal's period.

Q1

While vision models typically resolve small, local structures (e.g. edges) in early layers and focus on shapes and objects in later layers, the order appear reversed in Chronos. We offer an intuitive explanation. For language and vision, finer structures must be resolved first to complete the task, with each layer providing the information that later layers build on. For time series modeling, the logic is reversed: small-scaled structures are meaningless without the larger scaled patterns they are part of. A local fluctuation can only be discerned if it is placed it within the signal's overall shape, such as the extrema or cycle. This is consistent with what we observe. Early blocks are dominated by value-based modes where tokens with equal values attend to each other, and all tokens attend to extrema, while later blocks concentrate attention on local neighborhoods and off-diagonal bands.

Q2

We agree that the field has moved quickly since 2024. Many of the recent models use patch embeddings and decoder-only or causal attention. Our appendix extends our metrics to such models, including one more recent model, Moirai-2, however, we will discuss post-2024 models explicitly in the related work and the future work sections. Our patch-size experiment (DqPt W3) shows that the same metrics can be applied to patch embedding models, as long as the patch size is accounted for relative to the signal's period. Applying the framework to newer models, and testing whether the same structural modes emerge in them, is an important direction for future work, and we will state this in the conclusion.

W2

We agree that our main analysis relies on controlled synthetic stimuli, and we chose them deliberately. We vary one parameter at a time for each family of stimuli so that any changes in the attention structure and the forecast can be attributed to said parameter. Locality (EOD, MDS) and periodicity (PS) are numerical measures on the attention matrices, and Table 1 relates them to forecast error through within-family rank correlations. For the rebuttal, we have also added two reference levels against which these measures can be judged: a noise floor for PS (the PS of structure-free attention) and, for locality, the attention the model gives a constant input, i.e. the locality its position bias provides alone.

Real series: To check whether the modes also appear outside the handselected stimuli, we ran Chronos-small on real benchmark series and on perturbed versions of our synthetic series. In every series with a daily cycle (Australian electricity, Traffic, 25 M4 hourly series), block-5 attention is periodic, with PS 3-13 times greater than the noise floor. S&P 500 shows no periodicity in any block, and Exchange rate only a weak peak (below the floor). Locality is present in all series, at or moderately above the constant-input level. Where block-5 attention picks up the daily period, the forecast reproduces it; for the series without periodicity, the forecast is flat. (Figure R1_W2_1, R1_W2_2)

These checks show that the structural modes are not specific to our stimuli. We do not use these to establish the link between the modes and forecast accuracy because with all the differences real data have, such a correlation would not be interpretable in the same way we establish it in the controlled families. A systematic study on large pretraining or benchmark corpora is beyond the scope of this work, and we will state this as a limitation.

Revision: We will add the real-series to the appendix, and state the limitation that the link between attention structure and forecast accuracy is established on controlled stimuli and remains to be tested on large benchmark datasets.

Q3

We thank the reviewer for these references and will add a related-work section that includes them. These works study what TSFMs represent internally:

  • Pandey et al. (2025) and Chemeris et al. (2026) use linear probes to test which properties of the input can be read from a model’s internal activations.
  • Divo et al. (2026) and Mishra (2026) use sparse autoencoders to identify interpretable features in Chronos models and, in Mishra’s case, how much forecasts depend on them.
  • Yu et al. (2026a, b) analyze how design choices such as patching, embeddings and loss shape a model’s representations, and how their rank changes with depth.

These works focus on internal activations. Our work instead studies the structure of context-to-context attention: which past time steps each token attends to, and how this depends on the input signal. It adds three things:

  • Interpretable, model-agnostic metrics of attention structure: Locality (EOD, MDS) and periodicity (PS) are defined directly on the attention matrices. They can be compared with the signal’s own timescales and applied to other models without training a probe or dictionary.
  • Controlled parametric stimuli: We vary one parameter at a time: the Jacobi modulus k, the period, the decay and lag of iterative series, and the logistic r. This shows how each structural mode emerges block by block and changes with the parameter.
  • Layer pruning tied to forecast accuracy: Pruning locates where each mode forms, separates the role of the relative position bias (computed in block 1 and shared by later blocks), and links encoder structure to forecast error.

Our observation that locality is the dominant mode in Chronos is consistent with Yu et al. (2026a), who argue that quantized embeddings such as Chronos’s encourage localized attention.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Main Revisions to Paper

We have narrowed the paper's central claim. The original abstract linked accurate forecasts to strong locality or clear periodicity. Using two stricter reference levels, the constant-input level for locality (what the position bias provides alone) and a noise floor for periodicity strength, we find that clear periodicity at block 5 goes with better forecasts (median skill 0.05 vs 0.59), also within families, while strong locality does not. We now read locality as the encoder's default mode. The abstract now reads: "For several stimulus families, forecast accuracy is better when the later blocks resolve clear periodicity, and worse when they do not, as for near-aperiodic or chaotic signals. Strong locality, by contrast, is not by itself associated with accurate forecasts." We have revised the introduction, Figure 1 and the discussion to match, added the analysis to Section 4.4 (Tables 2, 10), and added both reference levels to Figure 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.