acceptodds
Withdrawn from ICLR 2027

Understanding and Improving Zero-Shot Forecasting through Attention-Head Functions in Time-Series Foundation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Transformer-based Time-Series Foundation Models (T²SFMs) enable zero-shot forecasting across diverse domains by inferring temporal characteristics from unseen input contexts. However, how such capability is internally organized and utilized for forecasting remains poorly understood. Since self-attention performs input-dependent computation, attention heads provide a natural unit for investigating this question. We therefore introduce FBC, a Frequency-feature Based Causal framework for characterizing attention-head functions, revealing that individual heads are multifunctional yet functionally redundant, their functions exhibit both cross-domain stability and variation, and scaling functional heads causally alters forecasts. Together, these findings reveal how attention-head functions are organized and support zero-shot forecasting. Building on these, we propose SHIFT, a two-stage Soft-gated Head-wise Intervention Framework for improving T²SFMs' zero-shot forecasting capability. In calibration stage, SHIFT's soft-gated head-wise calibration module learns intervention vectors across observed domains and constructs a source-domain centroid bank. In inference stage, centroid-based retrieval module identifies a compatible source domain for unseen targets, and transfers its calibrated intervention vectors to modulate attention-head outputs, thereby enhancing zero-shot forecasting. Experiments on six T²SFMs across seven datasets demonstrate consistent zero-shot improvements, with average MSE and MASE reductions of up to 14.5% and 65.8%, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Time series foundation model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Qf1COpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how attention heads contribute to zero-shot forecasting in time-series foundation models (T²SFMs). It first proposes FBC, a frequency-feature-based causal framework that uses mean ablation to measure each head’s causal effect, then decomposes the resulting forecast changes into frequency bands such as level, low-frequency variation, seasonality, and residual components. Based on this, the paper argues that attention heads are multifunctional, redundant, domain-adaptive, and controllable through scaling. Building on these findings, the paper proposes SHIFT, a soft-gated head-wise intervention method. The pretrained T²SFM is kept frozen, while SHIFT learns lightweight head-wise scaling parameters from observed source domains. At inference time, it computes a target-domain activation centroid, retrieves the most compatible source-domain intervention, and applies the corresponding head-wise scaling.

Critical strengths
  1. The paper makes a clear attempt to connect analysis of head functions with an actual intervention method. A major strength is the interpretability-to-improvement pipeline. FBC is not only used for post-hoc analysis; its findings motivate SHIFT’s design, including head-wise scaling, shared/domain-specific intervention vectors, and soft gating.
Critical weaknesses
  1. SHIFT is only compared against the frozen vanilla TSFM, while it additionally uses thousands of labeled source-domain windows and unlabeled target contexts for retrieval. The paper also omits stronger baselines using the same source data, such as source fine-tuning, LoRA, LoFiT-style head intervention, in-context fine-tuning, layer-wise scaling, global head scaling, or simple output recalibration. Without these comparisons, it is hard to know whether the gains come from the proposed head-wise mechanism or from generic low-parameter source-domain calibration.

  2. The largest reported gains come mainly from near-domain transfers, such as ETTm1 to ETTm2 and BizITObs-Service to BizITObs-Application, and from cases where the vanilla model performs poorly. On more distinct targets such as Weather, Exchange, SZ-Taxi, and Bitbrains-Fast-Storage, the improvements are often only a few percent or even close to zero.

  3. Adaptivity is mainly analyzed between ETTh1 and ETTm1, which are closely related datasets from the same family, so calling this broad cross-domain adaptivity may be too strong. Redundancy is inferred from similar spectral profiles rather than causal compensation tests, and controllability is supported mainly by qualitative synthetic examples. The paper should provide more quantitative and diverse evidence for these mechanistic claims.

  4. The frequency-based functional score may be too narrow. Reducing head behavior to trend- versus periodicity-dominant roles may miss other important time-series behaviors, such as cross-variable dependency, regime shifts, anomalies, distribution shifts, and long-range events. The paper should also clarify how LOW and SEAS bands are separated, especially when the seasonal period may coincide with the lowest non-zero Fourier bin.

  5. The shared/domain-specific decomposition is not clearly identifiable during source-domain training. Since training uses the sum of the shared and source-specific vectors, shifting mass between them can leave the effective intervention similar. This weakens interpretations such as the shared vector being much smaller than domain-specific vectors. Moreover, the shared component is never trained alone, yet it is used alone in the fallback path.

  6. The zero-shot framing should be qualified. SHIFT does not use target labels, but it does learn from labeled source-domain windows and uses source validation data for early stopping.

  7. The paper reports many point estimates but does not provide repeated-seed results or confidence intervals. This matters because many improvements, especially on less related targets, are small.

Questions for authors
  1. Can the authors compare SHIFT with baselines that use the same labeled source data, such as source fine-tuning, LoRA, LoFiT-style head intervention, in-context fine-tuning, layer-wise scaling, or simple output recalibration?

  2. Can the authors report average gains over all model–target pairs and separate near-domain from far-domain transfer? The largest gains appear to come from closely related source–target pairs.

  3. Can the authors provide stronger evidence for the FBC mechanistic claims? Adaptivity is mainly shown on closely related ETT datasets, redundancy is based on spectral similarity, and controllability is mostly qualitative.

  4. Can the authors clarify how the LOW and SEAS frequency bands are defined, especially when the seasonal period coincides with the lowest non-zero Fourier bin?

  5. Can the authors address whether the shared and domain-specific intervention vectors are identifiable, since training only uses their combined effect?

  6. Can the authors clarify the zero-shot framing?

  7. Can the authors report repeated-seed results or confidence intervals, since many improvements on less related targets are small?

Reviewer_YCHHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what individual attention heads do in Transformer time-series foundation models. It proposes FBC, which mean-ablates heads, keeps the top heads by |ΔMSE|, and then measures how much of the ablation-induced forecast change falls into DC, low-frequency and seasonal bands, summarized as a score r (periodicity vs trend). Using ETTh1 and ETTm1, the authors report three findings: heads are multifunctional and redundant, their roles are partly stable and partly flip across datasets, and scaling a head changes the forecast in line with its role. They then propose SHIFT, which learns per-head multiplicative scalars (shared + source-specific, with soft gates initialized from the FBC head set) on a few source datasets, and at test time picks a source-specific vector by cosine similarity of head activation norms, falling back to the shared vector if a threshold is not met. Experiments on six TSFMs report gains on held-out TSLib and GIFT-Eval datasets.

Critical strengths

Strength 1. Head-level analysis of TSFMs is a nice angle. Applying mean ablation plus a frequency-band decomposition of the ablation effect is simple and fairly intuitive, and I liked that it is done across five or six different backbones rather than one. The synthetic scaling demo in Figure 3(b,c) makes the “trend head vs periodicity head” story concrete.

Strength 2. Very cheap intervention that helps across backbones. SHIFT trains around (n_s+2)N scalars with the backbone frozen, and the TSLib gains in Table 1 show up for almost every model and horizon. The mechanistic case study on ETTm2 (Figure 5) tying the largest adjustments to periodicity heads for Moirai and trend heads for TimesFM is a good attempt at closing the loop between analysis and method.

Critical weaknesses

Weakness 1. The only baseline is the vanilla frozen model. This is my main issue. SHIFT uses labeled source data (3,000 to 6,000 windows per source, early stopping on source validation), so the fair comparison is with other ways of using that same source data and then applying the model to the targets: LoRA or LayerNorm/bias tuning on the sources, a linear output calibration, or even simply scaling all heads without the FBC-based gate initialization. Without these I cannot tell whether the gains come from head-wise intervention specifically, or just from any small amount of tuning on electricity data that happens to be close to the targets. I was also a bit surprised there is no row in Table 1 for shared-only vs full SHIFT, since Figure 5a shows this only for one dataset.

Weakness 2. The “zero-shot / unseen domain” framing is stronger than what the splits test. Looking at Table 18, the retrieval essentially matches ETTm2 to ETTm1 and ETTh2 to ETTh1 for every model, and BizITObs-Application to BizITObs-Service. These are sibling datasets with the same sampling rate, so basically the near-in-distribution case. This is exactly where the large numbers come from (ETTm2 up to 14.5%, BizITObs-Application up to 65.8%), while the genuinely different targets (Weather, Exchange, SZ-Taxi, Fast-Storage) fall back to shared-only and gains shrink to around 0 to 4% for most models (e.g. Moirai and Time-MoE at 0.4% on Weather). The 65.8% for Chronos also starts from a vanilla MASE of 8.44 on a stream where TimesFM gets 2.21, so it looks more like fixing a pathological failure using a sister stream (the source BizITObs-Service shows the same -72.8% in Table 11) than improved zero-shot ability. The percentile q also differs per model (10, 25, 50), and “selected using source data only” is not explained, which matters because the threshold decides which targets get the big gain.

Smaller issues: all results appear to be single runs with no variance, which is a problem for the many sub-1% deltas (I may have missed it, but I saw no seeds anywhere). The FBC findings rest on only two ETT datasets, and the link from “entanglement/adaptivity” to the actual design (per-head scalars with shared + specific parts) feels more like post hoc motivation than a derivation.

Questions for authors

Q1. Can you add baselines that use the same source data, e.g. LoRA or norm/bias tuning on ETTm1/ETTh1/ECL evaluated on the same targets, and a version that scales all heads with random or uniform gate init?

Q2. How exactly was q chosen per model using only source data? And what happens on ETTm2/ETTh2 if you force shared-only, i.e. how much of the headline gain comes from the retrieved source-specific vector?

Q3. Could you report a leave-one-family-out setting (e.g. no ETT sources when testing on ETTm2/ETTh2, no BizITObs sources when testing on BizITObs-Application)? I think this would tell much more about real zero-shot transfer.

Reviewer_fq2UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SHIFT, a head-wise intervention framework for improving zero-shot forecasting with T$^2$SFMs. The authors analyze attention-head functions through mean ablation and frequency analysis, then learn shared and domain-specific head scaling factors and soft gates while keeping the backbone frozen. SHIFT transfers these interventions to unseen domains using activation-based retrieval, with experiments on six T$^2$SFMs and seven held-out datasets reporting improved forecasting performance.

Critical strengths

1. Well-Motivated Head-Level Analysis

The paper investigates how attention heads in Transformer-based time-series foundation models contribute to forecasting, with the aim of using head-level interventions to improve predictions. The head-scaling experiments show how modifying individual heads changes predictions, providing more direct evidence of head influence than attention visualization alone.

2. Simple Intervention Design with Broad Evaluation

SHIFT keeps the backbone frozen and learns only head-wise scaling factors and soft gates. The method is evaluated on six T$^2$SFMs and seven held-out target datasets, showing overall forecasting improvements without target labels or target-specific optimization. Its small number of trainable parameters and applicability across different backbones make the approach practically appealing.

Critical weaknesses

1. Insufficient Evidence Connecting the Functional Analysis to SHIFT’s Improvements

The main results establish improvements over the frozen backbone, but do not sufficiently distinguish the proposed mechanism from general calibration on source data. Comparisons with simple output affine calibration or layer-wise scaling under the same source-data and training budget would help determine whether head-wise intervention provides a distinct advantage.

2. Inconsistency Between the Retrieval Definition and Reported Results

Equations 12–15 define the retrieval fingerprints and centroids using averages of L2 norms, making all coordinates nonnegative. Their cosine similarities should therefore be nonnegative whenever the centroids are nonzero. However, Tables 18–19 report multiple negative values without describing centering, standardization, or another transformation that would explain them.

3. Minor Issues

  • The main results and gating ablations do not report variability across repeated calibration runs or sampled source windows. This is particularly relevant when the differences are small.
Questions for authors
  1. Could the authors provide comparisons with simple output affine calibration or layer-wise scaling using the same source data and a comparable training budget?

  2. My understanding of Equations 12–15 is that the activation-norm centroids have nonnegative coordinates, which would imply nonnegative cosine similarities. However, Tables 18–19 report negative values. Could the authors clarify this discrepancy, or point me to any explanation I may have overlooked?

  3. Could the authors report the variability of the results across repeated calibration runs or source-window samples, particularly for settings with small improvements? This would help assess whether these gains are consistent across runs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.