- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how attention heads contribute to zero-shot forecasting in time-series foundation models (T²SFMs). It first proposes FBC, a frequency-feature-based causal framework that uses mean ablation to measure each head’s causal effect, then decomposes the resulting forecast changes into frequency bands such as level, low-frequency variation, seasonality, and residual components. Based on this, the paper argues that attention heads are multifunctional, redundant, domain-adaptive, and controllable through scaling. Building on these findings, the paper proposes SHIFT, a soft-gated head-wise intervention method. The pretrained T²SFM is kept frozen, while SHIFT learns lightweight head-wise scaling parameters from observed source domains. At inference time, it computes a target-domain activation centroid, retrieves the most compatible source-domain intervention, and applies the corresponding head-wise scaling.
Critical strengths
- The paper makes a clear attempt to connect analysis of head functions with an actual intervention method. A major strength is the interpretability-to-improvement pipeline. FBC is not only used for post-hoc analysis; its findings motivate SHIFT’s design, including head-wise scaling, shared/domain-specific intervention vectors, and soft gating.
Critical weaknesses
-
SHIFT is only compared against the frozen vanilla TSFM, while it additionally uses thousands of labeled source-domain windows and unlabeled target contexts for retrieval. The paper also omits stronger baselines using the same source data, such as source fine-tuning, LoRA, LoFiT-style head intervention, in-context fine-tuning, layer-wise scaling, global head scaling, or simple output recalibration. Without these comparisons, it is hard to know whether the gains come from the proposed head-wise mechanism or from generic low-parameter source-domain calibration.
-
The largest reported gains come mainly from near-domain transfers, such as ETTm1 to ETTm2 and BizITObs-Service to BizITObs-Application, and from cases where the vanilla model performs poorly. On more distinct targets such as Weather, Exchange, SZ-Taxi, and Bitbrains-Fast-Storage, the improvements are often only a few percent or even close to zero.
-
Adaptivity is mainly analyzed between ETTh1 and ETTm1, which are closely related datasets from the same family, so calling this broad cross-domain adaptivity may be too strong. Redundancy is inferred from similar spectral profiles rather than causal compensation tests, and controllability is supported mainly by qualitative synthetic examples. The paper should provide more quantitative and diverse evidence for these mechanistic claims.
-
The frequency-based functional score may be too narrow. Reducing head behavior to trend- versus periodicity-dominant roles may miss other important time-series behaviors, such as cross-variable dependency, regime shifts, anomalies, distribution shifts, and long-range events. The paper should also clarify how LOW and SEAS bands are separated, especially when the seasonal period may coincide with the lowest non-zero Fourier bin.
-
The shared/domain-specific decomposition is not clearly identifiable during source-domain training. Since training uses the sum of the shared and source-specific vectors, shifting mass between them can leave the effective intervention similar. This weakens interpretations such as the shared vector being much smaller than domain-specific vectors. Moreover, the shared component is never trained alone, yet it is used alone in the fallback path.
-
The zero-shot framing should be qualified. SHIFT does not use target labels, but it does learn from labeled source-domain windows and uses source validation data for early stopping.
-
The paper reports many point estimates but does not provide repeated-seed results or confidence intervals. This matters because many improvements, especially on less related targets, are small.
Questions for authors
-
Can the authors compare SHIFT with baselines that use the same labeled source data, such as source fine-tuning, LoRA, LoFiT-style head intervention, in-context fine-tuning, layer-wise scaling, or simple output recalibration?
-
Can the authors report average gains over all model–target pairs and separate near-domain from far-domain transfer? The largest gains appear to come from closely related source–target pairs.
-
Can the authors provide stronger evidence for the FBC mechanistic claims? Adaptivity is mainly shown on closely related ETT datasets, redundancy is based on spectral similarity, and controllability is mostly qualitative.
-
Can the authors clarify how the LOW and SEAS frequency bands are defined, especially when the seasonal period coincides with the lowest non-zero Fourier bin?
-
Can the authors address whether the shared and domain-specific intervention vectors are identifiable, since training only uses their combined effect?
-
Can the authors clarify the zero-shot framing?
-
Can the authors report repeated-seed results or confidence intervals, since many improvements on less related targets are small?