- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates whether time-series foundation models can be truncated once forecasting information becomes recoverable at intermediate layers. It introduces the forecasting tunnel to identify such depths, analyzes the mismatch between intermediate representations and the pretrained forecasting pathway, and uses lightweight affine alignment to enable effective truncation. Experiments on three TSFMs and 14 datasets show that near-final forecasting information often emerges well before the final layer, and aligned truncation can substantially reduce inference cost with limited performance degradation.
Critical strengths
The proposed forecasting tunnel provides a simple functional criterion for identifying when near-final forecasting performance becomes recoverable, and the paper shows that this notion is distinct from representation similarity or effective-rank saturation. This is consequential because it gives a more directly actionable way to study removable depth in TSFMs.
The paper does not stop at layerwise probing: it identifies the compatibility gap between intermediate states and the pretrained forecasting pathway, introduces lightweight affine alignment, and validates actual physical truncation and end-to-end speedups. This makes the analysis operational rather than purely diagnostic.
Critical weaknesses
A key appeal of TSFMs is their zero-shot forecasting ability, since they are not necessarily most useful in full-shot settings. However, the tunnel entrance varies substantially across datasets and architectures, from very early to near-final layers. Although the proposed adapter does not require ground-truth labels, it still needs target-domain samples to be passed through the full model to fit a dataset-specific adapter. This adds extra adaptation cost for every new dataset and therefore weakens the practical value of the method as a zero-shot acceleration approach.
A key concern is that the proposed method does not seem to exploit any characteristics specific to time series. The observation that different layers contribute unequally, and that some layers may be redundant or removable, is already well established across other modalities such as language and vision. In this sense, the forecasting tunnel, layerwise probing, alignment, and truncation framework appear largely modality-agnostic. The paper would be stronger if it clarified whether this layer-wise redundancy reflects any intrinsic dynamics of TSFMs, or whether it is simply another instance of a more general deep-network phenomenon.
Another concern is that the truncation strategy only keeps a contiguous prefix of the network and removes all later layers. This implicitly assumes that layer importance is roughly ordered from early to late. However, prior studies in deep models have shown that layer contributions can be highly non-monotonic, with some middle or later layers being more important than others. Therefore, restricting the search to prefix truncation may overlook better layer-selection or layer-skipping strategies.
Questions for authors
-
Have you tried non-contiguous layer selection or skipping specific intermediate layers, rather than always truncating from the end? If so, how does the performance compare?
-
What aspects of the proposed method are specific to time series?
2 comments
Q2. What aspects of the proposed method are specific to time series?
We thank the reviewer for raising this question - we agree that the general framework of layerwise probing, alignment, and truncation can be considered modality-agnostic. Our contribution is to show that identifying removable depth in TSFMs requires accounting for their forecasting architectures, and that ignoring these differences can lead to misleading conclusions about forecast recoverability.
We probe the hidden states used by each model's native forecasting head: forecast-slot tokens in Chronos-2, the last context token in TimesFM-3, and the final input token of each forecasting pass in TiRex.
To test the importance of this choice, we added a comparison with generic content-mean pooling across all datasets (Appendix B.1.5). Relative to our architecture-aware readouts, generic pooling increases median final-block test loss by 27.2%, 15.9%, and 20.9% for Chronos-2, TimesFM-3, and TiRex, respectively. Even when controlling for readout structure and parameter count, the increases remain 11.1%, 15.9%, and 15.6%, so the state selection matters independently of readout capacity.
Generic pooling also produces earlier apparent 5% tunnel entrances, shifting the median from 29% to 8% of depth for Chronos-2 and from 79% to 21% for TiRex. However, its final-block test loss already exceeds that of the architecture-aware readout at the latter's tunnel entrance. We therefore interpret these earlier entrances as saturation of a weaker readout, rather than earlier recovery of accurate forecasts.
Overall, while the underlying techniques are general, our additional experiments provide stronger evidence that forecasting-aware state selection substantially affects both prediction accuracy and the identification of redundant depth. These results reinforce the importance of adapting layerwise analysis to the forecasting mechanisms of TSFMs rather than relying on generic probing strategies.
Dataset-specific calibration
We agree that fitting an adapter separately for each dataset limits zero-shot use, so we ran leave-one-dataset-out transfer for Chronos-2 at L4, L6, and L9. For each held-out target, an adapter is trained on the other 13 datasets, or 12 when the two Electricity datasets must be excluded together, and evaluated without target-specific fitting or selection (Appendix B.4.2).
The transferred adapter increases median test MASE by +12.7%, +9.0%, and +4.0% at L4, L6, and L9, compared with +37.0%, +20.1%, and +17.1% for the hard cut, and +3.1%, +2.3%, and +2.2% for dataset-specific adapters. An adapter therefore can transfer to an unseen target at mild truncation (+4.0% at 1.28×), although calibration on unlabeled target windows remains better at aggressive cuts. For Chronos-2, that calibration takes one pass of the full model over 1,394 windows (about one second at batch size 256 on an A100) and about one minute of fitting. The cost can be amortized over repeated forecasts on the same target, but we agree it cannot be ignored in a zero-shot setting.
We have added these results and updated the limitations. Together, these results demonstrate viable paths to acceleration both with and without target-specific calibration.
Q1. Have you tried non-contiguous layer selection or skipping specific intermediate layers, rather than always truncating from the end? If so, how does the performance compare?
We thank the reviewer for suggesting this comparison. In the updated version, we compared prefix truncation with three fixed alternatives for Chronos-2: uniformly spaced blocks, early-plus-late blocks, and a contiguous middle window. Each configuration retains 4, 6, or 9 of the 12 blocks in their original order and uses the same label-free adapter and selection procedure. Measured speedups at matched retained depth differ by at most 6% (Appendix B.4.3).
Median test MASE increase over the full model, with the hard-cut result in parentheses:
At aggressive compression (four retained blocks, 2.43× speedup), the prefix performs best, and it also achieves the lowest median degradation at six blocks At nine blocks, the reviewer is right: retaining later blocks while skipping intermediate ones performs better (+0.9% versus +2.2% for the prefix), and the uniform configuration's hard cut is already close to the full model.
Additional single-block removal experiments confirm that block importance is not monotonic across depth in Chronos-2. Removing the final block increases median MASE by +18.5% (worse on all datasets), removing block 1 by +5.2%, and removing any individual block from 2 to 11 by at most +2.2%. Thus, the first and final blocks are more consequential than most intermediate blocks, which is consistent with the compatibility gap: the final block produces states that the pretrained forecasting head can use directly, whereas prefix truncation removes that block and relies on alignment.
While these experiments cover only four predefined masks on Chronos-2 rather than an exhaustive search, they show that the best layer-selection strategy can depend on the compression level. In our experiments, prefix truncation performs best at aggressive compression, while non-contiguous selection can further improve accuracy at milder cuts. We have added these results and clarified the limitations of our prefix-only approach, highlighting additional opportunities for efficient inference through more flexible layer selection.