acceptodds
Under review as a conference paper at ICLR 2027

Truncate to Accelerate: From Forecast Recoverability to Efficient Time-Series Foundation Models

Abstract

In large language models, intermediate representations have been used both to study how task-relevant information evolves with depth and to accelerate inference. In time-series foundation models (TSFMs), however, these questions remain largely unexplored. We address this gap through layerwise probing, representation geometry, and lightweight adaptation across three state-of-the-art TSFMs: Chronos-2, TimesFM-3, and TiRex, evaluated on a collection of datasets with diverse sampling frequencies. First, layerwise forecasting readouts reveal a forecasting tunnel: across nearly all model-dataset pairs, intermediate depths already recover performance within 5% of the final-block readout. Next, we find that functional saturation coexists with continued geometric evolution. Finally, intermediate states can contain recoverable forecast information, but feeding them directly into the model's native forecasting pathway degrades performance relative to the full model. Lightweight affine alignment substantially reduces this gap and exposes a measured performance-compute frontier: aligned Chronos-2 remains within a median 3.1% of the full model after retaining only 4 of its 12 blocks, while running faster, with the achievable tradeoff varying substantially across architectures. Moreover, cuts inside the forecasting tunnel are typically better tolerated at matched depth, linking early recoverability to practical truncation. Together, these results suggest that much of the forecast-relevant information becomes recoverable before the final block and offer a path toward both understanding and accelerating time-series foundation models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model compression, efficient inference, inference acceleration, truncation, recoverability, time-series foundation models, time-series forecasting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dHTDOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether time-series foundation models can be truncated once forecasting information becomes recoverable at intermediate layers. It introduces the forecasting tunnel to identify such depths, analyzes the mismatch between intermediate representations and the pretrained forecasting pathway, and uses lightweight affine alignment to enable effective truncation. Experiments on three TSFMs and 14 datasets show that near-final forecasting information often emerges well before the final layer, and aligned truncation can substantially reduce inference cost with limited performance degradation.

Critical strengths

The proposed forecasting tunnel provides a simple functional criterion for identifying when near-final forecasting performance becomes recoverable, and the paper shows that this notion is distinct from representation similarity or effective-rank saturation. This is consequential because it gives a more directly actionable way to study removable depth in TSFMs.

The paper does not stop at layerwise probing: it identifies the compatibility gap between intermediate states and the pretrained forecasting pathway, introduces lightweight affine alignment, and validates actual physical truncation and end-to-end speedups. This makes the analysis operational rather than purely diagnostic.

Critical weaknesses

A key appeal of TSFMs is their zero-shot forecasting ability, since they are not necessarily most useful in full-shot settings. However, the tunnel entrance varies substantially across datasets and architectures, from very early to near-final layers. Although the proposed adapter does not require ground-truth labels, it still needs target-domain samples to be passed through the full model to fit a dataset-specific adapter. This adds extra adaptation cost for every new dataset and therefore weakens the practical value of the method as a zero-shot acceleration approach.

A key concern is that the proposed method does not seem to exploit any characteristics specific to time series. The observation that different layers contribute unequally, and that some layers may be redundant or removable, is already well established across other modalities such as language and vision. In this sense, the forecasting tunnel, layerwise probing, alignment, and truncation framework appear largely modality-agnostic. The paper would be stronger if it clarified whether this layer-wise redundancy reflects any intrinsic dynamics of TSFMs, or whether it is simply another instance of a more general deep-network phenomenon.

Another concern is that the truncation strategy only keeps a contiguous prefix of the network and removes all later layers. This implicitly assumes that layer importance is roughly ordered from early to late. However, prior studies in deep models have shown that layer contributions can be highly non-monotonic, with some middle or later layers being more important than others. Therefore, restricting the search to prefix truncation may overlook better layer-selection or layer-skipping strategies.

Questions for authors
  1. Have you tried non-contiguous layer selection or skipping specific intermediate layers, rather than always truncating from the end? If so, how does the performance compare?

  2. What aspects of the proposed method are specific to time series?

2 comments

Dataset-specific calibration

We agree that fitting an adapter separately for each dataset limits zero-shot use, so we ran leave-one-dataset-out transfer for Chronos-2 at L4, L6, and L9. For each held-out target, an adapter is trained on the other 13 datasets, or 12 when the two Electricity datasets must be excluded together, and evaluated without target-specific fitting or selection (Appendix B.4.2).

The transferred adapter increases median test MASE by +12.7%, +9.0%, and +4.0% at L4, L6, and L9, compared with +37.0%, +20.1%, and +17.1% for the hard cut, and +3.1%, +2.3%, and +2.2% for dataset-specific adapters. An adapter therefore can transfer to an unseen target at mild truncation (+4.0% at 1.28×), although calibration on unlabeled target windows remains better at aggressive cuts. For Chronos-2, that calibration takes one pass of the full model over 1,394 windows (about one second at batch size 256 on an A100) and about one minute of fitting. The cost can be amortized over repeated forecasts on the same target, but we agree it cannot be ignored in a zero-shot setting.

We have added these results and updated the limitations. Together, these results demonstrate viable paths to acceleration both with and without target-specific calibration.

Q1. Have you tried non-contiguous layer selection or skipping specific intermediate layers, rather than always truncating from the end? If so, how does the performance compare?

We thank the reviewer for suggesting this comparison. In the updated version, we compared prefix truncation with three fixed alternatives for Chronos-2: uniformly spaced blocks, early-plus-late blocks, and a contiguous middle window. Each configuration retains 4, 6, or 9 of the 12 blocks in their original order and uses the same label-free adapter and selection procedure. Measured speedups at matched retained depth differ by at most 6% (Appendix B.4.3).

Median test MASE increase over the full model, with the hard-cut result in parentheses:

Blocks keptPrefixUniformEarly + lateMiddle window
4+3.1% (+37.0%)+8.9% (+42.2%)+10.1% (+31.3%)+7.2% (+63.3%)
6+2.3% (+20.1%)+4.2% (+17.5%)+7.6% (+23.1%)+7.6% (+52.0%)
9+2.2% (+17.1%)+0.9% (+1.1%)+0.9% (+2.1%)+3.4% (+20.2%)

At aggressive compression (four retained blocks, 2.43× speedup), the prefix performs best, and it also achieves the lowest median degradation at six blocks At nine blocks, the reviewer is right: retaining later blocks while skipping intermediate ones performs better (+0.9% versus +2.2% for the prefix), and the uniform configuration's hard cut is already close to the full model.

Additional single-block removal experiments confirm that block importance is not monotonic across depth in Chronos-2. Removing the final block increases median MASE by +18.5% (worse on all datasets), removing block 1 by +5.2%, and removing any individual block from 2 to 11 by at most +2.2%. Thus, the first and final blocks are more consequential than most intermediate blocks, which is consistent with the compatibility gap: the final block produces states that the pretrained forecasting head can use directly, whereas prefix truncation removes that block and relies on alignment.

While these experiments cover only four predefined masks on Chronos-2 rather than an exhaustive search, they show that the best layer-selection strategy can depend on the compression level. In our experiments, prefix truncation performs best at aggressive compression, while non-contiguous selection can further improve accuracy at milder cuts. We have added these results and clarified the limitations of our prefix-only approach, highlighting additional opportunities for efficient inference through more flexible layer selection.

Q2. What aspects of the proposed method are specific to time series?

We thank the reviewer for raising this question - we agree that the general framework of layerwise probing, alignment, and truncation can be considered modality-agnostic. Our contribution is to show that identifying removable depth in TSFMs requires accounting for their forecasting architectures, and that ignoring these differences can lead to misleading conclusions about forecast recoverability.

We probe the hidden states used by each model's native forecasting head: forecast-slot tokens in Chronos-2, the last context token in TimesFM-3, and the final input token of each forecasting pass in TiRex.

To test the importance of this choice, we added a comparison with generic content-mean pooling across all datasets (Appendix B.1.5). Relative to our architecture-aware readouts, generic pooling increases median final-block test loss by 27.2%, 15.9%, and 20.9% for Chronos-2, TimesFM-3, and TiRex, respectively. Even when controlling for readout structure and parameter count, the increases remain 11.1%, 15.9%, and 15.6%, so the state selection matters independently of readout capacity.

Generic pooling also produces earlier apparent 5% tunnel entrances, shifting the median from 29% to 8% of depth for Chronos-2 and from 79% to 21% for TiRex. However, its final-block test loss already exceeds that of the architecture-aware readout at the latter's tunnel entrance. We therefore interpret these earlier entrances as saturation of a weaker readout, rather than earlier recovery of accurate forecasts.

Overall, while the underlying techniques are general, our additional experiments provide stronger evidence that forecasting-aware state selection substantially affects both prediction accuracy and the identification of redundant depth. These results reinforce the importance of adapting layerwise analysis to the forecasting mechanisms of TSFMs rather than relying on generic probing strategies.

Reviewer_96qNOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates where forecast-relevant information becomes available inside three frozen time-series foundation models, Chronos-2, TimesFM-3, and TiRex, across 14 datasets with fixed C=512 and H=64. The authors define the sustained forecasting tunnel as the earliest depth after which layerwise linear quantile readouts stay within 5% of the final-block readout, finding it begins before the final block in 40 of 42 model-dataset pairs while debiased CKA and effective rank fail to mark the entrance. A label-free affine adapter trained to match the full model's forecasts bridges the gap of direct truncation, allowing physically truncated Chronos-2 to keep 4 of 12 blocks for a 2.43x speedup at a 3.1% median MASE increase, and TiRex to reach 1.71x speedup at +8.5% MASE.

Critical strengths

The experimental protocol is rigorous and transparent, relying strictly on validation data for selection, evaluating on identical checksummed windows, building physically truncated models for end-to-end timing, and candidly reporting pretraining overlap and negative results.

Post-hoc affine alignment offers a practical way to get multiple operating points from a single Chronos-2 checkpoint, such as 1.78x speedup at +2.3% error or 2.43x at +3.1%, while cleanly separating linear recoverability from native head compatibility.

Critical weaknesses

The tunnel is defined against a handicapped reference because the 5% threshold is measured relative to the final-block linear probe rather than the native model, yet Table 2 shows this probe is already 11.6% to 17.0% worse in MASE than the full model, and 7 pairs enter the tunnel at input layer L0, showing that being within 5% of the final probe does not mean near-native performance.

The tunnel fails to locate good cut points, as Chronos-2 has median entrances around L3 to L10 while the best trade-offs are at L4 to L6, TimesFM-3 still loses 40% to 68% at most depths inside the tunnel, and the inside-versus-before comparison in Table 14 is a post-hoc observation over small groups of 2 to 6 datasets confounded by task difficulty.

The aligned pathway obscures adaptation costs and removes the zero-shot property because the adapter is trained per dataset and per depth on 1,394 in-domain windows as a small distillation, and aligned TimesFM-3 at L0 with +31.3% error beats intermediate layers L2 to L17, suggesting the adapter is learning a shallow forecaster from input embeddings rather than exploiting intermediate states.

Questions for authors

How do the tunnel entrances change if the 5% tolerance is measured relative to the native full-model performance instead of the final-block linear probe, and how many of the 42 pairs still exhibit a tunnel?

Does an adapter trained on 1,394 windows from one dataset or pooled data transfer to unseen datasets, and can you provide leave-one-dataset-out results at a few depths to show whether a single truncated checkpoint is deployable?

Why does aligned TimesFM-3 at L0 with +31.3% error outperform most intermediate depths from L2 to L17, and is the adapter essentially learning a linear forecaster directly from the input patch embedding?

2 comments

Q1. How many tunnels remain when the 5% tolerance is measured against native-model performance?

We thank the reviewer for suggesting this comparison. We repeated the calculation using the native full model's validation loss as the reference, keeping the readouts and sustained criterion unchanged. Under this definition, 6/42 model-dataset pairs satisfy the 5% criterion, all with pre-final entrances.

This alternative definition measures something different from our forecasting tunnel: it requires the final-block linear readout itself to match the native model within 5%. However, the independently trained readouts have median validation losses 17.7%, 20.3%, and 12.7% above the native models for Chronos-2, TimesFM-3, and TiRex, respectively. This gap does not necessarily indicate a lack of recoverable forecast information. For example, TimesFM-3's native forecasting head is itself linear, yet performs better than our independently trained linear readout.

Our original definition instead compares readouts trained under the same procedure across depths, measuring when additional blocks stop improving recoverable forecasting performance. Under this definition, 40/42 pairs have pre-final tunnels, with a median test-loss increase of only +1.3% at the validation-selected entrance relative to the final-block readout.

We agree that performance relative to the native model is important for assessing practical truncation. We therefore also examine native-pathway compatibility through alignment. Using the full model as the reference, the aligned pathway reaches within 5% of native performance before the final block in 33/42 pairs: 13/14 for Chronos-2, 9/14 for TiRex, and 11/14 for TimesFM-3 with the corrected adapter. Together, the results show that early forecast recoverability can often be translated into near-native forecasting performance through lightweight alignment.

Q2. Do single-source or pooled adapters transfer to unseen datasets, and is a truncated checkpoint deployable?

We thank the reviewer for raising this question. We evaluated leave-one-dataset-out transfer for Chronos-2 at L4, L6, and L9, corresponding to speedups of 2.43×, 1.78×, and 1.28× (Appendix B.4.2). For each target, we train an adapter on the other 13 datasets, or 12 when the two Electricity datasets are excluded together.

Median test MASE increase over the full model across 14 datasets:

MethodL4 (2.43×)L6 (1.78×)L9 (1.28×)
Hard cut+37.0%+20.1%+17.1%
Transferred adapter+12.7%+9.0%+4.0%
Dataset-specific adapter+3.1%+2.3%+2.2%
Transferred within 5%4/144/149/14

Transfer improves over the hard cut on 13/14 datasets at each depth. At L9, the median gap to dataset-specific calibration is only 1.7 percentage points, increasing to 7.1 at L4.

We also tested single-source transfer at L6 on four targets. Across 12 source–target combinations, MASE degradation ranges from +5.0% to +50.7%, averaging +19.7%, compared with +19.8% for the hard cut. Pooling 4, 8, or 13 sources reduces the mean to +12.7%, +13.2%, and +12.8%, respectively. Since the number of training windows also increases, we cannot isolate the effect of source diversity.

These experiments show transfer to unseen datasets, while evaluating a single fixed adapter across multiple unseen targets remains a useful next step. When target calibration is available, it requires only 1,394 unlabeled windows, approximately one second for the full-model pass, and one minute of fitting on an A100.

Overall, pooled adapters can provide useful acceleration without target-specific fitting, particularly at mild truncation, while inexpensive calibration enables better accuracy at more aggressive cuts.

Q3. Why does TimesFM-3 perform better at L0, and is it learning a linear forecaster?

We thank the reviewer for pointing this out. TimesFM-3's linear head makes the affine adapter an affine forecaster on the L0 patch embedding. We investigated this behavior and revised the regularization to shrink the adapter's linear map toward zero rather than the identity, keeping the data and validation protocol unchanged (Appendix B.3.1).

The revised median test MASE increases at L5/L10/L15 are +10.6%/+11.3%/+8.5%, compared with +69.8%/+67.3%/+47.9% originally. L0 is no longer unusually favorable, and the corrected adapter outperforms the supervised readout on 10-12 of 14 datasets at depths from L1. The correction was selected post hoc, as disclosed in the manuscript. All 98 physical truncation checks agree with offline mean test MASE within 4.5 × 10⁻⁵ relative error. We have included these results and updated the corresponding figures in the revised version.

Thanks for the response. I’ve decided to keep my positive rating.

Reviewer_2p99OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

paper investigates whether intermediate representations in time-series foundation models contain sufficient information for forecasting. It introduce a forecasting tunnel based on layerwise readouts and use affine alignment to enable early truncation. Experiments on Chronos-2, TimesFM-3, and TiRex across 14 datasets show early forecast recoverability and architecture dependent inference speedups, reaching 2.43x for Chronos-2 with a 3.1% median MASE increase.

Critical strengths

The distinction between forecast recoverability and compatibility with the pretrained prediction head is insightful. The experiment cover 3 different architectures, include representation geometry analysis, and measure actual inference speedups rather than relying on theoretical complexity. The results for Chronos-2 demonstrate a useful practical tradeoff.

Critical weaknesses

main concern is that the proposed affine adapters are fitted separately for each dataset, limiting the claim of general purpose foundation model acceleration. Furthermore, several evaluation datasets appear in the documented pretraining corpora, making it difficult to establish generalization to genuinely unseen distributions. methodological novelty is moderate, as layerwise probing, representation alignment, and model truncation are established techniques. The gains are also architecture dependent: TimesFM-3 shows substantial accuracy degradation despite truncation, while the separately pretrained Chronos-2-small offers a better accuracy speed tradeoff at comparable latency. The forecasting tunnel does not yet provide a reliable practical rule for selecting truncation depth.

Questions for authors

Can the authors clarify whether an adapter trained on one dataset transfers to unseen datasets? Can the forecasting tunnel reliably select a truncation depth without separately evaluating multiple truncated models? How do the conclusions change when considering only model dataset pairs documented as held out from pretraining?

2 comments

Q1. Can the authors clarify whether an adapter trained on one dataset transfers to unseen datasets?

We thank the reviewer for raising this question. In the revised version, we have tested transfer from both single and multiple source datasets for Chronos-2. In particular, we ran leave-one-dataset-out transfer at L4, L6, and L9, corresponding to measured speedups of 2.43×, 1.78×, and 1.28× (Appendix B.4.2). For each held-out target, we train an adapter on the other 13 datasets, or 12 when the two Electricity datasets are excluded together because they share meters.

Median test MASE increase over the full model across 14 datasets:

MethodL4 (2.43×)L6 (1.78×)L9 (1.28×)
Hard cut+37.0%+20.1%+17.1%
Transferred adapter+12.7%+9.0%+4.0%
Dataset-specific adapter+3.1%+2.3%+2.2%

The transferred adapter improves over the hard cut on 13/14 datasets at each depth, recovering a median 70–80% of the hard-cut gap. Dataset-specific adapters remain better, by a median 7.1, 5.1, and 1.7 percentage points at L4, L6, and L9.

To address transfer from a single dataset directly, we also ran an additional experiment at L6 on four targets: LOOP Seattle, London Smart Meters, BOOM, and M5. Across 12 single-source transfer configurations, MASE degradation ranges from +5.0% to +50.7%, with a mean of +19.7%, compared with +19.8% for the hard cut. Pooling 4, 8, or 13 source datasets reduces the mean degradation to +12.7%, +13.2%, and +12.8%, respectively. Thus, pooled training performs substantially better than single-source transfer in this experiment. However, the number of training windows also grows with the number of sources, so we cannot isolate the effect of source diversity.

Overall, our experiments show that adapters can transfer to unseen datasets without target-specific fitting, particularly when trained on multiple sources and at mild truncation (+4.0% at 1.28×). At more aggressive cuts, calibration on unlabeled target windows remains more accurate. We have added these results and updated the corresponding limitations in Appendix B.4.2.

Q2. Can the forecasting tunnel reliably select a truncation depth without separately evaluating multiple truncated models?

While the tunnel entrance does not necessarily correspond to the best truncation depth, we find that it provides useful information about which depths are more likely to tolerate truncation.

We compare datasets at the same cut depth, separating those where the cut lies inside the validation-selected 5% tunnel from those where it lies before the tunnel (Section 4.4, Appendix B.4.1). For Chronos-2 at L4, the median test MASE increase is +2.2% for the 8 datasets where L4 lies inside the tunnel, compared with +14.2% for the other 6. Similarly, for TiRex at L6, the increases are +4.4% and +12.3%. We observe lower median degradation inside the tunnel at all compared depths for Chronos-2, TiRex, and the corrected TimesFM-3 pathway.

The tunnel provides information about truncation tolerance, but does not guarantee that a particular cut will preserve full-model accuracy.

Q3. How do the conclusions change when considering only model-dataset pairs documented as held out from pretraining?

We thank the reviewer for suggesting this comparison - we restricted the analysis to the 9 of 42 model-dataset pairs documented as held out from the relevant real-data pretraining corpora: 4 for Chronos-2, 3 for TimesFM-3, and 2 for TiRex (Appendix A.3). We do not treat datasets that are simply not listed in the pretraining documentation as confirmed held out.

The main findings remain similar: the 5% forecasting tunnel begins before the final block in 8/9 held-out pairs, with TiRex on Monash Traffic being the exception. At the validation-selected tunnel entrance, test readout loss increases by a median of only +1.3% relative to the final-block readout.

Alignment also improves truncation performance in this subset. At half the model depth, median test MASE degradation is +10.6% for the hard cut versus +1.1% with alignment for Chronos-2; 89× versus +4.5% for TimesFM-3; and +138% versus +15.6% for TiRex.

These results are consistent with our conclusions on the full benchmark, but the held-out subsets are small. We therefore cannot establish broad generalization from these nine pairs alone. Nevertheless, both early forecast recoverability and the benefit of alignment are observed on datasets documented as excluded from pretraining.

The added transfer experiments and documented heldout analysis substantially address my concerns, while clarifying the accuracy cost of transfer and the tunnel’s limited role in selecting truncation depth. Methodological novelty and practical gains remain moderate and architecture dependent, but the evidence supports retaining my weak-acceptance recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.