acceptodds
Under review as a conference paper at ICLR 2027

PTEF: Efficient Long-term Time Series Forecasting via Layer Pruned Text Embeddings

Abstract

Time series forecasting plays a crucial role across diverse industries. Existing LLM-based approaches suffer from large model sizes and significant computational costs that limit their applicability in resource-constrained environments. We propose Pruned Text Embeddings for Time Series Forecasting (PTEF), a lightweight architecture using frozen pruned text embedding models to encode numerical sequences, exploiting implicit numerical and structural patterns acquired from large text corpora, as in zero-shot text-to-number forecasting. PTEF applies channel-independence to treat each input channel as a standalone sequence, encodes it via a frozen pruned embedding model in a single forward pass, and passes the resulting embedding vectors through a compact CNN-based forecasting head. This design decouples representation learning from forecasting, reducing trainable parameters to just 0.094M. In few-shot settings using only 10% of training data, PTEF achieves the best average ranking across long-term forecasting benchmarks, outperforming larger LLM-based models including Time-LLM and LLM4TS, while remaining competitive in full-shot settings despite its significantly smaller capacity. Using only a small embedding models such as all-MiniLM-L6-v2, PTEF demonstrates strong generalisation across multiple embedding architectures. Ablation studies confirm that pre-trained text embeddings are the primary driver of PTEF's performance, validating the effectiveness of cross-modal transfer from text to numerical time series.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series, Forecasting, Text Embedding, LLM, Pruning, Efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_a1w1OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes PTEF, a lightweight framework for long-term time series forecasting using pretrained text embeddings. The method converts numerical time series into text, processes them using a frozen, layer-pruned text embedding model, and combines the resulting embeddings with the original time series through residual connections. A small CNN and linear prediction head are then used for forecasting. The authors evaluate PTEF on five benchmarks under few-shot and full-shot settings, reporting competitive few-shot performance with relatively few trainable parameters. Additional experiments investigate the effects of text embeddings, layer pruning, and different model components.

Critical strengths

The paper is generally clear and easy to follow.

The proposed framework is relatively simple and achieves competitive few-shot forecasting results with a small number of trainable parameters.

The authors provide several ablation studies on text embeddings, layer pruning, and forecasting heads.

Critical weaknesses
  1. Limited novelty. The main idea is highly related to LETS-C (ACL 2025), which already uses pretrained text embeddings for time series modeling. Most other components, including RevIN, channel independence, layer pruning, and CNN-based prediction, are also existing techniques. The paper does not clearly establish a substantial methodological contribution beyond adapting these ideas to forecasting.

  2. Potential input truncation. The look-back window is 336, while the selected MiniLM embedding model supports only 256 tokens. Since numerical values may require multiple tokens, a substantial portion of the input could be truncated. The paper does not adequately investigate this issue or clarify how much historical information the embedding model actually receives.

  3. Inconsistent parameter reporting. The claimed 0.094M trainable parameters appear to correspond to the shortest prediction horizon. According to the output head formulation, the parameter count should increase significantly for longer horizons, potentially exceeding 1.5M at horizon 720.

  4. Insufficient experiments. The evaluation lacks stronger and more recent forecasting baselines, such as iTransformer and TimeMixer. The experiments are also limited to Weather and four closely related ETT datasets. Moreover, the full-shot results are not particularly competitive, with PTEF performing worse than LLM4TS and Time-LLM in average ranking.

  5. Incomplete ablation studies. Although adding text embeddings improves performance, the experiments do not establish that the improvements come from pretrained language knowledge. Comparisons with randomly initialized encoders or random frozen embeddings are missing. The no-embedding baseline also changes the timestamp encoding, making the comparison less controlled.

  6. Insufficient efficiency and statistical analysis. The reported fast inference time mainly reflects cached embeddings rather than complete end-to-end inference for new inputs. Direct latency comparisons with lightweight forecasting models are missing. In addition, most results rely on a fixed seed, and the limited statistical analysis does not adequately support the small improvements reported in some few-shot settings.

Questions for authors

see weaknesses

3 comments

Dear reviewer, Thank you for the feedback.

-The number of trainable parameters for the different horizons:

HorizonPTEF trainable parametersLLM4TS trainable parameters
960.094 M3.4 M
1920.22 M6.19 M
3360.49 M10.84 M
7201.59 M23.22 M
  • We performed the experiment with random embedding while keeping everything else unchanged to check how text embedding performs against random embedding (table below results for 3 datasets).
DatasetHorizonRandom Emb (MSE/MAE)PTEF (MSE/MAE)
Weather960.161 / 0.2110.157 / 0.207
Weather1920.205 / 0.2490.201 / 0.245
Weather3360.255 / 0.2870.252 / 0.285
Weather7200.329 / 0.3390.326 / 0.337
ETTh1960.426 / 0.4360.418 / 0.427
ETTh11920.509 / 0.4860.505 / 0.483
ETTh13360.533 / 0.5010.530 / 0.498
ETTh17200.685 / 0.5840.640 / 0.564
ETTh2960.319 / 0.3660.316 / 0.365
ETTh21920.395 / 0.4070.388 / 0.403
ETTh23360.407 / 0.4270.405 / 0.431
ETTh27200.458 / 0.4700.452 / 0.466
  • Due the limited sequence length of the encoder, part of the input is truncated (2/3), it's one of the limitations of these approach as it consumes a considerable amount of tokens.

  • Inference time per batch = 128 on ETTh2, Tout = 96: 0.084 second for the whole pipeline jo cached embeddings. LLM4TS 0.024 seconds.

  • Confidence Interval analysis for the ETTm1 | Horizon | MSE Avg. (95% CI) | MAE Avg. (95% CI) | |----------|----------|----------| | 96 | 0.344 [0.343, 0.344] | 0.375 [0.374, 0.376] | | 192 | 0.371 [0.369, 0.374] | 0.391 [0.389, 0.392] | | 336 | 0.404 [0.402, 0.407] | 0.410 [0.409, 0.411] | | 720 | 0.464 [0.461, 0.468] | 0.447 [0.445, 0.449] |

Thank you for the additional experiments and clarifications.

The parameter counts are now clarified, and the random embedding experiment provides useful additional evidence. However, the improvements over random embeddings are relatively small in most settings, so the benefit of pretrained language representations remains limited. More importantly, the reported end-to-end inference time (0.084s) is higher than that of LLM4TS (0.024s), which weakens the claimed efficiency advantage.

Several other concerns also remain unresolved. The authors acknowledge that approximately two-thirds of the input is truncated, but do not investigate its impact on forecasting performance. The limited novelty relative to LETS-C and the lack of comparisons with recent forecasting baselines have not been adequately addressed. The additional confidence intervals are also insufficient to establish the significance of the reported improvements.

Overall, while the response addresses some technical questions, my main concerns about novelty, experimental validation, and practical efficiency remain. I therefore maintain my original score of 2 .

Thank you for the feedback, apologies for not reply to your other comments we were running additional experiments.

  1. We agree PTEF builds directly on LETS-C, which established that frozen text embeddings fused with raw values via addition are useful time-series representations; we do not claim the embedding-plus-fusion idea as new, and will state this explicitly. Our contribution is to test whether this transfer survives the move to long-term forecasting, and how much of the encoder is needed. Three points go beyond LETS-C:
  • Forecasting, not labelling - Forecasting maps each channel to a full future sequence rather than a class label. This requires per-channel encoding, horizon-dependent output heads, and handling a 336-step window under a 256-token budget (we now report this truncation and its effect, see question 2).
  • How much encoder is needed - Our layer-retention study shows that keeping 1 of 6 MiniLM layers retains most of full-encoder accuracy at 17% of the MACs validating that numerical tokens requires less processing depth. LETS-C uses complete encoders and does not study this.
  • Data-scarce forecasting - With 10% training data, PTEF ranks first against LLM-based forecasters (Time-LLM, LLM4TS) and numerical baselines.

We will revise the introduction and related work to position PTEF as an empirical study of efficient text-embedding transfer for forecasting that extends LETS-C, rather than as a new representation method.

  • We did run further experiments to include the few shot for TimeMixer and iTransformer, we will revise the paper accordingly and include these 2 additional baselines + random embedding experiment (question 5 table). | Dataset | PTEF (ours) (MSE/MAE) | TimeMixer (MSE/MAE) | iTransformer (MSE/MAE) | |----------|----------|----------|----------| | Weather | 0.234 / 0.269 | 0.252 / 0.283 | 0.270 / 0.302 | | ETTh1 | 0.523 / 0.493 | 0.757 / 0.602 | 0.718 / 0.578 | | ETTh2 | 0.390 / 0.416 | 0.453 / 0.449 | 0.434 / 0.447 | | ETTm1 | 0.396 / 0.405 | 0.561 / 0.496 | 0.507 / 0.474 | | ETTm2 | 0.277 / 0.325 | 0.306 / 0.347 | 0.295 / 0.343 | | Avg. Rank | 2.000 / 2.200 | 9.000 / 9.200 | 8.200 / 8.400 |
  • We acknowledge that PTEF is not competitive with LLM4TS and Time-LLM in the full-shot setting, its primary claim is data efficiency in few-shot forecasting.
  • Inference time on the same hardware per batch = 128 on ETTh2, Tout = 96 including the computation of the embedding of a new input: PTEF’s latency is 0.084 s, versus 0.012 s for DLinear, 0.017 s for PatchTST, and 0.024 s for LLM4TS. PTEF is therefore slower because of text-embedding extraction.

  • We experimented with 4 seeds below the full statistical analysis and it will be included in the revised version of the paper. MSE: | Dataset | MSE Average | 95% Confidence Interval | |----------|----------:|----------| | Weather | 0.2340 | [0.2336, 0.2343] | | ETTh1 | 0.5267 | [0.5167, 0.5367] | | ETTh2 | 0.3880 | [0.3758, 0.4002] | | ETTm1 | 0.3955 | [0.3941, 0.3968] | | ETTm2 | 0.2770 | [0.2764, 0.2776] |

MAE:

DatasetMAE Average95% Confidence Interval
Weather0.2685[0.2680, 0.2690]
ETTh10.4958[0.4908, 0.5008]
ETTh20.4135[0.4055, 0.4216]
ETTm10.4055[0.4049, 0.4061]
ETTm20.3249[0.3245, 0.3254]
Reviewer_2fuYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PTEF asks whether a heavily pruned text encoder can provide useful time-series features at lower cost. It serializes normalized observations and a timestamp, retains one MiniLM layer, combines its embedding with numerical input, and trains a small CNN/linear forecasting head. Experiments examine forecasting, few-shot learning, pruning, and cost.

Critical strengths
  1. The question is worthwhile and practical. Testing how much of a pretrained text encoder forecasting actually needs can yield useful knowledge even if the final model does not lead every accuracy benchmark. The layer-retention study directly investigates this question.
  2. The paper provides evidence of feasibility. Forecasting and few-shot results show that a shallow frozen representation can work with a small head. It also reports full-model MAC estimates and acknowledges that simple numerical models can be cheaper.
Critical weaknesses
  1. The main practical claim requires a clearer accuracy–cost comparison for the actual pruned configuration. The no-embedding ablation uses the full encoder and changes timestamp representation and feature combination, so it does not isolate the value added by the deployed one-layer embedding. The pruning study supports retaining fewer layers, but does not settle whether those features improve forecasting enough to justify their cost over a compact numerical model.
  2. The practical inference advantage needs clearer comparative validation. The within-model caching comparison and cross-model MAC comparison do not directly establish an end-to-end latency advantage over competing forecasters. For newly arriving windows requiring fresh encoding, the relevant comparison should include formatting, tokenization, embedding, and prediction under matched hardware and batch sizes.
Questions for authors
  1. Does the benefit of the deployed one-layer text embedding remain when the numerical path, timestamp information, fusion dimensions, and forecasting head are held fixed? Which result isolates this contribution?
  2. Using fresh inputs without cached embeddings, how does the complete pipeline’s latency compare with LLM4TS and a compact numerical forecaster under matched hardware, batch size, and forecast horizon? Under what conditions can embeddings be reused when the input window or timestamp changes?
2 comments

Dear Reviewer,

Thank you for the constructive reviews.

  1. We performed the experiment with random embedding while keeping everything else unchanged to check how text embedding performs against random embedding (table below results for 3 datasets).
DatasetHorizonRandom Emb (MSE/MAE)PTEF (MSE/MAE)
Weather960.161 / 0.2110.157 / 0.207
Weather1920.205 / 0.2490.201 / 0.245
Weather3360.255 / 0.2870.252 / 0.285
Weather7200.329 / 0.3390.326 / 0.337
ETTh1960.426 / 0.4360.418 / 0.427
ETTh11920.509 / 0.4860.505 / 0.483
ETTh13360.533 / 0.5010.530 / 0.498
ETTh17200.685 / 0.5840.640 / 0.564
ETTh2960.319 / 0.3660.316 / 0.365
ETTh21920.395 / 0.4070.388 / 0.403
ETTh23360.407 / 0.4270.405 / 0.431
ETTh27200.458 / 0.4700.452 / 0.466
  • Inference time per batch = 128 on ETTh2, Tout = 96: PTEF 0.084 second for the whole pipeline. LLM4TS 0.024 second.
  • If the input window or timestamp changes the embedding must be recalculated.

Thanks for the authors' efforts. The experimental results for random embedding is now sufficient. However, the inference time is higher than that of the LLM4TS. Therefore, I decided to maintain current score.

Reviewer_UXesOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PTEF, a lightweight time series forecasting model that uses frozen and pruned pretrained text embeddings combined with a CNN-based forecasting head. The main objective is to reduce the computational cost and number of trainable parameters while maintaining forecasting accuracy. The authors evaluate the proposed method on five benchmark datasets in few-shot and full-shot settings and report competitive results, particularly when using only 10% of the training data.

Critical strengths
  1. Computational efficiency:The proposed method uses a lightweight architecture with a small number of trainable parameters compared to larger LLM-based forecasting models.
  2. Few-shot forecasting performance:The proposed method achieves competitive results using only 10% of the training data across five benchmark datasets. These results show the potential of using pretrained text embeddings for time series forecasting when limited training data is available.
Critical weaknesses
  1. Limited novelty: LETS also uses pretrained text embeddings and a CNN-based architecture for time series analysis. Although PTEF extends this approach to forecasting and applies layer pruning, the main methodological novelty is not sufficiently clear. This limits the contribution of the paper compared to existing works.

  2. Insufficient experimental validation: The paper does not compare with several recent and efficient forecasting models. In addition, the ablation studies do not clearly show whether the improvements come from pretrained text embeddings or simply from additional feature representations. These missing comparisons make it difficult to fully support the claimed advantages of the proposed method.

Questions for authors

1.Evidence for the main claim: The embedding ablation removes the encoder and also changes timestamp handling, while using the full encoder rather than the proposed pruned model. What evidence shows that the improvement comes from language pretraining? Could you compare the pruned pretrained encoder with the same frozen, randomly initialized encoder while keeping all other components unchanged? 2. Clarification of the actual input: The input contains 336 numerical values, but Table 11 gives MiniLM a limit of 256 tokens. How much of each sequence actually reaches the text encoder, and which part is kept after truncation? 3. Justification of the model design: After mean pooling, embedding coordinates represent latent features rather than time steps. What justifies adding them directly to the raw sequence and applying local convolution across the resulting vector? Would the improvement remain after a fixed permutation of the embedding coordinates, with the forecasting head retrained? 4. Computational efficiency and baseline comparisons: Could the authors report the number of trainable parameters for different prediction horizons and the end-to-end inference time, including embedding computation for new inputs? It would be helpful if the authors could include more recent papers in this domain, particularly those published from 2025 onward 6- lastly, the paper "Are Language Models Actually Useful for Time Series Forecasting?" (NeurIPS 2024) shows that pretrained language models do not necessarily improve forecasting performance compared to simpler models. Although PTEF uses a smaller text embedding model, how do the authors address these findings? Have they compared the pretrained embeddings with randomly initialized embeddings to demonstrate the actual benefit of text pretraining?

2 comments

Dear Reviewer,

Thank you for the constructive reviews.

  1. We performed the experiment with random embedding while keeping everything else unchanged to check how text embedding performs against random embedding (table below results for 3 datasets), results shows text embedding performs better than random embedding.
DatasetHorizonRandom Emb (MSE/MAE)PTEF (MSE/MAE)
Weather960.161 / 0.2110.157 / 0.207
Weather1920.205 / 0.2490.201 / 0.245
Weather3360.255 / 0.2870.252 / 0.285
Weather7200.329 / 0.3390.326 / 0.337
ETTh1960.426 / 0.4360.418 / 0.427
ETTh11920.509 / 0.4860.505 / 0.483
ETTh13360.533 / 0.5010.530 / 0.498
ETTh17200.685 / 0.5840.640 / 0.564
ETTh2960.319 / 0.3660.316 / 0.365
ETTh21920.395 / 0.4070.388 / 0.403
ETTh23360.407 / 0.4270.405 / 0.431
ETTh27200.458 / 0.4700.452 / 0.466
  1. Due the limited sequence length of the encoder, part of the input is truncated (2/3), it's one of the limitations of this approach as it consumes a considerable amount of tokens.
  2. We tested the injection of the raw sequences using 2 approaches:
  • Concatenation in the channel dimension and feeding it to the convolution layer.
  • Point-wise addition

Point-wise addition was better and less expensive, other methods can be explored such cross attention.

  • We performed the experiment with randomly permuted embedding coordinates, we found a decrease in performance for some datasets, table below. | Dataset | Permuted Embed Coord (MSE/MAE) | PTEF (ours) (MSE/MAE) | |----------|----------|----------| | Weather | 0.235 / 0.269 | 0.234 / 0.269 | | ETTh1 | 0.538 / 0.500 | 0.523 / 0.493 | | ETTh2 | 0.392 / 0.416 | 0.390 / 0.416 | | ETTm1 | 0.396 / 0.406 | 0.396 / 0.405 | | ETTm2 | 0.277 / 0.324 | 0.277 / 0.325 | | Avg. Rank | 3.200 / 3.000 | 2.200 / 2.600 |
  1. Below the number of trainable parameters for the different horizons:
HorizonPTEF trainable parametersLLM4TS trainable parameters
960.094 M3.4 M
1920.22 M6.19 M
3360.49 M10.84 M
7201.59 M23.22 M
  • Inference time on the same hardware per batch = 128 on ETTh2, Tout = 96 including the computation of the embedding of a new input: PTEF’s latency is 0.084 s and 0.024 s for LLM4TS. PTEF is therefore slower because of text-embedding extraction.
  • The findings of "Are Language Models Actually Useful for Time Series Forecasting?" (NeurIPS 2024): we replaced the encoder with randomly initialized embeddings (Question 1).

Apologies for not replying to all the comments before, as we were working on additional experiments.

Weaknesses:

  1. We agree PTEF builds directly on LETS-C, which established that frozen text embeddings fused with raw values via addition are useful time-series representations; we do not claim the embedding-plus-fusion idea as new, and will state this explicitly. Our contribution is to test whether this transfer survives the move to long-term forecasting, and how much of the encoder is needed. Three points go beyond LETS-C:
  • Forecasting, not labelling - Forecasting maps each channel to a full future sequence rather than a class label. This requires per-channel encoding, horizon-dependent output heads, and handling a 336-step window under a 256-token budget (we now report this truncation and its effect, see question 2).
  • How much encoder is needed - Our layer-retention study shows that keeping 1 of 6 MiniLM layers retains most of full-encoder accuracy at 17% of the MACs validating that numerical tokens requires less processing depth. LETS-C uses complete encoders and does not study this.
  • Data-scarce forecasting - With 10% training data, PTEF ranks first against LLM-based forecasters (Time-LLM, LLM4TS) and numerical baselines.

We will revise the introduction and related work to position PTEF as an empirical study of efficient text-embedding transfer for forecasting that extends LETS-C, rather than as a new representation method.

  1. We did run further experiments to include the few shot for TimeMixer and iTransformer, we will revise the paper accordingly and include these 2 additional baselines (below table) + random embedding experiment (question 1 table). | Dataset | PTEF (ours) (MSE/MAE) | TimeMixer (MSE/MAE) | iTransformer (MSE/MAE) | |----------|----------|----------|----------| | Weather | 0.234 / 0.269 | 0.252 / 0.283 | 0.270 / 0.302 | | ETTh1 | 0.523 / 0.493 | 0.757 / 0.602 | 0.718 / 0.578 | | ETTh2 | 0.390 / 0.416 | 0.453 / 0.449 | 0.434 / 0.447 | | ETTm1 | 0.396 / 0.405 | 0.561 / 0.496 | 0.507 / 0.474 | | ETTm2 | 0.277 / 0.325 | 0.306 / 0.347 | 0.295 / 0.343 | | Avg. Rank | 2.000 / 2.200 | 9.000 / 9.200 | 8.200 / 8.400 |

Questions:

  1. We performed the experiment with random embedding while keeping everything else unchanged to check how text embedding performs against random embedding (table below results for 3 datasets), results shows text embedding performs better than random embedding.
DatasetHorizonRandom Emb (MSE/MAE)PTEF (MSE/MAE)
Weather960.161 / 0.2110.157 / 0.207
Weather1920.205 / 0.2490.201 / 0.245
Weather3360.255 / 0.2870.252 / 0.285
Weather7200.329 / 0.3390.326 / 0.337
ETTh1960.426 / 0.4360.418 / 0.427
ETTh11920.509 / 0.4860.505 / 0.483
ETTh13360.533 / 0.5010.530 / 0.498
ETTh17200.685 / 0.5840.640 / 0.564
ETTh2960.319 / 0.3660.316 / 0.365
ETTh21920.395 / 0.4070.388 / 0.403
ETTh23360.407 / 0.4270.405 / 0.431
ETTh27200.458 / 0.4700.452 / 0.466
  1. Due the limited sequence length of the encoder, part of the input is truncated (2/3), it's one of the limitations of this approach as it consumes a considerable amount of tokens.
  2. We tested the injection of the raw sequences using 2 approaches:
  • Concatenation in the channel dimension and feeding it to the convolution layer.
  • Point-wise addition

Point-wise addition was better and less expensive, other methods can be explored such cross attention.

  • We performed the experiment with randomly permuted embedding coordinates, we found a decrease in performance for some datasets, table below. | Dataset | Permuted Embed Coord (MSE/MAE) | PTEF (ours) (MSE/MAE) | |----------|----------|----------| | Weather | 0.235 / 0.269 | 0.234 / 0.269 | | ETTh1 | 0.538 / 0.500 | 0.523 / 0.493 | | ETTh2 | 0.392 / 0.416 | 0.390 / 0.416 | | ETTm1 | 0.396 / 0.406 | 0.396 / 0.405 | | ETTm2 | 0.277 / 0.324 | 0.277 / 0.325 | | Avg. Rank | 3.200 / 3.000 | 2.200 / 2.600 |
  1. Below the number of trainable parameters for the different horizons:
HorizonPTEF trainable parametersLLM4TS trainable parameters
960.094 M3.4 M
1920.22 M6.19 M
3360.49 M10.84 M
7201.59 M23.22 M
  • Inference time on the same hardware per batch = 128 on ETTh2, Tout = 96 including the computation of the embedding of a new input: PTEF’s latency is 0.084 s, versus 0.012 s for DLinear, 0.017 s for PatchTST, and 0.024 s for LLM4TS. PTEF is therefore slower because of text-embedding extraction.
  • The findings of "Are Language Models Actually Useful for Time Series Forecasting?" (NeurIPS 2024): we replaced the encoder with randomly initialized embeddings (Question 1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.