PTEF: Efficient Long-term Time Series Forecasting via Layer Pruned Text Embeddings

Time series forecasting plays a crucial role across diverse industries. Existing LLM-based approaches suffer from large model sizes and significant computational costs that limit their applicability in resource-constrained environments. We propose Pruned Text Embeddings for Time Series Forecasting (PTEF), a lightweight architecture using frozen pruned text embedding models to encode numerical sequences, exploiting implicit numerical and structural patterns acquired from large text corpora, as in zero-shot text-to-number forecasting. PTEF applies channel-independence to treat each input channel as a standalone sequence, encodes it via a frozen pruned embedding model in a single forward pass, and passes the resulting embedding vectors through a compact CNN-based forecasting head. This design decouples representation learning from forecasting, reducing trainable parameters to just 0.094M. In few-shot settings using only 10% of training data, PTEF achieves the best average ranking across long-term forecasting benchmarks, outperforming larger LLM-based models including Time-LLM and LLM4TS, while remaining competitive in full-shot settings despite its significantly smaller capacity. Using only a small embedding models such as all-MiniLM-L6-v2, PTEF demonstrates strong generalisation across multiple embedding architectures. Ablation studies confirm that pre-trained text embeddings are the primary driver of PTEF's performance, validating the effectiveness of cross-modal transfer from text to numerical time series.