RAC-Net: Patch-based Multivariate Time Series Forecasting via Codebook Routing and Small Language Model Distillation

The computational overhead of large pre-trained language models and the resource demands of real-world multivariate time-series forecasting (MTSF) can be addressed using the knowledge distillation (KD) framework. Existing approaches provide limited investigation of cached semantic supervision for lightweight forecasting without language-model inference. To address these limitations, the study proposes regime-aware codebook (RAC-Net), a semantic representation distillation framework that uses a small language model (SLM) and a lightweight patch-based Transformer network for MTSF. RAC-Net first leverages a pre-trained small language model (SLM) prompted with statistical trends to perform semantic reasoning and produce high-fidelity latent representations. A lightweight student backbone model is developed with a low-rank codebook routing mechanism. The directional cosine alignment loss function aligns the student routing representations with the latent representations of the SLM teacher model. After training, the teacher model is decoupled, leaving the lightweight trained student model for inference. Extensive evaluation across seven publicly available datasets demonstrates that RAC-Net achieves competitive forecasting performance with inference latencies of 2.9–13.67 per multivariate sample. On a high-dimensional Electricity dataset at a long horizon of 720, RAC-Net achieves 23.17% and 16% relative improvements in mean squared error compared to PatchTST and iTransformer, respectively, with 0.87G inference MAC operations.