acceptodds
Under review as a conference paper at ICLR 2027

RAC-Net: Patch-based Multivariate Time Series Forecasting via Codebook Routing and Small Language Model Distillation

Abstract

The computational overhead of large pre-trained language models and the resource demands of real-world multivariate time-series forecasting (MTSF) can be addressed using the knowledge distillation (KD) framework. Existing approaches provide limited investigation of cached semantic supervision for lightweight forecasting without language-model inference. To address these limitations, the study proposes regime-aware codebook (RAC-Net), a semantic representation distillation framework that uses a small language model (SLM) and a lightweight patch-based Transformer network for MTSF. RAC-Net first leverages a pre-trained small language model (SLM) prompted with statistical trends to perform semantic reasoning and produce high-fidelity latent representations. A lightweight student backbone model is developed with a low-rank codebook routing mechanism. The directional cosine alignment loss function aligns the student routing representations with the latent representations of the SLM teacher model. After training, the teacher model is decoupled, leaving the lightweight trained student model for inference. Extensive evaluation across seven publicly available datasets demonstrates that RAC-Net achieves competitive forecasting performance with inference latencies of 2.9–13.67 per multivariate sample. On a high-dimensional Electricity dataset at a long horizon of 720, RAC-Net achieves 23.17% and 16% relative improvements in mean squared error compared to PatchTST and iTransformer, respectively, with 0.87G inference MAC operations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
time series forecasting, knowledge distillation, small language model, multivariate forecasting, transformer, patch-transformer, cross channel attention

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aLxiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RAC-Net, a knowledge distillation framework for multivariate long-term time-series forecasting in which a small language model (SLM) is used only offline. Standardised training windows of length L = 96 from all channels are clustered with MiniBatch k-means into M = 128 centroids. For each centroid, a frozen Qwen-2.5-0.5B receives a prompt with the trend class, the least-squares trend slope, the dominant FFT period and the standard deviation, and generates up to 48 rationale tokens with greedy decoding. The top-layer hidden states are mean-pooled, mapped to 32 dimensions by a non-trainable orthogonal projection, normalised onto the unit hypersphere and cached. During training, the teacher target for each window is the cached vector of its nearest centroid, as defined in Eq. 13.

The student is a patch-based Transformer with channel embeddings and cross-channel attention at each temporal patch position. It adds a dynamic low-rank mixture codebook, in which a router over K = 16 rank-4 adapters produces a low-rank perturbation of the pooled representation. This perturbation is expanded across patches and added through a non-linear gated residual. The loss is MSE + 2.0 × MAE plus a directional cosine alignment term and an entropy-based routing diversity term, as given in Eqs. 26 to 30 and Table 5. The SLM is not used during student training or inference.

The experiments use seven benchmarks, ETTh1, ETTh2, ETTm1, ETTm2, Weather, Electricity and Traffic, with L = 96 and H ∈ {96, 192, 336, 720}. Tables 1 and 12 compare RAC-Net with nine baselines, using numbers taken from the published papers. Table 2 ablates distillation at H = 720 with no-distillation, MSE-only, MLP-statistics and random target vector variants, and Table 11 repeats this over four seeds. Table 3 and Fig. 2 compare parameters and MACs on Electricity and ETTh1. The appendices cover the offline centroid budget, codebook routing dynamics, context length, rank and codebook size, error bars over seeds and topology preservation under the target projection. The main result is on Electricity at H = 720, where RAC-Net reaches an MSE of 0.189 against 0.246 for PatchTST and 0.225 for iTransformer, with 1.55M parameters and 0.87G MACs.

Critical strengths

The SLM runs once, offline, on 128 centroid prompts, and Table 6 reports 0.098 s for M = 128. Neither student training nor inference needs a forward pass through the SLM. By contrast, Section 2 notes that TimeKD and CALF run an active forward pass through the LLM during student training. The deployed model is small. Table 3 gives 0.50M parameters on ETTh1 and 1.55M on Electricity. Inference cost is one of the main practical obstacles for language-model-assisted forecasting, so this is a real advantage, and the design is simple to reproduce.

On Electricity at H = 720, RAC-Net reaches an MSE of 0.189, with a four-seed mean of 0.190 ± 0.002 in Table 11. The strongest listed baseline, TQNet, gets 0.201 in Table 12. RAC-Net also has the best Electricity average in Table 1, with an MSE of 0.161 and an MAE of 0.256. The paper includes no-distillation, MLP-statistics and random target vector controls and a four-seed analysis. In Table 11, the full model has the lowest or tied-lowest mean error among these variants on all three datasets. These results are the main empirical support for the approach.

Critical weaknesses

The effect of distillation is small. Over four seeds at H = 720, Table 11 gives full versus no-distillation MSE of 0.488 ± 0.003 vs 0.491 ± 0.007 on ETTh1, 0.334 ± 0.011 vs 0.339 ± 0.016 on Weather, and 0.190 ± 0.002 vs 0.192 ± 0.001 on Electricity. On Weather, the MSE-only variant reaches the same 0.334 as the full model. In the single-seed Table 2, the gains range from 0.49% to 2.23%, and on Traffic distillation raises the MSE from 0.532 to 0.546. On Electricity, the no-distillation model at 0.192 already accounts for about 95% of the drop from PatchTST's 0.246 to 0.189. The Electricity gain therefore comes mainly from the student model and the training objective, not from the teacher.

The teacher sees very little. Eq. 7 and Table 13 show that for each centroid the SLM receives only the trend class, the slope, the dominant period and the standard deviation. Both rationales in Table 13 simply restate these numbers. Each window's target is the cached vector of its nearest centroid, and each centroid is itself a window of length L. The paper does not show that this target carries the dataset-level regime dynamics beyond L that Section 1 gives as the motivation.

The alignment scores say little about what is distilled. Appendix G and Fig. 6 show that the cached targets are very similar to one another. Their pairwise cosine similarity lies in [0.65, 1.00], and after projection it stays roughly above 0.75 on ETTh1 and above 0.87 on Electricity. A CosSim of 0.96 or higher in Table 2 is therefore weak evidence that the SLM's knowledge reaches the student. The random target vector control uses isotropic random unit vectors with a very different geometry and reaches a CosSim of only 0.20 to 0.61, so it does not isolate the content of the SLM targets. No experiment removes the codebook and router, or the cross-channel attention.

The training objectives differ. Appendix A.3 states that baseline results are taken directly from the published literature, and their training losses are not given. RAC-Net is trained with MSE + 2.0 × MAE according to Table 5, and its results come from a single run with seed 0. Section 4.1 itself credits this objective for RAC-Net's MAE advantage. RAC-Net is in the top two for MAE on all seven datasets but best in MSE only on Electricity. On Traffic, its MSE of 0.468 is worse than SOFTs at 0.409, S-Mamba and RAFT at 0.414, iTransformer at 0.428 and TQNet at 0.445, while its MAE of 0.267 ties the best. Without a comparison under the same objective, the MAE ranking cannot be credited to the architecture.

The results depend on the seed. With seed 0, Table 7 puts RAC-Net ahead of PatchTST at L = 720, with 0.414 vs 0.420 on ETTh1 and 0.223 vs 0.224 on Weather. The four-seed means in Table 10 are 0.428 and 0.230, which reverses both comparisons. Appendix F calls the variance negligible, yet ETTh1 at L = 720, H = 720 gives 0.479 ± 0.021.

The MAC counts are lower than the stated architecture allows. Take the ETT configuration in Table 5, with P = 8 patches, C = 7, d_model = 64, d_ffn = 256 and two layers, and count in the standard way. The encoder FFN layers alone need about 3.67M MACs, more than the 2.80M total that Table 9 reports at H = 96. Adding the attention projections gives about 5.5M, more than the 5.04M total at H = 720. For Weather, with P = 21, C = 21, d_model = 128 and d_ffn = 512, the encoder linear layers alone need about 173M MACs, against 56.66M in Table 8. The paper reports neither the baseline configurations nor the counting tool. Some baseline entries also need explaining. PatchTST has 13.89M parameters on ETTh1 but 10.74M on Electricity, and 411.61G MACs for a single Electricity sample. Table 3 leaves out the most accurate baselines of Table 1, including TQNet, whose title names efficiency as its goal. Inference latencies of 2.9 ms and 13.67 ms are given only for RAC-Net, with no protocol and no baseline timings.

Efficiency is the paper's second main claim, so ratios such as 17.72× fewer MACs than LMP-Net cannot be relied on until the counting is sorted out.

Questions for authors

Which tool and operator list did you use to count MACs, and which configuration did you use for each baseline? Please show how the RAC-Net totals follow from the Table 5 configuration. The count should include all channels, both FFN projections, the attention projections and score products, the cross-channel attention and the forecast head.

For each baseline in Table 1, which paper do the numbers come from, and which loss was used? Please report the full RAC-Net with λ_MAE = 0, keeping the distillation and diversity terms, on all seven datasets and four horizons in the format of Table 1. The existing MSE-only variant also removes distillation, so it changes two things at once. Another option is to retrain at least PatchTST and TQNet with MSE + 2.0 × MAE.

Can Tables 1 and 7 report four-seed means for RAC-Net, as Table 10 does?

Please give the paired per-seed differences between the full and no-distillation models. Please also add controls that keep the target geometry, such as shuffling which target goes to which centroid, using a constant mean target, and using the prompt embedding without generation. Is the MLP in the MLP-statistics control trained or frozen, and what does κ mean among its inputs?

Appendix C reports that the soft probability distribution reaches a perplexity of at least 15.98 out of 16, so the routing probabilities averaged over the test set are close to uniform. Fig. 3, however, shows very uneven top-1 selection frequencies. These averages do not tell us how sharp the routing is for each sample. Please report the per-sample routing entropy or the mean maximum routing weight, and compare the routed perturbation with a uniform mix of the K adapters. This would show whether routing adds anything beyond a single averaged adapter.

4 comments
AuthorsOpenReview ↗

Response to Reviewer aLxi (Part 1)

Thank you very much for your encouraging assessment and constructive feedback on our submission. We have updated the manuscript to address your helpful suggestions. We truly appreciate the time and effort you invested in helping us make the paper even stronger.

Q1 Response: We thank the anonymous reviewer for their careful audit of our computational complexity numbers. We would like to clarify the mistake in the previous MACs calculation and provide the updated version. In previous simulations, we measured MACs using torch.utils.flop_counter. FlopCounterMode in evaluation mode. With PyTorch 2.13, evaluating nn.TransformerEncoder under torch.no_grad() dispatches execution to a fused C++ kernel (aten::_transformer_encoder_layer_fwd) whose internal matrix multiplications are not registered in FlopCounterMode and were counted as zero, leaving only the embedding, routing, combine-gate, and forecast-head MACs.

To correct this, we updated our profiling protocol by explicitly disabling fused fast-path kernels (torch.backends.mha.set_fastpath_enabled(False)) and utilising sdpa_kernel(SDPBackend.MATH) during profiling. FlopCounterMode now accurately captures every linear projection, attention score product (QK^⊤ and AV), FFN projection, cross-channel attention layer, codebook router/adapter, combine gate, and forecast head across all C channels.

As stated in Appendix A.3, cross-channel attention is implemented for large-scale multivariate datasets, and MACs are calculated on a per-sample multivariate forward pass (B=1, covering all channels C) and parametric matrix multiplications. We have updated Table 3, Table 9, Table 10, and Figure 2 with these corrected numbers.

Analytical MAC Verification ETTh1 Dataset (C=7,L=96) Using the Table 5 hyperparameters (P_len, Stride=12⟹P=8 patches, d_model=64, d_ffn=256, N=2 layers, K=16, r=4:

Patch Embedding: 7×8×(12×64)=0.043M  
Temporal Transformer Encoder (N=2)
Q,K,V,O projections: 7×2×8×(4×64^2 )=1.835M 
Attention score products (QK^⊤,AV): 7×2×(2×8^2×64)=0.115M
FFN projections: 7×2×8×(2×64×256)=3.670M
Codebook Router & Low-Rank Mixture Modulation: 7×(64⋅32+32⋅16+16⋅(2⋅64⋅4)=0.075M
Combine Gate: 7×8×(128×64)=0.459M
Forecast Head (H=720): 7×(512×720)=2.580M
Exact Total MACs for ETTh1: 8.77M/0.008G  (H=720).

Electricity (ECL) Dataset (C=321,L=96) Using the Table 5 hyperparameters (P_len=16, stride=4⟹P=21 patches, d_model=128, d_ffn=512, N=2, K=16 ,r=4 with cross-channel attention active):

Patch Embeddings: 321×21×(16×128)=0.014G  
Temporal Transformer Encoder (N=2,P=21)
Q,K,V,O projections: 321×2×21×(4×128^2 )=0.884G 
Attention scores (QK^⊤,AV): 321×2×(2×21^2×128)=0.072G 
FFN projections (128→512→128): 321×2×21×(2×128×512)=1.767G 
Cross-Channel Attention (P=21,C=321):
Q,K,V,O projections: 21×321×(4×128^2 )=0.442G 
Cross-channel attention scores (C×C): 21×(2×321^2×128)=0.554G  
Codebook Router & Low-Rank Modulation (K=16,r=4):( 321×(128×64+64×16+16×(2×128×4)=0.008G
Combine Gate (P=21): 321×21×(256×128)=0.221G
Two-Layer Forecast Head (P=21,H=720): 0.280G
Exact Total MACs for ECL: 4.24G

To provide a transparent comparison in Table 3, competing baselines’ numbers were not cited from external papers but were re-evaluated locally using the original study parameter configuration on our system under identical conditions. All baselines—including TQNet, Pathformer, Crossformer, PatchTST, iTransformer, and LMP-Net—were executed locally on the same NVIDIA GB10 GPU under a unified configuration (L=96,H=720,B=1) using their standard default hyperparameters from official repositories and the exact same FlopCounterMode counting rules.

Comparing ETTh1, and high-dimensional Electricity regimes highlights RAC-Net’s superior parameter-efficient scalability as variable cardinality increases. While channel-independent MLPs such as LMP-Net have better MSE on small 7-channel data (0.18M params, 0.472 MSE), scaling to 321 channels forces LMP-Net to expand its latent MLP capacity to 3.01M parameters while still suffering higher forecasting error (0.219 MSE) due to the absence of cross-variate modelling. In comparison, by sharing compact patch-codebook weights alongside cross-channel attention, RAC-Net scales to C = 321 with 48.5% fewer parameters (1.55M) and achieves a 13.70% relative reduction in MSE (0.189) over LMP-Net. Similarly, the TQNet report better MACs and parameter count on both datasets; however, RAC-Net outperforms TQNet in the forecasting task with a relative improvement of 7.35% in the high-dimensional dataset. Consequently, competing networks may offer isolated advantages in low-dimensional settings or raw footprint metrics; RAC-Net provides the most effective balance of parameter efficiency and predictive accuracy as data complexity scales.

AuthorsOpenReview ↗

Response to Reviewer aLxi (Part 2)

Q2 Response: We thank the reviewer for valuable feedback regarding baseline sources, task loss configurations, and fair comparison protocols. Baseline Sources and Task Losses in Table 1: As noted in our simulation setup (Appendix A.3), all baseline numbers reported in Table 1 are drawn directly from the official published literature where the data split and experimental setup align with our configurations. Regarding training objectives, all standard forecasting baselines-including GTR, S-Mamba, TQNet, RAFT, SOFTS, iTransformer, PatchTST, and DLinear-are optimised using standard point-wise Mean Squared Error (L2L_2) loss, whereas LMP-Net incorporates structural alignment and distillation loss alongside its primary forecasting MSE task loss.

Disentangling the Optimisation Objectives & Controlled Evaluation: To address the reviewer's concern that our composite task loss (L2L_2 + 2.0×L12.0 \times L_1 ) might introduce an unfair advantage, we performed a controlled re-evaluation across all 7 benchmark datasets (Appendix H, Table 14). We evaluated:

RAC-Net with L2L_2 (λMAE=0\lambda_{MAE} =0): Keeping our semantic distillation and diversity terms completely intact while removing the L1L_1 task loss component. Composite Loss Baselines (L2+2.0×L1L_2+ 2.0 \times L_1): Retraining primary baselines (LMP-Net and PatchTST) locally using the identical composite objective. Table 14 in Appendix H reports the network’s performance with L2L_2 and composite loss functions. Results demonstrate that with a L2L_2 loss, RAC-Net retains superior performance and competitive margins over baselines on high-dimensional datasets (achieving 0.167 and 0.465 in MSE vs. LMP-Net's 0.179 and 0.479, and PatchTST's 0.205 and 0.481 on Electricity and Traffic datasets, respectively). Moreover, when standard architectures (like PatchTST and LMP-Net) are trained with the composite objective function, optimisation stability degrades significantly, causing errors to rise sharply (e.g., PatchTST's ETTh1 MSE jumps from 0.469 to 0.519 and LMP-Net’s ETTh1 jumps from 0.430 to 0.461). In comparison, RAC-Net safely capitalises on joint pointwise and macro-structural regularisation. Because our student representation space is continuously anchored to smooth, SLM-grounded macro-regime centroids, it provides a robust geometric regulariser that supports effective joint optimisation without degrading forecasting performance.

Q3 Response: Thank you for the comment. In the revised article, we have updated Table 8 (previously Table 7, Appendix D) with 4-seed mean values. The updated multi-seed evaluations in Table 8 demonstrate that RAC-Net maintains consistent performance trends across all look-back windows. The MSE of RAC-Net for H = 720 drops from 0.488 to 0.479 (for ETTh1) and 0.334 to 0.321 (for Weather) by increasing the context window from 96 to 720. For the longer horizons, Weather (H=720), the four-seed performance lags behind PatchTST. However, the four-seed aggregation confirms that RAC-Net achieves competitive forecasting performance across random initialisation without severe performance degradation. Due to the inherently high-dimensional nature of our dataset, running a full 4-seed analysis across all Table 1 experiments represents an overwhelming computational burden that is unfortunately prohibitive within the short ICLR rebuttal window. However, on the representative dataset (ETTh1, Weather and Electricity), Tables 11, 12, and 13 report multi-seed results.

Q5 Response: Thank you for the valuable comment. To address this, we have computed the mean maximum routing weight across the test set and directly evaluated RAC-Net against a baseline utilising a static, uniform routing mechanism where all K=16 adapters are assigned a fixed, equal weight of (1/16=0.0625) (Appendix C, Table 7). The uniform routing baseline is trained separately with similar hyperparameter settings.

The per-sample maximum routing weights range from 0.072 to 0.096 (representing a 15% to 54% relative lift above the uniform (0.0625) baseline), confirming that the router operates in a calibrated soft-mixture regime, preserving high global codebook utilisation (preventing mode collapse) while applying consistent, input-dependent modulation rather than collapsing into a static uniform blend. Moreover, Table 7 demonstrates that RAC-Net consistently outperforms the uniform routing baseline across nearly all evaluation metrics on ETTh1, and delivers substantial gains in short-to-medium horizons on Weather (e.g., lowering Horizon 96 MSE to 0.153 compared to 0.161 for Uniform Routing). Moreover, on all 8 evaluations, RAC-Net outperforms uniform routing in MAE. Overall, these empirical comparisons indicate that input-dependent routing provides consistent MAE improvements and competitive MSE gains over a static uniform adapter average across most forecasting horizons.

AuthorsOpenReview ↗

Response to Reviewer aLxi (Part 3)

Q4 Response: Thank you for the comment.

  1. Paired Per-Seed Evaluation: Full RAC-Net vs No-Distillation For paired per-seed evaluation, we added Table 13 in Appendix F of the revised manuscript. The paired per-seed analysis demonstrates the RAC-Net’s consistent performance gains across the vast majority of initialisations, with minor fluctuations observed on individual seeds (e.g., a marginal MSE shift of (-0.41%) on ETTh1 for Seed 2025, and (-0.35%) on ECL for Seed 2024). This is due to standard random initialisation noise inherent to deep forecasting architectures. More importantly, even in these rare instances where MSE varies slightly, the MAE remains robustly stable and virtually identical to the baseline, confirming that distillation consistently preserves overall predictive stability while delivering a net positive benefit across the aggregate distribution of seeds.

  2. MLP statistics: The MLP control test is completely frozen and untrained. It functions as a static, randomly initialised projection mapping of statistical summaries onto the 32-dimensional target space without gradient updates during student training. The term κ denotes kurtosis, which measures distribution tailedness, heavy-tailed fluctuations, intermittent extreme values and transient anomaly behaviours within the look-back window. The frozen, randomly initialised MLP is intentionally not trained; it serves as a static projection baseline rather than a learnable model. Its purpose is to project statistical summaries (such as mean, variance, skewness, and kurtosis) into the 32-dimensional target space via a fixed, non-linear random kernel. Keeping the MLP frozen isolates the contribution of our proposed reasoning codebook and distillation framework. If the MLP were jointly trained, it would learn task-specific representations that conflate standard representation learning with our reasoning codebook routing mechanism, making it impossible to evaluate whether performance variation stems from the teacher's guidance or simply from additional trainable parameters. Thus, a fixed random projection provides a rigorous control baseline to demonstrate that structured LLM reasoning targets offer superior guidance over unstructured or random feature mappings.

  3. Shuffled-embedding test: In the revised manuscript, we implemented a shuffled embedding test (Section 4.2, Table 2). This test randomly permutes and shuffles teacher reasoning targets across centroids. We updated Table 2 with the shuffled embedding test. With shuffle embeddings, the RAC-Net performance degrades while the network reports high cosine similarity. This is due to the multi-objective function implemented in the RAC-Net architecture.

L_distil = 1.0 - (1 / (B · C)) · ∑_{i=1}^{B·C} [ (z_student^(i) · z_teacher^(i)) / (‖z_student^(i)‖₂ · ‖z_teacher^(i)‖₂) ]

The backpropagation minimises the cosine distance term; however, the forced alignment with shuffled embeddings causes forecasting performance degradation. We note one exception on the Weather benchmark, where shuffled embeddings achieve a lower MSE (0.328 vs. 0.350), though true semantic alignment still yields a better MAE (0.342 vs. 0.349). Because Weather data is dominated by smooth diurnal cycles, permuting target anchors acts as an implicit smoothing constraint that dampens outlier penalties (lowering MSE) at the cost of higher average step-wise error (worse MAE). In contrast, on high-dimensional datasets with distinct operating regimes, preserving true semantic correspondence between centroids and SLM rationales is essential for optimal accuracy. Across ETTh2 and Traffic datasets, true semantic distillation reports relative improvements of 1.88% and 1.97% compared to shuffled embeddings in MSE, respectively.

We thank the reviewer for these insightful control suggestions. Our current ablation suite already evaluates no-distillation, random embeddings, frozen MLP-statistics, and shuffled embeddings, which systematically isolate the effects of target presence, target geometry, and semantic alignment. Specifically, shuffled embeddings preserve the exact spectral and statistical geometry of the prompt-conditioned embedding space while removing semantic correspondence, whereas random embeddings and no-distillation establish the lower bound when structured target geometry is absent. Due to computational constraints across multi-dimensional benchmarks and the rebuttal discussion period, we focused our evaluation on these four primary controls and note constant-mean and prompt-only ablations as valuable extensions for future work.

AuthorsOpenReview ↗

Summary of Responses to Reviewer aLxi (For Reviewer aLxi and Area Chair)

Dear Area Chair and Reviewer aLxi,

First, we sincerely thank Reviewer aLxi for their thorough initial review and for highlighting key areas—such as MAC profiling verification, loss disentanglement, and target-geometry controls—that helped us significantly strengthen our experimental validation.

As the discussion period closes, we provide a brief summary for the Area Chair of the completed updates addressing Reviewer aLxi’s points in our three-part rebuttal posted on November 12, as the reviewer has not yet had the opportunity to follow up:

• Exact MACs & Unified Baseline Profiling (Tables 3, 9, 10 & Fig. 2): We identified and fixed the PyTorch fused-kernel (aten::_transformer_encoder_layer_fwd) profiling omission, providing exact analytical MAC breakdowns (8.77M on ETTh1; 4.24G on ECL) and unified GPU re-evaluations across TQNet, PatchTST, iTransformer, Crossformer, Pathformer, and LMP-Net.

• Loss Objective Disentanglement (Table 14, Appendix H): We evaluated MSE-trained RAC-Net (λMAE=0\lambda_{\text{MAE}}=0) across all 7 datasets, retrained baselines with MSE + 2⋅MAE and compared networks with the composite loss function and standard L2L_2 loss function, and reported the average values across four horizons as requested by the reviewer (Table 1 format).

• Semantic Target & Geometry Controls (Table 2, Table 13): We added shuffled-centroid target controls, clarified the frozen MLP-statistics control, and reported paired per-seed differences across 4 seeds.

• Multi-Seed Look-Back & Routing Sharpness (Tables 7 & 8): We updated Table 8 with 4-seed means across look-back windows and added a static uniform-routing baseline (Table 7), showing per-sample max routing weights achieve a 15%–54% lift over uniform (1/16) and outperform static uniform averaging.

We thank Reviewer aLxi again for their valuable technical feedback, and we kindly ask the Area Chair to take these completed experiments and revisions into account during the meta-review.

Reviewer_5DPSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RAC-Net, which uses cluster centroids to summarize temporal patterns in time series and queries a small language model offline using the centroids’ statistical features to generate and cache semantic representations. Through a cosine alignment loss, these representations provide auxiliary supervision for a lightweight patch-based Transformer forecaster equipped with a dynamic low-rank mixture codebook routing mechanism. During inference, only the trained forecaster is used, without requiring language model calls.

Critical strengths

The paper cleverly uses cluster centroids to summarize representative patterns in time series and, through knowledge distillation, employs semantic representations generated offline by a small language model as auxiliary supervision for a lightweight forecaster. This design introduces semantic information into time-series forecasting while avoiding the additional computational overhead of invoking a language model during inference, achieving a favorable balance between forecasting performance and computational efficiency.

Critical weaknesses
  1. The paper lacks direct comparisons with relevant time series forecasting methods that use language models. Moreover, its reported performance advantages are mainly reflected in MAE, while its MSE results are not the best on most datasets. Additional comparisons would help demonstrate the method’s forecasting performance more comprehensively. Nevertheless, I recognize the paper’s core methodological contribution and will not base my score solely on whether it achieves the best forecasting results.

  2. The paper uses semantic representations generated by a language model to provide auxiliary supervision. In my view, its main contribution depends to some extent on the LLM, yet the paper does not discuss the use of different LLMs. Such a discussion is important for clarifying the applicability of this mechanism, and I consider its absence a limitation of the current work.

  3. In the paper, some equations are not smoothly integrated with the surrounding text, and the punctuation used when introducing equations and providing subsequent explanations is not always consistent, making the logical connections in some passages difficult to follow. In addition, the spacing between some subsection headings, body text, and equations is rather tight, resulting in a somewhat cramped layout in places. Clearer transitions and explanations linking the text and equations would improve the paper’s readability.

Questions for authors

Q1. The paper uses the small language model Qwen2.5-0.5B to generate textual representations describing temporal patterns based on cluster centroids, and uses the corresponding hidden representations produced by the language model as auxiliary supervision. Is this method currently limited to Qwen models?

Q2. Table 5 in the appendix reports the values of the loss weights. Could the authors further explain the rationale behind these choices and whether the model’s forecasting performance is sensitive to these weights?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 5DPS

We are thankful to the anonymous reviewer for encouraging assessment and constructive feedback on our submission. We have updated the manuscript to address your helpful suggestions. We truly appreciate the time and effort you invested to enhance the manuscript.

Q1 Response: We thank the reviewer for this valuable question. RAC-Net is not restricted to Qwen and can accommodate different pre-trained SLMs as semantic teachers. To demonstrate this flexibility, we extended our experiments to four SLMs: Qwen2.5-0.5B, Llama-3.2-1B, Phi-3-mini-4k, and SmolLM2-360M. As shown in the additional Table 16 (Appendix I) included in the revised manuscript, RAC-Net achieves comparable forecasting performance across all four models on ETTh1, Weather, and Electricity. Notably, Llama-based RAC-Net achieves a 2.87% reduction in average MSE on Weather compared with Qwen (0.237 versus 0.244). These results demonstrate that RAC-Net's semantic distillation framework is not dependent on a specific SLM architecture.

Q2 Response: Thank you for the valuable comment. MAE loss: This serves as the primary task loss. Setting it higher than the other weights ensures that the backbone remains strictly anchored to pointwise forecasting precision, preventing the model from sacrificing reconstruction accuracy for secondary objectives.

Distillation loss: This aligns the network's latent features with the semantic rationales extracted from the Teacher SLM (Qwen-2.5-0.5B). For complex, high-dimensional datasets like Electricity and Traffic, we found that reducing this weight to 0.5 prevents the text-distillation constraint from over-regularising the backbone, allowing the model sufficient capacity to capture intricate physical temporal patterns.

Entropy loss: This acts as a diversity regulariser for our dynamic codebook routing module. It prevents routing collapse (where the router delegates all input windows to a trivial subset of the K=16 adapters), ensuring that all codebook entries are utilised dynamically.

In the updated manuscript, we added a comprehensive evaluation comparing RAC-Net against baseline techniques trained under varying loss functions (Appendix H, Table 14). The results indicate that removing the L_1 task loss component (λ_"MAE" =0) negatively affects the model's forecasting accuracy, confirming the necessity of joint pointwise (L_1) and structural (L_2) optimisation. Similarly, our paired per-seed evaluations with no semantic distillation (λ_"distill" =0) module demonstrate that across the vast majority of random initialisations, removing teacher guidance degrades the network's overall stability and predictive performance (Appendix F, Table 13).

Moreover, we thoroughly read the revised manuscript, removed typos, ensured consistent punctuation, proper spacing and equation formatting and explanation,

Reviewer_5DPSOpenReview ↗

Response to authors

I thank the authors for their detailed response. Most of my concerns have been addressed. However, I would still encourage the authors to include comparisons with one or two representative LLM-based forecasting methods (See weakness 1). Given that the proposed approach leverages LLMs, such comparisons are important for a comprehensive evaluation, particularly as large-scale pretraining may provide stronger transferable representations and generalization capabilities than lightweight time-series forecasting models.

AuthorsOpenReview ↗

Response to Reviewer 5DPS

Response: We are grateful to the reviewer for their valuable suggestions, and we apologise for not highlighting this comparison in greater detail in our previous response. In the revised manuscript, we have both discussed language-assisted models, LangTime, Time-FFM, and UniTime, in the Related Work (Section 2, Line 111-117) and included direct empirical comparisons against them in Appendix H (Table 15). All three baselines incorporate LLMs directly into their forecasting pipelines—specifically, LangTime leverages Qwen-0.5B-Instruct, while Time-FFM and UniTime utilise GPT-2.

As shown in Appendix H (Table 15) (reproduced below for convenience), RAC-Net achieves superior overall forecasting performance across the six benchmark datasets. While LangTime achieves a slightly lower MSE on the ETTh1 dataset, RAC-Net outperforms all three language-assisted baselines across the remaining datasets. Notably, on the high-dimensional Electricity dataset, RAC-Net achieves a 19.9% relative reduction in MSE compared to LangTime (0.161 vs. 0.201).

Excerpt from Appendix H (Table 15): Average MSE / MAE across prediction horizons H∈{96,192,336,720}H \in \{96, 192, 336, 720\} (lower is better; best results in bold).

DatasetRAC-Net (Ours)LangTimeTime-FFMUniTime
ETTh10.441 / 0.4310.435 / 0.4230.442 / 0.4340.440 / 0.448
ETTh20.369 / 0.3950.375 / 0.3910.382 / 0.4060.378 / 0.403
ETTm10.375 / 0.3840.397 / 0.3910.399 / 0.4020.385 / 0.399
ETTm20.281 / 0.3230.284 / 0.3210.286 / 0.3320.293 / 0.334
Weather0.244 / 0.2670.252 / 0.2700.270 / 0.2880.253 / 0.276
Electricity0.161 / 0.2560.201 / 0.2850.216 / 0.2990.216 / 0.305
Reviewer_sswkOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The article presents a patch-based multivariate time series prediction framework, which shows notable performance improvements through codebook routing and small language model distillation.

Critical strengths
  1. The distillation framework for small language models hasn't been deeply explored, and the direction the author looked into has some real-world significance.
Critical weaknesses
  1. Firstly, the proposal put forward in the article seems to offer only marginal gains in terms of performance. Especially the cover of the MSE, it seems to be worn out. This shows that the predictions involve a lot of extreme outcomes.

  2. The concepts used in the text seem a bit imprecise. A codebook is often used to describe the learnable codebook in VQ-VAE. This kind of description can cause a lot of misunderstanding.

  3. I looked at the image on the right in Figure 2. I noticed that the method proposed in this paper is largely behind LMP-Net. Why smaller datasets don't offer performance or efficiency benefits. This raises questions about the method's generalisability.

  4. The research on related work is insufficient, and some representative explorations of cross-architecture distillation (TimeDIstill KDD 2026) haven't been included in the discussion.

Questions for authors

See Weaknesses

9 comments
AuthorsOpenReview ↗

Response to Reviewer sswk

We appreciate the anonymous reviewer's thoughtful and constructive feedback. We have carefully considered the reviewer's insights and updated the manuscript to address the raised concerns. We truly appreciate the time and effort you took to help us strengthen this work.

Q1 Response: We thank the reviewer for valuable feedback regarding Mean Squared Error (MSE) and extreme outcomes. We like to clarify the performance of RAC-Net in these scenarios. RAC-Net employs a composite forecasting objective, MSE+ 2*MAE, alongside semantic distillation and codebook diversity regularisation. While MSE penalises large forecasting deviations quadratically, MAE applies a linear penalty to absolute errors. Their combination balances sensitivity to large deviations with overall pointwise forecasting accuracy. As reported in Table 1, RAC-Net achieves top-two MAE performance across all seven benchmark datasets. However, we acknowledge that these improvements do not always translate into equivalent reductions in MSE. We also recognise that aggregate MSE and MAE alone cannot establish whether a model produces unusually frequent extreme prediction errors. Although MSE improvements are modest on some datasets, RAC-Net demonstrates substantial gains on the high-dimensional Electricity benchmark. Averaged over four forecasting horizons, RAC-Net achieves an MSE of 0.161, compared with 0.179 for LMP-Net, 0.201 for LangTime, and 0.205 for PatchTST. These correspond to relative MSE reductions of 10.06%, 19.90%, and 21.46%, respectively. Furthermore, RAC-Net achieves an MSE of 0.244 on the Weather datasets, compared with 0.246 for LMP-Net. These findings demonstrate that the proposed framework can improve squared-error performance, although the magnitude of the improvement varies across datasets. To strengthen the experimental evaluation, we have extended the comparison to additional language-model-assisted forecasting approaches, including LangTime, Time-FFM, and UniTime, alongside the knowledge-distillation baseline LMP-Net (Appendix H, Table 15). The additional results provide further context for RAC-Net's performance across different benchmarks.

Q2 Response: We agree that this terminology is commonly associated with vector quantisation methods, such as VQ-VAE, and appreciate the opportunity to clarify the distinction. In RAC-Net, the Dynamic Low-Rank Mixture Codebook refers to a learnable bank of low-rank adapter transformations, rather than a conventional vector-quantisation codebook. Specifically, as defined in Section 3.4, each entry consists of two trainable low-rank matrices, A_k and B_k. Unlike VQ-VAE, which quantises continuous representations by selecting discrete embedding vectors, RAC-Net employs a softmax-based routing mechanism to compute input-dependent mixture weights over these adapters. To avoid conceptual confusion, we have revised the manuscript to explicitly define the codebook as a soft-routed low-rank adapter bank and distinguish it from conventional vector-quantisation codebooks (Section 3.4).

Q3 Response: We thank the reviewer for the valuable feedback. We believe the observed performance split between ETTh1 (C=7) and Electricity (C=321) does not indicate a flaw in generalisability; rather, it reflects a clear, expected architectural trade-off between local parameter memorisation and scalable cross-variate modelling. LMP-Net processes channels independently using large dense MLPs without cross-channel attention. On low-dimensional datasets like ETTh1, this acts as an effective localised regulariser with negligible cost. However, omitting cross-channel interactions degrades LMP-Net’s performance in high-dimensional spaces. On the Electricity benchmark, its parameter count scales to 3.01M, while its error rises to 0.219 MSE. RAC-Net addresses high-dimensional complexity by sharing a compact, regularised feature space across channels. On Electricity dataset, RAC-Net uses 48.5% fewer parameters than LMP-Net while reducing MSE by 13.7%. On ETTh1, RAC-Net maintains practical efficiency with nearly identical inference latency (0.17 ms vs 0.16 ms) and comparable MAE. Moreover, Table 1’s averaged four-horizon results highlight that RAC-Net achieves better MAE than LMP-Net even on low-dimensional ETT datasets. While RAC-Net does not outperform LMP-Net on every metric across all datasets, nevertheless, it offers competitive performance on low-dimensional benchmarks and performance scalability on the higher-dimensional Electricity benchmark.

Q4 Response: We agree that recent studies on cross-architecture knowledge distillation are relevant to our work. Accordingly, we have expanded Section 2 (Related Work) to include TimeDistill (2026), LangTime (2025), Time-FFM (2024), and UniTime (2024). We have also clarified how RAC-Net differs from these approaches, particularly through its offline SLM-based semantic distillation and lightweight dynamic codebook routing mechanism (Section 2, Lines 111-117 and 135-145).

Thanks to the authors for their effort and timely responses.

I think the author’s responses to weaknesses 2 and 4 are sufficient; these two weaknesses have been addressed.

However, weaknesses 1 and 3 haven't been effectively dealt with.

First, for Weakness 1, the improvement only appears in a few datasets. I also checked Table 15, and except for the ECL dataset, the other datasets still show only marginal improvements. And there are no metrics like standard deviation to prove the confidence in the slight improvement.

Also, regarding weakness 2, I think the author's response still doesn't really address my concerns. Improving a single metric can't be taken as an effective measure of performance improvement, and there’s no confidence analysis. From the current results, I think the method proposed in this paper shows decent MAE, while the MSE stays at a relatively low level, which raises huge challenges and doubts about the relevance and predictability of the prediction results of the proposed model.

Also, the paper's revisions don't have any markings, which makes it really hard for me to compare with the previous draft and re-evaluate.

All in all, based on these considerations, I’ve decided to stick with my previous score.

AuthorsOpenReview ↗

Response to Reviewer sswk

We sincerely thank the reviewer for their follow-up and for confirming that weaknesses 2 and 4 have been resolved. We apologise for not highlighting the changes in the revised PDF—we have now uploaded an updated manuscript with all revisions clearly marked in blue for easy verification. Regarding your remaining concerns on statistical confidence (Mean ± Std), MSE vs. MAE behaviour, and cross-dataset generalisability (Weaknesses 1 & 3), we would like to highlight key multi-seed statistical analyses already included in the appendix alongside additional clarifications:

  1. Multi-Seed Confidence & Statistical Stability (Appendix F, Tables 11–13)

To verify that RAC-Net's gains are statistically reliable rather than random seed variance, we conducted 4-seed evaluations �0��0� across multiple datasets and horizons in Appendix F (Pages 23–24): Low Variance Across Horizons (Table 11): On ETTh1 (L=96)(L=96), RAC-Net achieves a tight 4-seed average of (0.444±0.002)(0.444 \pm 0.002) MSE and �1��1� MAE. On Weather (L=96)(L=96), it achieves �2��2� MSE and $(0.270 \pm 0.003)$ MAE.

Consistent Distillation Gains Over Ablations (Table 12): At �3��3�, full RAC-Net outperforms the no-distillation baseline (�4�)(�4�) and statistical/random controls with substantially lower variance—for instance, on ETTh1, RAC-Net achieves �5��5� MSE vs. �6��6� (MLP-Statistics) and �7��7� (Random Vectors).

Paired Per-Seed Consistency (Table 13): Paired seed-by-seed comparisons show that semantic distillation improves MSE on 10 out of 12 seed-dataset pairs across ETTh1, Weather, and Electricity (+0.81%, +1.53%, and +0.95% mean relative MSE gains, respectively).

  1. MSE Predictability & Absence of Extreme Outliers (Tables 1, 14, and 17): We respectfully clarify that RAC-Net does not sacrifice MSE or suffer from extreme prediction outliers.

MSE penalises errors quadratically, a model prone to extreme outliers would exhibit severe MSE spikes compared to baselines. Instead, while no single baseline achieves a clean sweep across every dataset due to varying temporal and covariate dynamics, RAC-Net ranks 1st overall in combined top-2 finishes across all 7 benchmarks in Table 1 (10 top-2 placements: 3 in MSE, 7 in MAE)—ahead of GTR (7: 3 in MSE, 4 in MAE), LMP-Net (4: 2 in MSE, 2 in MAE), and TQNet (4: 1 in MSE, 3 in MAE).

Per-Horizon MSE Wins Beyond Electricity (Table 17, Page 27): Looking at individual horizons (L=96)(L=96), RAC-Net achieves lower MSE than LMP-Net across five of the seven datasets—including ETTm1 (H=96)(H=96): 0.307vs.0.3170.307 vs. 0.317; (H=336)(H=336): 0.385vs.0.3880.385 vs. 0.388; (H=720)(H=720): 0.445vs.0.4510.445 vs. 0.451, ETTm2 (H=96)(H=96): 0.175vs.0.1780.175 vs. 0.178, Weather (H=96):(H=96): 0.153vs.0.1630.153 vs. 0.163; (H=192)(H=192): 0.204vs.0.2100.204 vs. 0.210, Electricity (all four horizons), and Traffic (H=96,192,336)(H=96, 192, 336)—while outperforming LMP-Net in MAE on 24 out of 28 settings. Furthermore, RAC-Net outperforms standard Transformer baselines such as PatchTST and iTransformer in MSE on multiple dataset-horizon pairs, confirming consistent squared-error accuracy rather than isolated gains.

Pure (L2)(L_{2}) Ablation (Table 14, Page 25): When baseline models are trained with the exact same joint optimisation objective (L2+2L1)(L_2 + 2L_1), their MSE degrades significantly across datasets (e.g., LMP-Net worsens from (0.430→0.461)(0.430 \rightarrow 0.461) on ETTh1 and (0.179→0.195)(0.179 \rightarrow 0.195) on Electricity; PatchTST worsens from (0.469→0.519)(0.469 \rightarrow 0.519) on ETTh1 and (0.205→0.232)(0.205 \rightarrow 0.232) on Electricity. In contrast, RAC-Net maintains strong, competitive MSE under the same joint objective—confirming that our semantic distillation and codebook offer forecasts without inducing extreme prediction outliers.

  1. Generalisability Across Small and Large Datasets (Weakness 3): Rather than failing on smaller datasets, RAC-Net provides a unified, scalable architecture across covariate scales (C=7)(C=7) to (C=862)(C=862):

On low-dimensional datasets (ETTh1, ETTh2, ETTm1, ETTm2, (C=7)(C=7)), RAC-Net matches LMP-Net's sub-millisecond latency (0.17 ms vs. 0.16 ms, Table 3) while achieving better 4-horizon average MAE on all four ETT datasets (Table 1) and better MSE on ETTm1 (0.375 vs. 0.379).

On high-dimensional datasets (Electricity, (C=321)(C=321), where channel-independent MLPs like LMP-Net bloat to 3.01M parameters and degrade in accuracy, RAC-Net cuts parameters by 48.5% (1.55M) and improves MSE by 10.1% on average (0.161 vs. 0.179) and 13.7% at (H=720)(H=720) (0.189 vs. 0.219).

We hope that highlighting these multi-seed confidence intervals (Tables 11–13), per-horizon MSE results (Table 17), and the blue-highlighted revisions in the updated PDF resolve your remaining concerns. We would be grateful if you would reconsider your evaluation in light of these clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.