- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes RAC-Net, a knowledge distillation framework for multivariate long-term time-series forecasting in which a small language model (SLM) is used only offline. Standardised training windows of length L = 96 from all channels are clustered with MiniBatch k-means into M = 128 centroids. For each centroid, a frozen Qwen-2.5-0.5B receives a prompt with the trend class, the least-squares trend slope, the dominant FFT period and the standard deviation, and generates up to 48 rationale tokens with greedy decoding. The top-layer hidden states are mean-pooled, mapped to 32 dimensions by a non-trainable orthogonal projection, normalised onto the unit hypersphere and cached. During training, the teacher target for each window is the cached vector of its nearest centroid, as defined in Eq. 13.
The student is a patch-based Transformer with channel embeddings and cross-channel attention at each temporal patch position. It adds a dynamic low-rank mixture codebook, in which a router over K = 16 rank-4 adapters produces a low-rank perturbation of the pooled representation. This perturbation is expanded across patches and added through a non-linear gated residual. The loss is MSE + 2.0 × MAE plus a directional cosine alignment term and an entropy-based routing diversity term, as given in Eqs. 26 to 30 and Table 5. The SLM is not used during student training or inference.
The experiments use seven benchmarks, ETTh1, ETTh2, ETTm1, ETTm2, Weather, Electricity and Traffic, with L = 96 and H ∈ {96, 192, 336, 720}. Tables 1 and 12 compare RAC-Net with nine baselines, using numbers taken from the published papers. Table 2 ablates distillation at H = 720 with no-distillation, MSE-only, MLP-statistics and random target vector variants, and Table 11 repeats this over four seeds. Table 3 and Fig. 2 compare parameters and MACs on Electricity and ETTh1. The appendices cover the offline centroid budget, codebook routing dynamics, context length, rank and codebook size, error bars over seeds and topology preservation under the target projection. The main result is on Electricity at H = 720, where RAC-Net reaches an MSE of 0.189 against 0.246 for PatchTST and 0.225 for iTransformer, with 1.55M parameters and 0.87G MACs.
Critical strengths
The SLM runs once, offline, on 128 centroid prompts, and Table 6 reports 0.098 s for M = 128. Neither student training nor inference needs a forward pass through the SLM. By contrast, Section 2 notes that TimeKD and CALF run an active forward pass through the LLM during student training. The deployed model is small. Table 3 gives 0.50M parameters on ETTh1 and 1.55M on Electricity. Inference cost is one of the main practical obstacles for language-model-assisted forecasting, so this is a real advantage, and the design is simple to reproduce.
On Electricity at H = 720, RAC-Net reaches an MSE of 0.189, with a four-seed mean of 0.190 ± 0.002 in Table 11. The strongest listed baseline, TQNet, gets 0.201 in Table 12. RAC-Net also has the best Electricity average in Table 1, with an MSE of 0.161 and an MAE of 0.256. The paper includes no-distillation, MLP-statistics and random target vector controls and a four-seed analysis. In Table 11, the full model has the lowest or tied-lowest mean error among these variants on all three datasets. These results are the main empirical support for the approach.
Critical weaknesses
The effect of distillation is small. Over four seeds at H = 720, Table 11 gives full versus no-distillation MSE of 0.488 ± 0.003 vs 0.491 ± 0.007 on ETTh1, 0.334 ± 0.011 vs 0.339 ± 0.016 on Weather, and 0.190 ± 0.002 vs 0.192 ± 0.001 on Electricity. On Weather, the MSE-only variant reaches the same 0.334 as the full model. In the single-seed Table 2, the gains range from 0.49% to 2.23%, and on Traffic distillation raises the MSE from 0.532 to 0.546. On Electricity, the no-distillation model at 0.192 already accounts for about 95% of the drop from PatchTST's 0.246 to 0.189. The Electricity gain therefore comes mainly from the student model and the training objective, not from the teacher.
The teacher sees very little. Eq. 7 and Table 13 show that for each centroid the SLM receives only the trend class, the slope, the dominant period and the standard deviation. Both rationales in Table 13 simply restate these numbers. Each window's target is the cached vector of its nearest centroid, and each centroid is itself a window of length L. The paper does not show that this target carries the dataset-level regime dynamics beyond L that Section 1 gives as the motivation.
The alignment scores say little about what is distilled. Appendix G and Fig. 6 show that the cached targets are very similar to one another. Their pairwise cosine similarity lies in [0.65, 1.00], and after projection it stays roughly above 0.75 on ETTh1 and above 0.87 on Electricity. A CosSim of 0.96 or higher in Table 2 is therefore weak evidence that the SLM's knowledge reaches the student. The random target vector control uses isotropic random unit vectors with a very different geometry and reaches a CosSim of only 0.20 to 0.61, so it does not isolate the content of the SLM targets. No experiment removes the codebook and router, or the cross-channel attention.
The training objectives differ. Appendix A.3 states that baseline results are taken directly from the published literature, and their training losses are not given. RAC-Net is trained with MSE + 2.0 × MAE according to Table 5, and its results come from a single run with seed 0. Section 4.1 itself credits this objective for RAC-Net's MAE advantage. RAC-Net is in the top two for MAE on all seven datasets but best in MSE only on Electricity. On Traffic, its MSE of 0.468 is worse than SOFTs at 0.409, S-Mamba and RAFT at 0.414, iTransformer at 0.428 and TQNet at 0.445, while its MAE of 0.267 ties the best. Without a comparison under the same objective, the MAE ranking cannot be credited to the architecture.
The results depend on the seed. With seed 0, Table 7 puts RAC-Net ahead of PatchTST at L = 720, with 0.414 vs 0.420 on ETTh1 and 0.223 vs 0.224 on Weather. The four-seed means in Table 10 are 0.428 and 0.230, which reverses both comparisons. Appendix F calls the variance negligible, yet ETTh1 at L = 720, H = 720 gives 0.479 ± 0.021.
The MAC counts are lower than the stated architecture allows. Take the ETT configuration in Table 5, with P = 8 patches, C = 7, d_model = 64, d_ffn = 256 and two layers, and count in the standard way. The encoder FFN layers alone need about 3.67M MACs, more than the 2.80M total that Table 9 reports at H = 96. Adding the attention projections gives about 5.5M, more than the 5.04M total at H = 720. For Weather, with P = 21, C = 21, d_model = 128 and d_ffn = 512, the encoder linear layers alone need about 173M MACs, against 56.66M in Table 8. The paper reports neither the baseline configurations nor the counting tool. Some baseline entries also need explaining. PatchTST has 13.89M parameters on ETTh1 but 10.74M on Electricity, and 411.61G MACs for a single Electricity sample. Table 3 leaves out the most accurate baselines of Table 1, including TQNet, whose title names efficiency as its goal. Inference latencies of 2.9 ms and 13.67 ms are given only for RAC-Net, with no protocol and no baseline timings.
Efficiency is the paper's second main claim, so ratios such as 17.72× fewer MACs than LMP-Net cannot be relied on until the counting is sorted out.
Questions for authors
Which tool and operator list did you use to count MACs, and which configuration did you use for each baseline? Please show how the RAC-Net totals follow from the Table 5 configuration. The count should include all channels, both FFN projections, the attention projections and score products, the cross-channel attention and the forecast head.
For each baseline in Table 1, which paper do the numbers come from, and which loss was used? Please report the full RAC-Net with λ_MAE = 0, keeping the distillation and diversity terms, on all seven datasets and four horizons in the format of Table 1. The existing MSE-only variant also removes distillation, so it changes two things at once. Another option is to retrain at least PatchTST and TQNet with MSE + 2.0 × MAE.
Can Tables 1 and 7 report four-seed means for RAC-Net, as Table 10 does?
Please give the paired per-seed differences between the full and no-distillation models. Please also add controls that keep the target geometry, such as shuffling which target goes to which centroid, using a constant mean target, and using the prompt embedding without generation. Is the MLP in the MLP-statistics control trained or frozen, and what does κ mean among its inputs?
Appendix C reports that the soft probability distribution reaches a perplexity of at least 15.98 out of 16, so the routing probabilities averaged over the test set are close to uniform. Fig. 3, however, shows very uneven top-1 selection frequencies. These averages do not tell us how sharp the routing is for each sample. Please report the per-sample routing entropy or the mean maximum routing weight, and compare the routed perturbation with a uniform mix of the K adapters. This would show whether routing adds anything beyond a single averaged adapter.
4 comments
Response to Reviewer aLxi (Part 2)
Q2 Response: We thank the reviewer for valuable feedback regarding baseline sources, task loss configurations, and fair comparison protocols. Baseline Sources and Task Losses in Table 1: As noted in our simulation setup (Appendix A.3), all baseline numbers reported in Table 1 are drawn directly from the official published literature where the data split and experimental setup align with our configurations. Regarding training objectives, all standard forecasting baselines-including GTR, S-Mamba, TQNet, RAFT, SOFTS, iTransformer, PatchTST, and DLinear-are optimised using standard point-wise Mean Squared Error () loss, whereas LMP-Net incorporates structural alignment and distillation loss alongside its primary forecasting MSE task loss.
Disentangling the Optimisation Objectives & Controlled Evaluation: To address the reviewer's concern that our composite task loss ( + ) might introduce an unfair advantage, we performed a controlled re-evaluation across all 7 benchmark datasets (Appendix H, Table 14). We evaluated:
RAC-Net with (): Keeping our semantic distillation and diversity terms completely intact while removing the task loss component. Composite Loss Baselines (): Retraining primary baselines (LMP-Net and PatchTST) locally using the identical composite objective. Table 14 in Appendix H reports the network’s performance with and composite loss functions. Results demonstrate that with a loss, RAC-Net retains superior performance and competitive margins over baselines on high-dimensional datasets (achieving 0.167 and 0.465 in MSE vs. LMP-Net's 0.179 and 0.479, and PatchTST's 0.205 and 0.481 on Electricity and Traffic datasets, respectively). Moreover, when standard architectures (like PatchTST and LMP-Net) are trained with the composite objective function, optimisation stability degrades significantly, causing errors to rise sharply (e.g., PatchTST's ETTh1 MSE jumps from 0.469 to 0.519 and LMP-Net’s ETTh1 jumps from 0.430 to 0.461). In comparison, RAC-Net safely capitalises on joint pointwise and macro-structural regularisation. Because our student representation space is continuously anchored to smooth, SLM-grounded macro-regime centroids, it provides a robust geometric regulariser that supports effective joint optimisation without degrading forecasting performance.
Q3 Response: Thank you for the comment. In the revised article, we have updated Table 8 (previously Table 7, Appendix D) with 4-seed mean values. The updated multi-seed evaluations in Table 8 demonstrate that RAC-Net maintains consistent performance trends across all look-back windows. The MSE of RAC-Net for H = 720 drops from 0.488 to 0.479 (for ETTh1) and 0.334 to 0.321 (for Weather) by increasing the context window from 96 to 720. For the longer horizons, Weather (H=720), the four-seed performance lags behind PatchTST. However, the four-seed aggregation confirms that RAC-Net achieves competitive forecasting performance across random initialisation without severe performance degradation. Due to the inherently high-dimensional nature of our dataset, running a full 4-seed analysis across all Table 1 experiments represents an overwhelming computational burden that is unfortunately prohibitive within the short ICLR rebuttal window. However, on the representative dataset (ETTh1, Weather and Electricity), Tables 11, 12, and 13 report multi-seed results.
Q5 Response: Thank you for the valuable comment. To address this, we have computed the mean maximum routing weight across the test set and directly evaluated RAC-Net against a baseline utilising a static, uniform routing mechanism where all K=16 adapters are assigned a fixed, equal weight of (1/16=0.0625) (Appendix C, Table 7). The uniform routing baseline is trained separately with similar hyperparameter settings.
The per-sample maximum routing weights range from 0.072 to 0.096 (representing a 15% to 54% relative lift above the uniform (0.0625) baseline), confirming that the router operates in a calibrated soft-mixture regime, preserving high global codebook utilisation (preventing mode collapse) while applying consistent, input-dependent modulation rather than collapsing into a static uniform blend. Moreover, Table 7 demonstrates that RAC-Net consistently outperforms the uniform routing baseline across nearly all evaluation metrics on ETTh1, and delivers substantial gains in short-to-medium horizons on Weather (e.g., lowering Horizon 96 MSE to 0.153 compared to 0.161 for Uniform Routing). Moreover, on all 8 evaluations, RAC-Net outperforms uniform routing in MAE. Overall, these empirical comparisons indicate that input-dependent routing provides consistent MAE improvements and competitive MSE gains over a static uniform adapter average across most forecasting horizons.
Response to Reviewer aLxi (Part 3)
Q4 Response: Thank you for the comment.
-
Paired Per-Seed Evaluation: Full RAC-Net vs No-Distillation For paired per-seed evaluation, we added Table 13 in Appendix F of the revised manuscript. The paired per-seed analysis demonstrates the RAC-Net’s consistent performance gains across the vast majority of initialisations, with minor fluctuations observed on individual seeds (e.g., a marginal MSE shift of (-0.41%) on ETTh1 for Seed 2025, and (-0.35%) on ECL for Seed 2024). This is due to standard random initialisation noise inherent to deep forecasting architectures. More importantly, even in these rare instances where MSE varies slightly, the MAE remains robustly stable and virtually identical to the baseline, confirming that distillation consistently preserves overall predictive stability while delivering a net positive benefit across the aggregate distribution of seeds.
-
MLP statistics: The MLP control test is completely frozen and untrained. It functions as a static, randomly initialised projection mapping of statistical summaries onto the 32-dimensional target space without gradient updates during student training. The term κ denotes kurtosis, which measures distribution tailedness, heavy-tailed fluctuations, intermittent extreme values and transient anomaly behaviours within the look-back window. The frozen, randomly initialised MLP is intentionally not trained; it serves as a static projection baseline rather than a learnable model. Its purpose is to project statistical summaries (such as mean, variance, skewness, and kurtosis) into the 32-dimensional target space via a fixed, non-linear random kernel. Keeping the MLP frozen isolates the contribution of our proposed reasoning codebook and distillation framework. If the MLP were jointly trained, it would learn task-specific representations that conflate standard representation learning with our reasoning codebook routing mechanism, making it impossible to evaluate whether performance variation stems from the teacher's guidance or simply from additional trainable parameters. Thus, a fixed random projection provides a rigorous control baseline to demonstrate that structured LLM reasoning targets offer superior guidance over unstructured or random feature mappings.
-
Shuffled-embedding test: In the revised manuscript, we implemented a shuffled embedding test (Section 4.2, Table 2). This test randomly permutes and shuffles teacher reasoning targets across centroids. We updated Table 2 with the shuffled embedding test. With shuffle embeddings, the RAC-Net performance degrades while the network reports high cosine similarity. This is due to the multi-objective function implemented in the RAC-Net architecture.
L_distil = 1.0 - (1 / (B · C)) · ∑_{i=1}^{B·C} [ (z_student^(i) · z_teacher^(i)) / (‖z_student^(i)‖₂ · ‖z_teacher^(i)‖₂) ]
The backpropagation minimises the cosine distance term; however, the forced alignment with shuffled embeddings causes forecasting performance degradation. We note one exception on the Weather benchmark, where shuffled embeddings achieve a lower MSE (0.328 vs. 0.350), though true semantic alignment still yields a better MAE (0.342 vs. 0.349). Because Weather data is dominated by smooth diurnal cycles, permuting target anchors acts as an implicit smoothing constraint that dampens outlier penalties (lowering MSE) at the cost of higher average step-wise error (worse MAE). In contrast, on high-dimensional datasets with distinct operating regimes, preserving true semantic correspondence between centroids and SLM rationales is essential for optimal accuracy. Across ETTh2 and Traffic datasets, true semantic distillation reports relative improvements of 1.88% and 1.97% compared to shuffled embeddings in MSE, respectively.
We thank the reviewer for these insightful control suggestions. Our current ablation suite already evaluates no-distillation, random embeddings, frozen MLP-statistics, and shuffled embeddings, which systematically isolate the effects of target presence, target geometry, and semantic alignment. Specifically, shuffled embeddings preserve the exact spectral and statistical geometry of the prompt-conditioned embedding space while removing semantic correspondence, whereas random embeddings and no-distillation establish the lower bound when structured target geometry is absent. Due to computational constraints across multi-dimensional benchmarks and the rebuttal discussion period, we focused our evaluation on these four primary controls and note constant-mean and prompt-only ablations as valuable extensions for future work.
Summary of Responses to Reviewer aLxi (For Reviewer aLxi and Area Chair)
Dear Area Chair and Reviewer aLxi,
First, we sincerely thank Reviewer aLxi for their thorough initial review and for highlighting key areas—such as MAC profiling verification, loss disentanglement, and target-geometry controls—that helped us significantly strengthen our experimental validation.
As the discussion period closes, we provide a brief summary for the Area Chair of the completed updates addressing Reviewer aLxi’s points in our three-part rebuttal posted on November 12, as the reviewer has not yet had the opportunity to follow up:
• Exact MACs & Unified Baseline Profiling (Tables 3, 9, 10 & Fig. 2): We identified and fixed the PyTorch fused-kernel (aten::_transformer_encoder_layer_fwd) profiling omission, providing exact analytical MAC breakdowns (8.77M on ETTh1; 4.24G on ECL) and unified GPU re-evaluations across TQNet, PatchTST, iTransformer, Crossformer, Pathformer, and LMP-Net.
• Loss Objective Disentanglement (Table 14, Appendix H): We evaluated MSE-trained RAC-Net () across all 7 datasets, retrained baselines with MSE + 2⋅MAE and compared networks with the composite loss function and standard loss function, and reported the average values across four horizons as requested by the reviewer (Table 1 format).
• Semantic Target & Geometry Controls (Table 2, Table 13): We added shuffled-centroid target controls, clarified the frozen MLP-statistics control, and reported paired per-seed differences across 4 seeds.
• Multi-Seed Look-Back & Routing Sharpness (Tables 7 & 8): We updated Table 8 with 4-seed means across look-back windows and added a static uniform-routing baseline (Table 7), showing per-sample max routing weights achieve a 15%–54% lift over uniform (1/16) and outperform static uniform averaging.
We thank Reviewer aLxi again for their valuable technical feedback, and we kindly ask the Area Chair to take these completed experiments and revisions into account during the meta-review.
Response to Reviewer aLxi (Part 1)
Thank you very much for your encouraging assessment and constructive feedback on our submission. We have updated the manuscript to address your helpful suggestions. We truly appreciate the time and effort you invested in helping us make the paper even stronger.
Q1 Response: We thank the anonymous reviewer for their careful audit of our computational complexity numbers. We would like to clarify the mistake in the previous MACs calculation and provide the updated version. In previous simulations, we measured MACs using torch.utils.flop_counter. FlopCounterMode in evaluation mode. With PyTorch 2.13, evaluating nn.TransformerEncoder under torch.no_grad() dispatches execution to a fused C++ kernel (aten::_transformer_encoder_layer_fwd) whose internal matrix multiplications are not registered in FlopCounterMode and were counted as zero, leaving only the embedding, routing, combine-gate, and forecast-head MACs.
To correct this, we updated our profiling protocol by explicitly disabling fused fast-path kernels (torch.backends.mha.set_fastpath_enabled(False)) and utilising sdpa_kernel(SDPBackend.MATH) during profiling. FlopCounterMode now accurately captures every linear projection, attention score product (QK^⊤ and AV), FFN projection, cross-channel attention layer, codebook router/adapter, combine gate, and forecast head across all C channels.
As stated in Appendix A.3, cross-channel attention is implemented for large-scale multivariate datasets, and MACs are calculated on a per-sample multivariate forward pass (B=1, covering all channels C) and parametric matrix multiplications. We have updated Table 3, Table 9, Table 10, and Figure 2 with these corrected numbers.
Analytical MAC Verification ETTh1 Dataset (C=7,L=96) Using the Table 5 hyperparameters (P_len, Stride=12⟹P=8 patches, d_model=64, d_ffn=256, N=2 layers, K=16, r=4:
Electricity (ECL) Dataset (C=321,L=96) Using the Table 5 hyperparameters (P_len=16, stride=4⟹P=21 patches, d_model=128, d_ffn=512, N=2, K=16 ,r=4 with cross-channel attention active):
To provide a transparent comparison in Table 3, competing baselines’ numbers were not cited from external papers but were re-evaluated locally using the original study parameter configuration on our system under identical conditions. All baselines—including TQNet, Pathformer, Crossformer, PatchTST, iTransformer, and LMP-Net—were executed locally on the same NVIDIA GB10 GPU under a unified configuration (L=96,H=720,B=1) using their standard default hyperparameters from official repositories and the exact same FlopCounterMode counting rules.
Comparing ETTh1, and high-dimensional Electricity regimes highlights RAC-Net’s superior parameter-efficient scalability as variable cardinality increases. While channel-independent MLPs such as LMP-Net have better MSE on small 7-channel data (0.18M params, 0.472 MSE), scaling to 321 channels forces LMP-Net to expand its latent MLP capacity to 3.01M parameters while still suffering higher forecasting error (0.219 MSE) due to the absence of cross-variate modelling. In comparison, by sharing compact patch-codebook weights alongside cross-channel attention, RAC-Net scales to C = 321 with 48.5% fewer parameters (1.55M) and achieves a 13.70% relative reduction in MSE (0.189) over LMP-Net. Similarly, the TQNet report better MACs and parameter count on both datasets; however, RAC-Net outperforms TQNet in the forecasting task with a relative improvement of 7.35% in the high-dimensional dataset. Consequently, competing networks may offer isolated advantages in low-dimensional settings or raw footprint metrics; RAC-Net provides the most effective balance of parameter efficiency and predictive accuracy as data complexity scales.